diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 05fd66c..93210da 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -11,12 +11,17 @@ jobs: runs-on: macos-14 steps: - uses: actions/checkout@v4 + with: + fetch-depth: 0 # validate_records.py checks that every recorded commit exists - uses: actions/setup-python@v5 with: python-version: "3.12" - name: Install project - run: python -m pip install -e ".[dev]" - - name: Lint - run: ruff check . - - name: Test - run: pytest -q + run: python -m pip install -r requirements/release-lock.txt && python -m pip install -e . --no-deps + - name: Hardware report + # Hosted macOS arm64 runners report MPS as available but cannot allocate on it; + # MPS tests are then skipped with this reason (run them on Apple Silicon with + # MINILM_REQUIRE_MPS=1). CPU correctness tests always run. + run: python tests/hardware.py + - name: Release verification (lint, tests, strict record validation, report drift) + run: ./scripts/verify_release.sh diff --git a/README.md b/README.md index 4d5501a..9a04115 100644 --- a/README.md +++ b/README.md @@ -1,84 +1,91 @@ # MiniLM Lab -**Build a tiny language model. Measure what actually helps.** +A small decoder-only Transformer, trained on byte-level science prose, used as a laboratory for measuring how implementation choices affect quality, latency, throughput and memory. Every reported number is recomputed from a validated record in [`experiments/records/`](experiments/records/). -Created and maintained by [Anvit Devadiga](https://github.com/AnvitDevadiga). +## Architecture -[![CI](https://github.com/AnvitDevadiga/MiniLM-Lab/actions/workflows/ci.yml/badge.svg)](https://github.com/AnvitDevadiga/MiniLM-Lab/actions/workflows/ci.yml) [![Python 3.12](https://img.shields.io/badge/Python-3.12-111111)](https://www.python.org/) [![MIT](https://img.shields.io/badge/License-MIT-111111)](LICENSE) +![MiniLM Lab architecture](docs/figures/architecture-light.svg#gh-light-mode-only) +![MiniLM Lab architecture](docs/figures/architecture-dark.svg#gh-dark-mode-only) -MiniLM Lab is a small, inspectable language-model project. It follows the entire path from text to next-byte prediction, then tests whether engineering ideas—like reusing previous attention calculations or storing smaller weights—actually help. It is designed for a laptop budget, not to imitate a commercial chatbot. +Defaults: d = 192, 4 layers, 6 heads, context 128, vocabulary 256 (UTF-8 bytes). Options: RoPE instead of learned positions, LayerNorm instead of RMSNorm, GELU MLP instead of SwiGLU. -![MiniLM Lab architecture](docs/figures/architecture.svg) +## Key Results -### What did the experiment find? +Apple M4 (16 GB), CPU and MPS, Python 3.12.14, torch 2.14.1. Full tables: [EXPERIMENT_RESULTS.md](docs/EXPERIMENT_RESULTS.md). -The model trained on public-domain science writing by Darwin and Einstein. Paragraphs from **both** books were held out, so the numbers below are not training-set scores. [Sources and exact file hashes](data/science/manifest.json) · [Full experimental method](docs/experiment_protocol.md). - -![Learning curve](docs/figures/learning.svg) - -At the final checkpoint, held-out perplexity was **17.58** across **112,111 next-byte predictions**. Perplexity measures prediction uncertainty; lower is better, but it is not a score for how helpful or factual a chatbot is. - -![KV-cache benchmark](docs/figures/speed.svg) - -The KV cache avoids recomputing attention over the full prompt after every generated token. In this controlled CPU test, it cut generation time from **119.4 ms to 33.4 ms**. +| Experiment | Result | +| --- | --- | +| Attention backends, fp32, 3 trained seeds | identical held-out perplexity, top-1 predictions and greedy text across `sdpa`, `mha` and `reference` | +| Reference ÷ SDPA forward time, 2,048 tokens | 3.4–3.9× (CPU), 3.9–5.4× (MPS); SDPA uses 8–22× less extra memory at batch 8 | +| Training, legacy MHA vs SDPA (corpus v1, 3 seeds) | validation perplexity 19.21 ± 1.75 (MHA) vs 19.04 ± 1.37 (SDPA); training step 5–28% faster with SDPA on CPU, within 5% on MPS | +| KV cache vs recomputation, 64 tokens | 2.2–23× faster than recomputation on CPU; slower on MPS for batch-1 prompts of 1–16 tokens (0.88×) | +| Seed variance, published v1 configuration | seeds 42/43/44 give held-out perplexity 17.58 / 18.99 / 21.07; the published 17.58 is the best of three | +| Held-out test book (corpus v2, Faraday) | test perplexity is 15.66 ± 2.58 (MHA) and 15.60 ± 2.63 (SDPA) | -![INT8 storage and latency trade-off](docs/figures/tradeoff.svg) +Corpus v1 and v2 use different data and preprocessing; their perplexities are not comparable. -Weight-only INT8 reduced stored model tensors by **53.6%**, but the reference implementation ran **slower** because it dequantizes weights during each forward pass. Smaller files do not automatically mean faster inference. +## Implementation -These are **one-seed CPU measurements**, not M4/MPS results or claims of general language ability. The training corpus is about **1 MB**. [Read the report, including limitations](docs/research_report.md) or inspect the [raw measured record](docs/data/science_run.json). +| Component | Implementation | +| --- | --- | +| Model | decoder-only, pre-norm residual blocks, RMSNorm, SwiGLU, learned positions or RoPE, tied embeddings ([`model.py`](src/minilm_lab/model.py)) | +| Attention backends | `sdpa` (default), `mha` (legacy path of all v1 results; alias `auto`), `reference`; same weights, selected at run time ([diagram](docs/figures/attention-backends-light.svg) · [dark](docs/figures/attention-backends-dark.svg)) | +| Inference | KV cache with prefill, single-token decode and sliding-window refresh ([diagram](docs/figures/kv-cache-light.svg) · [dark](docs/figures/kv-cache-dark.svg)); uncached sampling with temperature, top-k and top-p; cached sampling with temperature and greedy decoding | +| Training | AdamW, gradient accumulation, full-split checkpoint selection, bitwise resume with optimizer and RNG state | +| Evaluation | every held-out next byte scored once; perplexity and bits per byte | +| Provenance | append-only records, deterministic model fingerprints, audited baseline, strict validation ([diagram](docs/figures/reproducibility-light.svg) · [dark](docs/figures/reproducibility-dark.svg)) | -## Reproduce on a MacBook Air +## Quick Start -You need Python 3.12 and the ability to install PyTorch. The commands below run the checks, prepare the pinned corpus, train the small model, and score the held-out text. Training time varies by device. +Python 3.12. The lock file pins the exact packages used for every record. ```bash -python3.12 -m venv .venv -source .venv/bin/activate -python -m pip install -e '.[dev]' -make check -python scripts/prepare_science_corpus.py -python scripts/train_tiny.py data/science/train.txt \ - --validation-text data/science/validation.txt \ - --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 \ - --output-dir artifacts/runs/science-v2 -python scripts/evaluate.py data/science/train.txt \ - --validation-text data/science/validation.txt \ - --checkpoint artifacts/runs/science-v2/best_checkpoint.pt +python3.12 -m venv .venv && source .venv/bin/activate +python -m pip install -r requirements/release-lock.txt +python -m pip install -e . --no-deps +make check # ruff + test suite + +python scripts/train_tiny.py data/science-v2/train.txt \ + --validation-text data/science-v2/validation.txt \ + --steps 200 --device cpu --output-dir artifacts/runs/quickstart +python scripts/evaluate.py data/science-v2/train.txt \ + --validation-text data/science-v2/validation.txt --device cpu \ + --checkpoint artifacts/runs/quickstart/best_checkpoint.pt +python scripts/generate.py "Natural selection" --checkpoint artifacts/runs/quickstart/best_checkpoint.pt ``` -Training chooses Apple MPS when available and otherwise runs on CPU. The published run used CPU because MPS was unavailable in its execution environment. Use `--device cpu` to match that hardware path. The corpus preparation script downloads the two source texts and records SHA-256 hashes. Source files can change upstream, so check the hashes before comparing a new run to the published one. +## Reproducibility -## System +```bash +./scripts/verify_release.sh # lint, tests, strict record validation, report drift +python scripts/validate_records.py --strict # model identities verified from artifacts or the pinned baseline -| Component | Implementation | -| --- | --- | -| Tokenization | UTF-8 byte tokenizer; educational BPE alternative | -| Decoder | causal multi-head attention, learned positions or RoPE, RMSNorm, SwiGLU, tied token/output embeddings | -| Training | AdamW, true optimizer-step accumulation, deterministic full-split checkpoint selection, resume with optimizer and RNG state | -| Evaluation | every held-out next token scored once; perplexity and bits per byte | -| Generation | temperature, top-k, top-p; KV cache with sliding-window refresh | -| Experiments | repeated decode benchmark; portable MLP weight-only INT8 comparison; LoRA linear building block | +# published v1 run: legacy backend selected explicitly; bitwise on CPU with torch 2.14 (perplexity 17.58) +python scripts/train_tiny.py data/science/train.txt --validation-text data/science/validation.txt \ + --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 \ + --device cpu --seed 42 --attention-backend mha --output-dir artifacts/runs/science-v1-s42 -PyTorch provides tensor operations, automatic differentiation, multi-head attention and SDPA. The project exposes the model and systems decisions around those primitives. It is not a claim to reimplement an entire deep-learning framework. +PYTHON=python ./experiments/sdpa_v1.sh # v1: MHA vs SDPA training, 3 seeds +PYTHON=python ./experiments/v2_protocol.sh # v2: verify corpus, train, select on validation, test once +``` -## Read the evidence +Checkpoint files are not distributed, and they embed elapsed time, so a regenerated file does not reproduce its recorded SHA-256. Models are compared by a deterministic fingerprint instead (`minilm-model-identity/v1`). Historical fingerprints are anchored by an audited baseline whose hash is pinned in code: an integrity anchor in reviewed code, not a cryptographic signature. Rules: [RECORD_SCHEMA.md](docs/RECORD_SCHEMA.md) · checks: [RELEASE_CHECKLIST.md](docs/RELEASE_CHECKLIST.md). -- [Research paper (PDF)](output/pdf/MiniLM-Lab-Research-Paper.pdf) -- [Research report](docs/research_report.md) -- [Raw science run record](docs/data/science_run.json) -- [Experiment protocol](docs/experiment_protocol.md) -- [Implementation audit](docs/implementation_audit.md) -- [2026 positioning](docs/positioning_2026.md) -- [Roadmap](docs/roadmap.md) -- [Contributing](CONTRIBUTING.md) +## Documentation -The research paper is generated from the tracked results and benchmark protocol with -\`python -m pip install -e '.[report]'\` followed by -\`python scripts/build_research_report.py\`. +- [Experiment results](docs/EXPERIMENT_RESULTS.md): all tables, method and limitations +- [Research report](docs/research_report.md) and [paper (PDF, v1 only)](output/pdf/MiniLM-Lab-Research-Paper.pdf) +- [Implementation audit](docs/implementation_audit.md): evidence for each correctness claim +- [Record schema and provenance](docs/RECORD_SCHEMA.md) · [Release checklist](docs/RELEASE_CHECKLIST.md) +- [Experiment protocol](docs/experiment_protocol.md) · [Roadmap](docs/roadmap.md) · [Contributing](CONTRIBUTING.md) +- Corpus manifests: [v1](data/science/manifest.json) · [v2](data/science-v2/manifest.json) -## Scope +## Limitations -The corpus contains about 1 MB of training text. The model is a controlled educational system for studying language-model mechanics and inference trade-offs. Generated text should not be treated as reliable scientific information. +- **Small, under-trained models.** Every run is 1,000 optimizer steps on about 1 MB of text and spends most of it on an early loss plateau; results are seed-sensitive, and three seeds give a spread, not a confidence interval. +- **One held-out test book.** The v2 test set measures transfer to one unseen author and topic, not broad generalization. Its results are published and can no longer guide tuning under an untouched-test claim. +- **Legacy v1 corpus.** 26% of v1 held-out bytes are Darwin's alphabetical index, and v1 validation both selects checkpoints and is reported; v2 fixes both. +- **One machine.** Timings come from one laptop; MPS findings reflect a tiny model where launch overhead dominates; CUDA is untested; no cross-device or cross-version bitwise reproducibility is claimed. The September KV-cache speedup (3.58×) re-measures at 3.40× on the same hardware. +- **Trust anchor.** Without the original checkpoints, historical model fingerprints rest on the pinned audited baseline. MIT licensed. © Anvit Devadiga. diff --git a/data/science-v2/darwin-1859-validation.txt b/data/science-v2/darwin-1859-validation.txt new file mode 100644 index 0000000..a970577 --- /dev/null +++ b/data/science-v2/darwin-1859-validation.txt @@ -0,0 +1,1391 @@ +We are so much accustomed to see differences in structure between the +embryo and the adult, and likewise a close similarity in the embryos of +widely different animals within the same class, that we might be led to +look at these facts as necessarily contingent in some manner on growth. +But there is no obvious reason why, for instance, the wing of a bat, or +the fin of a porpoise, should not have been sketched out with all the +parts in proper proportion, as soon as any structure became visible in +the embryo. And in some whole groups of animals and in certain members +of other groups, the embryo does not at any period differ widely from +the adult: thus Owen has remarked in regard to cuttle-fish, “there is +no metamorphosis; the cephalopodic character is manifested long before +the parts of the embryo are completed;” and again in spiders, “there is +nothing worthy to be called a metamorphosis.” The larvæ of insects, +whether adapted to the most diverse and active habits, or quite +inactive, being fed by their parents or placed in the midst of proper +nutriment, yet nearly all pass through a similar worm-like stage of +development; but in some few cases, as in that of Aphis, if we look to +the admirable drawings by Professor Huxley of the development of this +insect, we see no trace of the vermiform stage. + +How, then, can we explain these several facts in embryology,—namely the +very general, but not universal difference in structure between the +embryo and the adult;—of parts in the same individual embryo, which +ultimately become very unlike and serve for diverse purposes, being at +this early period of growth alike;—of embryos of different species +within the same class, generally, but not universally, resembling each +other;—of the structure of the embryo not being closely related to its +conditions of existence, except when the +embryo becomes at any period of life active and has to provide for +itself;—of the embryo apparently having sometimes a higher organisation +than the mature animal, into which it is developed. I believe that all +these facts can be explained, as follows, on the view of descent with +modification. + +It is commonly assumed, perhaps from monstrosities often affecting the +embryo at a very early period, that slight variations necessarily +appear at an equally early period. But we have little evidence on this +head—indeed the evidence rather points the other way; for it is +notorious that breeders of cattle, horses, and various fancy animals, +cannot positively tell, until some time after the animal has been born, +what its merits or form will ultimately turn out. We see this plainly +in our own children; we cannot always tell whether the child will be +tall or short, or what its precise features will be. The question is +not, at what period of life any variation has been caused, but at what +period it is fully displayed. The cause may have acted, and I believe +generally has acted, even before the embryo is formed; and the +variation may be due to the male and female sexual elements having been +affected by the conditions to which either parent, or their ancestors, +have been exposed. Nevertheless an effect thus caused at a very early +period, even before the formation of the embryo, may appear late in +life; as when an hereditary disease, which appears in old age alone, +has been communicated to the offspring from the reproductive element of +one parent. Or again, as when the horns of cross-bred cattle have been +affected by the shape of the horns of either parent. For the welfare of +a very young animal, as long as it remains in its mother’s womb, or in +the egg, or as long as it is nourished and protected by its parent, it +must be quite unimportant whether most of its characters are fully +acquired a little earlier or later in life. It would not signify, for +instance, to a bird which obtained its food best by having a long beak, +whether or not it assumed a beak of this particular length, as long as +it was fed by its parents. Hence, I conclude, that it is quite +possible, that each of the many successive modifications, by which each +species has acquired its present structure, may have supervened at a +not very early period of life; and some direct evidence from our +domestic animals supports this view. But in other cases it is quite +possible that each successive modification, or most of them, may have +appeared at an extremely early period. + +I have stated in the first chapter, that there is some evidence to +render it probable, that at whatever age any variation first appears in +the parent, it tends to reappear at a corresponding age in the +offspring. Certain variations can only appear at corresponding ages, +for instance, peculiarities in the caterpillar, cocoon, or imago states +of the silk-moth; or, again, in the horns of almost full-grown cattle. +But further than this, variations which, for all that we can see, might +have appeared earlier or later in life, tend to appear at a +corresponding age in the offspring and parent. I am far from meaning +that this is invariably the case; and I could give a good many cases of +variations (taking the word in the largest sense) which have supervened +at an earlier age in the child than in the parent. + +These two principles, if their truth be admitted, will, I believe, +explain all the above specified leading facts in embryology. But first +let us look at a few analogous cases in domestic varieties. Some +authors who have written on Dogs, maintain that the greyhound and +bulldog, though appearing so different, are really varieties most +closely allied, and have probably descended from +the same wild stock; hence I was curious to see how far their puppies +differed from each other: I was told by breeders that they differed +just as much as their parents, and this, judging by the eye, seemed +almost to be the case; but on actually measuring the old dogs and their +six-days old puppies, I found that the puppies had not nearly acquired +their full amount of proportional difference. So, again, I was told +that the foals of cart and race-horses differed as much as the +full-grown animals; and this surprised me greatly, as I think it +probable that the difference between these two breeds has been wholly +caused by selection under domestication; but having had careful +measurements made of the dam and of a three-days old colt of a race and +heavy cart-horse, I find that the colts have by no means acquired their +full amount of proportional difference. + +As the evidence appears to me conclusive, that the several domestic +breeds of Pigeon have descended from one wild species, I compared young +pigeons of various breeds, within twelve hours after being hatched; I +carefully measured the proportions (but will not here give details) of +the beak, width of mouth, length of nostril and of eyelid, size of feet +and length of leg, in the wild stock, in pouters, fantails, runts, +barbs, dragons, carriers, and tumblers. Now some of these birds, when +mature, differ so extraordinarily in length and form of beak, that they +would, I cannot doubt, be ranked in distinct genera, had they been +natural productions. But when the nestling birds of these several +breeds were placed in a row, though most of them could be distinguished +from each other, yet their proportional differences in the above +specified several points were incomparably less than in the full-grown +birds. Some characteristic points of difference—for instance, that of +the width of mouth—could hardly be detected in the young. +But there was one remarkable exception to this rule, for the young of +the short-faced tumbler differed from the young of the wild rock-pigeon +and of the other breeds, in all its proportions, almost exactly as much +as in the adult state. + +The two principles above given seem to me to explain these facts in +regard to the later embryonic stages of our domestic varieties. +Fanciers select their horses, dogs, and pigeons, for breeding, when +they are nearly grown up: they are indifferent whether the desired +qualities and structures have been acquired earlier or later in life, +if the full-grown animal possesses them. And the cases just given, more +especially that of pigeons, seem to show that the characteristic +differences which give value to each breed, and which have been +accumulated by man’s selection, have not generally first appeared at an +early period of life, and have been inherited by the offspring at a +corresponding not early period. But the case of the short-faced +tumbler, which when twelve hours old had acquired its proper +proportions, proves that this is not the universal rule; for here the +characteristic differences must either have appeared at an earlier +period than usual, or, if not so, the differences must have been +inherited, not at the corresponding, but at an earlier age. + +Now let us apply these facts and the above two principles—which latter, +though not proved true, can be shown to be in some degree probable—to +species in a state of nature. Let us take a genus of birds, descended +on my theory from some one parent-species, and of which the several new +species have become modified through natural selection in accordance +with their diverse habits. Then, from the many slight successive steps +of variation having supervened at a rather late age, and having been +inherited at a corresponding +age, the young of the new species of our supposed genus will manifestly +tend to resemble each other much more closely than do the adults, just +as we have seen in the case of pigeons. We may extend this view to +whole families or even classes. The fore-limbs, for instance, which +served as legs in the parent-species, may become, by a long course of +modification, adapted in one descendant to act as hands, in another as +paddles, in another as wings; and on the above two principles—namely of +each successive modification supervening at a rather late age, and +being inherited at a corresponding late age—the fore-limbs in the +embryos of the several descendants of the parent-species will still +resemble each other closely, for they will not have been modified. But +in each individual new species, the embryonic fore-limbs will differ +greatly from the fore-limbs in the mature animal; the limbs in the +latter having undergone much modification at a rather late period of +life, and having thus been converted into hands, or paddles, or wings. +Whatever influence long-continued exercise or use on the one hand, and +disuse on the other, may have in modifying an organ, such influence +will mainly affect the mature animal, which has come to its full powers +of activity and has to gain its own living; and the effects thus +produced will be inherited at a corresponding mature age. Whereas the +young will remain unmodified, or be modified in a lesser degree, by the +effects of use and disuse. + +In certain cases the successive steps of variation might supervene, +from causes of which we are wholly ignorant, at a very early period of +life, or each step might be inherited at an earlier period than that at +which it first appeared. In either case (as with the short-faced +tumbler) the young or embryo would closely +resemble the mature parent-form. We have seen that this is the rule of +development in certain whole groups of animals, as with cuttle-fish and +spiders, and with a few members of the great class of insects, as with +Aphis. With respect to the final cause of the young in these cases not +undergoing any metamorphosis, or closely resembling their parents from +their earliest age, we can see that this would result from the two +following contingencies; firstly, from the young, during a course of +modification carried on for many generations, having to provide for +their own wants at a very early stage of development, and secondly, +from their following exactly the same habits of life with their +parents; for in this case, it would be indispensable for the existence +of the species, that the child should be modified at a very early age +in the same manner with its parents, in accordance with their similar +habits. Some further explanation, however, of the embryo not undergoing +any metamorphosis is perhaps requisite. If, on the other hand, it +profited the young to follow habits of life in any degree different +from those of their parent, and consequently to be constructed in a +slightly different manner, then, on the principle of inheritance at +corresponding ages, the active young or larvæ might easily be rendered +by natural selection different to any conceivable extent from their +parents. Such differences might, also, become correlated with +successive stages of development; so that the larvæ, in the first +stage, might differ greatly from the larvæ in the second stage, as we +have seen to be the case with cirripedes. The adult might become fitted +for sites or habits, in which organs of locomotion or of the senses, +etc., would be useless; and in this case the final metamorphosis would +be said to be retrograde. + +As all the organic beings, extinct and recent, which +have ever lived on this earth have to be classed together, and as all +have been connected by the finest gradations, the best, or indeed, if +our collections were nearly perfect, the only possible arrangement, +would be genealogical. Descent being on my view the hidden bond of +connexion which naturalists have been seeking under the term of the +natural system. On this view we can understand how it is that, in the +eyes of most naturalists, the structure of the embryo is even more +important for classification than that of the adult. For the embryo is +the animal in its less modified state; and in so far it reveals the +structure of its progenitor. In two groups of animal, however much they +may at present differ from each other in structure and habits, if they +pass through the same or similar embryonic stages, we may feel assured +that they have both descended from the same or nearly similar parents, +and are therefore in that degree closely related. Thus, community in +embryonic structure reveals community of descent. It will reveal this +community of descent, however much the structure of the adult may have +been modified and obscured; we have seen, for instance, that cirripedes +can at once be recognised by their larvæ as belonging to the great +class of crustaceans. As the embryonic state of each species and group +of species partially shows us the structure of their less modified +ancient progenitors, we can clearly see why ancient and extinct forms +of life should resemble the embryos of their descendants,—our existing +species. Agassiz believes this to be a law of nature; but I am bound to +confess that I only hope to see the law hereafter proved true. It can +be proved true in those cases alone in which the ancient state, now +supposed to be represented in many embryos, has not been obliterated, +either by the successive variations in a long course of modification +having supervened +at a very early age, or by the variations having been inherited at an +earlier period than that at which they first appeared. It should also +be borne in mind, that the supposed law of resemblance of ancient forms +of life to the embryonic stages of recent forms, may be true, but yet, +owing to the geological record not extending far enough back in time, +may remain for a long period, or for ever, incapable of demonstration. + +Thus, as it seems to me, the leading facts in embryology, which are +second in importance to none in natural history, are explained on the +principle of slight modifications not appearing, in the many +descendants from some one ancient progenitor, at a very early period in +the life of each, though perhaps caused at the earliest, and being +inherited at a corresponding not early period. Embryology rises greatly +in interest, when we thus look at the embryo as a picture, more or less +obscured, of the common parent-form of each great class of animals. + +_Rudimentary, atrophied, or aborted organs_.—Organs or parts in this +strange condition, bearing the stamp of inutility, are extremely common +throughout nature. For instance, rudimentary mammæ are very general in +the males of mammals: I presume that the “bastard-wing” in birds may be +safely considered as a digit in a rudimentary state: in very many +snakes one lobe of the lungs is rudimentary; in other snakes there are +rudiments of the pelvis and hind limbs. Some of the cases of +rudimentary organs are extremely curious; for instance, the presence of +teeth in foetal whales, which when grown up have not a tooth in their +heads; and the presence of teeth, which never cut through the gums, in +the upper jaws of our unborn calves. It has even been stated on good +authority that rudiments of teeth can be detected +in the beaks of certain embryonic birds. Nothing can be plainer than +that wings are formed for flight, yet in how many insects do we see +wings so reduced in size as to be utterly incapable of flight, and not +rarely lying under wing-cases, firmly soldered together! + +The meaning of rudimentary organs is often quite unmistakeable: for +instance there are beetles of the same genus (and even of the same +species) resembling each other most closely in all respects, one of +which will have full-sized wings, and another mere rudiments of +membrane; and here it is impossible to doubt, that the rudiments +represent wings. Rudimentary organs sometimes retain their +potentiality, and are merely not developed: this seems to be the case +with the mammæ of male mammals, for many instances are on record of +these organs having become well developed in full-grown males, and +having secreted milk. So again there are normally four developed and +two rudimentary teats in the udders of the genus Bos, but in our +domestic cows the two sometimes become developed and give milk. In +individual plants of the same species the petals sometimes occur as +mere rudiments, and sometimes in a well-developed state. In plants with +separated sexes, the male flowers often have a rudiment of a pistil; +and Kölreuter found that by crossing such male plants with an +hermaphrodite species, the rudiment of the pistil in the hybrid +offspring was much increased in size; and this shows that the rudiment +and the perfect pistil are essentially alike in nature. + +An organ serving for two purposes, may become rudimentary or utterly +aborted for one, even the more important purpose; and remain perfectly +efficient for the other. Thus in plants, the office of the pistil is to +allow the pollen-tubes to reach the ovules protected in the ovarium at +its base. The pistil consists of a stigma +supported on the style; but in some Compositæ, the male florets, which +of course cannot be fecundated, have a pistil, which is in a +rudimentary state, for it is not crowned with a stigma; but the style +remains well developed, and is clothed with hairs as in other +compositæ, for the purpose of brushing the pollen out of the +surrounding anthers. Again, an organ may become rudimentary for its +proper purpose, and be used for a distinct object: in certain fish the +swim-bladder seems to be rudimentary for its proper function of giving +buoyancy, but has become converted into a nascent breathing organ or +lung. Other similar instances could be given. + +Rudimentary organs in the individuals of the same species are very +liable to vary in degree of development and in other respects. +Moreover, in closely allied species, the degree to which the same organ +has been rendered rudimentary occasionally differs much. This latter +fact is well exemplified in the state of the wings of the female moths +in certain groups. Rudimentary organs may be utterly aborted; and this +implies, that we find in an animal or plant no trace of an organ, which +analogy would lead us to expect to find, and which is occasionally +found in monstrous individuals of the species. Thus in the snapdragon +(antirrhinum) we generally do not find a rudiment of a fifth stamen; +but this may sometimes be seen. In tracing the homologies of the same +part in different members of a class, nothing is more common, or more +necessary, than the use and discovery of rudiments. This is well shown +in the drawings given by Owen of the bones of the leg of the horse, ox, +and rhinoceros. + +It is an important fact that rudimentary organs, such as teeth in the +upper jaws of whales and ruminants, can often be detected in the +embryo, but afterwards wholly disappear. It is also, I believe, a +universal +rule, that a rudimentary part or organ is of greater size relatively to +the adjoining parts in the embryo, than in the adult; so that the organ +at this early age is less rudimentary, or even cannot be said to be in +any degree rudimentary. Hence, also, a rudimentary organ in the adult, +is often said to have retained its embryonic condition. + +I have now given the leading facts with respect to rudimentary organs. +In reflecting on them, every one must be struck with astonishment: for +the same reasoning power which tells us plainly that most parts and +organs are exquisitely adapted for certain purposes, tells us with +equal plainness that these rudimentary or atrophied organs, are +imperfect and useless. In works on natural history rudimentary organs +are generally said to have been created “for the sake of symmetry,” or +in order “to complete the scheme of nature;” but this seems to me no +explanation, merely a restatement of the fact. Would it be thought +sufficient to say that because planets revolve in elliptic courses +round the sun, satellites follow the same course round the planets, for +the sake of symmetry, and to complete the scheme of nature? An eminent +physiologist accounts for the presence of rudimentary organs, by +supposing that they serve to excrete matter in excess, or injurious to +the system; but can we suppose that the minute papilla, which often +represents the pistil in male flowers, and which is formed merely of +cellular tissue, can thus act? Can we suppose that the formation of +rudimentary teeth which are subsequently absorbed, can be of any +service to the rapidly growing embryonic calf by the excretion of +precious phosphate of lime? When a man’s fingers have been amputated, +imperfect nails sometimes appear on the stumps: I could as soon believe +that these vestiges of nails have appeared, not from unknown laws +of growth, but in order to excrete horny matter, as that the +rudimentary nails on the fin of the manatee were formed for this +purpose. + +On my view of descent with modification, the origin of rudimentary +organs is simple. We have plenty of cases of rudimentary organs in our +domestic productions,—as the stump of a tail in tailless breeds,—the +vestige of an ear in earless breeds,—the reappearance of minute +dangling horns in hornless breeds of cattle, more especially, according +to Youatt, in young animals,—and the state of the whole flower in the +cauliflower. We often see rudiments of various parts in monsters. But I +doubt whether any of these cases throw light on the origin of +rudimentary organs in a state of nature, further than by showing that +rudiments can be produced; for I doubt whether species under nature +ever undergo abrupt changes. I believe that disuse has been the main +agency; that it has led in successive generations to the gradual +reduction of various organs, until they have become rudimentary,—as in +the case of the eyes of animals inhabiting dark caverns, and of the +wings of birds inhabiting oceanic islands, which have seldom been +forced to take flight, and have ultimately lost the power of flying. +Again, an organ useful under certain conditions, might become injurious +under others, as with the wings of beetles living on small and exposed +islands; and in this case natural selection would continue slowly to +reduce the organ, until it was rendered harmless and rudimentary. + +Any change in function, which can be effected by insensibly small +steps, is within the power of natural selection; so that an organ +rendered, during changed habits of life, useless or injurious for one +purpose, might easily be modified and used for another purpose. Or an +organ might be retained for one alone of its +former functions. An organ, when rendered useless, may well be +variable, for its variations cannot be checked by natural selection. At +whatever period of life disuse or selection reduces an organ, and this +will generally be when the being has come to maturity and to its full +powers of action, the principle of inheritance at corresponding ages +will reproduce the organ in its reduced state at the same age, and +consequently will seldom affect or reduce it in the embryo. Thus we can +understand the greater relative size of rudimentary organs in the +embryo, and their lesser relative size in the adult. But if each step +of the process of reduction were to be inherited, not at the +corresponding age, but at an extremely early period of life (as we have +good reason to believe to be possible) the rudimentary part would tend +to be wholly lost, and we should have a case of complete abortion. The +principle, also, of economy, explained in a former chapter, by which +the materials forming any part or structure, if not useful to the +possessor, will be saved as far as is possible, will probably often +come into play; and this will tend to cause the entire obliteration of +a rudimentary organ. + +As the presence of rudimentary organs is thus due to the tendency in +every part of the organisation, which has long existed, to be +inherited—we can understand, on the genealogical view of +classification, how it is that systematists have found rudimentary +parts as useful as, or even sometimes more useful than, parts of high +physiological importance. Rudimentary organs may be compared with the +letters in a word, still retained in the spelling, but become useless +in the pronunciation, but which serve as a clue in seeking for its +derivation. On the view of descent with modification, we may conclude +that the existence of organs in a rudimentary, imperfect, and useless +condition, or quite aborted, far +from presenting a strange difficulty, as they assuredly do on the +ordinary doctrine of creation, might even have been anticipated, and +can be accounted for by the laws of inheritance. + +_Summary_.—In this chapter I have attempted to show, that the +subordination of group to group in all organisms throughout all time; +that the nature of the relationship, by which all living and extinct +beings are united by complex, radiating, and circuitous lines of +affinities into one grand system; the rules followed and the +difficulties encountered by naturalists in their classifications; the +value set upon characters, if constant and prevalent, whether of high +vital importance, or of the most trifling importance, or, as in +rudimentary organs, of no importance; the wide opposition in value +between analogical or adaptive characters, and characters of true +affinity; and other such rules;—all naturally follow on the view of the +common parentage of those forms which are considered by naturalists as +allied, together with their modification through natural selection, +with its contingencies of extinction and divergence of character. In +considering this view of classification, it should be borne in mind +that the element of descent has been universally used in ranking +together the sexes, ages, and acknowledged varieties of the same +species, however different they may be in structure. If we extend the +use of this element of descent,—the only certainly known cause of +similarity in organic beings,—we shall understand what is meant by the +natural system: it is genealogical in its attempted arrangement, with +the grades of acquired difference marked by the terms varieties, +species, genera, families, orders, and classes. + +On this same view of descent with modification, all the great facts in +Morphology become intelligible,—whether +we look to the same pattern displayed in the homologous organs, to +whatever purpose applied, of the different species of a class; or to +the homologous parts constructed on the same pattern in each individual +animal and plant. + +On the principle of successive slight variations, not necessarily or +generally supervening at a very early period of life, and being +inherited at a corresponding period, we can understand the great +leading facts in Embryology; namely, the resemblance in an individual +embryo of the homologous parts, which when matured will become widely +different from each other in structure and function; and the +resemblance in different species of a class of the homologous parts or +organs, though fitted in the adult members for purposes as different as +possible. Larvæ are active embryos, which have become specially +modified in relation to their habits of life, through the principle of +modifications being inherited at corresponding ages. On this same +principle—and bearing in mind, that when organs are reduced in size, +either from disuse or selection, it will generally be at that period of +life when the being has to provide for its own wants, and bearing in +mind how strong is the principle of inheritance—the occurrence of +rudimentary organs and their final abortion, present to us no +inexplicable difficulties; on the contrary, their presence might have +been even anticipated. The importance of embryological characters and +of rudimentary organs in classification is intelligible, on the view +that an arrangement is only so far natural as it is genealogical. + +Finally, the several classes of facts which have been considered in +this chapter, seem to me to proclaim so plainly, that the innumerable +species, genera, and families of organic beings, with which this world +is +peopled, have all descended, each within its own class or group, from +common parents, and have all been modified in the course of descent, +that I should without hesitation adopt this view, even if it were +unsupported by other facts or arguments. + +Recapitulation of the difficulties on the theory of Natural Selection. +Recapitulation of the general and special circumstances in its favour. +Causes of the general belief in the immutability of species. How far +the theory of natural selection may be extended. Effects of its +adoption on the study of Natural history. Concluding remarks. + +As this whole volume is one long argument, it may be convenient to the +reader to have the leading facts and inferences briefly recapitulated. + +That many and grave objections may be advanced against the theory of +descent with modification through natural selection, I do not deny. I +have endeavoured to give to them their full force. Nothing at first can +appear more difficult to believe than that the more complex organs and +instincts should have been perfected, not by means superior to, though +analogous with, human reason, but by the accumulation of innumerable +slight variations, each good for the individual possessor. +Nevertheless, this difficulty, though appearing to our imagination +insuperably great, cannot be considered real if we admit the following +propositions, namely,—that gradations in the perfection of any organ or +instinct, which we may consider, either do now exist or could have +existed, each good of its kind,—that all organs and instincts are, in +ever so slight a degree, variable,—and, lastly, that there is a +struggle for existence leading to the preservation of each profitable +deviation of structure or instinct. The truth of these propositions +cannot, I think, be disputed. + +It is, no doubt, extremely difficult even to conjecture by what +gradations many structures have been perfected, more especially amongst +broken and failing groups of organic beings; but we see so many strange +gradations in nature, as is proclaimed by the canon, “Natura non facit +saltum,” that we ought to be extremely cautious in saying that any +organ or instinct, or any whole being, could not have arrived at its +present state by many graduated steps. There are, it must be admitted, +cases of special difficulty on the theory of natural selection; and one +of the most curious of these is the existence of two or three defined +castes of workers or sterile females in the same community of ants; but +I have attempted to show how this difficulty can be mastered. + +With respect to the almost universal sterility of species when first +crossed, which forms so remarkable a contrast with the almost universal +fertility of varieties when crossed, I must refer the reader to the +recapitulation of the facts given at the end of the eighth chapter, +which seem to me conclusively to show that this sterility is no more a +special endowment than is the incapacity of two trees to be grafted +together, but that it is incidental on constitutional differences in +the reproductive systems of the intercrossed species. We see the truth +of this conclusion in the vast difference in the result, when the same +two species are crossed reciprocally; that is, when one species is +first used as the father and then as the mother. + +The fertility of varieties when intercrossed and of their mongrel +offspring cannot be considered as universal; nor is their very general +fertility surprising when we remember that it is not likely that either +their constitutions or their reproductive systems should have been +profoundly modified. Moreover, most of the +varieties which have been experimentised on have been produced under +domestication; and as domestication apparently tends to eliminate +sterility, we ought not to expect it also to produce sterility. + +The sterility of hybrids is a very different case from that of first +crosses, for their reproductive organs are more or less functionally +impotent; whereas in first crosses the organs on both sides are in a +perfect condition. As we continually see that organisms of all kinds +are rendered in some degree sterile from their constitutions having +been disturbed by slightly different and new conditions of life, we +need not feel surprise at hybrids being in some degree sterile, for +their constitutions can hardly fail to have been disturbed from being +compounded of two distinct organisations. This parallelism is supported +by another parallel, but directly opposite, class of facts; namely, +that the vigour and fertility of all organic beings are increased by +slight changes in their conditions of life, and that the offspring of +slightly modified forms or varieties acquire from being crossed +increased vigour and fertility. So that, on the one hand, considerable +changes in the conditions of life and crosses between greatly modified +forms, lessen fertility; and on the other hand, lesser changes in the +conditions of life and crosses between less modified forms, increase +fertility. + +Turning to geographical distribution, the difficulties encountered on +the theory of descent with modification are grave enough. All the +individuals of the same species, and all the species of the same genus, +or even higher group, must have descended from common parents; and +therefore, in however distant and isolated parts of the world they are +now found, they must in the course of successive generations have +passed from some one part to the others. We are often wholly unable +even to conjecture how this could have been effected. Yet, as we have +reason to believe that some species have retained the same specific +form for very long periods, enormously long as measured by years, too +much stress ought not to be laid on the occasional wide diffusion of +the same species; for during very long periods of time there will +always be a good chance for wide migration by many means. A broken or +interrupted range may often be accounted for by the extinction of the +species in the intermediate regions. It cannot be denied that we are as +yet very ignorant of the full extent of the various climatal and +geographical changes which have affected the earth during modern +periods; and such changes will obviously have greatly facilitated +migration. As an example, I have attempted to show how potent has been +the influence of the Glacial period on the distribution both of the +same and of representative species throughout the world. We are as yet +profoundly ignorant of the many occasional means of transport. With +respect to distinct species of the same genus inhabiting very distant +and isolated regions, as the process of modification has necessarily +been slow, all the means of migration will have been possible during a +very long period; and consequently the difficulty of the wide diffusion +of species of the same genus is in some degree lessened. + +As on the theory of natural selection an interminable number of +intermediate forms must have existed, linking together all the species +in each group by gradations as fine as our present varieties, it may be +asked, Why do we not see these linking forms all around us? Why are not +all organic beings blended together in an inextricable chaos? With +respect to existing forms, we should remember that we have no right to +expect (excepting in rare cases) to discover _directly_ connecting +links between them, but only between each and some extinct and +supplanted form. Even on a wide area, which has during a long period +remained continuous, and of which the climate and other conditions of +life change insensibly in going from a district occupied by one species +into another district occupied by a closely allied species, we have no +just right to expect often to find intermediate varieties in the +intermediate zone. For we have reason to believe that only a few +species are undergoing change at any one period; and all changes are +slowly effected. I have also shown that the intermediate varieties +which will at first probably exist in the intermediate zones, will be +liable to be supplanted by the allied forms on either hand; and the +latter, from existing in greater numbers, will generally be modified +and improved at a quicker rate than the intermediate varieties, which +exist in lesser numbers; so that the intermediate varieties will, in +the long run, be supplanted and exterminated. + +On this doctrine of the extermination of an infinitude of connecting +links, between the living and extinct inhabitants of the world, and at +each successive period between the extinct and still older species, why +is not every geological formation charged with such links? Why does not +every collection of fossil remains afford plain evidence of the +gradation and mutation of the forms of life? We meet with no such +evidence, and this is the most obvious and forcible of the many +objections which may be urged against my theory. Why, again, do whole +groups of allied species appear, though certainly they often falsely +appear, to have come in suddenly on the several geological stages? Why +do we not find great piles of strata beneath the Silurian system, +stored with the remains of the progenitors of the Silurian groups of +fossils? For certainly on my theory such +strata must somewhere have been deposited at these ancient and utterly +unknown epochs in the world’s history. + +I can answer these questions and grave objections only on the +supposition that the geological record is far more imperfect than most +geologists believe. It cannot be objected that there has not been time +sufficient for any amount of organic change; for the lapse of time has +been so great as to be utterly inappreciable by the human intellect. +The number of specimens in all our museums is absolutely as nothing +compared with the countless generations of countless species which +certainly have existed. We should not be able to recognise a species as +the parent of any one or more species if we were to examine them ever +so closely, unless we likewise possessed many of the intermediate links +between their past or parent and present states; and these many links +we could hardly ever expect to discover, owing to the imperfection of +the geological record. Numerous existing doubtful forms could be named +which are probably varieties; but who will pretend that in future ages +so many fossil links will be discovered, that naturalists will be able +to decide, on the common view, whether or not these doubtful forms are +varieties? As long as most of the links between any two species are +unknown, if any one link or intermediate variety be discovered, it will +simply be classed as another and distinct species. Only a small portion +of the world has been geologically explored. Only organic beings of +certain classes can be preserved in a fossil condition, at least in any +great number. Widely ranging species vary most, and varieties are often +at first local,—both causes rendering the discovery of intermediate +links less likely. Local varieties will not spread into other and +distant regions until they are considerably modified and improved; +and when they do spread, if discovered in a geological formation, they +will appear as if suddenly created there, and will be simply classed as +new species. Most formations have been intermittent in their +accumulation; and their duration, I am inclined to believe, has been +shorter than the average duration of specific forms. Successive +formations are separated from each other by enormous blank intervals of +time; for fossiliferous formations, thick enough to resist future +degradation, can be accumulated only where much sediment is deposited +on the subsiding bed of the sea. During the alternate periods of +elevation and of stationary level the record will be blank. During +these latter periods there will probably be more variability in the +forms of life; during periods of subsidence, more extinction. + +With respect to the absence of fossiliferous formations beneath the +lowest Silurian strata, I can only recur to the hypothesis given in the +ninth chapter. That the geological record is imperfect all will admit; +but that it is imperfect to the degree which I require, few will be +inclined to admit. If we look to long enough intervals of time, geology +plainly declares that all species have changed; and they have changed +in the manner which my theory requires, for they have changed slowly +and in a graduated manner. We clearly see this in the fossil remains +from consecutive formations invariably being much more closely related +to each other, than are the fossils from formations distant from each +other in time. + +Such is the sum of the several chief objections and difficulties which +may justly be urged against my theory; and I have now briefly +recapitulated the answers and explanations which can be given to them. +I have felt these difficulties far too heavily during many years to +doubt their weight. But it deserves especial notice that the more +important objections relate to questions on which we are confessedly +ignorant; nor do we know how ignorant we are. We do not know all the +possible transitional gradations between the simplest and the most +perfect organs; it cannot be pretended that we know all the varied +means of Distribution during the long lapse of years, or that we know +how imperfect the Geological Record is. Grave as these several +difficulties are, in my judgment they do not overthrow the theory of +descent with modification. + +Now let us turn to the other side of the argument. Under domestication +we see much variability. This seems to be mainly due to the +reproductive system being eminently susceptible to changes in the +conditions of life; so that this system, when not rendered impotent, +fails to reproduce offspring exactly like the parent-form. Variability +is governed by many complex laws,—by correlation of growth, by use and +disuse, and by the direct action of the physical conditions of life. +There is much difficulty in ascertaining how much modification our +domestic productions have undergone; but we may safely infer that the +amount has been large, and that modifications can be inherited for long +periods. As long as the conditions of life remain the same, we have +reason to believe that a modification, which has already been inherited +for many generations, may continue to be inherited for an almost +infinite number of generations. On the other hand we have evidence that +variability, when it has once come into play, does not wholly cease; +for new varieties are still occasionally produced by our most anciently +domesticated productions. + +Man does not actually produce variability; he only +unintentionally exposes organic beings to new conditions of life, and +then nature acts on the organisation, and causes variability. But man +can and does select the variations given to him by nature, and thus +accumulate them in any desired manner. He thus adapts animals and +plants for his own benefit or pleasure. He may do this methodically, or +he may do it unconsciously by preserving the individuals most useful to +him at the time, without any thought of altering the breed. It is +certain that he can largely influence the character of a breed by +selecting, in each successive generation, individual differences so +slight as to be quite inappreciable by an uneducated eye. This process +of selection has been the great agency in the production of the most +distinct and useful domestic breeds. That many of the breeds produced +by man have to a large extent the character of natural species, is +shown by the inextricable doubts whether very many of them are +varieties or aboriginal species. + +There is no obvious reason why the principles which have acted so +efficiently under domestication should not have acted under nature. In +the preservation of favoured individuals and races, during the +constantly-recurrent Struggle for Existence, we see the most powerful +and ever-acting means of selection. The struggle for existence +inevitably follows from the high geometrical ratio of increase which is +common to all organic beings. This high rate of increase is proved by +calculation, by the effects of a succession of peculiar seasons, and by +the results of naturalisation, as explained in the third chapter. More +individuals are born than can possibly survive. A grain in the balance +will determine which individual shall live and which shall die,—which +variety or species shall increase in number, and which shall decrease, +or finally become extinct. As the individuals +of the same species come in all respects into the closest competition +with each other, the struggle will generally be most severe between +them; it will be almost equally severe between the varieties of the +same species, and next in severity between the species of the same +genus. But the struggle will often be very severe between beings most +remote in the scale of nature. The slightest advantage in one being, at +any age or during any season, over those with which it comes into +competition, or better adaptation in however slight a degree to the +surrounding physical conditions, will turn the balance. + +With animals having separated sexes there will in most cases be a +struggle between the males for possession of the females. The most +vigorous individuals, or those which have most successfully struggled +with their conditions of life, will generally leave most progeny. But +success will often depend on having special weapons or means of +defence, or on the charms of the males; and the slightest advantage +will lead to victory. + +As geology plainly proclaims that each land has undergone great +physical changes, we might have expected that organic beings would have +varied under nature, in the same way as they generally have varied +under the changed conditions of domestication. And if there be any +variability under nature, it would be an unaccountable fact if natural +selection had not come into play. It has often been asserted, but the +assertion is quite incapable of proof, that the amount of variation +under nature is a strictly limited quantity. Man, though acting on +external characters alone and often capriciously, can produce within a +short period a great result by adding up mere individual differences in +his domestic productions; and every one admits that there are at least +individual differences in species under nature. But, besides such +differences, all naturalists +have admitted the existence of varieties, which they think sufficiently +distinct to be worthy of record in systematic works. No one can draw +any clear distinction between individual differences and slight +varieties; or between more plainly marked varieties and sub-species, +and species. Let it be observed how naturalists differ in the rank +which they assign to the many representative forms in Europe and North +America. + +If then we have under nature variability and a powerful agent always +ready to act and select, why should we doubt that variations in any way +useful to beings, under their excessively complex relations of life, +would be preserved, accumulated, and inherited? Why, if man can by +patience select variations most useful to himself, should nature fail +in selecting variations useful, under changing conditions of life, to +her living products? What limit can be put to this power, acting during +long ages and rigidly scrutinising the whole constitution, structure, +and habits of each creature,—favouring the good and rejecting the bad? +I can see no limit to this power, in slowly and beautifully adapting +each form to the most complex relations of life. The theory of natural +selection, even if we looked no further than this, seems to me to be in +itself probable. I have already recapitulated, as fairly as I could, +the opposed difficulties and objections: now let us turn to the special +facts and arguments in favour of the theory. + +On the view that species are only strongly marked and permanent +varieties, and that each species first existed as a variety, we can see +why it is that no line of demarcation can be drawn between species, +commonly supposed to have been produced by special acts of creation, +and varieties which are acknowledged to have been produced by secondary +laws. On this same view we can understand how it is that in each region +where many species of a genus have been produced, and where they now +flourish, these same species should present many varieties; for where +the manufactory of species has been active, we might expect, as a +general rule, to find it still in action; and this is the case if +varieties be incipient species. Moreover, the species of the larger +genera, which afford the greater number of varieties or incipient +species, retain to a certain degree the character of varieties; for +they differ from each other by a less amount of difference than do the +species of smaller genera. The closely allied species also of the +larger genera apparently have restricted ranges, and they are clustered +in little groups round other species—in which respects they resemble +varieties. These are strange relations on the view of each species +having been independently created, but are intelligible if all species +first existed as varieties. + +As each species tends by its geometrical ratio of reproduction to +increase inordinately in number; and as the modified descendants of +each species will be enabled to increase by so much the more as they +become more diversified in habits and structure, so as to be enabled to +seize on many and widely different places in the economy of nature, +there will be a constant tendency in natural selection to preserve the +most divergent offspring of any one species. Hence during a +long-continued course of modification, the slight differences, +characteristic of varieties of the same species, tend to be augmented +into the greater differences characteristic of species of the same +genus. New and improved varieties will inevitably supplant and +exterminate the older, less improved and intermediate varieties; and +thus species are rendered to a large extent defined and distinct +objects. Dominant species belonging to the larger groups tend to give +birth to new and dominant +forms; so that each large group tends to become still larger, and at +the same time more divergent in character. But as all groups cannot +thus succeed in increasing in size, for the world would not hold them, +the more dominant groups beat the less dominant. This tendency in the +large groups to go on increasing in size and diverging in character, +together with the almost inevitable contingency of much extinction, +explains the arrangement of all the forms of life, in groups +subordinate to groups, all within a few great classes, which we now see +everywhere around us, and which has prevailed throughout all time. This +grand fact of the grouping of all organic beings seems to me utterly +inexplicable on the theory of creation. + +As natural selection acts solely by accumulating slight, successive, +favourable variations, it can produce no great or sudden modification; +it can act only by very short and slow steps. Hence the canon of +“Natura non facit saltum,” which every fresh addition to our knowledge +tends to make more strictly correct, is on this theory simply +intelligible. We can plainly see why nature is prodigal in variety, +though niggard in innovation. But why this should be a law of nature if +each species has been independently created, no man can explain. + +Many other facts are, as it seems to me, explicable on this theory. How +strange it is that a bird, under the form of woodpecker, should have +been created to prey on insects on the ground; that upland geese, which +never or rarely swim, should have been created with webbed feet; that a +thrush should have been created to dive and feed on sub-aquatic +insects; and that a petrel should have been created with habits and +structure fitting it for the life of an auk or grebe! and so on in +endless other cases. But on the view of each +species constantly trying to increase in number, with natural selection +always ready to adapt the slowly varying descendants of each to any +unoccupied or ill-occupied place in nature, these facts cease to be +strange, or perhaps might even have been anticipated. + +As natural selection acts by competition, it adapts the inhabitants of +each country only in relation to the degree of perfection of their +associates; so that we need feel no surprise at the inhabitants of any +one country, although on the ordinary view supposed to have been +specially created and adapted for that country, being beaten and +supplanted by the naturalised productions from another land. Nor ought +we to marvel if all the contrivances in nature be not, as far as we can +judge, absolutely perfect; and if some of them be abhorrent to our +ideas of fitness. We need not marvel at the sting of the bee causing +the bee’s own death; at drones being produced in such vast numbers for +one single act, and being then slaughtered by their sterile sisters; at +the astonishing waste of pollen by our fir-trees; at the instinctive +hatred of the queen bee for her own fertile daughters; at ichneumonidæ +feeding within the live bodies of caterpillars; and at other such +cases. The wonder indeed is, on the theory of natural selection, that +more cases of the want of absolute perfection have not been observed. + +The complex and little known laws governing variation are the same, as +far as we can see, with the laws which have governed the production of +so-called specific forms. In both cases physical conditions seem to +have produced but little direct effect; yet when varieties enter any +zone, they occasionally assume some of the characters of the species +proper to that zone. In both varieties and species, use and disuse seem +to have produced some effect; for it is difficult to resist this +conclusion +when we look, for instance, at the logger-headed duck, which has wings +incapable of flight, in nearly the same condition as in the domestic +duck; or when we look at the burrowing tucutucu, which is occasionally +blind, and then at certain moles, which are habitually blind and have +their eyes covered with skin; or when we look at the blind animals +inhabiting the dark caves of America and Europe. In both varieties and +species correlation of growth seems to have played a most important +part, so that when one part has been modified other parts are +necessarily modified. In both varieties and species reversions to +long-lost characters occur. How inexplicable on the theory of creation +is the occasional appearance of stripes on the shoulder and legs of the +several species of the horse-genus and in their hybrids! How simply is +this fact explained if we believe that these species have descended +from a striped progenitor, in the same manner as the several domestic +breeds of pigeon have descended from the blue and barred rock-pigeon! + +On the ordinary view of each species having been independently created, +why should the specific characters, or those by which the species of +the same genus differ from each other, be more variable than the +generic characters in which they all agree? Why, for instance, should +the colour of a flower be more likely to vary in any one species of a +genus, if the other species, supposed to have been created +independently, have differently coloured flowers, than if all the +species of the genus have the same coloured flowers? If species are +only well-marked varieties, of which the characters have become in a +high degree permanent, we can understand this fact; for they have +already varied since they branched off from a common progenitor in +certain characters, by which they have come to be specifically distinct +from each other; +and therefore these same characters would be more likely still to be +variable than the generic characters which have been inherited without +change for an enormous period. It is inexplicable on the theory of +creation why a part developed in a very unusual manner in any one +species of a genus, and therefore, as we may naturally infer, of great +importance to the species, should be eminently liable to variation; +but, on my view, this part has undergone, since the several species +branched off from a common progenitor, an unusual amount of variability +and modification, and therefore we might expect this part generally to +be still variable. But a part may be developed in the most unusual +manner, like the wing of a bat, and yet not be more variable than any +other structure, if the part be common to many subordinate forms, that +is, if it has been inherited for a very long period; for in this case +it will have been rendered constant by long-continued natural +selection. + +Glancing at instincts, marvellous as some are, they offer no greater +difficulty than does corporeal structure on the theory of the natural +selection of successive, slight, but profitable modifications. We can +thus understand why nature moves by graduated steps in endowing +different animals of the same class with their several instincts. I +have attempted to show how much light the principle of gradation throws +on the admirable architectural powers of the hive-bee. Habit no doubt +sometimes comes into play in modifying instincts; but it certainly is +not indispensable, as we see, in the case of neuter insects, which +leave no progeny to inherit the effects of long-continued habit. On the +view of all the species of the same genus having descended from a +common parent, and having inherited much in common, we can understand +how it is that allied species, when placed under considerably different +conditions of life, +yet should follow nearly the same instincts; why the thrush of South +America, for instance, lines her nest with mud like our British +species. On the view of instincts having been slowly acquired through +natural selection we need not marvel at some instincts being apparently +not perfect and liable to mistakes, and at many instincts causing other +animals to suffer. + +If species be only well-marked and permanent varieties, we can at once +see why their crossed offspring should follow the same complex laws in +their degrees and kinds of resemblance to their parents,—in being +absorbed into each other by successive crosses, and in other such +points,—as do the crossed offspring of acknowledged varieties. On the +other hand, these would be strange facts if species have been +independently created, and varieties have been produced by secondary +laws. + +If we admit that the geological record is imperfect in an extreme +degree, then such facts as the record gives, support the theory of +descent with modification. New species have come on the stage slowly +and at successive intervals; and the amount of change, after equal +intervals of time, is widely different in different groups. The +extinction of species and of whole groups of species, which has played +so conspicuous a part in the history of the organic world, almost +inevitably follows on the principle of natural selection; for old forms +will be supplanted by new and improved forms. Neither single species +nor groups of species reappear when the chain of ordinary generation +has once been broken. The gradual diffusion of dominant forms, with the +slow modification of their descendants, causes the forms of life, after +long intervals of time, to appear as if they had changed simultaneously +throughout the world. The fact of the fossil remains of each formation +being in some degree intermediate in character between the +fossils in the formations above and below, is simply explained by their +intermediate position in the chain of descent. The grand fact that all +extinct organic beings belong to the same system with recent beings, +falling either into the same or into intermediate groups, follows from +the living and the extinct being the offspring of common parents. As +the groups which have descended from an ancient progenitor have +generally diverged in character, the progenitor with its early +descendants will often be intermediate in character in comparison with +its later descendants; and thus we can see why the more ancient a +fossil is, the oftener it stands in some degree intermediate between +existing and allied groups. Recent forms are generally looked at as +being, in some vague sense, higher than ancient and extinct forms; and +they are in so far higher as the later and more improved forms have +conquered the older and less improved organic beings in the struggle +for life. Lastly, the law of the long endurance of allied forms on the +same continent,—of marsupials in Australia, of edentata in America, and +other such cases,—is intelligible, for within a confined country, the +recent and the extinct will naturally be allied by descent. + +Looking to geographical distribution, if we admit that there has been +during the long course of ages much migration from one part of the +world to another, owing to former climatal and geographical changes and +to the many occasional and unknown means of dispersal, then we can +understand, on the theory of descent with modification, most of the +great leading facts in Distribution. We can see why there should be so +striking a parallelism in the distribution of organic beings throughout +space, and in their geological succession throughout time; for in both +cases the beings have been connected by the bond of ordinary +generation, and the means of +modification have been the same. We see the full meaning of the +wonderful fact, which must have struck every traveller, namely, that on +the same continent, under the most diverse conditions, under heat and +cold, on mountain and lowland, on deserts and marshes, most of the +inhabitants within each great class are plainly related; for they will +generally be descendants of the same progenitors and early colonists. +On this same principle of former migration, combined in most cases with +modification, we can understand, by the aid of the Glacial period, the +identity of some few plants, and the close alliance of many others, on +the most distant mountains, under the most different climates; and +likewise the close alliance of some of the inhabitants of the sea in +the northern and southern temperate zones, though separated by the +whole intertropical ocean. Although two areas may present the same +physical conditions of life, we need feel no surprise at their +inhabitants being widely different, if they have been for a long period +completely separated from each other; for as the relation of organism +to organism is the most important of all relations, and as the two +areas will have received colonists from some third source or from each +other, at various periods and in different proportions, the course of +modification in the two areas will inevitably be different. + +On this view of migration, with subsequent modification, we can see why +oceanic islands should be inhabited by few species, but of these, that +many should be peculiar. We can clearly see why those animals which +cannot cross wide spaces of ocean, as frogs and terrestrial mammals, +should not inhabit oceanic islands; and why, on the other hand, new and +peculiar species of bats, which can traverse the ocean, should so often +be found on islands far distant from any continent. Such facts +as the presence of peculiar species of bats, and the absence of all +other mammals, on oceanic islands, are utterly inexplicable on the +theory of independent acts of creation. + +The existence of closely allied or representative species in any two +areas, implies, on the theory of descent with modification, that the +same parents formerly inhabited both areas; and we almost invariably +find that wherever many closely allied species inhabit two areas, some +identical species common to both still exist. Wherever many closely +allied yet distinct species occur, many doubtful forms and varieties of +the same species likewise occur. It is a rule of high generality that +the inhabitants of each area are related to the inhabitants of the +nearest source whence immigrants might have been derived. We see this +in nearly all the plants and animals of the Galapagos archipelago, of +Juan Fernandez, and of the other American islands being related in the +most striking manner to the plants and animals of the neighbouring +American mainland; and those of the Cape de Verde archipelago and other +African islands to the African mainland. It must be admitted that these +facts receive no explanation on the theory of creation. + +The fact, as we have seen, that all past and present organic beings +constitute one grand natural system, with group subordinate to group, +and with extinct groups often falling in between recent groups, is +intelligible on the theory of natural selection with its contingencies +of extinction and divergence of character. On these same principles we +see how it is, that the mutual affinities of the species and genera +within each class are so complex and circuitous. We see why certain +characters are far more serviceable than others for classification;—why +adaptive characters, though of paramount importance to the being, are +of hardly any +importance in classification; why characters derived from rudimentary +parts, though of no service to the being, are often of high +classificatory value; and why embryological characters are the most +valuable of all. The real affinities of all organic beings are due to +inheritance or community of descent. The natural system is a +genealogical arrangement, in which we have to discover the lines of +descent by the most permanent characters, however slight their vital +importance may be. + +The framework of bones being the same in the hand of a man, wing of a +bat, fin of the porpoise, and leg of the horse,—the same number of +vertebræ forming the neck of the giraffe and of the elephant,—and +innumerable other such facts, at once explain themselves on the theory +of descent with slow and slight successive modifications. The +similarity of pattern in the wing and leg of a bat, though used for +such different purpose,—in the jaws and legs of a crab,—in the petals, +stamens, and pistils of a flower, is likewise intelligible on the view +of the gradual modification of parts or organs, which were alike in the +early progenitor of each class. On the principle of successive +variations not always supervening at an early age, and being inherited +at a corresponding not early period of life, we can clearly see why the +embryos of mammals, birds, reptiles, and fishes should be so closely +alike, and should be so unlike the adult forms. We may cease marvelling +at the embryo of an air-breathing mammal or bird having branchial slits +and arteries running in loops, like those in a fish which has to +breathe the air dissolved in water, by the aid of well-developed +branchiæ. + +Disuse, aided sometimes by natural selection, will often tend to reduce +an organ, when it has become useless by changed habits or under changed +conditions +of life; and we can clearly understand on this view the meaning of +rudimentary organs. But disuse and selection will generally act on each +creature, when it has come to maturity and has to play its full part in +the struggle for existence, and will thus have little power of acting +on an organ during early life; hence the organ will not be much reduced +or rendered rudimentary at this early age. The calf, for instance, has +inherited teeth, which never cut through the gums of the upper jaw, +from an early progenitor having well-developed teeth; and we may +believe, that the teeth in the mature animal were reduced, during +successive generations, by disuse or by the tongue and palate having +been fitted by natural selection to browse without their aid; whereas +in the calf, the teeth have been left untouched by selection or disuse, +and on the principle of inheritance at corresponding ages have been +inherited from a remote period to the present day. On the view of each +organic being and each separate organ having been specially created, +how utterly inexplicable it is that parts, like the teeth in the +embryonic calf or like the shrivelled wings under the soldered +wing-covers of some beetles, should thus so frequently bear the plain +stamp of inutility! Nature may be said to have taken pains to reveal, +by rudimentary organs and by homologous structures, her scheme of +modification, which it seems that we wilfully will not understand. + +I have now recapitulated the chief facts and considerations which have +thoroughly convinced me that species have changed, and are still slowly +changing by the preservation and accumulation of successive slight +favourable variations. Why, it may be asked, have all the most eminent +living naturalists and geologists rejected this view of the mutability +of species? It cannot be +asserted that organic beings in a state of nature are subject to no +variation; it cannot be proved that the amount of variation in the +course of long ages is a limited quantity; no clear distinction has +been, or can be, drawn between species and well-marked varieties. It +cannot be maintained that species when intercrossed are invariably +sterile, and varieties invariably fertile; or that sterility is a +special endowment and sign of creation. The belief that species were +immutable productions was almost unavoidable as long as the history of +the world was thought to be of short duration; and now that we have +acquired some idea of the lapse of time, we are too apt to assume, +without proof, that the geological record is so perfect that it would +have afforded us plain evidence of the mutation of species, if they had +undergone mutation. + +But the chief cause of our natural unwillingness to admit that one +species has given birth to other and distinct species, is that we are +always slow in admitting any great change of which we do not see the +intermediate steps. The difficulty is the same as that felt by so many +geologists, when Lyell first insisted that long lines of inland cliffs +had been formed, and great valleys excavated, by the slow action of the +coast-waves. The mind cannot possibly grasp the full meaning of the +term of a hundred million years; it cannot add up and perceive the full +effects of many slight variations, accumulated during an almost +infinite number of generations. + +Although I am fully convinced of the truth of the views given in this +volume under the form of an abstract, I by no means expect to convince +experienced naturalists whose minds are stocked with a multitude of +facts all viewed, during a long course of years, from a point of view +directly opposite to mine. It is so easy +to hide our ignorance under such expressions as the “plan of creation,” +“unity of design,” etc., and to think that we give an explanation when +we only restate a fact. Any one whose disposition leads him to attach +more weight to unexplained difficulties than to the explanation of a +certain number of facts will certainly reject my theory. A few +naturalists, endowed with much flexibility of mind, and who have +already begun to doubt on the immutability of species, may be +influenced by this volume; but I look with confidence to the future, to +young and rising naturalists, who will be able to view both sides of +the question with impartiality. Whoever is led to believe that species +are mutable will do good service by conscientiously expressing his +conviction; for only thus can the load of prejudice by which this +subject is overwhelmed be removed. + +Several eminent naturalists have of late published their belief that a +multitude of reputed species in each genus are not real species; but +that other species are real, that is, have been independently created. +This seems to me a strange conclusion to arrive at. They admit that a +multitude of forms, which till lately they themselves thought were +special creations, and which are still thus looked at by the majority +of naturalists, and which consequently have every external +characteristic feature of true species,—they admit that these have been +produced by variation, but they refuse to extend the same view to other +and very slightly different forms. Nevertheless they do not pretend +that they can define, or even conjecture, which are the created forms +of life, and which are those produced by secondary laws. They admit +variation as a _vera causa_ in one case, they arbitrarily reject it in +another, without assigning any distinction in the two cases. The day +will come when this will be given as a curious illustration of +the blindness of preconceived opinion. These authors seem no more +startled at a miraculous act of creation than at an ordinary birth. But +do they really believe that at innumerable periods in the earth’s +history certain elemental atoms have been commanded suddenly to flash +into living tissues? Do they believe that at each supposed act of +creation one individual or many were produced? Were all the infinitely +numerous kinds of animals and plants created as eggs or seed, or as +full grown? and in the case of mammals, were they created bearing the +false marks of nourishment from the mother’s womb? Although naturalists +very properly demand a full explanation of every difficulty from those +who believe in the mutability of species, on their own side they ignore +the whole subject of the first appearance of species in what they +consider reverent silence. + +It may be asked how far I extend the doctrine of the modification of +species. The question is difficult to answer, because the more distinct +the forms are which we may consider, by so much the arguments fall away +in force. But some arguments of the greatest weight extend very far. +All the members of whole classes can be connected together by chains of +affinities, and all can be classified on the same principle, in groups +subordinate to groups. Fossil remains sometimes tend to fill up very +wide intervals between existing orders. Organs in a rudimentary +condition plainly show that an early progenitor had the organ in a +fully developed state; and this in some instances necessarily implies +an enormous amount of modification in the descendants. Throughout whole +classes various structures are formed on the same pattern, and at an +embryonic age the species closely resemble each other. Therefore I +cannot doubt that the theory of descent with modification +embraces all the members of the same class. I believe that animals have +descended from at most only four or five progenitors, and plants from +an equal or lesser number. + +Analogy would lead me one step further, namely, to the belief that all +animals and plants have descended from some one prototype. But analogy +may be a deceitful guide. Nevertheless all living things have much in +common, in their chemical composition, their germinal vesicles, their +cellular structure, and their laws of growth and reproduction. We see +this even in so trifling a circumstance as that the same poison often +similarly affects plants and animals; or that the poison secreted by +the gall-fly produces monstrous growths on the wild rose or oak-tree. +Therefore I should infer from analogy that probably all the organic +beings which have ever lived on this earth have descended from some one +primordial form, into which life was first breathed. + +When the views entertained in this volume on the origin of species, or +when analogous views are generally admitted, we can dimly foresee that +there will be a considerable revolution in natural history. +Systematists will be able to pursue their labours as at present; but +they will not be incessantly haunted by the shadowy doubt whether this +or that form be in essence a species. This I feel sure, and I speak +after experience, will be no slight relief. The endless disputes +whether or not some fifty species of British brambles are true species +will cease. Systematists will have only to decide (not that this will +be easy) whether any form be sufficiently constant and distinct from +other forms, to be capable of definition; and if definable, whether the +differences be sufficiently important to deserve a specific name. This +latter point will become a far more essential consideration +than it is at present; for differences, however slight, between any two +forms, if not blended by intermediate gradations, are looked at by most +naturalists as sufficient to raise both forms to the rank of species. +Hereafter we shall be compelled to acknowledge that the only +distinction between species and well-marked varieties is, that the +latter are known, or believed, to be connected at the present day by +intermediate gradations, whereas species were formerly thus connected. +Hence, without quite rejecting the consideration of the present +existence of intermediate gradations between any two forms, we shall be +led to weigh more carefully and to value higher the actual amount of +difference between them. It is quite possible that forms now generally +acknowledged to be merely varieties may hereafter be thought worthy of +specific names, as with the primrose and cowslip; and in this case +scientific and common language will come into accordance. In short, we +shall have to treat species in the same manner as those naturalists +treat genera, who admit that genera are merely artificial combinations +made for convenience. This may not be a cheering prospect; but we shall +at least be freed from the vain search for the undiscovered and +undiscoverable essence of the term species. + +The other and more general departments of natural history will rise +greatly in interest. The terms used by naturalists of affinity, +relationship, community of type, paternity, morphology, adaptive +characters, rudimentary and aborted organs, etc., will cease to be +metaphorical, and will have a plain signification. When we no longer +look at an organic being as a savage looks at a ship, as at something +wholly beyond his comprehension; when we regard every production of +nature as one which has had a history; when we contemplate every +complex structure +and instinct as the summing up of many contrivances, each useful to the +possessor, nearly in the same way as when we look at any great +mechanical invention as the summing up of the labour, the experience, +the reason, and even the blunders of numerous workmen; when we thus +view each organic being, how far more interesting, I speak from +experience, will the study of natural history become! + +A grand and almost untrodden field of inquiry will be opened, on the +causes and laws of variation, on correlation of growth, on the effects +of use and disuse, on the direct action of external conditions, and so +forth. The study of domestic productions will rise immensely in value. +A new variety raised by man will be a far more important and +interesting subject for study than one more species added to the +infinitude of already recorded species. Our classifications will come +to be, as far as they can be so made, genealogies; and will then truly +give what may be called the plan of creation. The rules for classifying +will no doubt become simpler when we have a definite object in view. We +possess no pedigrees or armorial bearings; and we have to discover and +trace the many diverging lines of descent in our natural genealogies, +by characters of any kind which have long been inherited. Rudimentary +organs will speak infallibly with respect to the nature of long-lost +structures. Species and groups of species, which are called aberrant, +and which may fancifully be called living fossils, will aid us in +forming a picture of the ancient forms of life. Embryology will reveal +to us the structure, in some degree obscured, of the prototypes of each +great class. + +When we can feel assured that all the individuals of the same species, +and all the closely allied species of most genera, have within a not +very remote period descended +from one parent, and have migrated from some one birthplace; and when +we better know the many means of migration, then, by the light which +geology now throws, and will continue to throw, on former changes of +climate and of the level of the land, we shall surely be enabled to +trace in an admirable manner the former migrations of the inhabitants +of the whole world. Even at present, by comparing the differences of +the inhabitants of the sea on the opposite sides of a continent, and +the nature of the various inhabitants of that continent in relation to +their apparent means of immigration, some light can be thrown on +ancient geography. + +The noble science of Geology loses glory from the extreme imperfection +of the record. The crust of the earth with its embedded remains must +not be looked at as a well-filled museum, but as a poor collection made +at hazard and at rare intervals. The accumulation of each great +fossiliferous formation will be recognised as having depended on an +unusual concurrence of circumstances, and the blank intervals between +the successive stages as having been of vast duration. But we shall be +able to gauge with some security the duration of these intervals by a +comparison of the preceding and succeeding organic forms. We must be +cautious in attempting to correlate as strictly contemporaneous two +formations, which include few identical species, by the general +succession of their forms of life. As species are produced and +exterminated by slowly acting and still existing causes, and not by +miraculous acts of creation and by catastrophes; and as the most +important of all causes of organic change is one which is almost +independent of altered and perhaps suddenly altered physical +conditions, namely, the mutual relation of organism to organism,—the +improvement of one being entailing the improvement or the extermination +of +others; it follows, that the amount of organic change in the fossils of +consecutive formations probably serves as a fair measure of the lapse +of actual time. A number of species, however, keeping in a body might +remain for a long period unchanged, whilst within this same period, +several of these species, by migrating into new countries and coming +into competition with foreign associates, might become modified; so +that we must not overrate the accuracy of organic change as a measure +of time. During early periods of the earth’s history, when the forms of +life were probably fewer and simpler, the rate of change was probably +slower; and at the first dawn of life, when very few forms of the +simplest structure existed, the rate of change may have been slow in an +extreme degree. The whole history of the world, as at present known, +although of a length quite incomprehensible by us, will hereafter be +recognised as a mere fragment of time, compared with the ages which +have elapsed since the first creature, the progenitor of innumerable +extinct and living descendants, was created. + +In the distant future I see open fields for far more important +researches. Psychology will be based on a new foundation, that of the +necessary acquirement of each mental power and capacity by gradation. +Light will be thrown on the origin of man and his history. + +Authors of the highest eminence seem to be fully satisfied with the +view that each species has been independently created. To my mind it +accords better with what we know of the laws impressed on matter by the +Creator, that the production and extinction of the past and present +inhabitants of the world should have been due to secondary causes, like +those determining the birth and death of the individual. When I view +all beings not as special creations, but as the lineal descendants of +some few beings which lived long before the +first bed of the Silurian system was deposited, they seem to me to +become ennobled. Judging from the past, we may safely infer that not +one living species will transmit its unaltered likeness to a distant +futurity. And of the species now living very few will transmit progeny +of any kind to a far distant futurity; for the manner in which all +organic beings are grouped, shows that the greater number of species of +each genus, and all the species of many genera, have left no +descendants, but have become utterly extinct. We can so far take a +prophetic glance into futurity as to foretel that it will be the common +and widely-spread species, belonging to the larger and dominant groups, +which will ultimately prevail and procreate new and dominant species. +As all the living forms of life are the lineal descendants of those +which lived long before the Silurian epoch, we may feel certain that +the ordinary succession by generation has never once been broken, and +that no cataclysm has desolated the whole world. Hence we may look with +some confidence to a secure future of equally inappreciable length. And +as natural selection works solely by and for the good of each being, +all corporeal and mental endowments will tend to progress towards +perfection. + +It is interesting to contemplate an entangled bank, clothed with many +plants of many kinds, with birds singing on the bushes, with various +insects flitting about, and with worms crawling through the damp earth, +and to reflect that these elaborately constructed forms, so different +from each other, and dependent on each other in so complex a manner, +have all been produced by laws acting around us. These laws, taken in +the largest sense, being Growth with Reproduction; Inheritance which is +almost implied by reproduction; Variability from the indirect and +direct action of the external conditions +of life, and from use and disuse; a Ratio of Increase so high as to +lead to a Struggle for Life, and as a consequence to Natural Selection, +entailing Divergence of Character and the Extinction of less-improved +forms. Thus, from the war of nature, from famine and death, the most +exalted object which we are capable of conceiving, namely, the +production of the higher animals, directly follows. There is grandeur +in this view of life, with its several powers, having been originally +breathed into a few forms or into one; and that, whilst this planet has +gone cycling on according to the fixed law of gravity, from so simple a +beginning endless forms most beautiful and most wonderful have been, +and are being, evolved. diff --git a/data/science-v2/einstein-1924-validation.txt b/data/science-v2/einstein-1924-validation.txt new file mode 100644 index 0000000..c4761a7 --- /dev/null +++ b/data/science-v2/einstein-1924-validation.txt @@ -0,0 +1,345 @@ +This four-dimensional “world” bears a close similarity to the +three-dimensional “space” of (Euclidean) analytical geometry. If we +introduce into the latter a new Cartesian co-ordinate system (_x′_1, +_x′_2, _x′_3) with the same origin, then _x′_1, _x′_2, _x′_3, are +linear homogeneous functions of _x_1, _x_2, _x_3 which identically +satisfy the equation + +_x_1′2 + _x_2′2 + _x_3′2 = _x_12 + _x_22 + _x_32 + +The analogy with (12) is a complete one. We can regard Minkowski’s +“world” in a formal manner as a four-dimensional Euclidean space (with +an imaginary time coordinate); the Lorentz transformation corresponds +to a “rotation” of the co-ordinate system in the four-dimensional +“world.” + +From a systematic theoretical point of view, we may imagine the process +of evolution of an empirical science to be a continuous process of +induction. Theories are evolved and are expressed in short compass as +statements of a large number of individual observations in the form of +empirical laws, from which the general laws can be ascertained by +comparison. Regarded in this way, the development of a science bears +some resemblance to the compilation of a classified catalogue. It is, +as it were, a purely empirical enterprise. + +But this point of view by no means embraces the whole of the actual +process; for it slurs over the important part played by intuition and +deductive thought in the development of an exact science. As soon as a +science has emerged from its initial stages, theoretical advances are +no longer achieved merely by a process of arrangement. Guided by +empirical data, the investigator rather develops a system of thought +which, in general, is built up logically from a small number of +fundamental assumptions, the so-called axioms. We call such a system of +thought a _theory_. The theory finds the justification for its +existence in the fact that it correlates a large number of single +observations, and it is just here that the “truth” of the theory lies. + +Corresponding to the same complex of empirical data, there may be +several theories, which differ from one another to a considerable +extent. But as regards the deductions from the theories which are +capable of being tested, the agreement between the theories may be so +complete that it becomes difficult to find any deductions in which the +two theories differ from each other. As an example, a case of general +interest is available in the province of biology, in the Darwinian +theory of the development of species by selection in the struggle for +existence, and in the theory of development which is based on the +hypothesis of the hereditary transmission of acquired characters. + +We have another instance of far-reaching agreement between the +deductions from two theories in Newtonian mechanics on the one hand, +and the general theory of relativity on the other. This agreement goes +so far, that up to the present we have been able to find only a few +deductions from the general theory of relativity which are capable of +investigation, and to which the physics of pre-relativity days does not +also lead, and this despite the profound difference in the fundamental +assumptions of the two theories. In what follows, we shall again +consider these important deductions, and we shall also discuss the +empirical evidence appertaining to them which has hitherto been +obtained. + +(_a_) Motion of the Perihelion of Mercury + +According to Newtonian mechanics and Newton’s law of gravitation, a +planet which is revolving round the sun would describe an ellipse round +the latter, or, more correctly, round the common centre of gravity of +the sun and the planet. In such a system, the sun, or the common centre +of gravity, lies in one of the foci of the orbital ellipse in such a +manner that, in the course of a planet-year, the distance sun-planet +grows from a minimum to a maximum, and then decreases again to a +minimum. If instead of Newton’s law we insert a somewhat different law +of attraction into the calculation, we find that, according to this new +law, the motion would still take place in such a manner that the +distance sun-planet exhibits periodic variations; but in this case the +angle described by the line joining sun and planet during such a period +(from perihelion—closest proximity to the sun—to perihelion) would +differ from 360°. The line of the orbit would not then be a closed one +but in the course of time it would fill up an annular part of the +orbital plane, viz. between the circle of least and the circle of +greatest distance of the planet from the sun. + +According also to the general theory of relativity, which differs of +course from the theory of Newton, a small variation from the +Newton-Kepler motion of a planet in its orbit should take place, and in +such away, that the angle described by the radius sun-planet between +one perhelion and the next should exceed that corresponding to one +complete revolution by an amount given by + +image051 + +(_N.B._—One complete revolution corresponds to the angle 2π in the +absolute angular measure customary in physics, and the above expression +given the amount by which the radius sun-planet exceeds this angle +during the interval between one perihelion and the next.) In this +expression _a_ represents the major semi-axis of the ellipse, _e_ its +eccentricity, _c_ the velocity of light, and _T_ the period of +revolution of the planet. Our result may also be stated as follows: +According to the general theory of relativity, the major axis of the +ellipse rotates round the sun in the same sense as the orbital motion +of the planet. Theory requires that this rotation should amount to 43 +seconds of arc per century for the planet Mercury, but for the other +Planets of our solar system its magnitude should be so small that it +would necessarily escape detection.[26] + + [26] Especially since the next planet Venus has an orbit that is + almost an exact circle, which makes it more difficult to locate the + perihelion with precision. + +In point of fact, astronomers have found that the theory of Newton does +not suffice to calculate the observed motion of Mercury with an +exactness corresponding to that of the delicacy of observation +attainable at the present time. After taking account of all the +disturbing influences exerted on Mercury by the remaining planets, it +was found (Leverrier: 1859; and Newcomb: 1895) that an unexplained +perihelial movement of the orbit of Mercury remained over, the amount +of which does not differ sensibly from the above mentioned +43 seconds +of arc per century. The uncertainty of the empirical result amounts to +a few seconds only. + +(_b_) Deflection of Light by a Gravitational Field + +image052 + +In Section XXII it has been already mentioned that according to the +general theory of relativity, a ray of light will experience a +curvature of its path when passing through a gravitational field, this +curvature being similar to that experienced by the path of a body which +is projected through a gravitational field. As a result of this theory, +we should expect that a ray of light which is passing close to a +heavenly body would be deviated towards the latter. For a ray of light +which passes the sun at a distance of Δ sun-radii from its centre, the +angle of deflection (α) should amount to + +image053 + +It may be added that, according to the theory, half of this deflection +is produced by the Newtonian field of attraction of the sun, and the +other half by the geometrical modification (“curvature”) of space +caused by the sun. + +This result admits of an experimental test by means of the photographic +registration of stars during a total eclipse of the sun. The only +reason why we must wait for a total eclipse is because at every other +time the atmosphere is so strongly illuminated by the light from the +sun that the stars situated near the sun’s disc are invisible. The +predicted effect can be seen clearly from the accompanying diagram. If +the sun (_S_) were not present, a star which is practically infinitely +distant would be seen in the direction _D_1, as observed front the +earth. But as a consequence of the deflection of light from the star by +the sun, the star will be seen in the direction _D_2, _i.e._ at a +somewhat greater distance from the centre of the sun than corresponds +to its real position. + +In practice, the question is tested in the following way. The stars in +the neighbourhood of the sun are photographed during a solar eclipse. + +In addition, a second photograph of the same stars is taken when the +sun is situated at another position in the sky, _i.e._ a few months +earlier or later. As compared with the standard photograph, the +positions of the stars on the eclipse-photograph ought to appear +displaced radially outwards (away from the centre of the sun) by an +amount corresponding to the angle _a_. + +We are indebted to the [British] Royal Society and to the Royal +Astronomical Society for the investigation of this important deduction. +Undaunted by the [first world] war and by difficulties of both a +material and a psychological nature aroused by the war, these societies +equipped two expeditions—to Sobral (Brazil), and to the island of +Principe (West Africa)—and sent several of Britain’s most celebrated +astronomers (Eddington, Cottingham, Crommelin, Davidson), in order to +obtain photographs of the solar eclipse of 29th May, 1919. The relative +discrepancies to be expected between the stellar photographs obtained +during the eclipse and the comparison photographs amounted to a few +hundredths of a millimetre only. Thus great accuracy was necessary in +making the adjustments required for the taking of the photographs, and +in their subsequent measurement. + +The results of the measurements confirmed the theory in a thoroughly +satisfactory manner. The rectangular components of the observed and of +the calculated deviations of the stars (in seconds of arc) are set +forth in the following table of results: + +image054 + +(_c_) Displacement of Spectral Lines Towards the Red + +In Section XXIII it has been shown that in a system _K′_ which is in +rotation with regard to a Galileian system _K_, clocks of identical +construction, and which are considered at rest with respect to the +rotating reference-body, go at rates which are dependent on the +positions of the clocks. We shall now examine this dependence +quantitatively. A clock, which is situated at a distance r from the +centre of the disc, has a velocity relative to _K_ which is given by + +_v_ = ω_r_, + +where ω represents the angular velocity of rotation of the disc _K′_ +with respect to _K_. If _v_0, represents the number of ticks of the +clock per unit time (“rate” of the clock) relative to _K_ when the +clock is at rest, then the “rate” of the clock (_v_) when it is moving +relative to _K_ with a velocity _v_, but at rest with respect to the +disc, will, in accordance with Section XII, be given by + +image055 + +or with sufficient accuracy by + +image056 + +This expression may also be stated in the following form: + +image057 + +If we represent the difference of potential of the centrifugal force +between the position of the clock and the centre of the disc by φ, +_i.e._ the work, considered negatively, which must be performed on the +unit of mass against the centrifugal force in order to transport it +from the position of the clock on the rotating disc to the centre of +the disc, then we have + +image058 + +From this it follows that + +image059 + +In the first place, we see from this expression that two clocks of +identical construction will go at different rates when situated at +different distances from the centre of the disc. This result is also +valid from the standpoint of an observer who is rotating with the disc. + +Now, as judged from the disc, the latter is in a gravitational field of +potential φ, hence the result we have obtained will hold quite +generally for gravitational fields. Furthermore, we can regard an atom +which is emitting spectral lines as a clock, so that the following +statement will hold: + +_An atom absorbs or emits light of a frequency which is dependent on +the potential of the gravitational field in which it is situated._ + +The frequency of an atom situated on the surface of a heavenly body +will be somewhat less than the frequency of an atom of the same element +which is situated in free space (or on the surface of a smaller +celestial body). + +Now φ = – _K (M/r)_, where _K_ is Newton’s constant of gravitation, and +_M_ is the mass of the heavenly body. Thus a displacement towards the +red ought to take place for spectral lines produced at the surface of +stars as compared with the spectral lines of the same element produced +at the surface of the earth, the amount of this displacement being + +image060 + +For the sun, the displacement towards the red predicted by theory +amounts to about two millionths of the wave-length. A trustworthy +calculation is not possible in the case of the stars, because in +general neither the mass _M_ nor the radius _r_ are known. + +It is an open question whether or not this effect exists, and at the +present time (1920) astronomers are working with great zeal towards the +solution. Owing to the smallness of the effect in the case of the sun, +it is difficult to form an opinion as to its existence. Whereas Grebe +and Bachem (Bonn), as a result of their own measurements and those of +Evershed and Schwarzschild on the cyanogen bands, have placed the +existence of the effect almost beyond doubt, while other investigators, +particularly St. John, have been led to the opposite opinion in +consequence of their measurements. + +Mean displacements of lines towards the less refrangible end of the +spectrum are certainly revealed by statistical investigations of the +fixed stars; but up to the present the examination of the available +data does not allow of any definite decision being arrived at, as to +whether or not these displacements are to be referred in reality to the +effect of gravitation. The results of observation have been collected +together, and discussed in detail from the standpoint of the question +which has been engaging our attention here, in a paper by E. Freundlich +entitled “Zur Prüfung der allgemeinen Relativitäts-Theorie” (_Die +Naturwissenschaften_, 1919, No. 35, p. 520: Julius Springer, Berlin). + +At all events, a definite decision will be reached during the next few +years. If the displacement of spectral lines towards the red by the +gravitational potential does not exist, then the general theory of +relativity will be untenable. On the other hand, if the cause of the +displacement of spectral lines be definitely traced to the +gravitational potential, then the study of this displacement will +furnish us with important information as to the mass of the heavenly +bodies.[27] + + [27] The displacement of spectral lines towards the red end of the + spectrum was definitely established by Adams in 1924, by observations + on the dense companion of Sirius, for which the effect is about thirty + times greater than for the Sun. R.W.L.—translator + +Since the publication of the first edition of this little book, our +knowledge about the structure of space in the large (“cosmological +problem”) has had an important development, which ought to be mentioned +even in a popular presentation of the subject. + +My original considerations on the subject were based on two hypotheses: + +(1) There exists an average density of matter in the whole of space +which is everywhere the same and different from zero. + +(2) The magnitude (“radius”) of space is independent of time. + +Both these hypotheses proved to be consistent, according to the general +theory of relativity, but only after a hypothetical term was added to +the field equations, a term which was not required by the theory as +such nor did it seem natural from a theoretical point of view +(“cosmological term of the field equations”). + +Hypothesis (2) appeared unavoidable to me at the time, since I thought +that one would get into bottomless speculations if one departed from +it. + +However, already in the ’twenties, the Russian mathematician Friedman +showed that a different hypothesis was natural from a purely +theoretical point of view. He realized that it was possible to preserve +hypothesis (1) without introducing the less natural cosmological term +into the field equations of gravitation, if one was ready to drop +hypothesis (2). Namely, the original field equations admit a solution +in which the “world radius” depends on time (expanding space). In that +sense one can say, according to Friedman, that the theory demands an +expansion of space. + +A few years later Hubble showed, by a special investigation of the +extra-galactic nebulae (“milky ways”), that the spectral lines emitted +showed a red shift which increased regularly with the distance of the +nebulae. This can be interpreted in regard to our present knowledge +only in the sense of Doppler’s principle, as an expansive motion of the +system of stars in the large—as required, according to Friedman, by the +field equations of gravitation. Hubble’s discovery can, therefore, be +considered to some extent as a confirmation of the theory. + +There does arise, however, a strange difficulty. The interpretation of +the galactic line-shift discovered by Hubble as an expansion (which can +hardly be doubted from a theoretical point of view), leads to an origin +of this expansion which lies “only” about 109 years ago, while physical +astronomy makes it appear likely that the development of individual +stars and systems of stars takes considerably longer. It is in no way +known how this incongruity is to be overcome. + +I further want to remark that the theory of expanding space, together +with the empirical data of astronomy, permit no decision to be reached +about the finite or infinite character of (three-dimensional) space, +while the original “static” hypothesis of space yielded the closure +(finiteness) of space. diff --git a/data/science-v2/manifest.json b/data/science-v2/manifest.json new file mode 100644 index 0000000..7c421d3 --- /dev/null +++ b/data/science-v2/manifest.json @@ -0,0 +1,148 @@ +{ + "protocol": "science-v2", + "legacy_protocol": "science-v1 in data/science/ (unchanged)", + "rules": [ + "extract the Project Gutenberg body; verify its SHA-256", + "keep text from the start marker line (last of the expected occurrences) up to the end marker line", + "split paragraphs on blank lines", + "drop paragraphs with no lowercase letter (headings, numerals)", + "drop paragraphs entirely enclosed in square brackets (illustrations, editorial notes)", + "drop paragraphs of two or more lines where at least half of the lines end in a number (index, contents, tables)", + "train/validation books: first ~90% of kept bytes at a paragraph boundary to train, rest to validation", + "test book: all kept paragraphs; never used for training, checkpoint selection or tuning", + "remove held-out paragraphs that are exact (normalized) or near (>=50% word 8-gram) duplicates of training or validation text" + ], + "usage": { + "train": "optimization only", + "validation": "checkpoint selection", + "test": "final evaluation only, once per selected checkpoint" + }, + "sources": [ + { + "name": "darwin-1859", + "title": "On the Origin of Species (1859)", + "author": "Charles Darwin", + "role": "train_validation", + "url": "https://www.gutenberg.org/cache/epub/1228/pg1228.txt", + "raw_sha256_observed": "362142cb7c6f90850477520cfe2a62fe423ae92ea8eb6d9cd8b30ddb39de60b9", + "body_sha256": "9ff49617f2a0b52b18a2a3b6f09c8a323358160eb98e4e336969b1100768d73f", + "start_marker": { + "line": "ON THE ORIGIN OF SPECIES.", + "occurrences": 1 + }, + "end_marker": { + "line": "INDEX.", + "occurrences": 1 + }, + "main_text_paragraphs": 711, + "dropped_paragraphs": { + "no_lowercase_heading": 15 + }, + "kept_paragraphs": 696, + "train_paragraphs": 623 + }, + { + "name": "einstein-1924", + "title": "Relativity: The Special and General Theory (Lawson translation)", + "author": "Albert Einstein", + "role": "train_validation", + "url": "https://www.gutenberg.org/cache/epub/30155/pg30155.txt", + "raw_sha256_observed": "86ed8156239455cbb6ed33e06097707d3ab7c40d46e9aa5ff32d3c3f94ef68fd", + "body_sha256": "384e707c77e521bec3cceee2b2b6a0e88f271e0f27752a3ec3790ba06554a36e", + "start_marker": { + "line": "PREFACE", + "occurrences": 1 + }, + "end_marker": { + "line": "_K_ = co-ordinate system", + "occurrences": 1 + }, + "main_text_paragraphs": 542, + "dropped_paragraphs": { + "no_lowercase_heading": 79 + }, + "kept_paragraphs": 463, + "train_paragraphs": 404 + }, + { + "name": "faraday-1861", + "title": "The Chemical History of a Candle", + "author": "Michael Faraday", + "role": "test", + "url": "https://www.gutenberg.org/cache/epub/14474/pg14474.txt", + "raw_sha256_observed": "ee17c709443e87dd8d36e2f8d7065e54344bc5182e0dc856c08f058fbd9b013f", + "body_sha256": "5580d9bfa409ee7f2f7d79b3f27921470a692d6e549a55052adf959547d7ade5", + "start_marker": { + "line": "THE CHEMICAL HISTORY OF A CANDLE", + "occurrences": 1 + }, + "end_marker": { + "line": "NOTES.", + "occurrences": 2 + }, + "main_text_paragraphs": 217, + "dropped_paragraphs": { + "bracketed_editorial": 40, + "no_lowercase_heading": 15, + "numeric_list_or_table": 2 + }, + "kept_paragraphs": 160 + } + ], + "files": { + "train.txt": { + "sha256": "65182db3ac268ec990f79a32fd9151c0de332fb8c411138af9ef965c9f80265e", + "bytes": 969506 + }, + "validation.txt": { + "sha256": "6457fe8eaf9011ef75600da24ed0925cfb5e9464fa1c9e453785eecfc9592b17", + "bytes": 105666 + }, + "test.txt": { + "sha256": "9e98ba26c35173180e6a43cc3b49096bea57f62cb1e8013bf5ac1e0b03d53d6b", + "bytes": 211849 + }, + "darwin-1859-validation.txt": { + "sha256": "dea22adfd1f580ec7f3c6a87011bf250436e49371e38efa44209942a4120d59a", + "bytes": 87711 + }, + "einstein-1924-validation.txt": { + "sha256": "c6728bc6890bc714df5ebbe2030766126ad4f08b57c15b551fb948e4e405f9c8", + "bytes": 17953 + } + }, + "contamination_checks": { + "darwin-1859_validation_vs_train": { + "paragraphs": 73, + "exact_duplicates": 0, + "near_duplicates": 0, + "ngram_n": 8, + "near_threshold": 0.5, + "mean_ngram_overlap": 0.006235369217520088 + }, + "einstein-1924_validation_vs_train": { + "paragraphs": 59, + "exact_duplicates": 0, + "near_duplicates": 0, + "ngram_n": 8, + "near_threshold": 0.5, + "mean_ngram_overlap": 0.009755296544751041 + }, + "test_vs_train": { + "paragraphs": 160, + "exact_duplicates": 0, + "near_duplicates": 0, + "ngram_n": 8, + "near_threshold": 0.5, + "mean_ngram_overlap": 1.812480742392112e-05 + }, + "test_vs_validation": { + "paragraphs": 160, + "exact_duplicates": 0, + "near_duplicates": 0, + "ngram_n": 8, + "near_threshold": 0.5, + "mean_ngram_overlap": 1.812480742392112e-05 + } + } +} diff --git a/data/science-v2/test.txt b/data/science-v2/test.txt new file mode 100644 index 0000000..fc4e887 --- /dev/null +++ b/data/science-v2/test.txt @@ -0,0 +1,3166 @@ +I purpose, in return for the honour you do us by coming to see what are +our proceedings here, to bring before you, in the course of these +lectures, the Chemical History of a Candle. I have taken this subject on a +former occasion; and were it left to my own will, I should prefer to +repeat it almost every year—so abundant is the interest that attaches +itself to the subject, so wonderful are the varieties of outlet which it +offers into the various departments of philosophy. There is not a law +under which any part of this universe is governed which does not come into +play, and is touched upon in these phenomena. There is no better, there is +no more open door by which you can enter into the study of natural +philosophy, than by considering the physical phenomena of a candle. I +trust, therefore, I shall not disappoint you in choosing this for my +subject rather than any newer topic, which could not be better, were it +even so good. + +And before proceeding, let me say this also—that though our subject be so +great, and our intention that of treating it honestly, seriously, and +philosophically, yet I mean to pass away from all those who are seniors +amongst us. I claim the privilege of speaking to juveniles as a juvenile +myself. I have done so on former occasions—and, if you please, I shall do +so again. And though I stand here with the knowledge of having the words I +utter given to the world, yet that shall not deter me from speaking in the +same familiar way to those whom I esteem nearest to me on this occasion. + +And now, my boys and girls, I must first tell you of what candles are +made. Some are great curiosities. I have here some bits of timber, +branches of trees particularly famous for their burning. And here you see +a piece of that very curious substance taken out of some of the bogs in +Ireland, called _candle-wood_,—a hard, strong, excellent wood, evidently +fitted for good work as a resister of force, and yet withal burning so +well that where it is found they make splinters of it, and torches, since +it burns like a candle, and gives a very good light indeed. And in this +wood we have one of the most beautiful illustrations of the general nature +of a candle that I can possibly give. The fuel provided, the means of +bringing that fuel to the place of chemical action, the regular and +gradual supply of air to that place of action—heat and light—all +produced by a little piece of wood of this kind, forming, in fact, a +natural candle. + +But we must speak of candles as they are in commerce. Here are a couple of +candles commonly called dips. They are made of lengths of cotton cut off, +hung up by a loop, dipped into melted tallow, taken out again and cooled, +then re-dipped until there is an accumulation of tallow round the cotton. +In order that you may have an idea of the various characters of these +candles, you see these which I hold in my hand—they are very small, and +very curious. They are, or were, the candles used by the miners in coal +mines. In olden times the miner had to find his own candles; and it was +supposed that a small candle would not so soon set fire to the fire-damp +in the coal mines as a large one; and for that reason, as well as for +economy’s sake, he had candles made of this sort—20, 30, 40, or 60 to the +pound. They have been replaced since then by the steel-mill, and then by +the Davy-lamp, and other safety-lamps of various kinds. I have here a +candle that was taken out of the _Royal George_[1], it is said, by Colonel +Pasley. It has been sunk in the sea for many years, subject to the action +of salt water. It shews you how well candles may be preserved; for though +it is cracked about and broken a good deal, yet, when lighted, it goes on +burning regularly, and the tallow resumes its natural condition as soon as +it is fused. + +Mr. Field, of Lambeth, has supplied me abundantly with beautiful +illustrations of the candle and its materials. I shall therefore now refer +to them. And, first, there is the suet—the fat of the ox—Russian tallow, +I believe, employed in the manufacture of these dips, which Gay Lussac, or +some one who entrusted him with his knowledge, converted into that +beautiful substance, stearin, which you see lying beside it. A candle, you +know, is not now a greasy thing like an ordinary tallow candle, but a +clean thing, and you may almost scrape off and pulverise the drops which +fall from it without soiling anything. This is the process he +adopted[2]:—The fat or tallow is first boiled with quick-lime, and made +into a soap, and then the soap is decomposed by sulphuric acid, which +takes away the lime, and leaves the fat re-arranged as stearic acid, +whilst a quantity of glycerin is produced at the same time. +Glycerin—absolutely a sugar, or a substance similar to sugar—comes out +of the tallow in this chemical change. The oil is then pressed out of it; +and you see here this series of pressed cakes, shewing how beautifully the +impurities are carried out by the oily part as the pressure goes on +increasing, and at last you have left that substance which is melted, and +cast into candles as here represented. The candle I have in my hand is a +stearin candle, made of stearin from tallow in the way I have told you. +Then here is a sperm candle, which comes from the purified oil of the +spermaceti whale. Here also are yellow bees-wax and refined bees-wax, from +which candles are made. Here, too, is that curious substance called +paraffin, and some paraffin candles made of paraffin obtained from the +bogs of Ireland. I have here also a substance brought from Japan, since we +have forced an entrance into that out-of-the-way place—a sort of wax +which a kind friend has sent me, and which forms a new material for the +manufacture of candles. + +And how are these candles made? I have told you about dips, and I will +shew you how moulds are made. Let us imagine any of these candles to be +made of materials which can be cast. “Cast!” you say. “Why, a candle is a +thing that melts; and surely if you can melt it, you can cast it.” Not so. +It is wonderful, in the progress of manufacture, and in the consideration +of the means best fitted to produce the required result, how things turn +up which one would not expect beforehand. Candles cannot always be cast. A +wax candle can never be cast. It is made by a particular process, which I +can illustrate in a minute or two: but I must not spend much time on it. +Wax is a thing which, burning so well, and melting so easily in a candle, +cannot be cast. However, let us take a material that can be cast. Here is +a frame, with a number of moulds fastened in it. The first thing to be +done is to put a wick through them. Here is one—a plaited wick, which +does not require snuffing[3]—supported by a little wire. It goes to the +bottom, where it is pegged in—the little peg holding the cotton tight, +and stopping the aperture, so that nothing fluid shall run out. At the +upper part there is a little bar placed across, which stretches the cotton +and holds it in the mould. The tallow is then melted, and the moulds are +filled. After a certain time, when the moulds are cool, the excess of +tallow is poured off at one corner, and then cleaned off altogether, and +the ends of the wick cut away. The candles alone then remain in the mould, +and you have only to upset them, as I am doing, when out they tumble, for +the candles are made in the form of cones, being narrower at the top than +at the bottom; so that what with their form and their own shrinking, they +only need a little shaking, and out they fall. In the same way are made +these candles of stearin and of paraffin. It is a curious thing to see how +wax candles are made. A lot of cottons are hung upon frames, as you see +here, and covered with metal tags at the ends to keep the wax from +covering the cotton in those places. These are carried to a heater, where +the wax is melted. As you see, the frames can turn round; and as they +turn, a man takes a vessel of wax and pours it first down one, and then +the next and the next, and so on. When he has gone once round, if it is +sufficiently cool, he gives the first a second coat, and so on until they +are all of the required thickness. When they have been thus clothed, or +fed, or made up to that thickness, they are taken off, and placed +elsewhere. I have here, by the kindness of Mr. Field, several specimens of +these candles. Here is one only half-finished. They are then taken down, +and well rolled upon a fine stone slab, and the conical top is moulded by +properly shaped tubes, and the bottoms cut off and trimmed. This is done +so beautifully that they can make candles in this way weighing exactly +four, or six, to the pound, or any number they please. + +We must not, however, take up more time about the mere manufacture, but go +a little further into the matter. I have not yet referred you to luxuries +in candles (for there is such a thing as luxury in candles). See how +beautifully these are coloured: you see here mauve, magenta, and all the +chemical colours recently introduced, applied to candles. You observe, +also, different forms employed. Here is a fluted pillar most beautifully +shaped; and I have also here some candles sent me by Mr. Pearsall, which +are ornamented with designs upon them, so that as they burn you have as it +were a glowing sun above, and a bouquet of flowers beneath. All, however, +that is fine and beautiful is not useful. These fluted candles, pretty as +they are, are bad candles; they are bad because of their external shape. +Nevertheless, I shew you these specimens sent to me from kind friends on +all sides, that you may see what is done, and what may be done in this or +that direction; although, as I have said, when we come to these +refinements, we are obliged to sacrifice a little in utility. + +Now, as to the light of the candle. We will light one or two, and set them +at work in the performance of their proper functions. You observe a candle +is a very different thing from a lamp. With a lamp you take a little oil, +fill your vessel, put in a little moss or some cotton prepared by +artificial means, and then light the top of the wick. When the flame runs +down the cotton to the oil, it gets extinguished, but it goes on burning +in the part above. Now, I have no doubt you will ask, how is it that the +oil, which will not burn of itself, gets up to the top of the cotton, +where it will burn? We shall presently examine that; but there is a much +more wonderful thing about the burning of a candle than this. You have +here a solid substance with no vessel to contain it; and how is it that +this solid substance can get up to the place where the flame is? How is it +that this solid gets there, it not being a fluid? or, when it is made a +fluid, then how is it that it keeps together? This is a wonderful thing +about a candle. + +We have here a good deal of wind, which will help us in some of our +illustrations, but tease us in others; for the sake, therefore, of a +little regularity, and to simplify the matter, I shall make a quiet +flame—for who can study a subject when there are difficulties in the way +not belonging to it? Here is a clever invention of some costermonger or +street stander in the market-place for the shading of their candles on +Saturday nights, when they are selling their greens, or potatoes, or fish. +I have very often admired it. They put a lamp-glass round the candle, +supported on a kind of gallery, which clasps it, and it can be slipped up +and down as required. By the use of this lamp-glass, employed in the same +way, you have a steady flame, which you can look at, and carefully +examine, as I hope you will do, at home. + +You see, then, in the first instance, that a beautiful cup is formed. As +the air comes to the candle it moves upwards by the force of the current +which the heat of the candle produces, and it so cools all the sides of +the wax, tallow, or fuel, as to keep the edge much cooler than the part +within; the part within melts by the flame that runs down the wick as far +as it can go before it is extinguished, but the part on the outside does +not melt. If I made a current in one direction, my cup would be lop-sided, +and the fluid would consequently run over,—for the same force of gravity +which holds worlds together holds this fluid in a horizontal position, and +if the cup be not horizontal, of course the fluid will run away in +guttering. You see, therefore, that the cup is formed by this beautifully +regular ascending current of air playing upon all sides, which keeps the +exterior of the candle cool. No fuel would serve for a candle which has +not the property of giving this cup, except such fuel as the Irish +bogwood, where the material itself is like a sponge, and holds its own +fuel. You see now why you would have had such a bad result if you were to +burn these beautiful candles that I have shewn you, which are irregular, +intermittent in their shape, and cannot therefore have that nicely-formed +edge to the cup which is the great beauty in a candle. I hope you will now +see that the perfection of a process—that is, its utility—is the better +point of beauty about it. It is not the best looking thing, but the best +acting thing, which is the most advantageous to us. This good-looking +candle is a bad burning one. There will be a guttering round about it +because of the irregularity of the stream of air and the badness of the +cup which is formed thereby. You may see some pretty examples (and I trust +you will notice these instances) of the action of the ascending current +when you have A little gutter run down the side of a candle, making it +thicker there than it is elsewhere. As the candle goes on burning, that +keeps its place and forms a little pillar sticking up by the side, +because, as it rises higher above the rest of the wax or fuel, the air +gets better round it, and it is more cooled and better able to resist the +action of the heat at a little distance. Now, the greatest mistakes and +faults with regard to candles, as in many other things, often bring with +them instruction which we should not receive if they had not occurred. We +come here to be philosophers; and I hope you will always remember that +whenever a result happens, especially if it be new, you should say, “What +is the cause? Why does it occur?” and you will in the course of time find +out the reason. + +Then, there is another point about these candles which will answer a +question,—that is, as to the way in which this fluid gets out of the cup, +up the wick, and into the place of combustion. You know that the flames on +these burning wicks in candles made of bees-wax, stearin, or spermaceti, do +not run down to the wax or other matter, and melt it all away, but keep to +their own right place. They are fenced off from the fluid below, and do +not encroach on the cup at the sides. I cannot imagine a more beautiful +example than the condition of adjustment under which a candle makes one +part subserve to the other to the very end of its action. A combustible +thing like that, burning away gradually, never being intruded upon by the +flame, is a very beautiful sight; especially when you come to learn what a +vigorous thing flame is—what power it has of destroying the wax itself +when it gets hold of it, and of disturbing its proper form if it come only +too near. + +But how does the flame get hold of the fuel? There is a beautiful point +about that—_capillary attraction_[4]. “Capillary attraction!” you +say,—“the attraction of hairs.” Well, never mind the name: it was given +in old times, before we had a good understanding of what the real power +was. It is by what is called capillary attraction that the fuel is +conveyed to the part where combustion goes on, and is deposited there, not +in a careless way, but very beautifully in the very midst of the centre of +action which takes place around it. Now, I am going to give you one or two +instances of capillary attraction. It is that kind of action or attraction +which makes two things that do not dissolve in each other still hold +together. When you wash your hands, you wet them thoroughly; you take a +little soap to make the adhesion better, and you find your hand remains +wet. This is by that kind of attraction of which I am about to speak. And, +what is more, if your hands are not soiled (as they almost always are by +the usages of life), if you put your finger into a little warm water, the +water will creep a little way up the finger, though you may not stop to +examine it. I have here a substance which is rather porous—a column of +salt—and I will pour into the plate at the bottom, not water, as it +appears, but a saturated solution of salt which cannot absorb more; so +that the action which you see will not be due to its dissolving anything. +We may consider the plate to be the candle, and the salt the wick, and +this solution the melted tallow. (I have coloured the fluid, that you may +see the action better.) You observe that, now I pour in the fluid, it +rises and gradually creeps up the salt higher and higher; and provided the +column does not tumble over, it will go to the top. + +If this blue solution were combustible, and we were to place a wick at the +top of the salt, it would burn as it entered into the wick. It is a most +curious thing to see this kind of action taking place, and to observe how +singular some of the circumstances are about it. When you wash your hands, +you take a towel to wipe off the water; and it is by that kind of wetting, +or that kind of attraction which makes the towel become wet with water, +that the wick is made wet with the tallow. I have known some careless boys +and girls (indeed, I have known it happen to careful people as well) who, +having washed their hands and wiped them with a towel, have thrown the +towel over the side of the basin, and before long it has drawn all the +water out of the basin and conveyed it to the floor, because it happened +to be thrown over the side in such a way as to serve the purpose of a +syphon.[5] That you may the better see the way in which the substances act +one upon another, I have here a vessel made of wire gauze filled with +water, and you may compare it in its action to the cotton in one respect, +or to a piece of calico in the other. In fact, wicks are sometimes made of +a kind of wire gauze. You will observe that this vessel is a porous thing; +for if I pour a little water on to the top, it will run out at the bottom. +You would be puzzled for a good while if I asked you what the state of +this vessel is, what is inside it, and why it is there? The vessel is full +of water, and yet you see the water goes in and runs out as if it were +empty. In order to prove this to you, I have only to empty it. The reason +is this,—the wire, being once wetted, remains wet; the meshes are so +small that the fluid is attracted so strongly from the one side to the +other, as to remain in the vessel although it is porous. In like manner +the particles of melted tallow ascend the cotton and get to the top; other +particles then follow by their mutual attraction for each other, and as +they reach the flame they are gradually burned. + +Here is another application of the same principle. You see this bit of +cane. I have seen boys about the streets, who are very anxious to appear +like men, take a piece of cane, and light it and smoke it, as an imitation +of a cigar. They are enabled to do so by the permeability of the cane in +one direction, and by its capillarity. If I place this piece of cane on a +plate containing some camphine (which is very much like paraffin in its +general character), exactly in the same manner as the blue fluid rose +through the salt will this fluid rise through the piece of cane. There +being no pores at the side, the fluid cannot go in that direction, but +must pass through its length. Already the fluid is at the top of the cane: +now I can light it and make it serve as a candle. The fluid has risen by +the capillary attraction of the piece of cane, just as it does through the +cotton in the candle. + +Now, the only reason why the candle does not burn all down the side of the +wick is, that the melted tallow extinguishes the flame. You know that a +candle, if turned upside-down, so as to allow the fuel to run upon the +wick, will be put out. The reason is, that the flame has not had time to +make the fuel hot enough to burn, as it does above, where it is carried in +small quantities into the wick, and has all the effect of the heat +exercised upon it. + +There is another condition which you must learn as regards the candle, +without which you would not be able fully to understand the philosophy of +it, and that is the vaporous condition of the fuel. In order that you may +understand that, let me shew you a very pretty, but very common-place +experiment. If you blow a candle out cleverly, you will see the vapour +rise from it. You have, I know, often smelt the vapour of a blown-out +candle—and a very bad smell it is; but if you blow it out cleverly, you +will be able to see pretty well the vapour into which this solid matter is +transformed. I will blow out one of these candles in such a way as not to +disturb the air around it, by the continuing action of my breath; and now, +if I hold a lighted taper two or three inches from the wick, you will +observe a train of fire going through the air till it reaches the candle. +I am obliged to be quick and ready, because, if I allow the vapour time to +cool, it becomes condensed into a liquid or solid, or the stream of +combustible matter gets disturbed. + +Now, as to the shape or form of the flame. It concerns us much to know +about the condition which the matter of the candle finally assumes at the +top of the wick—where you have such beauty and brightness as nothing but +combustion or flame can produce. + +You have the glittering beauty of gold and silver, and the still higher +lustre of jewels, like the ruby and diamond; but none of these rival the +brilliancy and beauty of flame. What diamond can shine like flame? It owes +its lustre at night-time to the very flame shining upon it. The flame +shines in darkness, but the light which the diamond has is as nothing +until the flame shine upon it, when it is brilliant again. The candle +alone shines by itself, and for itself, or for those who have arranged the +materials. Now, let us look a little at the form of the flame as you see +it under the glass shade. It is steady and equal; and its general form is +that which is represented in the diagram, varying with atmospheric +disturbances, and also varying according to the size of the candle. It is +a bright oblong—brighter at the top than towards the bottom—with the +wick in the middle, and besides the wick in the middle, certain darker +parts towards the bottom, where the ignition is not so perfect as in the +part above. + +I have a drawing here, sketched many years ago by Hooke, when he made his +investigations. It is the drawing of the flame of a lamp, but it will +apply to the flame of a candle. The cup of the candle is the vessel or +lamp, the melted spermaceti is the oil, and the wick is common to both. +Upon that he sets this little flame, and then he represents what is +true—a certain quantity of matter rising about it which you do not see, +and which, if you have not been here before, or are not familiar with the +subject, you will not know of. He has here represented the parts of the +surrounding atmosphere that are very essential to the flame, and that are +always present with it. There is a current formed, which draws the flame +out—for the flame which you see is really drawn out by the current, and +drawn upward to a great height—just as Hooke has here shewn you by that +prolongation of the current in the diagram. You may see this by taking a +lighted candle, and putting it in the sun so as to get its shadow thrown +on a piece of paper. How remarkable it is that that thing which is light +enough to produce shadows of other objects, can be made to throw its own +shadow on a piece of white paper or card, so that you can actually see +streaming round the flame something which is not part of the flame, but is +ascending and drawing the flame upwards. Now, I am going to imitate the +sunlight, by applying the voltaic battery to the electric lamp. You now +see our sun, and its great luminosity; and by placing a candle between it +and the screen, we get the shadow of the flame. + +You observe the shadow of the candle and of the wick; then there is a +darkish part, as represented in the diagram, and then a part which is more +distinct. Curiously enough, however, what we see in the shadow as the +darkest part of the flame is, in reality, the brightest part; and here you +see streaming upwards the ascending current of hot air, as shewn by +Hooke, which draws out the flame, supplies it with air, and cools the +sides of the cup of melted fuel. + +I can give you here a little further illustration, for the purpose of +shewing you how flame goes up or down; according to the current. I have +here a flame—it is not a candle-flame—but you can, no doubt, by this +time, generalise enough to be able to compare one thing with another. What +I am about to do is to change the ascending current that takes the flame +upwards into a descending current. This I can easily do by the little +apparatus you see before me. The flame, as I have said, is not a candle +flame, but it is produced by alcohol, so that it shall not smoke too much. +I will also colour the flame with another substance[6], so that you may +trace its course; for with the spirit alone you could hardly see well +enough to have the opportunity of tracing its direction. By lighting this +spirit-of-wine, we have then a flame produced; and you observe that when +held in the air, it naturally goes upwards. + +You understand now easily enough why flames go up under ordinary +circumstances—it is because of the draught of air by which the combustion +is formed. But now, by blowing the flame down, you see I am enabled to +make it go downwards into this little chimney—the direction of the +current being changed. Before we have concluded this course of lectures, +we shall shew you a lamp in which the flame goes up and the smoke goes +down, or the flame goes down and the smoke goes up. You see, then, that we +have the power in this way of varying the flame in different directions. + +There are now some other points that I must bring before you. Many of the +flames you see here vary very much in their shape by the currents of air +blowing around them in different directions; but we can, if we like, make +flames so that they will look like fixtures, and we can photograph +them—indeed, we have to photograph them—so that they become fixed to us, +if we wish to find out everything concerning them. That, however, is not +the only thing I wish to mention. If I take a flame sufficiently large, it +does not keep that homogeneous, that uniform condition of shape, but it +breaks out with a power of life which is quite wonderful. I am about to +use another kind of fuel, but one which is truly and fairly a +representative of the wax or tallow of a candle. I have here a large ball +of cotton, which will serve as a wick. And, now that I have immersed it in +spirit and applied a light to it, in what way does it differ from an +ordinary candle? Why, it differs very much in one respect, that we have a +vivacity and power about it, a beauty and a life entirely different from +the light presented by a candle. You see those fine tongues of flame +rising up. You have the same general disposition of the mass of the flame +from below upwards; but, in addition to that, you have this remarkable +breaking out into tongues which you do not perceive in the case of a +candle. Now, why is this? I must explain it to you, because when you +understand that perfectly, you will be able to follow me better in what I +have to say hereafter. I suppose some here will have made for themselves +the experiment I am going to shew you. Am I right in supposing that +anybody here has played at snapdragon? I do not know a more beautiful +illustration of the philosophy of flame, as to a certain part of its +history, than the game of snapdragon. First, here is the dish; and let me +say, that when you play snapdragon properly, you ought to have the dish +well-warmed; you ought also to have warm plums and warm brandy, which, +however, I have not got. When you have put the spirit into the dish, you +have the cup and the fuel; and are not the raisins acting like the wicks? +I now throw the plums into the dish, and light the spirit, and you see +those beautiful tongues of flame that I refer to. You have the air +creeping in over the edge of the dish forming these tongues. Why? Because, +through the force of the current and the irregularity of the action of the +flame, it cannot flow in one uniform stream. The air flows in so +irregularly that you have what would otherwise be a single image, broken +up into a variety of forms, and each of these little tongues has an +independent existence of its own. Indeed, I might say, you have here a +multitude of independent candles. You must not imagine, because you see +these tongues all at once, that the flame is of this particular shape. A +flame of that shape is never so at any one time. Never is a body of flame, +like that which you just saw rising from the ball, of the shape it appears +to you. It consists of a multitude of different shapes, succeeding each +other so fast that the eye is only able to take cognisance of them all at +once. In former times, I purposely analysed a flame of that general +character, and the diagram shews you the different parts of which it is +composed. They do not occur all at once: it is only because we see these +shapes in such rapid succession, that they seem to us to exist all at one +time. + +It is too bad that we have not got further than my game of snapdragon; but +we must not, under any circumstances, keep you beyond your time. It will +be a lesson to me in future to hold you more strictly to the philosophy of +the thing, than to take up your time so much with these illustrations. + +We were occupied the last time we met in considering the general character +and arrangement as regards the fluid portion of a candle, and the way in +which that fluid got into the place of combustion. You see, when we have a +candle burning fairly in a regular, steady atmosphere, it will have a +shape something like the one shewn in the diagram, and will look pretty +uniform, although very curious in its character. And now, I have to ask +your attention to the means by which we are enabled to ascertain what +happens in any particular part of the flame—why it happens, what it does +in happening, and where, after all, the whole candle goes to: because, as +you know very well, a candle being brought before us and burned, +disappears, if burned properly, without the least trace of dirt in the +candlestick—and this is a very curious circumstance. In order, then, to +examine this candle carefully, I have arranged certain apparatus, the use +of which you will see as I go on. Here is a candle: I am about to put the +end of this glass tube into the middle of the flame—into that part which +old Hooke has represented in the diagram as being rather dark, and which +you can see at any time, if you will look at a candle carefully, without +blowing it about. We will examine this dark part first. + +Now, I take this bent glass tube, and introduce one end into that part of +the flame, and you see at once that something is coming from the flame, +out at the other end of the tube; and if I put a flask there, and leave it +for a little while, you will see that something from the middle part of +the flame is gradually drawn out, and goes through the tube and into that +flask, and there behaves very differently from what it does in the open +air. It not only escapes from the end of the tube, but falls down to the +bottom of the flask like a heavy substance, as indeed it is. We find that +this is the wax of the candle made into a vaporous fluid—not a gas. (You +must learn the difference between a gas and a vapour: a gas remains +permanent, a vapour is something that will condense.) If you blow out a +candle, you perceive a very nasty smell, resulting from the condensation +of this vapour. That is very different from what you have outside the +flame; and, in order to make that more clear to you, I am about to produce +and set fire to a larger portion of this vapour—for what we have in the +small way in a candle, to understand thoroughly, we must, as philosophers, +produce in a larger way, if needful, that we may examine the different +parts. And now Mr. Anderson will give me a source of heat, and I am about +to shew you what that vapour is. Here is some wax in a glass flask, and I +am going to make it hot, as the inside of that candle-flame is hot, and +the matter about the wick is hot. [The Lecturer placed some wax in a glass +flask, and heated it over a lamp.] Now, I dare say that is hot enough for +me. You see that the wax I put in it has become fluid, and there is a +little smoke coming from it. We shall very soon have the vapour rising up. +I will make it still hotter, and now we get more of it, so that I can +actually pour the vapour out of the flask into that basin, and set it on +fire there. This, then, is exactly the same kind of vapour as we have in +the middle of the candle; and that you may be sure this is the case, let +us try whether we have not got here, in this flask, a real combustible +vapour out of the middle of the candle. [Taking the flask into which the +tube from the candle proceeded, and introducing a lighted taper.] See how +it burns. Now, this is the vapour from the middle of the candle, produced +by its own heat; and that is one of the first things you have to consider +with respect to the progress of the wax in the course of its combustion, +and as regards the changes it undergoes. I will arrange another tube +carefully in the flame, and I should not wonder if we were able, by a +little care, to get that vapour to pass through the tube to the other +extremity, where we will light it, and obtain absolutely the flame of the +candle at a place distant from it. Now, look at that. Is not that a very +pretty experiment? Talk about laying on gas—why, we can actually lay on a +candle! And you see from this that there are clearly two different kinds +of action—one the _production_ of the vapour, and the other the +_combustion_ of it—both of which take place in particular parts of the +candle. + +I shall get no vapour from that part which is already burnt. If I raise +the tube (fig. 7) to the upper part of the flame, so soon as the vapour +has been swept out, what comes away will be no longer combustible: It is +already burned. How burned? Why, burned thus:—In the middle of the flame, +where the wick is, there is this combustible vapour; on the outside of the +flame is the air which we shall find necessary for the burning of the +candle; between the two, intense chemical action takes place, whereby the +air and the fuel act upon each other, and at the very same time that we +obtain light the vapour inside is destroyed. If you examine where the heat +of a candle is, you will find it very curiously arranged. Suppose I take +this candle, and hold a piece of paper close upon the flame, where is the +heat of that flame? Do you not see that it is _not_ in the inside? It is +in a ring, exactly in the place where I told you the chemical action was; +and even in my irregular mode of making the experiment, if there is not +too much disturbance, there will always be a ring. This is a good +experiment for you to make at home. Take a strip of paper, have the air in +the room quiet, and put the piece of paper right across the middle of the +flame (I must not talk while I make the experiment), and you will find +that it is burnt in two places, and that it is not burnt, or very little +so, in the middle; and when you have tried the experiment once or twice, +so as to make it nicely, you will be very interested to see where the heat +is, and to find that it is where the air and the fuel come together. + +This is most important for us as we proceed with our subject. Air is +absolutely necessary for combustion; and, what is more, I must have you +understand that _fresh_ air is necessary, or else we should be imperfect +in our reasoning and our experiments. Here is a jar of air. I place it +over a candle, and it burns very nicely in it at first, shewing that what +I have said about it is true; but there will soon be a change. See how the +flame is drawing upwards, presently fading, and at last going out. And +going out, why? Not because it wants air merely, for the jar is as full +now as it was before; but it wants pure, fresh air. The jar is full of +air, partly changed, partly not changed; but it does not contain +sufficient of the fresh air which is necessary for the combustion of a +candle. These are all points which we, as young chemists, have to gather +up; and if we look a little more closely into this kind of action, we +shall find certain steps of reasoning extremely interesting. For instance, +here is the oil-lamp I shewed you—an excellent lamp for our +experiments—the old Argand lamp. I now make it like a candle [obstructing +the passage of air into the centre of the flame]; there is the cotton; +there is the oil rising up it; and there is the conical flame. It burns +poorly, because there is a partial restraint of air. I have allowed no air +to get to it, save round the outside of the flame, and it does not burn +well. I cannot admit more air from the outside, because the wick is large; +but if, as Argand did so cleverly, I open a passage to the middle of the +flame, and so let air come in there, you will see how much more +beautifully it burns. If I shut the air off, look how it smokes; and why? +We have now some very interesting points to study. We have the case of the +combustion of a candle; we have the case of a candle being put out by the +want of air; and we have now the case of imperfect combustion; and this is +to us so interesting, that I want you to understand it as thoroughly as +you do the case of a candle burning in its best possible manner. I will +now make a great flame, because we need the largest possible +illustrations. Here is a larger wick [burning turpentine on a ball of +cotton]. All these things are the same as candles, after all. If we have +larger wicks, we must have a larger supply of air, or we shall have less +perfect combustion. Look now at this black substance going up into the +atmosphere; there is a regular stream of it. I have provided means to +carry off the imperfectly burned part, lest it should annoy you. Look at +the soots that fly off from the flame: see what an imperfect combustion it +is, because it cannot get enough air. What, then, is happening? Why, +certain things which are necessary to the combustion of a candle are +absent, and very bad results are accordingly produced; but we see what +happens to a candle when it is burnt in a pure and proper state of air. At +the time when I shewed you this charring by the ring of flame on the one +side of the paper, I might have also shewn you, by turning to the other +side, that the burning of a candle produces the same kind of +soot—charcoal or carbon. + +But, before I shew that, let me explain to you—as it is quite necessary +for our purpose—that, though I take a candle and give you, as the general +result, its combustion in the form of a flame, we must see whether +combustion is always in this condition, or whether there are other +conditions of flame; and we shall soon discover that there are, and that +they are most important to us. I think, perhaps, the best illustration of +such a point to us, as juveniles, is to shew the result of strong +contrast. Here is a little gunpowder. You know that gunpowder burns with +flame—we may fairly call it flame. It contains carbon and other +materials, which altogether cause it to burn with a flame. And here is +some pulverised iron, or iron filings. Now, I purpose burning these two +things together. I have a little mortar in which I will mix them. (Before +I go into these experiments, let me hope that none of you, by trying to +repeat them, for fun’s sake, will do any harm. These things may all be +very properly used if you take care; but without that, much mischief will +be done.) Well, then, here is a little gunpowder, which I put at the +bottom of that little wooden vessel, and mix the iron filings up with it, +my object being to make the gunpowder set fire to the filings and burn +them in the air, and thereby shew the difference between substances +burning with flame and not with flame. Here is the mixture; and when I set +fire to it, you must watch the combustion, and you will see that it is of +two kinds. You will see the gunpowder burning with a flame, and the +filings thrown up. You will see them burning too, but without the +production of flame. They will each burn separately. [The Lecturer then +ignited the mixture.] There is the gunpowder, which burns with a flame; +and there are the filings—they burn with a different kind of combustion. +You see, then, these two great distinctions; and upon these differences +depend all the utility and all the beauty of flame which we use for the +purpose of giving out light. When we use oil, or gas, or candle, for the +purpose of illumination, their fitness all depends upon these different +kinds of combustion. + +There are such curious conditions of flame, that it requires some +cleverness and nicety of discrimination to distinguish the kinds of +combustion one from another. For instance, here is a powder which is very +combustible, consisting, as you see, of separate little particles. It is +called _lycopodium_[7], and each of these particles can produce a vapour, +and produce its own flame; but, to see them burning, you would imagine it +was all one flame. I will now set fire to a quantity, and you will see the +effect. We saw a cloud of flame, apparently in one body; but that rushing +noise [referring to the sound produced by the burning] was a proof that +the combustion was not a continuous or regular one. This is the lightning +of the pantomimes, and a very good imitation. [The experiment was twice +repeated by blowing lycopodium from a glass tube through a spirit-flame.] +This is not an example of combustion like that of the filings I have been +speaking of, to which we must now return. + +Suppose I take a candle, and examine that part of it which appears +brightest to our eyes. Why, there I get these black particles, which +already you have seen many times evolved from the flame, and which I am +now about to evolve in a different way. I will take this candle and clear +away the gutterage, which occurs by reason of the currents of air; and if +I now arrange a glass tube so as just to dip into this luminous part, as +in our first experiment, only higher, you see the result. In place of +having the same white vapour that you had before, you will now have a +black vapour. There it goes, as black as ink. It is certainly very +different from the white vapour; and when we put a light to it, we shall +find that it does not burn, but that it puts the light out. Well, these +particles, as I said before, are just the smoke of the candle; and this +brings to mind that old employment which Dean Swift recommended to +servants for their amusement, namely, writing on the ceiling of a room +with a candle. But what is that black substance? Why, it is the same +carbon which exists in the candle. How comes it out of the candle? It +evidently existed in the candle, or else we should not have had it here. +And now I want you to follow me in this explanation. You would hardly +think that all those substances which fly about London, in the form of +soots and blacks, are the very beauty and life of the flame, and which are +burned in it as those iron filings were burned here. Here is a piece of +wire gauze, which will not let the flame go through it; and I think you +will see, almost immediately, that when I bring it low enough to touch +that part of the flame which is otherwise so bright, that it quells and +quenches it at once, and allows a volume of smoke to rise up. + +I want you now to follow me in this point,—that whenever a substance +burns, as the iron filings burnt in the flame of gunpowder, without +assuming the vaporous state (whether it becomes liquid or remains solid), +it becomes exceedingly luminous. I have here taken three or four examples +apart from the candle, on purpose to illustrate this point to you; because +what I have to say is applicable to all substances, whether they burn or +whether they do not burn,—that they are exceedingly bright if they retain +their solid state, and that it is to this presence of solid particles in +the candle-flame that it owes its brilliancy. + +Here is a platinum-wire, a body which does not change by heat. If I heat +it in this flame, see how exceedingly luminous it becomes. I will make the +flame dim, for the purpose of giving a little light only, and yet you will +see that the heat which it can give to that platinum-wire, though far less +than the heat it has itself, is able to raise the platinum-wire to a far +higher state of effulgence. This flame has carbon in it; but I will take +one that has no carbon in it. There is a material, a kind of fuel—a +vapour, or gas, whichever you like to call it—in that vessel, and it has +no solid particles in it; so I take that because it is an example of flame +itself burning without any solid matter whatever; and if I now put this +solid substance in it, you see what an intense heat it has, and how +brightly it causes the solid body to glow. This is the pipe through which +we convey this particular gas, which we call hydrogen, and which you shall +know all about next time we meet. And here is a substance called oxygen, +by means of which this hydrogen can burn; and although we produce, by +their mixture, far greater heat[8] than you can obtain from the candle, +yet there is very little light. If, however, I take a solid substance, and +put that into it, we produce an intense light If I take a piece of lime, a +substance which will not burn, and which will not vaporise by the heat +(and because it does not vaporise, remains solid, and remains heated), you +will soon observe what happens as to its glowing. I have here a most +intense heat, produced by the burning of hydrogen in contact with the +oxygen; but there is as yet very little light—not for want of heat, but +for want of particles which can retain their solid state; but when I hold +this piece of lime in the flame of the hydrogen as it burns in the oxygen, +see how it glows! This is the glorious lime-light, which rivals the +voltaic-light, and which is almost equal to sunlight. I have here a piece +of carbon or charcoal, which will burn and give us light exactly in the +same manner as if it were burnt as part of a candle. The heat that is in +the flame of a candle decomposes the vapour of the wax, and sets free the +carbon particles—they rise up heated and glowing as this now glows, and +then enter into the air. But the particles when burnt never pass off from +a candle in the form of carbon. They go off into the air as a perfectly +invisible substance, about which we shall know hereafter. + +Is it not beautiful to think that such a process is going on, and that +such a dirty thing as charcoal can become so incandescent? You see it +comes to this—that all bright flames contain these solid particles; all +things that burn and produce solid particles, either during the time they +are burning, as in the candle, or immediately after being burnt, as in the +case of the gunpowder and iron-filings,—all these things give us this +glorious and beautiful light. + +I will give you a few illustrations. Here is a piece of phosphorus, which +burns with a bright flame. Very well; we may now conclude that phosphorus +will produce, either at the moment that it is burning or afterwards, these +solid particles. Here is the phosphorus lighted, and I cover it over with +this glass for the purpose of keeping in what is produced. What is all +that smoke? That smoke consists of those very particles which are produced +by the combustion of the phosphorus. Here, again, are two substances. This +is chlorate of potassa, and this other sulphuret of antimony. I shall mix +these together a little, and then they may be burnt in many ways. I shall +touch them with a drop of sulphuric acid, for the purpose of giving you an +illustration of chemical action, and they will instantly burn[9]. [The +Lecturer then ignited the mixture by means of sulphuric acid.] Now, from +the appearance of things, you can judge for yourselves whether they +produce solid matter in burning. I have given you the train of reasoning +which will enable you to say whether they do or do not; for what is this +bright flame but the solid particles passing off? + +Mr. Anderson has in the furnace a very hot crucible,—I am about to throw +into it some zinc filings, and they will burn with a flame like gunpowder. +I make this experiment because you can make it well at home. Now, I want +you to see what will be the result of the combustion of this zinc. Here it +is burning—burning beautifully like a candle, I may say. But what is all +that smoke, and what are those little clouds of wool which will come to +you if you cannot come to them, and make themselves sensible to you in the +form of the old philosophic wool, as it was called? We shall have left in +that crucible, also, a quantity of this woolly matter. But I will take a +piece of this same zinc and make an experiment a little more closely at +home, as it were. You will have here the same thing happening. Here is the +piece of zinc, there [pointing to a jet of hydrogen] is the furnace, and +we will set to work and try and burn the metal. It glows, you see: there +is the combustion, and there is the white substance into which it burns. +And so, if I take that flame of hydrogen as the representative of a +candle, and shew you a substance like zinc burning in the flame, you will +see that it was merely during the action of combustion that this substance +glowed—while it was kept hot; and if I take a flame of hydrogen, and put +this white substance from the zinc into it, look how beautifully it glows, +and just because it is a solid substance. + +I will now take such a flame as I had a moment since, and set free from it +the particles of carbon. Here is some camphine, which will burn with a +smoke; but if I send these particles of smoke through this pipe into the +hydrogen flame, you will see they will burn and become luminous, because +we heat them a second time. There they are. Those are the particles of +carbon re-ignited a second time. They are those particles which you can +easily see by holding a piece of paper behind them, and which, whilst they +are in the flame, are ignited by the heat produced, and, when so ignited, +produce this brightness. When the particles are not separated, you get no +brightness. The flame of coal-gas owes its brightness to the separation, +during combustion, of these particles of carbon, which are equally in that +as in a candle. I can very quickly alter that arrangement. Here, for +instance, is a bright flame of gas. Supposing I add so much air to the +flame as to cause it all to burn before those particles are set free, I +shall not have this brightness; and I can do that in this way:—If I place +over the jet this wire-gauze cap, as you see, and then light the gas over +it, it burns with a non-luminous flame, owing to its having plenty of air +mixed with it before it burns; and if I raise the gauze, you see it does +not burn below[10]. There is plenty of carbon in the gas; but, because the +atmosphere can get to it, and mix with it before it burns, you see how +pale and blue the flame is. And if I blow upon a bright gas-flame, so as +to consume all this carbon before it gets heated to the glowing point, it +will also burn blue: [The Lecturer illustrated his remarks by blowing on +the gas-light.] The only reason why I have not the same bright light when +I thus blow upon the flame is, that the carbon meets with sufficient air +to burn it before it gets separated in the flame in a free state. The +difference is solely due to the solid particles not being separated before +the gas is burnt. + +You observe that there are certain products as the result of the +combustion of a candle, and that of these products one portion may be +considered as charcoal, or soot; that charcoal, when afterwards burnt, +produces some other product; and it concerns us very much now to ascertain +what that other product is. We shewed that something was going away; and I +want you now to understand how much is going up into the air; and for that +purpose we will have combustion on a little larger scale. From that candle +ascends heated air, and two or three experiments will shew you the +ascending current; but, in order to give you a notion of the quantity of +matter which ascends in this way, I will make an experiment by which I +shall try to imprison some of the products of this combustion. For this +purpose I have here what boys call a fire-balloon. I use this fire-balloon +merely as a sort of measure of the result of the combustion we are +considering; and I am about to make a flame in such an easy and simple +manner as shall best serve my present purpose. This plate shall be the +“cup,” we will so say, of the candle; this spirit shall be our fuel; and I +am about to place this chimney over it, because it is better for me to do +so than to let things proceed at random. + +Mr. Anderson will now light the fuel, and here at the top we shall get the +results of the combustion. What we get at the top of that tube is exactly +the same, generally speaking, as you get from the combustion of a candle; +but we do not get a luminous flame here, because we use a substance which +is feeble in carbon. I am about to put this balloon—not into action, +because that is not my object—but to shew you the effect which results +from the action of those products which arise from the candle, as they +arise here from the furnace. [The balloon was held over the chimney, when +it immediately commenced to fill.] You see how it is disposed to ascend; +but we must not let it up, because it might come in contact with those +upper gas-lights, and that would be very inconvenient. [The upper +gas-lights were turned out, at the request of the Lecturer, and the +balloon was allowed to ascend.] Does not that shew you what a large bulk +of matter is being evolved? Now, there is going through this tube [placing +a large glass tube over a candle] all the products of that candle, and you +will presently see that the tube will become quite opaque. Suppose I take +another candle, and place it under a jar, and then put a light on the +other side, just to shew you what is going on. You see that the sides of +the jar become cloudy, and the light begins to burn feebly. It is the +products, you see, which make the light so dim, and this is the same thing +which makes the sides of the jar so opaque. If you go home and take a +spoon that has been in the cold air, and hold it over a candle—not so as +to soot it—you will find that it becomes dim, just as that jar is dim. If +you can get a silver dish, or something of that kind, you will make the +experiment still better. And now, just to carry your thoughts forward to +the time we shall next meet, let me tell you that it is _water_ which +causes the dimness; and when we next meet. I will shew you that we can +make it, without difficulty, assume the form of a liquid. + +I dare say you will remember that when we parted we had just mentioned the +word “products” from the candle. For when a candle burns we found we were +able, by nice adjustment, to get various products from it. There was one +substance which was not obtained when the candle was burning properly, +which was charcoal or smoke; and there was some other substance that went +upwards from the flame which did not appear as smoke, but took some other +form, and made part of that general current which, ascending from the +candle upwards, becomes invisible, and escapes. There were also other +products to mention. You remember that in that rising current having its +origin at the candle, we found that one part was condensable against a +cold spoon, or against a clean plate, or any other cold thing, and another +part was incondensable. + +We will first take the condensable part, and examine it; and, strange to +say, we find that that part of the product is just water—nothing but +water. On the last occasion I spoke of it incidentally, merely saying that +water was produced among the condensable products of the candle; but +to-day I wish to draw your attention to water, that we may examine it +carefully, especially in relation to this subject, and also with respect +to its general existence on the surface of the globe. + +Now, having previously arranged an experiment for the purpose of +condensing water from the products of the candle, my next point will be to +shew you this water; and perhaps one of the best means that I can adopt +for shewing its presence to so many at once, is to exhibit a very visible +action of water, and then to apply that test to what is collected as a +drop at the bottom of that vessel. I have here a chemical substance, +discovered by Sir Humphrey Davy, which has a very energetic action upon +water, which I shall use as a test of the presence of water. If I take a +little piece of it—it is called potassium, as coming from potash,—if I +take a little piece of it, and throw it into that basin, you see how it +shews the presence of water by lighting up and floating about, burning +with a violent flame. I am now going to take away the candle which has +been burning beneath the vessel containing ice and salt, and you see a +drop of water—a condensed product of the candle—hanging from under the +surface of the dish. + +I will shew you that potassium has the same action upon it as upon the +water in that basin in the experiment we have just tried. See, it takes +fire, and burns in just the same manner. I will take another drop upon +this glass slab, and when I put the potassium on to it, you see at once, +from its taking fire, that there is water present. Now, that water was +produced by the candle. In the same manner, if I put this spirit-lamp +under that jar, you will soon see the latter become damp, from the dew +which is deposited upon it—that dew being the result of combustion; and I +have no doubt you will shortly see by the drops of water which fall upon +the paper below, that there is a good deal of water produced from the +combustion of the lamp. I will let it remain, and you can afterwards see +how much water has been collected. So, if I take a gas-lamp, and put any +cooling arrangement over it, I shall get water—water being likewise +produced from the combustion of gas. Here, in this bottle, is a quantity +of water—perfectly pure, distilled water, produced from the combustion of +a gas-lamp—in no point different from the water that you distil from the +river, or ocean, or spring, but exactly the same thing. Water is one +individual thing—it never changes. We can add to it by careful +adjustment, for a little while, or we can take it apart, and get other +things from it; but water, as water, remains always the same, either in a +solid, liquid, or fluid state. Here, again [holding another bottle], is +some water produced by the combustion of an oil-lamp. A pint of oil, when +burnt fairly and properly, produces rather more than a pint of water. +Here, again, is some water, produced by a rather long experiment from a +wax candle. And so we can go on with almost all combustible substances, +and find that if they burn with a flame, as a candle, they produce water. +You may make these experiments yourselves. The head of a poker is a very +good thing to try with, and if it remains cold long enough over the +candle, you may get water condensed in drops on it; or a spoon or ladle, +or anything else may be used, provided it be clean, and can carry off the +heat, and so condense the water. + +And now—to go into the history of this wonderful production of water from +combustibles, and by combustion—I must first of all tell you that this +water may exist in different conditions; and although you may now be +acquainted with all its forms, they still require us to give a little +attention to them for the present, so that we may perceive how the water, +whilst it goes through its Protean changes, is entirely and absolutely the +same thing, whether it is produced from a candle, by combustion, or from +the rivers or ocean. + +First of all, water, when at the coldest, is ice. Now, we philosophers—I +hope that I may class you and myself together in this case—speak of water +as water, whether it be in its solid, or liquid, or gaseous state,—we +speak of it chemically as water. Water is a thing compounded of two +substances, one of which we have derived from the candle, and the other we +shall find elsewhere. Water may occur as ice; and you have had most +excellent opportunities lately of seeing this. Ice changes back into +water—for we had on our last Sabbath a strong instance of this change, by +the sad catastrophe which occurred in our own house, as well as in the +houses of many of our friends,—ice changes back into water when the +temperature is raised: water also changes into steam when it is warmed +enough. The water which we have here before us is in its densest +state[11], and although it changes in weight, in condition, in form, and +in many other qualities, it still is water; and whether we alter it into +ice by cooling, or whether we change it into steam by heat, it increases +in volume,—in the one case very strangely and powerfully, and in the +other case very largely and wonderfully. For instance, I will now take +this tin cylinder, and pour a little water into it; and seeing how much +water I pour in, you may easily estimate for yourselves how high it will +rise in the vessel: it will cover the bottom about two inches. I am now +about to convert the water into steam, for the purpose of shewing to you +the different volumes which water occupies in its different states of +water and steam. + +Let us now take the case of water changing into ice: we can effect that by +cooling it in a mixture of salt and pounded ice[12]; and I shall do so to +shew you the expansion of water into a thing of larger bulk when it is so +changed. These bottles [holding one] are made of strong cast iron, very +strong and very thick—I suppose they are the third of an inch in +thickness; they are very carefully filled with water, so as to exclude all +air, and then they are screwed down tight. We shall see that when we +freeze the water in these iron vessels, they will not be able to hold the +ice, and the expansion within them will break them in pieces as these +[pointing to some fragments] are broken, which have been bottles of +exactly the same kind. I am about to put these two bottles into that +mixture of ice and salt, for the purpose of shewing that when water +becomes ice, it changes in volume in this extraordinary way. + +In the mean time look at the change which has taken place in the water to +which we have applied heat—it is losing its fluid state. You may tell +this by two or three circumstances. I have covered the mouth of this glass +flask, in which water is boiling, with a watch-glass. Do you see what +happens? It rattles away like a valve chattering, because the steam rising +from the boiling water sends the valve up and down, and forces itself out, +and so makes it clatter. You can very easily perceive that the flask is +quite full of steam, or else it would not force its way out. You see, +also, that the flask contains a substance very much larger than the water, +for it fills the whole of the flask over and over again, and there it is +blowing away into the air; and yet you cannot observe any great diminution +in the bulk of the water, which shews you that its change of bulk is very +great when it becomes steam. + +I have put our iron bottles containing water into this freezing mixture, +that you may see what happens. No communication will take place, you +observe, between the water in the bottles and the ice in the outer vessel. +But there will be a conveyance of heat from the one to the other; and if +we are successful—we are making our experiment in very great haste—I +expect you will by-and-by, so soon as the cold has taken possession of the +bottles and their contents, hear a pop on the occasion of the bursting of +the one bottle or the other; and, when we come to examine the bottles, we +shall find their contents masses of ice, partly enclosed by the covering +of iron which is too small for them, because the ice is larger in bulk +than the water. You know very well that ice floats upon water: if a boy +falls through a hole into the water, he tries to get on the ice again to +float him up. Why does the ice float?—think of that, and philosophise. +Because the ice is larger than the quantity of water which can produce it; +and therefore the ice weighs the lighter, and the water is the heavier. + +To return now to the action of heat on water. See what a stream of vapour +is issuing from this tin vessel! You observe, we must have made it quite +full of steam to have it sent out in that great quantity. And now, as we +can convert the water into steam by heat, we convert it back into liquid +water by the application of cold. And if we take a glass, or any other +cold thing, and hold it over this steam, see how soon it gets damp with +water; it will condense it until the glass is warm—it condenses the water +which is now running down the sides of it. I have here another experiment +to shew the condensation of water from a vaporous state back into a liquid +state, in the same way as the vapour, one of the products of the candle, +was condensed against the bottom of the dish, and obtained in the form of +water; and to shew you how truly and thoroughly these changes take place, +I will take this tin flask, which is now full of steam, and close the top. +We shall see what takes place when we cause this water or steam to return +back to the fluid state by pouring some cold water on the outside. [The +Lecturer poured the cold water over the vessel, when it immediately +collapsed.] You see what has happened. If I had closed the stopper, and +still kept the heat applied to it, it would have burst the vessel; yet, +when the steam returns to the state of water, the vessel collapses, there +being a vacuum produced inside by the condensation of the steam. I shew +you these experiments for the purpose of pointing out that in all these +occurrences there is nothing that changes the water into any other +thing—it still remains water; and so the vessel is obliged to give way, +and is crushed inwards, as in the other case, by the further application +of heat, it would have been blown outwards. + +And what do you think the bulk of that water is when it assumes the +vaporous condition? You see that cube [pointing to a cubic foot]. There, +by its side, is a cubic inch, exactly the same shape as the cubic foot, +and that bulk of water [the cubic inch] is sufficient to expand into that +bulk [the cubic foot] of steam; and, on the contrary, the application of +cold will contract that large quantity of steam into this small quantity +of water. + +[One of the iron bottles burst at that moment.] Ah! There is one of our +bottles burst, and here you see is a crack down one side an eighth of an +inch in width. [The other now exploded, sending the freezing mixture in +all directions.] This other bottle is also broken; although the iron was +nearly half-an-inch thick, the ice has burst it asunder. These changes +always take place in water: they do not require to be always produced by +artificial means,—we only use them here because we want to produce a +small winter round that little bottle, instead of a long and severe one. +But if you go to Canada, or to the North, you will find the temperature +there out of doors will do the same thing as has been done here by the +freezing mixture. + +To return to our quiet philosophy. We shall not in future be deceived, +therefore, by any changes that are produced in water. Water is the same +everywhere, whether produced from the ocean or from the flame of the +candle. Where, then, is this water which we get from a candle? I must +anticipate a little, and tell you. It evidently comes, as to part of it, +from the candle; but is it within the candle beforehand? No. It is not in +the candle; and it is not in the air round about the candle which is +necessary for its combustion. It is neither in one nor the other, but it +comes from their conjoint action, a part from the candle, a part from the +air; and this we have now to trace, so that we may understand thoroughly +what is the chemical history of a candle when we have it burning on our +table. How shall we get at this? I myself know plenty of ways, but I want +_you_ to get at it from the association in your own minds of what I have +already told you. + +I think you can see a little in this way. We had just now the case of a +substance which acted upon the water in the way that Sir Humphrey Davy +shewed us[13], and which I am now going to recall to your minds by making +again an experiment upon that dish. It is a thing which we have to handle +very carefully, for you see, if I allow a little splash of water to come +upon this mass, it sets fire to part of it; and if there were free access +of air, it would quickly set fire to the whole. Now, this is a metal—a +beautiful and bright metal—which rapidly changes in the air, and, as you +know, rapidly changes in water. I will put a piece on the water, and you +see it burns beautifully, making a floating lamp, using the water in the +place of air. Again, if we take a few iron filings or turnings, and put +them in water, we find that they likewise undergo an alteration. They do +not change so much as this potassium does, but they change somewhat in the +same way; they become rusty, and shew an action upon the water, though in +a different degree of intensity to what this beautiful metal does: but +they act upon the water in the same manner generally as this potassium. I +want you to put these different facts together in your minds. I have +another metal here [zinc], and when we examined it with regard to the +solid substance produced by its combustion, we had an opportunity of +seeing that it burned; and I suppose, if I take a little strip of this +zinc and put it over the candle, you will see something half-way, as it +were, between the combustion of potassium on the water and the action of +iron,—you see there is a sort of combustion. It has burned, leaving a +white ash or residuum, and here also we find that the metal has a certain +amount of action upon water. + +By degrees we have learned how to modify the action of these different +substances, and to make them tell us what we want to know. And now, first +of all, I take iron. It is a common thing in all chemical reactions, where +we get any result of this kind, to find that it is increased by the action +of heat; and if we want to examine minutely and carefully the action of +bodies one upon another, we often have to refer to the action of heat. You +are aware, I believe, that iron-filings burn beautifully in the air; but I +am about to shew you an experiment of this kind, because it will impress +upon you what I am going to say about iron in its action on water. If I +take a flame and make it hollow;—you know why, because I want to get air +to it and into it, and therefore I make it hollow—and then take a few +iron-filings and drop them into the flame, you see how well they burn. +That combustion results from the chemical action which is going on when we +ignite those particles. And so we proceed to consider these different +effects, and ascertain what iron will do when it meets with water. It will +tell us the story so beautifully, so gradually and regularly, that I think +it will please you very much. + +I have here a furnace with a pipe going through it like an iron +gun barrel, and I have stuffed that barrel full of bright iron-turnings, +and placed it across the fire, to be made red-hot. We can either send air +through the barrel to come in contact with the iron, or we can send steam +from this little boiler at the end of the barrel. Here is a stop-cock +which shuts off the steam from the barrel until we wish to admit it. There +is some water in these glass jars, which I have coloured blue, so that you +may see what happens. Now, you know very well that any steam I might send +through that barrel, if it went through into the water, would be +condensed; for you have seen that steam cannot retain its gaseous form if +it be cooled down. + +You saw it here [pointing to the tin flask] crushing itself into a small +bulk, and causing the flask holding it to collapse; so that if I were to +send steam through that barrel, it would be condensed—supposing the +barrel were cold: it is, therefore, heated to perform the experiment I am +now about to shew you. I am going to send the steam through the barrel in +small quantities; and you shall judge for yourselves, when you see it +issue from the other end, whether it still remains steam. Steam is +condensable into water, and when you lower the temperature of steam, you +convert it back into fluid water; but I have lowered the temperature of +the gas which I have collected in this jar, by passing it through water +after it has traversed the iron barrel, and still it does not change back +into water. I will take another test and apply to this gas. (I hold the +jar in an inverted position, or my substance would escape.) If I now apply +a light to the mouth of the jar, it ignites with a slight noise. That +tells you that it is not steam. Steam puts out a fire—it does not burn; +but you saw that what I had in that jar burnt. We may obtain this +substance equally from water produced from the candle-flame as from any +other source. When it is obtained by the action of the iron upon the +aqueous vapour, it leaves the iron in a state very similar to that in +which these filings were after they were burnt. It makes the iron heavier +than it was before. So long as the iron remains in the tube and is heated, +and is cooled again without the access of air or water, it does not change +in its weight; but after having had this current of steam passed over it, +it then comes out heavier that it was before, having taken something out +of the steam, and having allowed something else to pass forth, which we +see here. And now, as we have another jar full, I will shew you something +most interesting. It is a combustible gas; and I might at once take this +jar and set fire to the contents, and shew you that it is combustible; but +I intend to shew you more if I can. It is also a very light substance. +Steam will condense: this body will rise in the air, and not condense. + +Suppose I take another glass jar, empty of all but air: if I examine it +with a taper, I shall find that it contains nothing but air. I will now +take this jar full of the gas that I am speaking of, and deal with it as +though it were a light body. I will hold both upside-down, and turn the +one up under the other; and that which did contain the gas procured from +the steam, what does it contain now? You will find it now only contains +air. But look! Here is the combustible substance [taking the other jar] +which I have poured out of the one jar into the other. It still preserves +its quality, and condition, and independence, and therefore is the more +worthy of our consideration, as belonging to the products of a candle. + +Now, this substance which we have just prepared by the action of iron on +the steam or water, we can also get by means of those other things which +you have already seen act so well upon the water. If I take a piece of +potassium, and make the necessary arrangements, it will produce this gas; +and if, instead, a piece of zinc, I find, when I come to examine it very +carefully, that the main reason why this zinc cannot act upon the water +continuously as the other metal does, is because the result of the action +of the water envelopes the zinc in a kind of protecting coat. We have +learned in consequence, that if we put into our vessel only the zinc and +water, they by themselves do not give rise to much action, and we get no +result. But suppose I proceed to dissolve off this varnish—this +encumbering substance—which I can do by a little acid; the moment I do +this, I find the zinc acting upon the water exactly as the iron did, but +at the common temperature. The acid in no way is altered, except in its +combination with the oxide of zinc, which is produced. I have now poured +the acid into the glass, and the effect is as though I were applying heat +to cause this boiling up. There is something coming off from the zinc very +abundantly, which is not steam. There is a jar full of it; and you will +find that I have exactly the same combustible substance remaining in the +vessel, when I hold it upside-down, that I produced during the experiment +with the iron barrel. This is what we get from water—the same substance +which is contained in the candle. + +Let us now trace distinctly the connection between these two points. This +is hydrogen—a body classed among those things which in Chemistry we call +elements, because we can get nothing else out of them. A candle is not an +elementary body, because we can get carbon out of it; we can get this +hydrogen out of it, or at least out of the water which it supplies. And +this gas has been so named hydrogen, because it is that element which, in +association with another, generates water. [Footnote: [greek: hudos], +“water,” and [greek: gennao], “I generate.”] Mr. Anderson having now been +able to get two or three jars of gas, we shall have a few experiments to +make, and I want to shew you the best way of making these experiments. I +am not afraid to shew you, for I wish you to make experiments, if you will +only make them with care and attention, and the assent of those around +you. As we advance in Chemistry, we are obliged to deal with substances +which are rather injurious, if in their wrong places—the acids, and heat, +and combustible things we use, might do harm if carelessly employed. If +you want to make hydrogen, you can make it easily from bits of zinc, and +sulphuric or muriatic acid. Here is what in former times was called the +“philosopher’s candle.” It is a little phial with a cork, and a tube or +pipe passing through it. + +And I am now putting a few little pieces of zinc into it. This little +instrument I am going to apply to a useful purpose in our +demonstrations—for I want to shew you that you can prepare hydrogen, and +make some experiments with it as you please at your own homes. Let me here +tell you why I am so careful to fill this phial nearly, and yet not quite +full. I do it because the evolved gas, which, as you have seen, is very +combustible, is explosive to a considerable extent when mixed with air, +and might lead to harm, if you were to apply a light to the end of that +pipe before all the air had been swept out of the space above the water. I +am now about to pour in the sulphuric acid. I have used very little zinc, +and more sulphuric acid and water, because I want to keep it at work for +some time. I therefore take care in this way to modify the proportions of +the ingredients, so that I may have a regular supply—not too quick, and +not too slow. Supposing I now take a glass and put it upside-down over the +end of the tube, because the hydrogen is light I expect that it will +remain in that vessel a little while. We will now test the contents of our +glass to see if there be hydrogen in it. I think I am safe in saying we +have caught some [applying a light]. There it is, you see. I will now +apply a light to the top of the tube. There is the hydrogen burning. There +is our philosophical candle. It is a foolish feeble sort of a flame, you +may say; but it is so hot that scarcely any common flame gives out so much +heat. It goes on burning regularly, and I am now about to put that flame +to burn under a certain arrangement, in order that we may examine its +results and make use of the information which we may thereby acquire. +Inasmuch as the candle produces water, and this gas comes out of the +water, let us see what this gives us by the same process of combustion +that the candle went through when it burnt in the atmosphere; and for that +purpose I am going to put the lamp under this apparatus, in order to +condense whatever may arise from the combustion within it In the course of +a short time you will see moisture appearing in the cylinder, and you will +get the water running down the side; and the water from this hydrogen +flame will have absolutely the same effect upon all our tests, being +obtained by the same general process as in the former case. This hydrogen +is a very beautiful substance. It is so light that it carries things up: +it is far lighter than the atmosphere; and I dare say I can shew you this +by an experiment which, if you are very clever, some of you may even have +skill enough to repeat. Here is our generator of hydrogen, and here are +some soap-suds. I have an india-rubber tube connected with the hydrogen +generator, and at the end of the tube is a tobacco-pipe. + +I can thus put the pipe into the suds, and blow bubbles by means of the +hydrogen. You observe how the bubbles fall downwards when I blow them with +my warm breath; but notice the difference when I blow them with hydrogen. +[The Lecturer here blew bubbles with hydrogen, which rose to the roof of +the theatre.] It shews you how light this gas must be in order to carry +with it not merely the ordinary soap-bubble, but the larger portion of a +drop hanging to the bottom of it. I can shew its lightness in a better way +than this; larger bubbles than these may be so lifted up; indeed, in +former times balloons used to be filled with this gas. Mr. Anderson will +fasten this tube on to our generator, and we shall have a stream of +hydrogen here with which we can charge this balloon made of collodion. I +need not even be very careful to get all the air out, for I know the power +of this gas to carry it up. [Two collodion balloons were inflated, and +sent up, one being held by a string.] Here is another larger one made of +thin membrane, which we will fill and allow to ascend. You will see they +will all remain floating about until the gas escapes. + +What, then, are the comparative weights of these substances? I have a +table here which will shew you the proportion which their weights bear to +each other. I have taken a pint and a cubic foot as the measures, and have +placed opposite to them the respective figures. A pint measure of this +hydrogen weighs three-quarters of our smallest weight (a grain), and a +cubic foot weighs one-twelfth of an ounce; whereas a pint of water weighs +8,750 grains, and a cubic foot of water weighs almost 1,000 ounces. You +see, therefore, what a vast difference there is between the weight of a +cubic foot of water and a cubic foot of hydrogen. + +Hydrogen gives rise to no substance that can become solid, either during +combustion or afterwards as a product of its combustion. But when it +burns, it produces water only; and if we take a cold glass and put it over +the flame, it becomes damp, and you have water, produced immediately in +appreciable quantity; and nothing is produced by its combustion but the +same water which you have seen the flame of the candle produce. It is +important to remember that this hydrogen is the only thing in nature which +furnishes water as the sole product of combustion. + +And now we must endeavour to find some additional proof of the general +character and composition of water; and for this purpose I will keep you a +little longer, so that at our next meeting we may be better prepared for +the subject. We have the power of arranging the zinc which you have seen +acting upon the water by the assistance of an acid, in such a manner as to +cause all the power to be evolved in the place where we require it I have +behind me a voltaic pile, and I am just about to shew you, at the end of +this lecture, its character and power, that you may see what we shall have +to deal with when next we meet. I hold here the extremities of the wires +which transport the power from behind me, and which I shall cause to act +on the water. + +We have previously seen what a power of combustion is possessed by the +potassium, or the zinc, or the iron-filings; but none of them shew such +energy as this. [The Lecturer here made contact between the two terminal +wires of the battery, when a brilliant flash of light was produced.] This +light is, in fact, produced by a forty-zinc power of burning: it is a +power that I can carry about in my hands, through these wires, at +pleasure—although, if I applied it wrongly to myself, it would destroy me +in an instant, for it is a most intense thing, and the power you see here +put forth while you count five [bringing the poles in contact, and +exhibiting the electric light] is equivalent to the power of several +thunder-storms, so great is its force[14]. And that you may see what +intense energy it has, I will take the ends of the wires which convey the +power from the battery, and with it I dare say I can burn this iron file. +Now, this is a chemical power, and one which, when we next meet, I shall +apply to water, and shew you what results we are able to produce. + +I see you are not tired of the candle yet, or I am sure you would not be +interested in the subject in the way you are. When our candle was burning, +we found it produced water exactly like the water we have around us; and +by further examination of this water we found in it that curious body, +hydrogen—that light substance of which there is some in this jar. We +afterwards saw the burning powers of that hydrogen, and that it produced +water. And I think I introduced to your notice an apparatus which I very +briefly said was an arrangement of chemical force, or power, or energy, so +adjusted as to convey its power to us in these wires; and I said I should +use that force to pull the water to pieces, to see what else there was in +the water besides hydrogen; because, you remember, when we passed the +water through the iron tube, we by no means got the weight of water back +which we put in, in the form of steam, though we had a very large quantity +of gas evolved. We have now to see what is the other substance present. +That you may understand the character and use of this instrument, let us +make an experiment or two. Let us put together, first of all, some +substances, knowing what they are, and then see what that instrument does +to them. There is some copper (observe the various changes which it can +undergo), and here is some nitric acid, and you will find that this, being +a strong chemical agent, will act very powerfully when I add it to the +copper. It is now sending forth a beautiful red vapour; but as we do not +want that vapour, Mr. Anderson will hold it near the chimney for a short +time, that we may have the use and beauty of the experiment without the +annoyance. The copper which I have put into the flask will dissolve: it +will change the acid and the water into a blue fluid, containing copper +and other things; and I propose then shewing you how this voltaic battery +deals with it; and in the mean time we will arrange another kind of +experiment for you to see what power it has. This is a substance which is +to us like water—that is to say, it contains bodies which we do not know +of as yet, as water contains a body which we do not know as yet. Now, this +solution of a salt[15] I will put upon paper, and spread about, and apply +the power of the battery to it, and observe what will happen. Three or +four important things will happen which we shall take advantage of. I +place this wetted paper upon a sheet of tinfoil, which is convenient for +keeping all clean, and also for the advantageous application of the power; +and this solution, you see, is not at all affected by being put upon the +paper or tinfoil, nor by anything else I have brought in contact with it +yet, and which, therefore, is free to us to use as regards that +instrument. But first let us see that our instrument is in order. Here are +our wires. Let us see whether it is in the state in which it was last +time. We can soon tell. As yet, when I bring them together, we have no +power, because the conveyers—what we call the electrodes—the passages or +ways for the electricity—are stopped; but now Mr. Anderson by that +[referring to a sudden flash at the ends of the wires] has given me a +telegram to say that it is ready. Before I begin our experiment I will get +Mr. Anderson to break contact again at the battery behind me, and we will +put a platinum-wire across to connect the poles, and then if I find I can +ignite a pretty good length of this wire, we shall be safe in our +experiment. Now you will see the power. [The connection was established, +and the intermediate wire became red-hot.] There is the power running +beautifully through the wire, which I have made thin on purpose to shew +you that we have those powerful forces; and now, having that power, we +will proceed with it to the examination of water. + +I have here two pieces of platinum, and if I lay them down upon this piece +of paper [the moistened paper on the tinfoil], you will see no action; and +if I take them up, there is no change that you can see, but the +arrangement remains just as it was before. But, now, see what happens: if +I take these two poles and put either one or the other of them down +separately on the platinum-plates, they do nothing for me, both are +perfectly without action; but if I let them both be in contact at the same +moment, see what happens [a brown spot appeared under each pole of the +battery]. Look here at the effect that takes place, and see how I have +pulled something apart from the white—something brown; and I have no +doubt, if I were to arrange it thus, and were to put one of the poles to +the tinfoil on the other side of the paper—why, I get such a beautiful +action upon the paper, that I am going to see whether I cannot write with +it—a telegram, if you please. [The Lecturer here traced the word +“juvenile” on the paper with one of the terminal wires.] See there how +beautifully we can get our results! + +You see we have here drawn something, which we have not known about +before, out of this solution. Let us now take that flask from Mr. +Andersen’s hands, and see what we can draw out of that. This, you know, is +a liquid which we have just made up from copper and nitric acid, whilst +our other experiments were in hand; and though I am making this experiment +very hastily, and may bungle a little, yet I prefer to let you see what I +do rather than prepare it beforehand. + +Now, see what happens. These two platinum-plates are the two ends (or I +will make them so immediately) of this apparatus; and I am about to put +them in contact with that solution just as we did a moment ago on the +paper. It does not matter to us whether the solution be on the paper or +whether it be in the jar, so long as we bring the ends of the apparatus to +it. If I put the two platinums in by themselves, they come out as clean +and as white as they go in [inserting them into the fluid without +connecting them with the battery]; but when we take the power and lay that +on [the platinums were connected with the battery and again dipped into +the solution], this, you see [exhibiting one of the platinums], is at once +turned into copper, as it were: it has become like a plate of copper; and +that [exhibiting the other piece of platinum] has come out quite clean. If +I take this coppered piece and change sides, the copper will leave the +right-hand side and come over to the left side; what was before the +coppered plate comes out clean, and the plate which was clean comes out +coated with copper; and thus you see that the same copper we put into this +solution we can also take out of it by means of this instrument. + +Putting that solution aside, let us now see what effect this instrument +will have upon water. Here are two little platinum-plates which I intend +to make the ends of the battery, and this (C) is a little vessel so shaped +as to enable me to take it to pieces, and shew you its construction. In +these two cups (A and B) I pour mercury, which touches the ends of the +wires connected with the platinum-plates. In the vessel (C) I pour some +water containing a little acid (but which is put only for the purpose of +facilitating the action; it undergoes no change in the process), and +connected with the top of the vessel is a bent glass tube (D), which may +remind you of the pipe which was connected with the gun barrel in our +furnace experiment, and which now passes under the jar (F). I have now +adjusted this apparatus, and we will proceed to affect the water in some +way or other. In the other case, I sent the water through a tube which was +made red-hot; I am now going to pass the electricity through the contents +of this vessel. Perhaps I may boil the water; if I do boil the water, I +shall get steam; and you know that steam condenses when it gets cold, and +you will therefore see by that whether I do boil the water or not. +Perhaps, however, I shall not boil the water, but produce some other +effect. You shall have the experiment and see. There is one wire which I +will put to this side (A), and here is the other wire which I will put to +the other side (B), and you will soon see whether any disturbance takes +place. Here it is seeming to boil up famously; but does it boil? Let us +see whether that which goes out is steam or not. I think you will soon see +the jar (F) will be filled with vapour, if that which rises from the water +is steam. But can it be steam? Why, certainly not; because there it +remains, you see, unchanged. There it is standing over the water, and it +cannot therefore be steam, but must be a permanent gas of some sort What +is it? Is it hydrogen? Is it anything else? Well, we will examine it. If +it is hydrogen, it will burn. [The Lecturer then ignited a portion of the +gas collected, which burnt with an explosion.] + +It is certainly something combustible, but not combustible in the way that +hydrogen is. Hydrogen would not have given you that noise; but the colour +of that light, when the thing did burn, was like that of hydrogen: it +will, however, burn without contact with the air. That is why I have +chosen this other form of apparatus, for the purpose of pointing out to +you what are the particular circumstances of this experiment. In place of +an open vessel I have taken one that is closed (our battery is so +beautifully active that we are even boiling the mercury, and getting all +things right—not wrong, but vigorously right); and I am going to shew you +that that gas, whatever it may be, can burn without air, and in that +respect differs from a candle, which cannot burn without the air. And our +manner of doing this is as follows:—I have here a glass vessel (G) which +is fitted with two platinum-wires (IK), through which I can apply +electricity; and we can put the vessel on the air-pump and exhaust the +air, and when we have taken the air out we can bring it here and fasten it +on to this jar (F), and let into the vessel that gas which was formed by +the action of the voltaic battery upon the water, and which we have +produced by changing the water into it,—for I may go as far as this, and +say we have really, by that experiment, changed the water into that gas. +We have not only altered its condition, but we have changed it really and +truly into that gaseous substance, and all the water is there which was +decomposed by the experiment. As I screw this vessel (GH) on here (H), and +make the tubes well connected, and when I open the stop-cocks (HHH), if +you watch the level of the water (in F), you will see that the gas will +rise. I will now close the stop-cocks, as I have drawn up as much as the +vessel can hold, and being safely conveyed into that chamber, I will pass +into it an electric spark from this Leyden jar (L), when the vessel, which +is now quite clear and bright, will become dim. There will be no sound, +for the vessel is strong enough to confine the explosion. [A spark was +then passed through the jar, when the explosive mixture was ignited.] Did +you see that brilliant light? If I again screw the vessel on to the jar, +and open these stop-cocks, you will see that the gas will rise a second +time. [The stop-cocks were then opened.] Those gases [referring to the +gases first collected in the jar, and which had just been ignited by the +electric spark] have disappeared, as you see: their place is vacant, and +fresh gas has gone in. Water has been formed from them; and if we repeat +our operation [repeating the last experiment], I shall have another +vacancy, as you will see by the water rising. I always have an empty +vessel after the explosion, because the vapour or gas into which that +water has been resolved by the battery explodes under the influence of the +spark, and changes into water; and by-and-by you will see in this upper +vessel some drops of water trickling down the sides and collecting at the +bottom. + +We are here dealing with water entirely, without reference to the +atmosphere. The water of the candle had the atmosphere helping to produce +it; but in this way it can be produced independently of the air. Water, +therefore, ought to contain that other substance which the candle takes +from the air, and which, combining with the hydrogen, produces water. + +Just now you saw that one end of this battery took hold of the copper, +extracting it from the vessel which contained the blue solution. It was +effected by this wire; and surely we may say, if the battery has such +power with a metallic solution which we made and unmade, may we not find +that it is possible to split asunder the component parts of the water, and +put them into this place and that place? Suppose I take the poles—the +metallic ends of this battery—and see what will happen with the water in +this apparatus (fig. 20), where we have separated the two ends far apart. + +I place one here (at A), and the other there (at B), and I have little +shelves with holes which I can put upon each pole, and so arrange them +that whatever escapes from the two ends of the battery will appear as +separate gases; for you saw that the water did not become vaporous, but +gaseous. The wires are now in perfect and proper connection with the +vessel containing the water; and you see the bubbles rising: let us +collect these bubbles and see what they are. Here is a glass cylinder (O); +I fill it with water and put it over one end (A) of the pile; and I will +take another (H) and put it over the other end (B) of the pile. And so now +we have a double apparatus, with both places delivering gas. Both these +jars will fill with gas. There they go, that to the right (H) filling very +rapidly; the one to the left (O) filling not so rapidly; and though I have +allowed some bubbles to escape, yet still the action is going on pretty +regularly; and were it not that one is rather smaller than the other, you +would see that I should have twice as much in this (H) as I have in that +(O). Both these gases are colourless; they stand over the water without +condensing; they are alike in all things—I mean in all _apparent_ things; +and we have here an opportunity of examining these bodies and ascertaining +what they are. Their bulk is large, and we can easily apply experiments to +them. I will take this jar (H) first, and will ask you to be prepared to +recognise hydrogen. + +Think of all its qualities—the light gas which stood well in inverted +vessels, burning with a pale flame at the mouth of the jar—and see +whether this gas does not satisfy all these conditions. If it be hydrogen, +it will remain here while I hold this jar inverted. [A light was then +applied, when the hydrogen burnt] What is there now in the other jar? You +know that the two together made an explosive mixture. But what can this +be which we find as the other constituent in water, and which must +therefore be that substance which made the hydrogen burn? We know that the +water we put into the vessel consisted of the two things together. We find +one of these is hydrogen: what must that other be which was in the water +before the experiment, and which we now have by itself? I am about to put +this lighted splinter of wood into the gas. The gas itself will not burn, +but it will make the splinter of wood burn. [The Lecturer ignited the end +of the wood, and introduced it into the jar of gas.] See how it +invigorates the combustion of the wood, and how it makes it burn far +better than the air would make it burn; and now you see by itself that +every other substance which is contained in the water, and which, when the +water was formed by the burning of the candle, must have been taken from +the atmosphere. What shall we call it, A, B, or C? Let us call it O—call +it “Oxygen:” it is a very good distinct-sounding name. This, then, is the +oxygen which was present in the water, forming so large a part of it. + +We shall now begin to understand more clearly our experiments and +researches; because, when we have examined these things once or twice, we +shall soon see why a candle burns in the air. When we have in this way +analysed the water—that is to say, separated, or electrolysed its parts +out of it—we get two volumes of hydrogen, and one of the body that burns +it. And these two are represented to us on the following diagram, with +their weights also stated; and we shall find that the oxygen is a very +heavy body by comparison with the hydrogen. It is the other element in +water. + +I had better, perhaps, tell you now how we get this oxygen abundantly, +having shewn you how we can separate it from the water. Oxygen, as you +will immediately imagine, exists in the atmosphere; for how should the +candle burn to produce water without it? + + _____________________ + | | | + | 1 | 8 | + | | | + | | Oxygen. | Oxygen, . . . . 88.9 + | | | + | |_________| Hydrogen, . . . 11.1 + | Hydrogen. | ————— + | | 9 Water,. . . . . 100.0 + | | + | | + | | + |___________| + +Such a thing would be absolutely impossible, and chemically impossible, +without oxygen. + +Can we get it from the air? Well, there are some very complicated and +difficult processes by which we can get it from the air; but we have +better processes. There is a substance called the black oxide of +manganese: it is a very black-looking mineral, but very useful, and when +made red-hot it gives out oxygen. Here is an iron bottle which has had +some of this substance put into it, and there is a tube fixed to it, and a +fire ready made, and Mr. Anderson will put that retort into the fire, for +it is made of iron, and can stand the heat. Here is a salt called chlorate +of potassa, which is now made in large quantities for bleaching, and +chemical and medical uses, and for pyrotechnic and other purposes. I will +take some and mix it with some of the oxide of manganese (oxide of copper, +or oxide of iron would do as well); and if I put these together in a +retort, far less than a red heat is sufficient to evolve this oxygen from +the mixture. I am not preparing to make much, because we only want +sufficient for our experiments; only, as you will see immediately, if I +use too small a charge, the first portion of the gas will be mixed with +the air already in the retort, and I should be obliged to sacrifice the +first portion of the gas, because it would be so much diluted with air; +the first portion must therefore be thrown away. You will find in this +case, that a common spirit-lamp is quite sufficient for me to get the +oxygen, and so we shall have two processes going on for its preparation. +See how freely the gas is coming over from that small portion of the +mixture. We will examine it, and see what are its properties. Now, in this +way we are producing, as you will observe, a gas just like the one we had +in the experiment with the battery, transparent, undissolved by water, and +presenting the ordinary visible properties of the atmosphere. (As this +first jar contains the air, together with the first portions of the oxygen +set free during the preparation, we will carry it out of the way, and be +prepared to make our experiments in a regular, dignified manner.) And, +inasmuch as that power of making wood, wax, or other things burn, was so +marked in the oxygen we obtained by means of the voltaic battery from +water, we may expect to find the same property here. We will try it You +see there is the combustion of a lighted taper in air, and here is its +combustion in this gas [lowering the taper into the jar]. See how brightly +and how beautifully it burns! You can also see more than this,—you will +perceive it is a heavy gas, whilst the hydrogen would go up like a +balloon, or even faster than a balloon, when not encumbered with the +weight of the envelope. + +You may easily see that although we obtained from water twice as much in +volume of the hydrogen as of oxygen, it does not follow that we have twice +as much in weight—because one is heavy, and the other a very light gas. +We have means of weighing gases or air; but without stopping to explain, +that, let me just tell you what their respective weights are. The weight +of a pint of hydrogen is three-quarters of a grain; the weight of the same +quantity of oxygen is nearly twelve grains. This is a very great +difference. The weight of a cubit foot of hydrogen is one-twelfth of an +ounce; and the weight of a cubit foot of oxygen is one ounce and a third. +And so on we might come to masses of matter which may be weighed in the +balance, and which we can take account of as to hundredweights and as to +tons, as you will see almost immediately. + +Now, as regards this very property of oxygen supporting combustion, which +we may compare to air, I will take a piece of candle to shew it you in a +rough way, and the result will be rough. There is our candle burning in +the air: how will it burn in oxygen? I have here a jar of this gas, and I +am about to put it over the candle for you to compare the action of this +gas with that of the air. Why, look at it: it looks something like the +light you saw at the poles of the voltaic battery. Think how vigorous that +action must be! And yet, during all that action, nothing more is produced +than what is produced by the burning of the candle in air. We have the +same production of water, and the same phenomena exactly, when we use this +gas instead of air, as we have when the candle is burnt in air. + +But now we have got a knowledge of this new substance, we can look at it a +little more distinctly, in order to satisfy ourselves that we have got a +good general understanding of this part of the product of a candle. It is +wonderful how great the supporting powers of this substance are as regards +combustion. For instance, here is a lamp which, simple though it be, is +the original, I may say, of a great variety of lamps which are constructed +for divers purposes—for light-houses, microscopic illuminations, and +other uses; and if it were proposed to make it burn very brightly, you +would say, “If a candle burnt better in oxygen, will not a lamp do the +same?” Why, it will do so. Mr. Anderson will give me a tube coming from +our oxygen reservoir, and I am about to apply it to this flame, which I +will previously make burn badly on purpose. There comes the oxygen: what a +combustion that makes! But if I shut it off, what becomes of the lamp? [The +flow of oxygen was stopped, and the lamp relapsed to its former dimness.] +It is wonderful how, by means of oxygen, we get combustion accelerated. +But it does not affect merely the combustion of hydrogen, or carbon, or +the candle; but it exalts all combustions of the common kind. We will take +one which relates to iron, for instance, as you have already seen iron +burn a little in the atmosphere. Here is a jar of oxygen, and this is a +piece of iron wire; but if it were a bar as thick as my wrist, it would +burn the same. + +I first attach a little piece of wood to the iron, I then set the wood on +fire and let them both down together into the jar. The wood is now alight, +and there it burns as wood should burn in oxygen; but it will soon +communicate its combustion to the iron. The iron is now burning +brilliantly, and will continue so for a long time. As long as we supply +oxygen, so long can we carry on the combustion of the iron, until the +latter is consumed. + +We will now put that on one side, and take some other substance; but we +must limit our experiments, for we have not time to spare for all the +illustrations you would have a right to if we had more time. We will take +a piece of sulphur—you know how sulphur burns in the air—well, we put it +into the oxygen, and you will see that whatever can burn in air, can burn +with a far greater intensity in oxygen, leading you to think that perhaps +the atmosphere itself owes all its power of combustion to this gas. The +sulphur is now burning very quietly in the oxygen; but you cannot for a +moment mistake the very high and increased action which takes place when +it is so burnt, instead of being burnt merely in common air. + +I am now about to shew you the combustion of another +substance—phosphorus. I can do it better for you here than you can do it +at home. This is a very combustible substance; and if it be so combustible +in air, what might you expect it would be in oxygen? I am about to shew it +to you not in its fullest intensity, for if I did so we should almost blow +the apparatus up—I may even now crack the jar, though I do not want to +break things carelessly. You see how it burns in the air. But what a +glorious light it gives out when I introduce it into oxygen! [Introducing +the lighted phosphorus into the jar of oxygen.] There you see the solid +particles going off which cause that combustion to be so brilliantly +luminous. + +Thus far we have tested this power of oxygen, and the high combustion it +produces by means of other substances. We must now, for a little while +longer, look at it as respects the hydrogen. You know, when we allowed the +oxygen and the hydrogen derived from the water to mix and burn together, +we had a little explosion. You remember, also, that when I burnt the +oxygen and the hydrogen in a jet together, we got very little light, but +great heat. I am now about to set fire to oxygen and hydrogen, mixed in +the proportion in which they occur in water. Here is a vessel containing +one volume of oxygen and two volumes of hydrogen. This mixture is exactly +of the same nature as the gas we just now obtained from the voltaic +battery: it would be far too much to burn at once; I have therefore +arranged to blow soap-bubbles with it, and burn those bubbles, that we may +see by a general experiment or two how this oxygen supports the combustion +of the hydrogen. First of all, we will see whether we can blow a bubble. +Well, there goes the gas [causing it to issue through a tobacco-pipe into +some soap-suds]. Here I have a bubble. I am receiving them on my hand: and +you will perhaps think I am acting oddly in this experiment; but it is to +shew you that we must not always trust to noise and sounds, but rather to +real facts. [Exploding a bubble on the palm of his hand.] I am afraid to +fire a bubble from the end of the pipe, because the explosion would pass +up into the jar and blow it to pieces. This oxygen then will unite with +the hydrogen, as you see by the phenomena, and hear by the sound, with the +utmost readiness of action, and all its powers are then taken up in its +neutralisation of the qualities of the hydrogen. + +So now I think you will perceive the whole history of water with reference +to oxygen and the air, from what we have before said. Why does a piece of +potassium decompose water? Because it finds oxygen in the water. What is +set free when I put it in the water, as I am about to do again? It sets +free hydrogen, and the hydrogen burns; but the potassium itself combines +with oxygen; and this piece of potassium, in taking the water apart—the +water, you may say, derived from the combustion of the candle—takes away +the oxygen which the candle took from the air, and so sets the hydrogen +free; and even if I take a piece of ice, and put a piece of potassium upon +it, the beautiful affinities by which the oxygen and the hydrogen are +related are such, that the ice will absolutely set fire to the potassium. +I shew this to you to-day, in order to enlarge your ideas of these things, +and that you may see how greatly results are modified by circumstances. +There is the potassium on the ice, producing a sort of volcanic action. + +It will be my place, when next we meet, having pointed out these anomalous +actions, to shew you that none of these extra and strange effects are met +with by us—that none of these strange and injurious actions take place +when we are burning, not merely a candle, but gas in our streets, or fuel +in our fireplaces, so long as we confine ourselves within the laws that +Nature has made for our guidance. + +We have now seen that we can produce hydrogen and oxygen from the water +that we obtained from the candle. Hydrogen, you know, comes from the +candle, and oxygen, you believe, comes from the air. But then you have a +right to ask me, “How is it that the air and the oxygen do not equally +well burn the candle?” If you remember what happened when I put a jar of +oxygen over a piece of candle, you recollect there was a very different +kind of combustion to that which took place in the air. Now, why is this? +It is a very important question, and one I shall endeavour to make you +understand: it relates most intimately to the nature of the atmosphere, +and is most important to us. + +We have several tests for oxygen besides the mere burning of bodies. You +have seen a candle burnt in oxygen, or in the air; you have seen +phosphorus burnt in the air, or in oxygen; and you have seen iron-filings +burnt in oxygen. But we have other tests besides these, and I am about to +refer to one or two of them for the purpose of carrying your conviction +and your experience further. Here we have a vessel of oxygen. I will shew +its presence to you: if I take a little spark and put it into that oxygen, +you know, by the experience you gained the last time we met, what will +happen; if I put that spark into the jar, it will tell you whether we have +oxygen here or not. Yes! We have proved it by combustion; and now here is +another test for oxygen, which is a very curious and useful one. I have +here two jars full of gas, with a plate between them to prevent their +mixing; I take the plate away, and the gases are creeping one into the +other. “What happens?” say you: “they together produce no such combustion +as was seen in the case of the candle.” But see how the presence of oxygen +is told by its association with this other substance[14]. What a +beautifully coloured gas I have obtained in this way, shewing me the +presence of the oxygen! In the same way we can try this experiment by +mixing common air with this test-gas. Here is a jar containing air—such +air as the candle would burn in—and here is a jar or bottle containing +the test-gas. I let them come together over water, and you see the result: +the contents of the test-bottle are flowing into the jar of air, and you +see I obtain exactly the same kind of action as before, and that shews me +that there is oxygen in the air—the very same substance that has been +already obtained by us from the water produced by the candle. But then, +beyond that, how is it that the candle does not burn in air as well as in +oxygen? We will come to that point at once. I have here two jars; they are +filled to the same height with gas, and the appearance to the eye is alike +in both, and I really do not know at present which of these jars contains +oxygen and which contains air, although I know they have previously been +filled with these gases. But here is our test-gas, and I am going to work +with the two jars, in order to examine whether there is any difference +between them in the quality of reddening this gas. I am now going to turn +this test-gas into one of the jars, and observe what happens. There is +reddening, you see; there is then oxygen present. We will now test the +other jar; but you see this is not so distinctly red as the first: and, +further, this curious thing happens,—if I take these two gases and shake +them well together with water, we shall absorb the red gas; and then, if I +put in more of this test-gas and shake again, we shall absorb more; and I +can go on as long as there be any oxygen present to produce that effect. +If I let in air, it will not matter; but the moment I introduce water, the +red gas disappears; and I may go on in this way, putting in more and more +of the test-gas, until I come to something left behind which will not +redden any longer by the use of that particular body that rendered the air +and the oxygen red. Why is that? You see in a moment it is because there +is, besides oxygen, something else present which is left behind. I will +let a little more air into the jar, and if it turns red you will know that +some of that reddening gas is still present, and that consequently it was +not for the want of this producing body that that air was left behind. + +Now, you will begin to understand what I am about to say. You saw that +when I burnt phosphorus in a jar, as the smoke produced by the phosphorus +and the oxygen of the air condensed, it left a good deal of gas unburnt, +just as this red gas left something untouched,—there was, in fact, this +gas left behind, which the phosphorus cannot touch, which the reddening +gas cannot touch, and this something is not oxygen, and yet is part of the +atmosphere. + +So that is one way of opening out air into the two things of which it is +composed—oxygen, which burns our candles, our phosphorus, or anything +else; and this other substance—nitrogen—which will not burn them. This +other part of the air is by far the larger proportion, and it is a very +curious body, when we come to examine it; it is remarkably curious, and +yet you say, perhaps, that it is very uninteresting. It is uninteresting +in some respects because of this—that it shews no brilliant effects of +combustion. If I test it with a taper as I do oxygen and hydrogen, it +does not burn like hydrogen, nor does it make the taper burn like oxygen. +Try it in any way I will, it does neither the one thing nor the other: it +will not take fire; it will not let the taper burn; it puts out the +combustion of everything. There is nothing that will burn in it in common +circumstances. It has no smell; it is not sour; it does not dissolve in +water; it is neither an acid nor an alkali; it is as indifferent to all +our organs as it is possible for a thing to be. And you might say, “It is +nothing; it is not worth chemical attention; what does it do in the air?” +Ah! then come our beautiful and fine results shewn us by an observant +philosophy. Suppose, in place of having nitrogen, or nitrogen and oxygen, +we had pure oxygen as our atmosphere; what would become of us? You know +very well that a piece of iron lit in a jar of oxygen goes on burning to +the end. When you see a fire in an iron grate, imagine where the grate +would go to if the whole of the atmosphere were oxygen. The grate would +burn up more powerfully than the coals—for the iron of the grate itself +is even more combustible than the coals which we burn in it. A fire put +into the middle of a locomotive would be a fire in a magazine of fuel, if +the atmosphere were oxygen. The nitrogen lowers it down and makes it +moderate and useful for us, and then, with all that, it takes away with it +the fumes that you have seen produced from the candle, disperses them +throughout the whole of the atmosphere, and carries them away to places +where they are wanted to perform a great and glorious purpose of good to +man, for the sustenance of vegetation; and thus does a most wonderful +work, although you say, on examining it, “Why, it is a perfectly +indifferent thing.” This nitrogen in its ordinary state is an inactive +element; no action short of the most intense electric force, and then in +the most infinitely small degree, can cause the nitrogen to combine +directly with the other element of the atmosphere, or with other things +round about it; it is a perfectly indifferent, and therefore to say, a +safe substance. + +But before I take you to that result, I must tell you about the atmosphere +itself. I have written on this diagram the composition of one hundred +parts of atmospheric air:— + +It is a true analysis of the atmosphere, so far as regards the quantity of +oxygen and the quantity of nitrogen present. By our analysis, we find that +5 pints of the atmosphere contain only 1 pint of oxygen, and 4 pints, or 4 +parts, of nitrogen by bulk. That is our analysis of the atmosphere. It +requires all that quantity of nitrogen to reduce the oxygen down, so as to +be able to supply the candle properly with fuel, so as to supply us with +an atmosphere which our lungs can healthily and safely breathe; for it is +just as important to make the oxygen right for us to breathe, as it is to +make the atmosphere right for the burning of the fire and the candle. + +But now for this atmosphere. First of all, let me tell you the weight of +these gases. A pint of nitrogen weighs 10⁴⁄₁₀ grains, or a cubic foot +weighs 1⅙ ounce. That is the weight of the nitrogen. The oxygen is +heavier: a pint of it weighs 11⁹⁄₁₀ grains, and a cubic foot weighs 1¾ +ounce. A pint of air weighs about 10⁷⁄₁₀ grains, and a cubic foot 1⅕ +ounce. + +You have asked me several times, and I am very glad you have, “How do you +weigh gases?” I will shew you; it is very simple, and easily done. Here is +a balance, and here a copper bottle, made as light as we can consistent +with due strength, turned very nicely in the lathe, and made perfectly +air-tight, with a stop-cock, which we can open and shut, which at present +is open, and therefore allows the bottle to be full of air. I have here a +nicely-adjusted balance, in which I think the bottle, in its present +condition, will be balanced by the weight on the other side. And here is a +pump by which we can force the air into this bottle, and with it we will +force in a certain number of volumes of air, as measured by the pump. +[Twenty measures were pumped in.] We will shut that in and put it in the +balance. See how it sinks: it is much heavier than it was. By what? By +the air that we have forced into it by the pump. There is not a greater +_bulk_ of air, but there is the same bulk of _heavier_ air, because we +have forced in air upon it. And that you may have a fair notion in your +mind as to how much this air measures, here is a jar full of water. We +will open that copper vessel into this jar, and let the air return to its +former state. All I have to do now is to screw them tightly together, and +to turn the taps, when there, you see, is the bulk of the twenty pumps of +air which I forced into the bottle; and to make sure that we have been +quite correct in what we have been doing, we will take the bottle again to +the balance, and, if it is now counterpoised by the original weight, we +shall be quite sure we have made our experiment correctly. + +It is balanced; so, you see, we can find out the weight of the extra +volumes of air forced in, in that way, and by that means we are able to +ascertain that a cubic foot of air weighs 1⅕ ounce. But that small +experiment will by no means convey to your mind the whole literal truth of +this matter. It is wonderful how it accumulates when you come to larger +volumes. This bulk of air [a cubic foot] weighs 1⅕ ounce. What do you +think of the contents of that box above there, which I have had made for +the purpose? The air which is within that box weighs one pound—a full +pound; and I have calculated the weight of the air in this room,—you +would hardly imagine it, but it is above a ton. So rapidly do the weights +rise up, and so important is the presence of the atmosphere, and of the +oxygen and the nitrogen in it, and the use it performs in conveying things +to and fro from place to place, and carrying bad vapours to places where +they will do good instead of harm. + +Having given you that little illustration with respect to the weight of +the air, let me shew you certain consequences of it. You have a right to +them, because you would not understand so much without it. Do you remember +this kind of experiment? Have you ever seen it? Suppose I take a pump +somewhat similar to the one I had a little while ago to force air into the +bottle, and suppose I place it in such a manner that by certain +arrangements I can apply my hand to it: my hand moves about in the air so +easily that it seems to feel nothing, and I can hardly get velocity enough +by any motion of my own in the atmosphere to make sure that there is much +resistance to it. + +But, when I put my hand here [on the air-pump receiver, which was +afterwards exhausted], you see what happens. Why is my hand fastened to +this place, and why am I able to pull this pump about? And see! how is it +that I can hardly get my hand away? Why is this? It is the weight of the +air—the weight of the air that is above. I have another experiment here, +which I think will explain to you more about it. When the air is pumped +from underneath the bladder which is stretched over this glass, you will +see the effect in another shape: the top is quite flat at present, but I +will make a very little motion with the pump, and now look at it—see how +it has gone down, see how it is bent in. You will see the bladder go in +more and more, until at last I expect it will be driven in and broken by +the force of the atmosphere pressing upon it. + +[The bladder at last broke with a loud report.] Now, that was done +entirely by the weight of the air pressing on it, and you can easily +understand how that is. The particles that are piled up in the atmosphere +stand upon each other, as these five cubes do. You can easily conceive +that four of these five cubes are resting upon the bottom one, and if I +take that away, the others will all sink down. So it is with the +atmosphere: the air that is above is sustained by the air that is beneath; +and when the air is pumped away from beneath them, the change occurs which +you saw when I placed my hand on the air-pump, and which you saw in the +case of the bladder, and which you shall see better here. I have tied over +this jar a piece of sheet india-rubber, and I am now about to take away +the air from the inside of the jar; and if you will watch the +india-rubber—which acts as a partition between the air below and the air +above—you will see, when I pump, how the pressure shews itself. See where +it is going to—I can actually put my hand into the jar; and yet this +result is only caused by the great and powerful action of the air above. +How beautifully it shews this curious circumstance! + +Here is something that you can have a pull at, when I have finished +to-day. It is a little apparatus of two hollow brass hemispheres, closely +fitted together, and having connected with it a pipe and a cock, through +which we can exhaust the air from the inside; and although the two halves +are so easily taken apart, while the air is left within, yet you will see, +when we exhaust it by-and-by, no power of any two of you will be able to +pull them apart. Every square inch of surface that is contained in the +area of that vessel sustains fifteen pounds by weight, or nearly so, when +the air is taken out; and you may try your strength presently in seeing +whether you can overcome that pressure of the atmosphere. + +Here is another very pretty thing—the boys’ sucker, only refined by the +philosopher. We young ones have a perfect right to take toys, and make +them into philosophy, inasmuch as now-a-days we are turning philosophy +into toys. Here is a sucker, only it is made of india-rubber: if I clap it +upon the table, you see at once it holds. Why does it hold? I can slip it +about, and yet if I try to pull it up, it seems as if it would pull the +table with it I can easily make it slip about from place to place; but +only when I bring it to the edge of the table can I get it off. It is only +kept down by the pressure of the atmosphere above. We have a couple of +them; and if you take these two and press them together, you will see how +firmly they stick. And, indeed, we may use them as they are proposed to be +used, to stick against windows, or against walls, where they will adhere +for an evening, and serve to hang anything on that you want. I think, +however, that you boys ought to be shewn experiments that you can make at +home; and so here is a very pretty experiment in illustration of the +pressure of the atmosphere. Here is a tumbler of water. Suppose I were to +ask you to turn that tumbler upside-down, so that the water should not +fall out, and yet not be kept in by your hand, but merely by using the +pressure of the atmosphere. Could you do that? Take a wine-glass, either +quite full or half-full of water, and put a flat card on the top, turn it +upside-down, and then see what becomes of the card and of the water. The +air cannot get in because the water by its capillary attraction round the +edge keeps it out. + +I think this will give you a correct notion of what you may call the +materiality of the air; and when I tell you that the box holds a pound of +it, and this room more than a ton, you will begin to think that air is +something very serious. I will make another experiment, to convince you of +this positive resistance. There is that beautiful experiment of the +popgun, made so well and so easily, you know, out of a quill, or a tube, +or anything of that kind,—where we take a slice of potato, for instance, +or an apple, and take the tube and cut out a pellet, as I have now done, +and push it to one end. I have made that end tight; and now I take another +piece and put it in: it will confine the air that is within the tube +perfectly and completely for our purpose; and I shall now find it +absolutely impossible by any force of mine to drive that little pellet +close up to the other. It cannot be done. I may press the air to a certain +extent, but if I go on pressing, long before it comes to the second, the +confined air will drive the front one out with a force something like that +of gunpowder; for gunpowder is in part dependent upon the same action that +you see here exemplified. + +I saw the other day an experiment which pleased me much, as I thought it +would serve our purpose here. (I ought to have held my tongue for four or +five minutes before beginning this experiment, because it depends upon my +lungs for success.) By the proper application of air I expect to be able +to drive this egg out of one cup into the other by the force of my breath; +but if I fail, it is in a good cause; and I do not promise success, +because I have been talking more than I ought to do to make the experiment +succeed. + +You see that the air which I blow goes downwards between the egg and the +cup, and makes a blast under the egg, and is thus able to lift a heavy +thing—for a full egg is a very heavy thing for air to lift. If you want +to make the experiment, you had better boil the egg quite hard first, and +then you may very safely try to blow it from one cup to the other, with a +little care. + +I have now kept you long enough upon this property of the weight of the +air, but there is another thing I should like to mention. You saw the way +in which, in this popgun, I was able to drive the second piece of potato +half or two-thirds of an inch before the first piece started, by virtue of +the elasticity of the air—just as I pressed into the copper bottle the +particles of air by means of the pump. Now, this depends upon a wonderful +property in the air, namely, its elasticity; and I should like to give you +a good illustration of this. If I take anything that confines the air +properly, as this membrane, which also is able to contract and expand so +as to give us a measure of the elasticity of the air, and confine in this +bladder a certain portion of air; and then, if we take the atmosphere off +from the outside of it, just as in these cases we put the pressure on—if +we take the pressure off, you will see how it will then go on expanding +and expanding, larger and larger, until it will fill the whole of this +bell-jar, shewing you that wonderful property of the air, its elasticity, +its compressibility, and expansibility, to an exceedingly large extent, +and which is very essential for the purposes and services it performs in +the economy of creation. + +We will now turn to another very important part of our subject, +remembering that we have examined the candle in its burning, and have +found that it gives rise to various products. We have the products, you +know, of soot, of water, and of something else which you have not yet +examined. We have collected the water, but have allowed the other things +to go into the air. Let us now examine some of these other products. + +Here is an experiment which I think will help you in part in this way. We +will put our candle there, and place over it a chimney, thus. I think my +candle will go on burning, because the air-passage is open at the bottom +and the top. In the first place, you see the moisture appearing—that you +know about. It is water produced from the candle by the action of the air +upon its hydrogen. But, besides that, something is going out at the top: +it is not moisture—it is not water—it is not condensable; and yet, after +all, it has very singular properties. You will find that the air coming +out of the top of our chimney is nearly sufficient to blow the light out I +am holding to it; and if I put the light fairly opposed to the current, it +will blow it quite out. You will say that is as it should be; and I am +supposing that you think it ought to do so, because the nitrogen does not +support combustion, and ought to put the candle out, since the candle will +not burn in nitrogen. + +But is there nothing else there than nitrogen? I must now anticipate—that +is to say, I must use my own knowledge to supply you with the means that +we adopt for the purpose of ascertaining these things, and examining such +gases as these. I will take an empty bottle—here is one—and if I hold it +over this chimney, I shall get the combustion of the candle below sending +its results into the bottle above; and we shall soon find that this bottle +contains, not merely an air that is bad as regards the combustion of a +taper put into it, but having other properties. + +Let me take a little quick-lime and pour some common water on to it—the +commonest water will do. I will stir it a moment, then pour it upon a +piece of filtering paper in a funnel, and we shall very quickly have a +clear water proceeding to the bottle below, as I have here. I have plenty +of this water in another bottle; but, nevertheless, I should like to use +the lime-water that was prepared before you, so that you may see what its +uses are. If I take some of this beautiful clear lime-water, and pour it +into this jar, which has collected the air from the candle, you will see a +change coming about. Do you see that the water has become quite milky? +Observe, that will not happen with air merely. Here is a bottle filled +with air; and if I put a little lime-water into it, neither the oxygen nor +the nitrogen, nor anything else that is in that quantity of air, will make +any change in the lime-water. It remains perfectly clear, and no shaking +of that quantity of lime-water with that quantity of air in its common +state will cause any change; but if I take this bottle with the +lime-water, and hold it so as to get the general products of the candle in +contact with it, in a very short time we shall have it milky. There is the +chalk, consisting of the lime which we used in making the lime-water, +combined with something that came from the candle—that other product +which we are in search of, and which I want to tell you about to-day. This +is a substance made visible to us by its action, which is not the action +of the lime-water either upon the oxygen or upon the nitrogen, nor upon +the water itself, but it is something new to us from the candle. And then +we find this white powder, produced by the lime-water and the vapour from +the candle, appears to us very much like whitening or chalk, and, when +examined, it does prove to be exactly the same substance as whitening or +chalk. So we are led, or have been led, to observe upon the various +circumstances of this experiment, and to trace this production of chalk to +its various causes, to give us the true knowledge of the nature of this +combustion of the candle—to find that this substance, issuing from the +candle, is exactly the same as that substance which would issue from a +retort, if I were to put some chalk into it with a little moisture, and +make it red-hot: you would then find that exactly the same substance would +issue from it as from the candle. + +But we have a better means of getting this substance, and in greater +quantity, so as to ascertain what its general characters are. We find this +substance in very great abundance in a multitude of cases where you would +least expect it. All limestones contain a great deal of this gas which +issues from the candle, and which we call _carbonic acid_. All chalks, all +shells, all corals contain a great quantity of this curious air. We find +it fixed in these stones; for which reason Dr. Black called it “fixed +air”—finding it in these fixed things like marble and chalk. He called it +fixed air, because it lost its quality of air, and assumed the condition +of a solid body. We can easily get this air from marble. Here is a jar +containing a little muriatic acid, and here is a taper which, if I put it +into that jar, will shew only the presence of common air. There is, you +see, pure air down to the bottom; the jar is full of it Here is a +substance—marble[17], a very beautiful and superior marble—and if I put +these pieces of marble into the jar, a great boiling apparently goes on. +That, however, is not steam—it is a gas that is rising up; and if I now +search the jar by a candle, I shall have exactly the same effect produced +upon the taper as I had from the air which issued from the end of the +chimney over the burning candle. It is exactly the same action, and caused +by the very same substance that issued from the candle; and in this way we +can get carbonic acid in great abundance—we have already nearly filled +the jar. We also find that this gas is not merely contained in marble. +Here is a vessel in which I have put some common whitening—chalk, which +has been washed in water and deprived of its coarser particles, and so +supplied to the plasterer as whitening. Here is a large jar containing +this whitening and water, and I have here some strong sulphuric acid, +which is the acid you might have to use if you were to make these +experiments (only, in using this acid with limestone, the body that is +produced is an insoluble substance, whereas the muriatic acid produces a +soluble substance that does not so much thicken the water). And you will +seek out a reason why I take this kind of apparatus for the purpose of +shewing this experiment. I do it because you may repeat in a small way +what I am about to do in a large one. You will have here just the same +kind of action; and I am evolving in this large jar carbonic acid, exactly +the same in its nature and properties as the gas which we obtained from +the combustion of the candle in the atmosphere. And no matter how +different the two methods by which we prepare this carbonic acid, you will +see, when we get to the end of our subject, that it is all exactly the +same, whether prepared in the one way or in the other. + +We will now proceed to the next experiment with regard to this gas. What +is its nature? Here is one of the vessels full, and we will try it, as we +have done so many other gases, by combustion. You see it is not +combustible, nor does it support combustion. Neither, as we know, does it +dissolve much in water, because we collect it over water very easily. +Then, you know that it has an effect, and becomes white in contact with +lime-water; and when it does become white in that way, it becomes one of +the constituents to make carbonate of lime or limestone. + +The next thing I must shew you is, that it really does dissolve a little +in water, and therefore that it is unlike oxygen and hydrogen in that +respect I have here an apparatus by which we can produce this solution. In +the lower part of this apparatus is marble and acid, and in the upper part +cold water. The valves are so arranged that the gas can get from one to +the other. I will set it in action now, and you can see the gas bubbling +up through the water, as it has been doing all night long, and by this +time we shall find that we have this substance dissolved in the water. If +I take a glass and draw off some of the water, I find that it tastes a +little acid to the mouth: it is impregnated with carbonic acid; and if I +now apply a little lime-water to it, that will give us a test of its +presence. This water will make the lime-water turbid and white, which is +proof of the presence of carbonic acid. + +Then it is a very weighty gas—it is heavier than the atmosphere. I have +put their respective weights at the lower part of this table, along with, +for comparison, the weights of the other gases we have been examining:— + + Pint. Cubic Foot. +Hydrogen, . . . . 3/4 grains. 1/12 ounce. +Oxygen, . . . . 11 9/10 „ 1 1/3 „ +Nitrogen, . . . . 10 4/10 „ 1 1/6 „ +Air,. . . . . . 10 7/10 „ 1 1/5 „ +Carbonic acid, . . 16 1/3 „ 1 9/10 „ + +A pint of it weighs 16⅓ grains, and a cubic foot weighs 1⁹⁄₁₀ ounce, +almost two ounces. You can see by many experiments that this is a heavy +gas. Suppose I take a glass containing nothing else but air, and from this +vessel containing the carbonic acid I attempt to pour a little of this gas +into that glass; I wonder whether any has gone in or not. I cannot tell by +the appearance, but I can in this way [introducing the taper]. Yes, there +it is, you see; and if I were to examine it by lime-water, I should find +it by that test also. I will take this little bucket, and put it down into +the well of carbonic acid—indeed, we too often have real wells of +carbonic acid—and now, if there is any carbonic acid, I must have got to +it by this time, and it will be in this bucket, which we will examine with +a taper. There it is, you see; it is full of carbonic acid. + +There is another experiment by which I will shew you its weight. I have +here a jar suspended at one end of a balance—it is now equipoised; but +when I pour this carbonic acid into the jar on the one side which now +contains air, you will see it sink down at once, because of the carbonic +acid that I pour into it. And now, if I examine this jar with the lighted +taper, I shall find that the carbonic acid has fallen into it, and it no +longer has any power of supporting the combustion. If I blow a +soap-bubble, which of course will be filled with air, and let it fall into +this jar of carbonic acid, it will float. + +But I shall first of all take one of these little balloons filled with +air. I am not quite sure where the carbonic acid is; we will just try the +depth, and see whereabouts is its level. There, you see, we have this +bladder floating on the carbonic acid; and if I evolve some more of the +carbonic acid, the bladder will be lifted up higher. There it goes—the +jar is nearly full; and now I will see whether I can blow a soap-bubble on +that, and float it in the same way. [The Lecturer here blew a soap-bubble, +and allowed it to fall into the jar of carbonic acid, when it floated in +it midway.] It is floating, as the balloon floated, by virtue of the +greater weight of the carbonic acid than of the air. And now, having so +far given you the history of the carbonic acid—as to its sources in the +candle, as to its physical properties and weight—when we next meet I +shall shew you of what it is composed, and where it gets its elements +from. + +A lady, who honours me by her presence at these Lectures, has conferred a +still further obligation by sending me these two candles, which are from +Japan, and, I presume, are made of that substance to which I referred in a +former lecture. You see that they are even far more highly ornamented than +the French candles; and, I suppose, are candles of luxury, judging from +their appearance. They have a remarkable peculiarity about them—namely, a +hollow wick,—that beautiful peculiarity which Argand introduced into the +lamp, and made so valuable. To those who receive such presents from the +East, I may just say that this and such like materials gradually undergo a +change which gives them on the surface a dull and dead appearance; but +they may easily be restored to their original beauty, if the surface be +rubbed with a clean cloth or silk handkerchief, so as to polish the little +rugosity or roughness: this will restore the beauty of the colours. I have +so rubbed one of these candles, and you see the difference between it and +the other which has not been polished, but which may be restored by the +same process. Observe, also, that these moulded candles from Japan are +made more conical than the moulded candles in this part of the world. + +I told you, when we last met, a good deal about carbonic acid. We found, +by the lime-water test, that when the vapour from the top of the candle or +lamp was received into bottles, and tested by this solution of lime-water +(the composition of which I explained to you, and which you can make for +yourselves), we had that white opacity which was in fact calcareous +matter, like shells and corals, and many of the rocks and minerals in the +earth. But I have not yet told you fully and clearly the chemical history +of this substance—carbonic acid—as we have it from the candle, and I +must now resume that subject. We have seen the products, and the nature of +them, as they issue from the candle. We have traced the water to its +elements, and now we have to see where are the elements of the carbonic +acid supplied by the candle. A few experiments will shew this. You +remember that when a candle burns badly, it produces smoke; but if it is +burning well, there is no smoke. And you know that the brightness of the +candle is due to this smoke, which becomes ignited. Here is an experiment +to prove this: so long as the smoke remains in the flame of the candle and +becomes ignited, it gives a beautiful light, and never appears to us in +the form of black particles. I will light some fuel, which is extravagant +in its burning. This will serve our purpose—a little turpentine on a +sponge. You see the smoke rising from it, and floating into the air in +large quantities; and, remember now, the carbonic acid that we have from +the candle is from such smoke as that. To make that evident to you, I will +introduce this turpentine burning on the sponge into a flask where I have +plenty of oxygen, the rich part of the atmosphere, and you now see that +the smoke is all consumed. This is the first part of our experiment; and +now, what follows? The carbon which you saw flying off from the turpentine +flame in the air is now entirely burned in this oxygen, and we shall find +that it will, by this rough and temporary experiment, give us exactly the +same conclusion and result as we had from the combustion of the candle. +The reason why I make the experiment in this manner is solely that I may +cause the steps of our demonstration to be so simple that you can never +for a moment lose the train of reasoning, if you only pay attention. All +the carbon which is burned in oxygen, or air, comes out as carbonic acid, +whilst those particles which are not so burned shew you the second +substance in the carbonic acid—namely, the carbon—that body which made +the flame so bright whilst there was plenty of air, but which was thrown +off in excess when there was not oxygen enough to burn it. + +I have also to shew you a little more distinctly the history of carbon and +oxygen, in their union to make carbonic acid. You are now better able to +understand this than before, and I have prepared three or four experiments +by way of illustration. This jar is filled with oxygen, and here is some +carbon which has been placed in a crucible, for the purpose of being made +red-hot. I keep my jar dry, and venture to give you a result imperfect in +some degree, in order that I may make the experiment brighter. I am about +to put the oxygen and the carbon together. That this is carbon (common +charcoal pulverised), you will see by the way in which it burns in the air +[letting some of the red-hot charcoal fall out of the crucible]. I am now +about to burn it in oxygen gas, and look at the difference. It may appear +to you at a distance as if it were burning with a flame; but it is not so. +Every little piece of charcoal is burning as a spark, and whilst it so +burns it is producing carbonic acid. I specially want these two or three +experiments to point out what I shall dwell upon more distinctly +by-and-by—that carbon burns in this way, and not as a flame. + +Instead of taking many particles of carbon to burn, I will take a rather +large piece, which will enable you to see the form and size; and to trace +the effects very decidedly. Here is the jar of oxygen, and here is the +piece of charcoal, to which I have fastened a little piece of wood, which +I can set fire to, and so commence the combustion, which I could not +conveniently do without. You now see the charcoal burning, but not as a +flame (or if there be a flame, it is the smallest possible one, which I +know the cause of—namely, the formation of a little carbonic oxide close +upon the surface of the carbon). It goes on burning, you see, slowly +producing carbonic acid by the union of this carbon or charcoal (they are +equivalent terms) with the oxygen. I have here another piece of charcoal, +a piece of bark, which has the quality of being blown to pieces—exploding +as it burns. By the effect of the heat, we shall reduce the lump of carbon +into particles that will fly off; still every particle, equally with the +whole mass, burns in this peculiar way: it burns as a coal, and not like a +flame. You observe a multitude of little combustions going on, but no +flame. I do not know a finer experiment than this, to shew that carbon +burns with a spark. + +Here, then, is carbonic acid formed from its elements. It is produced at +once; and if we examined it by lime-water, you will see that we have the +same substance which I have previously described to you. By putting +together 6 parts of carbon by weight (whether it comes from the flame of a +candle or from powdered charcoal) and 16 parts of oxygen by weight, we +have 22 parts of carbonic acid; and, as we saw last time, the 22 parts of +carbonic acid, combined with 28 parts of lime, produced common carbonate +of lime. If you were to examine an oyster-shell, and weigh the component +parts, you would find that every 50 parts would give 6 of carbon and 16 of +oxygen, combined with 28 of lime. However, I do not want to trouble you +with these minutiæ—it is only the general philosophy of the matter that +we can now go into. See how finely the carbon is dissolving away [pointing +to the lump of charcoal burning quietly in the jar of oxygen]. You may say +that the charcoal is actually dissolving in the air round about; and if +that were perfectly pure charcoal, which we can easily prepare, there +would be no residue whatever. When we have a perfectly cleansed and +purified piece of carbon, there is no ash left. The carbon burns as a +solid dense body, that heat alone cannot change as to its solidity, and +yet it passes away into vapour that never condenses into solid or liquid +under ordinary circumstances; and what is more curious still, is the fact +that the oxygen does not change in its bulk by the solution of the carbon +in it. Just as the bulk is at first, so it is at last, only it has become +carbonic acid. + +There is another experiment which I must give you before you are fully +acquainted with the general nature of carbonic acid. Being a compound +body, consisting of carbon and oxygen, carbonic acid is a body that we +ought to be able to take asunder. And so we can. As we did with water, so +we can with carbonic acid—take the two parts asunder. The simplest and +quickest way is to act upon the carbonic acid by a substance that can +attract the oxygen from it, and leave the carbon behind. You recollect +that I took potassium and put it upon water or ice, and you saw that it +could take the oxygen from the hydrogen. Now, suppose we do something of +the same kind here with this carbonic acid. You know carbonic acid to be a +heavy gas. I will not test it with lime-water, as that will interfere with +our subsequent experiments; but I think the heaviness of the gas and the +power of extinguishing flame will be sufficient for our purpose. I +introduce a flame into the gas, and you will see whether it will be put +out. You see the light is extinguished. Indeed, the gas may, perhaps, put +out phosphorus, which, you know, has a pretty strong combustion. Here is a +piece of phosphorus heated to a high degree. I introduce it into gas, and +you observe the light is put out; but it will take fire again in the air, +because there it re-enters into combustion. Now, let me take a piece of +potassium, a substance which, even at common temperatures, can act upon +carbonic acid, though not sufficiently for our present purpose, because it +soon gets covered with a protecting coat; but if we warm it up to the +burning point in air, as we have a fair right to do, and as we have done +with phosphorus, you will see that it can burn in carbonic acid; and if it +burns, it will burn by taking oxygen, so that you will see what is left +behind. I am going, then, to burn this potassium in the carbonic acid, as +a proof of the existence of oxygen in the carbonic acid. [In the +preliminary process of heating, the potassium exploded.] Sometimes we get +an awkward piece of potassium that explodes, or something like it, when it +burns. I will take another piece; and now that it is heated, I introduce +it into the jar, and you perceive that it burns in the carbonic acid—not +so well as in the air, because the carbonic acid contains the oxygen +combined; but it does burn, and takes away the oxygen. If I now put this +potassium into water, I find that, besides the potash formed (which you +need not trouble about), there is a quantity of carbon produced. I have +here made the experiment in a very rough way; but I assure you that if I +were to make it carefully, devoting a day to it, instead of five minutes, +we should get all the proper amount of charcoal left in the spoon, or in +the place where the potassium was burned, so that there could be no doubt +as to the result. Here, then, is the carbon obtained from the carbonic +acid, as a common black substance; so that you have the entire proof of +the nature of carbonic acid as consisting of carbon and oxygen. And now, I +may tell you, that _whenever_ carbon burns under common circumstances, it +produces carbonic acid. + +Suppose I take this piece of wood, and put it into a bottle with +lime-water. I might shake that lime-water up with wood and the atmosphere +as long as I pleased, it would still remain clear as you see it; but +suppose I burn the piece of wood in the air of that bottle. You, of +course, know I get water. Do I get carbonic acid? [The experiment was +performed.] There it is, you see—that is to say, the carbonate lime, +which results from carbonic acid, and that carbonic acid must be formed +from the carbon which comes from the wood, from the candle, or any other +thing. Indeed, you have yourselves frequently tried a very pretty +experiment, by which you may see the carbon in wood. If you take a piece +of wood, and partly burn it, and then blow it out, you have carbon left. +There are things that do not shew carbon in this way. A candle does not +shew it, but it contains carbon. Here also is a jar of coal-gas, which +produces carbonic acid abundantly. You do not see the carbon, but we can +soon shew it to you. I will light it, and as long as there is any gas in +this cylinder it will go on burning. You see no carbon, but you see a +flame; and because that is bright, it will lead you to guess that there is +carbon in the flame. But I will shew it to you by another process. I have +some of the same gas in another vessel, mixed with a body that will burn +the hydrogen of the gas, but will not burn the carbon. I will light them +with a burning taper, and you perceive the hydrogen is consumed, but not +the carbon, which is left behind as a dense black smoke. I hope that by +these three or four experiments you will learn to see when carbon is +present, and understand what are the products of combustion, when gas or +other bodies are thoroughly burned in the air. + +Before we leave the subject of carbon, let us make a few experiments and +remarks upon its wonderful condition as respects ordinary combustion. I +have shewn you that the carbon in burning burns only as a solid body, and +yet you perceive that, after it is burned, it ceases to be a solid. There +are very few fuels that act like this. It is, in fact, only that great +source of fuel, the carbonaceous series, the coals, charcoals, and woods, +that can do it. I do not know that there is any other elementary substance +besides carbon that burns with these conditions; and if it had not been +so, what would happen to us? Suppose all fuel had been like iron, which, +when it burns, burns into a solid substance. We could not then have such a +combustion as you have in this fireplace. Here also is another kind of +fuel which burns very well—as well as, if not better, than carbon—so +well, indeed, as to take fire of itself when it is in the air, as you see +[breaking a tube full of lead pyrophorus]. This substance is lead, and you +see how wonderfully combustible it is. It is very much divided, and is +like a heap of coals in the fireplace; the air can get to its surface and +inside, and so it burns. But why does it not burn in that way now, when it +is lying in a mass? [emptying the contents of the tube in a heap on to a +plate of iron]. Simply because the air cannot get to it. Though it can +produce a great heat, the great heat which we want in our furnaces and +under our boilers, still that which is produced cannot get away from the +portion which remains unburned underneath, and that portion, therefore, is +prevented from coming in contact with the atmosphere, and cannot be +consumed. How different is that from carbon. Carbon burns just in the same +way as this lead does, and so gives an intense fire in the furnace, or +wherever you choose to burn it; but then the body produced by its +combustion passes away, and the remaining carbon is left clear. I shewed +you how carbon went on dissolving in the oxygen, leaving no ash; whereas +here [pointing to the heap of pyrophorus] we have actually more ash than +fuel, for it is heavier by the amount of the oxygen which has united with +it. Thus you see the difference between carbon and lead or iron: if we +choose iron, which gives so wonderful a result in our application of this +fuel, either as light or heat. If, when the carbon burnt, the product went +off as a solid body, you would have had the room filled with an opaque +substance, as in the case of the phosphorus; but when carbon burns, +everything passes up into the atmosphere. It is in a fixed, almost +unchangeable condition before the combustion; but afterwards it is in the +form of gas, which it is very difficult (though we have succeeded) to +produce in a solid or a liquid state. + +Now, I must take you to a very interesting part of our subject—to the +relation between the combustion of a candle and that living kind of +combustion which goes on within us. In every one of us there is a living +process of combustion going on very similar to that of a candle; and I +must try to make that plain to you. For it is not merely true in a +poetical sense—the relation of the life of man to a taper; and if you +will follow, I think I can make this clear. In order to make the relation +very plain, I have devised a little apparatus which we can soon build up +before you. Here is a board and a groove cut in it, and I can close the +groove at the top part by a little cover. I can then continue the groove +as a channel by a glass tube at each end, there being a free passage +through the whole. Suppose I take a taper or candle (we can now be liberal +in our use of the word “candle,” since we understand what it means), and +place it in one of the tubes; it will go on, you see, burning very well. +You observe that the air which feeds the flame passes down the tube at one +end, then goes along the horizontal tube, and ascends the tube at the +other end in which the taper is placed. + +If I stop the aperture through which the air enters, I stop combustion, as +you perceive. I stop the supply of air, and consequently the candle goes +out. But, now, what will you think of this fact? In a former experiment I +shewed you the air going from one burning candle to a second candle. If I +took the air proceeding from another candle, and sent it down by a +complicated arrangement into this tube, I should put this burning candle +out. But what will you say when I tell you that my breath will put out +that candle? I do not mean by blowing at all, but simply that the nature +of my breath is such that a candle cannot burn in it. I will now hold my +mouth over the aperture, and without blowing the flame in any way, let no +air enter the tube but what comes from my mouth. You see the result. I did +not blow the candle out. I merely let the air which I expired pass into +the aperture, and the result was that the light went out for want of +oxygen, and for no other reason. Something or other—namely, my lungs—had +taken away the oxygen from the air, and there was no more to supply the +combustion of the candle. It is, I think, very pretty to see the time it +takes before the bad air which I throw into this part of the apparatus has +reached the candle. The candle at first goes on burning, but so soon as +the air has had time to reach it, it goes out. And, now, I will shew you +another experiment, because this is an important part of our philosophy. +Here is a jar which contains fresh air, as you can see by the circumstance +of a candle or gas-light burning it. I make it close for a little time, +and by means of a pipe I get my mouth over it so that I can inhale the +air. By putting it over water, in the way that you see, I am able to draw +up this air (supposing the cork to be quite tight), take it into my lungs, +and throw it back into the jar. + +We can then examine it, and see the result. You observe, I first take up +the air, and then throw it back, as is evident from the ascent and descent +of the water; and now, by putting a taper into the air, you will see the +state in which it is, by the light being extinguished. Even one +inspiration, you see, has completely spoiled this air, so that it is no +use my trying to breathe it a second time. Now, you understand the ground +of the impropriety of many of the arrangements among the houses of the +poorer classes, by which the air is breathed over and over again, for the +want of a supply, by means of proper ventilation, sufficient to produce a +good result. You see how bad the air becomes by a single breathing; so +that you can easily understand how essential fresh air is to us. + +To pursue this a little further, let us see what will happen with +lime-water. Here is a globe which contains a little lime-water, and it is +so arranged as regards the pipes, as to give access to the air within, so +that we can ascertain the effect of respired or unrespired air upon it. Of +course, I can either draw in air (through A), and so make the air that +feeds my lungs go through the lime-water, or I can force the air out of my +lungs through the tube (B), which goes to the bottom, and so shew its +effect upon the lime-water. + +You will observe that, however long I draw the external air into the +lime-water, and then through it to my lungs, I shall produce no effect +upon the water—it will not make the lime-water turbid; but if I throw the +air _from_ my lungs through the lime-water, several times in succession, +you see how white and milky the water is getting, shewing the effect which +expired air has had upon it; and now you begin to know that the atmosphere +which we have spoiled by respiration is spoiled by carbonic acid, for you +see it here in contact with the lime-water. + +I have here two bottles, one containing lime-water and the other common +water, and tubes which pass into the bottles and connect them. The +apparatus is very rough, but it is useful notwithstanding. + +If I take these two bottles, inhaling here and exhaling there, the +arrangement of the tubes will prevent the air going backwards. The air +coming in will go to my mouth and lungs, and in going out, will pass +through the lime-water, so that I can go on breathing and making an +experiment, very refined in its nature, and very good in its results. You +will observe that the good air has done nothing to the lime-water; in the +other case nothing has come to the lime-water but my respiration, and you +see the difference in the two cases. + +Let us now go a little further. What is all this process going on within +us which we cannot do without, either day or night, which is so provided +for by the Author of all things that He has arranged that it shall be +independent of all will? If we restrain our respiration, as we can to a +certain extent, we should destroy ourselves. When we are asleep, the +organs of respiration, and the parts that are associated with them, still +go on with their action—so necessary is this process of respiration to +us, this contact of the air with the lungs. I must tell you, in the +briefest possible manner, what this process is. We consume food: the food +goes through that strange set of vessels and organs within us, and is +brought into various parts of the system, into the digestive parts +especially; and alternately the portion which is so changed is carried +through our lungs by one set of vessels, while the air that we inhale and +exhale is drawn into and thrown out of the lungs by another set of +vessels, so that the air and the food come close together, separated only +by an exceedingly thin surface: the air can thus act upon the blood by +this process, producing precisely the same results in kind as we have seen +in the case of the candle. The candle combines with parts of the air, +forming carbonic acid, and evolves heat; so in the lungs there is this +curious, wonderful change taking place. The air entering, combines with +the carbon (not carbon in a free state, but, as in this case, placed ready +for action at the moment), and makes carbonic acid, and is so thrown out +into the atmosphere, and thus this singular result takes place: we may +thus look upon the food as fuel. Let me take that piece of sugar, which +will serve my purpose. It is a compound of carbon, hydrogen, and oxygen, +similar to a candle, as containing the same elements, though not in the +same proportion—the proportions being as shewn in this table:— + + Carbon, . . . . 72 + _ + Hydrogen, . . . 11 | + | 99 + Oxygen, . . . . 88_| + +This is, indeed, a very curious thing, which you can well remember, for +the oxygen and hydrogen are in exactly the proportions which form water, +so that sugar may be said to be compounded of 72 parts of carbon and 99 +parts of water; and it is the carbon in the sugar that combines with the +oxygen carried in by the air in the process of respiration—so making us +like candles—producing these actions, warmth, and far more wonderful +results besides, for the sustenance of the system, by a most beautiful and +simple process. To make this still more striking, I will take a little +sugar; or, to hasten the experiment, I will use some syrup, which contains +about three-fourths of sugar and a little water. If I put a little oil of +vitriol on it, it takes away the water, and leaves the carbon in a black +mass. [The Lecturer mixed the two together.] You see how the carbon is +coming out, and before long we shall have a solid mass of charcoal, all of +which has come out of sugar. Sugar, as you know, is food, and here we have +absolutely a solid lump of carbon where you would not have expected it. +And if I make arrangements so as to oxidize the carbon of sugar, we shall +have a much more striking result Here is sugar, and I have here an +oxidizer—a quicker one than the atmosphere; and so we shall oxidize this +fuel by a process different from respiration in its form, though not +different in its kind. It is the combustion of the carbon by the contact +of oxygen which the body has supplied to it. If I set this into action at +once, you will see combustion produced. Just what occurs in my +lungs—taking in oxygen from another source, namely, the atmosphere—takes +place here by a more rapid process. + +You will be astonished when I tell you what this curious play of carbon +amounts to. A candle will burn some four, five, six, or seven hours. What, +then, must be the daily amount of carbon going up into the air in the way +of carbonic acid! What a quantity of carbon must go from each of us in +respiration! What a wonderful change of carbon must take place under these +circumstances of combustion or respiration! A man in twenty-four hours +converts as much as seven ounces of carbon into carbonic acid; a milch cow +will convert seventy ounces, and a horse seventy-nine ounces, solely by +the act of respiration. That is, the horse in twenty-four hours burns +seventy-nine ounces of charcoal, or carbon, in his organs of respiration, +to supply his natural warmth in that time. All the warm-blooded animals +get their warmth in this way, by the conversion of carbon, not in a free +state, but in a state of combination. And what an extraordinary notion +this gives us of the alterations going on in our atmosphere. As much as +5,000,000 pounds, or 548 tons, of carbonic acid is formed by respiration +in London alone in twenty-four hours. And where does all this go? Up into +the air. If the carbon had been like the lead which I shewed you, or the +iron which, in burning, produces a solid substance, what would happen? +Combustion could not go on. As charcoal burns, it becomes a vapour and +passes off into the atmosphere, which is the great vehicle, the great +carrier for conveying it away to other places. Then, what becomes of it? +Wonderful is it to find that the change produced by respiration, which +seems so injurious to us (for we cannot breathe air twice over), is the +very life and support of plants and vegetables that grow upon the surface +of the earth. It is the same also under the surface, in the great bodies +of water; for fishes and other animals respire upon the same principle, +though not exactly by contact with the open air. + +Such fish as I have here [pointing to a globe of gold-fish] respire by the +oxygen which is dissolved from the air by the water, and form carbonic +acid; and they all move about to produce the one great work of making the +animal and vegetable kingdoms subservient to each other. And all the +plants growing upon the surface of the earth, like that which I have +brought here to serve as an illustration, absorb carbon. These leaves are +taking up their carbon from the atmosphere, to which we have given it in +the form of carbonic acid, and they are growing and prospering. Give them +a pure air like ours, and they could not live in it; give them carbon with +other matters, and they live and rejoice. This piece of wood gets all its +carbon, as the trees and plants get theirs, from the atmosphere, which, as +we have seen, carries away what is bad for us and at the same time good +for them,—what is disease to the one being health to the other. So are we +made dependent, not merely upon our fellow-creatures, but upon our +fellow-existers, all Nature being tied together by the laws that make one +part conduce to the good of another. + +There is another little point which I must mention before we draw to a +close—a point which concerns the whole of these operations, and most +curious and beautiful it is to see it clustering upon and associated with +the bodies that concern us—oxygen, hydrogen, and carbon, in different +states of their existence. I shewed you just now some powdered lead, which +I set burning[18]; and you saw that the moment the fuel was brought to the +air, it acted, even before it got out of the bottle—the moment the air +crept in, it acted. Now, there is a case of chemical affinity by which all +our operations proceed. When we breathe, the same operation is going on +within us. When we burn a candle, the attraction of the different parts +one to the other is going on. Here it is going on in this case of the +lead; and it is a beautiful instance of chemical affinity. If the products +of combustion rose off from the surface, the lead would take fire, and go +on burning to the end; but you remember that we have this difference +between charcoal and lead—that, while the lead can start into action at +once, if there be access of air to it, the carbon will remain days, weeks, +months, or years. The manuscripts of Herculaneum were written with +carbonaceous ink, and there they have been for 1,800 years or more, not +having been at all changed by the atmosphere, though coming in contact +with it under various circumstances. Now, what is the circumstance which +makes the lead and carbon differ in this respect? It is a striking thing +to see that the matter which is appointed to serve the purpose of fuel +_waits_ in its action: it does not start off burning, like the lead and +many other things that I could shew you; but which I have not encumbered +the table with; but it waits for action. This waiting is a curious and +wonderful thing. Candles—those Japanese candles, for instance—do not +start into action at once, like the lead or iron (for iron finely divided +does the same thing as lead), but there they wait for years, perhaps for +ages, without undergoing any alteration. I have here a supply of coal-gas. +The jet is giving forth the gas, but you see it does not take fire—it +comes out into the air, but it waits till it is hot enough before it +burns. If I make it hot enough, it takes fire. If I blow it out, the gas +that is issuing forth waits till the light is applied to it again. It is +curious to see how different substances wait—how some will wait till the +temperature is raised a little, and others till it is raised a good deal. +I have here a little gunpowder and some gun-cotton; even these things +differ in the conditions under which they will burn. The gunpowder is +composed of carbon and other substances, making it highly combustible; and +the gun-cotton is another combustible preparation. They are both waiting, +but they will start into activity at different degrees of heat, or under +different conditions. By applying a heated wire to them, we shall see +which will start first [touching the gun-cotton with the hot iron]. You +see the gun-cotton has gone off, but not even the hottest part of the wire +is now hot enough to fire the gunpowder. How beautifully that shews you +the difference in the degree in which bodies act in this way! In the one +case the substance will wait any time until the associated bodies are made +active by heat; but in the other, as in the process of respiration, it +waits no time. In the lungs, as soon as the air enters, it unites with the +carbon; even in the lowest temperature which the body can bear short of +being frozen, the action begins at once, producing the carbonic acid of +respiration: and so all things go on fitly and properly. Thus you see the +analogy between respiration and combustion is rendered still more +beautiful and striking. Indeed, all I can say to you at the end of these +lectures (for we must come to an end at one time or other) is to express a +wish that you may, in your generation, be fit to compare to a candle; that +you may, like it, shine as lights to those about you; that, in all your +actions, you may justify the beauty of the taper by making your deeds +honourable and effectual in the discharge of your duty to your fellow-men. + +Whether I was to have the honour of appearing before you this evening or +not, seemed to be doubtful upon one or two points. One of these I will +mention immediately; the other may or may not appear during the course of +the hour that follows. The first point is this. When I was tempted to +promise this subject for your attention this evening, it was founded upon +a promise, and a full intent of performing that promise, on the part of my +friend Deville, of Paris, to come here to shew before you a phenomenon in +metallurgic chemistry not common. In that I have been disappointed. His +intention was to have fused here some thirty or forty pounds of platinum, +and so to have made manifest, through my mouth and my statement, the +principles of a new process in metallurgy, in relation to this beautiful, +magnificent, and valuable metal; but circumstances over which neither he +nor I, nor others concerned, have sufficient control, have prevented the +fulfilment of that intention; and the period at which I learned the fact +was so recent, that I could hardly leave my place here to be filled by +another, or permit you, who in your kindness have come to hear what might +be said, to remain unreceived in the best manner possible to me under the +circumstances. I therefore propose to state, as well as I can, what the +principles are on which M. Deville proceeds, by means of drawings, and +some subordinate or inferior experiments. The metal platinum, of which you +see some very fine specimens on the table, has been known to us about a +hundred years. It has been wrought in a beautiful way in this country, in +France, and elsewhere, and supplied to the consumer in ingots of this +kind, or in plates, such as we have here, or in masses, that by their very +fall upon the table indicate the great weight of the substance, which is, +indeed, nearly at the head of all substances in that respect. This +substance has been given to us hitherto mainly through the philosophy of +Dr. Wollaston, whom many of us know, and it is obtained in great purity +and beauty. It is a very remarkable metal in many points, besides its +known special uses. It usually comes to us in grains. Here is a very fine +specimen of native platinum in grains. Here is also a nugget or ingot, and +here are some small pieces gathered out of certain alluvial soils in +Brazil, Mexico, California, and the Uralian districts of Russia. + +It is strange that this metal is almost always found associated with some +four or five other metals, most curious in their qualities and +characteristics. They are called platiniferous metals; and they have not +only the relation of being always found associated in this manner, but +they have other relations of a curious nature, which I shall point out to +you by a reference to one of the tables behind me. This substance is +always native—it is always in the metallic state; and the metals with +which it is found connected, and which are rarely found elsewhere, are +palladium, rhodium, iridium, osmium, and ruthenium. We have the names in +one of the tables arranged in two columns, representing, as you see, two +groups—platinum, iridium, and osmium constituting one group; and +ruthenium, rhodium, and palladium the other. Three of these have the +chemical equivalent of 98½, and the others a chemical equivalent of +about half that number. Then the metals of one group have an extreme +specific gravity—platinum being, in fact, the lightest of the three, or +as light as the lightest. Osmium has a specific gravity of 21.4, and is +the heaviest body in nature; platinum is 21.15, and iridium the same; the +specific gravity of the other three being only about half that, namely, +11.3, 12.1, and 11.8. Then there is this curious relation, that palladium +and iridium are very much alike, so that you would scarcely know one from +the other, though one has only half the weight of the other, and only half +the equivalent power. So with iridium and rhodium, and osmium and +ruthenium, which are so closely allied that they make pairs, being +separated each from its own group. Then these metals are the most +infusible that we possess. Osmium is the most difficult to fuse: indeed, I +believe it never has been fused, while every other metal has. Ruthenium +comes next, iridium next, rhodium next, platinum next (so that it ranks +here as a pretty fusible metal, and yet we have been long accustomed to +speak of the infusibility of platinum), and next comes palladium, which is +the most fusible metal of the whole. It is a curious thing to see this +fine association of physical properties coming out in metals which are +grouped together somehow or other in nature, but, no doubt, by causes +which are related to analogous properties in their situation on the +surface of the earth, for it is in alluvial soils that these things are +found. + +Now, with regard to this substance, let me tell you briefly how we get it. +The process used to be this. The ore which I shewed you just now was +taken, and digested in nitro-muriatic acid of a certain strength, and +partly converted into a solution, with the leaving behind of certain +bodies that I have upon the table. The platinum being dissolved with care +in acids, to the solution the muriate of ammonia was added, as I am about +to add it here. A yellow precipitate was then thrown down, as you perceive +is the case now; and this, carefully washed and cleansed, gave us that +body [pointing to a specimen of the chloride of platinum and ammonium], +the other elements, or nearly all, being ejected. This substance being +heated, gave us what we call platinum sponge, or platinum in the metallic +state, so finely divided as to form a kind of heavy mass or sponge, which, +at the time that Dr. Wollaston first sent it forth, was not fusible for +the market or in the manufacturers’ workshops, inasmuch as the temperature +required was so high, and there were no furnaces that could bring the mass +into a globule, and cause the parts to adhere together. Most of our metals +that we obtain from nature, and work in our shops, are brought at last +into a mass by fusion. I am not aware that there is in the arts or +sciences any other than iron which is not so. Soft iron we do not bring +together by fusion, but by a process which is analogous to the one that +was followed in the case of platinum, namely, welding; for these divided +grains of spongy platinum having been well washed and sunk in water for +the purpose of excluding air, and pressed together, and heated, and +hammered, and pressed again, until they come into a pretty close, dense, +compact mass, did so cohere, that when the mass was put into the furnace +of charcoal, and raised to a high temperature, the particles, at first +infinitely divided—for they were chemically divided—adhered the one to +the other, each to all the rest, until they made that kind of substance +which you see here, which will bear rolling and expansion of every kind. +No other process than that has hitherto been adopted for the purpose of +obtaining this substance from the particles by solution, precipitation, +ignition, and welding. It certainly is a very fine thing to see that we +may so fully depend upon the properties of the various substances we have +to deal with; that we can, by carrying out our processes, obtain a +material like this, allowing of division and extension under a rolling +mill—a material of the finest possible kind, the parts being held +together, not with interstices, not with porosity, but so continuous that +no fluids can pass between them; and, as Dr. Wollaston beautifully shewed, +a globule of platinum fused by the voltaic battery and the oxyhydrogen +blowpipe, when drawn into a wire, was not sounder or stronger than this +wire made by the curious coalescence of the particles by the sticking +power that they had at high temperatures. This is the process adopted by +Messrs. Johnson and Matthey, to whose great kindness I am indebted for +these ingots and for the valuable assistance I have received in the +illustrations. + +The treatment, however, that I have to bring before you is of another +kind; and it is in the hope that we shall be able before long to have such +a thing as the manufacture of platinum of this kind, that I am encouraged +to come before you, and tell you how far Deville has gone in the matter, +and to give you illustrations of the principles on which he proceeds. I +think it is but fair that you should see an experiment shewing you the way +in which we get the adhesion of platinum. Probably you all know of the +welding of iron: you go into the smith’s shop, and you see him put the +handle of a poker on to the stem, and by a little management and the +application of heat he makes them one. You have no doubt seen him put the +iron into the fire and sprinkle a little sand upon it. He does not know +the philosophy he calls into play when he sprinkles a little sand over the +oxide of iron, but he has a fine philosophy there, or practises it, when +he gets his welding. I can shew you here this beautiful circumstance of +the sticking together of the particles up to the fullest possible +intensity of their combination. If you were to go into the workshops of +Mr. Matthey, and see them hammering and welding away, you would see the +value of the experiment I am about to shew you. I have here some +platinum-wire. This is a metal which resists the action of acids, resists +oxidation by heat, and change of any sort; and which, therefore, I may +heat in the atmosphere without any flux. I bend the wire so as to make the +ends cross: these I make hot by means of the blowpipe, and then, by giving +them a tap with a hammer, I shall make them into one piece. Now that the +pieces are united, I shall have great difficulty in pulling them apart, +though they are joined only at the point where the two cylindrical +surfaces came together. And now I have succeeded in pulling the wire +apart, the division is not at the point of welding, but where the force of +the pincers has cut it, so that the junction we have effected is a +complete one. This, then, is the principle of the manufacture and +production of platinum in the old way. + +The treatment which Deville proposes to carry out, and which he has +carried out to a rather large extent in reference to the Russian supply of +platinum, is one altogether by heat, having little or no reference to the +use of acids. That you may know what the problem is, look at this table, +which gives you the composition of such a piece of platinum ore as I +shewed you just now. Wherever it comes from, the composition is as +complicated, though the proportions vary:— + +This refers to the Uralian ore. In that state of combination, as shewn in +the table, the iridium and osmium are found combined in crystals, +sometimes to the amount of 0.5 per cent., and sometimes 3 or 4 per cent. +Now, this Deville proposes to deal with in the dry way, in the place of +dealing with it by any acid. + +I have here another kind of platinum; and I shew it to you for this +reason. The Russian Government, having large stores of platinum in their +dominions, have obtained it in a metallic state, and worked it into coin. +The coin I have in my hand is a twelve silver rouble piece. The rouble is +worth three shillings, and this coin is, therefore, of the value of +thirty-six shillings. The smaller coin is worth half that sum; and the +other, half of that. The metal, however, is unfit for coinage. When you +have the two metals, gold and silver, used for coinage, you have a little +confusion in the value of the two in the market; but when you have three +precious metals (for you may call platinum a precious metal) worked into +coin, they will be sure to run counter to one another. Indeed, the case +did happen, that the price of platinum coin fixed by the Government was +such, that it was worth while to purchase platinum in other countries, and +make coin of it, and then take it into that country and circulate it. The +result was, that the Russian Government stopped the issue. The composition +of this coin is—platinum, 97.0; iridium, 1.2; rhodium, 0.5; palladium, +0.25; a little copper, and a little iron. It is, in fact, bad platinum: it +scales, and it has an unfitness for commercial use and in the laboratory, +which the other well-purified platinum has not. It wants working over +again. + +Now, Deville’s process depends upon three points,—upon intense heat, +blowpipe action, and the volatility of certain metals. We know that there +are plenty of metals that are volatile; but this, I think, is the first +time that it has been proposed to use the volatility of certain +metals—such as gold and palladium—for the purpose of driving them off +and leaving something else behind. He counts largely upon the volatility +of metals which we have not been in the habit of considering volatile, but +which we have rather looked upon as fixed; and I must endeavour to +illustrate these three points by a few experiments. Perhaps I can best +show you what is required in the process of heating platinum by using that +source of heat which we have here, and which seems to be almost +illimitable—namely, the voltaic battery; for it is only in consequence of +the heat that the voltaic battery affects the platinum. By applying the +two extremities of the battery to this piece of platinum-wire, you will +see what result we shall obtain. You perceive that we can take about this +heating agent wherever we like, and deal with it as we please, limiting it +in any way. I am obliged to deal carefully with it; but even that +circumstance will have an interest for you in watching the experiment. +Contact is now made. The electric current, when compressed into thin +conducting-wires offering resistance, evolves heat to a large extent; and +this is the power by which we work. You see the intense glow immediately +imparted to the wire; and if I applied the heat continuously, the effect +of the current would be to melt the wire. As soon as the contact is +broken, the wire resumes its former appearance; and now that we make +contact again, you perceive the glow as before. [The experiment was +repeated several times in rapid succession.] You can see a line of light, +though you can scarcely perceive the wire; and now that it has melted with +the great heat, if you examine it, you will perceive that it is indeed a +set of irregularities from end to end—a set of little spheres, which are +strung upon an axis of platinum running through it. It is that wire which +Mr. Grove described as being produced at the moment when fusion of the +whole mass is commencing. In the same manner, if I take a tolerably thick +piece of platinum, and subject it to the heat that can be produced by this +battery, you will see the brilliancy of the effect produced. I shall put +on a pair of spectacles for the experiment, as there is an injurious +effect of the voltaic spark upon the eyes, if the action is continued; and +it is neither policy nor bravery to subject any organ to unnecessary +danger; and I want, at all events, to keep the full use of my eyes to the +end of the lecture. + +You now see the action of the heat upon the piece of platinum—heat so +great as to break in pieces the plate on which the drops of metal fall. +You perceive, then, that we have sufficiently powerful sources of heat in +nature to deal with platinum. I have here an apparatus by which the same +thing can be shewn. Here is a piece of platinum, which is put into a +crucible of carbon made at the end of one pole of the battery, and you +will see the brilliant light that will be produced. There is our furnace, +and the platinum is rapidly getting heated; and now you perceive that it +is melted, and throwing off little particles. What a magnificent +philosophical instrument this is. When you look at the result, which is +lying upon the charcoal, you will see a beautifully fused piece of +platinum. It is now a fiery globule, with a surface so bright, and smooth, +and reflecting, that I cannot tell whether it is transparent, or opaque, +or what. This, then, will give you an idea of what has to be done by any +process that pretends to deal with thirty, or forty, or fifty pounds of +platinum at once. + +Let me now tell you briefly what Deville proposes to do. First of all, he +takes this ore, with its impurities, and mixes it (as he finds it +essential and best) with its own weight of sulphuret of lead—lead +combined with sulphur. Both the lead and the sulphur are wanted; for the +iron that is there present, as you see by the table, is one of the most +annoying substances in the treatment that you can imagine, because it is +not volatile; and while the iron remains adhering to the platinum, the +platinum will not flow readily. It cannot be sent away by a high +temperature—sent into the atmosphere so as to leave the platinum behind. +Well, then, a hundred parts of ore and a hundred parts of sulphuret of +lead, with about fifty parts of metallic lead, being all mingled together +in a crucible, the sulphur of the sulphuret takes the iron, the copper, +and some of the other metals and impurities, and combines with them to +form a slag; and as it goes on boiling and oxidising, it carries off the +iron, and so a great cleansing takes place. + +Now, you ought to know that these metals, such as platinum, iridium, and +palladium, have a strong affinity for such metals as lead and tin, and +upon this a great deal depends. Very much depends upon the platinum +throwing out its impurities of iron and so forth, by being taken up with +the lead present in it. That you may have a notion of the great power that +platinum has of combining with other metals, I will refer you to a little +of the chemist’s experience—his bad experience. He knows very well that +if he takes a piece of platinum-foil, and heats a piece of lead upon it, +or if he takes a piece of platinum-foil, such as we have here, and heats +things upon it that have lead in them, his platinum is destroyed. I have +here a piece of platinum, and if I apply the heat of the spirit-lamp to +it, in consequence of the presence of this little piece of lead which I +will place on it, I shall make a hole in the metal. The heat of the lamp +itself would do no harm to the platinum, nor would other chemical means; +but because there is a little lead present, and there is an affinity +between the two substances, the bodies fuse together at once. You see the +hole I have made. It is large enough to put your finger in, though the +platinum itself was, as you saw, almost infusible, except by the voltaic +battery. For the purpose of shewing this fact in a more striking manner, I +have taken pieces of platinum-foil, tinfoil, and lead-foil, and rolled +them together; and if I apply the blowpipe to them, you will have, in +fact, a repetition on a larger scale of the experiment you saw just now +when the lead and platinum came together, and one spoiled the other. When +the metals are laid one upon the other, and folded together and heat +applied, you will not only see that the platinum runs to waste, but that +at the time when the platinum and lead are combined there is ignition +produced—there is a power of sustaining combustion. I have taken a large +piece, that you may see the phenomenon on a large scale. You saw the +ignition and the explosion which followed, of which we have here the +results—the consequence of the chemical affinity between the platinum and +the metals combined with it, which is the thing upon which Deville founds +his first result. + +When he has melted these substances and stirred them well up, and so +obtained a complete mixture, he throws in air upon the surface to burn off +all the sulphur from the remaining sulphuret of lead; and at last he gets +an ingot of lead with platinum—much lead, comparatively, and little +platinum. He gets that in the crucible with a lot of scoriæ and other +things, which he treats afterwards. It is that platiniferous lead which we +have to deal with in our future process. Now, let me tell you what he does +with it. His first object is to get rid of the lead. He has thrown out all +the iron, and a number of other things, and he has got this kind of +compound indicated in the table. He may get it as high as 78 per cent. of +platinum, and 22 of lead; or 5, or 10, or 15 of platinum, and 95, or 90, +or 85 of lead (which he calls weak platinum), and he then places it in the +kind of vessel that you see before you. Suppose we had the mixture here; +we should have to make it hot, and then throw in air upon the surface. The +combustible metal—that is, the lead—and the part that will oxidise, are +thoroughly oxidised; the litharge would flow out in a fused state into a +vessel placed to receive it, and the platinum remains behind. + +Here is the process which Deville adopts for the purpose of casting off +the lead, after he has got out the platinum from the ore. (Having made use +of your friend, you get rid of him as quickly as you can.) He gets his +heat by applying the combination of oxygen and hydrogen, or of carburetted +fuel, for the purpose of producing a fire. I have here a source of +coal-gas; there I have a source of hydrogen; and here I have a source of +oxygen. I have here also one of the blowpipes used by Deville in his +process for working platinum in the way I have spoken of. There are two +pipes, and one of them goes to the source of coal-gas, and the other to +the supply of oxygen. + +By uniting these we obtain a flame of such a heat as to melt platinum. You +will, perhaps, hardly imagine what the heat is, unless you have some proof +of it; but you will soon see that I have actually the power of melting +platinum. Here is a piece of platinum-foil running like wax under the +flame which I am bringing to bear against it. The question, however, is +whether we shall get heat enough to melt, not this small quantity, but +large masses—many pounds of the metal. Having obtained heat like this, +the next consideration is what vessel is he to employ which could retain +the platinum when so heated, or bear the effects of the flame? Such +vessels are happily well supplied at Paris, and are formed of a substance +which surrounds Paris; it is a kind of chalk (called, I believe, by +geologists, _calcaire grossière_), and it has the property of enduring an +extreme degree of heat. I am now going to get the highest heat that we can +obtain. First, I shew you the combustion of hydrogen by itself. I have not +a large supply, because the coal-gas is sufficient for most of our +purposes. If I put a piece of lime obtained from this chalk into the gas, +you see we get a pretty hot flame, which would burn one’s fingers a good +deal But now let me subject a piece of it to the joint action of oxygen +and hydrogen. I do this for the purpose of shewing you the value of lime +as a material for the furnaces and chambers that are to contain the +substances to be operated on, and that are consequently to sustain the +action of this extreme heat. Here we have the hydrogen and the oxygen, +which will give the most intense heat that can be obtained by chemical +action; and if I put a piece of lime into the flame, we get what is called +the lime-light. Now, with all the beauty and intensity of action which you +perceive, there is no sensible deterioration of the lime except by the +mechanical force of the current of gases rushing from the jet against the +lime, sweeping away such particles as are not strongly aggregated. “Vapour +of lime” some call it; and it may be so, but there is no other change of +the lime than that under the action of heat of this highly-exalted +chemical condition, though almost any other substance would melt at once. + +Then, as to the way in which the heat is applied to the substance. It is +all very well for me to take a piece of antimony, and fuse it in the flame +of a blowpipe. But if I tried this piece in the ordinary lamp flame, I +should do nothing; if I tried a smaller piece, I should do little or +nothing; and if I tried a still smaller piece, I should do little or +nothing; yet I have a condition which will represent what Deville carries +to the highest possible extent, and which we all carry to the highest +extent, in the use of the blowpipe. Suppose I take this piece of antimony: +I shall not be able to melt it in that flame of the candle by merely +holding it there; yet, by taking pains, we can even melt a piece of +platinum there. This is a preparation which I made for the purpose of +proving the fusibility of platinum in a common candle. There is a piece of +wire, drawn by that ingenious process of Dr. Wollaston’s, not more than +the three-thousandth part of an inch in diameter. He put the wire into the +middle of a cylinder of silver, and drew both together until the whole +compound was exceedingly thin; and then he dissolved away the silver by +nitric acid. There was left in the centre a substance which I can scarcely +see with an eye-glass, but which I know is there, and which I can make +visible, as you see, by putting it into the candle, where the heat makes +it glow like a spark. I have again and again tried this experiment +up-stairs in my own room, and have easily fused this platinum-wire by a +common candle. You see we have, therefore, heat enough in the candle, as +in the voltaic battery, or in the highly-exalted combustion of the +blowpipe, but we do not supply a continuous source of heat. In the very +act of this becoming ignited, the heat radiates so fast that you cannot +accumulate enough to cause the fusion of the wire, except under the most +careful arrangement. Thus I cannot melt that piece of antimony by simply +putting it into the candle; but if I put it upon charcoal, and drive the +fiery current against it, there will be heat enough to melt it. The beauty +of the blowpipe is, that it sends hot air (making hot air by the +combustion of the flame) against the thing to be heated. I have only to +hold the antimony in the course of that current, and particle by particle +of the current impinges upon the antimony, and so we get it melted. You +now see it red-hot, and I have no doubt it will continue to burn if I +withdraw it from the flame and continue to force the air on it. Now, you +see it burning without any heat but that of its own combustion, which I am +keeping up by sending the air against it. It would go out in a moment if I +took away the current of air from it; but there it is burning, and the +more air I give it, by this or any other action, the better it is. So, +then, we have here not merely a mighty source of heat, but a means of +driving the heat forcibly against substances. + +Let me shew you another experiment with a piece of iron. It will serve two +purposes—shewing you what the blowpipe does as a source of heat, and what +it does by sending that heat where it is wanted. I have taken iron in +contrast with silver or other metals, that you may see the difference of +action, and so be more interested in the experiment. Here is our fuel, the +coal-gas; and here our oxygen. Having thus my power of heat, I apply it to +the iron, which, as you see, soon gets red-hot. It is now flowing about +like a globule of melted mercury. But observe, I cannot raise any vapour: +it is now covered with a coat of melted oxide, and unless I have a great +power in my blowpipe, it is hardly possible to break through it. Now, +then, you see these beautiful sparks: you have not only a beautiful kind +of combustion, but you see the iron is being driven off, not producing +smoke, but burning in a fixed condition. How different this is from the +action of some other metals—that piece of antimony, for instance, which +we saw just now throwing off abundance of fumes. We can, of course, burn +away this iron by giving plenty of air to it; but with the bodies which +Deville wants to expose to this intense heat he has not that means: the +gas itself must have power enough to drive off the slag which forms on the +surface of the metal, and power to impinge upon the platinum so as to get +the full contact that he wants for the fusion to take place. We see here, +then, the means to which he resorts—oxygen, and either coal-gas or +water-gas[19], or pure hydrogen, for producing heat, and the blowpipe for +the purpose of impelling the heated current upon the metals. + +I have two or three rough drawings here, representing the kind of furnaces +which he employs. They are larger, however, than the actual furnaces he +uses. Even the furnace in which he carries on that most serious operation +of fusing fifty pounds of platinum at once is not much more than half the +size of the drawing. It is made of a piece of lime below and a piece of +lime above. You see how beautifully lime sustains heat without altering in +shape; and you may have thought how beautifully it prevents the +dissipation of the heat by its very bad conducting powers. + +While the front part of the lime which you saw here was so highly ignited, +I could at any moment touch the back of it without feeling any annoyance +from the heat So, by having a chamber of lime of this sort, he is able to +get a vessel to contain these metals with scarcely any loss of heat. He +puts the blowpipes through these apertures, and sends down these gases +upon the metals, which are gradually melted. He then puts in more metal +through a hole at the top. The results of the combustion issue out of the +aperture which you see represented. If there be strips of platinum, he +pushes them through the mouth out of which the heated current is coming, +and there they get red-hot and white-hot before they get into the bath of +platinum. So he is able to fuse a large body of platinum in this manner. +When the platinum is melted, he takes off the top and pours out from the +bottom piece, like a crucible, and makes his cast. This is the furnace by +which he fuses his forty pounds or fifty pounds of platinum at once. The +metal is raised to a heat that no eye can bear. There is no light and +shadow, no chiaro-oscuro there; all is the same intensity of glow. You +look in, and you cannot see where the metal or the lime is; it is all as +one. We have, therefore, a platform with a handle, which turns upon an +axis, that coincides with the gutter that is formed for the pouring of the +metal; and when all is known to be ready, by means of dark glasses, the +workmen take off the top piece and lift up the handle, and the mould being +then placed in a proper position, he knows that the issue of the metal +will be exactly in the line of the axis. No injury has ever happened from +the use of this plan. You know with what care it is necessary to carry +such a vessel of mercury as we have here, for fear of turning it over on +one side or the other; but if it be a vessel of melted platinum, the very +greatest care must be used, because the substance is twice as heavy: yet +no injury has been done to any of the workmen in this operation. + +I have said that Deville depends upon intense heat for carrying off +vapour; and this brings me to the point of shewing how vapours are carried +off. Here is a basin of mercury, which boils easily, as you know, and +gives us the opportunity of observing the facts and principles which are +to guide us. I have here two poles of the battery, and if I bring them +into contact with the mercury, see what a development of vapour we have. +The mercury is flying off rapidly; and I might, if I pleased, put all the +company around me in a bath of mercury vapour. And so, if we take this +piece of lead and treat it in the same way, it will also give off vapour. +Observe the fumes that rise from it; and even if it was so far enclosed +from the air that you could not form any litharge, you would still have +those abundant fumes flying off. I may also take a piece of gold, and shew +you the same thing. I have here a piece of gold which I put upon a clean +surface of Paris limestone. Applying the heat of the blowpipe to it, you +see how the heat drives off the vapour; and if you notice at the end of +the Lecture, you will observe on the stone a purple patch of condensed +gold. Thus you see a proof of the volatilisation of gold. It is the same +with silver. You will not be startled if I sometimes use one agent and +sometimes another to illustrate a particular point. The volatility of gold +and silver is the same thing, whether it be effected by the voltaic +battery or by the blowpipe. [A lump of silver was placed in a charcoal +crucible between the poles of a voltaic battery.] Now, look at the fumes +of silver, and observe the peculiar and beautiful green colour which they +produce. We shall now shew you this same process of boiling the silver, +cast on a screen from the electric lamp which you have before you; and +while Dr. Tyndall is kindly getting the lamp ready for this purpose, let +me tell you that Deville proposes to throw out in this way all these +extraneous things that I have spoken of, except two—namely, iridium and +rhodium. It so happens, as he says, that iridium and rhodium do make the +metal more capable of resisting the attacks of acids than platinum itself. +Alloys are compounded up to 25 per cent. of rhodium and iridium, by which +the chemical inaction of the platinum is increased, and also its +malleability and other physical properties. [The image of the voltaic +discharge through vapour of silver was now thrown upon the screen.] What +you have now on the screen is an inverted image of what you saw when we +heated the silver before. The fine stream that you see around the silver +is the discharge of the electric force that takes place, giving you that +glorious green light which you see in the ray; and if Dr. Tyndall will +open the top of the lamp, you will see the quantity of fumes that will +come out of the aperture, shewing you at once the volatility of silver. + +I have now finished this imperfect account. It is but an apology for not +having brought the process itself before you. I have done the best I could +under the circumstances; and I know your kindness well, for if I were not +aware that I might trust to it, I would not appear here so often as I have +done. The gradual loss of memory and of my other faculties is making +itself painfully evident to me, and requires, every time I appear before +you, the continued remembrance of your kindness to enable me to get +through my task. If I should happen to go on too long, or should fail in +doing what you might desire, remember it is yourselves who are chargeable, +by wishing me to remain. I have desired to retire, as I think every man +ought to do before his faculties become impaired; but I must confess that +the affection I have for this place, and for those who frequent this +place, is such, that I hardly know when the proper time has arrived. diff --git a/data/science-v2/train.txt b/data/science-v2/train.txt new file mode 100644 index 0000000..0d449e3 --- /dev/null +++ b/data/science-v2/train.txt @@ -0,0 +1,15725 @@ +When on board H.M.S. ‘Beagle,’ as naturalist, I was much struck with +certain facts in the distribution of the inhabitants of South America, +and in the geological relations of the present to the past inhabitants +of that continent. These facts seemed to me to throw some light on the +origin of species—that mystery of mysteries, as it has been called by +one of our greatest philosophers. On my return home, it occurred to me, +in 1837, that something might perhaps be made out on this question by +patiently accumulating and reflecting on all sorts of facts which could +possibly have any bearing on it. After five years’ work I allowed +myself to speculate on the subject, and drew up some short notes; these +I enlarged in 1844 into a sketch of the conclusions, which then seemed +to me probable: from that period to the present day I have steadily +pursued the same object. I hope that I may be excused for entering on +these personal details, as I give them to show that I have not been +hasty in coming to a decision. + +My work is now nearly finished; but as it will take me two or three +more years to complete it, and as my health is far from strong, I have +been urged to publish this Abstract. I have more especially been +induced to do this, as Mr. Wallace, who is now studying the +natural history of the Malay archipelago, has arrived at almost exactly +the same general conclusions that I have on the origin of species. Last +year he sent to me a memoir on this subject, with a request that I +would forward it to Sir Charles Lyell, who sent it to the Linnean +Society, and it is published in the third volume of the Journal of that +Society. Sir C. Lyell and Dr. Hooker, who both knew of my work—the +latter having read my sketch of 1844—honoured me by thinking it +advisable to publish, with Mr. Wallace’s excellent memoir, some brief +extracts from my manuscripts. + +This Abstract, which I now publish, must necessarily be imperfect. I +cannot here give references and authorities for my several statements; +and I must trust to the reader reposing some confidence in my accuracy. +No doubt errors will have crept in, though I hope I have always been +cautious in trusting to good authorities alone. I can here give only +the general conclusions at which I have arrived, with a few facts in +illustration, but which, I hope, in most cases will suffice. No one can +feel more sensible than I do of the necessity of hereafter publishing +in detail all the facts, with references, on which my conclusions have +been grounded; and I hope in a future work to do this. For I am well +aware that scarcely a single point is discussed in this volume on which +facts cannot be adduced, often apparently leading to conclusions +directly opposite to those at which I have arrived. A fair result can +be obtained only by fully stating and balancing the facts and arguments +on both sides of each question; and this cannot possibly be here done. + +I much regret that want of space prevents my having the satisfaction of +acknowledging the generous assistance which I have received from very +many naturalists, some of them personally unknown to me. I cannot, +however, +let this opportunity pass without expressing my deep obligations to Dr. +Hooker, who for the last fifteen years has aided me in every possible +way by his large stores of knowledge and his excellent judgment. + +In considering the Origin of Species, it is quite conceivable that a +naturalist, reflecting on the mutual affinities of organic beings, on +their embryological relations, their geographical distribution, +geological succession, and other such facts, might come to the +conclusion that each species had not been independently created, but +had descended, like varieties, from other species. Nevertheless, such a +conclusion, even if well founded, would be unsatisfactory, until it +could be shown how the innumerable species inhabiting this world have +been modified, so as to acquire that perfection of structure and +coadaptation which most justly excites our admiration. Naturalists +continually refer to external conditions, such as climate, food, etc., +as the only possible cause of variation. In one very limited sense, as +we shall hereafter see, this may be true; but it is preposterous to +attribute to mere external conditions, the structure, for instance, of +the woodpecker, with its feet, tail, beak, and tongue, so admirably +adapted to catch insects under the bark of trees. In the case of the +misseltoe, which draws its nourishment from certain trees, which has +seeds that must be transported by certain birds, and which has flowers +with separate sexes absolutely requiring the agency of certain insects +to bring pollen from one flower to the other, it is equally +preposterous to account for the structure of this parasite, with its +relations to several distinct organic beings, by the effects of +external conditions, or of habit, or of the volition of the plant +itself. + +The author of the ‘Vestiges of Creation’ would, I presume, say that, +after a certain unknown number of +generations, some bird had given birth to a woodpecker, and some plant +to the misseltoe, and that these had been produced perfect as we now +see them; but this assumption seems to me to be no explanation, for it +leaves the case of the coadaptations of organic beings to each other +and to their physical conditions of life, untouched and unexplained. + +It is, therefore, of the highest importance to gain a clear insight +into the means of modification and coadaptation. At the commencement of +my observations it seemed to me probable that a careful study of +domesticated animals and of cultivated plants would offer the best +chance of making out this obscure problem. Nor have I been +disappointed; in this and in all other perplexing cases I have +invariably found that our knowledge, imperfect though it be, of +variation under domestication, afforded the best and safest clue. I may +venture to express my conviction of the high value of such studies, +although they have been very commonly neglected by naturalists. + +From these considerations, I shall devote the first chapter of this +Abstract to Variation under Domestication. We shall thus see that a +large amount of hereditary modification is at least possible, and, what +is equally or more important, we shall see how great is the power of +man in accumulating by his Selection successive slight variations. I +will then pass on to the variability of species in a state of nature; +but I shall, unfortunately, be compelled to treat this subject far too +briefly, as it can be treated properly only by giving long catalogues +of facts. We shall, however, be enabled to discuss what circumstances +are most favourable to variation. In the next chapter the Struggle for +Existence amongst all organic beings throughout the world, which +inevitably follows from their high geometrical powers of +increase, will be treated of. This is the doctrine of Malthus, applied +to the whole animal and vegetable kingdoms. As many more individuals of +each species are born than can possibly survive; and as, consequently, +there is a frequently recurring struggle for existence, it follows that +any being, if it vary however slightly in any manner profitable to +itself, under the complex and sometimes varying conditions of life, +will have a better chance of surviving, and thus be _naturally +selected_. From the strong principle of inheritance, any selected +variety will tend to propagate its new and modified form. + +This fundamental subject of Natural Selection will be treated at some +length in the fourth chapter; and we shall then see how Natural +Selection almost inevitably causes much Extinction of the less improved +forms of life and induces what I have called Divergence of Character. +In the next chapter I shall discuss the complex and little known laws +of variation and of correlation of growth. In the four succeeding +chapters, the most apparent and gravest difficulties on the theory will +be given: namely, first, the difficulties of transitions, or in +understanding how a simple being or a simple organ can be changed and +perfected into a highly developed being or elaborately constructed +organ; secondly the subject of Instinct, or the mental powers of +animals, thirdly, Hybridism, or the infertility of species and the +fertility of varieties when intercrossed; and fourthly, the +imperfection of the Geological Record. In the next chapter I shall +consider the geological succession of organic beings throughout time; +in the eleventh and twelfth, their geographical distribution throughout +space; in the thirteenth, their classification or mutual affinities, +both when mature and in an embryonic condition. In the last chapter I +shall give a +brief recapitulation of the whole work, and a few concluding remarks. + +No one ought to feel surprise at much remaining as yet unexplained in +regard to the origin of species and varieties, if he makes due +allowance for our profound ignorance in regard to the mutual relations +of all the beings which live around us. Who can explain why one species +ranges widely and is very numerous, and why another allied species has +a narrow range and is rare? Yet these relations are of the highest +importance, for they determine the present welfare, and, as I believe, +the future success and modification of every inhabitant of this world. +Still less do we know of the mutual relations of the innumerable +inhabitants of the world during the many past geological epochs in its +history. Although much remains obscure, and will long remain obscure, I +can entertain no doubt, after the most deliberate study and +dispassionate judgment of which I am capable, that the view which most +naturalists entertain, and which I formerly entertained—namely, that +each species has been independently created—is erroneous. I am fully +convinced that species are not immutable; but that those belonging to +what are called the same genera are lineal descendants of some other +and generally extinct species, in the same manner as the acknowledged +varieties of any one species are the descendants of that species. +Furthermore, I am convinced that Natural Selection has been the main +but not exclusive means of modification. + +Causes of Variability. Effects of Habit. Correlation of Growth. +Inheritance. Character of Domestic Varieties. Difficulty of +distinguishing between Varieties and Species. Origin of Domestic +Varieties from one or more Species. Domestic Pigeons, their Differences +and Origin. Principle of Selection anciently followed, its Effects. +Methodical and Unconscious Selection. Unknown Origin of our Domestic +Productions. Circumstances favourable to Man’s power of Selection. + +When we look to the individuals of the same variety or sub-variety of +our older cultivated plants and animals, one of the first points which +strikes us, is, that they generally differ much more from each other, +than do the individuals of any one species or variety in a state of +nature. When we reflect on the vast diversity of the plants and animals +which have been cultivated, and which have varied during all ages under +the most different climates and treatment, I think we are driven to +conclude that this greater variability is simply due to our domestic +productions having been raised under conditions of life not so uniform +as, and somewhat different from, those to which the parent-species have +been exposed under nature. There is, also, I think, some probability in +the view propounded by Andrew Knight, that this variability may be +partly connected with excess of food. It seems pretty clear that +organic beings must be exposed during several generations to the new +conditions of life to cause any appreciable amount of variation; and +that when the organisation has once begun to vary, it generally +continues to vary for many generations. +No case is on record of a variable being ceasing to be variable under +cultivation. Our oldest cultivated plants, such as wheat, still often +yield new varieties: our oldest domesticated animals are still capable +of rapid improvement or modification. + +It has been disputed at what period of life the causes of variability, +whatever they may be, generally act; whether during the early or late +period of development of the embryo, or at the instant of conception. +Geoffroy St. Hilaire’s experiments show that unnatural treatment of the +embryo causes monstrosities; and monstrosities cannot be separated by +any clear line of distinction from mere variations. But I am strongly +inclined to suspect that the most frequent cause of variability may be +attributed to the male and female reproductive elements having been +affected prior to the act of conception. Several reasons make me +believe in this; but the chief one is the remarkable effect which +confinement or cultivation has on the functions of the reproductive +system; this system appearing to be far more susceptible than any other +part of the organisation, to the action of any change in the conditions +of life. Nothing is more easy than to tame an animal, and few things +more difficult than to get it to breed freely under confinement, even +in the many cases when the male and female unite. How many animals +there are which will not breed, though living long under not very close +confinement in their native country! This is generally attributed to +vitiated instincts; but how many cultivated plants display the utmost +vigour, and yet rarely or never seed! In some few such cases it has +been found out that very trifling changes, such as a little more or +less water at some particular period of growth, will determine whether +or not the plant sets a seed. I cannot here enter on the copious +details which I have collected on +this curious subject; but to show how singular the laws are which +determine the reproduction of animals under confinement, I may just +mention that carnivorous animals, even from the tropics, breed in this +country pretty freely under confinement, with the exception of the +plantigrades or bear family; whereas, carnivorous birds, with the +rarest exceptions, hardly ever lay fertile eggs. Many exotic plants +have pollen utterly worthless, in the same exact condition as in the +most sterile hybrids. When, on the one hand, we see domesticated +animals and plants, though often weak and sickly, yet breeding quite +freely under confinement; and when, on the other hand, we see +individuals, though taken young from a state of nature, perfectly +tamed, long-lived, and healthy (of which I could give numerous +instances), yet having their reproductive system so seriously affected +by unperceived causes as to fail in acting, we need not be surprised at +this system, when it does act under confinement, acting not quite +regularly, and producing offspring not perfectly like their parents or +variable. + +Sterility has been said to be the bane of horticulture; but on this +view we owe variability to the same cause which produces sterility; and +variability is the source of all the choicest productions of the +garden. I may add, that as some organisms will breed most freely under +the most unnatural conditions (for instance, the rabbit and ferret kept +in hutches), showing that their reproductive system has not been thus +affected; so will some animals and plants withstand domestication or +cultivation, and vary very slightly—perhaps hardly more than in a state +of nature. + +A long list could easily be given of “sporting plants;” by this term +gardeners mean a single bud or offset, which suddenly assumes a new and +sometimes very different character from that of the rest of the plant. +Such buds can be propagated by grafting, etc., and sometimes by seed. +These “sports” are extremely rare under nature, but far from rare under +cultivation; and in this case we see that the treatment of the parent +has affected a bud or offset, and not the ovules or pollen. But it is +the opinion of most physiologists that there is no essential difference +between a bud and an ovule in their earliest stages of formation; so +that, in fact, “sports” support my view, that variability may be +largely attributed to the ovules or pollen, or to both, having been +affected by the treatment of the parent prior to the act of conception. +These cases anyhow show that variation is not necessarily connected, as +some authors have supposed, with the act of generation. + +Seedlings from the same fruit, and the young of the same litter, +sometimes differ considerably from each other, though both the young +and the parents, as Müller has remarked, have apparently been exposed +to exactly the same conditions of life; and this shows how unimportant +the direct effects of the conditions of life are in comparison with the +laws of reproduction, and of growth, and of inheritance; for had the +action of the conditions been direct, if any of the young had varied, +all would probably have varied in the same manner. To judge how much, +in the case of any variation, we should attribute to the direct action +of heat, moisture, light, food, etc., is most difficult: my impression +is, that with animals such agencies have produced very little direct +effect, though apparently more in the case of plants. Under this point +of view, Mr. Buckman’s recent experiments on plants seem extremely +valuable. When all or nearly all the individuals exposed to certain +conditions are affected in the same way, the change at first appears to +be directly due to such conditions; but in some cases it can be shown +that quite opposite conditions produce +similar changes of structure. Nevertheless some slight amount of change +may, I think, be attributed to the direct action of the conditions of +life—as, in some cases, increased size from amount of food, colour from +particular kinds of food and from light, and perhaps the thickness of +fur from climate. + +Habit also has a decided influence, as in the period of flowering with +plants when transported from one climate to another. In animals it has +a more marked effect; for instance, I find in the domestic duck that +the bones of the wing weigh less and the bones of the leg more, in +proportion to the whole skeleton, than do the same bones in the +wild-duck; and I presume that this change may be safely attributed to +the domestic duck flying much less, and walking more, than its wild +parent. The great and inherited development of the udders in cows and +goats in countries where they are habitually milked, in comparison with +the state of these organs in other countries, is another instance of +the effect of use. Not a single domestic animal can be named which has +not in some country drooping ears; and the view suggested by some +authors, that the drooping is due to the disuse of the muscles of the +ear, from the animals not being much alarmed by danger, seems probable. + +There are many laws regulating variation, some few of which can be +dimly seen, and will be hereafter briefly mentioned. I will here only +allude to what may be called correlation of growth. Any change in the +embryo or larva will almost certainly entail changes in the mature +animal. In monstrosities, the correlations between quite distinct parts +are very curious; and many instances are given in Isidore Geoffroy St. +Hilaire’s great work on this subject. Breeders believe that long limbs +are almost always accompanied by an elongated head. Some instances of +correlation are quite whimsical; thus +cats with blue eyes are invariably deaf; colour and constitutional +peculiarities go together, of which many remarkable cases could be +given amongst animals and plants. From the facts collected by +Heusinger, it appears that white sheep and pigs are differently +affected from coloured individuals by certain vegetable poisons. +Hairless dogs have imperfect teeth; long-haired and coarse-haired +animals are apt to have, as is asserted, long or many horns; pigeons +with feathered feet have skin between their outer toes; pigeons with +short beaks have small feet, and those with long beaks large feet. +Hence, if man goes on selecting, and thus augmenting, any peculiarity, +he will almost certainly unconsciously modify other parts of the +structure, owing to the mysterious laws of the correlation of growth. + +The result of the various, quite unknown, or dimly seen laws of +variation is infinitely complex and diversified. It is well worth while +carefully to study the several treatises published on some of our old +cultivated plants, as on the hyacinth, potato, even the dahlia, etc.; +and it is really surprising to note the endless points in structure and +constitution in which the varieties and sub-varieties differ slightly +from each other. The whole organisation seems to have become plastic, +and tends to depart in some small degree from that of the parental +type. + +Any variation which is not inherited is unimportant for us. But the +number and diversity of inheritable deviations of structure, both those +of slight and those of considerable physiological importance, is +endless. Dr. Prosper Lucas’s treatise, in two large volumes, is the +fullest and the best on this subject. No breeder doubts how strong is +the tendency to inheritance: like produces like is his fundamental +belief: doubts have been thrown on this principle by theoretical +writers alone. When a +deviation appears not unfrequently, and we see it in the father and +child, we cannot tell whether it may not be due to the same original +cause acting on both; but when amongst individuals, apparently exposed +to the same conditions, any very rare deviation, due to some +extraordinary combination of circumstances, appears in the parent—say, +once amongst several million individuals—and it reappears in the child, +the mere doctrine of chances almost compels us to attribute its +reappearance to inheritance. Every one must have heard of cases of +albinism, prickly skin, hairy bodies, etc., appearing in several +members of the same family. If strange and rare deviations of structure +are truly inherited, less strange and commoner deviations may be freely +admitted to be inheritable. Perhaps the correct way of viewing the +whole subject, would be, to look at the inheritance of every character +whatever as the rule, and non-inheritance as the anomaly. + +The laws governing inheritance are quite unknown; no one can say why +the same peculiarity in different individuals of the same species, and +in individuals of different species, is sometimes inherited and +sometimes not so; why the child often reverts in certain characters to +its grandfather or grandmother or other much more remote ancestor; why +a peculiarity is often transmitted from one sex to both sexes or to one +sex alone, more commonly but not exclusively to the like sex. It is a +fact of some little importance to us, that peculiarities appearing in +the males of our domestic breeds are often transmitted either +exclusively, or in a much greater degree, to males alone. A much more +important rule, which I think may be trusted, is that, at whatever +period of life a peculiarity first appears, it tends to appear in the +offspring at a corresponding age, though sometimes earlier. In many +cases this could +not be otherwise: thus the inherited peculiarities in the horns of +cattle could appear only in the offspring when nearly mature; +peculiarities in the silkworm are known to appear at the corresponding +caterpillar or cocoon stage. But hereditary diseases and some other +facts make me believe that the rule has a wider extension, and that +when there is no apparent reason why a peculiarity should appear at any +particular age, yet that it does tend to appear in the offspring at the +same period at which it first appeared in the parent. I believe this +rule to be of the highest importance in explaining the laws of +embryology. These remarks are of course confined to the first +_appearance_ of the peculiarity, and not to its primary cause, which +may have acted on the ovules or male element; in nearly the same manner +as in the crossed offspring from a short-horned cow by a long-horned +bull, the greater length of horn, though appearing late in life, is +clearly due to the male element. + +Having alluded to the subject of reversion, I may here refer to a +statement often made by naturalists—namely, that our domestic +varieties, when run wild, gradually but certainly revert in character +to their aboriginal stocks. Hence it has been argued that no deductions +can be drawn from domestic races to species in a state of nature. I +have in vain endeavoured to discover on what decisive facts the above +statement has so often and so boldly been made. There would be great +difficulty in proving its truth: we may safely conclude that very many +of the most strongly-marked domestic varieties could not possibly live +in a wild state. In many cases we do not know what the aboriginal stock +was, and so could not tell whether or not nearly perfect reversion had +ensued. It would be quite necessary, in order to prevent the effects of +intercrossing, that only a +single variety should be turned loose in its new home. Nevertheless, as +our varieties certainly do occasionally revert in some of their +characters to ancestral forms, it seems to me not improbable, that if +we could succeed in naturalising, or were to cultivate, during many +generations, the several races, for instance, of the cabbage, in very +poor soil (in which case, however, some effect would have to be +attributed to the direct action of the poor soil), that they would to a +large extent, or even wholly, revert to the wild aboriginal stock. +Whether or not the experiment would succeed, is not of great importance +for our line of argument; for by the experiment itself the conditions +of life are changed. If it could be shown that our domestic varieties +manifested a strong tendency to reversion,—that is, to lose their +acquired characters, whilst kept under unchanged conditions, and whilst +kept in a considerable body, so that free intercrossing might check, by +blending together, any slight deviations of structure, in such case, I +grant that we could deduce nothing from domestic varieties in regard to +species. But there is not a shadow of evidence in favour of this view: +to assert that we could not breed our cart and race-horses, long and +short-horned cattle, and poultry of various breeds, and esculent +vegetables, for an almost infinite number of generations, would be +opposed to all experience. I may add, that when under nature the +conditions of life do change, variations and reversions of character +probably do occur; but natural selection, as will hereafter be +explained, will determine how far the new characters thus arising shall +be preserved. + +When we look to the hereditary varieties or races of our domestic +animals and plants, and compare them with species closely allied +together, we generally perceive in each domestic race, as already +remarked, less uniformity of character than in true species. Domestic +races of +the same species, also, often have a somewhat monstrous character; by +which I mean, that, although differing from each other, and from the +other species of the same genus, in several trifling respects, they +often differ in an extreme degree in some one part, both when compared +one with another, and more especially when compared with all the +species in nature to which they are nearest allied. With these +exceptions (and with that of the perfect fertility of varieties when +crossed,—a subject hereafter to be discussed), domestic races of the +same species differ from each other in the same manner as, only in most +cases in a lesser degree than, do closely-allied species of the same +genus in a state of nature. I think this must be admitted, when we find +that there are hardly any domestic races, either amongst animals or +plants, which have not been ranked by some competent judges as mere +varieties, and by other competent judges as the descendants of +aboriginally distinct species. If any marked distinction existed +between domestic races and species, this source of doubt could not so +perpetually recur. It has often been stated that domestic races do not +differ from each other in characters of generic value. I think it could +be shown that this statement is hardly correct; but naturalists differ +most widely in determining what characters are of generic value; all +such valuations being at present empirical. Moreover, on the view of +the origin of genera which I shall presently give, we have no right to +expect often to meet with generic differences in our domesticated +productions. + +When we attempt to estimate the amount of structural difference between +the domestic races of the same species, we are soon involved in doubt, +from not knowing whether they have descended from one or several +parent-species. This point, if it could be cleared up, would be +interesting; if, for instance, it could be shown that the greyhound, +bloodhound, terrier, spaniel, and bull-dog, which we all know propagate +their kind so truly, were the offspring of any single species, then +such facts would have great weight in making us doubt about the +immutability of the many very closely allied and natural species—for +instance, of the many foxes—inhabiting different quarters of the world. +I do not believe, as we shall presently see, that all our dogs have +descended from any one wild species; but, in the case of some other +domestic races, there is presumptive, or even strong, evidence in +favour of this view. + +It has often been assumed that man has chosen for domestication animals +and plants having an extraordinary inherent tendency to vary, and +likewise to withstand diverse climates. I do not dispute that these +capacities have added largely to the value of most of our domesticated +productions; but how could a savage possibly know, when he first tamed +an animal, whether it would vary in succeeding generations, and whether +it would endure other climates? Has the little variability of the ass +or guinea-fowl, or the small power of endurance of warmth by the +rein-deer, or of cold by the common camel, prevented their +domestication? I cannot doubt that if other animals and plants, equal +in number to our domesticated productions, and belonging to equally +diverse classes and countries, were taken from a state of nature, and +could be made to breed for an equal number of generations under +domestication, they would vary on an average as largely as the parent +species of our existing domesticated productions have varied. + +In the case of most of our anciently domesticated animals and plants, I +do not think it is possible to come to any definite conclusion, whether +they have descended from one or several species. The argument mainly +relied on by those who believe in the multiple origin +of our domestic animals is, that we find in the most ancient records, +more especially on the monuments of Egypt, much diversity in the +breeds; and that some of the breeds closely resemble, perhaps are +identical with, those still existing. Even if this latter fact were +found more strictly and generally true than seems to me to be the case, +what does it show, but that some of our breeds originated there, four +or five thousand years ago? But Mr. Horner’s researches have rendered +it in some degree probable that man sufficiently civilized to have +manufactured pottery existed in the valley of the Nile thirteen or +fourteen thousand years ago; and who will pretend to say how long +before these ancient periods, savages, like those of Tierra del Fuego +or Australia, who possess a semi-domestic dog, may not have existed in +Egypt? + +The whole subject must, I think, remain vague; nevertheless, I may, +without here entering on any details, state that, from geographical and +other considerations, I think it highly probable that our domestic dogs +have descended from several wild species. In regard to sheep and goats +I can form no opinion. I should think, from facts communicated to me by +Mr. Blyth, on the habits, voice, and constitution, etc., of the humped +Indian cattle, that these had descended from a different aboriginal +stock from our European cattle; and several competent judges believe +that these latter have had more than one wild parent. With respect to +horses, from reasons which I cannot give here, I am doubtfully inclined +to believe, in opposition to several authors, that all the races have +descended from one wild stock. Mr. Blyth, whose opinion, from his large +and varied stores of knowledge, I should value more than that of almost +any one, thinks that all the breeds of poultry have proceeded from the +common wild +Indian fowl (Gallus bankiva). In regard to ducks and rabbits, the +breeds of which differ considerably from each other in structure, I do +not doubt that they all have descended from the common wild duck and +rabbit. + +The doctrine of the origin of our several domestic races from several +aboriginal stocks, has been carried to an absurd extreme by some +authors. They believe that every race which breeds true, let the +distinctive characters be ever so slight, has had its wild prototype. +At this rate there must have existed at least a score of species of +wild cattle, as many sheep, and several goats in Europe alone, and +several even within Great Britain. One author believes that there +formerly existed in Great Britain eleven wild species of sheep peculiar +to it! When we bear in mind that Britain has now hardly one peculiar +mammal, and France but few distinct from those of Germany and +conversely, and so with Hungary, Spain, etc., but that each of these +kingdoms possesses several peculiar breeds of cattle, sheep, etc., we +must admit that many domestic breeds have originated in Europe; for +whence could they have been derived, as these several countries do not +possess a number of peculiar species as distinct parent-stocks? So it +is in India. Even in the case of the domestic dogs of the whole world, +which I fully admit have probably descended from several wild species, +I cannot doubt that there has been an immense amount of inherited +variation. Who can believe that animals closely resembling the Italian +greyhound, the bloodhound, the bull-dog, or Blenheim spaniel, etc.—so +unlike all wild Canidæ—ever existed freely in a state of nature? It has +often been loosely said that all our races of dogs have been produced +by the crossing of a few aboriginal species; but by crossing we can get +only forms in some degree intermediate between their parents; and if we +account for our several domestic races by this process, we must admit +the former existence of the most extreme forms, as the Italian +greyhound, bloodhound, bull-dog, etc., in the wild state. Moreover, the +possibility of making distinct races by crossing has been greatly +exaggerated. There can be no doubt that a race may be modified by +occasional crosses, if aided by the careful selection of those +individual mongrels, which present any desired character; but that a +race could be obtained nearly intermediate between two extremely +different races or species, I can hardly believe. Sir J. Sebright +expressly experimentised for this object, and failed. The offspring +from the first cross between two pure breeds is tolerably and sometimes +(as I have found with pigeons) extremely uniform, and everything seems +simple enough; but when these mongrels are crossed one with another for +several generations, hardly two of them will be alike, and then the +extreme difficulty, or rather utter hopelessness, of the task becomes +apparent. Certainly, a breed intermediate between _two very distinct_ +breeds could not be got without extreme care and long-continued +selection; nor can I find a single case on record of a permanent race +having been thus formed. + +_On the Breeds of the Domestic Pigeon_.—Believing that it is always +best to study some special group, I have, after deliberation, taken up +domestic pigeons. I have kept every breed which I could purchase or +obtain, and have been most kindly favoured with skins from several +quarters of the world, more especially by the Honourable W. Elliot from +India, and by the Honourable C. Murray from Persia. Many treatises in +different languages have been published on pigeons, and some of them +are very important, as being of considerable antiquity. I have +associated with several eminent fanciers, and have been permitted to +join two +of the London Pigeon Clubs. The diversity of the breeds is something +astonishing. Compare the English carrier and the short-faced tumbler, +and see the wonderful difference in their beaks, entailing +corresponding differences in their skulls. The carrier, more especially +the male bird, is also remarkable from the wonderful development of the +carunculated skin about the head, and this is accompanied by greatly +elongated eyelids, very large external orifices to the nostrils, and a +wide gape of mouth. The short-faced tumbler has a beak in outline +almost like that of a finch; and the common tumbler has the singular +and strictly inherited habit of flying at a great height in a compact +flock, and tumbling in the air head over heels. The runt is a bird of +great size, with long, massive beak and large feet; some of the +sub-breeds of runts have very long necks, others very long wings and +tails, others singularly short tails. The barb is allied to the +carrier, but, instead of a very long beak, has a very short and very +broad one. The pouter has a much elongated body, wings, and legs; and +its enormously developed crop, which it glories in inflating, may well +excite astonishment and even laughter. The turbit has a very short and +conical beak, with a line of reversed feathers down the breast; and it +has the habit of continually expanding slightly the upper part of the +oesophagus. The Jacobin has the feathers so much reversed along the +back of the neck that they form a hood, and it has, proportionally to +its size, much elongated wing and tail feathers. The trumpeter and +laugher, as their names express, utter a very different coo from the +other breeds. The fantail has thirty or even forty tail-feathers, +instead of twelve or fourteen, the normal number in all members of the +great pigeon family; and these feathers are kept expanded, and are +carried so erect that in good birds the head and tail +touch; the oil-gland is quite aborted. Several other less distinct +breeds might have been specified. + +In the skeletons of the several breeds, the development of the bones of +the face in length and breadth and curvature differs enormously. The +shape, as well as the breadth and length of the ramus of the lower jaw, +varies in a highly remarkable manner. The number of the caudal and +sacral vertebræ vary; as does the number of the ribs, together with +their relative breadth and the presence of processes. The size and +shape of the apertures in the sternum are highly variable; so is the +degree of divergence and relative size of the two arms of the furcula. +The proportional width of the gape of mouth, the proportional length of +the eyelids, of the orifice of the nostrils, of the tongue (not always +in strict correlation with the length of beak), the size of the crop +and of the upper part of the oesophagus; the development and abortion +of the oil-gland; the number of the primary wing and caudal feathers; +the relative length of wing and tail to each other and to the body; the +relative length of leg and of the feet; the number of scutellæ on the +toes, the development of skin between the toes, are all points of +structure which are variable. The period at which the perfect plumage +is acquired varies, as does the state of the down with which the +nestling birds are clothed when hatched. The shape and size of the eggs +vary. The manner of flight differs remarkably; as does in some breeds +the voice and disposition. Lastly, in certain breeds, the males and +females have come to differ to a slight degree from each other. + +Altogether at least a score of pigeons might be chosen, which if shown +to an ornithologist, and he were told that they were wild birds, would +certainly, I think, be ranked by him as well-defined species. Moreover, +I do not believe that any ornithologist would place touch; the +oil-gland is quite aborted. Several other less distinct breeds might +have been specified. +the English carrier, the short-faced tumbler, the runt, the barb, +pouter, and fantail in the same genus; more especially as in each of +these breeds several truly-inherited sub-breeds, or species as he might +have called them, could be shown him. + +Great as the differences are between the breeds of pigeons, I am fully +convinced that the common opinion of naturalists is correct, namely, +that all have descended from the rock-pigeon (Columba livia), including +under this term several geographical races or sub-species, which differ +from each other in the most trifling respects. As several of the +reasons which have led me to this belief are in some degree applicable +in other cases, I will here briefly give them. If the several breeds +are not varieties, and have not proceeded from the rock-pigeon, they +must have descended from at least seven or eight aboriginal stocks; for +it is impossible to make the present domestic breeds by the crossing of +any lesser number: how, for instance, could a pouter be produced by +crossing two breeds unless one of the parent-stocks possessed the +characteristic enormous crop? The supposed aboriginal stocks must all +have been rock-pigeons, that is, not breeding or willingly perching on +trees. But besides C. livia, with its geographical sub-species, only +two or three other species of rock-pigeons are known; and these have +not any of the characters of the domestic breeds. Hence the supposed +aboriginal stocks must either still exist in the countries where they +were originally domesticated, and yet be unknown to ornithologists; and +this, considering their size, habits, and remarkable characters, seems +very improbable; or they must have become extinct in the wild state. +But birds breeding on precipices, and good fliers, are unlikely to be +exterminated; and the common rock-pigeon, which has the same habits +with the domestic breeds, has not been exterminated +even on several of the smaller British islets, or on the shores of the +Mediterranean. Hence the supposed extermination of so many species +having similar habits with the rock-pigeon seems to me a very rash +assumption. Moreover, the several above-named domesticated breeds have +been transported to all parts of the world, and, therefore, some of +them must have been carried back again into their native country; but +not one has ever become wild or feral, though the dovecot-pigeon, which +is the rock-pigeon in a very slightly altered state, has become feral +in several places. Again, all recent experience shows that it is most +difficult to get any wild animal to breed freely under domestication; +yet on the hypothesis of the multiple origin of our pigeons, it must be +assumed that at least seven or eight species were so thoroughly +domesticated in ancient times by half-civilized man, as to be quite +prolific under confinement. + +An argument, as it seems to me, of great weight, and applicable in +several other cases, is, that the above-specified breeds, though +agreeing generally in constitution, habits, voice, colouring, and in +most parts of their structure, with the wild rock-pigeon, yet are +certainly highly abnormal in other parts of their structure: we may +look in vain throughout the whole great family of Columbidæ for a beak +like that of the English carrier, or that of the short-faced tumbler, +or barb; for reversed feathers like those of the jacobin; for a crop +like that of the pouter; for tail-feathers like those of the fantail. +Hence it must be assumed not only that half-civilized man succeeded in +thoroughly domesticating several species, but that he intentionally or +by chance picked out extraordinarily abnormal species; and further, +that these very species have since all become extinct or unknown. So +many strange contingencies seem to me improbable in the highest degree. + +Some facts in regard to the colouring of pigeons well deserve +consideration. The rock-pigeon is of a slaty-blue, and has a white rump +(the Indian sub-species, C. intermedia of Strickland, having it +bluish); the tail has a terminal dark bar, with the bases of the outer +feathers externally edged with white; the wings have two black bars; +some semi-domestic breeds and some apparently truly wild breeds have, +besides the two black bars, the wings chequered with black. These +several marks do not occur together in any other species of the whole +family. Now, in every one of the domestic breeds, taking thoroughly +well-bred birds, all the above marks, even to the white edging of the +outer tail-feathers, sometimes concur perfectly developed. Moreover, +when two birds belonging to two distinct breeds are crossed, neither of +which is blue or has any of the above-specified marks, the mongrel +offspring are very apt suddenly to acquire these characters; for +instance, I crossed some uniformly white fantails with some uniformly +black barbs, and they produced mottled brown and black birds; these I +again crossed together, and one grandchild of the pure white fantail +and pure black barb was of as beautiful a blue colour, with the white +rump, double black wing-bar, and barred and white-edged tail-feathers, +as any wild rock-pigeon! We can understand these facts, on the +well-known principle of reversion to ancestral characters, if all the +domestic breeds have descended from the rock-pigeon. But if we deny +this, we must make one of the two following highly improbable +suppositions. Either, firstly, that all the several imagined aboriginal +stocks were coloured and marked like the rock-pigeon, although no other +existing species is thus coloured and marked, so that in each separate +breed there might be a tendency to revert to the very same colours and +markings. Or, secondly, +that each breed, even the purest, has within a dozen or, at most, +within a score of generations, been crossed by the rock-pigeon: I say +within a dozen or twenty generations, for we know of no fact +countenancing the belief that the child ever reverts to some one +ancestor, removed by a greater number of generations. In a breed which +has been crossed only once with some distinct breed, the tendency to +reversion to any character derived from such cross will naturally +become less and less, as in each succeeding generation there will be +less of the foreign blood; but when there has been no cross with a +distinct breed, and there is a tendency in both parents to revert to a +character, which has been lost during some former generation, this +tendency, for all that we can see to the contrary, may be transmitted +undiminished for an indefinite number of generations. These two +distinct cases are often confounded in treatises on inheritance. + +Lastly, the hybrids or mongrels from between all the domestic breeds of +pigeons are perfectly fertile. I can state this from my own +observations, purposely made on the most distinct breeds. Now, it is +difficult, perhaps impossible, to bring forward one case of the hybrid +offspring of two animals _clearly distinct_ being themselves perfectly +fertile. Some authors believe that long-continued domestication +eliminates this strong tendency to sterility: from the history of the +dog I think there is some probability in this hypothesis, if applied to +species closely related together, though it is unsupported by a single +experiment. But to extend the hypothesis so far as to suppose that +species, aboriginally as distinct as carriers, tumblers, pouters, and +fantails now are, should yield offspring perfectly fertile, _inter se_, +seems to me rash in the extreme. + +From these several reasons, namely, the improbability of man having +formerly got seven or eight supposed +species of pigeons to breed freely under domestication; these supposed +species being quite unknown in a wild state, and their becoming nowhere +feral; these species having very abnormal characters in certain +respects, as compared with all other Columbidæ, though so like in most +other respects to the rock-pigeon; the blue colour and various marks +occasionally appearing in all the breeds, both when kept pure and when +crossed; the mongrel offspring being perfectly fertile;—from these +several reasons, taken together, I can feel no doubt that all our +domestic breeds have descended from the Columba livia with its +geographical sub-species. + +In favour of this view, I may add, firstly, that C. livia, or the +rock-pigeon, has been found capable of domestication in Europe and in +India; and that it agrees in habits and in a great number of points of +structure with all the domestic breeds. Secondly, although an English +carrier or short-faced tumbler differs immensely in certain characters +from the rock-pigeon, yet by comparing the several sub-breeds of these +breeds, more especially those brought from distant countries, we can +make an almost perfect series between the extremes of structure. +Thirdly, those characters which are mainly distinctive of each breed, +for instance the wattle and length of beak of the carrier, the +shortness of that of the tumbler, and the number of tail-feathers in +the fantail, are in each breed eminently variable; and the explanation +of this fact will be obvious when we come to treat of selection. +Fourthly, pigeons have been watched, and tended with the utmost care, +and loved by many people. They have been domesticated for thousands of +years in several quarters of the world; the earliest known record of +pigeons is in the fifth Aegyptian dynasty, about 3000 B.C., as was +pointed out to me by Professor Lepsius; but Mr. Birch informs me that +pigeons are given in a bill +of fare in the previous dynasty. In the time of the Romans, as we hear +from Pliny, immense prices were given for pigeons; “nay, they are come +to this pass, that they can reckon up their pedigree and race.” Pigeons +were much valued by Akber Khan in India, about the year 1600; never +less than 20,000 pigeons were taken with the court. “The monarchs of +Iran and Turan sent him some very rare birds;” and, continues the +courtly historian, “His Majesty by crossing the breeds, which method +was never practised before, has improved them astonishingly.” About +this same period the Dutch were as eager about pigeons as were the old +Romans. The paramount importance of these considerations in explaining +the immense amount of variation which pigeons have undergone, will be +obvious when we treat of Selection. We shall then, also, see how it is +that the breeds so often have a somewhat monstrous character. It is +also a most favourable circumstance for the production of distinct +breeds, that male and female pigeons can be easily mated for life; and +thus different breeds can be kept together in the same aviary. + +I have discussed the probable origin of domestic pigeons at some, yet +quite insufficient, length; because when I first kept pigeons and +watched the several kinds, knowing well how true they bred, I felt +fully as much difficulty in believing that they could ever have +descended from a common parent, as any naturalist could in coming to a +similar conclusion in regard to the many species of finches, or other +large groups of birds, in nature. One circumstance has struck me much; +namely, that all the breeders of the various domestic animals and the +cultivators of plants, with whom I have ever conversed, or whose +treatises I have read, are firmly convinced that the several breeds to +which each has attended, are descended from so many aboriginally +distinct species. +Ask, as I have asked, a celebrated raiser of Hereford cattle, whether +his cattle might not have descended from long horns, and he will laugh +you to scorn. I have never met a pigeon, or poultry, or duck, or rabbit +fancier, who was not fully convinced that each main breed was descended +from a distinct species. Van Mons, in his treatise on pears and apples, +shows how utterly he disbelieves that the several sorts, for instance a +Ribston-pippin or Codlin-apple, could ever have proceeded from the +seeds of the same tree. Innumerable other examples could be given. The +explanation, I think, is simple: from long-continued study they are +strongly impressed with the differences between the several races; and +though they well know that each race varies slightly, for they win +their prizes by selecting such slight differences, yet they ignore all +general arguments, and refuse to sum up in their minds slight +differences accumulated during many successive generations. May not +those naturalists who, knowing far less of the laws of inheritance than +does the breeder, and knowing no more than he does of the intermediate +links in the long lines of descent, yet admit that many of our domestic +races have descended from the same parents—may they not learn a lesson +of caution, when they deride the idea of species in a state of nature +being lineal descendants of other species? + +_Selection_.—Let us now briefly consider the steps by which domestic +races have been produced, either from one or from several allied +species. Some little effect may, perhaps, be attributed to the direct +action of the external conditions of life, and some little to habit; +but he would be a bold man who would account by such agencies for the +differences of a dray and race horse, a greyhound and bloodhound, a +carrier and tumbler pigeon. One of the most remarkable features in our +domesticated races +is that we see in them adaptation, not indeed to the animal’s or +plant’s own good, but to man’s use or fancy. Some variations useful to +him have probably arisen suddenly, or by one step; many botanists, for +instance, believe that the fuller’s teazle, with its hooks, which +cannot be rivalled by any mechanical contrivance, is only a variety of +the wild Dipsacus; and this amount of change may have suddenly arisen +in a seedling. So it has probably been with the turnspit dog; and this +is known to have been the case with the ancon sheep. But when we +compare the dray-horse and race-horse, the dromedary and camel, the +various breeds of sheep fitted either for cultivated land or mountain +pasture, with the wool of one breed good for one purpose, and that of +another breed for another purpose; when we compare the many breeds of +dogs, each good for man in very different ways; when we compare the +game-cock, so pertinacious in battle, with other breeds so little +quarrelsome, with “everlasting layers” which never desire to sit, and +with the bantam so small and elegant; when we compare the host of +agricultural, culinary, orchard, and flower-garden races of plants, +most useful to man at different seasons and for different purposes, or +so beautiful in his eyes, we must, I think, look further than to mere +variability. We cannot suppose that all the breeds were suddenly +produced as perfect and as useful as we now see them; indeed, in +several cases, we know that this has not been their history. The key is +man’s power of accumulative selection: nature gives successive +variations; man adds them up in certain directions useful to him. In +this sense he may be said to make for himself useful breeds. + +The great power of this principle of selection is not hypothetical. It +is certain that several of our eminent breeders have, even within a +single lifetime, modified to +a large extent some breeds of cattle and sheep. In order fully to +realise what they have done, it is almost necessary to read several of +the many treatises devoted to this subject, and to inspect the animals. +Breeders habitually speak of an animal’s organisation as something +quite plastic, which they can model almost as they please. If I had +space I could quote numerous passages to this effect from highly +competent authorities. Youatt, who was probably better acquainted with +the works of agriculturalists than almost any other individual, and who +was himself a very good judge of an animal, speaks of the principle of +selection as “that which enables the agriculturist, not only to modify +the character of his flock, but to change it altogether. It is the +magician’s wand, by means of which he may summon into life whatever +form and mould he pleases.” Lord Somerville, speaking of what breeders +have done for sheep, says:—“It would seem as if they had chalked out +upon a wall a form perfect in itself, and then had given it existence.” +That most skilful breeder, Sir John Sebright, used to say, with respect +to pigeons, that “he would produce any given feather in three years, +but it would take him six years to obtain head and beak.” In Saxony the +importance of the principle of selection in regard to merino sheep is +so fully recognised, that men follow it as a trade: the sheep are +placed on a table and are studied, like a picture by a connoisseur; +this is done three times at intervals of months, and the sheep are each +time marked and classed, so that the very best may ultimately be +selected for breeding. + +What English breeders have actually effected is proved by the enormous +prices given for animals with a good pedigree; and these have now been +exported to almost every quarter of the world. The improvement is by no +means generally due to crossing different breeds; +all the best breeders are strongly opposed to this practice, except +sometimes amongst closely allied sub-breeds. And when a cross has been +made, the closest selection is far more indispensable even than in +ordinary cases. If selection consisted merely in separating some very +distinct variety, and breeding from it, the principle would be so +obvious as hardly to be worth notice; but its importance consists in +the great effect produced by the accumulation in one direction, during +successive generations, of differences absolutely inappreciable by an +uneducated eye—differences which I for one have vainly attempted to +appreciate. Not one man in a thousand has accuracy of eye and judgment +sufficient to become an eminent breeder. If gifted with these +qualities, and he studies his subject for years, and devotes his +lifetime to it with indomitable perseverance, he will succeed, and may +make great improvements; if he wants any of these qualities, he will +assuredly fail. Few would readily believe in the natural capacity and +years of practice requisite to become even a skilful pigeon-fancier. + +The same principles are followed by horticulturists; but the variations +are here often more abrupt. No one supposes that our choicest +productions have been produced by a single variation from the +aboriginal stock. We have proofs that this is not so in some cases, in +which exact records have been kept; thus, to give a very trifling +instance, the steadily-increasing size of the common gooseberry may be +quoted. We see an astonishing improvement in many florists’ flowers, +when the flowers of the present day are compared with drawings made +only twenty or thirty years ago. When a race of plants is once pretty +well established, the seed-raisers do not pick out the best plants, but +merely go over their seed-beds, and pull up the “rogues,” as they call +the plants that deviate from the proper standard. With animals this +kind of selection is, in fact, also followed; for hardly any one is so +careless as to allow his worst animals to breed. + +In regard to plants, there is another means of observing the +accumulated effects of selection—namely, by comparing the diversity of +flowers in the different varieties of the same species in the +flower-garden; the diversity of leaves, pods, or tubers, or whatever +part is valued, in the kitchen-garden, in comparison with the flowers +of the same varieties; and the diversity of fruit of the same species +in the orchard, in comparison with the leaves and flowers of the same +set of varieties. See how different the leaves of the cabbage are, and +how extremely alike the flowers; how unlike the flowers of the +heartsease are, and how alike the leaves; how much the fruit of the +different kinds of gooseberries differ in size, colour, shape, and +hairiness, and yet the flowers present very slight differences. It is +not that the varieties which differ largely in some one point do not +differ at all in other points; this is hardly ever, perhaps never, the +case. The laws of correlation of growth, the importance of which should +never be overlooked, will ensure some differences; but, as a general +rule, I cannot doubt that the continued selection of slight variations, +either in the leaves, the flowers, or the fruit, will produce races +differing from each other chiefly in these characters. + +It may be objected that the principle of selection has been reduced to +methodical practice for scarcely more than three-quarters of a century; +it has certainly been more attended to of late years, and many +treatises have been published on the subject; and the result, I may +add, has been, in a corresponding degree, rapid and important. But it +is very far from true that the principle is a modern discovery. I could +give several references to the full acknowledgment of the importance of +the principle in works of high antiquity. In rude and +barbarous periods of English history choice animals were often +imported, and laws were passed to prevent their exportation: the +destruction of horses under a certain size was ordered, and this may be +compared to the “roguing” of plants by nurserymen. The principle of +selection I find distinctly given in an ancient Chinese encyclopædia. +Explicit rules are laid down by some of the Roman classical writers. +From passages in Genesis, it is clear that the colour of domestic +animals was at that early period attended to. Savages now sometimes +cross their dogs with wild canine animals, to improve the breed, and +they formerly did so, as is attested by passages in Pliny. The savages +in South Africa match their draught cattle by colour, as do some of the +Esquimaux their teams of dogs. Livingstone shows how much good domestic +breeds are valued by the negroes of the interior of Africa who have not +associated with Europeans. Some of these facts do not show actual +selection, but they show that the breeding of domestic animals was +carefully attended to in ancient times, and is now attended to by the +lowest savages. It would, indeed, have been a strange fact, had +attention not been paid to breeding, for the inheritance of good and +bad qualities is so obvious. + +At the present time, eminent breeders try by methodical selection, with +a distinct object in view, to make a new strain or sub-breed, superior +to anything existing in the country. But, for our purpose, a kind of +Selection, which may be called Unconscious, and which results from +every one trying to possess and breed from the best individual animals, +is more important. Thus, a man who intends keeping pointers naturally +tries to get as good dogs as he can, and afterwards breeds from his own +best dogs, but he has no wish or expectation of permanently altering +the breed. Nevertheless I cannot +doubt that this process, continued during centuries, would improve and +modify any breed, in the same way as Bakewell, Collins, etc., by this +very same process, only carried on more methodically, did greatly +modify, even during their own lifetimes, the forms and qualities of +their cattle. Slow and insensible changes of this kind could never be +recognised unless actual measurements or careful drawings of the breeds +in question had been made long ago, which might serve for comparison. +In some cases, however, unchanged or but little changed individuals of +the same breed may be found in less civilised districts, where the +breed has been less improved. There is reason to believe that King +Charles’s spaniel has been unconsciously modified to a large extent +since the time of that monarch. Some highly competent authorities are +convinced that the setter is directly derived from the spaniel, and has +probably been slowly altered from it. It is known that the English +pointer has been greatly changed within the last century, and in this +case the change has, it is believed, been chiefly effected by crosses +with the fox-hound; but what concerns us is, that the change has been +effected unconsciously and gradually, and yet so effectually, that, +though the old Spanish pointer certainly came from Spain, Mr. Borrow +has not seen, as I am informed by him, any native dog in Spain like our +pointer. + +By a similar process of selection, and by careful training, the whole +body of English racehorses have come to surpass in fleetness and size +the parent Arab stock, so that the latter, by the regulations for the +Goodwood Races, are favoured in the weights they carry. Lord Spencer +and others have shown how the cattle of England have increased in +weight and in early maturity, compared with the stock formerly kept in +this country. By comparing the accounts given in old pigeon treatises +of carriers +and tumblers with these breeds as now existing in Britain, India, and +Persia, we can, I think, clearly trace the stages through which they +have insensibly passed, and come to differ so greatly from the +rock-pigeon. + +Youatt gives an excellent illustration of the effects of a course of +selection, which may be considered as unconsciously followed, in so far +that the breeders could never have expected or even have wished to have +produced the result which ensued—namely, the production of two distinct +strains. The two flocks of Leicester sheep kept by Mr. Buckley and Mr. +Burgess, as Mr. Youatt remarks, “have been purely bred from the +original stock of Mr. Bakewell for upwards of fifty years. There is not +a suspicion existing in the mind of any one at all acquainted with the +subject that the owner of either of them has deviated in any one +instance from the pure blood of Mr. Bakewell’s flock, and yet the +difference between the sheep possessed by these two gentlemen is so +great that they have the appearance of being quite different +varieties.” + +If there exist savages so barbarous as never to think of the inherited +character of the offspring of their domestic animals, yet any one +animal particularly useful to them, for any special purpose, would be +carefully preserved during famines and other accidents, to which +savages are so liable, and such choice animals would thus generally +leave more offspring than the inferior ones; so that in this case there +would be a kind of unconscious selection going on. We see the value set +on animals even by the barbarians of Tierra del Fuego, by their killing +and devouring their old women, in times of dearth, as of less value +than their dogs. + +In plants the same gradual process of improvement, through the +occasional preservation of the best individuals, whether or not +sufficiently distinct to be ranked +at their first appearance as distinct varieties, and whether or not two +or more species or races have become blended together by crossing, may +plainly be recognised in the increased size and beauty which we now see +in the varieties of the heartsease, rose, pelargonium, dahlia, and +other plants, when compared with the older varieties or with their +parent-stocks. No one would ever expect to get a first-rate heartsease +or dahlia from the seed of a wild plant. No one would expect to raise a +first-rate melting pear from the seed of a wild pear, though he might +succeed from a poor seedling growing wild, if it had come from a +garden-stock. The pear, though cultivated in classical times, appears, +from Pliny’s description, to have been a fruit of very inferior +quality. I have seen great surprise expressed in horticultural works at +the wonderful skill of gardeners, in having produced such splendid +results from such poor materials; but the art, I cannot doubt, has been +simple, and, as far as the final result is concerned, has been followed +almost unconsciously. It has consisted in always cultivating the best +known variety, sowing its seeds, and, when a slightly better variety +has chanced to appear, selecting it, and so onwards. But the gardeners +of the classical period, who cultivated the best pear they could +procure, never thought what splendid fruit we should eat; though we owe +our excellent fruit, in some small degree, to their having naturally +chosen and preserved the best varieties they could anywhere find. + +A large amount of change in our cultivated plants, thus slowly and +unconsciously accumulated, explains, as I believe, the well-known fact, +that in a vast number of cases we cannot recognise, and therefore do +not know, the wild parent-stocks of the plants which have been longest +cultivated in our flower and kitchen gardens. If it has taken centuries +or thousands of years to improve +or modify most of our plants up to their present standard of usefulness +to man, we can understand how it is that neither Australia, the Cape of +Good Hope, nor any other region inhabited by quite uncivilised man, has +afforded us a single plant worth culture. It is not that these +countries, so rich in species, do not by a strange chance possess the +aboriginal stocks of any useful plants, but that the native plants have +not been improved by continued selection up to a standard of perfection +comparable with that given to the plants in countries anciently +civilised. + +In regard to the domestic animals kept by uncivilised man, it should +not be overlooked that they almost always have to struggle for their +own food, at least during certain seasons. And in two countries very +differently circumstanced, individuals of the same species, having +slightly different constitutions or structure, would often succeed +better in the one country than in the other, and thus by a process of +“natural selection,” as will hereafter be more fully explained, two +sub-breeds might be formed. This, perhaps, partly explains what has +been remarked by some authors, namely, that the varieties kept by +savages have more of the character of species than the varieties kept +in civilised countries. + +On the view here given of the all-important part which selection by man +has played, it becomes at once obvious, how it is that our domestic +races show adaptation in their structure or in their habits to man’s +wants or fancies. We can, I think, further understand the frequently +abnormal character of our domestic races, and likewise their +differences being so great in external characters and relatively so +slight in internal parts or organs. Man can hardly select, or only with +much difficulty, any deviation of structure excepting such as is +externally visible; and indeed he rarely cares for what is internal. He +can never act by selection, excepting on variations +which are first given to him in some slight degree by nature. No man +would ever try to make a fantail, till he saw a pigeon with a tail +developed in some slight degree in an unusual manner, or a pouter till +he saw a pigeon with a crop of somewhat unusual size; and the more +abnormal or unusual any character was when it first appeared, the more +likely it would be to catch his attention. But to use such an +expression as trying to make a fantail, is, I have no doubt, in most +cases, utterly incorrect. The man who first selected a pigeon with a +slightly larger tail, never dreamed what the descendants of that pigeon +would become through long-continued, partly unconscious and partly +methodical selection. Perhaps the parent bird of all fantails had only +fourteen tail-feathers somewhat expanded, like the present Java +fantail, or like individuals of other and distinct breeds, in which as +many as seventeen tail-feathers have been counted. Perhaps the first +pouter-pigeon did not inflate its crop much more than the turbit now +does the upper part of its oesophagus,—a habit which is disregarded by +all fanciers, as it is not one of the points of the breed. + +Nor let it be thought that some great deviation of structure would be +necessary to catch the fancier’s eye: he perceives extremely small +differences, and it is in human nature to value any novelty, however +slight, in one’s own possession. Nor must the value which would +formerly be set on any slight differences in the individuals of the +same species, be judged of by the value which would now be set on them, +after several breeds have once fairly been established. Many slight +differences might, and indeed do now, arise amongst pigeons, which are +rejected as faults or deviations from the standard of perfection of +each breed. The common goose has not given rise to any marked +varieties; hence the Thoulouse and the common breed, which differ only +in colour, that +most fleeting of characters, have lately been exhibited as distinct at +our poultry-shows. + +I think these views further explain what has sometimes been +noticed—namely that we know nothing about the origin or history of any +of our domestic breeds. But, in fact, a breed, like a dialect of a +language, can hardly be said to have had a definite origin. A man +preserves and breeds from an individual with some slight deviation of +structure, or takes more care than usual in matching his best animals +and thus improves them, and the improved individuals slowly spread in +the immediate neighbourhood. But as yet they will hardly have a +distinct name, and from being only slightly valued, their history will +be disregarded. When further improved by the same slow and gradual +process, they will spread more widely, and will get recognised as +something distinct and valuable, and will then probably first receive a +provincial name. In semi-civilised countries, with little free +communication, the spreading and knowledge of any new sub-breed will be +a slow process. As soon as the points of value of the new sub-breed are +once fully acknowledged, the principle, as I have called it, of +unconscious selection will always tend,—perhaps more at one period than +at another, as the breed rises or falls in fashion,—perhaps more in one +district than in another, according to the state of civilisation of the +inhabitants—slowly to add to the characteristic features of the breed, +whatever they may be. But the chance will be infinitely small of any +record having been preserved of such slow, varying, and insensible +changes. + +I must now say a few words on the circumstances, favourable, or the +reverse, to man’s power of selection. A high degree of variability is +obviously favourable, as freely giving the materials for selection to +work on; not that mere individual differences are not amply +sufficient, with extreme care, to allow of the accumulation of a large +amount of modification in almost any desired direction. But as +variations manifestly useful or pleasing to man appear only +occasionally, the chance of their appearance will be much increased by +a large number of individuals being kept; and hence this comes to be of +the highest importance to success. On this principle Marshall has +remarked, with respect to the sheep of parts of Yorkshire, that “as +they generally belong to poor people, and are mostly _in small lots_, +they never can be improved.” On the other hand, nurserymen, from +raising large stocks of the same plants, are generally far more +successful than amateurs in getting new and valuable varieties. The +keeping of a large number of individuals of a species in any country +requires that the species should be placed under favourable conditions +of life, so as to breed freely in that country. When the individuals of +any species are scanty, all the individuals, whatever their quality may +be, will generally be allowed to breed, and this will effectually +prevent selection. But probably the most important point of all, is, +that the animal or plant should be so highly useful to man, or so much +valued by him, that the closest attention should be paid to even the +slightest deviation in the qualities or structure of each individual. +Unless such attention be paid nothing can be effected. I have seen it +gravely remarked, that it was most fortunate that the strawberry began +to vary just when gardeners began to attend closely to this plant. No +doubt the strawberry had always varied since it was cultivated, but the +slight varieties had been neglected. As soon, however, as gardeners +picked out individual plants with slightly larger, earlier, or better +fruit, and raised seedlings from them, and again picked out the best +seedlings and bred from them, then, there appeared (aided by some +crossing with distinct species) those many admirable varieties of the +strawberry which have been raised during the last thirty or forty +years. + +In the case of animals with separate sexes, facility in preventing +crosses is an important element of success in the formation of new +races,—at least, in a country which is already stocked with other +races. In this respect enclosure of the land plays a part. Wandering +savages or the inhabitants of open plains rarely possess more than one +breed of the same species. Pigeons can be mated for life, and this is a +great convenience to the fancier, for thus many races may be kept true, +though mingled in the same aviary; and this circumstance must have +largely favoured the improvement and formation of new breeds. Pigeons, +I may add, can be propagated in great numbers and at a very quick rate, +and inferior birds may be freely rejected, as when killed they serve +for food. On the other hand, cats, from their nocturnal rambling +habits, cannot be matched, and, although so much valued by women and +children, we hardly ever see a distinct breed kept up; such breeds as +we do sometimes see are almost always imported from some other country, +often from islands. Although I do not doubt that some domestic animals +vary less than others, yet the rarity or absence of distinct breeds of +the cat, the donkey, peacock, goose, etc., may be attributed in main +part to selection not having been brought into play: in cats, from the +difficulty in pairing them; in donkeys, from only a few being kept by +poor people, and little attention paid to their breeding; in peacocks, +from not being very easily reared and a large stock not kept; in geese, +from being valuable only for two purposes, food and feathers, and more +especially from no pleasure having been felt in the display of distinct +breeds. + +To sum up on the origin of our Domestic Races of animals and plants. I +believe that the conditions of life, from their action on the +reproductive system, are so far of the highest importance as causing +variability. I do not believe that variability is an inherent and +necessary contingency, under all circumstances, with all organic +beings, as some authors have thought. The effects of variability are +modified by various degrees of inheritance and of reversion. +Variability is governed by many unknown laws, more especially by that +of correlation of growth. Something may be attributed to the direct +action of the conditions of life. Something must be attributed to use +and disuse. The final result is thus rendered infinitely complex. In +some cases, I do not doubt that the intercrossing of species, +aboriginally distinct, has played an important part in the origin of +our domestic productions. When in any country several domestic breeds +have once been established, their occasional intercrossing, with the +aid of selection, has, no doubt, largely aided in the formation of new +sub-breeds; but the importance of the crossing of varieties has, I +believe, been greatly exaggerated, both in regard to animals and to +those plants which are propagated by seed. In plants which are +temporarily propagated by cuttings, buds, etc., the importance of the +crossing both of distinct species and of varieties is immense; for the +cultivator here quite disregards the extreme variability both of +hybrids and mongrels, and the frequent sterility of hybrids; but the +cases of plants not propagated by seed are of little importance to us, +for their endurance is only temporary. Over all these causes of Change +I am convinced that the accumulative action of Selection, whether +applied methodically and more quickly, or unconsciously and more +slowly, but more efficiently, is by far the predominant Power. + +Variability. Individual differences. Doubtful species. Wide ranging, +much diffused, and common species vary most. Species of the larger +genera in any country vary more than the species of the smaller genera. +Many of the species of the larger genera resemble varieties in being +very closely, but unequally, related to each other, and in having +restricted ranges. + +Before applying the principles arrived at in the last chapter to +organic beings in a state of nature, we must briefly discuss whether +these latter are subject to any variation. To treat this subject at all +properly, a long catalogue of dry facts should be given; but these I +shall reserve for my future work. Nor shall I here discuss the various +definitions which have been given of the term species. No one +definition has as yet satisfied all naturalists; yet every naturalist +knows vaguely what he means when he speaks of a species. Generally the +term includes the unknown element of a distinct act of creation. The +term “variety” is almost equally difficult to define; but here +community of descent is almost universally implied, though it can +rarely be proved. We have also what are called monstrosities; but they +graduate into varieties. By a monstrosity I presume is meant some +considerable deviation of structure in one part, either injurious to or +not useful to the species, and not generally propagated. Some authors +use the term “variation” in a technical sense, as implying a +modification directly due to the physical conditions of life; and +“variations” in this sense are supposed not to be inherited: but who +can say that the dwarfed condition of shells in the brackish waters of +the Baltic, or dwarfed plants on Alpine summits, or the thicker fur of +an animal from far northwards, would not in some cases be inherited for +at least some few generations? and in this case I presume that the form +would be called a variety. + +Again, we have many slight differences which may be called individual +differences, such as are known frequently to appear in the offspring +from the same parents, or which may be presumed to have thus arisen, +from being frequently observed in the individuals of the same species +inhabiting the same confined locality. No one supposes that all the +individuals of the same species are cast in the very same mould. These +individual differences are highly important for us, as they afford +materials for natural selection to accumulate, in the same manner as +man can accumulate in any given direction individual differences in his +domesticated productions. These individual differences generally affect +what naturalists consider unimportant parts; but I could show by a long +catalogue of facts, that parts which must be called important, whether +viewed under a physiological or classificatory point of view, sometimes +vary in the individuals of the same species. I am convinced that the +most experienced naturalist would be surprised at the number of the +cases of variability, even in important parts of structure, which he +could collect on good authority, as I have collected, during a course +of years. It should be remembered that systematists are far from +pleased at finding variability in important characters, and that there +are not many men who will laboriously examine internal and important +organs, and compare them in many specimens of the same species. I +should never have expected that the branching of the main nerves close +to the great central ganglion of an insect would have been variable in +the same species; I should have expected that changes of this nature +could have been effected only by slow degrees: yet quite recently Mr. +Lubbock has shown a degree of variability in these main nerves in +Coccus, which may almost be compared to the irregular branching of the +stem of a tree. This philosophical naturalist, I may add, has also +quite recently shown that the muscles in the larvæ of certain insects +are very far from uniform. Authors sometimes argue in a circle when +they state that important organs never vary; for these same authors +practically rank that character as important (as some few naturalists +have honestly confessed) which does not vary; and, under this point of +view, no instance of an important part varying will ever be found: but +under any other point of view many instances assuredly can be given. + +There is one point connected with individual differences, which seems +to me extremely perplexing: I refer to those genera which have +sometimes been called “protean” or “polymorphic,” in which the species +present an inordinate amount of variation; and hardly two naturalists +can agree which forms to rank as species and which as varieties. We may +instance Rubus, Rosa, and Hieracium amongst plants, several genera of +insects, and several genera of Brachiopod shells. In most polymorphic +genera some of the species have fixed and definite characters. Genera +which are polymorphic in one country seem to be, with some few +exceptions, polymorphic in other countries, and likewise, judging from +Brachiopod shells, at former periods of time. These facts seem to be +very perplexing, for they seem to show that this kind of variability is +independent of the conditions of life. I am inclined to suspect that we +see in these polymorphic genera variations in points of structure which +are of no service or disservice to the species, and which consequently +have not been seized on and rendered definite by natural selection, as +hereafter will be explained. + +Those forms which possess in some considerable degree the character of +species, but which are so closely similar to some other forms, or are +so closely linked to them by intermediate gradations, that naturalists +do not like to rank them as distinct species, are in several respects +the most important for us. We have every reason to believe that many of +these doubtful and closely-allied forms have permanently retained their +characters in their own country for a long time; for as long, as far as +we know, as have good and true species. Practically, when a naturalist +can unite two forms together by others having intermediate characters, +he treats the one as a variety of the other, ranking the most common, +but sometimes the one first described, as the species, and the other as +the variety. But cases of great difficulty, which I will not here +enumerate, sometimes occur in deciding whether or not to rank one form +as a variety of another, even when they are closely connected by +intermediate links; nor will the commonly-assumed hybrid nature of the +intermediate links always remove the difficulty. In very many cases, +however, one form is ranked as a variety of another, not because the +intermediate links have actually been found, but because analogy leads +the observer to suppose either that they do now somewhere exist, or may +formerly have existed; and here a wide door for the entry of doubt and +conjecture is opened. + +Hence, in determining whether a form should be ranked as a species or a +variety, the opinion of naturalists having sound judgment and wide +experience seems the only guide to follow. We must, however, in many +cases, decide by a majority of naturalists, for few well-marked and +well-known varieties can be named which have not been ranked as species +by at least some competent judges. + +That varieties of this doubtful nature are far from uncommon cannot be +disputed. Compare the several floras of Great Britain, of France or of +the United States, drawn up by different botanists, and see what a +surprising number of forms have been ranked by one botanist as good +species, and by another as mere varieties. Mr. H. C. Watson, to whom I +lie under deep obligation for assistance of all kinds, has marked for +me 182 British plants, which are generally considered as varieties, but +which have all been ranked by botanists as species; and in making this +list he has omitted many trifling varieties, but which nevertheless +have been ranked by some botanists as species, and he has entirely +omitted several highly polymorphic genera. Under genera, including the +most polymorphic forms, Mr. Babington gives 251 species, whereas Mr. +Bentham gives only 112,—a difference of 139 doubtful forms! Amongst +animals which unite for each birth, and which are highly locomotive, +doubtful forms, ranked by one zoologist as a species and by another as +a variety, can rarely be found within the same country, but are common +in separated areas. How many of those birds and insects in North +America and Europe, which differ very slightly from each other, have +been ranked by one eminent naturalist as undoubted species, and by +another as varieties, or, as they are often called, as geographical +races! Many years ago, when comparing, and seeing others compare, the +birds from the separate islands of the Galapagos Archipelago, both one +with another, and with those from the American mainland, I was much +struck how entirely vague and arbitrary is the distinction between +species and varieties. On the islets of the little Madeira group there +are many insects which are characterized as varieties in Mr. +Wollaston’s admirable work, but which it cannot +be doubted would be ranked as distinct species by many entomologists. +Even Ireland has a few animals, now generally regarded as varieties, +but which have been ranked as species by some zoologists. Several most +experienced ornithologists consider our British red grouse as only a +strongly-marked race of a Norwegian species, whereas the greater number +rank it as an undoubted species peculiar to Great Britain. A wide +distance between the homes of two doubtful forms leads many naturalists +to rank both as distinct species; but what distance, it has been well +asked, will suffice? if that between America and Europe is ample, will +that between the Continent and the Azores, or Madeira, or the Canaries, +or Ireland, be sufficient? It must be admitted that many forms, +considered by highly-competent judges as varieties, have so perfectly +the character of species that they are ranked by other highly-competent +judges as good and true species. But to discuss whether they are +rightly called species or varieties, before any definition of these +terms has been generally accepted, is vainly to beat the air. + +Many of the cases of strongly-marked varieties or doubtful species well +deserve consideration; for several interesting lines of argument, from +geographical distribution, analogical variation, hybridism, etc., have +been brought to bear on the attempt to determine their rank. I will +here give only a single instance,—the well-known one of the primrose +and cowslip, or Primula veris and elatior. These plants differ +considerably in appearance; they have a different flavour and emit a +different odour; they flower at slightly different periods; they grow +in somewhat different stations; they ascend mountains to different +heights; they have different geographical ranges; and lastly, according +to very numerous experiments made during several years by +that most careful observer Gärtner, they can be crossed only with much +difficulty. We could hardly wish for better evidence of the two forms +being specifically distinct. On the other hand, they are united by many +intermediate links, and it is very doubtful whether these links are +hybrids; and there is, as it seems to me, an overwhelming amount of +experimental evidence, showing that they descend from common parents, +and consequently must be ranked as varieties. + +Close investigation, in most cases, will bring naturalists to an +agreement how to rank doubtful forms. Yet it must be confessed, that it +is in the best-known countries that we find the greatest number of +forms of doubtful value. I have been struck with the fact, that if any +animal or plant in a state of nature be highly useful to man, or from +any cause closely attract his attention, varieties of it will almost +universally be found recorded. These varieties, moreover, will be often +ranked by some authors as species. Look at the common oak, how closely +it has been studied; yet a German author makes more than a dozen +species out of forms, which are very generally considered as varieties; +and in this country the highest botanical authorities and practical men +can be quoted to show that the sessile and pedunculated oaks are either +good and distinct species or mere varieties. + +When a young naturalist commences the study of a group of organisms +quite unknown to him, he is at first much perplexed to determine what +differences to consider as specific, and what as varieties; for he +knows nothing of the amount and kind of variation to which the group is +subject; and this shows, at least, how very generally there is some +variation. But if he confine his attention to one class within one +country, he will soon make up his mind how to rank most of the doubtful +forms. His +general tendency will be to make many species, for he will become +impressed, just like the pigeon or poultry-fancier before alluded to, +with the amount of difference in the forms which he is continually +studying; and he has little general knowledge of analogical variation +in other groups and in other countries, by which to correct his first +impressions. As he extends the range of his observations, he will meet +with more cases of difficulty; for he will encounter a greater number +of closely-allied forms. But if his observations be widely extended, he +will in the end generally be enabled to make up his own mind which to +call varieties and which species; but he will succeed in this at the +expense of admitting much variation,—and the truth of this admission +will often be disputed by other naturalists. When, moreover, he comes +to study allied forms brought from countries not now continuous, in +which case he can hardly hope to find the intermediate links between +his doubtful forms, he will have to trust almost entirely to analogy, +and his difficulties will rise to a climax. + +Certainly no clear line of demarcation has as yet been drawn between +species and sub-species—that is, the forms which in the opinion of some +naturalists come very near to, but do not quite arrive at the rank of +species; or, again, between sub-species and well-marked varieties, or +between lesser varieties and individual differences. These differences +blend into each other in an insensible series; and a series impresses +the mind with the idea of an actual passage. + +Hence I look at individual differences, though of small interest to the +systematist, as of high importance for us, as being the first step +towards such slight varieties as are barely thought worth recording in +works on natural history. And I look at varieties which are in any +degree more distinct and permanent, as steps leading to more +strongly marked and more permanent varieties; and at these latter, as +leading to sub-species, and to species. The passage from one stage of +difference to another and higher stage may be, in some cases, due +merely to the long-continued action of different physical conditions in +two different regions; but I have not much faith in this view; and I +attribute the passage of a variety, from a state in which it differs +very slightly from its parent to one in which it differs more, to the +action of natural selection in accumulating (as will hereafter be more +fully explained) differences of structure in certain definite +directions. Hence I believe a well-marked variety may be justly called +an incipient species; but whether this belief be justifiable must be +judged of by the general weight of the several facts and views given +throughout this work. + +It need not be supposed that all varieties or incipient species +necessarily attain the rank of species. They may whilst in this +incipient state become extinct, or they may endure as varieties for +very long periods, as has been shown to be the case by Mr. Wollaston +with the varieties of certain fossil land-shells in Madeira. If a +variety were to flourish so as to exceed in numbers the parent species, +it would then rank as the species, and the species as the variety; or +it might come to supplant and exterminate the parent species; or both +might co-exist, and both rank as independent species. But we shall +hereafter have to return to this subject. + +From these remarks it will be seen that I look at the term species, as +one arbitrarily given for the sake of convenience to a set of +individuals closely resembling each other, and that it does not +essentially differ from the term variety, which is given to less +distinct and more fluctuating forms. The term variety, again, in +comparison with mere individual differences, is also applied +arbitrarily, and for mere convenience sake. + +Guided by theoretical considerations, I thought that some interesting +results might be obtained in regard to the nature and relations of the +species which vary most, by tabulating all the varieties in several +well-worked floras. At first this seemed a simple task; but Mr. H. C. +Watson, to whom I am much indebted for valuable advice and assistance +on this subject, soon convinced me that there were many difficulties, +as did subsequently Dr. Hooker, even in stronger terms. I shall reserve +for my future work the discussion of these difficulties, and the tables +themselves of the proportional numbers of the varying species. Dr. +Hooker permits me to add, that after having carefully read my +manuscript, and examined the tables, he thinks that the following +statements are fairly well established. The whole subject, however, +treated as it necessarily here is with much brevity, is rather +perplexing, and allusions cannot be avoided to the “struggle for +existence,” “divergence of character,” and other questions, hereafter +to be discussed. + +Alph. De Candolle and others have shown that plants which have very +wide ranges generally present varieties; and this might have been +expected, as they become exposed to diverse physical conditions, and as +they come into competition (which, as we shall hereafter see, is a far +more important circumstance) with different sets of organic beings. But +my tables further show that, in any limited country, the species which +are most common, that is abound most in individuals, and the species +which are most widely diffused within their own country (and this is a +different consideration from wide range, and to a certain extent from +commonness), often give rise to varieties sufficiently well-marked to +have been recorded in botanical works. Hence it is the most +flourishing, or, as they may be called, the dominant species,—those +which range widely over the world, are the most diffused in their own +country, and are the most numerous in individuals,—which oftenest +produce well-marked varieties, or, as I consider them, incipient +species. And this, perhaps, might have been anticipated; for, as +varieties, in order to become in any degree permanent, necessarily have +to struggle with the other inhabitants of the country, the species +which are already dominant will be the most likely to yield offspring +which, though in some slight degree modified, will still inherit those +advantages that enabled their parents to become dominant over their +compatriots. + +If the plants inhabiting a country and described in any Flora be +divided into two equal masses, all those in the larger genera being +placed on one side, and all those in the smaller genera on the other +side, a somewhat larger number of the very common and much diffused or +dominant species will be found on the side of the larger genera. This, +again, might have been anticipated; for the mere fact of many species +of the same genus inhabiting any country, shows that there is something +in the organic or inorganic conditions of that country favourable to +the genus; and, consequently, we might have expected to have found in +the larger genera, or those including many species, a large +proportional number of dominant species. But so many causes tend to +obscure this result, that I am surprised that my tables show even a +small majority on the side of the larger genera. I will here allude to +only two causes of obscurity. Fresh-water and salt-loving plants have +generally very wide ranges and are much diffused, but this seems to be +connected with the nature of the stations inhabited by them, and has +little or no relation to the size of the genera to which the species +belong. Again, plants low in the scale of organisation are +generally much more widely diffused than plants higher in the scale; +and here again there is no close relation to the size of the genera. +The cause of lowly-organised plants ranging widely will be discussed in +our chapter on geographical distribution. + +From looking at species as only strongly-marked and well-defined +varieties, I was led to anticipate that the species of the larger +genera in each country would oftener present varieties, than the +species of the smaller genera; for wherever many closely related +species (_i.e._ species of the same genus) have been formed, many +varieties or incipient species ought, as a general rule, to be now +forming. Where many large trees grow, we expect to find saplings. Where +many species of a genus have been formed through variation, +circumstances have been favourable for variation; and hence we might +expect that the circumstances would generally be still favourable to +variation. On the other hand, if we look at each species as a special +act of creation, there is no apparent reason why more varieties should +occur in a group having many species, than in one having few. + +To test the truth of this anticipation I have arranged the plants of +twelve countries, and the coleopterous insects of two districts, into +two nearly equal masses, the species of the larger genera on one side, +and those of the smaller genera on the other side, and it has +invariably proved to be the case that a larger proportion of the +species on the side of the larger genera present varieties, than on the +side of the smaller genera. Moreover, the species of the large genera +which present any varieties, invariably present a larger average number +of varieties than do the species of the small genera. Both these +results follow when another division is made, and when all the smallest +genera, with from only one to four species, are absolutely excluded +from the tables. These +facts are of plain signification on the view that species are only +strongly marked and permanent varieties; for wherever many species of +the same genus have been formed, or where, if we may use the +expression, the manufactory of species has been active, we ought +generally to find the manufactory still in action, more especially as +we have every reason to believe the process of manufacturing new +species to be a slow one. And this certainly is the case, if varieties +be looked at as incipient species; for my tables clearly show as a +general rule that, wherever many species of a genus have been formed, +the species of that genus present a number of varieties, that is of +incipient species, beyond the average. It is not that all large genera +are now varying much, and are thus increasing in the number of their +species, or that no small genera are now varying and increasing; for if +this had been so, it would have been fatal to my theory; inasmuch as +geology plainly tells us that small genera have in the lapse of time +often increased greatly in size; and that large genera have often come +to their maxima, declined, and disappeared. All that we want to show +is, that where many species of a genus have been formed, on an average +many are still forming; and this holds good. + +There are other relations between the species of large genera and their +recorded varieties which deserve notice. We have seen that there is no +infallible criterion by which to distinguish species and well-marked +varieties; and in those cases in which intermediate links have not been +found between doubtful forms, naturalists are compelled to come to a +determination by the amount of difference between them, judging by +analogy whether or not the amount suffices to raise one or both to the +rank of species. Hence the amount of difference is one very important +criterion in settling whether two forms should +be ranked as species or varieties. Now Fries has remarked in regard to +plants, and Westwood in regard to insects, that in large genera the +amount of difference between the species is often exceedingly small. I +have endeavoured to test this numerically by averages, and, as far as +my imperfect results go, they always confirm the view. I have also +consulted some sagacious and most experienced observers, and, after +deliberation, they concur in this view. In this respect, therefore, the +species of the larger genera resemble varieties, more than do the +species of the smaller genera. Or the case may be put in another way, +and it may be said, that in the larger genera, in which a number of +varieties or incipient species greater than the average are now +manufacturing, many of the species already manufactured still to a +certain extent resemble varieties, for they differ from each other by a +less than usual amount of difference. + +Moreover, the species of the large genera are related to each other, in +the same manner as the varieties of any one species are related to each +other. No naturalist pretends that all the species of a genus are +equally distinct from each other; they may generally be divided into +sub-genera, or sections, or lesser groups. As Fries has well remarked, +little groups of species are generally clustered like satellites around +certain other species. And what are varieties but groups of forms, +unequally related to each other, and clustered round certain forms—that +is, round their parent-species? Undoubtedly there is one most important +point of difference between varieties and species; namely, that the +amount of difference between varieties, when compared with each other +or with their parent-species, is much less than that between the +species of the same genus. But when we come to discuss the principle, +as I call it, of Divergence of Character, +we shall see how this may be explained, and how the lesser differences +between varieties will tend to increase into the greater differences +between species. + +There is one other point which seems to me worth notice. Varieties +generally have much restricted ranges: this statement is indeed +scarcely more than a truism, for if a variety were found to have a +wider range than that of its supposed parent-species, their +denominations ought to be reversed. But there is also reason to +believe, that those species which are very closely allied to other +species, and in so far resemble varieties, often have much restricted +ranges. For instance, Mr. H. C. Watson has marked for me in the +well-sifted London Catalogue of plants (4th edition) 63 plants which +are therein ranked as species, but which he considers as so closely +allied to other species as to be of doubtful value: these 63 reputed +species range on an average over 6.9 of the provinces into which Mr. +Watson has divided Great Britain. Now, in this same catalogue, 53 +acknowledged varieties are recorded, and these range over 7.7 +provinces; whereas, the species to which these varieties belong range +over 14.3 provinces. So that the acknowledged varieties have very +nearly the same restricted average range, as have those very closely +allied forms, marked for me by Mr. Watson as doubtful species, but +which are almost universally ranked by British botanists as good and +true species. + +Finally, then, varieties have the same general characters as species, +for they cannot be distinguished from species,—except, firstly, by the +discovery of intermediate linking forms, and the occurrence of such +links cannot affect the actual characters of the forms which they +connect; and except, secondly, by a certain amount of +difference, for two forms, if differing very little, are generally +ranked as varieties, notwithstanding that intermediate linking forms +have not been discovered; but the amount of difference considered +necessary to give to two forms the rank of species is quite indefinite. +In genera having more than the average number of species in any +country, the species of these genera have more than the average number +of varieties. In large genera the species are apt to be closely, but +unequally, allied together, forming little clusters round certain +species. Species very closely allied to other species apparently have +restricted ranges. In all these several respects the species of large +genera present a strong analogy with varieties. And we can clearly +understand these analogies, if species have once existed as varieties, +and have thus originated: whereas, these analogies are utterly +inexplicable if each species has been independently created. + +We have, also, seen that it is the most flourishing and dominant +species of the larger genera which on an average vary most; and +varieties, as we shall hereafter see, tend to become converted into new +and distinct species. The larger genera thus tend to become larger; and +throughout nature the forms of life which are now dominant tend to +become still more dominant by leaving many modified and dominant +descendants. But by steps hereafter to be explained, the larger genera +also tend to break up into smaller genera. And thus, the forms of life +throughout the universe become divided into groups subordinate to +groups. + +Bears on natural selection. The term used in a wide sense. Geometrical +powers of increase. Rapid increase of naturalised animals and plants. +Nature of the checks to increase. Competition universal. Effects of +climate. Protection from the number of individuals. Complex relations +of all animals and plants throughout nature. Struggle for life most +severe between individuals and varieties of the same species; often +severe between species of the same genus. The relation of organism to +organism the most important of all relations. + +Before entering on the subject of this chapter, I must make a few +preliminary remarks, to show how the struggle for existence bears on +Natural Selection. It has been seen in the last chapter that amongst +organic beings in a state of nature there is some individual +variability; indeed I am not aware that this has ever been disputed. It +is immaterial for us whether a multitude of doubtful forms be called +species or sub-species or varieties; what rank, for instance, the two +or three hundred doubtful forms of British plants are entitled to hold, +if the existence of any well-marked varieties be admitted. But the mere +existence of individual variability and of some few well-marked +varieties, though necessary as the foundation for the work, helps us +but little in understanding how species arise in nature. How have all +those exquisite adaptations of one part of the organisation to another +part, and to the conditions of life, and of one distinct organic being +to another being, been perfected? We see these beautiful co-adaptations +most plainly in the woodpecker and missletoe; and only a little less +plainly in the humblest parasite which clings +to the hairs of a quadruped or feathers of a bird; in the structure of +the beetle which dives through the water; in the plumed seed which is +wafted by the gentlest breeze; in short, we see beautiful adaptations +everywhere and in every part of the organic world. + +Again, it may be asked, how is it that varieties, which I have called +incipient species, become ultimately converted into good and distinct +species, which in most cases obviously differ from each other far more +than do the varieties of the same species? How do those groups of +species, which constitute what are called distinct genera, and which +differ from each other more than do the species of the same genus, +arise? All these results, as we shall more fully see in the next +chapter, follow inevitably from the struggle for life. Owing to this +struggle for life, any variation, however slight and from whatever +cause proceeding, if it be in any degree profitable to an individual of +any species, in its infinitely complex relations to other organic +beings and to external nature, will tend to the preservation of that +individual, and will generally be inherited by its offspring. The +offspring, also, will thus have a better chance of surviving, for, of +the many individuals of any species which are periodically born, but a +small number can survive. I have called this principle, by which each +slight variation, if useful, is preserved, by the term of Natural +Selection, in order to mark its relation to man’s power of selection. +We have seen that man by selection can certainly produce great results, +and can adapt organic beings to his own uses, through the accumulation +of slight but useful variations, given to him by the hand of Nature. +But Natural Selection, as we shall hereafter see, is a power +incessantly ready for action, and is as immeasurably superior to man’s +feeble efforts, as the works of Nature are to those of Art. + +We will now discuss in a little more detail the struggle for existence. +In my future work this subject shall be treated, as it well deserves, +at much greater length. The elder De Candolle and Lyell have largely +and philosophically shown that all organic beings are exposed to severe +competition. In regard to plants, no one has treated this subject with +more spirit and ability than W. Herbert, Dean of Manchester, evidently +the result of his great horticultural knowledge. Nothing is easier than +to admit in words the truth of the universal struggle for life, or more +difficult—at least I have found it so—than constantly to bear this +conclusion in mind. Yet unless it be thoroughly engrained in the mind, +I am convinced that the whole economy of nature, with every fact on +distribution, rarity, abundance, extinction, and variation, will be +dimly seen or quite misunderstood. We behold the face of nature bright +with gladness, we often see superabundance of food; we do not see, or +we forget, that the birds which are idly singing round us mostly live +on insects or seeds, and are thus constantly destroying life; or we +forget how largely these songsters, or their eggs, or their nestlings, +are destroyed by birds and beasts of prey; we do not always bear in +mind, that though food may be now superabundant, it is not so at all +seasons of each recurring year. + +I should premise that I use the term Struggle for Existence in a large +and metaphorical sense, including dependence of one being on another, +and including (which is more important) not only the life of the +individual, but success in leaving progeny. Two canine animals in a +time of dearth, may be truly said to struggle with each other which +shall get food and live. But a plant on the edge of a desert is said to +struggle for life against the drought, though more properly it should +be said to be dependent on the moisture. A +plant which annually produces a thousand seeds, of which on an average +only one comes to maturity, may be more truly said to struggle with the +plants of the same and other kinds which already clothe the ground. The +missletoe is dependent on the apple and a few other trees, but can only +in a far-fetched sense be said to struggle with these trees, for if too +many of these parasites grow on the same tree, it will languish and +die. But several seedling missletoes, growing close together on the +same branch, may more truly be said to struggle with each other. As the +missletoe is disseminated by birds, its existence depends on birds; and +it may metaphorically be said to struggle with other fruit-bearing +plants, in order to tempt birds to devour and thus disseminate its +seeds rather than those of other plants. In these several senses, which +pass into each other, I use for convenience sake the general term of +struggle for existence. + +A struggle for existence inevitably follows from the high rate at which +all organic beings tend to increase. Every being, which during its +natural lifetime produces several eggs or seeds, must suffer +destruction during some period of its life, and during some season or +occasional year, otherwise, on the principle of geometrical increase, +its numbers would quickly become so inordinately great that no country +could support the product. Hence, as more individuals are produced than +can possibly survive, there must in every case be a struggle for +existence, either one individual with another of the same species, or +with the individuals of distinct species, or with the physical +conditions of life. It is the doctrine of Malthus applied with manifold +force to the whole animal and vegetable kingdoms; for in this case +there can be no artificial increase of food, and no prudential +restraint from marriage. Although some species may be +now increasing, more or less rapidly, in numbers, all cannot do so, for +the world would not hold them. + +There is no exception to the rule that every organic being naturally +increases at so high a rate, that if not destroyed, the earth would +soon be covered by the progeny of a single pair. Even slow-breeding man +has doubled in twenty-five years, and at this rate, in a few thousand +years, there would literally not be standing room for his progeny. +Linnæus has calculated that if an annual plant produced only two +seeds—and there is no plant so unproductive as this—and their seedlings +next year produced two, and so on, then in twenty years there would be +a million plants. The elephant is reckoned to be the slowest breeder of +all known animals, and I have taken some pains to estimate its probable +minimum rate of natural increase: it will be under the mark to assume +that it breeds when thirty years old, and goes on breeding till ninety +years old, bringing forth three pair of young in this interval; if this +be so, at the end of the fifth century there would be alive fifteen +million elephants, descended from the first pair. + +But we have better evidence on this subject than mere theoretical +calculations, namely, the numerous recorded cases of the astonishingly +rapid increase of various animals in a state of nature, when +circumstances have been favourable to them during two or three +following seasons. Still more striking is the evidence from our +domestic animals of many kinds which have run wild in several parts of +the world: if the statements of the rate of increase of slow-breeding +cattle and horses in South America, and latterly in Australia, had not +been well authenticated, they would have been quite incredible. So it +is with plants: cases could be given of introduced plants which have +become common throughout whole islands in a period of less than ten +years. Several +of the plants now most numerous over the wide plains of La Plata, +clothing square leagues of surface almost to the exclusion of all other +plants, have been introduced from Europe; and there are plants which +now range in India, as I hear from Dr. Falconer, from Cape Comorin to +the Himalaya, which have been imported from America since its +discovery. In such cases, and endless instances could be given, no one +supposes that the fertility of these animals or plants has been +suddenly and temporarily increased in any sensible degree. The obvious +explanation is that the conditions of life have been very favourable, +and that there has consequently been less destruction of the old and +young, and that nearly all the young have been enabled to breed. In +such cases the geometrical ratio of increase, the result of which never +fails to be surprising, simply explains the extraordinarily rapid +increase and wide diffusion of naturalised productions in their new +homes. + +In a state of nature almost every plant produces seed, and amongst +animals there are very few which do not annually pair. Hence we may +confidently assert, that all plants and animals are tending to increase +at a geometrical ratio, that all would most rapidly stock every station +in which they could any how exist, and that the geometrical tendency to +increase must be checked by destruction at some period of life. Our +familiarity with the larger domestic animals tends, I think, to mislead +us: we see no great destruction falling on them, and we forget that +thousands are annually slaughtered for food, and that in a state of +nature an equal number would have somehow to be disposed of. + +The only difference between organisms which annually produce eggs or +seeds by the thousand, and those which produce extremely few, is, that +the slow-breeders would require a few more years to people, under +favourable +conditions, a whole district, let it be ever so large. The condor lays +a couple of eggs and the ostrich a score, and yet in the same country +the condor may be the more numerous of the two: the Fulmar petrel lays +but one egg, yet it is believed to be the most numerous bird in the +world. One fly deposits hundreds of eggs, and another, like the +hippobosca, a single one; but this difference does not determine how +many individuals of the two species can be supported in a district. A +large number of eggs is of some importance to those species, which +depend on a rapidly fluctuating amount of food, for it allows them +rapidly to increase in number. But the real importance of a large +number of eggs or seeds is to make up for much destruction at some +period of life; and this period in the great majority of cases is an +early one. If an animal can in any way protect its own eggs or young, a +small number may be produced, and yet the average stock be fully kept +up; but if many eggs or young are destroyed, many must be produced, or +the species will become extinct. It would suffice to keep up the full +number of a tree, which lived on an average for a thousand years, if a +single seed were produced once in a thousand years, supposing that this +seed were never destroyed, and could be ensured to germinate in a +fitting place. So that in all cases, the average number of any animal +or plant depends only indirectly on the number of its eggs or seeds. + +In looking at Nature, it is most necessary to keep the foregoing +considerations always in mind—never to forget that every single organic +being around us may be said to be striving to the utmost to increase in +numbers; that each lives by a struggle at some period of its life; that +heavy destruction inevitably falls either on the young or old, during +each generation or at recurrent intervals. Lighten any check, mitigate +the +destruction ever so little, and the number of the species will almost +instantaneously increase to any amount. The face of Nature may be +compared to a yielding surface, with ten thousand sharp wedges packed +close together and driven inwards by incessant blows, sometimes one +wedge being struck, and then another with greater force. + +What checks the natural tendency of each species to increase in number +is most obscure. Look at the most vigorous species; by as much as it +swarms in numbers, by so much will its tendency to increase be still +further increased. We know not exactly what the checks are in even one +single instance. Nor will this surprise any one who reflects how +ignorant we are on this head, even in regard to mankind, so +incomparably better known than any other animal. This subject has been +ably treated by several authors, and I shall, in my future work, +discuss some of the checks at considerable length, more especially in +regard to the feral animals of South America. Here I will make only a +few remarks, just to recall to the reader’s mind some of the chief +points. Eggs or very young animals seem generally to suffer most, but +this is not invariably the case. With plants there is a vast +destruction of seeds, but, from some observations which I have made, I +believe that it is the seedlings which suffer most from germinating in +ground already thickly stocked with other plants. Seedlings, also, are +destroyed in vast numbers by various enemies; for instance, on a piece +of ground three feet long and two wide, dug and cleared, and where +there could be no choking from other plants, I marked all the seedlings +of our native weeds as they came up, and out of the 357 no less than +295 were destroyed, chiefly by slugs and insects. If turf which has +long been mown, and the case would be the same with turf closely +browsed by quadrupeds, be let to grow, +the more vigorous plants gradually kill the less vigorous, though fully +grown, plants: thus out of twenty species growing on a little plot of +turf (three feet by four) nine species perished from the other species +being allowed to grow up freely. + +The amount of food for each species of course gives the extreme limit +to which each can increase; but very frequently it is not the obtaining +food, but the serving as prey to other animals, which determines the +average numbers of a species. Thus, there seems to be little doubt that +the stock of partridges, grouse, and hares on any large estate depends +chiefly on the destruction of vermin. If not one head of game were shot +during the next twenty years in England, and, at the same time, if no +vermin were destroyed, there would, in all probability, be less game +than at present, although hundreds of thousands of game animals are now +annually killed. On the other hand, in some cases, as with the elephant +and rhinoceros, none are destroyed by beasts of prey: even the tiger in +India most rarely dares to attack a young elephant protected by its +dam. + +Climate plays an important part in determining the average numbers of a +species, and periodical seasons of extreme cold or drought, I believe +to be the most effective of all checks. I estimated that the winter of +1854-55 destroyed four-fifths of the birds in my own grounds; and this +is a tremendous destruction, when we remember that ten per cent. is an +extraordinarily severe mortality from epidemics with man. The action of +climate seems at first sight to be quite independent of the struggle +for existence; but in so far as climate chiefly acts in reducing food, +it brings on the most severe struggle between the individuals, whether +of the same or of distinct species, which subsist on the same kind of +food. Even when climate, for instance extreme +cold, acts directly, it will be the least vigorous, or those which have +got least food through the advancing winter, which will suffer most. +When we travel from south to north, or from a damp region to a dry, we +invariably see some species gradually getting rarer and rarer, and +finally disappearing; and the change of climate being conspicuous, we +are tempted to attribute the whole effect to its direct action. But +this is a very false view: we forget that each species, even where it +most abounds, is constantly suffering enormous destruction at some +period of its life, from enemies or from competitors for the same place +and food; and if these enemies or competitors be in the least degree +favoured by any slight change of climate, they will increase in +numbers, and, as each area is already fully stocked with inhabitants, +the other species will decrease. When we travel southward and see a +species decreasing in numbers, we may feel sure that the cause lies +quite as much in other species being favoured, as in this one being +hurt. So it is when we travel northward, but in a somewhat lesser +degree, for the number of species of all kinds, and therefore of +competitors, decreases northwards; hence in going northward, or in +ascending a mountain, we far oftener meet with stunted forms, due to +the _directly_ injurious action of climate, than we do in proceeding +southwards or in descending a mountain. When we reach the Arctic +regions, or snow-capped summits, or absolute deserts, the struggle for +life is almost exclusively with the elements. + +That climate acts in main part indirectly by favouring other species, +we may clearly see in the prodigious number of plants in our gardens +which can perfectly well endure our climate, but which never become +naturalised, for they cannot compete with our native plants, nor resist +destruction by our native animals. + +When a species, owing to highly favourable circumstances, increases +inordinately in numbers in a small tract, epidemics—at least, this +seems generally to occur with our game animals—often ensue: and here we +have a limiting check independent of the struggle for life. But even +some of these so-called epidemics appear to be due to parasitic worms, +which have from some cause, possibly in part through facility of +diffusion amongst the crowded animals, been disproportionably favoured: +and here comes in a sort of struggle between the parasite and its prey. + +On the other hand, in many cases, a large stock of individuals of the +same species, relatively to the numbers of its enemies, is absolutely +necessary for its preservation. Thus we can easily raise plenty of corn +and rape-seed, etc., in our fields, because the seeds are in great +excess compared with the number of birds which feed on them; nor can +the birds, though having a superabundance of food at this one season, +increase in number proportionally to the supply of seed, as their +numbers are checked during winter: but any one who has tried, knows how +troublesome it is to get seed from a few wheat or other such plants in +a garden; I have in this case lost every single seed. This view of the +necessity of a large stock of the same species for its preservation, +explains, I believe, some singular facts in nature, such as that of +very rare plants being sometimes extremely abundant in the few spots +where they do occur; and that of some social plants being social, that +is, abounding in individuals, even on the extreme confines of their +range. For in such cases, we may believe, that a plant could exist only +where the conditions of its life were so favourable that many could +exist together, and thus save each other from utter destruction. I +should add that the good effects of frequent intercrossing, and the ill +effects +of close interbreeding, probably come into play in some of these cases; +but on this intricate subject I will not here enlarge. + +Many cases are on record showing how complex and unexpected are the +checks and relations between organic beings, which have to struggle +together in the same country. I will give only a single instance, +which, though a simple one, has interested me. In Staffordshire, on the +estate of a relation where I had ample means of investigation, there +was a large and extremely barren heath, which had never been touched by +the hand of man; but several hundred acres of exactly the same nature +had been enclosed twenty-five years previously and planted with Scotch +fir. The change in the native vegetation of the planted part of the +heath was most remarkable, more than is generally seen in passing from +one quite different soil to another: not only the proportional numbers +of the heath-plants were wholly changed, but twelve species of plants +(not counting grasses and carices) flourished in the plantations, which +could not be found on the heath. The effect on the insects must have +been still greater, for six insectivorous birds were very common in the +plantations, which were not to be seen on the heath; and the heath was +frequented by two or three distinct insectivorous birds. Here we see +how potent has been the effect of the introduction of a single tree, +nothing whatever else having been done, with the exception that the +land had been enclosed, so that cattle could not enter. But how +important an element enclosure is, I plainly saw near Farnham, in +Surrey. Here there are extensive heaths, with a few clumps of old +Scotch firs on the distant hill-tops: within the last ten years large +spaces have been enclosed, and self-sown firs are now springing up in +multitudes, so close together that all cannot live. +When I ascertained that these young trees had not been sown or planted, +I was so much surprised at their numbers that I went to several points +of view, whence I could examine hundreds of acres of the unenclosed +heath, and literally I could not see a single Scotch fir, except the +old planted clumps. But on looking closely between the stems of the +heath, I found a multitude of seedlings and little trees, which had +been perpetually browsed down by the cattle. In one square yard, at a +point some hundred yards distant from one of the old clumps, I counted +thirty-two little trees; and one of them, judging from the rings of +growth, had during twenty-six years tried to raise its head above the +stems of the heath, and had failed. No wonder that, as soon as the land +was enclosed, it became thickly clothed with vigorously growing young +firs. Yet the heath was so extremely barren and so extensive that no +one would ever have imagined that cattle would have so closely and +effectually searched it for food. + +Here we see that cattle absolutely determine the existence of the +Scotch fir; but in several parts of the world insects determine the +existence of cattle. Perhaps Paraguay offers the most curious instance +of this; for here neither cattle nor horses nor dogs have ever run +wild, though they swarm southward and northward in a feral state; and +Azara and Rengger have shown that this is caused by the greater number +in Paraguay of a certain fly, which lays its eggs in the navels of +these animals when first born. The increase of these flies, numerous as +they are, must be habitually checked by some means, probably by birds. +Hence, if certain insectivorous birds (whose numbers are probably +regulated by hawks or beasts of prey) were to increase in Paraguay, the +flies would decrease—then cattle and horses would become feral, and +this would certainly greatly alter (as +indeed I have observed in parts of South America) the vegetation: this +again would largely affect the insects; and this, as we just have seen +in Staffordshire, the insectivorous birds, and so onwards in +ever-increasing circles of complexity. We began this series by +insectivorous birds, and we have ended with them. Not that in nature +the relations can ever be as simple as this. Battle within battle must +ever be recurring with varying success; and yet in the long-run the +forces are so nicely balanced, that the face of nature remains uniform +for long periods of time, though assuredly the merest trifle would +often give the victory to one organic being over another. Nevertheless +so profound is our ignorance, and so high our presumption, that we +marvel when we hear of the extinction of an organic being; and as we do +not see the cause, we invoke cataclysms to desolate the world, or +invent laws on the duration of the forms of life! + +I am tempted to give one more instance showing how plants and animals, +most remote in the scale of nature, are bound together by a web of +complex relations. I shall hereafter have occasion to show that the +exotic Lobelia fulgens, in this part of England, is never visited by +insects, and consequently, from its peculiar structure, never can set a +seed. Many of our orchidaceous plants absolutely require the visits of +moths to remove their pollen-masses and thus to fertilise them. I have, +also, reason to believe that humble-bees are indispensable to the +fertilisation of the heartsease (Viola tricolor), for other bees do not +visit this flower. From experiments which I have tried, I have found +that the visits of bees, if not indispensable, are at least highly +beneficial to the fertilisation of our clovers; but humble-bees alone +visit the common red clover (Trifolium pratense), as other bees cannot +reach the nectar. Hence I have very little doubt, that if the whole +genus of humble-bees became +extinct or very rare in England, the heartsease and red clover would +become very rare, or wholly disappear. The number of humble-bees in any +district depends in a great degree on the number of field-mice, which +destroy their combs and nests; and Mr. H. Newman, who has long attended +to the habits of humble-bees, believes that “more than two thirds of +them are thus destroyed all over England.” Now the number of mice is +largely dependent, as every one knows, on the number of cats; and Mr. +Newman says, “Near villages and small towns I have found the nests of +humble-bees more numerous than elsewhere, which I attribute to the +number of cats that destroy the mice.” Hence it is quite credible that +the presence of a feline animal in large numbers in a district might +determine, through the intervention first of mice and then of bees, the +frequency of certain flowers in that district! + +In the case of every species, many different checks, acting at +different periods of life, and during different seasons or years, +probably come into play; some one check or some few being generally the +most potent, but all concurring in determining the average number or +even the existence of the species. In some cases it can be shown that +widely-different checks act on the same species in different districts. +When we look at the plants and bushes clothing an entangled bank, we +are tempted to attribute their proportional numbers and kinds to what +we call chance. But how false a view is this! Every one has heard that +when an American forest is cut down, a very different vegetation +springs up; but it has been observed that the trees now growing on the +ancient Indian mounds, in the Southern United States, display the same +beautiful diversity and proportion of kinds as in the surrounding +virgin forests. What a struggle between the several kinds of trees +must here have gone on during long centuries, each annually scattering +its seeds by the thousand; what war between insect and insect—between +insects, snails, and other animals with birds and beasts of prey—all +striving to increase, and all feeding on each other or on the trees or +their seeds and seedlings, or on the other plants which first clothed +the ground and thus checked the growth of the trees! Throw up a handful +of feathers, and all must fall to the ground according to definite +laws; but how simple is this problem compared to the action and +reaction of the innumerable plants and animals which have determined, +in the course of centuries, the proportional numbers and kinds of trees +now growing on the old Indian ruins! + +The dependency of one organic being on another, as of a parasite on its +prey, lies generally between beings remote in the scale of nature. This +is often the case with those which may strictly be said to struggle +with each other for existence, as in the case of locusts and +grass-feeding quadrupeds. But the struggle almost invariably will be +most severe between the individuals of the same species, for they +frequent the same districts, require the same food, and are exposed to +the same dangers. In the case of varieties of the same species, the +struggle will generally be almost equally severe, and we sometimes see +the contest soon decided: for instance, if several varieties of wheat +be sown together, and the mixed seed be resown, some of the varieties +which best suit the soil or climate, or are naturally the most fertile, +will beat the others and so yield more seed, and will consequently in a +few years quite supplant the other varieties. To keep up a mixed stock +of even such extremely close varieties as the variously coloured +sweet-peas, they must be each year harvested separately, and the seed +then mixed in due proportion, +otherwise the weaker kinds will steadily decrease in numbers and +disappear. So again with the varieties of sheep: it has been asserted +that certain mountain-varieties will starve out other +mountain-varieties, so that they cannot be kept together. The same +result has followed from keeping together different varieties of the +medicinal leech. It may even be doubted whether the varieties of any +one of our domestic plants or animals have so exactly the same +strength, habits, and constitution, that the original proportions of a +mixed stock could be kept up for half a dozen generations, if they were +allowed to struggle together, like beings in a state of nature, and if +the seed or young were not annually sorted. + +As species of the same genus have usually, though by no means +invariably, some similarity in habits and constitution, and always in +structure, the struggle will generally be more severe between species +of the same genus, when they come into competition with each other, +than between species of distinct genera. We see this in the recent +extension over parts of the United States of one species of swallow +having caused the decrease of another species. The recent increase of +the missel-thrush in parts of Scotland has caused the decrease of the +song-thrush. How frequently we hear of one species of rat taking the +place of another species under the most different climates! In Russia +the small Asiatic cockroach has everywhere driven before it its great +congener. One species of charlock will supplant another, and so in +other cases. We can dimly see why the competition should be most severe +between allied forms, which fill nearly the same place in the economy +of nature; but probably in no one case could we precisely say why one +species has been victorious over another in the great battle of life. + +A corollary of the highest importance may be deduced from the foregoing +remarks, namely, that the structure of every organic being is related, +in the most essential yet often hidden manner, to that of all other +organic beings, with which it comes into competition for food or +residence, or from which it has to escape, or on which it preys. This +is obvious in the structure of the teeth and talons of the tiger; and +in that of the legs and claws of the parasite which clings to the hair +on the tiger’s body. But in the beautifully plumed seed of the +dandelion, and in the flattened and fringed legs of the water-beetle, +the relation seems at first confined to the elements of air and water. +Yet the advantage of plumed seeds no doubt stands in the closest +relation to the land being already thickly clothed by other plants; so +that the seeds may be widely distributed and fall on unoccupied ground. +In the water-beetle, the structure of its legs, so well adapted for +diving, allows it to compete with other aquatic insects, to hunt for +its own prey, and to escape serving as prey to other animals. + +The store of nutriment laid up within the seeds of many plants seems at +first sight to have no sort of relation to other plants. But from the +strong growth of young plants produced from such seeds (as peas and +beans), when sown in the midst of long grass, I suspect that the chief +use of the nutriment in the seed is to favour the growth of the young +seedling, whilst struggling with other plants growing vigorously all +around. + +Look at a plant in the midst of its range, why does it not double or +quadruple its numbers? We know that it can perfectly well withstand a +little more heat or cold, dampness or dryness, for elsewhere it ranges +into slightly hotter or colder, damper or drier districts. In this case +we can clearly see that if we wished in imagination to give the plant +the power of increasing in number, we should have to give it some +advantage over its competitors, or over the animals which preyed on it. +On the confines of its geographical range, a change of constitution +with respect to climate would clearly be an advantage to our plant; but +we have reason to believe that only a few plants or animals range so +far, that they are destroyed by the rigour of the climate alone. Not +until we reach the extreme confines of life, in the arctic regions or +on the borders of an utter desert, will competition cease. The land may +be extremely cold or dry, yet there will be competition between some +few species, or between the individuals of the same species, for the +warmest or dampest spots. + +Hence, also, we can see that when a plant or animal is placed in a new +country amongst new competitors, though the climate may be exactly the +same as in its former home, yet the conditions of its life will +generally be changed in an essential manner. If we wished to increase +its average numbers in its new home, we should have to modify it in a +different way to what we should have done in its native country; for we +should have to give it some advantage over a different set of +competitors or enemies. + +It is good thus to try in our imagination to give any form some +advantage over another. Probably in no single instance should we know +what to do, so as to succeed. It will convince us of our ignorance on +the mutual relations of all organic beings; a conviction as necessary, +as it seems to be difficult to acquire. All that we can do, is to keep +steadily in mind that each organic being is striving to increase at a +geometrical +ratio; that each at some period of its life, during some season of the +year, during each generation or at intervals, has to struggle for life, +and to suffer great destruction. When we reflect on this struggle, we +may console ourselves with the full belief, that the war of nature is +not incessant, that no fear is felt, that death is generally prompt, +and that the vigorous, the healthy, and the happy survive and multiply. + +Natural Selection: its power compared with man’s selection, its power +on characters of trifling importance, its power at all ages and on both +sexes. Sexual Selection. On the generality of intercrosses between +individuals of the same species. Circumstances favourable and +unfavourable to Natural Selection, namely, intercrossing, isolation, +number of individuals. Slow action. Extinction caused by Natural +Selection. Divergence of Character, related to the diversity of +inhabitants of any small area, and to naturalisation. Action of Natural +Selection, through Divergence of Character and Extinction, on the +descendants from a common parent. Explains the Grouping of all organic +beings. + +How will the struggle for existence, discussed too briefly in the last +chapter, act in regard to variation? Can the principle of selection, +which we have seen is so potent in the hands of man, apply in nature? I +think we shall see that it can act most effectually. Let it be borne in +mind in what an endless number of strange peculiarities our domestic +productions, and, in a lesser degree, those under nature, vary; and how +strong the hereditary tendency is. Under domestication, it may be truly +said that the whole organisation becomes in some degree plastic. Let it +be borne in mind how infinitely complex and close-fitting are the +mutual relations of all organic beings to each other and to their +physical conditions of life. Can it, then, be thought improbable, +seeing that variations useful to man have undoubtedly occurred, that +other variations useful in some way to each being in the great and +complex battle of life, should sometimes occur in the course of +thousands of generations? If such do occur, can we doubt (remembering +that many more individuals are born than can possibly survive) that +individuals having any advantage, however slight, over others, would +have the best chance of surviving and of procreating their kind? On the +other hand, we may feel sure that any variation in the least degree +injurious would be rigidly destroyed. This preservation of favourable +variations and the rejection of injurious variations, I call Natural +Selection. Variations neither useful nor injurious would not be +affected by natural selection, and would be left a fluctuating element, +as perhaps we see in the species called polymorphic. + +We shall best understand the probable course of natural selection by +taking the case of a country undergoing some physical change, for +instance, of climate. The proportional numbers of its inhabitants would +almost immediately undergo a change, and some species might become +extinct. We may conclude, from what we have seen of the intimate and +complex manner in which the inhabitants of each country are bound +together, that any change in the numerical proportions of some of the +inhabitants, independently of the change of climate itself, would most +seriously affect many of the others. If the country were open on its +borders, new forms would certainly immigrate, and this also would +seriously disturb the relations of some of the former inhabitants. Let +it be remembered how powerful the influence of a single introduced tree +or mammal has been shown to be. But in the case of an island, or of a +country partly surrounded by barriers, into which new and better +adapted forms could not freely enter, we should then have places in the +economy of nature which would assuredly be better filled up, if some of +the original inhabitants were in some manner modified; for, had the +area been open to immigration, these same +places would have been seized on by intruders. In such case, every +slight modification, which in the course of ages chanced to arise, and +which in any way favoured the individuals of any of the species, by +better adapting them to their altered conditions, would tend to be +preserved; and natural selection would thus have free scope for the +work of improvement. + +We have reason to believe, as stated in the first chapter, that a +change in the conditions of life, by specially acting on the +reproductive system, causes or increases variability; and in the +foregoing case the conditions of life are supposed to have undergone a +change, and this would manifestly be favourable to natural selection, +by giving a better chance of profitable variations occurring; and +unless profitable variations do occur, natural selection can do +nothing. Not that, as I believe, any extreme amount of variability is +necessary; as man can certainly produce great results by adding up in +any given direction mere individual differences, so could Nature, but +far more easily, from having incomparably longer time at her disposal. +Nor do I believe that any great physical change, as of climate, or any +unusual degree of isolation to check immigration, is actually necessary +to produce new and unoccupied places for natural selection to fill up +by modifying and improving some of the varying inhabitants. For as all +the inhabitants of each country are struggling together with nicely +balanced forces, extremely slight modifications in the structure or +habits of one inhabitant would often give it an advantage over others; +and still further modifications of the same kind would often still +further increase the advantage. No country can be named in which all +the native inhabitants are now so perfectly adapted to each other and +to the physical conditions under which they live, that none of +them could anyhow be improved; for in all countries, the natives have +been so far conquered by naturalised productions, that they have +allowed foreigners to take firm possession of the land. And as +foreigners have thus everywhere beaten some of the natives, we may +safely conclude that the natives might have been modified with +advantage, so as to have better resisted such intruders. + +As man can produce and certainly has produced a great result by his +methodical and unconscious means of selection, what may not nature +effect? Man can act only on external and visible characters: nature +cares nothing for appearances, except in so far as they may be useful +to any being. She can act on every internal organ, on every shade of +constitutional difference, on the whole machinery of life. Man selects +only for his own good; Nature only for that of the being which she +tends. Every selected character is fully exercised by her; and the +being is placed under well-suited conditions of life. Man keeps the +natives of many climates in the same country; he seldom exercises each +selected character in some peculiar and fitting manner; he feeds a long +and a short beaked pigeon on the same food; he does not exercise a +long-backed or long-legged quadruped in any peculiar manner; he exposes +sheep with long and short wool to the same climate. He does not allow +the most vigorous males to struggle for the females. He does not +rigidly destroy all inferior animals, but protects during each varying +season, as far as lies in his power, all his productions. He often +begins his selection by some half-monstrous form; or at least by some +modification prominent enough to catch his eye, or to be plainly useful +to him. Under nature, the slightest difference of structure or +constitution may well turn the nicely-balanced scale in the +struggle for life, and so be preserved. How fleeting are the wishes and +efforts of man! how short his time! and consequently how poor will his +products be, compared with those accumulated by nature during whole +geological periods. Can we wonder, then, that nature’s productions +should be far “truer” in character than man’s productions; that they +should be infinitely better adapted to the most complex conditions of +life, and should plainly bear the stamp of far higher workmanship? + +It may be said that natural selection is daily and hourly scrutinising, +throughout the world, every variation, even the slightest; rejecting +that which is bad, preserving and adding up all that is good; silently +and insensibly working, whenever and wherever opportunity offers, at +the improvement of each organic being in relation to its organic and +inorganic conditions of life. We see nothing of these slow changes in +progress, until the hand of time has marked the long lapse of ages, and +then so imperfect is our view into long past geological ages, that we +only see that the forms of life are now different from what they +formerly were. + +Although natural selection can act only through and for the good of +each being, yet characters and structures, which we are apt to consider +as of very trifling importance, may thus be acted on. When we see +leaf-eating insects green, and bark-feeders mottled-grey; the alpine +ptarmigan white in winter, the red-grouse the colour of heather, and +the black-grouse that of peaty earth, we must believe that these tints +are of service to these birds and insects in preserving them from +danger. Grouse, if not destroyed at some period of their lives, would +increase in countless numbers; they are known to suffer largely from +birds of prey; and hawks are guided by eyesight to their prey,—so much +so, that on +parts of the Continent persons are warned not to keep white pigeons, as +being the most liable to destruction. Hence I can see no reason to +doubt that natural selection might be most effective in giving the +proper colour to each kind of grouse, and in keeping that colour, when +once acquired, true and constant. Nor ought we to think that the +occasional destruction of an animal of any particular colour would +produce little effect: we should remember how essential it is in a +flock of white sheep to destroy every lamb with the faintest trace of +black. In plants the down on the fruit and the colour of the flesh are +considered by botanists as characters of the most trifling importance: +yet we hear from an excellent horticulturist, Downing, that in the +United States smooth-skinned fruits suffer far more from a beetle, a +curculio, than those with down; that purple plums suffer far more from +a certain disease than yellow plums; whereas another disease attacks +yellow-fleshed peaches far more than those with other coloured flesh. +If, with all the aids of art, these slight differences make a great +difference in cultivating the several varieties, assuredly, in a state +of nature, where the trees would have to struggle with other trees and +with a host of enemies, such differences would effectually settle which +variety, whether a smooth or downy, a yellow or purple fleshed fruit, +should succeed. + +In looking at many small points of difference between species, which, +as far as our ignorance permits us to judge, seem to be quite +unimportant, we must not forget that climate, food, etc., probably +produce some slight and direct effect. It is, however, far more +necessary to bear in mind that there are many unknown laws of +correlation of growth, which, when one part of the organisation is +modified through variation, and the modifications are accumulated by +natural selection for +the good of the being, will cause other modifications, often of the +most unexpected nature. + +As we see that those variations which under domestication appear at any +particular period of life, tend to reappear in the offspring at the +same period;—for instance, in the seeds of the many varieties of our +culinary and agricultural plants; in the caterpillar and cocoon stages +of the varieties of the silkworm; in the eggs of poultry, and in the +colour of the down of their chickens; in the horns of our sheep and +cattle when nearly adult;—so in a state of nature, natural selection +will be enabled to act on and modify organic beings at any age, by the +accumulation of profitable variations at that age, and by their +inheritance at a corresponding age. If it profit a plant to have its +seeds more and more widely disseminated by the wind, I can see no +greater difficulty in this being effected through natural selection, +than in the cotton-planter increasing and improving by selection the +down in the pods on his cotton-trees. Natural selection may modify and +adapt the larva of an insect to a score of contingencies, wholly +different from those which concern the mature insect. These +modifications will no doubt affect, through the laws of correlation, +the structure of the adult; and probably in the case of those insects +which live only for a few hours, and which never feed, a large part of +their structure is merely the correlated result of successive changes +in the structure of their larvæ. So, conversely, modifications in the +adult will probably often affect the structure of the larva; but in all +cases natural selection will ensure that modifications consequent on +other modifications at a different period of life, shall not be in the +least degree injurious: for if they became so, they would cause the +extinction of the species. + +Natural selection will modify the structure of the +young in relation to the parent, and of the parent in relation to the +young. In social animals it will adapt the structure of each individual +for the benefit of the community; if each in consequence profits by the +selected change. What natural selection cannot do, is to modify the +structure of one species, without giving it any advantage, for the good +of another species; and though statements to this effect may be found +in works of natural history, I cannot find one case which will bear +investigation. A structure used only once in an animal’s whole life, if +of high importance to it, might be modified to any extent by natural +selection; for instance, the great jaws possessed by certain insects, +and used exclusively for opening the cocoon—or the hard tip to the beak +of nestling birds, used for breaking the egg. It has been asserted, +that of the best short-beaked tumbler-pigeons more perish in the egg +than are able to get out of it; so that fanciers assist in the act of +hatching. Now, if nature had to make the beak of a full-grown pigeon +very short for the bird’s own advantage, the process of modification +would be very slow, and there would be simultaneously the most rigorous +selection of the young birds within the egg, which had the most +powerful and hardest beaks, for all with weak beaks would inevitably +perish: or, more delicate and more easily broken shells might be +selected, the thickness of the shell being known to vary like every +other structure. + +_Sexual Selection_.—Inasmuch as peculiarities often appear under +domestication in one sex and become hereditarily attached to that sex, +the same fact probably occurs under nature, and if so, natural +selection will be able to modify one sex in its functional relations to +the other sex, or in relation to wholly different habits of life in the +two sexes, as is sometimes the case +with insects. And this leads me to say a few words on what I call +Sexual Selection. This depends, not on a struggle for existence, but on +a struggle between the males for possession of the females; the result +is not death to the unsuccessful competitor, but few or no offspring. +Sexual selection is, therefore, less rigorous than natural selection. +Generally, the most vigorous males, those which are best fitted for +their places in nature, will leave most progeny. But in many cases, +victory will depend not on general vigour, but on having special +weapons, confined to the male sex. A hornless stag or spurless cock +would have a poor chance of leaving offspring. Sexual selection by +always allowing the victor to breed might surely give indomitable +courage, length to the spur, and strength to the wing to strike in the +spurred leg, as well as the brutal cock-fighter, who knows well that he +can improve his breed by careful selection of the best cocks. How low +in the scale of nature this law of battle descends, I know not; male +alligators have been described as fighting, bellowing, and whirling +round, like Indians in a war-dance, for the possession of the females; +male salmons have been seen fighting all day long; male stag-beetles +often bear wounds from the huge mandibles of other males. The war is, +perhaps, severest between the males of polygamous animals, and these +seem oftenest provided with special weapons. The males of carnivorous +animals are already well armed; though to them and to others, special +means of defence may be given through means of sexual selection, as the +mane to the lion, the shoulder-pad to the boar, and the hooked jaw to +the male salmon; for the shield may be as important for victory, as the +sword or spear. + +Amongst birds, the contest is often of a more peaceful character. All +those who have attended to the subject, +believe that there is the severest rivalry between the males of many +species to attract by singing the females. The rock-thrush of Guiana, +birds of Paradise, and some others, congregate; and successive males +display their gorgeous plumage and perform strange antics before the +females, which standing by as spectators, at last choose the most +attractive partner. Those who have closely attended to birds in +confinement well know that they often take individual preferences and +dislikes: thus Sir R. Heron has described how one pied peacock was +eminently attractive to all his hen birds. It may appear childish to +attribute any effect to such apparently weak means: I cannot here enter +on the details necessary to support this view; but if man can in a +short time give elegant carriage and beauty to his bantams, according +to his standard of beauty, I can see no good reason to doubt that +female birds, by selecting, during thousands of generations, the most +melodious or beautiful males, according to their standard of beauty, +might produce a marked effect. I strongly suspect that some well-known +laws with respect to the plumage of male and female birds, in +comparison with the plumage of the young, can be explained on the view +of plumage having been chiefly modified by sexual selection, acting +when the birds have come to the breeding age or during the breeding +season; the modifications thus produced being inherited at +corresponding ages or seasons, either by the males alone, or by the +males and females; but I have not space here to enter on this subject. + +Thus it is, as I believe, that when the males and females of any animal +have the same general habits of life, but differ in structure, colour, +or ornament, such differences have been mainly caused by sexual +selection; that is, individual males have had, in successive +generations, some slight advantage over other +males, in their weapons, means of defence, or charms; and have +transmitted these advantages to their male offspring. Yet, I would not +wish to attribute all such sexual differences to this agency: for we +see peculiarities arising and becoming attached to the male sex in our +domestic animals (as the wattle in male carriers, horn-like +protuberances in the cocks of certain fowls, etc.), which we cannot +believe to be either useful to the males in battle, or attractive to +the females. We see analogous cases under nature, for instance, the +tuft of hair on the breast of the turkey-cock, which can hardly be +either useful or ornamental to this bird;—indeed, had the tuft appeared +under domestication, it would have been called a monstrosity. + +_Illustrations of the action of Natural Selection_.—In order to make it +clear how, as I believe, natural selection acts, I must beg permission +to give one or two imaginary illustrations. Let us take the case of a +wolf, which preys on various animals, securing some by craft, some by +strength, and some by fleetness; and let us suppose that the fleetest +prey, a deer for instance, had from any change in the country increased +in numbers, or that other prey had decreased in numbers, during that +season of the year when the wolf is hardest pressed for food. I can +under such circumstances see no reason to doubt that the swiftest and +slimmest wolves would have the best chance of surviving, and so be +preserved or selected,—provided always that they retained strength to +master their prey at this or at some other period of the year, when +they might be compelled to prey on other animals. I can see no more +reason to doubt this, than that man can improve the fleetness of his +greyhounds by careful and methodical selection, or by that unconscious +selection which results from each man trying +to keep the best dogs without any thought of modifying the breed. + +Even without any change in the proportional numbers of the animals on +which our wolf preyed, a cub might be born with an innate tendency to +pursue certain kinds of prey. Nor can this be thought very improbable; +for we often observe great differences in the natural tendencies of our +domestic animals; one cat, for instance, taking to catch rats, another +mice; one cat, according to Mr. St. John, bringing home winged game, +another hares or rabbits, and another hunting on marshy ground and +almost nightly catching woodcocks or snipes. The tendency to catch rats +rather than mice is known to be inherited. Now, if any slight innate +change of habit or of structure benefited an individual wolf, it would +have the best chance of surviving and of leaving offspring. Some of its +young would probably inherit the same habits or structure, and by the +repetition of this process, a new variety might be formed which would +either supplant or coexist with the parent-form of wolf. Or, again, the +wolves inhabiting a mountainous district, and those frequenting the +lowlands, would naturally be forced to hunt different prey; and from +the continued preservation of the individuals best fitted for the two +sites, two varieties might slowly be formed. These varieties would +cross and blend where they met; but to this subject of intercrossing we +shall soon have to return. I may add, that, according to Mr. Pierce, +there are two varieties of the wolf inhabiting the Catskill Mountains +in the United States, one with a light greyhound-like form, which +pursues deer, and the other more bulky, with shorter legs, which more +frequently attacks the shepherd’s flocks. + +Let us now take a more complex case. Certain plants excrete a sweet +juice, apparently for the sake of eliminating something injurious from +their sap: this is +effected by glands at the base of the stipules in some Leguminosæ, and +at the back of the leaf of the common laurel. This juice, though small +in quantity, is greedily sought by insects. Let us now suppose a little +sweet juice or nectar to be excreted by the inner bases of the petals +of a flower. In this case insects in seeking the nectar would get +dusted with pollen, and would certainly often transport the pollen from +one flower to the stigma of another flower. The flowers of two distinct +individuals of the same species would thus get crossed; and the act of +crossing, we have good reason to believe (as will hereafter be more +fully alluded to), would produce very vigorous seedlings, which +consequently would have the best chance of flourishing and surviving. +Some of these seedlings would probably inherit the nectar-excreting +power. Those individual flowers which had the largest glands or +nectaries, and which excreted most nectar, would be oftenest visited by +insects, and would be oftenest crossed; and so in the long-run would +gain the upper hand. Those flowers, also, which had their stamens and +pistils placed, in relation to the size and habits of the particular +insects which visited them, so as to favour in any degree the +transportal of their pollen from flower to flower, would likewise be +favoured or selected. We might have taken the case of insects visiting +flowers for the sake of collecting pollen instead of nectar; and as +pollen is formed for the sole object of fertilisation, its destruction +appears a simple loss to the plant; yet if a little pollen were +carried, at first occasionally and then habitually, by the +pollen-devouring insects from flower to flower, and a cross thus +effected, although nine-tenths of the pollen were destroyed, it might +still be a great gain to the plant; and those individuals which +produced more and more pollen, and had larger and larger anthers, would +be selected. + +When our plant, by this process of the continued preservation or +natural selection of more and more attractive flowers, had been +rendered highly attractive to insects, they would, unintentionally on +their part, regularly carry pollen from flower to flower; and that they +can most effectually do this, I could easily show by many striking +instances. I will give only one—not as a very striking case, but as +likewise illustrating one step in the separation of the sexes of +plants, presently to be alluded to. Some holly-trees bear only male +flowers, which have four stamens producing rather a small quantity of +pollen, and a rudimentary pistil; other holly-trees bear only female +flowers; these have a full-sized pistil, and four stamens with +shrivelled anthers, in which not a grain of pollen can be detected. +Having found a female tree exactly sixty yards from a male tree, I put +the stigmas of twenty flowers, taken from different branches, under the +microscope, and on all, without exception, there were pollen-grains, +and on some a profusion of pollen. As the wind had set for several days +from the female to the male tree, the pollen could not thus have been +carried. The weather had been cold and boisterous, and therefore not +favourable to bees, nevertheless every female flower which I examined +had been effectually fertilised by the bees, accidentally dusted with +pollen, having flown from tree to tree in search of nectar. But to +return to our imaginary case: as soon as the plant had been rendered so +highly attractive to insects that pollen was regularly carried from +flower to flower, another process might commence. No naturalist doubts +the advantage of what has been called the “physiological division of +labour;” hence we may believe that it would be advantageous to a plant +to produce stamens alone in one flower or on one whole plant, and +pistils alone in +another flower or on another plant. In plants under culture and placed +under new conditions of life, sometimes the male organs and sometimes +the female organs become more or less impotent; now if we suppose this +to occur in ever so slight a degree under nature, then as pollen is +already carried regularly from flower to flower, and as a more complete +separation of the sexes of our plant would be advantageous on the +principle of the division of labour, individuals with this tendency +more and more increased, would be continually favoured or selected, +until at last a complete separation of the sexes would be effected. + +Let us now turn to the nectar-feeding insects in our imaginary case: we +may suppose the plant of which we have been slowly increasing the +nectar by continued selection, to be a common plant; and that certain +insects depended in main part on its nectar for food. I could give many +facts, showing how anxious bees are to save time; for instance, their +habit of cutting holes and sucking the nectar at the bases of certain +flowers, which they can, with a very little more trouble, enter by the +mouth. Bearing such facts in mind, I can see no reason to doubt that an +accidental deviation in the size and form of the body, or in the +curvature and length of the proboscis, etc., far too slight to be +appreciated by us, might profit a bee or other insect, so that an +individual so characterised would be able to obtain its food more +quickly, and so have a better chance of living and leaving descendants. +Its descendants would probably inherit a tendency to a similar slight +deviation of structure. The tubes of the corollas of the common red and +incarnate clovers (Trifolium pratense and incarnatum) do not on a hasty +glance appear to differ in length; yet the hive-bee can easily suck the +nectar out of the incarnate clover, but not out of the common red +clover, which is visited by humble-bees alone; so that whole fields of +the red clover offer in vain an abundant supply of precious nectar to +the hive-bee. Thus it might be a great advantage to the hive-bee to +have a slightly longer or differently constructed proboscis. On the +other hand, I have found by experiment that the fertility of clover +greatly depends on bees visiting and moving parts of the corolla, so as +to push the pollen on to the stigmatic surface. Hence, again, if +humble-bees were to become rare in any country, it might be a great +advantage to the red clover to have a shorter or more deeply divided +tube to its corolla, so that the hive-bee could visit its flowers. Thus +I can understand how a flower and a bee might slowly become, either +simultaneously or one after the other, modified and adapted in the most +perfect manner to each other, by the continued preservation of +individuals presenting mutual and slightly favourable deviations of +structure. + +I am well aware that this doctrine of natural selection, exemplified in +the above imaginary instances, is open to the same objections which +were at first urged against Sir Charles Lyell’s noble views on “the +modern changes of the earth, as illustrative of geology;” but we now +very seldom hear the action, for instance, of the coast-waves, called a +trifling and insignificant cause, when applied to the excavation of +gigantic valleys or to the formation of the longest lines of inland +cliffs. Natural selection can act only by the preservation and +accumulation of infinitesimally small inherited modifications, each +profitable to the preserved being; and as modern geology has almost +banished such views as the excavation of a great valley by a single +diluvial wave, so will natural selection, if it be a true principle, +banish the belief of the continued creation of new organic +beings, or of any great and sudden modification in their structure. + +_On the Intercrossing of Individuals_.—I must here introduce a short +digression. In the case of animals and plants with separated sexes, it +is of course obvious that two individuals must always unite for each +birth; but in the case of hermaphrodites this is far from obvious. +Nevertheless I am strongly inclined to believe that with all +hermaphrodites two individuals, either occasionally or habitually, +concur for the reproduction of their kind. This view, I may add, was +first suggested by Andrew Knight. We shall presently see its +importance; but I must here treat the subject with extreme brevity, +though I have the materials prepared for an ample discussion. All +vertebrate animals, all insects, and some other large groups of +animals, pair for each birth. Modern research has much diminished the +number of supposed hermaphrodites, and of real hermaphrodites a large +number pair; that is, two individuals regularly unite for reproduction, +which is all that concerns us. But still there are many hermaphrodite +animals which certainly do not habitually pair, and a vast majority of +plants are hermaphrodites. What reason, it may be asked, is there for +supposing in these cases that two individuals ever concur in +reproduction? As it is impossible here to enter on details, I must +trust to some general considerations alone. + +In the first place, I have collected so large a body of facts, showing, +in accordance with the almost universal belief of breeders, that with +animals and plants a cross between different varieties, or between +individuals of the same variety but of another strain, gives vigour and +fertility to the offspring; and on the other hand, that _close_ +interbreeding diminishes vigour and fertility; that +these facts alone incline me to believe that it is a general law of +nature (utterly ignorant though we be of the meaning of the law) that +no organic being self-fertilises itself for an eternity of generations; +but that a cross with another individual is occasionally—perhaps at +very long intervals—indispensable. + +On the belief that this is a law of nature, we can, I think, understand +several large classes of facts, such as the following, which on any +other view are inexplicable. Every hybridizer knows how unfavourable +exposure to wet is to the fertilisation of a flower, yet what a +multitude of flowers have their anthers and stigmas fully exposed to +the weather! but if an occasional cross be indispensable, the fullest +freedom for the entrance of pollen from another individual will explain +this state of exposure, more especially as the plant’s own anthers and +pistil generally stand so close together that self-fertilisation seems +almost inevitable. Many flowers, on the other hand, have their organs +of fructification closely enclosed, as in the great papilionaceous or +pea-family; but in several, perhaps in all, such flowers, there is a +very curious adaptation between the structure of the flower and the +manner in which bees suck the nectar; for, in doing this, they either +push the flower’s own pollen on the stigma, or bring pollen from +another flower. So necessary are the visits of bees to papilionaceous +flowers, that I have found, by experiments published elsewhere, that +their fertility is greatly diminished if these visits be prevented. +Now, it is scarcely possible that bees should fly from flower to +flower, and not carry pollen from one to the other, to the great good, +as I believe, of the plant. Bees will act like a camel-hair pencil, and +it is quite sufficient just to touch the anthers of one flower and then +the stigma of another with the same brush to ensure fertilisation; but +it must not be +supposed that bees would thus produce a multitude of hybrids between +distinct species; for if you bring on the same brush a plant’s own +pollen and pollen from another species, the former will have such a +prepotent effect, that it will invariably and completely destroy, as +has been shown by Gärtner, any influence from the foreign pollen. + +When the stamens of a flower suddenly spring towards the pistil, or +slowly move one after the other towards it, the contrivance seems +adapted solely to ensure self-fertilisation; and no doubt it is useful +for this end: but, the agency of insects is often required to cause the +stamens to spring forward, as Kölreuter has shown to be the case with +the barberry; and curiously in this very genus, which seems to have a +special contrivance for self-fertilisation, it is well known that if +very closely-allied forms or varieties are planted near each other, it +is hardly possible to raise pure seedlings, so largely do they +naturally cross. In many other cases, far from there being any aids for +self-fertilisation, there are special contrivances, as I could show +from the writings of C. C. Sprengel and from my own observations, which +effectually prevent the stigma receiving pollen from its own flower: +for instance, in Lobelia fulgens, there is a really beautiful and +elaborate contrivance by which every one of the infinitely numerous +pollen-granules are swept out of the conjoined anthers of each flower, +before the stigma of that individual flower is ready to receive them; +and as this flower is never visited, at least in my garden, by insects, +it never sets a seed, though by placing pollen from one flower on the +stigma of another, I raised plenty of seedlings; and whilst another +species of Lobelia growing close by, which is visited by bees, seeds +freely. In very many other cases, though there be no special mechanical +contrivance to prevent the stigma of a flower receiving its own pollen, +yet, as +C. C. Sprengel has shown, and as I can confirm, either the anthers +burst before the stigma is ready for fertilisation, or the stigma is +ready before the pollen of that flower is ready, so that these plants +have in fact separated sexes, and must habitually be crossed. How +strange are these facts! How strange that the pollen and stigmatic +surface of the same flower, though placed so close together, as if for +the very purpose of self-fertilisation, should in so many cases be +mutually useless to each other! How simply are these facts explained on +the view of an occasional cross with a distinct individual being +advantageous or indispensable! + +If several varieties of the cabbage, radish, onion, and of some other +plants, be allowed to seed near each other, a large majority, as I have +found, of the seedlings thus raised will turn out mongrels: for +instance, I raised 233 seedling cabbages from some plants of different +varieties growing near each other, and of these only 78 were true to +their kind, and some even of these were not perfectly true. Yet the +pistil of each cabbage-flower is surrounded not only by its own six +stamens, but by those of the many other flowers on the same plant. How, +then, comes it that such a vast number of the seedlings are +mongrelized? I suspect that it must arise from the pollen of a distinct +_variety_ having a prepotent effect over a flower’s own pollen; and +that this is part of the general law of good being derived from the +intercrossing of distinct individuals of the same species. When +distinct _species_ are crossed the case is directly the reverse, for a +plant’s own pollen is always prepotent over foreign pollen; but to this +subject we shall return in a future chapter. + +In the case of a gigantic tree covered with innumerable flowers, it may +be objected that pollen could seldom be carried from tree to tree, and +at most only from flower +to flower on the same tree, and that flowers on the same tree can be +considered as distinct individuals only in a limited sense. I believe +this objection to be valid, but that nature has largely provided +against it by giving to trees a strong tendency to bear flowers with +separated sexes. When the sexes are separated, although the male and +female flowers may be produced on the same tree, we can see that pollen +must be regularly carried from flower to flower; and this will give a +better chance of pollen being occasionally carried from tree to tree. +That trees belonging to all Orders have their sexes more often +separated than other plants, I find to be the case in this country; and +at my request Dr. Hooker tabulated the trees of New Zealand, and Dr. +Asa Gray those of the United States, and the result was as I +anticipated. On the other hand, Dr. Hooker has recently informed me +that he finds that the rule does not hold in Australia; and I have made +these few remarks on the sexes of trees simply to call attention to the +subject. + +Turning for a very brief space to animals: on the land there are some +hermaphrodites, as land-mollusca and earth-worms; but these all pair. +As yet I have not found a single case of a terrestrial animal which +fertilises itself. We can understand this remarkable fact, which offers +so strong a contrast with terrestrial plants, on the view of an +occasional cross being indispensable, by considering the medium in +which terrestrial animals live, and the nature of the fertilising +element; for we know of no means, analogous to the action of insects +and of the wind in the case of plants, by which an occasional cross +could be effected with terrestrial animals without the concurrence of +two individuals. Of aquatic animals, there are many self-fertilising +hermaphrodites; but here currents in the water offer an obvious means +for an occasional cross. And, as in the case of flowers, I have as yet +failed, after consultation with one of the highest authorities, namely, +Professor Huxley, to discover a single case of an hermaphrodite animal +with the organs of reproduction so perfectly enclosed within the body, +that access from without and the occasional influence of a distinct +individual can be shown to be physically impossible. Cirripedes long +appeared to me to present a case of very great difficulty under this +point of view; but I have been enabled, by a fortunate chance, +elsewhere to prove that two individuals, though both are +self-fertilising hermaphrodites, do sometimes cross. + +It must have struck most naturalists as a strange anomaly that, in the +case of both animals and plants, species of the same family and even of +the same genus, though agreeing closely with each other in almost their +whole organisation, yet are not rarely, some of them hermaphrodites, +and some of them unisexual. But if, in fact, all hermaphrodites do +occasionally intercross with other individuals, the difference between +hermaphrodites and unisexual species, as far as function is concerned, +becomes very small. + +From these several considerations and from the many special facts which +I have collected, but which I am not here able to give, I am strongly +inclined to suspect that, both in the vegetable and animal kingdoms, an +occasional intercross with a distinct individual is a law of nature. I +am well aware that there are, on this view, many cases of difficulty, +some of which I am trying to investigate. Finally then, we may conclude +that in many organic beings, a cross between two individuals is an +obvious necessity for each birth; in many others it occurs perhaps only +at long intervals; but in none, as I suspect, can self-fertilisation go +on for perpetuity. + +_Circumstances favourable to Natural Selection_.—This +is an extremely intricate subject. A large amount of inheritable and +diversified variability is favourable, but I believe mere individual +differences suffice for the work. A large number of individuals, by +giving a better chance for the appearance within any given period of +profitable variations, will compensate for a lesser amount of +variability in each individual, and is, I believe, an extremely +important element of success. Though nature grants vast periods of time +for the work of natural selection, she does not grant an indefinite +period; for as all organic beings are striving, it may be said, to +seize on each place in the economy of nature, if any one species does +not become modified and improved in a corresponding degree with its +competitors, it will soon be exterminated. + +In man’s methodical selection, a breeder selects for some definite +object, and free intercrossing will wholly stop his work. But when many +men, without intending to alter the breed, have a nearly common +standard of perfection, and all try to get and breed from the best +animals, much improvement and modification surely but slowly follow +from this unconscious process of selection, notwithstanding a large +amount of crossing with inferior animals. Thus it will be in nature; +for within a confined area, with some place in its polity not so +perfectly occupied as might be, natural selection will always tend to +preserve all the individuals varying in the right direction, though in +different degrees, so as better to fill up the unoccupied place. But if +the area be large, its several districts will almost certainly present +different conditions of life; and then if natural selection be +modifying and improving a species in the several districts, there will +be intercrossing with the other individuals of the same species on the +confines of each. And in this case the effects of intercrossing can +hardly be counterbalanced +by natural selection always tending to modify all the individuals in +each district in exactly the same manner to the conditions of each; for +in a continuous area, the conditions will generally graduate away +insensibly from one district to another. The intercrossing will most +affect those animals which unite for each birth, which wander much, and +which do not breed at a very quick rate. Hence in animals of this +nature, for instance in birds, varieties will generally be confined to +separated countries; and this I believe to be the case. In +hermaphrodite organisms which cross only occasionally, and likewise in +animals which unite for each birth, but which wander little and which +can increase at a very rapid rate, a new and improved variety might be +quickly formed on any one spot, and might there maintain itself in a +body, so that whatever intercrossing took place would be chiefly +between the individuals of the same new variety. A local variety when +once thus formed might subsequently slowly spread to other districts. +On the above principle, nurserymen always prefer getting seed from a +large body of plants of the same variety, as the chance of +intercrossing with other varieties is thus lessened. + +Even in the case of slow-breeding animals, which unite for each birth, +we must not overrate the effects of intercrosses in retarding natural +selection; for I can bring a considerable catalogue of facts, showing +that within the same area, varieties of the same animal can long remain +distinct, from haunting different stations, from breeding at slightly +different seasons, or from varieties of the same kind preferring to +pair together. + +Intercrossing plays a very important part in nature in keeping the +individuals of the same species, or of the same variety, true and +uniform in character. It will obviously thus act far more efficiently +with those animals +which unite for each birth; but I have already attempted to show that +we have reason to believe that occasional intercrosses take place with +all animals and with all plants. Even if these take place only at long +intervals, I am convinced that the young thus produced will gain so +much in vigour and fertility over the offspring from long-continued +self-fertilisation, that they will have a better chance of surviving +and propagating their kind; and thus, in the long run, the influence of +intercrosses, even at rare intervals, will be great. If there exist +organic beings which never intercross, uniformity of character can be +retained amongst them, as long as their conditions of life remain the +same, only through the principle of inheritance, and through natural +selection destroying any which depart from the proper type; but if +their conditions of life change and they undergo modification, +uniformity of character can be given to their modified offspring, +solely by natural selection preserving the same favourable variations. + +Isolation, also, is an important element in the process of natural +selection. In a confined or isolated area, if not very large, the +organic and inorganic conditions of life will generally be in a great +degree uniform; so that natural selection will tend to modify all the +individuals of a varying species throughout the area in the same manner +in relation to the same conditions. Intercrosses, also, with the +individuals of the same species, which otherwise would have inhabited +the surrounding and differently circumstanced districts, will be +prevented. But isolation probably acts more efficiently in checking the +immigration of better adapted organisms, after any physical change, +such as of climate or elevation of the land, etc.; and thus new places +in the natural economy of the country are left open for the old +inhabitants to struggle for, and become adapted to, through +modifications +in their structure and constitution. Lastly, isolation, by checking +immigration and consequently competition, will give time for any new +variety to be slowly improved; and this may sometimes be of importance +in the production of new species. If, however, an isolated area be very +small, either from being surrounded by barriers, or from having very +peculiar physical conditions, the total number of the individuals +supported on it will necessarily be very small; and fewness of +individuals will greatly retard the production of new species through +natural selection, by decreasing the chance of the appearance of +favourable variations. + +If we turn to nature to test the truth of these remarks, and look at +any small isolated area, such as an oceanic island, although the total +number of the species inhabiting it, will be found to be small, as we +shall see in our chapter on geographical distribution; yet of these +species a very large proportion are endemic,—that is, have been +produced there, and nowhere else. Hence an oceanic island at first +sight seems to have been highly favourable for the production of new +species. But we may thus greatly deceive ourselves, for to ascertain +whether a small isolated area, or a large open area like a continent, +has been most favourable for the production of new organic forms, we +ought to make the comparison within equal times; and this we are +incapable of doing. + +Although I do not doubt that isolation is of considerable importance in +the production of new species, on the whole I am inclined to believe +that largeness of area is of more importance, more especially in the +production of species, which will prove capable of enduring for a long +period, and of spreading widely. Throughout a great and open area, not +only will there be a better chance of favourable variations arising +from the large number of individuals of the same species +there supported, but the conditions of life are infinitely complex from +the large number of already existing species; and if some of these many +species become modified and improved, others will have to be improved +in a corresponding degree or they will be exterminated. Each new form, +also, as soon as it has been much improved, will be able to spread over +the open and continuous area, and will thus come into competition with +many others. Hence more new places will be formed, and the competition +to fill them will be more severe, on a large than on a small and +isolated area. Moreover, great areas, though now continuous, owing to +oscillations of level, will often have recently existed in a broken +condition, so that the good effects of isolation will generally, to a +certain extent, have concurred. Finally, I conclude that, although +small isolated areas probably have been in some respects highly +favourable for the production of new species, yet that the course of +modification will generally have been more rapid on large areas; and +what is more important, that the new forms produced on large areas, +which already have been victorious over many competitors, will be those +that will spread most widely, will give rise to most new varieties and +species, and will thus play an important part in the changing history +of the organic world. + +We can, perhaps, on these views, understand some facts which will be +again alluded to in our chapter on geographical distribution; for +instance, that the productions of the smaller continent of Australia +have formerly yielded, and apparently are now yielding, before those of +the larger Europæo-Asiatic area. Thus, also, it is that continental +productions have everywhere become so largely naturalised on islands. +On a small island, the race for life will have been less severe, and +there will have been less modification and less extermination. +Hence, perhaps, it comes that the flora of Madeira, according to Oswald +Heer, resembles the extinct tertiary flora of Europe. All fresh-water +basins, taken together, make a small area compared with that of the sea +or of the land; and, consequently, the competition between fresh-water +productions will have been less severe than elsewhere; new forms will +have been more slowly formed, and old forms more slowly exterminated. +And it is in fresh water that we find seven genera of Ganoid fishes, +remnants of a once preponderant order: and in fresh water we find some +of the most anomalous forms now known in the world, as the +Ornithorhynchus and Lepidosiren, which, like fossils, connect to a +certain extent orders now widely separated in the natural scale. These +anomalous forms may almost be called living fossils; they have endured +to the present day, from having inhabited a confined area, and from +having thus been exposed to less severe competition. + +To sum up the circumstances favourable and unfavourable to natural +selection, as far as the extreme intricacy of the subject permits. I +conclude, looking to the future, that for terrestrial productions a +large continental area, which will probably undergo many oscillations +of level, and which consequently will exist for long periods in a +broken condition, will be the most favourable for the production of +many new forms of life, likely to endure long and to spread widely. For +the area will first have existed as a continent, and the inhabitants, +at this period numerous in individuals and kinds, will have been +subjected to very severe competition. When converted by subsidence into +large separate islands, there will still exist many individuals of the +same species on each island: intercrossing on the confines of the range +of each species will thus be checked: after physical changes of any +kind, immigration will be prevented, +so that new places in the polity of each island will have to be filled +up by modifications of the old inhabitants; and time will be allowed +for the varieties in each to become well modified and perfected. When, +by renewed elevation, the islands shall be re-converted into a +continental area, there will again be severe competition: the most +favoured or improved varieties will be enabled to spread: there will be +much extinction of the less improved forms, and the relative +proportional numbers of the various inhabitants of the renewed +continent will again be changed; and again there will be a fair field +for natural selection to improve still further the inhabitants, and +thus produce new species. + +That natural selection will always act with extreme slowness, I fully +admit. Its action depends on there being places in the polity of +nature, which can be better occupied by some of the inhabitants of the +country undergoing modification of some kind. The existence of such +places will often depend on physical changes, which are generally very +slow, and on the immigration of better adapted forms having been +checked. But the action of natural selection will probably still +oftener depend on some of the inhabitants becoming slowly modified; the +mutual relations of many of the other inhabitants being thus disturbed. +Nothing can be effected, unless favourable variations occur, and +variation itself is apparently always a very slow process. The process +will often be greatly retarded by free intercrossing. Many will exclaim +that these several causes are amply sufficient wholly to stop the +action of natural selection. I do not believe so. On the other hand, I +do believe that natural selection will always act very slowly, often +only at long intervals of time, and generally on only a very few of the +inhabitants of the same region at the same time. I further believe, +that this very slow, intermittent +action of natural selection accords perfectly well with what geology +tells us of the rate and manner at which the inhabitants of this world +have changed. + +Slow though the process of selection may be, if feeble man can do much +by his powers of artificial selection, I can see no limit to the amount +of change, to the beauty and infinite complexity of the coadaptations +between all organic beings, one with another and with their physical +conditions of life, which may be effected in the long course of time by +nature’s power of selection. + +_Extinction_.—This subject will be more fully discussed in our chapter +on Geology; but it must be here alluded to from being intimately +connected with natural selection. Natural selection acts solely through +the preservation of variations in some way advantageous, which +consequently endure. But as from the high geometrical powers of +increase of all organic beings, each area is already fully stocked with +inhabitants, it follows that as each selected and favoured form +increases in number, so will the less favoured forms decrease and +become rare. Rarity, as geology tells us, is the precursor to +extinction. We can, also, see that any form represented by few +individuals will, during fluctuations in the seasons or in the number +of its enemies, run a good chance of utter extinction. But we may go +further than this; for as new forms are continually and slowly being +produced, unless we believe that the number of specific forms goes on +perpetually and almost indefinitely increasing, numbers inevitably must +become extinct. That the number of specific forms has not indefinitely +increased, geology shows us plainly; and indeed we can see reason why +they should not have thus increased, for the number of places in the +polity of nature is not indefinitely great,—not that we +have any means of knowing that any one region has as yet got its +maximum of species. Probably no region is as yet fully stocked, for at +the Cape of Good Hope, where more species of plants are crowded +together than in any other quarter of the world, some foreign plants +have become naturalised, without causing, as far as we know, the +extinction of any natives. + +Furthermore, the species which are most numerous in individuals will +have the best chance of producing within any given period favourable +variations. We have evidence of this, in the facts given in the second +chapter, showing that it is the common species which afford the +greatest number of recorded varieties, or incipient species. Hence, +rare species will be less quickly modified or improved within any given +period, and they will consequently be beaten in the race for life by +the modified descendants of the commoner species. + +From these several considerations I think it inevitably follows, that +as new species in the course of time are formed through natural +selection, others will become rarer and rarer, and finally extinct. The +forms which stand in closest competition with those undergoing +modification and improvement, will naturally suffer most. And we have +seen in the chapter on the Struggle for Existence that it is the most +closely-allied forms,—varieties of the same species, and species of the +same genus or of related genera,—which, from having nearly the same +structure, constitution, and habits, generally come into the severest +competition with each other. Consequently, each new variety or species, +during the progress of its formation, will generally press hardest on +its nearest kindred, and tend to exterminate them. We see the same +process of extermination amongst our domesticated productions, through +the selection of improved forms by man. Many curious +instances could be given showing how quickly new breeds of cattle, +sheep, and other animals, and varieties of flowers, take the place of +older and inferior kinds. In Yorkshire, it is historically known that +the ancient black cattle were displaced by the long-horns, and that +these “were swept away by the short-horns” (I quote the words of an +agricultural writer) “as if by some murderous pestilence.” + +_Divergence of Character_.—The principle, which I have designated by +this term, is of high importance on my theory, and explains, as I +believe, several important facts. In the first place, varieties, even +strongly-marked ones, though having somewhat of the character of +species—as is shown by the hopeless doubts in many cases how to rank +them—yet certainly differ from each other far less than do good and +distinct species. Nevertheless, according to my view, varieties are +species in the process of formation, or are, as I have called them, +incipient species. How, then, does the lesser difference between +varieties become augmented into the greater difference between species? +That this does habitually happen, we must infer from most of the +innumerable species throughout nature presenting well-marked +differences; whereas varieties, the supposed prototypes and parents of +future well-marked species, present slight and ill-defined differences. +Mere chance, as we may call it, might cause one variety to differ in +some character from its parents, and the offspring of this variety +again to differ from its parent in the very same character and in a +greater degree; but this alone would never account for so habitual and +large an amount of difference as that between varieties of the same +species and species of the same genus. + +As has always been my practice, let us seek light on +this head from our domestic productions. We shall here find something +analogous. A fancier is struck by a pigeon having a slightly shorter +beak; another fancier is struck by a pigeon having a rather longer +beak; and on the acknowledged principle that “fanciers do not and will +not admire a medium standard, but like extremes,” they both go on (as +has actually occurred with tumbler-pigeons) choosing and breeding from +birds with longer and longer beaks, or with shorter and shorter beaks. +Again, we may suppose that at an early period one man preferred swifter +horses; another stronger and more bulky horses. The early differences +would be very slight; in the course of time, from the continued +selection of swifter horses by some breeders, and of stronger ones by +others, the differences would become greater, and would be noted as +forming two sub-breeds; finally, after the lapse of centuries, the +sub-breeds would become converted into two well-established and +distinct breeds. As the differences slowly become greater, the inferior +animals with intermediate characters, being neither very swift nor very +strong, will have been neglected, and will have tended to disappear. +Here, then, we see in man’s productions the action of what may be +called the principle of divergence, causing differences, at first +barely appreciable, steadily to increase, and the breeds to diverge in +character both from each other and from their common parent. + +But how, it may be asked, can any analogous principle apply in nature? +I believe it can and does apply most efficiently, from the simple +circumstance that the more diversified the descendants from any one +species become in structure, constitution, and habits, by so much will +they be better enabled to seize on many and widely diversified places +in the polity of nature, and so be enabled to increase in numbers. + +We can clearly see this in the case of animals with simple habits. Take +the case of a carnivorous quadruped, of which the number that can be +supported in any country has long ago arrived at its full average. If +its natural powers of increase be allowed to act, it can succeed in +increasing (the country not undergoing any change in its conditions) +only by its varying descendants seizing on places at present occupied +by other animals: some of them, for instance, being enabled to feed on +new kinds of prey, either dead or alive; some inhabiting new stations, +climbing trees, frequenting water, and some perhaps becoming less +carnivorous. The more diversified in habits and structure the +descendants of our carnivorous animal became, the more places they +would be enabled to occupy. What applies to one animal will apply +throughout all time to all animals—that is, if they vary—for otherwise +natural selection can do nothing. So it will be with plants. It has +been experimentally proved, that if a plot of ground be sown with one +species of grass, and a similar plot be sown with several distinct +genera of grasses, a greater number of plants and a greater weight of +dry herbage can thus be raised. The same has been found to hold good +when first one variety and then several mixed varieties of wheat have +been sown on equal spaces of ground. Hence, if any one species of grass +were to go on varying, and those varieties were continually selected +which differed from each other in at all the same manner as distinct +species and genera of grasses differ from each other, a greater number +of individual plants of this species of grass, including its modified +descendants, would succeed in living on the same piece of ground. And +we well know that each species and each variety of grass is annually +sowing almost countless seeds; and thus, as it may be said, is striving +its utmost to increase its numbers. Consequently, +I cannot doubt that in the course of many thousands of generations, the +most distinct varieties of any one species of grass would always have +the best chance of succeeding and of increasing in numbers, and thus of +supplanting the less distinct varieties; and varieties, when rendered +very distinct from each other, take the rank of species. + +The truth of the principle, that the greatest amount of life can be +supported by great diversification of structure, is seen under many +natural circumstances. In an extremely small area, especially if freely +open to immigration, and where the contest between individual and +individual must be severe, we always find great diversity in its +inhabitants. For instance, I found that a piece of turf, three feet by +four in size, which had been exposed for many years to exactly the same +conditions, supported twenty species of plants, and these belonged to +eighteen genera and to eight orders, which shows how much these plants +differed from each other. So it is with the plants and insects on small +and uniform islets; and so in small ponds of fresh water. Farmers find +that they can raise most food by a rotation of plants belonging to the +most different orders: nature follows what may be called a simultaneous +rotation. Most of the animals and plants which live close round any +small piece of ground, could live on it (supposing it not to be in any +way peculiar in its nature), and may be said to be striving to the +utmost to live there; but, it is seen, that where they come into the +closest competition with each other, the advantages of diversification +of structure, with the accompanying differences of habit and +constitution, determine that the inhabitants, which thus jostle each +other most closely, shall, as a general rule, belong to what we call +different genera and orders. + +The same principle is seen in the naturalisation of +plants through man’s agency in foreign lands. It might have been +expected that the plants which have succeeded in becoming naturalised +in any land would generally have been closely allied to the indigenes; +for these are commonly looked at as specially created and adapted for +their own country. It might, also, perhaps have been expected that +naturalised plants would have belonged to a few groups more especially +adapted to certain stations in their new homes. But the case is very +different; and Alph. De Candolle has well remarked in his great and +admirable work, that floras gain by naturalisation, proportionally with +the number of the native genera and species, far more in new genera +than in new species. To give a single instance: in the last edition of +Dr. Asa Gray’s ‘Manual of the Flora of the Northern United States,’ 260 +naturalised plants are enumerated, and these belong to 162 genera. We +thus see that these naturalised plants are of a highly diversified +nature. They differ, moreover, to a large extent from the indigenes, +for out of the 162 genera, no less than 100 genera are not there +indigenous, and thus a large proportional addition is made to the +genera of these States. + +By considering the nature of the plants or animals which have struggled +successfully with the indigenes of any country, and have there become +naturalised, we can gain some crude idea in what manner some of the +natives would have had to be modified, in order to have gained an +advantage over the other natives; and we may, I think, at least safely +infer that diversification of structure, amounting to new generic +differences, would have been profitable to them. + +The advantage of diversification in the inhabitants of the same region +is, in fact, the same as that of the physiological division of labour +in the organs of the same individual body—a subject so well elucidated +by +Milne Edwards. No physiologist doubts that a stomach by being adapted +to digest vegetable matter alone, or flesh alone, draws most nutriment +from these substances. So in the general economy of any land, the more +widely and perfectly the animals and plants are diversified for +different habits of life, so will a greater number of individuals be +capable of there supporting themselves. A set of animals, with their +organisation but little diversified, could hardly compete with a set +more perfectly diversified in structure. It may be doubted, for +instance, whether the Australian marsupials, which are divided into +groups differing but little from each other, and feebly representing, +as Mr. Waterhouse and others have remarked, our carnivorous, ruminant, +and rodent mammals, could successfully compete with these +well-pronounced orders. In the Australian mammals, we see the process +of diversification in an early and incomplete stage of development. +After the foregoing discussion, which ought to have been much +amplified, we may, I think, assume that the modified descendants of any +one species will succeed by so much the better as they become more +diversified in structure, and are thus enabled to encroach on places +occupied by other beings. Now let us see how this principle of great +benefit being derived from divergence of character, combined with the +principles of natural selection and of extinction, will tend to act. + +The accompanying diagram will aid us in understanding this rather +perplexing subject. Let A to L represent the species of a genus large +in its own country; these species are supposed to resemble each other +in unequal degrees, as is so generally the case in nature, and as is +represented in the diagram by the letters standing at unequal +distances. I have said a large genus, because we have seen in the +second chapter, +that on an average more of the species of large genera vary than of +small genera; and the varying species of the large genera present a +greater number of varieties. We have, also, seen that the species, +which are the commonest and the most widely-diffused, vary more than +rare species with restricted ranges. Let (A) be a common, +widely-diffused, and varying species, belonging to a genus large in its +own country. The little fan of diverging dotted lines of unequal +lengths proceeding from (A), may represent its varying offspring. The +variations are supposed to be extremely slight, but of the most +diversified nature; they are not supposed all to appear simultaneously, +but often after long intervals of time; nor are they all supposed to +endure for equal periods. Only those variations which are in some way +profitable will be preserved or naturally selected. And here the +importance of the principle of benefit being derived from divergence of +character comes in; for this will generally lead to the most different +or divergent variations (represented by the outer dotted lines) being +preserved and accumulated by natural selection. When a dotted line +reaches one of the horizontal lines, and is there marked by a small +numbered letter, a sufficient amount of variation is supposed to have +been accumulated to have formed a fairly well-marked variety, such as +would be thought worthy of record in a systematic work. + +The intervals between the horizontal lines in the diagram, may +represent each a thousand generations; but it would have been better if +each had represented ten thousand generations. After a thousand +generations, species (A) is supposed to have produced two fairly +well-marked varieties, namely _a_1 and _m_1. These two varieties will +generally continue to be exposed to the same conditions which made +their parents variable, +and the tendency to variability is in itself hereditary, consequently +they will tend to vary, and generally to vary in nearly the same manner +as their parents varied. Moreover, these two varieties, being only +slightly modified forms, will tend to inherit those advantages which +made their common parent (A) more numerous than most of the other +inhabitants of the same country; they will likewise partake of those +more general advantages which made the genus to which the +parent-species belonged, a large genus in its own country. And these +circumstances we know to be favourable to the production of new +varieties. + +If, then, these two varieties be variable, the most divergent of their +variations will generally be preserved during the next thousand +generations. And after this interval, variety _a_1 is supposed in the +diagram to have produced variety _a_2, which will, owing to the +principle of divergence, differ more from (A) than did variety _a_1. +Variety _m_1 is supposed to have produced two varieties, namely _m_2 +and _s_2, differing from each other, and more considerably from their +common parent (A). We may continue the process by similar steps for any +length of time; some of the varieties, after each thousand generations, +producing only a single variety, but in a more and more modified +condition, some producing two or three varieties, and some failing to +produce any. Thus the varieties or modified descendants, proceeding +from the common parent (A), will generally go on increasing in number +and diverging in character. In the diagram the process is represented +up to the ten-thousandth generation, and under a condensed and +simplified form up to the fourteen-thousandth generation. + +But I must here remark that I do not suppose that the process ever goes +on so regularly as is represented in the diagram, though in itself made +somewhat irregular. +I am far from thinking that the most divergent varieties will +invariably prevail and multiply: a medium form may often long endure, +and may or may not produce more than one modified descendant; for +natural selection will always act according to the nature of the places +which are either unoccupied or not perfectly occupied by other beings; +and this will depend on infinitely complex relations. But as a general +rule, the more diversified in structure the descendants from any one +species can be rendered, the more places they will be enabled to seize +on, and the more their modified progeny will be increased. In our +diagram the line of succession is broken at regular intervals by small +numbered letters marking the successive forms which have become +sufficiently distinct to be recorded as varieties. But these breaks are +imaginary, and might have been inserted anywhere, after intervals long +enough to have allowed the accumulation of a considerable amount of +divergent variation. + +As all the modified descendants from a common and widely-diffused +species, belonging to a large genus, will tend to partake of the same +advantages which made their parent successful in life, they will +generally go on multiplying in number as well as diverging in +character: this is represented in the diagram by the several divergent +branches proceeding from (A). The modified offspring from the later and +more highly improved branches in the lines of descent, will, it is +probable, often take the place of, and so destroy, the earlier and less +improved branches: this is represented in the diagram by some of the +lower branches not reaching to the upper horizontal lines. In some +cases I do not doubt that the process of modification will be confined +to a single line of descent, and the number of the descendants will not +be increased; although the amount +of divergent modification may have been increased in the successive +generations. This case would be represented in the diagram, if all the +lines proceeding from (A) were removed, excepting that from _a_1 to +_a_10. In the same way, for instance, the English race-horse and +English pointer have apparently both gone on slowly diverging in +character from their original stocks, without either having given off +any fresh branches or races. + +After ten thousand generations, species (A) is supposed to have +produced three forms, _a_10, _f_10, and _m_10, which, from having +diverged in character during the successive generations, will have come +to differ largely, but perhaps unequally, from each other and from +their common parent. If we suppose the amount of change between each +horizontal line in our diagram to be excessively small, these three +forms may still be only well-marked varieties; or they may have arrived +at the doubtful category of sub-species; but we have only to suppose +the steps in the process of modification to be more numerous or greater +in amount, to convert these three forms into well-defined species: thus +the diagram illustrates the steps by which the small differences +distinguishing varieties are increased into the larger differences +distinguishing species. By continuing the same process for a greater +number of generations (as shown in the diagram in a condensed and +simplified manner), we get eight species, marked by the letters between +_a_14 and _m_14, all descended from (A). Thus, as I believe, species +are multiplied and genera are formed. + +In a large genus it is probable that more than one species would vary. +In the diagram I have assumed that a second species (I) has produced, +by analogous steps, after ten thousand generations, either two +well-marked varieties (_w_10 and _z_10) or two species, according to +the amount of change supposed to be represented between +the horizontal lines. After fourteen thousand generations, six new +species, marked by the letters _n_14 to _z_14, are supposed to have +been produced. In each genus, the species, which are already extremely +different in character, will generally tend to produce the greatest +number of modified descendants; for these will have the best chance of +filling new and widely different places in the polity of nature: hence +in the diagram I have chosen the extreme species (A), and the nearly +extreme species (I), as those which have largely varied, and have given +rise to new varieties and species. The other nine species (marked by +capital letters) of our original genus, may for a long period continue +transmitting unaltered descendants; and this is shown in the diagram by +the dotted lines not prolonged far upwards from want of space. + +But during the process of modification, represented in the diagram, +another of our principles, namely that of extinction, will have played +an important part. As in each fully stocked country natural selection +necessarily acts by the selected form having some advantage in the +struggle for life over other forms, there will be a constant tendency +in the improved descendants of any one species to supplant and +exterminate in each stage of descent their predecessors and their +original parent. For it should be remembered that the competition will +generally be most severe between those forms which are most nearly +related to each other in habits, constitution, and structure. Hence all +the intermediate forms between the earlier and later states, that is +between the less and more improved state of a species, as well as the +original parent-species itself, will generally tend to become extinct. +So it probably will be with many whole collateral lines of descent, +which will be conquered by later and improved lines of descent. If, +however, the +modified offspring of a species get into some distinct country, or +become quickly adapted to some quite new station, in which child and +parent do not come into competition, both may continue to exist. + +If then our diagram be assumed to represent a considerable amount of +modification, species (A) and all the earlier varieties will have +become extinct, having been replaced by eight new species (_a_14 to +_m_14); and (I) will have been replaced by six (_n_14 to _z_14) new +species. + +But we may go further than this. The original species of our genus were +supposed to resemble each other in unequal degrees, as is so generally +the case in nature; species (A) being more nearly related to B, C, and +D, than to the other species; and species (I) more to G, H, K, L, than +to the others. These two species (A) and (I), were also supposed to be +very common and widely diffused species, so that they must originally +have had some advantage over most of the other species of the genus. +Their modified descendants, fourteen in number at the +fourteen-thousandth generation, will probably have inherited some of +the same advantages: they have also been modified and improved in a +diversified manner at each stage of descent, so as to have become +adapted to many related places in the natural economy of their country. +It seems, therefore, to me extremely probable that they will have taken +the places of, and thus exterminated, not only their parents (A) and +(I), but likewise some of the original species which were most nearly +related to their parents. Hence very few of the original species will +have transmitted offspring to the fourteen-thousandth generation. We +may suppose that only one (F), of the two species which were least +closely related to the other nine original species, has transmitted +descendants to this late stage of descent. + +The new species in our diagram descended from the original eleven +species, will now be fifteen in number. Owing to the divergent tendency +of natural selection, the extreme amount of difference in character +between species _a_14 and _z_14 will be much greater than that between +the most different of the original eleven species. The new species, +moreover, will be allied to each other in a widely different manner. Of +the eight descendants from (A) the three marked _a_14, _q_14, _p_14, +will be nearly related from having recently branched off from _a_10; +_b_14 and _f_14, from having diverged at an earlier period from _a_5, +will be in some degree distinct from the three first-named species; and +lastly, _o_14, _e_14, and _m_14, will be nearly related one to the +other, but from having diverged at the first commencement of the +process of modification, will be widely different from the other five +species, and may constitute a sub-genus or even a distinct genus. + +The six descendants from (I) will form two sub-genera or even genera. +But as the original species (I) differed largely from (A), standing +nearly at the extreme points of the original genus, the six descendants +from (I) will, owing to inheritance, differ considerably from the eight +descendants from (A); the two groups, moreover, are supposed to have +gone on diverging in different directions. The intermediate species, +also (and this is a very important consideration), which connected the +original species (A) and (I), have all become, excepting (F), extinct, +and have left no descendants. Hence the six new species descended from +(I), and the eight descended from (A), will have to be ranked as very +distinct genera, or even as distinct sub-families. + +Thus it is, as I believe, that two or more genera are produced by +descent, with modification, from two or more species of the same genus. +And the two or more +parent-species are supposed to have descended from some one species of +an earlier genus. In our diagram, this is indicated by the broken +lines, beneath the capital letters, converging in sub-branches +downwards towards a single point; this point representing a single +species, the supposed single parent of our several new sub-genera and +genera. + +It is worth while to reflect for a moment on the character of the new +species F14, which is supposed not to have diverged much in character, +but to have retained the form of (F), either unaltered or altered only +in a slight degree. In this case, its affinities to the other fourteen +new species will be of a curious and circuitous nature. Having +descended from a form which stood between the two parent-species (A) +and (I), now supposed to be extinct and unknown, it will be in some +degree intermediate in character between the two groups descended from +these species. But as these two groups have gone on diverging in +character from the type of their parents, the new species (F14) will +not be directly intermediate between them, but rather between types of +the two groups; and every naturalist will be able to bring some such +case before his mind. + +In the diagram, each horizontal line has hitherto been supposed to +represent a thousand generations, but each may represent a million or +hundred million generations, and likewise a section of the successive +strata of the earth’s crust including extinct remains. We shall, when +we come to our chapter on Geology, have to refer again to this subject, +and I think we shall then see that the diagram throws light on the +affinities of extinct beings, which, though generally belonging to the +same orders, or families, or genera, with those now living, yet are +often, in some degree, intermediate in character between existing +groups; and we can understand this fact, for +the extinct species lived at very ancient epochs when the branching +lines of descent had diverged less. + +I see no reason to limit the process of modification, as now explained, +to the formation of genera alone. If, in our diagram, we suppose the +amount of change represented by each successive group of diverging +dotted lines to be very great, the forms marked _a_14 to _p_14, those +marked _b_14 and _f_14, and those marked _o_14 to _m_14, will form +three very distinct genera. We shall also have two very distinct genera +descended from (I) and as these latter two genera, both from continued +divergence of character and from inheritance from a different parent, +will differ widely from the three genera descended from (A), the two +little groups of genera will form two distinct families, or even +orders, according to the amount of divergent modification supposed to +be represented in the diagram. And the two new families, or orders, +will have descended from two species of the original genus; and these +two species are supposed to have descended from one species of a still +more ancient and unknown genus. + +We have seen that in each country it is the species of the larger +genera which oftenest present varieties or incipient species. This, +indeed, might have been expected; for as natural selection acts through +one form having some advantage over other forms in the struggle for +existence, it will chiefly act on those which already have some +advantage; and the largeness of any group shows that its species have +inherited from a common ancestor some advantage in common. Hence, the +struggle for the production of new and modified descendants, will +mainly lie between the larger groups, which are all trying to increase +in number. One large group will slowly conquer another large group, +reduce its numbers, and thus lessen its chance of further variation and +improvement. Within the same large +group, the later and more highly perfected sub-groups, from branching +out and seizing on many new places in the polity of Nature, will +constantly tend to supplant and destroy the earlier and less improved +sub-groups. Small and broken groups and sub-groups will finally tend to +disappear. Looking to the future, we can predict that the groups of +organic beings which are now large and triumphant, and which are least +broken up, that is, which as yet have suffered least extinction, will +for a long period continue to increase. But which groups will +ultimately prevail, no man can predict; for we well know that many +groups, formerly most extensively developed, have now become extinct. +Looking still more remotely to the future, we may predict that, owing +to the continued and steady increase of the larger groups, a multitude +of smaller groups will become utterly extinct, and leave no modified +descendants; and consequently that of the species living at any one +period, extremely few will transmit descendants to a remote futurity. I +shall have to return to this subject in the chapter on Classification, +but I may add that on this view of extremely few of the more ancient +species having transmitted descendants, and on the view of all the +descendants of the same species making a class, we can understand how +it is that there exist but very few classes in each main division of +the animal and vegetable kingdoms. Although extremely few of the most +ancient species may now have living and modified descendants, yet at +the most remote geological period, the earth may have been as well +peopled with many species of many genera, families, orders, and +classes, as at the present day. + +_Summary of the Chapter_.—If during the long course of ages and under +varying conditions of life, organic beings +vary at all in the several parts of their organisation, and I think +this cannot be disputed; if there be, owing to the high geometrical +powers of increase of each species, at some age, season, or year, a +severe struggle for life, and this certainly cannot be disputed; then, +considering the infinite complexity of the relations of all organic +beings to each other and to their conditions of existence, causing an +infinite diversity in structure, constitution, and habits, to be +advantageous to them, I think it would be a most extraordinary fact if +no variation ever had occurred useful to each being’s own welfare, in +the same way as so many variations have occurred useful to man. But if +variations useful to any organic being do occur, assuredly individuals +thus characterised will have the best chance of being preserved in the +struggle for life; and from the strong principle of inheritance they +will tend to produce offspring similarly characterised. This principle +of preservation, I have called, for the sake of brevity, Natural +Selection. Natural selection, on the principle of qualities being +inherited at corresponding ages, can modify the egg, seed, or young, as +easily as the adult. Amongst many animals, sexual selection will give +its aid to ordinary selection, by assuring to the most vigorous and +best adapted males the greatest number of offspring. Sexual selection +will also give characters useful to the males alone, in their struggles +with other males. + +Whether natural selection has really thus acted in nature, in modifying +and adapting the various forms of life to their several conditions and +stations, must be judged of by the general tenour and balance of +evidence given in the following chapters. But we already see how it +entails extinction; and how largely extinction has acted in the world’s +history, geology plainly declares. Natural selection, also, leads to +divergence of +character; for more living beings can be supported on the same area the +more they diverge in structure, habits, and constitution, of which we +see proof by looking at the inhabitants of any small spot or at +naturalised productions. Therefore during the modification of the +descendants of any one species, and during the incessant struggle of +all species to increase in numbers, the more diversified these +descendants become, the better will be their chance of succeeding in +the battle of life. Thus the small differences distinguishing varieties +of the same species, will steadily tend to increase till they come to +equal the greater differences between species of the same genus, or +even of distinct genera. + +We have seen that it is the common, the widely-diffused, and +widely-ranging species, belonging to the larger genera, which vary +most; and these will tend to transmit to their modified offspring that +superiority which now makes them dominant in their own countries. +Natural selection, as has just been remarked, leads to divergence of +character and to much extinction of the less improved and intermediate +forms of life. On these principles, I believe, the nature of the +affinities of all organic beings may be explained. It is a truly +wonderful fact—the wonder of which we are apt to overlook from +familiarity—that all animals and all plants throughout all time and +space should be related to each other in group subordinate to group, in +the manner which we everywhere behold—namely, varieties of the same +species most closely related together, species of the same genus less +closely and unequally related together, forming sections and +sub-genera, species of distinct genera much less closely related, and +genera related in different degrees, forming sub-families, families, +orders, sub-classes, and classes. The several subordinate groups in any +class cannot be +ranked in a single file, but seem rather to be clustered round points, +and these round other points, and so on in almost endless cycles. On +the view that each species has been independently created, I can see no +explanation of this great fact in the classification of all organic +beings; but, to the best of my judgment, it is explained through +inheritance and the complex action of natural selection, entailing +extinction and divergence of character, as we have seen illustrated in +the diagram. + +The affinities of all the beings of the same class have sometimes been +represented by a great tree. I believe this simile largely speaks the +truth. The green and budding twigs may represent existing species; and +those produced during each former year may represent the long +succession of extinct species. At each period of growth all the growing +twigs have tried to branch out on all sides, and to overtop and kill +the surrounding twigs and branches, in the same manner as species and +groups of species have tried to overmaster other species in the great +battle for life. The limbs divided into great branches, and these into +lesser and lesser branches, were themselves once, when the tree was +small, budding twigs; and this connexion of the former and present buds +by ramifying branches may well represent the classification of all +extinct and living species in groups subordinate to groups. Of the many +twigs which flourished when the tree was a mere bush, only two or +three, now grown into great branches, yet survive and bear all the +other branches; so with the species which lived during long-past +geological periods, very few now have living and modified descendants. +From the first growth of the tree, many a limb and branch has decayed +and dropped off; and these lost branches of various sizes may represent +those whole orders, families, and genera which have now no living +representatives, and +which are known to us only from having been found in a fossil state. As +we here and there see a thin straggling branch springing from a fork +low down in a tree, and which by some chance has been favoured and is +still alive on its summit, so we occasionally see an animal like the +Ornithorhynchus or Lepidosiren, which in some small degree connects by +its affinities two large branches of life, and which has apparently +been saved from fatal competition by having inhabited a protected +station. As buds give rise by growth to fresh buds, and these, if +vigorous, branch out and overtop on all sides many a feebler branch, so +by generation I believe it has been with the great Tree of Life, which +fills with its dead and broken branches the crust of the earth, and +covers the surface with its ever branching and beautiful ramifications. + +Effects of external conditions. Use and disuse, combined with natural +selection; organs of flight and of vision. Acclimatisation. Correlation +of growth. Compensation and economy of growth. False correlations. +Multiple, rudimentary, and lowly organised structures variable. Parts +developed in an unusual manner are highly variable: specific characters +more variable than generic: secondary sexual characters variable. +Species of the same genus vary in an analogous manner. Reversions to +long lost characters. Summary. + +I have hitherto sometimes spoken as if the variations—so common and +multiform in organic beings under domestication, and in a lesser degree +in those in a state of nature—had been due to chance. This, of course, +is a wholly incorrect expression, but it serves to acknowledge plainly +our ignorance of the cause of each particular variation. Some authors +believe it to be as much the function of the reproductive system to +produce individual differences, or very slight deviations of structure, +as to make the child like its parents. But the much greater +variability, as well as the greater frequency of monstrosities, under +domestication or cultivation, than under nature, leads me to believe +that deviations of structure are in some way due to the nature of the +conditions of life, to which the parents and their more remote +ancestors have been exposed during several generations. I have remarked +in the first chapter—but a long catalogue of facts which cannot be here +given would be necessary to show the truth of the remark—that the +reproductive system is eminently susceptible to changes in the +conditions of life; and to +this system being functionally disturbed in the parents, I chiefly +attribute the varying or plastic condition of the offspring. The male +and female sexual elements seem to be affected before that union takes +place which is to form a new being. In the case of “sporting” plants, +the bud, which in its earliest condition does not apparently differ +essentially from an ovule, is alone affected. But why, because the +reproductive system is disturbed, this or that part should vary more or +less, we are profoundly ignorant. Nevertheless, we can here and there +dimly catch a faint ray of light, and we may feel sure that there must +be some cause for each deviation of structure, however slight. + +How much direct effect difference of climate, food, etc., produces on +any being is extremely doubtful. My impression is, that the effect is +extremely small in the case of animals, but perhaps rather more in that +of plants. We may, at least, safely conclude that such influences +cannot have produced the many striking and complex co-adaptations of +structure between one organic being and another, which we see +everywhere throughout nature. Some little influence may be attributed +to climate, food, etc.: thus, E. Forbes speaks confidently that shells +at their southern limit, and when living in shallow water, are more +brightly coloured than those of the same species further north or from +greater depths. Gould believes that birds of the same species are more +brightly coloured under a clear atmosphere, than when living on islands +or near the coast. So with insects, Wollaston is convinced that +residence near the sea affects their colours. Moquin-Tandon gives a +list of plants which when growing near the sea-shore have their leaves +in some degree fleshy, though not elsewhere fleshy. Several other such +cases could be given. + +The fact of varieties of one species, when they range +into the zone of habitation of other species, often acquiring in a very +slight degree some of the characters of such species, accords with our +view that species of all kinds are only well-marked and permanent +varieties. Thus the species of shells which are confined to tropical +and shallow seas are generally brighter-coloured than those confined to +cold and deeper seas. The birds which are confined to continents are, +according to Mr. Gould, brighter-coloured than those of islands. The +insect-species confined to sea-coasts, as every collector knows, are +often brassy or lurid. Plants which live exclusively on the sea-side +are very apt to have fleshy leaves. He who believes in the creation of +each species, will have to say that this shell, for instance, was +created with bright colours for a warm sea; but that this other shell +became bright-coloured by variation when it ranged into warmer or +shallower waters. + +When a variation is of the slightest use to a being, we cannot tell how +much of it to attribute to the accumulative action of natural +selection, and how much to the conditions of life. Thus, it is well +known to furriers that animals of the same species have thicker and +better fur the more severe the climate is under which they have lived; +but who can tell how much of this difference may be due to the +warmest-clad individuals having been favoured and preserved during many +generations, and how much to the direct action of the severe climate? +for it would appear that climate has some direct action on the hair of +our domestic quadrupeds. + +Instances could be given of the same variety being produced under +conditions of life as different as can well be conceived; and, on the +other hand, of different varieties being produced from the same species +under the same conditions. Such facts show how indirectly the +conditions of life must act. Again, innumerable instances are known to +every naturalist of species keeping true, or not varying at all, +although living under the most opposite climates. Such considerations +as these incline me to lay very little weight on the direct action of +the conditions of life. Indirectly, as already remarked, they seem to +play an important part in affecting the reproductive system, and in +thus inducing variability; and natural selection will then accumulate +all profitable variations, however slight, until they become plainly +developed and appreciable by us. + +_Effects of Use and Disuse_.—From the facts alluded to in the first +chapter, I think there can be little doubt that use in our domestic +animals strengthens and enlarges certain parts, and disuse diminishes +them; and that such modifications are inherited. Under free nature, we +can have no standard of comparison, by which to judge of the effects of +long-continued use or disuse, for we know not the parent-forms; but +many animals have structures which can be explained by the effects of +disuse. As Professor Owen has remarked, there is no greater anomaly in +nature than a bird that cannot fly; yet there are several in this +state. The logger-headed duck of South America can only flap along the +surface of the water, and has its wings in nearly the same condition as +the domestic Aylesbury duck. As the larger ground-feeding birds seldom +take flight except to escape danger, I believe that the nearly wingless +condition of several birds, which now inhabit or have lately inhabited +several oceanic islands, tenanted by no beast of prey, has been caused +by disuse. The ostrich indeed inhabits continents and is exposed to +danger from which it cannot escape by flight, but by kicking it can +defend itself from enemies, as well as any of the smaller +quadrupeds. We may imagine that the early progenitor of the ostrich had +habits like those of a bustard, and that as natural selection increased +in successive generations the size and weight of its body, its legs +were used more, and its wings less, until they became incapable of +flight. + +Kirby has remarked (and I have observed the same fact) that the +anterior tarsi, or feet, of many male dung-feeding beetles are very +often broken off; he examined seventeen specimens in his own +collection, and not one had even a relic left. In the Onites apelles +the tarsi are so habitually lost, that the insect has been described as +not having them. In some other genera they are present, but in a +rudimentary condition. In the Ateuchus or sacred beetle of the +Egyptians, they are totally deficient. There is not sufficient evidence +to induce us to believe that mutilations are ever inherited; and I +should prefer explaining the entire absence of the anterior tarsi in +Ateuchus, and their rudimentary condition in some other genera, by the +long-continued effects of disuse in their progenitors; for as the tarsi +are almost always lost in many dung-feeding beetles, they must be lost +early in life, and therefore cannot be much used by these insects. + +In some cases we might easily put down to disuse modifications of +structure which are wholly, or mainly, due to natural selection. Mr. +Wollaston has discovered the remarkable fact that 200 beetles, out of +the 550 species inhabiting Madeira, are so far deficient in wings that +they cannot fly; and that of the twenty-nine endemic genera, no less +than twenty-three genera have all their species in this condition! +Several facts, namely, that beetles in many parts of the world are very +frequently blown to sea and perish; that the beetles in Madeira, as +observed by Mr. Wollaston, lie much concealed, +until the wind lulls and the sun shines; that the proportion of +wingless beetles is larger on the exposed Dezertas than in Madeira +itself; and especially the extraordinary fact, so strongly insisted on +by Mr. Wollaston, of the almost entire absence of certain large groups +of beetles, elsewhere excessively numerous, and which groups have +habits of life almost necessitating frequent flight;—these several +considerations have made me believe that the wingless condition of so +many Madeira beetles is mainly due to the action of natural selection, +but combined probably with disuse. For during thousands of successive +generations each individual beetle which flew least, either from its +wings having been ever so little less perfectly developed or from +indolent habit, will have had the best chance of surviving from not +being blown out to sea; and, on the other hand, those beetles which +most readily took to flight will oftenest have been blown to sea and +thus have been destroyed. + +The insects in Madeira which are not ground-feeders, and which, as the +flower-feeding coleoptera and lepidoptera, must habitually use their +wings to gain their subsistence, have, as Mr. Wollaston suspects, their +wings not at all reduced, but even enlarged. This is quite compatible +with the action of natural selection. For when a new insect first +arrived on the island, the tendency of natural selection to enlarge or +to reduce the wings, would depend on whether a greater number of +individuals were saved by successfully battling with the winds, or by +giving up the attempt and rarely or never flying. As with mariners +shipwrecked near a coast, it would have been better for the good +swimmers if they had been able to swim still further, whereas it would +have been better for the bad swimmers if they had not been able to swim +at all and had stuck to the wreck. + +The eyes of moles and of some burrowing rodents are rudimentary in +size, and in some cases are quite covered up by skin and fur. This +state of the eyes is probably due to gradual reduction from disuse, but +aided perhaps by natural selection. In South America, a burrowing +rodent, the tuco-tuco, or Ctenomys, is even more subterranean in its +habits than the mole; and I was assured by a Spaniard, who had often +caught them, that they were frequently blind; one which I kept alive +was certainly in this condition, the cause, as appeared on dissection, +having been inflammation of the nictitating membrane. As frequent +inflammation of the eyes must be injurious to any animal, and as eyes +are certainly not indispensable to animals with subterranean habits, a +reduction in their size with the adhesion of the eyelids and growth of +fur over them, might in such case be an advantage; and if so, natural +selection would constantly aid the effects of disuse. + +It is well known that several animals, belonging to the most different +classes, which inhabit the caves of Styria and of Kentucky, are blind. +In some of the crabs the foot-stalk for the eye remains, though the eye +is gone; the stand for the telescope is there, though the telescope +with its glasses has been lost. As it is difficult to imagine that +eyes, though useless, could be in any way injurious to animals living +in darkness, I attribute their loss wholly to disuse. In one of the +blind animals, namely, the cave-rat, the eyes are of immense size; and +Professor Silliman thought that it regained, after living some days in +the light, some slight power of vision. In the same manner as in +Madeira the wings of some of the insects have been enlarged, and the +wings of others have been reduced by natural selection aided by use and +disuse, so in the case of the cave-rat natural selection seems to have +struggled with the loss of light and +to have increased the size of the eyes; whereas with all the other +inhabitants of the caves, disuse by itself seems to have done its work. + +It is difficult to imagine conditions of life more similar than deep +limestone caverns under a nearly similar climate; so that on the common +view of the blind animals having been separately created for the +American and European caverns, close similarity in their organisation +and affinities might have been expected; but, as Schiödte and others +have remarked, this is not the case, and the cave-insects of the two +continents are not more closely allied than might have been anticipated +from the general resemblance of the other inhabitants of North America +and Europe. On my view we must suppose that American animals, having +ordinary powers of vision, slowly migrated by successive generations +from the outer world into the deeper and deeper recesses of the +Kentucky caves, as did European animals into the caves of Europe. We +have some evidence of this gradation of habit; for, as Schiödte +remarks, “animals not far remote from ordinary forms, prepare the +transition from light to darkness. Next follow those that are +constructed for twilight; and, last of all, those destined for total +darkness.” By the time that an animal had reached, after numberless +generations, the deepest recesses, disuse will on this view have more +or less perfectly obliterated its eyes, and natural selection will +often have effected other changes, such as an increase in the length of +the antennæ or palpi, as a compensation for blindness. Notwithstanding +such modifications, we might expect still to see in the cave-animals of +America, affinities to the other inhabitants of that continent, and in +those of Europe, to the inhabitants of the European continent. And this +is the case with some of the American cave-animals, as I hear from +Professor Dana; and some of the European cave-insects are very closely +allied to those of the surrounding country. It would be most difficult +to give any rational explanation of the affinities of the blind +cave-animals to the other inhabitants of the two continents on the +ordinary view of their independent creation. That several of the +inhabitants of the caves of the Old and New Worlds should be closely +related, we might expect from the well-known relationship of most of +their other productions. Far from feeling any surprise that some of the +cave-animals should be very anomalous, as Agassiz has remarked in +regard to the blind fish, the Amblyopsis, and as is the case with the +blind Proteus with reference to the reptiles of Europe, I am only +surprised that more wrecks of ancient life have not been preserved, +owing to the less severe competition to which the inhabitants of these +dark abodes will probably have been exposed. + +_Acclimatisation_.—Habit is hereditary with plants, as in the period of +flowering, in the amount of rain requisite for seeds to germinate, in +the time of sleep, etc., and this leads me to say a few words on +acclimatisation. As it is extremely common for species of the same +genus to inhabit very hot and very cold countries, and as I believe +that all the species of the same genus have descended from a single +parent, if this view be correct, acclimatisation must be readily +effected during long-continued descent. It is notorious that each +species is adapted to the climate of its own home: species from an +arctic or even from a temperate region cannot endure a tropical +climate, or conversely. So again, many succulent plants cannot endure a +damp climate. But the degree of adaptation of species to the climates +under which they live is often overrated. +We may infer this from our frequent inability to predict whether or not +an imported plant will endure our climate, and from the number of +plants and animals brought from warmer countries which here enjoy good +health. We have reason to believe that species in a state of nature are +limited in their ranges by the competition of other organic beings +quite as much as, or more than, by adaptation to particular climates. +But whether or not the adaptation be generally very close, we have +evidence, in the case of some few plants, of their becoming, to a +certain extent, naturally habituated to different temperatures, or +becoming acclimatised: thus the pines and rhododendrons, raised from +seed collected by Dr. Hooker from trees growing at different heights on +the Himalaya, were found in this country to possess different +constitutional powers of resisting cold. Mr. Thwaites informs me that +he has observed similar facts in Ceylon, and analogous observations +have been made by Mr. H. C. Watson on European species of plants +brought from the Azores to England. In regard to animals, several +authentic cases could be given of species within historical times +having largely extended their range from warmer to cooler latitudes, +and conversely; but we do not positively know that these animals were +strictly adapted to their native climate, but in all ordinary cases we +assume such to be the case; nor do we know that they have subsequently +become acclimatised to their new homes. + +As I believe that our domestic animals were originally chosen by +uncivilised man because they were useful and bred readily under +confinement, and not because they were subsequently found capable of +far-extended transportation, I think the common and extraordinary +capacity in our domestic animals of not only withstanding the most +different climates but of being perfectly +fertile (a far severer test) under them, may be used as an argument +that a large proportion of other animals, now in a state of nature, +could easily be brought to bear widely different climates. We must not, +however, push the foregoing argument too far, on account of the +probable origin of some of our domestic animals from several wild +stocks: the blood, for instance, of a tropical and arctic wolf or wild +dog may perhaps be mingled in our domestic breeds. The rat and mouse +cannot be considered as domestic animals, but they have been +transported by man to many parts of the world, and now have a far wider +range than any other rodent, living free under the cold climate of +Faroe in the north and of the Falklands in the south, and on many +islands in the torrid zones. Hence I am inclined to look at adaptation +to any special climate as a quality readily grafted on an innate wide +flexibility of constitution, which is common to most animals. On this +view, the capacity of enduring the most different climates by man +himself and by his domestic animals, and such facts as that former +species of the elephant and rhinoceros were capable of enduring a +glacial climate, whereas the living species are now all tropical or +sub-tropical in their habits, ought not to be looked at as anomalies, +but merely as examples of a very common flexibility of constitution, +brought, under peculiar circumstances, into play. + +How much of the acclimatisation of species to any peculiar climate is +due to mere habit, and how much to the natural selection of varieties +having different innate constitutions, and how much to both means +combined, is a very obscure question. That habit or custom has some +influence I must believe, both from analogy, and from the incessant +advice given in agricultural works, even in the ancient Encyclopædias +of China, to be very cautious +in transposing animals from one district to another; for it is not +likely that man should have succeeded in selecting so many breeds and +sub-breeds with constitutions specially fitted for their own districts: +the result must, I think, be due to habit. On the other hand, I can see +no reason to doubt that natural selection will continually tend to +preserve those individuals which are born with constitutions best +adapted to their native countries. In treatises on many kinds of +cultivated plants, certain varieties are said to withstand certain +climates better than others: this is very strikingly shown in works on +fruit trees published in the United States, in which certain varieties +are habitually recommended for the northern, and others for the +southern States; and as most of these varieties are of recent origin, +they cannot owe their constitutional differences to habit. The case of +the Jerusalem artichoke, which is never propagated by seed, and of +which consequently new varieties have not been produced, has even been +advanced—for it is now as tender as ever it was—as proving that +acclimatisation cannot be effected! The case, also, of the kidney-bean +has been often cited for a similar purpose, and with much greater +weight; but until some one will sow, during a score of generations, his +kidney-beans so early that a very large proportion are destroyed by +frost, and then collect seed from the few survivors, with care to +prevent accidental crosses, and then again get seed from these +seedlings, with the same precautions, the experiment cannot be said to +have been even tried. Nor let it be supposed that no differences in the +constitution of seedling kidney-beans ever appear, for an account has +been published how much more hardy some seedlings appeared to be than +others. + +On the whole, I think we may conclude that habit, +use, and disuse, have, in some cases, played a considerable part in the +modification of the constitution, and of the structure of various +organs; but that the effects of use and disuse have often been largely +combined with, and sometimes overmastered by, the natural selection of +innate differences. + +_Correlation of Growth_.—I mean by this expression that the whole +organisation is so tied together during its growth and development, +that when slight variations in any one part occur, and are accumulated +through natural selection, other parts become modified. This is a very +important subject, most imperfectly understood. The most obvious case +is, that modifications accumulated solely for the good of the young or +larva, will, it may safely be concluded, affect the structure of the +adult; in the same manner as any malconformation affecting the early +embryo, seriously affects the whole organisation of the adult. The +several parts of the body which are homologous, and which, at an early +embryonic period, are alike, seem liable to vary in an allied manner: +we see this in the right and left sides of the body varying in the same +manner; in the front and hind legs, and even in the jaws and limbs, +varying together, for the lower jaw is believed to be homologous with +the limbs. These tendencies, I do not doubt, may be mastered more or +less completely by natural selection: thus a family of stags once +existed with an antler only on one side; and if this had been of any +great use to the breed it might probably have been rendered permanent +by natural selection. + +Homologous parts, as has been remarked by some authors, tend to cohere; +this is often seen in monstrous plants; and nothing is more common than +the union of homologous parts in normal structures, as the union of +the petals of the corolla into a tube. Hard parts seem to affect the +form of adjoining soft parts; it is believed by some authors that the +diversity in the shape of the pelvis in birds causes the remarkable +diversity in the shape of their kidneys. Others believe that the shape +of the pelvis in the human mother influences by pressure the shape of +the head of the child. In snakes, according to Schlegel, the shape of +the body and the manner of swallowing determine the position of several +of the most important viscera. + +The nature of the bond of correlation is very frequently quite obscure. +M. Is. Geoffroy St. Hilaire has forcibly remarked, that certain +malconformations very frequently, and that others rarely coexist, +without our being able to assign any reason. What can be more singular +than the relation between blue eyes and deafness in cats, and the +tortoise-shell colour with the female sex; the feathered feet and skin +between the outer toes in pigeons, and the presence of more or less +down on the young birds when first hatched, with the future colour of +their plumage; or, again, the relation between the hair and teeth in +the naked Turkish dog, though here probably homology comes into play? +With respect to this latter case of correlation, I think it can hardly +be accidental, that if we pick out the two orders of mammalia which are +most abnormal in their dermal coverings, viz. Cetacea (whales) and +Edentata (armadilloes, scaly ant-eaters, etc.), that these are likewise +the most abnormal in their teeth. + +I know of no case better adapted to show the importance of the laws of +correlation in modifying important structures, independently of utility +and, therefore, of natural selection, than that of the difference +between the outer and inner flowers in some Compositous and +Umbelliferous plants. Every one knows the +difference in the ray and central florets of, for instance, the daisy, +and this difference is often accompanied with the abortion of parts of +the flower. But, in some Compositous plants, the seeds also differ in +shape and sculpture; and even the ovary itself, with its accessory +parts, differs, as has been described by Cassini. These differences +have been attributed by some authors to pressure, and the shape of the +seeds in the ray-florets in some Compositæ countenances this idea; but, +in the case of the corolla of the Umbelliferæ, it is by no means, as +Dr. Hooker informs me, in species with the densest heads that the inner +and outer flowers most frequently differ. It might have been thought +that the development of the ray-petals by drawing nourishment from +certain other parts of the flower had caused their abortion; but in +some Compositæ there is a difference in the seeds of the outer and +inner florets without any difference in the corolla. Possibly, these +several differences may be connected with some difference in the flow +of nutriment towards the central and external flowers: we know, at +least, that in irregular flowers, those nearest to the axis are +oftenest subject to peloria, and become regular. I may add, as an +instance of this, and of a striking case of correlation, that I have +recently observed in some garden pelargoniums, that the central flower +of the truss often loses the patches of darker colour in the two upper +petals; and that when this occurs, the adherent nectary is quite +aborted; when the colour is absent from only one of the two upper +petals, the nectary is only much shortened. + +With respect to the difference in the corolla of the central and +exterior flowers of a head or umbel, I do not feel at all sure that C. +C. Sprengel’s idea that the ray-florets serve to attract insects, whose +agency is highly advantageous in the fertilisation of plants of +these two orders, is so far-fetched, as it may at first appear: and if +it be advantageous, natural selection may have come into play. But in +regard to the differences both in the internal and external structure +of the seeds, which are not always correlated with any differences in +the flowers, it seems impossible that they can be in any way +advantageous to the plant: yet in the Umbelliferæ these differences are +of such apparent importance—the seeds being in some cases, according to +Tausch, orthospermous in the exterior flowers and coelospermous in the +central flowers,—that the elder De Candolle founded his main divisions +of the order on analogous differences. Hence we see that modifications +of structure, viewed by systematists as of high value, may be wholly +due to unknown laws of correlated growth, and without being, as far as +we can see, of the slightest service to the species. + +We may often falsely attribute to correlation of growth, structures +which are common to whole groups of species, and which in truth are +simply due to inheritance; for an ancient progenitor may have acquired +through natural selection some one modification in structure, and, +after thousands of generations, some other and independent +modification; and these two modifications, having been transmitted to a +whole group of descendants with diverse habits, would naturally be +thought to be correlated in some necessary manner. So, again, I do not +doubt that some apparent correlations, occurring throughout whole +orders, are entirely due to the manner alone in which natural selection +can act. For instance, Alph. De Candolle has remarked that winged seeds +are never found in fruits which do not open: I should explain the rule +by the fact that seeds could not gradually become winged through +natural selection, except in fruits which opened; so that the +individual plants producing +seeds which were a little better fitted to be wafted further, might get +an advantage over those producing seed less fitted for dispersal; and +this process could not possibly go on in fruit which did not open. + +The elder Geoffroy and Goethe propounded, at about the same period, +their law of compensation or balancement of growth; or, as Goethe +expressed it, “in order to spend on one side, nature is forced to +economise on the other side.” I think this holds true to a certain +extent with our domestic productions: if nourishment flows to one part +or organ in excess, it rarely flows, at least in excess, to another +part; thus it is difficult to get a cow to give much milk and to fatten +readily. The same varieties of the cabbage do not yield abundant and +nutritious foliage and a copious supply of oil-bearing seeds. When the +seeds in our fruits become atrophied, the fruit itself gains largely in +size and quality. In our poultry, a large tuft of feathers on the head +is generally accompanied by a diminished comb, and a large beard by +diminished wattles. With species in a state of nature it can hardly be +maintained that the law is of universal application; but many good +observers, more especially botanists, believe in its truth. I will not, +however, here give any instances, for I see hardly any way of +distinguishing between the effects, on the one hand, of a part being +largely developed through natural selection and another and adjoining +part being reduced by this same process or by disuse, and, on the other +hand, the actual withdrawal of nutriment from one part owing to the +excess of growth in another and adjoining part. + +I suspect, also, that some of the cases of compensation which have been +advanced, and likewise some other facts, may be merged under a more +general principle, namely, that natural selection is continually trying +to economise in every part of the organisation. If under +changed conditions of life a structure before useful becomes less +useful, any diminution, however slight, in its development, will be +seized on by natural selection, for it will profit the individual not +to have its nutriment wasted in building up an useless structure. I can +thus only understand a fact with which I was much struck when examining +cirripedes, and of which many other instances could be given: namely, +that when a cirripede is parasitic within another and is thus +protected, it loses more or less completely its own shell or carapace. +This is the case with the male Ibla, and in a truly extraordinary +manner with the Proteolepas: for the carapace in all other cirripedes +consists of the three highly-important anterior segments of the head +enormously developed, and furnished with great nerves and muscles; but +in the parasitic and protected Proteolepas, the whole anterior part of +the head is reduced to the merest rudiment attached to the bases of the +prehensile antennæ. Now the saving of a large and complex structure, +when rendered superfluous by the parasitic habits of the Proteolepas, +though effected by slow steps, would be a decided advantage to each +successive individual of the species; for in the struggle for life to +which every animal is exposed, each individual Proteolepas would have a +better chance of supporting itself, by less nutriment being wasted in +developing a structure now become useless. + +Thus, as I believe, natural selection will always succeed in the long +run in reducing and saving every part of the organisation, as soon as +it is rendered superfluous, without by any means causing some other +part to be largely developed in a corresponding degree. And, +conversely, that natural selection may perfectly well succeed in +largely developing any organ, without requiring as a necessary +compensation the reduction of some adjoining part. + +It seems to be a rule, as remarked by Is. Geoffroy St. Hilaire, both in +varieties and in species, that when any part or organ is repeated many +times in the structure of the same individual (as the vertebræ in +snakes, and the stamens in polyandrous flowers) the number is variable; +whereas the number of the same part or organ, when it occurs in lesser +numbers, is constant. The same author and some botanists have further +remarked that multiple parts are also very liable to variation in +structure. Inasmuch as this “vegetative repetition,” to use Professor +Owen’s expression, seems to be a sign of low organisation; the +foregoing remark seems connected with the very general opinion of +naturalists, that beings low in the scale of nature are more variable +than those which are higher. I presume that lowness in this case means +that the several parts of the organisation have been but little +specialised for particular functions; and as long as the same part has +to perform diversified work, we can perhaps see why it should remain +variable, that is, why natural selection should have preserved or +rejected each little deviation of form less carefully than when the +part has to serve for one special purpose alone. In the same way that a +knife which has to cut all sorts of things may be of almost any shape; +whilst a tool for some particular object had better be of some +particular shape. Natural selection, it should never be forgotten, can +act on each part of each being, solely through and for its advantage. + +Rudimentary parts, it has been stated by some authors, and I believe +with truth, are apt to be highly variable. We shall have to recur to +the general subject of rudimentary and aborted organs; and I will here +only add that their variability seems to be owing to their uselessness, +and therefore to natural selection having no power to check deviations +in their structure. Thus +rudimentary parts are left to the free play of the various laws of +growth, to the effects of long-continued disuse, and to the tendency to +reversion. + +_A part developed in any species in an extraordinary degree or manner, +in comparison with the same part in allied species, tends to be highly +variable_.—Several years ago I was much struck with a remark, nearly to +the above effect, published by Mr. Waterhouse. I infer also from an +observation made by Professor Owen, with respect to the length of the +arms of the ourang-outang, that he has come to a nearly similar +conclusion. It is hopeless to attempt to convince any one of the truth +of this proposition without giving the long array of facts which I have +collected, and which cannot possibly be here introduced. I can only +state my conviction that it is a rule of high generality. I am aware of +several causes of error, but I hope that I have made due allowance for +them. It should be understood that the rule by no means applies to any +part, however unusually developed, unless it be unusually developed in +comparison with the same part in closely allied species. Thus, the +bat’s wing is a most abnormal structure in the class mammalia; but the +rule would not here apply, because there is a whole group of bats +having wings; it would apply only if some one species of bat had its +wings developed in some remarkable manner in comparison with the other +species of the same genus. The rule applies very strongly in the case +of secondary sexual characters, when displayed in any unusual manner. +The term, secondary sexual characters, used by Hunter, applies to +characters which are attached to one sex, but are not directly +connected with the act of reproduction. The rule applies to males and +females; but as females more rarely offer remarkable secondary sexual +characters, it applies +more rarely to them. The rule being so plainly applicable in the case +of secondary sexual characters, may be due to the great variability of +these characters, whether or not displayed in any unusual manner—of +which fact I think there can be little doubt. But that our rule is not +confined to secondary sexual characters is clearly shown in the case of +hermaphrodite cirripedes; and I may here add, that I particularly +attended to Mr. Waterhouse’s remark, whilst investigating this Order, +and I am fully convinced that the rule almost invariably holds good +with cirripedes. I shall, in my future work, give a list of the more +remarkable cases; I will here only briefly give one, as it illustrates +the rule in its largest application. The opercular valves of sessile +cirripedes (rock barnacles) are, in every sense of the word, very +important structures, and they differ extremely little even in +different genera; but in the several species of one genus, Pyrgoma, +these valves present a marvellous amount of diversification: the +homologous valves in the different species being sometimes wholly +unlike in shape; and the amount of variation in the individuals of +several of the species is so great, that it is no exaggeration to state +that the varieties differ more from each other in the characters of +these important valves than do other species of distinct genera. + +As birds within the same country vary in a remarkably small degree, I +have particularly attended to them, and the rule seems to me certainly +to hold good in this class. I cannot make out that it applies to +plants, and this would seriously have shaken my belief in its truth, +had not the great variability in plants made it particularly difficult +to compare their relative degrees of variability. + +When we see any part or organ developed in a remarkable degree or +manner in any species, the fair +presumption is that it is of high importance to that species; +nevertheless the part in this case is eminently liable to variation. +Why should this be so? On the view that each species has been +independently created, with all its parts as we now see them, I can see +no explanation. But on the view that groups of species have descended +from other species, and have been modified through natural selection, I +think we can obtain some light. In our domestic animals, if any part, +or the whole animal, be neglected and no selection be applied, that +part (for instance, the comb in the Dorking fowl) or the whole breed +will cease to have a nearly uniform character. The breed will then be +said to have degenerated. In rudimentary organs, and in those which +have been but little specialised for any particular purpose, and +perhaps in polymorphic groups, we see a nearly parallel natural case; +for in such cases natural selection either has not or cannot come into +full play, and thus the organisation is left in a fluctuating +condition. But what here more especially concerns us is, that in our +domestic animals those points, which at the present time are undergoing +rapid change by continued selection, are also eminently liable to +variation. Look at the breeds of the pigeon; see what a prodigious +amount of difference there is in the beak of the different tumblers, in +the beak and wattle of the different carriers, in the carriage and tail +of our fantails, etc., these being the points now mainly attended to by +English fanciers. Even in the sub-breeds, as in the short-faced +tumbler, it is notoriously difficult to breed them nearly to +perfection, and frequently individuals are born which depart widely +from the standard. There may be truly said to be a constant struggle +going on between, on the one hand, the tendency to reversion to a less +modified state, as well as an innate tendency to further +variability of all kinds, and, on the other hand, the power of steady +selection to keep the breed true. In the long run selection gains the +day, and we do not expect to fail so far as to breed a bird as coarse +as a common tumbler from a good short-faced strain. But as long as +selection is rapidly going on, there may always be expected to be much +variability in the structure undergoing modification. It further +deserves notice that these variable characters, produced by man’s +selection, sometimes become attached, from causes quite unknown to us, +more to one sex than to the other, generally to the male sex, as with +the wattle of carriers and the enlarged crop of pouters. + +Now let us turn to nature. When a part has been developed in an +extraordinary manner in any one species, compared with the other +species of the same genus, we may conclude that this part has undergone +an extraordinary amount of modification, since the period when the +species branched off from the common progenitor of the genus. This +period will seldom be remote in any extreme degree, as species very +rarely endure for more than one geological period. An extraordinary +amount of modification implies an unusually large and long-continued +amount of variability, which has continually been accumulated by +natural selection for the benefit of the species. But as the +variability of the extraordinarily-developed part or organ has been so +great and long-continued within a period not excessively remote, we +might, as a general rule, expect still to find more variability in such +parts than in other parts of the organisation, which have remained for +a much longer period nearly constant. And this, I am convinced, is the +case. That the struggle between natural selection on the one hand, and +the tendency to reversion and variability on the other hand, will in +the +course of time cease; and that the most abnormally developed organs may +be made constant, I can see no reason to doubt. Hence when an organ, +however abnormal it may be, has been transmitted in approximately the +same condition to many modified descendants, as in the case of the wing +of the bat, it must have existed, according to my theory, for an +immense period in nearly the same state; and thus it comes to be no +more variable than any other structure. It is only in those cases in +which the modification has been comparatively recent and +extraordinarily great that we ought to find the _generative +variability_, as it may be called, still present in a high degree. For +in this case the variability will seldom as yet have been fixed by the +continued selection of the individuals varying in the required manner +and degree, and by the continued rejection of those tending to revert +to a former and less modified condition. + +The principle included in these remarks may be extended. It is +notorious that specific characters are more variable than generic. To +explain by a simple example what is meant. If some species in a large +genus of plants had blue flowers and some had red, the colour would be +only a specific character, and no one would be surprised at one of the +blue species varying into red, or conversely; but if all the species +had blue flowers, the colour would become a generic character, and its +variation would be a more unusual circumstance. I have chosen this +example because an explanation is not in this case applicable, which +most naturalists would advance, namely, that specific characters are +more variable than generic, because they are taken from parts of less +physiological importance than those commonly used for classing genera. +I believe this explanation is partly, yet only indirectly, true; I +shall, however, have to return +to this subject in our chapter on Classification. It would be almost +superfluous to adduce evidence in support of the above statement, that +specific characters are more variable than generic; but I have +repeatedly noticed in works on natural history, that when an author has +remarked with surprise that some _important_ organ or part, which is +generally very constant throughout large groups of species, has +_differed_ considerably in closely-allied species, that it has, also, +been _variable_ in the individuals of some of the species. And this +fact shows that a character, which is generally of generic value, when +it sinks in value and becomes only of specific value, often becomes +variable, though its physiological importance may remain the same. +Something of the same kind applies to monstrosities: at least Is. +Geoffroy St. Hilaire seems to entertain no doubt, that the more an +organ normally differs in the different species of the same group, the +more subject it is to individual anomalies. + +On the ordinary view of each species having been independently created, +why should that part of the structure, which differs from the same part +in other independently-created species of the same genus, be more +variable than those parts which are closely alike in the several +species? I do not see that any explanation can be given. But on the +view of species being only strongly marked and fixed varieties, we +might surely expect to find them still often continuing to vary in +those parts of their structure which have varied within a moderately +recent period, and which have thus come to differ. Or to state the case +in another manner:—the points in which all the species of a genus +resemble each other, and in which they differ from the species of some +other genus, are called generic characters; and these characters in +common I attribute to inheritance from a common +progenitor, for it can rarely have happened that natural selection will +have modified several species, fitted to more or less widely-different +habits, in exactly the same manner: and as these so-called generic +characters have been inherited from a remote period, since that period +when the species first branched off from their common progenitor, and +subsequently have not varied or come to differ in any degree, or only +in a slight degree, it is not probable that they should vary at the +present day. On the other hand, the points in which species differ from +other species of the same genus, are called specific characters; and as +these specific characters have varied and come to differ within the +period of the branching off of the species from a common progenitor, it +is probable that they should still often be in some degree variable,—at +least more variable than those parts of the organisation which have for +a very long period remained constant. + +In connexion with the present subject, I will make only two other +remarks. I think it will be admitted, without my entering on details, +that secondary sexual characters are very variable; I think it also +will be admitted that species of the same group differ from each other +more widely in their secondary sexual characters, than in other parts +of their organisation; compare, for instance, the amount of difference +between the males of gallinaceous birds, in which secondary sexual +characters are strongly displayed, with the amount of difference +between their females; and the truth of this proposition will be +granted. The cause of the original variability of secondary sexual +characters is not manifest; but we can see why these characters should +not have been rendered as constant and uniform as other parts of the +organisation; for secondary sexual characters have been accumulated by +sexual selection, which +is less rigid in its action than ordinary selection, as it does not +entail death, but only gives fewer offspring to the less favoured +males. Whatever the cause may be of the variability of secondary sexual +characters, as they are highly variable, sexual selection will have had +a wide scope for action, and may thus readily have succeeded in giving +to the species of the same group a greater amount of difference in +their sexual characters, than in other parts of their structure. + +It is a remarkable fact, that the secondary sexual differences between +the two sexes of the same species are generally displayed in the very +same parts of the organisation in which the different species of the +same genus differ from each other. Of this fact I will give in +illustration two instances, the first which happen to stand on my list; +and as the differences in these cases are of a very unusual nature, the +relation can hardly be accidental. The same number of joints in the +tarsi is a character generally common to very large groups of beetles, +but in the Engidæ, as Westwood has remarked, the number varies greatly; +and the number likewise differs in the two sexes of the same species: +again in fossorial hymenoptera, the manner of neuration of the wings is +a character of the highest importance, because common to large groups; +but in certain genera the neuration differs in the different species, +and likewise in the two sexes of the same species. This relation has a +clear meaning on my view of the subject: I look at all the species of +the same genus as having as certainly descended from the same +progenitor, as have the two sexes of any one of the species. +Consequently, whatever part of the structure of the common progenitor, +or of its early descendants, became variable; variations of this part +would it is highly probable, be taken advantage of by natural and +sexual selection, in +order to fit the several species to their several places in the economy +of nature, and likewise to fit the two sexes of the same species to +each other, or to fit the males and females to different habits of +life, or the males to struggle with other males for the possession of +the females. + +Finally, then, I conclude that the greater variability of specific +characters, or those which distinguish species from species, than of +generic characters, or those which the species possess in common;—that +the frequent extreme variability of any part which is developed in a +species in an extraordinary manner in comparison with the same part in +its congeners; and the not great degree of variability in a part, +however extraordinarily it may be developed, if it be common to a whole +group of species;—that the great variability of secondary sexual +characters, and the great amount of difference in these same characters +between closely allied species;—that secondary sexual and ordinary +specific differences are generally displayed in the same parts of the +organisation,—are all principles closely connected together. All being +mainly due to the species of the same group having descended from a +common progenitor, from whom they have inherited much in common,—to +parts which have recently and largely varied being more likely still to +go on varying than parts which have long been inherited and have not +varied,—to natural selection having more or less completely, according +to the lapse of time, overmastered the tendency to reversion and to +further variability,—to sexual selection being less rigid than ordinary +selection,—and to variations in the same parts having been accumulated +by natural and sexual selection, and thus adapted for secondary sexual, +and for ordinary specific purposes. + +_Distinct species present analogous variations; and a variety of one +species often assumes some of the characters of an allied species, or +reverts to some of the characters of an early progenitor_.—These +propositions will be most readily understood by looking to our domestic +races. The most distinct breeds of pigeons, in countries most widely +apart, present sub-varieties with reversed feathers on the head and +feathers on the feet,—characters not possessed by the aboriginal +rock-pigeon; these then are analogous variations in two or more +distinct races. The frequent presence of fourteen or even sixteen +tail-feathers in the pouter, may be considered as a variation +representing the normal structure of another race, the fantail. I +presume that no one will doubt that all such analogous variations are +due to the several races of the pigeon having inherited from a common +parent the same constitution and tendency to variation, when acted on +by similar unknown influences. In the vegetable kingdom we have a case +of analogous variation, in the enlarged stems, or roots as commonly +called, of the Swedish turnip and Ruta baga, plants which several +botanists rank as varieties produced by cultivation from a common +parent: if this be not so, the case will then be one of analogous +variation in two so-called distinct species; and to these a third may +be added, namely, the common turnip. According to the ordinary view of +each species having been independently created, we should have to +attribute this similarity in the enlarged stems of these three plants, +not to the vera causa of community of descent, and a consequent +tendency to vary in a like manner, but to three separate yet closely +related acts of creation. + +With pigeons, however, we have another case, namely, the occasional +appearance in all the breeds, of slaty-blue birds with two black bars +on the wings, a white +rump, a bar at the end of the tail, with the outer feathers externally +edged near their bases with white. As all these marks are +characteristic of the parent rock-pigeon, I presume that no one will +doubt that this is a case of reversion, and not of a new yet analogous +variation appearing in the several breeds. We may I think confidently +come to this conclusion, because, as we have seen, these coloured marks +are eminently liable to appear in the crossed offspring of two distinct +and differently coloured breeds; and in this case there is nothing in +the external conditions of life to cause the reappearance of the +slaty-blue, with the several marks, beyond the influence of the mere +act of crossing on the laws of inheritance. + +No doubt it is a very surprising fact that characters should reappear +after having been lost for many, perhaps for hundreds of generations. +But when a breed has been crossed only once by some other breed, the +offspring occasionally show a tendency to revert in character to the +foreign breed for many generations—some say, for a dozen or even a +score of generations. After twelve generations, the proportion of +blood, to use a common expression, of any one ancestor, is only 1 in +2048; and yet, as we see, it is generally believed that a tendency to +reversion is retained by this very small proportion of foreign blood. +In a breed which has not been crossed, but in which _both_ parents have +lost some character which their progenitor possessed, the tendency, +whether strong or weak, to reproduce the lost character might be, as +was formerly remarked, for all that we can see to the contrary, +transmitted for almost any number of generations. When a character +which has been lost in a breed, reappears after a great number of +generations, the most probable hypothesis is, not that the offspring +suddenly takes after an ancestor some hundred generations +distant, but that in each successive generation there has been a +tendency to reproduce the character in question, which at last, under +unknown favourable conditions, gains an ascendancy. For instance, it is +probable that in each generation of the barb-pigeon, which produces +most rarely a blue and black-barred bird, there has been a tendency in +each generation in the plumage to assume this colour. This view is +hypothetical, but could be supported by some facts; and I can see no +more abstract improbability in a tendency to produce any character +being inherited for an endless number of generations, than in quite +useless or rudimentary organs being, as we all know them to be, thus +inherited. Indeed, we may sometimes observe a mere tendency to produce +a rudiment inherited: for instance, in the common snapdragon +(Antirrhinum) a rudiment of a fifth stamen so often appears, that this +plant must have an inherited tendency to produce it. + +As all the species of the same genus are supposed, on my theory, to +have descended from a common parent, it might be expected that they +would occasionally vary in an analogous manner; so that a variety of +one species would resemble in some of its characters another species; +this other species being on my view only a well-marked and permanent +variety. But characters thus gained would probably be of an unimportant +nature, for the presence of all important characters will be governed +by natural selection, in accordance with the diverse habits of the +species, and will not be left to the mutual action of the conditions of +life and of a similar inherited constitution. It might further be +expected that the species of the same genus would occasionally exhibit +reversions to lost ancestral characters. As, however, we never know the +exact character of the common ancestor of a group, we could not +distinguish these two +cases: if, for instance, we did not know that the rock-pigeon was not +feather-footed or turn-crowned, we could not have told, whether these +characters in our domestic breeds were reversions or only analogous +variations; but we might have inferred that the blueness was a case of +reversion, from the number of the markings, which are correlated with +the blue tint, and which it does not appear probable would all appear +together from simple variation. More especially we might have inferred +this, from the blue colour and marks so often appearing when distinct +breeds of diverse colours are crossed. Hence, though under nature it +must generally be left doubtful, what cases are reversions to an +anciently existing character, and what are new but analogous +variations, yet we ought, on my theory, sometimes to find the varying +offspring of a species assuming characters (either from reversion or +from analogous variation) which already occur in some other members of +the same group. And this undoubtedly is the case in nature. + +A considerable part of the difficulty in recognising a variable species +in our systematic works, is due to its varieties mocking, as it were, +some of the other species of the same genus. A considerable catalogue, +also, could be given of forms intermediate between two other forms, +which themselves must be doubtfully ranked as either varieties or +species; and this shows, unless all these forms be considered as +independently created species, that the one in varying has assumed some +of the characters of the other, so as to produce the intermediate form. +But the best evidence is afforded by parts or organs of an important +and uniform nature occasionally varying so as to acquire, in some +degree, the character of the same part or organ in an allied species. I +have collected a long list of such cases; but +here, as before, I lie under a great disadvantage in not being able to +give them. I can only repeat that such cases certainly do occur, and +seem to me very remarkable. + +I will, however, give one curious and complex case, not indeed as +affecting any important character, but from occurring in several +species of the same genus, partly under domestication and partly under +nature. It is a case apparently of reversion. The ass not rarely has +very distinct transverse bars on its legs, like those on the legs of a +zebra: it has been asserted that these are plainest in the foal, and +from inquiries which I have made, I believe this to be true. It has +also been asserted that the stripe on each shoulder is sometimes +double. The shoulder stripe is certainly very variable in length and +outline. A white ass, but _not_ an albino, has been described without +either spinal or shoulder-stripe; and these stripes are sometimes very +obscure, or actually quite lost, in dark-coloured asses. The koulan of +Pallas is said to have been seen with a double shoulder-stripe. The +hemionus has no shoulder-stripe; but traces of it, as stated by Mr. +Blyth and others, occasionally appear: and I have been informed by +Colonel Poole that the foals of this species are generally striped on +the legs, and faintly on the shoulder. The quagga, though so plainly +barred like a zebra over the body, is without bars on the legs; but Dr. +Gray has figured one specimen with very distinct zebra-like bars on the +hocks. + +With respect to the horse, I have collected cases in England of the +spinal stripe in horses of the most distinct breeds, and of _all_ +colours; transverse bars on the legs are not rare in duns, mouse-duns, +and in one instance in a chestnut: a faint shoulder-stripe may +sometimes be seen in duns, and I have seen a trace in a +bay horse. My son made a careful examination and sketch for me of a dun +Belgian cart-horse with a double stripe on each shoulder and with +leg-stripes; and a man, whom I can implicitly trust, has examined for +me a small dun Welch pony with _three_ short parallel stripes on each +shoulder. + +In the north-west part of India the Kattywar breed of horses is so +generally striped, that, as I hear from Colonel Poole, who examined the +breed for the Indian Government, a horse without stripes is not +considered as purely-bred. The spine is always striped; the legs are +generally barred; and the shoulder-stripe, which is sometimes double +and sometimes treble, is common; the side of the face, moreover, is +sometimes striped. The stripes are plainest in the foal; and sometimes +quite disappear in old horses. Colonel Poole has seen both gray and bay +Kattywar horses striped when first foaled. I have, also, reason to +suspect, from information given me by Mr. W. W. Edwards, that with the +English race-horse the spinal stripe is much commoner in the foal than +in the full-grown animal. Without here entering on further details, I +may state that I have collected cases of leg and shoulder stripes in +horses of very different breeds, in various countries from Britain to +Eastern China; and from Norway in the north to the Malay Archipelago in +the south. In all parts of the world these stripes occur far oftenest +in duns and mouse-duns; by the term dun a large range of colour is +included, from one between brown and black to a close approach to +cream-colour. + +I am aware that Colonel Hamilton Smith, who has written on this +subject, believes that the several breeds of the horse have descended +from several aboriginal species—one of which, the dun, was striped; and +that the above-described appearances are all due to ancient +crosses with the dun stock. But I am not at all satisfied with this +theory, and should be loth to apply it to breeds so distinct as the +heavy Belgian cart-horse, Welch ponies, cobs, the lanky Kattywar race, +etc., inhabiting the most distant parts of the world. + +Now let us turn to the effects of crossing the several species of the +horse-genus. Rollin asserts, that the common mule from the ass and +horse is particularly apt to have bars on its legs. I once saw a mule +with its legs so much striped that any one at first would have thought +that it must have been the product of a zebra; and Mr. W. C. Martin, in +his excellent treatise on the horse, has given a figure of a similar +mule. In four coloured drawings, which I have seen, of hybrids between +the ass and zebra, the legs were much more plainly barred than the rest +of the body; and in one of them there was a double shoulder-stripe. In +Lord Moreton’s famous hybrid from a chestnut mare and male quagga, the +hybrid, and even the pure offspring subsequently produced from the mare +by a black Arabian sire, were much more plainly barred across the legs +than is even the pure quagga. Lastly, and this is another most +remarkable case, a hybrid has been figured by Dr. Gray (and he informs +me that he knows of a second case) from the ass and the hemionus; and +this hybrid, though the ass seldom has stripes on its legs and the +hemionus has none and has not even a shoulder-stripe, nevertheless had +all four legs barred, and had three short shoulder-stripes, like those +on the dun Welch pony, and even had some zebra-like stripes on the +sides of its face. With respect to this last fact, I was so convinced +that not even a stripe of colour appears from what would commonly be +called an accident, that I was led solely from the occurrence of the +face-stripes on this hybrid from the ass and hemionus, +to ask Colonel Poole whether such face-stripes ever occur in the +eminently striped Kattywar breed of horses, and was, as we have seen, +answered in the affirmative. + +What now are we to say to these several facts? We see several very +distinct species of the horse-genus becoming, by simple variation, +striped on the legs like a zebra, or striped on the shoulders like an +ass. In the horse we see this tendency strong whenever a dun tint +appears—a tint which approaches to that of the general colouring of the +other species of the genus. The appearance of the stripes is not +accompanied by any change of form or by any other new character. We see +this tendency to become striped most strongly displayed in hybrids from +between several of the most distinct species. Now observe the case of +the several breeds of pigeons: they are descended from a pigeon +(including two or three sub-species or geographical races) of a bluish +colour, with certain bars and other marks; and when any breed assumes +by simple variation a bluish tint, these bars and other marks +invariably reappear; but without any other change of form or character. +When the oldest and truest breeds of various colours are crossed, we +see a strong tendency for the blue tint and bars and marks to reappear +in the mongrels. I have stated that the most probable hypothesis to +account for the reappearance of very ancient characters, is—that there +is a _tendency_ in the young of each successive generation to produce +the long-lost character, and that this tendency, from unknown causes, +sometimes prevails. And we have just seen that in several species of +the horse-genus the stripes are either plainer or appear more commonly +in the young than in the old. Call the breeds of pigeons, some of which +have bred true for centuries, species; and how exactly parallel is the +case with that of the species of the horse-genus! +For myself, I venture confidently to look back thousands on thousands +of generations, and I see an animal striped like a zebra, but perhaps +otherwise very differently constructed, the common parent of our +domestic horse, whether or not it be descended from one or more wild +stocks, of the ass, the hemionus, quagga, and zebra. + +He who believes that each equine species was independently created, +will, I presume, assert that each species has been created with a +tendency to vary, both under nature and under domestication, in this +particular manner, so as often to become striped like other species of +the genus; and that each has been created with a strong tendency, when +crossed with species inhabiting distant quarters of the world, to +produce hybrids resembling in their stripes, not their own parents, but +other species of the genus. To admit this view is, as it seems to me, +to reject a real for an unreal, or at least for an unknown, cause. It +makes the works of God a mere mockery and deception; I would almost as +soon believe with the old and ignorant cosmogonists, that fossil shells +had never lived, but had been created in stone so as to mock the shells +now living on the sea-shore. + +_Summary_.—Our ignorance of the laws of variation is profound. Not in +one case out of a hundred can we pretend to assign any reason why this +or that part differs, more or less, from the same part in the parents. +But whenever we have the means of instituting a comparison, the same +laws appear to have acted in producing the lesser differences between +varieties of the same species, and the greater differences between +species of the same genus. The external conditions of life, as climate +and food, etc., seem to have induced some slight modifications. Habit +in producing constitutional differences, +and use in strengthening, and disuse in weakening and diminishing +organs, seem to have been more potent in their effects. Homologous +parts tend to vary in the same way, and homologous parts tend to +cohere. Modifications in hard parts and in external parts sometimes +affect softer and internal parts. When one part is largely developed, +perhaps it tends to draw nourishment from the adjoining parts; and +every part of the structure which can be saved without detriment to the +individual, will be saved. Changes of structure at an early age will +generally affect parts subsequently developed; and there are very many +other correlations of growth, the nature of which we are utterly unable +to understand. Multiple parts are variable in number and in structure, +perhaps arising from such parts not having been closely specialised to +any particular function, so that their modifications have not been +closely checked by natural selection. It is probably from this same +cause that organic beings low in the scale of nature are more variable +than those which have their whole organisation more specialised, and +are higher in the scale. Rudimentary organs, from being useless, will +be disregarded by natural selection, and hence probably are variable. +Specific characters—that is, the characters which have come to differ +since the several species of the same genus branched off from a common +parent—are more variable than generic characters, or those which have +long been inherited, and have not differed within this same period. In +these remarks we have referred to special parts or organs being still +variable, because they have recently varied and thus come to differ; +but we have also seen in the second Chapter that the same principle +applies to the whole individual; for in a district where many species +of any genus are found—that is, where there has been much former +variation and differentiation, or where the manufactory of new specific +forms has been actively at work—there, on an average, we now find most +varieties or incipient species. Secondary sexual characters are highly +variable, and such characters differ much in the species of the same +group. Variability in the same parts of the organisation has generally +been taken advantage of in giving secondary sexual differences to the +sexes of the same species, and specific differences to the several +species of the same genus. Any part or organ developed to an +extraordinary size or in an extraordinary manner, in comparison with +the same part or organ in the allied species, must have gone through an +extraordinary amount of modification since the genus arose; and thus we +can understand why it should often still be variable in a much higher +degree than other parts; for variation is a long-continued and slow +process, and natural selection will in such cases not as yet have had +time to overcome the tendency to further variability and to reversion +to a less modified state. But when a species with any +extraordinarily-developed organ has become the parent of many modified +descendants—which on my view must be a very slow process, requiring a +long lapse of time—in this case, natural selection may readily have +succeeded in giving a fixed character to the organ, in however +extraordinary a manner it may be developed. Species inheriting nearly +the same constitution from a common parent and exposed to similar +influences will naturally tend to present analogous variations, and +these same species may occasionally revert to some of the characters of +their ancient progenitors. Although new and important modifications may +not arise from reversion and analogous variation, such modifications +will add to the beautiful and harmonious diversity of nature. + +Whatever the cause may be of each slight difference in the offspring +from their parents—and a cause for each must exist—it is the steady +accumulation, through natural selection, of such differences, when +beneficial to the individual, that gives rise to all the more important +modifications of structure, by which the innumerable beings on the face +of this earth are enabled to struggle with each other, and the best +adapted to survive. + +Difficulties on the theory of descent with modification. Transitions. +Absence or rarity of transitional varieties. Transitions in habits of +life. Diversified habits in the same species. Species with habits +widely different from those of their allies. Organs of extreme +perfection. Means of transition. Cases of difficulty. Natura non facit +saltum. Organs of small importance. Organs not in all cases absolutely +perfect. The law of Unity of Type and of the Conditions of Existence +embraced by the theory of Natural Selection. + +Long before having arrived at this part of my work, a crowd of +difficulties will have occurred to the reader. Some of them are so +grave that to this day I can never reflect on them without being +staggered; but, to the best of my judgment, the greater number are only +apparent, and those that are real are not, I think, fatal to my theory. + +These difficulties and objections may be classed under the following +heads:— + +Firstly, why, if species have descended from other species by +insensibly fine gradations, do we not everywhere see innumerable +transitional forms? Why is not all nature in confusion instead of the +species being, as we see them, well defined? + +Secondly, is it possible that an animal having, for instance, the +structure and habits of a bat, could have been formed by the +modification of some animal with wholly different habits? Can we +believe that natural selection could produce, on the one hand, organs +of trifling importance, such as the tail of a giraffe, which serves as +a fly-flapper, and, on the other hand, organs of +such wonderful structure, as the eye, of which we hardly as yet fully +understand the inimitable perfection? + +Thirdly, can instincts be acquired and modified through natural +selection? What shall we say to so marvellous an instinct as that which +leads the bee to make cells, which have practically anticipated the +discoveries of profound mathematicians? + +Fourthly, how can we account for species, when crossed, being sterile +and producing sterile offspring, whereas, when varieties are crossed, +their fertility is unimpaired? + +The two first heads shall be here discussed—Instinct and Hybridism in +separate chapters. + +_On the absence or rarity of transitional varieties._—As natural +selection acts solely by the preservation of profitable modifications, +each new form will tend in a fully-stocked country to take the place +of, and finally to exterminate, its own less improved parent or other +less-favoured forms with which it comes into competition. Thus +extinction and natural selection will, as we have seen, go hand in +hand. Hence, if we look at each species as descended from some other +unknown form, both the parent and all the transitional varieties will +generally have been exterminated by the very process of formation and +perfection of the new form. + +But, as by this theory innumerable transitional forms must have +existed, why do we not find them embedded in countless numbers in the +crust of the earth? It will be much more convenient to discuss this +question in the chapter on the Imperfection of the geological record; +and I will here only state that I believe the answer mainly lies in the +record being incomparably less perfect than is generally supposed; the +imperfection of the record being chiefly due to organic beings not +inhabiting +profound depths of the sea, and to their remains being embedded and +preserved to a future age only in masses of sediment sufficiently thick +and extensive to withstand an enormous amount of future degradation; +and such fossiliferous masses can be accumulated only where much +sediment is deposited on the shallow bed of the sea, whilst it slowly +subsides. These contingencies will concur only rarely, and after +enormously long intervals. Whilst the bed of the sea is stationary or +is rising, or when very little sediment is being deposited, there will +be blanks in our geological history. The crust of the earth is a vast +museum; but the natural collections have been made only at intervals of +time immensely remote. + +But it may be urged that when several closely-allied species inhabit +the same territory we surely ought to find at the present time many +transitional forms. Let us take a simple case: in travelling from north +to south over a continent, we generally meet at successive intervals +with closely allied or representative species, evidently filling nearly +the same place in the natural economy of the land. These representative +species often meet and interlock; and as the one becomes rarer and +rarer, the other becomes more and more frequent, till the one replaces +the other. But if we compare these species where they intermingle, they +are generally as absolutely distinct from each other in every detail of +structure as are specimens taken from the metropolis inhabited by each. +By my theory these allied species have descended from a common parent; +and during the process of modification, each has become adapted to the +conditions of life of its own region, and has supplanted and +exterminated its original parent and all the transitional varieties +between its past and present states. Hence we ought not to expect at +the +present time to meet with numerous transitional varieties in each +region, though they must have existed there, and may be embedded there +in a fossil condition. But in the intermediate region, having +intermediate conditions of life, why do we not now find closely-linking +intermediate varieties? This difficulty for a long time quite +confounded me. But I think it can be in large part explained. + +In the first place we should be extremely cautious in inferring, +because an area is now continuous, that it has been continuous during a +long period. Geology would lead us to believe that almost every +continent has been broken up into islands even during the later +tertiary periods; and in such islands distinct species might have been +separately formed without the possibility of intermediate varieties +existing in the intermediate zones. By changes in the form of the land +and of climate, marine areas now continuous must often have existed +within recent times in a far less continuous and uniform condition than +at present. But I will pass over this way of escaping from the +difficulty; for I believe that many perfectly defined species have been +formed on strictly continuous areas; though I do not doubt that the +formerly broken condition of areas now continuous has played an +important part in the formation of new species, more especially with +freely-crossing and wandering animals. + +In looking at species as they are now distributed over a wide area, we +generally find them tolerably numerous over a large territory, then +becoming somewhat abruptly rarer and rarer on the confines, and finally +disappearing. Hence the neutral territory between two representative +species is generally narrow in comparison with the territory proper to +each. We see the same fact in ascending mountains, and sometimes +it is quite remarkable how abruptly, as Alph. De Candolle has observed, +a common alpine species disappears. The same fact has been noticed by +Forbes in sounding the depths of the sea with the dredge. To those who +look at climate and the physical conditions of life as the +all-important elements of distribution, these facts ought to cause +surprise, as climate and height or depth graduate away insensibly. But +when we bear in mind that almost every species, even in its metropolis, +would increase immensely in numbers, were it not for other competing +species; that nearly all either prey on or serve as prey for others; in +short, that each organic being is either directly or indirectly related +in the most important manner to other organic beings, we must see that +the range of the inhabitants of any country by no means exclusively +depends on insensibly changing physical conditions, but in large part +on the presence of other species, on which it depends, or by which it +is destroyed, or with which it comes into competition; and as these +species are already defined objects (however they may have become so), +not blending one into another by insensible gradations, the range of +any one species, depending as it does on the range of others, will tend +to be sharply defined. Moreover, each species on the confines of its +range, where it exists in lessened numbers, will, during fluctuations +in the number of its enemies or of its prey, or in the seasons, be +extremely liable to utter extermination; and thus its geographical +range will come to be still more sharply defined. + +If I am right in believing that allied or representative species, when +inhabiting a continuous area, are generally so distributed that each +has a wide range, with a comparatively narrow neutral territory between +them, in which they become rather suddenly rarer and rarer; then, as +varieties do not essentially differ from species, +the same rule will probably apply to both; and if we in imagination +adapt a varying species to a very large area, we shall have to adapt +two varieties to two large areas, and a third variety to a narrow +intermediate zone. The intermediate variety, consequently, will exist +in lesser numbers from inhabiting a narrow and lesser area; and +practically, as far as I can make out, this rule holds good with +varieties in a state of nature. I have met with striking instances of +the rule in the case of varieties intermediate between well-marked +varieties in the genus Balanus. And it would appear from information +given me by Mr. Watson, Dr. Asa Gray, and Mr. Wollaston, that generally +when varieties intermediate between two other forms occur, they are +much rarer numerically than the forms which they connect. Now, if we +may trust these facts and inferences, and therefore conclude that +varieties linking two other varieties together have generally existed +in lesser numbers than the forms which they connect, then, I think, we +can understand why intermediate varieties should not endure for very +long periods;—why as a general rule they should be exterminated and +disappear, sooner than the forms which they originally linked together. + +For any form existing in lesser numbers would, as already remarked, run +a greater chance of being exterminated than one existing in large +numbers; and in this particular case the intermediate form would be +eminently liable to the inroads of closely allied forms existing on +both sides of it. But a far more important consideration, as I believe, +is that, during the process of further modification, by which two +varieties are supposed on my theory to be converted and perfected into +two distinct species, the two which exist in larger numbers from +inhabiting larger areas, will have a great advantage over the +intermediate variety, which exists +in smaller numbers in a narrow and intermediate zone. For forms +existing in larger numbers will always have a better chance, within any +given period, of presenting further favourable variations for natural +selection to seize on, than will the rarer forms which exist in lesser +numbers. Hence, the more common forms, in the race for life, will tend +to beat and supplant the less common forms, for these will be more +slowly modified and improved. It is the same principle which, as I +believe, accounts for the common species in each country, as shown in +the second chapter, presenting on an average a greater number of +well-marked varieties than do the rarer species. I may illustrate what +I mean by supposing three varieties of sheep to be kept, one adapted to +an extensive mountainous region; a second to a comparatively narrow, +hilly tract; and a third to wide plains at the base; and that the +inhabitants are all trying with equal steadiness and skill to improve +their stocks by selection; the chances in this case will be strongly in +favour of the great holders on the mountains or on the plains improving +their breeds more quickly than the small holders on the intermediate +narrow, hilly tract; and consequently the improved mountain or plain +breed will soon take the place of the less improved hill breed; and +thus the two breeds, which originally existed in greater numbers, will +come into close contact with each other, without the interposition of +the supplanted, intermediate hill-variety. + +To sum up, I believe that species come to be tolerably well-defined +objects, and do not at any one period present an inextricable chaos of +varying and intermediate links: firstly, because new varieties are very +slowly formed, for variation is a very slow process, and natural +selection can do nothing until favourable variations chance to occur, +and until a place in the natural polity +of the country can be better filled by some modification of some one or +more of its inhabitants. And such new places will depend on slow +changes of climate, or on the occasional immigration of new +inhabitants, and, probably, in a still more important degree, on some +of the old inhabitants becoming slowly modified, with the new forms +thus produced and the old ones acting and reacting on each other. So +that, in any one region and at any one time, we ought only to see a few +species presenting slight modifications of structure in some degree +permanent; and this assuredly we do see. + +Secondly, areas now continuous must often have existed within the +recent period in isolated portions, in which many forms, more +especially amongst the classes which unite for each birth and wander +much, may have separately been rendered sufficiently distinct to rank +as representative species. In this case, intermediate varieties between +the several representative species and their common parent, must +formerly have existed in each broken portion of the land, but these +links will have been supplanted and exterminated during the process of +natural selection, so that they will no longer exist in a living state. + +Thirdly, when two or more varieties have been formed in different +portions of a strictly continuous area, intermediate varieties will, it +is probable, at first have been formed in the intermediate zones, but +they will generally have had a short duration. For these intermediate +varieties will, from reasons already assigned (namely from what we know +of the actual distribution of closely allied or representative species, +and likewise of acknowledged varieties), exist in the intermediate +zones in lesser numbers than the varieties which they tend to connect. +From this cause alone the intermediate +varieties will be liable to accidental extermination; and during the +process of further modification through natural selection, they will +almost certainly be beaten and supplanted by the forms which they +connect; for these from existing in greater numbers will, in the +aggregate, present more variation, and thus be further improved through +natural selection and gain further advantages. + +Lastly, looking not to any one time, but to all time, if my theory be +true, numberless intermediate varieties, linking most closely all the +species of the same group together, must assuredly have existed; but +the very process of natural selection constantly tends, as has been so +often remarked, to exterminate the parent forms and the intermediate +links. Consequently evidence of their former existence could be found +only amongst fossil remains, which are preserved, as we shall in a +future chapter attempt to show, in an extremely imperfect and +intermittent record. + +_On the origin and transitions of organic beings with peculiar habits +and structure_.—It has been asked by the opponents of such views as I +hold, how, for instance, a land carnivorous animal could have been +converted into one with aquatic habits; for how could the animal in its +transitional state have subsisted? It would be easy to show that within +the same group carnivorous animals exist having every intermediate +grade between truly aquatic and strictly terrestrial habits; and as +each exists by a struggle for life, it is clear that each is well +adapted in its habits to its place in nature. Look at the Mustela vison +of North America, which has webbed feet and which resembles an otter in +its fur, short legs, and form of tail; during summer this animal dives +for and preys on fish, but during the long winter +it leaves the frozen waters, and preys like other polecats on mice and +land animals. If a different case had been taken, and it had been asked +how an insectivorous quadruped could possibly have been converted into +a flying bat, the question would have been far more difficult, and I +could have given no answer. Yet I think such difficulties have very +little weight. + +Here, as on other occasions, I lie under a heavy disadvantage, for out +of the many striking cases which I have collected, I can give only one +or two instances of transitional habits and structures in closely +allied species of the same genus; and of diversified habits, either +constant or occasional, in the same species. And it seems to me that +nothing less than a long list of such cases is sufficient to lessen the +difficulty in any particular case like that of the bat. + +Look at the family of squirrels; here we have the finest gradation from +animals with their tails only slightly flattened, and from others, as +Sir J. Richardson has remarked, with the posterior part of their bodies +rather wide and with the skin on their flanks rather full, to the +so-called flying squirrels; and flying squirrels have their limbs and +even the base of the tail united by a broad expanse of skin, which +serves as a parachute and allows them to glide through the air to an +astonishing distance from tree to tree. We cannot doubt that each +structure is of use to each kind of squirrel in its own country, by +enabling it to escape birds or beasts of prey, or to collect food more +quickly, or, as there is reason to believe, by lessening the danger +from occasional falls. But it does not follow from this fact that the +structure of each squirrel is the best that it is possible to conceive +under all natural conditions. Let the climate and vegetation change, +let other competing rodents or new beasts of prey immigrate, or old +ones +become modified, and all analogy would lead us to believe that some at +least of the squirrels would decrease in numbers or become +exterminated, unless they also became modified and improved in +structure in a corresponding manner. Therefore, I can see no +difficulty, more especially under changing conditions of life, in the +continued preservation of individuals with fuller and fuller +flank-membranes, each modification being useful, each being propagated, +until by the accumulated effects of this process of natural selection, +a perfect so-called flying squirrel was produced. + +Now look at the Galeopithecus or flying lemur, which formerly was +falsely ranked amongst bats. It has an extremely wide flank-membrane, +stretching from the corners of the jaw to the tail, and including the +limbs and the elongated fingers: the flank membrane is, also, furnished +with an extensor muscle. Although no graduated links of structure, +fitted for gliding through the air, now connect the Galeopithecus with +the other Lemuridæ, yet I can see no difficulty in supposing that such +links formerly existed, and that each had been formed by the same steps +as in the case of the less perfectly gliding squirrels; and that each +grade of structure had been useful to its possessor. Nor can I see any +insuperable difficulty in further believing it possible that the +membrane-connected fingers and fore-arm of the Galeopithecus might be +greatly lengthened by natural selection; and this, as far as the organs +of flight are concerned, would convert it into a bat. In bats which +have the wing-membrane extended from the top of the shoulder to the +tail, including the hind-legs, we perhaps see traces of an apparatus +originally constructed for gliding through the air rather than for +flight. + +If about a dozen genera of birds had become extinct or were unknown, +who would have ventured to have +surmised that birds might have existed which used their wings solely as +flappers, like the logger-headed duck (Micropterus of Eyton); as fins +in the water and front legs on the land, like the penguin; as sails, +like the ostrich; and functionally for no purpose, like the Apteryx. +Yet the structure of each of these birds is good for it, under the +conditions of life to which it is exposed, for each has to live by a +struggle; but it is not necessarily the best possible under all +possible conditions. It must not be inferred from these remarks that +any of the grades of wing-structure here alluded to, which perhaps may +all have resulted from disuse, indicate the natural steps by which +birds have acquired their perfect power of flight; but they serve, at +least, to show what diversified means of transition are possible. + +Seeing that a few members of such water-breathing classes as the +Crustacea and Mollusca are adapted to live on the land, and seeing that +we have flying birds and mammals, flying insects of the most +diversified types, and formerly had flying reptiles, it is conceivable +that flying-fish, which now glide far through the air, slightly rising +and turning by the aid of their fluttering fins, might have been +modified into perfectly winged animals. If this had been effected, who +would have ever imagined that in an early transitional state they had +been inhabitants of the open ocean, and had used their incipient organs +of flight exclusively, as far as we know, to escape being devoured by +other fish? + +When we see any structure highly perfected for any particular habit, as +the wings of a bird for flight, we should bear in mind that animals +displaying early transitional grades of the structure will seldom +continue to exist to the present day, for they will have been +supplanted by the very process of perfection through natural selection. +Furthermore, we may conclude that transitional +grades between structures fitted for very different habits of life will +rarely have been developed at an early period in great numbers and +under many subordinate forms. Thus, to return to our imaginary +illustration of the flying-fish, it does not seem probable that fishes +capable of true flight would have been developed under many subordinate +forms, for taking prey of many kinds in many ways, on the land and in +the water, until their organs of flight had come to a high stage of +perfection, so as to have given them a decided advantage over other +animals in the battle for life. Hence the chance of discovering species +with transitional grades of structure in a fossil condition will always +be less, from their having existed in lesser numbers, than in the case +of species with fully developed structures. + +I will now give two or three instances of diversified and of changed +habits in the individuals of the same species. When either case occurs, +it would be easy for natural selection to fit the animal, by some +modification of its structure, for its changed habits, or exclusively +for one of its several different habits. But it is difficult to tell, +and immaterial for us, whether habits generally change first and +structure afterwards; or whether slight modifications of structure lead +to changed habits; both probably often change almost simultaneously. Of +cases of changed habits it will suffice merely to allude to that of the +many British insects which now feed on exotic plants, or exclusively on +artificial substances. Of diversified habits innumerable instances +could be given: I have often watched a tyrant flycatcher (Saurophagus +sulphuratus) in South America, hovering over one spot and then +proceeding to another, like a kestrel, and at other times standing +stationary on the margin of water, and then dashing like a kingfisher +at a fish. In our own country the larger titmouse (Parus major) may be +seen climbing branches, almost like a creeper; it often, like a shrike, +kills small birds by blows on the head; and I have many times seen and +heard it hammering the seeds of the yew on a branch, and thus breaking +them like a nuthatch. In North America the black bear was seen by +Hearne swimming for hours with widely open mouth, thus catching, like a +whale, insects in the water. Even in so extreme a case as this, if the +supply of insects were constant, and if better adapted competitors did +not already exist in the country, I can see no difficulty in a race of +bears being rendered, by natural selection, more and more aquatic in +their structure and habits, with larger and larger mouths, till a +creature was produced as monstrous as a whale. + +As we sometimes see individuals of a species following habits widely +different from those both of their own species and of the other species +of the same genus, we might expect, on my theory, that such individuals +would occasionally have given rise to new species, having anomalous +habits, and with their structure either slightly or considerably +modified from that of their proper type. And such instances do occur in +nature. Can a more striking instance of adaptation be given than that +of a woodpecker for climbing trees and for seizing insects in the +chinks of the bark? Yet in North America there are woodpeckers which +feed largely on fruit, and others with elongated wings which chase +insects on the wing; and on the plains of La Plata, where not a tree +grows, there is a woodpecker, which in every essential part of its +organisation, even in its colouring, in the harsh tone of its voice, +and undulatory flight, told me plainly of its close blood-relationship +to our common species; yet it is a woodpecker which never climbs a +tree! + +Petrels are the most aërial and oceanic of birds, yet in the quiet +Sounds of Tierra del Fuego, the Puffinuria +berardi, in its general habits, in its astonishing power of diving, its +manner of swimming, and of flying when unwillingly it takes flight, +would be mistaken by any one for an auk or grebe; nevertheless, it is +essentially a petrel, but with many parts of its organisation +profoundly modified. On the other hand, the acutest observer by +examining the dead body of the water-ouzel would never have suspected +its sub-aquatic habits; yet this anomalous member of the strictly +terrestrial thrush family wholly subsists by diving,—grasping the +stones with its feet and using its wings under water. + +He who believes that each being has been created as we now see it, must +occasionally have felt surprise when he has met with an animal having +habits and structure not at all in agreement. What can be plainer than +that the webbed feet of ducks and geese are formed for swimming? yet +there are upland geese with webbed feet which rarely or never go near +the water; and no one except Audubon has seen the frigate-bird, which +has all its four toes webbed, alight on the surface of the sea. On the +other hand, grebes and coots are eminently aquatic, although their toes +are only bordered by membrane. What seems plainer than that the long +toes of grallatores are formed for walking over swamps and floating +plants, yet the water-hen is nearly as aquatic as the coot; and the +landrail nearly as terrestrial as the quail or partridge. In such +cases, and many others could be given, habits have changed without a +corresponding change of structure. The webbed feet of the upland goose +may be said to have become rudimentary in function, though not in +structure. In the frigate-bird, the deeply-scooped membrane between the +toes shows that structure has begun to change. + +He who believes in separate and innumerable acts of creation will say, +that in these cases it has pleased the +Creator to cause a being of one type to take the place of one of +another type; but this seems to me only restating the fact in dignified +language. He who believes in the struggle for existence and in the +principle of natural selection, will acknowledge that every organic +being is constantly endeavouring to increase in numbers; and that if +any one being vary ever so little, either in habits or structure, and +thus gain an advantage over some other inhabitant of the country, it +will seize on the place of that inhabitant, however different it may be +from its own place. Hence it will cause him no surprise that there +should be geese and frigate-birds with webbed feet, either living on +the dry land or most rarely alighting on the water; that there should +be long-toed corncrakes living in meadows instead of in swamps; that +there should be woodpeckers where not a tree grows; that there should +be diving thrushes, and petrels with the habits of auks. + +_Organs of extreme perfection and complication_.—To suppose that the +eye, with all its inimitable contrivances for adjusting the focus to +different distances, for admitting different amounts of light, and for +the correction of spherical and chromatic aberration, could have been +formed by natural selection, seems, I freely confess, absurd in the +highest possible degree. Yet reason tells me, that if numerous +gradations from a perfect and complex eye to one very imperfect and +simple, each grade being useful to its possessor, can be shown to +exist; if further, the eye does vary ever so slightly, and the +variations be inherited, which is certainly the case; and if any +variation or modification in the organ be ever useful to an animal +under changing conditions of life, then the difficulty of believing +that a perfect and complex eye could be formed by natural +selection, though insuperable by our imagination, can hardly be +considered real. How a nerve comes to be sensitive to light, hardly +concerns us more than how life itself first originated; but I may +remark that several facts make me suspect that any sensitive nerve may +be rendered sensitive to light, and likewise to those coarser +vibrations of the air which produce sound. + +In looking for the gradations by which an organ in any species has been +perfected, we ought to look exclusively to its lineal ancestors; but +this is scarcely ever possible, and we are forced in each case to look +to species of the same group, that is to the collateral descendants +from the same original parent-form, in order to see what gradations are +possible, and for the chance of some gradations having been transmitted +from the earlier stages of descent, in an unaltered or little altered +condition. Amongst existing Vertebrata, we find but a small amount of +gradation in the structure of the eye, and from fossil species we can +learn nothing on this head. In this great class we should probably have +to descend far beneath the lowest known fossiliferous stratum to +discover the earlier stages, by which the eye has been perfected. + +In the Articulata we can commence a series with an optic nerve merely +coated with pigment, and without any other mechanism; and from this low +stage, numerous gradations of structure, branching off in two +fundamentally different lines, can be shown to exist, until we reach a +moderately high stage of perfection. In certain crustaceans, for +instance, there is a double cornea, the inner one divided into facets, +within each of which there is a lens-shaped swelling. In other +crustaceans the transparent cones which are coated by pigment, and +which properly act only by excluding lateral pencils of light, are +convex at their upper ends +and must act by convergence; and at their lower ends there seems to be +an imperfect vitreous substance. With these facts, here far too briefly +and imperfectly given, which show that there is much graduated +diversity in the eyes of living crustaceans, and bearing in mind how +small the number of living animals is in proportion to those which have +become extinct, I can see no very great difficulty (not more than in +the case of many other structures) in believing that natural selection +has converted the simple apparatus of an optic nerve merely coated with +pigment and invested by transparent membrane, into an optical +instrument as perfect as is possessed by any member of the great +Articulate class. + +He who will go thus far, if he find on finishing this treatise that +large bodies of facts, otherwise inexplicable, can be explained by the +theory of descent, ought not to hesitate to go further, and to admit +that a structure even as perfect as the eye of an eagle might be formed +by natural selection, although in this case he does not know any of the +transitional grades. His reason ought to conquer his imagination; +though I have felt the difficulty far too keenly to be surprised at any +degree of hesitation in extending the principle of natural selection to +such startling lengths. + +It is scarcely possible to avoid comparing the eye to a telescope. We +know that this instrument has been perfected by the long-continued +efforts of the highest human intellects; and we naturally infer that +the eye has been formed by a somewhat analogous process. But may not +this inference be presumptuous? Have we any right to assume that the +Creator works by intellectual powers like those of man? If we must +compare the eye to an optical instrument, we ought in imagination to +take a thick layer of transparent tissue, with a nerve sensitive to +light beneath, and then suppose every +part of this layer to be continually changing slowly in density, so as +to separate into layers of different densities and thicknesses, placed +at different distances from each other, and with the surfaces of each +layer slowly changing in form. Further we must suppose that there is a +power always intently watching each slight accidental alteration in the +transparent layers; and carefully selecting each alteration which, +under varied circumstances, may in any way, or in any degree, tend to +produce a distincter image. We must suppose each new state of the +instrument to be multiplied by the million; and each to be preserved +till a better be produced, and then the old ones to be destroyed. In +living bodies, variation will cause the slight alterations, generation +will multiply them almost infinitely, and natural selection will pick +out with unerring skill each improvement. Let this process go on for +millions on millions of years; and during each year on millions of +individuals of many kinds; and may we not believe that a living optical +instrument might thus be formed as superior to one of glass, as the +works of the Creator are to those of man? + +If it could be demonstrated that any complex organ existed, which could +not possibly have been formed by numerous, successive, slight +modifications, my theory would absolutely break down. But I can find +out no such case. No doubt many organs exist of which we do not know +the transitional grades, more especially if we look to much-isolated +species, round which, according to my theory, there has been much +extinction. Or again, if we look to an organ common to all the members +of a large class, for in this latter case the organ must have been +first formed at an extremely remote period, since which all the many +members of the class have been developed; and in order to discover the +early transitional grades through which the organ has +passed, we should have to look to very ancient ancestral forms, long +since become extinct. + +We should be extremely cautious in concluding that an organ could not +have been formed by transitional gradations of some kind. Numerous +cases could be given amongst the lower animals of the same organ +performing at the same time wholly distinct functions; thus the +alimentary canal respires, digests, and excretes in the larva of the +dragon-fly and in the fish Cobites. In the Hydra, the animal may be +turned inside out, and the exterior surface will then digest and the +stomach respire. In such cases natural selection might easily +specialise, if any advantage were thus gained, a part or organ, which +had performed two functions, for one function alone, and thus wholly +change its nature by insensible steps. Two distinct organs sometimes +perform simultaneously the same function in the same individual; to +give one instance, there are fish with gills or branchiæ that breathe +the air dissolved in the water, at the same time that they breathe free +air in their swimbladders, this latter organ having a ductus +pneumaticus for its supply, and being divided by highly vascular +partitions. In these cases, one of the two organs might with ease be +modified and perfected so as to perform all the work by itself, being +aided during the process of modification by the other organ; and then +this other organ might be modified for some other and quite distinct +purpose, or be quite obliterated. + +The illustration of the swimbladder in fishes is a good one, because it +shows us clearly the highly important fact that an organ originally +constructed for one purpose, namely flotation, may be converted into +one for a wholly different purpose, namely respiration. The swimbladder +has, also, been worked in as an accessory to the auditory organs of +certain fish, or, for I do not know which +view is now generally held, a part of the auditory apparatus has been +worked in as a complement to the swimbladder. All physiologists admit +that the swimbladder is homologous, or “ideally similar,” in position +and structure with the lungs of the higher vertebrate animals: hence +there seems to me to be no great difficulty in believing that natural +selection has actually converted a swimbladder into a lung, or organ +used exclusively for respiration. + +I can, indeed, hardly doubt that all vertebrate animals having true +lungs have descended by ordinary generation from an ancient prototype, +of which we know nothing, furnished with a floating apparatus or +swimbladder. We can thus, as I infer from Professor Owen’s interesting +description of these parts, understand the strange fact that every +particle of food and drink which we swallow has to pass over the +orifice of the trachea, with some risk of falling into the lungs, +notwithstanding the beautiful contrivance by which the glottis is +closed. In the higher Vertebrata the branchiæ have wholly +disappeared—the slits on the sides of the neck and the loop-like course +of the arteries still marking in the embryo their former position. But +it is conceivable that the now utterly lost branchiæ might have been +gradually worked in by natural selection for some quite distinct +purpose: in the same manner as, on the view entertained by some +naturalists that the branchiæ and dorsal scales of Annelids are +homologous with the wings and wing-covers of insects, it is probable +that organs which at a very ancient period served for respiration have +been actually converted into organs of flight. + +In considering transitions of organs, it is so important to bear in +mind the probability of conversion from one function to another, that I +will give one more instance. Pedunculated cirripedes have two minute +folds of skin, +called by me the ovigerous frena, which serve, through the means of a +sticky secretion, to retain the eggs until they are hatched within the +sack. These cirripedes have no branchiæ, the whole surface of the body +and sack, including the small frena, serving for respiration. The +Balanidæ or sessile cirripedes, on the other hand, have no ovigerous +frena, the eggs lying loose at the bottom of the sack, in the +well-enclosed shell; but they have large folded branchiæ. Now I think +no one will dispute that the ovigerous frena in the one family are +strictly homologous with the branchiæ of the other family; indeed, they +graduate into each other. Therefore I do not doubt that little folds of +skin, which originally served as ovigerous frena, but which, likewise, +very slightly aided the act of respiration, have been gradually +converted by natural selection into branchiæ, simply through an +increase in their size and the obliteration of their adhesive glands. +If all pedunculated cirripedes had become extinct, and they have +already suffered far more extinction than have sessile cirripedes, who +would ever have imagined that the branchiæ in this latter family had +originally existed as organs for preventing the ova from being washed +out of the sack? + +Although we must be extremely cautious in concluding that any organ +could not possibly have been produced by successive transitional +gradations, yet, undoubtedly, grave cases of difficulty occur, some of +which will be discussed in my future work. + +One of the gravest is that of neuter insects, which are often very +differently constructed from either the males or fertile females; but +this case will be treated of in the next chapter. The electric organs +of fishes offer another case of special difficulty; it is impossible to +conceive by what steps these wondrous organs have been produced; but, +as Owen and others have remarked, +their intimate structure closely resembles that of common muscle; and +as it has lately been shown that Rays have an organ closely analogous +to the electric apparatus, and yet do not, as Matteuchi asserts, +discharge any electricity, we must own that we are far too ignorant to +argue that no transition of any kind is possible. + +The electric organs offer another and even more serious difficulty; for +they occur in only about a dozen fishes, of which several are widely +remote in their affinities. Generally when the same organ appears in +several members of the same class, especially if in members having very +different habits of life, we may attribute its presence to inheritance +from a common ancestor; and its absence in some of the members to its +loss through disuse or natural selection. But if the electric organs +had been inherited from one ancient progenitor thus provided, we might +have expected that all electric fishes would have been specially +related to each other. Nor does geology at all lead to the belief that +formerly most fishes had electric organs, which most of their modified +descendants have lost. The presence of luminous organs in a few +insects, belonging to different families and orders, offers a parallel +case of difficulty. Other cases could be given; for instance in plants, +the very curious contrivance of a mass of pollen-grains, borne on a +foot-stalk with a sticky gland at the end, is the same in Orchis and +Asclepias,—genera almost as remote as possible amongst flowering +plants. In all these cases of two very distinct species furnished with +apparently the same anomalous organ, it should be observed that, +although the general appearance and function of the organ may be the +same, yet some fundamental difference can generally be detected. I am +inclined to believe that in nearly the same way as two men have +sometimes independently hit on +the very same invention, so natural selection, working for the good of +each being and taking advantage of analogous variations, has sometimes +modified in very nearly the same manner two parts in two organic +beings, which owe but little of their structure in common to +inheritance from the same ancestor. + +Although in many cases it is most difficult to conjecture by what +transitions an organ could have arrived at its present state; yet, +considering that the proportion of living and known forms to the +extinct and unknown is very small, I have been astonished how rarely an +organ can be named, towards which no transitional grade is known to +lead. The truth of this remark is indeed shown by that old canon in +natural history of “Natura non facit saltum.” We meet with this +admission in the writings of almost every experienced naturalist; or, +as Milne Edwards has well expressed it, nature is prodigal in variety, +but niggard in innovation. Why, on the theory of Creation, should this +be so? Why should all the parts and organs of many independent beings, +each supposed to have been separately created for its proper place in +nature, be so invariably linked together by graduated steps? Why should +not Nature have taken a leap from structure to structure? On the theory +of natural selection, we can clearly understand why she should not; for +natural selection can act only by taking advantage of slight successive +variations; she can never take a leap, but must advance by the shortest +and slowest steps. + +_Organs of little apparent importance_.—As natural selection acts by +life and death,—by the preservation of individuals with any favourable +variation, and by the destruction of those with any unfavourable +deviation of structure,—I have sometimes felt much difficulty in +understanding the origin of simple parts, of which the importance does +not seem sufficient to cause the preservation of successively varying +individuals. I have sometimes felt as much difficulty, though of a very +different kind, on this head, as in the case of an organ as perfect and +complex as the eye. + +In the first place, we are much too ignorant in regard to the whole +economy of any one organic being, to say what slight modifications +would be of importance or not. In a former chapter I have given +instances of most trifling characters, such as the down on fruit and +the colour of the flesh, which, from determining the attacks of insects +or from being correlated with constitutional differences, might +assuredly be acted on by natural selection. The tail of the giraffe +looks like an artificially constructed fly-flapper; and it seems at +first incredible that this could have been adapted for its present +purpose by successive slight modifications, each better and better, for +so trifling an object as driving away flies; yet we should pause before +being too positive even in this case, for we know that the distribution +and existence of cattle and other animals in South America absolutely +depends on their power of resisting the attacks of insects: so that +individuals which could by any means defend themselves from these small +enemies, would be able to range into new pastures and thus gain a great +advantage. It is not that the larger quadrupeds are actually destroyed +(except in some rare cases) by the flies, but they are incessantly +harassed and their strength reduced, so that they are more subject to +disease, or not so well enabled in a coming dearth to search for food, +or to escape from beasts of prey. + +Organs now of trifling importance have probably in some cases been of +high importance to an early progenitor, and, after having been slowly +perfected at a +former period, have been transmitted in nearly the same state, although +now become of very slight use; and any actually injurious deviations in +their structure will always have been checked by natural selection. +Seeing how important an organ of locomotion the tail is in most aquatic +animals, its general presence and use for many purposes in so many land +animals, which in their lungs or modified swim-bladders betray their +aquatic origin, may perhaps be thus accounted for. A well-developed +tail having been formed in an aquatic animal, it might subsequently +come to be worked in for all sorts of purposes, as a fly-flapper, an +organ of prehension, or as an aid in turning, as with the dog, though +the aid must be slight, for the hare, with hardly any tail, can double +quickly enough. + +In the second place, we may sometimes attribute importance to +characters which are really of very little importance, and which have +originated from quite secondary causes, independently of natural +selection. We should remember that climate, food, etc., probably have +some little direct influence on the organisation; that characters +reappear from the law of reversion; that correlation of growth will +have had a most important influence in modifying various structures; +and finally, that sexual selection will often have largely modified the +external characters of animals having a will, to give one male an +advantage in fighting with another or in charming the females. Moreover +when a modification of structure has primarily arisen from the above or +other unknown causes, it may at first have been of no advantage to the +species, but may subsequently have been taken advantage of by the +descendants of the species under new conditions of life and with newly +acquired habits. + +To give a few instances to illustrate these latter +remarks. If green woodpeckers alone had existed, and we did not know +that there were many black and pied kinds, I dare say that we should +have thought that the green colour was a beautiful adaptation to hide +this tree-frequenting bird from its enemies; and consequently that it +was a character of importance and might have been acquired through +natural selection; as it is, I have no doubt that the colour is due to +some quite distinct cause, probably to sexual selection. A trailing +bamboo in the Malay Archipelago climbs the loftiest trees by the aid of +exquisitely constructed hooks clustered around the ends of the +branches, and this contrivance, no doubt, is of the highest service to +the plant; but as we see nearly similar hooks on many trees which are +not climbers, the hooks on the bamboo may have arisen from unknown laws +of growth, and have been subsequently taken advantage of by the plant +undergoing further modification and becoming a climber. The naked skin +on the head of a vulture is generally looked at as a direct adaptation +for wallowing in putridity; and so it may be, or it may possibly be due +to the direct action of putrid matter; but we should be very cautious +in drawing any such inference, when we see that the skin on the head of +the clean-feeding male turkey is likewise naked. The sutures in the +skulls of young mammals have been advanced as a beautiful adaptation +for aiding parturition, and no doubt they facilitate, or may be +indispensable for this act; but as sutures occur in the skulls of young +birds and reptiles, which have only to escape from a broken egg, we may +infer that this structure has arisen from the laws of growth, and has +been taken advantage of in the parturition of the higher animals. + +We are profoundly ignorant of the causes producing slight and +unimportant variations; and we are immediately +made conscious of this by reflecting on the differences in the breeds +of our domesticated animals in different countries,—more especially in +the less civilized countries where there has been but little artificial +selection. Careful observers are convinced that a damp climate affects +the growth of the hair, and that with the hair the horns are +correlated. Mountain breeds always differ from lowland breeds; and a +mountainous country would probably affect the hind limbs from +exercising them more, and possibly even the form of the pelvis; and +then by the law of homologous variation, the front limbs and even the +head would probably be affected. The shape, also, of the pelvis might +affect by pressure the shape of the head of the young in the womb. The +laborious breathing necessary in high regions would, we have some +reason to believe, increase the size of the chest; and again +correlation would come into play. Animals kept by savages in different +countries often have to struggle for their own subsistence, and would +be exposed to a certain extent to natural selection, and individuals +with slightly different constitutions would succeed best under +different climates; and there is reason to believe that constitution +and colour are correlated. A good observer, also, states that in cattle +susceptibility to the attacks of flies is correlated with colour, as is +the liability to be poisoned by certain plants; so that colour would be +thus subjected to the action of natural selection. But we are far too +ignorant to speculate on the relative importance of the several known +and unknown laws of variation; and I have here alluded to them only to +show that, if we are unable to account for the characteristic +differences of our domestic breeds, which nevertheless we generally +admit to have arisen through ordinary generation, we ought not to lay +too much stress on our +ignorance of the precise cause of the slight analogous differences +between species. I might have adduced for this same purpose the +differences between the races of man, which are so strongly marked; I +may add that some little light can apparently be thrown on the origin +of these differences, chiefly through sexual selection of a particular +kind, but without here entering on copious details my reasoning would +appear frivolous. + +The foregoing remarks lead me to say a few words on the protest lately +made by some naturalists, against the utilitarian doctrine that every +detail of structure has been produced for the good of its possessor. +They believe that very many structures have been created for beauty in +the eyes of man, or for mere variety. This doctrine, if true, would be +absolutely fatal to my theory. Yet I fully admit that many structures +are of no direct use to their possessors. Physical conditions probably +have had some little effect on structure, quite independently of any +good thus gained. Correlation of growth has no doubt played a most +important part, and a useful modification of one part will often have +entailed on other parts diversified changes of no direct use. So again +characters which formerly were useful, or which formerly had arisen +from correlation of growth, or from other unknown cause, may reappear +from the law of reversion, though now of no direct use. The effects of +sexual selection, when displayed in beauty to charm the females, can be +called useful only in rather a forced sense. But by far the most +important consideration is that the chief part of the organisation of +every being is simply due to inheritance; and consequently, though each +being assuredly is well fitted for its place in nature, many structures +now have no direct relation to the habits of life of each species. +Thus, we can hardly believe that the webbed feet of the upland +goose or of the frigate-bird are of special use to these birds; we +cannot believe that the same bones in the arm of the monkey, in the +fore leg of the horse, in the wing of the bat, and in the flipper of +the seal, are of special use to these animals. We may safely attribute +these structures to inheritance. But to the progenitor of the upland +goose and of the frigate-bird, webbed feet no doubt were as useful as +they now are to the most aquatic of existing birds. So we may believe +that the progenitor of the seal had not a flipper, but a foot with five +toes fitted for walking or grasping; and we may further venture to +believe that the several bones in the limbs of the monkey, horse, and +bat, which have been inherited from a common progenitor, were formerly +of more special use to that progenitor, or its progenitors, than they +now are to these animals having such widely diversified habits. +Therefore we may infer that these several bones might have been +acquired through natural selection, subjected formerly, as now, to the +several laws of inheritance, reversion, correlation of growth, etc. +Hence every detail of structure in every living creature (making some +little allowance for the direct action of physical conditions) may be +viewed, either as having been of special use to some ancestral form, or +as being now of special use to the descendants of this form—either +directly, or indirectly through the complex laws of growth. + +Natural selection cannot possibly produce any modification in any one +species exclusively for the good of another species; though throughout +nature one species incessantly takes advantage of, and profits by, the +structure of another. But natural selection can and does often produce +structures for the direct injury of other species, as we see in the +fang of the adder, and in the ovipositor of the ichneumon, by which its +eggs are deposited +in the living bodies of other insects. If it could be proved that any +part of the structure of any one species had been formed for the +exclusive good of another species, it would annihilate my theory, for +such could not have been produced through natural selection. Although +many statements may be found in works on natural history to this +effect, I cannot find even one which seems to me of any weight. It is +admitted that the rattlesnake has a poison-fang for its own defence and +for the destruction of its prey; but some authors suppose that at the +same time this snake is furnished with a rattle for its own injury, +namely, to warn its prey to escape. I would almost as soon believe that +the cat curls the end of its tail when preparing to spring, in order to +warn the doomed mouse. But I have not space here to enter on this and +other such cases. + +Natural selection will never produce in a being anything injurious to +itself, for natural selection acts solely by and for the good of each. +No organ will be formed, as Paley has remarked, for the purpose of +causing pain or for doing an injury to its possessor. If a fair balance +be struck between the good and evil caused by each part, each will be +found on the whole advantageous. After the lapse of time, under +changing conditions of life, if any part comes to be injurious, it will +be modified; or if it be not so, the being will become extinct, as +myriads have become extinct. + +Natural selection tends only to make each organic being as perfect as, +or slightly more perfect than, the other inhabitants of the same +country with which it has to struggle for existence. And we see that +this is the degree of perfection attained under nature. The endemic +productions of New Zealand, for instance, are perfect one compared with +another; but they are now rapidly yielding before the advancing legions +of plants +and animals introduced from Europe. Natural selection will not produce +absolute perfection, nor do we always meet, as far as we can judge, +with this high standard under nature. The correction for the aberration +of light is said, on high authority, not to be perfect even in that +most perfect organ, the eye. If our reason leads us to admire with +enthusiasm a multitude of inimitable contrivances in nature, this same +reason tells us, though we may easily err on both sides, that some +other contrivances are less perfect. Can we consider the sting of the +wasp or of the bee as perfect, which, when used against many attacking +animals, cannot be withdrawn, owing to the backward serratures, and so +inevitably causes the death of the insect by tearing out its viscera? + +If we look at the sting of the bee, as having originally existed in a +remote progenitor as a boring and serrated instrument, like that in so +many members of the same great order, and which has been modified but +not perfected for its present purpose, with the poison originally +adapted to cause galls subsequently intensified, we can perhaps +understand how it is that the use of the sting should so often cause +the insect’s own death: for if on the whole the power of stinging be +useful to the community, it will fulfil all the requirements of natural +selection, though it may cause the death of some few members. If we +admire the truly wonderful power of scent by which the males of many +insects find their females, can we admire the production for this +single purpose of thousands of drones, which are utterly useless to the +community for any other end, and which are ultimately slaughtered by +their industrious and sterile sisters? It may be difficult, but we +ought to admire the savage instinctive hatred of the queen-bee, which +urges her instantly to destroy the +young queens her daughters as soon as born, or to perish herself in the +combat; for undoubtedly this is for the good of the community; and +maternal love or maternal hatred, though the latter fortunately is most +rare, is all the same to the inexorable principle of natural selection. +If we admire the several ingenious contrivances, by which the flowers +of the orchis and of many other plants are fertilised through insect +agency, can we consider as equally perfect the elaboration by our +fir-trees of dense clouds of pollen, in order that a few granules may +be wafted by a chance breeze on to the ovules? + +_Summary of Chapter_.—We have in this chapter discussed some of the +difficulties and objections which may be urged against my theory. Many +of them are very grave; but I think that in the discussion light has +been thrown on several facts, which on the theory of independent acts +of creation are utterly obscure. We have seen that species at any one +period are not indefinitely variable, and are not linked together by a +multitude of intermediate gradations, partly because the process of +natural selection will always be very slow, and will act, at any one +time, only on a very few forms; and partly because the very process of +natural selection almost implies the continual supplanting and +extinction of preceding and intermediate gradations. Closely allied +species, now living on a continuous area, must often have been formed +when the area was not continuous, and when the conditions of life did +not insensibly graduate away from one part to another. When two +varieties are formed in two districts of a continuous area, an +intermediate variety will often be formed, fitted for an intermediate +zone; but from reasons assigned, the intermediate variety will usually +exist in lesser numbers than +the two forms which it connects; consequently the two latter, during +the course of further modification, from existing in greater numbers, +will have a great advantage over the less numerous intermediate +variety, and will thus generally succeed in supplanting and +exterminating it. + +We have seen in this chapter how cautious we should be in concluding +that the most different habits of life could not graduate into each +other; that a bat, for instance, could not have been formed by natural +selection from an animal which at first could only glide through the +air. + +We have seen that a species may under new conditions of life change its +habits, or have diversified habits, with some habits very unlike those +of its nearest congeners. Hence we can understand, bearing in mind that +each organic being is trying to live wherever it can live, how it has +arisen that there are upland geese with webbed feet, ground +woodpeckers, diving thrushes, and petrels with the habits of auks. + +Although the belief that an organ so perfect as the eye could have been +formed by natural selection, is more than enough to stagger any one; +yet in the case of any organ, if we know of a long series of gradations +in complexity, each good for its possessor, then, under changing +conditions of life, there is no logical impossibility in the +acquirement of any conceivable degree of perfection through natural +selection. In the cases in which we know of no intermediate or +transitional states, we should be very cautious in concluding that none +could have existed, for the homologies of many organs and their +intermediate states show that wonderful metamorphoses in function are +at least possible. For instance, a swim-bladder has apparently been +converted into an air-breathing lung. The same organ having performed +simultaneously very different functions, and then having been +specialised for one function; and two very distinct organs having +performed at the same time the same function, the one having been +perfected whilst aided by the other, must often have largely +facilitated transitions. + +We are far too ignorant, in almost every case, to be enabled to assert +that any part or organ is so unimportant for the welfare of a species, +that modifications in its structure could not have been slowly +accumulated by means of natural selection. But we may confidently +believe that many modifications, wholly due to the laws of growth, and +at first in no way advantageous to a species, have been subsequently +taken advantage of by the still further modified descendants of this +species. We may, also, believe that a part formerly of high importance +has often been retained (as the tail of an aquatic animal by its +terrestrial descendants), though it has become of such small importance +that it could not, in its present state, have been acquired by natural +selection,—a power which acts solely by the preservation of profitable +variations in the struggle for life. + +Natural selection will produce nothing in one species for the exclusive +good or injury of another; though it may well produce parts, organs, +and excretions highly useful or even indispensable, or highly injurious +to another species, but in all cases at the same time useful to the +owner. Natural selection in each well-stocked country, must act chiefly +through the competition of the inhabitants one with another, and +consequently will produce perfection, or strength in the battle for +life, only according to the standard of that country. Hence the +inhabitants of one country, generally the smaller one, will often +yield, as we see they do yield, to the inhabitants of another and +generally larger country. For in +the larger country there will have existed more individuals, and more +diversified forms, and the competition will have been severer, and thus +the standard of perfection will have been rendered higher. Natural +selection will not necessarily produce absolute perfection; nor, as far +as we can judge by our limited faculties, can absolute perfection be +everywhere found. + +On the theory of natural selection we can clearly understand the full +meaning of that old canon in natural history, “Natura non facit +saltum.” This canon, if we look only to the present inhabitants of the +world, is not strictly correct, but if we include all those of past +times, it must by my theory be strictly true. + +It is generally acknowledged that all organic beings have been formed +on two great laws—Unity of Type, and the Conditions of Existence. By +unity of type is meant that fundamental agreement in structure, which +we see in organic beings of the same class, and which is quite +independent of their habits of life. On my theory, unity of type is +explained by unity of descent. The expression of conditions of +existence, so often insisted on by the illustrious Cuvier, is fully +embraced by the principle of natural selection. For natural selection +acts by either now adapting the varying parts of each being to its +organic and inorganic conditions of life; or by having adapted them +during long-past periods of time: the adaptations being aided in some +cases by use and disuse, being slightly affected by the direct action +of the external conditions of life, and being in all cases subjected to +the several laws of growth. Hence, in fact, the law of the Conditions +of Existence is the higher law; as it includes, through the inheritance +of former adaptations, that of Unity of Type. + +Instincts comparable with habits, but different in their origin. +Instincts graduated. Aphides and ants. Instincts variable. Domestic +instincts, their origin. Natural instincts of the cuckoo, ostrich, and +parasitic bees. Slave-making ants. Hive-bee, its cell-making instinct. +Difficulties on the theory of the Natural Selection of instincts. +Neuter or sterile insects. Summary. + +The subject of instinct might have been worked into the previous +chapters; but I have thought that it would be more convenient to treat +the subject separately, especially as so wonderful an instinct as that +of the hive-bee making its cells will probably have occurred to many +readers, as a difficulty sufficient to overthrow my whole theory. I +must premise, that I have nothing to do with the origin of the primary +mental powers, any more than I have with that of life itself. We are +concerned only with the diversities of instinct and of the other mental +qualities of animals within the same class. + +I will not attempt any definition of instinct. It would be easy to show +that several distinct mental actions are commonly embraced by this +term; but every one understands what is meant, when it is said that +instinct impels the cuckoo to migrate and to lay her eggs in other +birds’ nests. An action, which we ourselves should require experience +to enable us to perform, when performed by an animal, more especially +by a very young one, without any experience, and when performed by many +individuals in the same way, without their knowing for what purpose it +is performed, is usually said to be instinctive. +But I could show that none of these characters of instinct are +universal. A little dose, as Pierre Huber expresses it, of judgment or +reason, often comes into play, even in animals very low in the scale of +nature. + +Frederick Cuvier and several of the older metaphysicians have compared +instinct with habit. This comparison gives, I think, a remarkably +accurate notion of the frame of mind under which an instinctive action +is performed, but not of its origin. How unconsciously many habitual +actions are performed, indeed not rarely in direct opposition to our +conscious will! yet they may be modified by the will or reason. Habits +easily become associated with other habits, and with certain periods of +time and states of the body. When once acquired, they often remain +constant throughout life. Several other points of resemblance between +instincts and habits could be pointed out. As in repeating a well-known +song, so in instincts, one action follows another by a sort of rhythm; +if a person be interrupted in a song, or in repeating anything by rote, +he is generally forced to go back to recover the habitual train of +thought: so P. Huber found it was with a caterpillar, which makes a +very complicated hammock; for if he took a caterpillar which had +completed its hammock up to, say, the sixth stage of construction, and +put it into a hammock completed up only to the third stage, the +caterpillar simply re-performed the fourth, fifth, and sixth stages of +construction. If, however, a caterpillar were taken out of a hammock +made up, for instance, to the third stage, and were put into one +finished up to the sixth stage, so that much of its work was already +done for it, far from feeling the benefit of this, it was much +embarrassed, and, in order to complete its hammock, seemed forced to +start from the third stage, where it had left off, and thus tried to +complete the already finished work. + +If we suppose any habitual action to become inherited—and I think it +can be shown that this does sometimes happen—then the resemblance +between what originally was a habit and an instinct becomes so close as +not to be distinguished. If Mozart, instead of playing the pianoforte +at three years old with wonderfully little practice, had played a tune +with no practice at all, he might truly be said to have done so +instinctively. But it would be the most serious error to suppose that +the greater number of instincts have been acquired by habit in one +generation, and then transmitted by inheritance to succeeding +generations. It can be clearly shown that the most wonderful instincts +with which we are acquainted, namely, those of the hive-bee and of many +ants, could not possibly have been thus acquired. + +It will be universally admitted that instincts are as important as +corporeal structure for the welfare of each species, under its present +conditions of life. Under changed conditions of life, it is at least +possible that slight modifications of instinct might be profitable to a +species; and if it can be shown that instincts do vary ever so little, +then I can see no difficulty in natural selection preserving and +continually accumulating variations of instinct to any extent that may +be profitable. It is thus, as I believe, that all the most complex and +wonderful instincts have originated. As modifications of corporeal +structure arise from, and are increased by, use or habit, and are +diminished or lost by disuse, so I do not doubt it has been with +instincts. But I believe that the effects of habit are of quite +subordinate importance to the effects of the natural selection of what +may be called accidental variations of instincts;—that is of variations +produced by the same unknown causes which produce slight deviations of +bodily structure. + +No complex instinct can possibly be produced through +natural selection, except by the slow and gradual accumulation of +numerous, slight, yet profitable, variations. Hence, as in the case of +corporeal structures, we ought to find in nature, not the actual +transitional gradations by which each complex instinct has been +acquired—for these could be found only in the lineal ancestors of each +species—but we ought to find in the collateral lines of descent some +evidence of such gradations; or we ought at least to be able to show +that gradations of some kind are possible; and this we certainly can +do. I have been surprised to find, making allowance for the instincts +of animals having been but little observed except in Europe and North +America, and for no instinct being known amongst extinct species, how +very generally gradations, leading to the most complex instincts, can +be discovered. The canon of “Natura non facit saltum” applies with +almost equal force to instincts as to bodily organs. Changes of +instinct may sometimes be facilitated by the same species having +different instincts at different periods of life, or at different +seasons of the year, or when placed under different circumstances, +etc.; in which case either one or the other instinct might be preserved +by natural selection. And such instances of diversity of instinct in +the same species can be shown to occur in nature. + +Again as in the case of corporeal structure, and conformably with my +theory, the instinct of each species is good for itself, but has never, +as far as we can judge, been produced for the exclusive good of others. +One of the strongest instances of an animal apparently performing an +action for the sole good of another, with which I am acquainted, is +that of aphides voluntarily yielding their sweet excretion to ants: +that they do so voluntarily, the following facts show. I removed all +the ants from a group of about a dozen aphides on a dock-plant, +and prevented their attendance during several hours. After this +interval, I felt sure that the aphides would want to excrete. I watched +them for some time through a lens, but not one excreted; I then tickled +and stroked them with a hair in the same manner, as well as I could, as +the ants do with their antennæ; but not one excreted. Afterwards I +allowed an ant to visit them, and it immediately seemed, by its eager +way of running about, to be well aware what a rich flock it had +discovered; it then began to play with its antennæ on the abdomen first +of one aphis and then of another; and each aphis, as soon as it felt +the antennæ, immediately lifted up its abdomen and excreted a limpid +drop of sweet juice, which was eagerly devoured by the ant. Even the +quite young aphides behaved in this manner, showing that the action was +instinctive, and not the result of experience. But as the excretion is +extremely viscid, it is probably a convenience to the aphides to have +it removed; and therefore probably the aphides do not instinctively +excrete for the sole good of the ants. Although I do not believe that +any animal in the world performs an action for the exclusive good of +another of a distinct species, yet each species tries to take advantage +of the instincts of others, as each takes advantage of the weaker +bodily structure of others. So again, in some few cases, certain +instincts cannot be considered as absolutely perfect; but as details on +this and other such points are not indispensable, they may be here +passed over. + +As some degree of variation in instincts under a state of nature, and +the inheritance of such variations, are indispensable for the action of +natural selection, as many instances as possible ought to have been +here given; but want of space prevents me. I can only assert, that +instincts certainly do vary—for instance, +the migratory instinct, both in extent and direction, and in its total +loss. So it is with the nests of birds, which vary partly in dependence +on the situations chosen, and on the nature and temperature of the +country inhabited, but often from causes wholly unknown to us: Audubon +has given several remarkable cases of differences in nests of the same +species in the northern and southern United States. Fear of any +particular enemy is certainly an instinctive quality, as may be seen in +nestling birds, though it is strengthened by experience, and by the +sight of fear of the same enemy in other animals. But fear of man is +slowly acquired, as I have elsewhere shown, by various animals +inhabiting desert islands; and we may see an instance of this, even in +England, in the greater wildness of all our large birds than of our +small birds; for the large birds have been most persecuted by man. We +may safely attribute the greater wildness of our large birds to this +cause; for in uninhabited islands large birds are not more fearful than +small; and the magpie, so wary in England, is tame in Norway, as is the +hooded crow in Egypt. + +That the general disposition of individuals of the same species, born +in a state of nature, is extremely diversified, can be shown by a +multitude of facts. Several cases also, could be given, of occasional +and strange habits in certain species, which might, if advantageous to +the species, give rise, through natural selection, to quite new +instincts. But I am well aware that these general statements, without +facts given in detail, can produce but a feeble effect on the reader’s +mind. I can only repeat my assurance, that I do not speak without good +evidence. + +The possibility, or even probability, of inherited variations of +instinct in a state of nature will be strengthened by briefly +considering a few cases under +domestication. We shall thus also be enabled to see the respective +parts which habit and the selection of so-called accidental variations +have played in modifying the mental qualities of our domestic animals. +A number of curious and authentic instances could be given of the +inheritance of all shades of disposition and tastes, and likewise of +the oddest tricks, associated with certain frames of mind or periods of +time. But let us look to the familiar case of the several breeds of +dogs: it cannot be doubted that young pointers (I have myself seen a +striking instance) will sometimes point and even back other dogs the +very first time that they are taken out; retrieving is certainly in +some degree inherited by retrievers; and a tendency to run round, +instead of at, a flock of sheep, by shepherd-dogs. I cannot see that +these actions, performed without experience by the young, and in nearly +the same manner by each individual, performed with eager delight by +each breed, and without the end being known,—for the young pointer can +no more know that he points to aid his master, than the white butterfly +knows why she lays her eggs on the leaf of the cabbage,—I cannot see +that these actions differ essentially from true instincts. If we were +to see one kind of wolf, when young and without any training, as soon +as it scented its prey, stand motionless like a statue, and then slowly +crawl forward with a peculiar gait; and another kind of wolf rushing +round, instead of at, a herd of deer, and driving them to a distant +point, we should assuredly call these actions instinctive. Domestic +instincts, as they may be called, are certainly far less fixed or +invariable than natural instincts; but they have been acted on by far +less rigorous selection, and have been transmitted for an incomparably +shorter period, under less fixed conditions of life. + +How strongly these domestic instincts, habits, and dispositions +are inherited, and how curiously they become mingled, is well shown +when different breeds of dogs are crossed. Thus it is known that a +cross with a bull-dog has affected for many generations the courage and +obstinacy of greyhounds; and a cross with a greyhound has given to a +whole family of shepherd-dogs a tendency to hunt hares. These domestic +instincts, when thus tested by crossing, resemble natural instincts, +which in a like manner become curiously blended together, and for a +long period exhibit traces of the instincts of either parent: for +example, Le Roy describes a dog, whose great-grandfather was a wolf, +and this dog showed a trace of its wild parentage only in one way, by +not coming in a straight line to his master when called. + +Domestic instincts are sometimes spoken of as actions which have become +inherited solely from long-continued and compulsory habit, but this, I +think, is not true. No one would ever have thought of teaching, or +probably could have taught, the tumbler-pigeon to tumble,—an action +which, as I have witnessed, is performed by young birds, that have +never seen a pigeon tumble. We may believe that some one pigeon showed +a slight tendency to this strange habit, and that the long-continued +selection of the best individuals in successive generations made +tumblers what they now are; and near Glasgow there are house-tumblers, +as I hear from Mr. Brent, which cannot fly eighteen inches high without +going head over heels. It may be doubted whether any one would have +thought of training a dog to point, had not some one dog naturally +shown a tendency in this line; and this is known occasionally to +happen, as I once saw in a pure terrier. When the first tendency was +once displayed, methodical selection and the inherited effects of +compulsory training in each successive generation would soon complete +the work; and unconscious +selection is still at work, as each man tries to procure, without +intending to improve the breed, dogs which will stand and hunt best. On +the other hand, habit alone in some cases has sufficed; no animal is +more difficult to tame than the young of the wild rabbit; scarcely any +animal is tamer than the young of the tame rabbit; but I do not suppose +that domestic rabbits have ever been selected for tameness; and I +presume that we must attribute the whole of the inherited change from +extreme wildness to extreme tameness, simply to habit and +long-continued close confinement. + +Natural instincts are lost under domestication: a remarkable instance +of this is seen in those breeds of fowls which very rarely or never +become “broody,” that is, never wish to sit on their eggs. Familiarity +alone prevents our seeing how universally and largely the minds of our +domestic animals have been modified by domestication. It is scarcely +possible to doubt that the love of man has become instinctive in the +dog. All wolves, foxes, jackals, and species of the cat genus, when +kept tame, are most eager to attack poultry, sheep, and pigs; and this +tendency has been found incurable in dogs which have been brought home +as puppies from countries, such as Tierra del Fuego and Australia, +where the savages do not keep these domestic animals. How rarely, on +the other hand, do our civilised dogs, even when quite young, require +to be taught not to attack poultry, sheep, and pigs! No doubt they +occasionally do make an attack, and are then beaten; and if not cured, +they are destroyed; so that habit, with some degree of selection, has +probably concurred in civilising by inheritance our dogs. On the other +hand, young chickens have lost, wholly by habit, that fear of the dog +and cat which no doubt was originally instinctive in them, in the same +way as it is so plainly instinctive in +young pheasants, though reared under a hen. It is not that chickens +have lost all fear, but fear only of dogs and cats, for if the hen +gives the danger-chuckle, they will run (more especially young turkeys) +from under her, and conceal themselves in the surrounding grass or +thickets; and this is evidently done for the instinctive purpose of +allowing, as we see in wild ground-birds, their mother to fly away. But +this instinct retained by our chickens has become useless under +domestication, for the mother-hen has almost lost by disuse the power +of flight. + +Hence, we may conclude, that domestic instincts have been acquired and +natural instincts have been lost partly by habit, and partly by man +selecting and accumulating during successive generations, peculiar +mental habits and actions, which at first appeared from what we must in +our ignorance call an accident. In some cases compulsory habit alone +has sufficed to produce such inherited mental changes; in other cases +compulsory habit has done nothing, and all has been the result of +selection, pursued both methodically and unconsciously; but in most +cases, probably, habit and selection have acted together. + +We shall, perhaps, best understand how instincts in a state of nature +have become modified by selection, by considering a few cases. I will +select only three, out of the several which I shall have to discuss in +my future work,—namely, the instinct which leads the cuckoo to lay her +eggs in other birds’ nests; the slave-making instinct of certain ants; +and the comb-making power of the hive-bee: these two latter instincts +have generally, and most justly, been ranked by naturalists as the most +wonderful of all known instincts. + +It is now commonly admitted that the more immediate and final cause of +the cuckoo’s instinct is, that +she lays her eggs, not daily, but at intervals of two or three days; so +that, if she were to make her own nest and sit on her own eggs, those +first laid would have to be left for some time unincubated, or there +would be eggs and young birds of different ages in the same nest. If +this were the case, the process of laying and hatching might be +inconveniently long, more especially as she has to migrate at a very +early period; and the first hatched young would probably have to be fed +by the male alone. But the American cuckoo is in this predicament; for +she makes her own nest and has eggs and young successively hatched, all +at the same time. It has been asserted that the American cuckoo +occasionally lays her eggs in other birds’ nests; but I hear on the +high authority of Dr. Brewer, that this is a mistake. Nevertheless, I +could give several instances of various birds which have been known +occasionally to lay their eggs in other birds’ nests. Now let us +suppose that the ancient progenitor of our European cuckoo had the +habits of the American cuckoo; but that occasionally she laid an egg in +another bird’s nest. If the old bird profited by this occasional habit, +or if the young were made more vigorous by advantage having been taken +of the mistaken maternal instinct of another bird, than by their own +mother’s care, encumbered as she can hardly fail to be by having eggs +and young of different ages at the same time; then the old birds or the +fostered young would gain an advantage. And analogy would lead me to +believe, that the young thus reared would be apt to follow by +inheritance the occasional and aberrant habit of their mother, and in +their turn would be apt to lay their eggs in other birds’ nests, and +thus be successful in rearing their young. By a continued process of +this nature, I believe that the strange instinct of our cuckoo could +be, and has been, +generated. I may add that, according to Dr. Gray and to some other +observers, the European cuckoo has not utterly lost all maternal love +and care for her own offspring. + +The occasional habit of birds laying their eggs in other birds’ nests, +either of the same or of a distinct species, is not very uncommon with +the Gallinaceæ; and this perhaps explains the origin of a singular +instinct in the allied group of ostriches. For several hen ostriches, +at least in the case of the American species, unite and lay first a few +eggs in one nest and then in another; and these are hatched by the +males. This instinct may probably be accounted for by the fact of the +hens laying a large number of eggs; but, as in the case of the cuckoo, +at intervals of two or three days. This instinct, however, of the +American ostrich has not as yet been perfected; for a surprising number +of eggs lie strewed over the plains, so that in one day’s hunting I +picked up no less than twenty lost and wasted eggs. + +Many bees are parasitic, and always lay their eggs in the nests of bees +of other kinds. This case is more remarkable than that of the cuckoo; +for these bees have not only their instincts but their structure +modified in accordance with their parasitic habits; for they do not +possess the pollen-collecting apparatus which would be necessary if +they had to store food for their own young. Some species, likewise, of +Sphegidæ (wasp-like insects) are parasitic on other species; and M. +Fabre has lately shown good reason for believing that although the +Tachytes nigra generally makes its own burrow and stores it with +paralysed prey for its own larvæ to feed on, yet that when this insect +finds a burrow already made and stored by another sphex, it takes +advantage of the prize, and becomes for the occasion parasitic. In this +case, as with the supposed case of the cuckoo, I can +see no difficulty in natural selection making an occasional habit +permanent, if of advantage to the species, and if the insect whose nest +and stored food are thus feloniously appropriated, be not thus +exterminated. + +_Slave-making instinct_.—This remarkable instinct was first discovered +in the Formica (Polyerges) rufescens by Pierre Huber, a better observer +even than his celebrated father. This ant is absolutely dependent on +its slaves; without their aid, the species would certainly become +extinct in a single year. The males and fertile females do no work. The +workers or sterile females, though most energetic and courageous in +capturing slaves, do no other work. They are incapable of making their +own nests, or of feeding their own larvæ. When the old nest is found +inconvenient, and they have to migrate, it is the slaves which +determine the migration, and actually carry their masters in their +jaws. So utterly helpless are the masters, that when Huber shut up +thirty of them without a slave, but with plenty of the food which they +like best, and with their larvæ and pupæ to stimulate them to work, +they did nothing; they could not even feed themselves, and many +perished of hunger. Huber then introduced a single slave (F. fusca), +and she instantly set to work, fed and saved the survivors; made some +cells and tended the larvæ, and put all to rights. What can be more +extraordinary than these well-ascertained facts? If we had not known of +any other slave-making ant, it would have been hopeless to have +speculated how so wonderful an instinct could have been perfected. + +Formica sanguinea was likewise first discovered by P. Huber to be a +slave-making ant. This species is found in the southern parts of +England, and its habits have been attended to by Mr. F. Smith, of the +British +Museum, to whom I am much indebted for information on this and other +subjects. Although fully trusting to the statements of Huber and Mr. +Smith, I tried to approach the subject in a sceptical frame of mind, as +any one may well be excused for doubting the truth of so extraordinary +and odious an instinct as that of making slaves. Hence I will give the +observations which I have myself made, in some little detail. I opened +fourteen nests of F. sanguinea, and found a few slaves in all. Males +and fertile females of the slave-species are found only in their own +proper communities, and have never been observed in the nests of F. +sanguinea. The slaves are black and not above half the size of their +red masters, so that the contrast in their appearance is very great. +When the nest is slightly disturbed, the slaves occasionally come out, +and like their masters are much agitated and defend the nest: when the +nest is much disturbed and the larvæ and pupæ are exposed, the slaves +work energetically with their masters in carrying them away to a place +of safety. Hence, it is clear, that the slaves feel quite at home. +During the months of June and July, on three successive years, I have +watched for many hours several nests in Surrey and Sussex, and never +saw a slave either leave or enter a nest. As, during these months, the +slaves are very few in number, I thought that they might behave +differently when more numerous; but Mr. Smith informs me that he has +watched the nests at various hours during May, June and August, both in +Surrey and Hampshire, and has never seen the slaves, though present in +large numbers in August, either leave or enter the nest. Hence he +considers them as strictly household slaves. The masters, on the other +hand, may be constantly seen bringing in materials for the nest, and +food of all kinds. During the present year, however, in the month +of July, I came across a community with an unusually large stock of +slaves, and I observed a few slaves mingled with their masters leaving +the nest, and marching along the same road to a tall Scotch-fir-tree, +twenty-five yards distant, which they ascended together, probably in +search of aphides or cocci. According to Huber, who had ample +opportunities for observation, in Switzerland the slaves habitually +work with their masters in making the nest, and they alone open and +close the doors in the morning and evening; and, as Huber expressly +states, their principal office is to search for aphides. This +difference in the usual habits of the masters and slaves in the two +countries, probably depends merely on the slaves being captured in +greater numbers in Switzerland than in England. + +One day I fortunately chanced to witness a migration from one nest to +another, and it was a most interesting spectacle to behold the masters +carefully carrying, as Huber has described, their slaves in their jaws. +Another day my attention was struck by about a score of the +slave-makers haunting the same spot, and evidently not in search of +food; they approached and were vigorously repulsed by an independent +community of the slave species (F. fusca); sometimes as many as three +of these ants clinging to the legs of the slave-making F. sanguinea. +The latter ruthlessly killed their small opponents, and carried their +dead bodies as food to their nest, twenty-nine yards distant; but they +were prevented from getting any pupæ to rear as slaves. I then dug up a +small parcel of the pupæ of F. fusca from another nest, and put them +down on a bare spot near the place of combat; they were eagerly seized, +and carried off by the tyrants, who perhaps fancied that, after all, +they had been victorious in their late combat. + +At the same time I laid on the same place a small parcel of the pupæ of +another species, F. flava, with a few of these little yellow ants still +clinging to the fragments of the nest. This species is sometimes, +though rarely, made into slaves, as has been described by Mr. Smith. +Although so small a species, it is very courageous, and I have seen it +ferociously attack other ants. In one instance I found to my surprise +an independent community of F. flava under a stone beneath a nest of +the slave-making F. sanguinea; and when I had accidentally disturbed +both nests, the little ants attacked their big neighbours with +surprising courage. Now I was curious to ascertain whether F. sanguinea +could distinguish the pupæ of F. fusca, which they habitually make into +slaves, from those of the little and furious F. flava, which they +rarely capture, and it was evident that they did at once distinguish +them: for we have seen that they eagerly and instantly seized the pupæ +of F. fusca, whereas they were much terrified when they came across the +pupæ, or even the earth from the nest of F. flava, and quickly ran +away; but in about a quarter of an hour, shortly after all the little +yellow ants had crawled away, they took heart and carried off the pupæ. + +One evening I visited another community of F. sanguinea, and found a +number of these ants entering their nest, carrying the dead bodies of +F. fusca (showing that it was not a migration) and numerous pupæ. I +traced the returning file burthened with booty, for about forty yards, +to a very thick clump of heath, whence I saw the last individual of F. +sanguinea emerge, carrying a pupa; but I was not able to find the +desolated nest in the thick heath. The nest, however, must have been +close at hand, for two or three individuals of F. fusca were rushing +about in the greatest agitation, and one was +perched motionless with its own pupa in its mouth on the top of a spray +of heath over its ravaged home. + +Such are the facts, though they did not need confirmation by me, in +regard to the wonderful instinct of making slaves. Let it be observed +what a contrast the instinctive habits of F. sanguinea present with +those of the F. rufescens. The latter does not build its own nest, does +not determine its own migrations, does not collect food for itself or +its young, and cannot even feed itself: it is absolutely dependent on +its numerous slaves. Formica sanguinea, on the other hand, possesses +much fewer slaves, and in the early part of the summer extremely few. +The masters determine when and where a new nest shall be formed, and +when they migrate, the masters carry the slaves. Both in Switzerland +and England the slaves seem to have the exclusive care of the larvæ, +and the masters alone go on slave-making expeditions. In Switzerland +the slaves and masters work together, making and bringing materials for +the nest: both, but chiefly the slaves, tend, and milk as it may be +called, their aphides; and thus both collect food for the community. In +England the masters alone usually leave the nest to collect building +materials and food for themselves, their slaves and larvæ. So that the +masters in this country receive much less service from their slaves +than they do in Switzerland. + +By what steps the instinct of F. sanguinea originated I will not +pretend to conjecture. But as ants, which are not slave-makers, will, +as I have seen, carry off pupæ of other species, if scattered near +their nests, it is possible that pupæ originally stored as food might +become developed; and the ants thus unintentionally reared would then +follow their proper instincts, and do what work they could. If their +presence proved useful to the species which had seized them—if it were +more advantageous +to this species to capture workers than to procreate them—the habit of +collecting pupæ originally for food might by natural selection be +strengthened and rendered permanent for the very different purpose of +raising slaves. When the instinct was once acquired, if carried out to +a much less extent even than in our British F. sanguinea, which, as we +have seen, is less aided by its slaves than the same species in +Switzerland, I can see no difficulty in natural selection increasing +and modifying the instinct—always supposing each modification to be of +use to the species—until an ant was formed as abjectly dependent on its +slaves as is the Formica rufescens. + +_Cell-making instinct of the Hive-Bee_.—I will not here enter on minute +details on this subject, but will merely give an outline of the +conclusions at which I have arrived. He must be a dull man who can +examine the exquisite structure of a comb, so beautifully adapted to +its end, without enthusiastic admiration. We hear from mathematicians +that bees have practically solved a recondite problem, and have made +their cells of the proper shape to hold the greatest possible amount of +honey, with the least possible consumption of precious wax in their +construction. It has been remarked that a skilful workman, with fitting +tools and measures, would find it very difficult to make cells of wax +of the true form, though this is perfectly effected by a crowd of bees +working in a dark hive. Grant whatever instincts you please, and it +seems at first quite inconceivable how they can make all the necessary +angles and planes, or even perceive when they are correctly made. But +the difficulty is not nearly so great as it at first appears: all this +beautiful work can be shown, I think, to follow from a few very simple +instincts. + +I was led to investigate this subject by Mr. Waterhouse, who has shown +that the form of the cell stands in close relation to the presence of +adjoining cells; and the following view may, perhaps, be considered +only as a modification of his theory. Let us look to the great +principle of gradation, and see whether Nature does not reveal to us +her method of work. At one end of a short series we have humble-bees, +which use their old cocoons to hold honey, sometimes adding to them +short tubes of wax, and likewise making separate and very irregular +rounded cells of wax. At the other end of the series we have the cells +of the hive-bee, placed in a double layer: each cell, as is well known, +is an hexagonal prism, with the basal edges of its six sides bevelled +so as to join on to a pyramid, formed of three rhombs. These rhombs +have certain angles, and the three which form the pyramidal base of a +single cell on one side of the comb, enter into the composition of the +bases of three adjoining cells on the opposite side. In the series +between the extreme perfection of the cells of the hive-bee and the +simplicity of those of the humble-bee, we have the cells of the Mexican +Melipona domestica, carefully described and figured by Pierre Huber. +The Melipona itself is intermediate in structure between the hive and +humble bee, but more nearly related to the latter: it forms a nearly +regular waxen comb of cylindrical cells, in which the young are +hatched, and, in addition, some large cells of wax for holding honey. +These latter cells are nearly spherical and of nearly equal sizes, and +are aggregated into an irregular mass. But the important point to +notice, is that these cells are always made at that degree of nearness +to each other, that they would have intersected or broken into each +other, if the spheres had been completed; but this is never permitted, +the bees building perfectly flat walls of wax between the spheres +which thus tend to intersect. Hence each cell consists of an outer +spherical portion and of two, three, or more perfectly flat surfaces, +according as the cell adjoins two, three or more other cells. When one +cell comes into contact with three other cells, which, from the spheres +being nearly of the same size, is very frequently and necessarily the +case, the three flat surfaces are united into a pyramid; and this +pyramid, as Huber has remarked, is manifestly a gross imitation of the +three-sided pyramidal basis of the cell of the hive-bee. As in the +cells of the hive-bee, so here, the three plane surfaces in any one +cell necessarily enter into the construction of three adjoining cells. +It is obvious that the Melipona saves wax by this manner of building; +for the flat walls between the adjoining cells are not double, but are +of the same thickness as the outer spherical portions, and yet each +flat portion forms a part of two cells. + +Reflecting on this case, it occurred to me that if the Melipona had +made its spheres at some given distance from each other, and had made +them of equal sizes and had arranged them symmetrically in a double +layer, the resulting structure would probably have been as perfect as +the comb of the hive-bee. Accordingly I wrote to Professor Miller, of +Cambridge, and this geometer has kindly read over the following +statement, drawn up from his information, and tells me that it is +strictly correct:— + +If a number of equal spheres be described with their centres placed in +two parallel layers; with the centre of each sphere at the distance of +radius x the square root of 2 or radius x 1.41421 (or at some lesser +distance), from the centres of the six surrounding spheres in the same +layer; and at the same distance from the centres of the adjoining +spheres in the other and parallel layer; then, if planes of +intersection between the several spheres in +both layers be formed, there will result a double layer of hexagonal +prisms united together by pyramidal bases formed of three rhombs; and +the rhombs and the sides of the hexagonal prisms will have every angle +identically the same with the best measurements which have been made of +the cells of the hive-bee. + +Hence we may safely conclude that if we could slightly modify the +instincts already possessed by the Melipona, and in themselves not very +wonderful, this bee would make a structure as wonderfully perfect as +that of the hive-bee. We must suppose the Melipona to make her cells +truly spherical, and of equal sizes; and this would not be very +surprising, seeing that she already does so to a certain extent, and +seeing what perfectly cylindrical burrows in wood many insects can +make, apparently by turning round on a fixed point. We must suppose the +Melipona to arrange her cells in level layers, as she already does her +cylindrical cells; and we must further suppose, and this is the +greatest difficulty, that she can somehow judge accurately at what +distance to stand from her fellow-labourers when several are making +their spheres; but she is already so far enabled to judge of distance, +that she always describes her spheres so as to intersect largely; and +then she unites the points of intersection by perfectly flat surfaces. +We have further to suppose, but this is no difficulty, that after +hexagonal prisms have been formed by the intersection of adjoining +spheres in the same layer, she can prolong the hexagon to any length +requisite to hold the stock of honey; in the same way as the rude +humble-bee adds cylinders of wax to the circular mouths of her old +cocoons. By such modifications of instincts in themselves not very +wonderful,—hardly more wonderful than those which guide a bird to make +its nest,—I believe that the hive-bee +has acquired, through natural selection, her inimitable architectural +powers. + +But this theory can be tested by experiment. Following the example of +Mr. Tegetmeier, I separated two combs, and put between them a long, +thick, square strip of wax: the bees instantly began to excavate minute +circular pits in it; and as they deepened these little pits, they made +them wider and wider until they were converted into shallow basins, +appearing to the eye perfectly true or parts of a sphere, and of about +the diameter of a cell. It was most interesting to me to observe that +wherever several bees had begun to excavate these basins near together, +they had begun their work at such a distance from each other, that by +the time the basins had acquired the above stated width (_i.e._ about +the width of an ordinary cell), and were in depth about one sixth of +the diameter of the sphere of which they formed a part, the rims of the +basins intersected or broke into each other. As soon as this occurred, +the bees ceased to excavate, and began to build up flat walls of wax on +the lines of intersection between the basins, so that each hexagonal +prism was built upon the festooned edge of a smooth basin, instead of +on the straight edges of a three-sided pyramid as in the case of +ordinary cells. + +I then put into the hive, instead of a thick, square piece of wax, a +thin and narrow, knife-edged ridge, coloured with vermilion. The bees +instantly began on both sides to excavate little basins near to each +other, in the same way as before; but the ridge of wax was so thin, +that the bottoms of the basins, if they had been excavated to the same +depth as in the former experiment, would have broken into each other +from the opposite sides. The bees, however, did not suffer this to +happen, and they stopped their excavations in due +time; so that the basins, as soon as they had been a little deepened, +came to have flat bottoms; and these flat bottoms, formed by thin +little plates of the vermilion wax having been left ungnawed, were +situated, as far as the eye could judge, exactly along the planes of +imaginary intersection between the basins on the opposite sides of the +ridge of wax. In parts, only little bits, in other parts, large +portions of a rhombic plate had been left between the opposed basins, +but the work, from the unnatural state of things, had not been neatly +performed. The bees must have worked at very nearly the same rate on +the opposite sides of the ridge of vermilion wax, as they circularly +gnawed away and deepened the basins on both sides, in order to have +succeeded in thus leaving flat plates between the basins, by stopping +work along the intermediate planes or planes of intersection. + +Considering how flexible thin wax is, I do not see that there is any +difficulty in the bees, whilst at work on the two sides of a strip of +wax, perceiving when they have gnawed the wax away to the proper +thinness, and then stopping their work. In ordinary combs it has +appeared to me that the bees do not always succeed in working at +exactly the same rate from the opposite sides; for I have noticed +half-completed rhombs at the base of a just-commenced cell, which were +slightly concave on one side, where I suppose that the bees had +excavated too quickly, and convex on the opposed side, where the bees +had worked less quickly. In one well-marked instance, I put the comb +back into the hive, and allowed the bees to go on working for a short +time, and again examined the cell, and I found that the rhombic plate +had been completed, and had become _perfectly flat:_ it was absolutely +impossible, from the extreme thinness of the little rhombic plate, that +they could have effected +this by gnawing away the convex side; and I suspect that the bees in +such cases stand in the opposed cells and push and bend the ductile and +warm wax (which as I have tried is easily done) into its proper +intermediate plane, and thus flatten it. + +From the experiment of the ridge of vermilion wax, we can clearly see +that if the bees were to build for themselves a thin wall of wax, they +could make their cells of the proper shape, by standing at the proper +distance from each other, by excavating at the same rate, and by +endeavouring to make equal spherical hollows, but never allowing the +spheres to break into each other. Now bees, as may be clearly seen by +examining the edge of a growing comb, do make a rough, circumferential +wall or rim all round the comb; and they gnaw into this from the +opposite sides, always working circularly as they deepen each cell. +They do not make the whole three-sided pyramidal base of any one cell +at the same time, but only the one rhombic plate which stands on the +extreme growing margin, or the two plates, as the case may be; and they +never complete the upper edges of the rhombic plates, until the +hexagonal walls are commenced. Some of these statements differ from +those made by the justly celebrated elder Huber, but I am convinced of +their accuracy; and if I had space, I could show that they are +conformable with my theory. + +Huber’s statement that the very first cell is excavated out of a little +parallel-sided wall of wax, is not, as far as I have seen, strictly +correct; the first commencement having always been a little hood of +wax; but I will not here enter on these details. We see how important a +part excavation plays in the construction of the cells; but it would be +a great error to suppose that the bees cannot build up a rough wall of +wax in the proper +position—that is, along the plane of intersection between two adjoining +spheres. I have several specimens showing clearly that they can do +this. Even in the rude circumferential rim or wall of wax round a +growing comb, flexures may sometimes be observed, corresponding in +position to the planes of the rhombic basal plates of future cells. But +the rough wall of wax has in every case to be finished off, by being +largely gnawed away on both sides. The manner in which the bees build +is curious; they always make the first rough wall from ten to twenty +times thicker than the excessively thin finished wall of the cell, +which will ultimately be left. We shall understand how they work, by +supposing masons first to pile up a broad ridge of cement, and then to +begin cutting it away equally on both sides near the ground, till a +smooth, very thin wall is left in the middle; the masons always piling +up the cut-away cement, and adding fresh cement, on the summit of the +ridge. We shall thus have a thin wall steadily growing upward; but +always crowned by a gigantic coping. From all the cells, both those +just commenced and those completed, being thus crowned by a strong +coping of wax, the bees can cluster and crawl over the comb without +injuring the delicate hexagonal walls, which are only about one +four-hundredth of an inch in thickness; the plates of the pyramidal +basis being about twice as thick. By this singular manner of building, +strength is continually given to the comb, with the utmost ultimate +economy of wax. + +It seems at first to add to the difficulty of understanding how the +cells are made, that a multitude of bees all work together; one bee +after working a short time at one cell going to another, so that, as +Huber has stated, a score of individuals work even at the commencement +of the first cell. I was able practically to show this fact, by +covering the edges of the hexagonal walls +of a single cell, or the extreme margin of the circumferential rim of a +growing comb, with an extremely thin layer of melted vermilion wax; and +I invariably found that the colour was most delicately diffused by the +bees—as delicately as a painter could have done with his brush—by atoms +of the coloured wax having been taken from the spot on which it had +been placed, and worked into the growing edges of the cells all round. +The work of construction seems to be a sort of balance struck between +many bees, all instinctively standing at the same relative distance +from each other, all trying to sweep equal spheres, and then building +up, or leaving ungnawed, the planes of intersection between these +spheres. It was really curious to note in cases of difficulty, as when +two pieces of comb met at an angle, how often the bees would entirely +pull down and rebuild in different ways the same cell, sometimes +recurring to a shape which they had at first rejected. + +When bees have a place on which they can stand in their proper +positions for working,—for instance, on a slip of wood, placed directly +under the middle of a comb growing downwards so that the comb has to be +built over one face of the slip—in this case the bees can lay the +foundations of one wall of a new hexagon, in its strictly proper place, +projecting beyond the other completed cells. It suffices that the bees +should be enabled to stand at their proper relative distances from each +other and from the walls of the last completed cells, and then, by +striking imaginary spheres, they can build up a wall intermediate +between two adjoining spheres; but, as far as I have seen, they never +gnaw away and finish off the angles of a cell till a large part both of +that cell and of the adjoining cells has been built. This capacity in +bees of laying down under certain circumstances a rough wall in its +proper place between two just-commenced +cells, is important, as it bears on a fact, which seems at first quite +subversive of the foregoing theory; namely, that the cells on the +extreme margin of wasp-combs are sometimes strictly hexagonal; but I +have not space here to enter on this subject. Nor does there seem to me +any great difficulty in a single insect (as in the case of a +queen-wasp) making hexagonal cells, if she work alternately on the +inside and outside of two or three cells commenced at the same time, +always standing at the proper relative distance from the parts of the +cells just begun, sweeping spheres or cylinders, and building up +intermediate planes. It is even conceivable that an insect might, by +fixing on a point at which to commence a cell, and then moving outside, +first to one point, and then to five other points, at the proper +relative distances from the central point and from each other, strike +the planes of intersection, and so make an isolated hexagon: but I am +not aware that any such case has been observed; nor would any good be +derived from a single hexagon being built, as in its construction more +materials would be required than for a cylinder. + +As natural selection acts only by the accumulation of slight +modifications of structure or instinct, each profitable to the +individual under its conditions of life, it may reasonably be asked, +how a long and graduated succession of modified architectural +instincts, all tending towards the present perfect plan of +construction, could have profited the progenitors of the hive-bee? I +think the answer is not difficult: it is known that bees are often hard +pressed to get sufficient nectar; and I am informed by Mr. Tegetmeier +that it has been experimentally found that no less than from twelve to +fifteen pounds of dry sugar are consumed by a hive of bees for the +secretion of each pound of wax; so that a prodigious quantity of fluid +nectar must be collected and consumed by the bees in a hive for +the secretion of the wax necessary for the construction of their combs. +Moreover, many bees have to remain idle for many days during the +process of secretion. A large store of honey is indispensable to +support a large stock of bees during the winter; and the security of +the hive is known mainly to depend on a large number of bees being +supported. Hence the saving of wax by largely saving honey must be a +most important element of success in any family of bees. Of course the +success of any species of bee may be dependent on the number of its +parasites or other enemies, or on quite distinct causes, and so be +altogether independent of the quantity of honey which the bees could +collect. But let us suppose that this latter circumstance determined, +as it probably often does determine, the numbers of a humble-bee which +could exist in a country; and let us further suppose that the community +lived throughout the winter, and consequently required a store of +honey: there can in this case be no doubt that it would be an advantage +to our humble-bee, if a slight modification of her instinct led her to +make her waxen cells near together, so as to intersect a little; for a +wall in common even to two adjoining cells, would save some little wax. +Hence it would continually be more and more advantageous to our +humble-bee, if she were to make her cells more and more regular, nearer +together, and aggregated into a mass, like the cells of the Melipona; +for in this case a large part of the bounding surface of each cell +would serve to bound other cells, and much wax would be saved. Again, +from the same cause, it would be advantageous to the Melipona, if she +were to make her cells closer together, and more regular in every way +than at present; for then, as we have seen, the spherical surfaces +would wholly disappear, and would all be replaced by plane surfaces; +and the Melipona +would make a comb as perfect as that of the hive-bee. Beyond this stage +of perfection in architecture, natural selection could not lead; for +the comb of the hive-bee, as far as we can see, is absolutely perfect +in economising wax. + +Thus, as I believe, the most wonderful of all known instincts, that of +the hive-bee, can be explained by natural selection having taken +advantage of numerous, successive, slight modifications of simpler +instincts; natural selection having by slow degrees, more and more +perfectly, led the bees to sweep equal spheres at a given distance from +each other in a double layer, and to build up and excavate the wax +along the planes of intersection. The bees, of course, no more knowing +that they swept their spheres at one particular distance from each +other, than they know what are the several angles of the hexagonal +prisms and of the basal rhombic plates. The motive power of the process +of natural selection having been economy of wax; that individual swarm +which wasted least honey in the secretion of wax, having succeeded +best, and having transmitted by inheritance its newly acquired +economical instinct to new swarms, which in their turn will have had +the best chance of succeeding in the struggle for existence. + +No doubt many instincts of very difficult explanation could be opposed +to the theory of natural selection,—cases, in which we cannot see how +an instinct could possibly have originated; cases, in which no +intermediate gradations are known to exist; cases of instinct of +apparently such trifling importance, that they could hardly have been +acted on by natural selection; cases of instincts almost identically +the same in animals so remote in the scale of nature, that we cannot +account +for their similarity by inheritance from a common parent, and must +therefore believe that they have been acquired by independent acts of +natural selection. I will not here enter on these several cases, but +will confine myself to one special difficulty, which at first appeared +to me insuperable, and actually fatal to my whole theory. I allude to +the neuters or sterile females in insect-communities: for these neuters +often differ widely in instinct and in structure from both the males +and fertile females, and yet, from being sterile, they cannot propagate +their kind. + +The subject well deserves to be discussed at great length, but I will +here take only a single case, that of working or sterile ants. How the +workers have been rendered sterile is a difficulty; but not much +greater than that of any other striking modification of structure; for +it can be shown that some insects and other articulate animals in a +state of nature occasionally become sterile; and if such insects had +been social, and it had been profitable to the community that a number +should have been annually born capable of work, but incapable of +procreation, I can see no very great difficulty in this being effected +by natural selection. But I must pass over this preliminary difficulty. +The great difficulty lies in the working ants differing widely from +both the males and the fertile females in structure, as in the shape of +the thorax and in being destitute of wings and sometimes of eyes, and +in instinct. As far as instinct alone is concerned, the prodigious +difference in this respect between the workers and the perfect females, +would have been far better exemplified by the hive-bee. If a working +ant or other neuter insect had been an animal in the ordinary state, I +should have unhesitatingly assumed that all its characters had been +slowly acquired through natural selection; namely, by an individual +having been born with some slight profitable modification of structure, +this being inherited by its offspring, which again varied and were +again selected, and so onwards. But with the working ant we have an +insect differing greatly from its parents, yet absolutely sterile; so +that it could never have transmitted successively acquired +modifications of structure or instinct to its progeny. It may well be +asked how is it possible to reconcile this case with the theory of +natural selection? + +First, let it be remembered that we have innumerable instances, both in +our domestic productions and in those in a state of nature, of all +sorts of differences of structure which have become correlated to +certain ages, and to either sex. We have differences correlated not +only to one sex, but to that short period alone when the reproductive +system is active, as in the nuptial plumage of many birds, and in the +hooked jaws of the male salmon. We have even slight differences in the +horns of different breeds of cattle in relation to an artificially +imperfect state of the male sex; for oxen of certain breeds have longer +horns than in other breeds, in comparison with the horns of the bulls +or cows of these same breeds. Hence I can see no real difficulty in any +character having become correlated with the sterile condition of +certain members of insect-communities: the difficulty lies in +understanding how such correlated modifications of structure could have +been slowly accumulated by natural selection. + +This difficulty, though appearing insuperable, is lessened, or, as I +believe, disappears, when it is remembered that selection may be +applied to the family, as well as to the individual, and may thus gain +the desired end. Thus, a well-flavoured vegetable is cooked, and the +individual is destroyed; but the horticulturist sows seeds of the same +stock, and confidently expects to +get nearly the same variety; breeders of cattle wish the flesh and fat +to be well marbled together; the animal has been slaughtered, but the +breeder goes with confidence to the same family. I have such faith in +the powers of selection, that I do not doubt that a breed of cattle, +always yielding oxen with extraordinarily long horns, could be slowly +formed by carefully watching which individual bulls and cows, when +matched, produced oxen with the longest horns; and yet no one ox could +ever have propagated its kind. Thus I believe it has been with social +insects: a slight modification of structure, or instinct, correlated +with the sterile condition of certain members of the community, has +been advantageous to the community: consequently the fertile males and +females of the same community flourished, and transmitted to their +fertile offspring a tendency to produce sterile members having the same +modification. And I believe that this process has been repeated, until +that prodigious amount of difference between the fertile and sterile +females of the same species has been produced, which we see in many +social insects. + +But we have not as yet touched on the climax of the difficulty; namely, +the fact that the neuters of several ants differ, not only from the +fertile females and males, but from each other, sometimes to an almost +incredible degree, and are thus divided into two or even three castes. +The castes, moreover, do not generally graduate into each other, but +are perfectly well defined; being as distinct from each other, as are +any two species of the same genus, or rather as any two genera of the +same family. Thus in Eciton, there are working and soldier neuters, +with jaws and instincts extraordinarily different: in Cryptocerus, the +workers of one caste alone carry a wonderful sort of shield on their +heads, the use of which is quite unknown: in the Mexican Myrmecocystus, +the workers of one caste never leave the nest; they are fed by the +workers of another caste, and they have an enormously developed abdomen +which secretes a sort of honey, supplying the place of that excreted by +the aphides, or the domestic cattle as they may be called, which our +European ants guard or imprison. + +It will indeed be thought that I have an overweening confidence in the +principle of natural selection, when I do not admit that such wonderful +and well-established facts at once annihilate my theory. In the simpler +case of neuter insects all of one caste or of the same kind, which have +been rendered by natural selection, as I believe to be quite possible, +different from the fertile males and females,—in this case, we may +safely conclude from the analogy of ordinary variations, that each +successive, slight, profitable modification did not probably at first +appear in all the individual neuters in the same nest, but in a few +alone; and that by the long-continued selection of the fertile parents +which produced most neuters with the profitable modification, all the +neuters ultimately came to have the desired character. On this view we +ought occasionally to find neuter-insects of the same species, in the +same nest, presenting gradations of structure; and this we do find, +even often, considering how few neuter-insects out of Europe have been +carefully examined. Mr. F. Smith has shown how surprisingly the neuters +of several British ants differ from each other in size and sometimes in +colour; and that the extreme forms can sometimes be perfectly linked +together by individuals taken out of the same nest: I have myself +compared perfect gradations of this kind. It often happens that the +larger or the smaller sized workers are the most numerous; or that both +large and small are numerous, with those of an intermediate size scanty +in numbers. Formica flava has larger and +smaller workers, with some of intermediate size; and, in this species, +as Mr. F. Smith has observed, the larger workers have simple eyes +(ocelli), which though small can be plainly distinguished, whereas the +smaller workers have their ocelli rudimentary. Having carefully +dissected several specimens of these workers, I can affirm that the +eyes are far more rudimentary in the smaller workers than can be +accounted for merely by their proportionally lesser size; and I fully +believe, though I dare not assert so positively, that the workers of +intermediate size have their ocelli in an exactly intermediate +condition. So that we here have two bodies of sterile workers in the +same nest, differing not only in size, but in their organs of vision, +yet connected by some few members in an intermediate condition. I may +digress by adding, that if the smaller workers had been the most useful +to the community, and those males and females had been continually +selected, which produced more and more of the smaller workers, until +all the workers had come to be in this condition; we should then have +had a species of ant with neuters very nearly in the same condition +with those of Myrmica. For the workers of Myrmica have not even +rudiments of ocelli, though the male and female ants of this genus have +well-developed ocelli. + +I may give one other case: so confidently did I expect to find +gradations in important points of structure between the different +castes of neuters in the same species, that I gladly availed myself of +Mr. F. Smith’s offer of numerous specimens from the same nest of the +driver ant (Anomma) of West Africa. The reader will perhaps best +appreciate the amount of difference in these workers, by my giving not +the actual measurements, but a strictly accurate illustration: the +difference was the same as if we were to see a set of workmen building +a house of whom many were five feet four inches high, and many sixteen +feet high; but we must suppose that the larger workmen had heads four +instead of three times as big as those of the smaller men, and jaws +nearly five times as big. The jaws, moreover, of the working ants of +the several sizes differed wonderfully in shape, and in the form and +number of the teeth. But the important fact for us is, that though the +workers can be grouped into castes of different sizes, yet they +graduate insensibly into each other, as does the widely-different +structure of their jaws. I speak confidently on this latter point, as +Mr. Lubbock made drawings for me with the camera lucida of the jaws +which I had dissected from the workers of the several sizes. + +With these facts before me, I believe that natural selection, by acting +on the fertile parents, could form a species which should regularly +produce neuters, either all of large size with one form of jaw, or all +of small size with jaws having a widely different structure; or lastly, +and this is our climax of difficulty, one set of workers of one size +and structure, and simultaneously another set of workers of a different +size and structure;—a graduated series having been first formed, as in +the case of the driver ant, and then the extreme forms, from being the +most useful to the community, having been produced in greater and +greater numbers through the natural selection of the parents which +generated them; until none with an intermediate structure were +produced. + +Thus, as I believe, the wonderful fact of two distinctly defined castes +of sterile workers existing in the same nest, both widely different +from each other and from their parents, has originated. We can see how +useful their production may have been to a social community of insects, +on the same principle that the division of +labour is useful to civilised man. As ants work by inherited instincts +and by inherited tools or weapons, and not by acquired knowledge and +manufactured instruments, a perfect division of labour could be +effected with them only by the workers being sterile; for had they been +fertile, they would have intercrossed, and their instincts and +structure would have become blended. And nature has, as I believe, +effected this admirable division of labour in the communities of ants, +by the means of natural selection. But I am bound to confess, that, +with all my faith in this principle, I should never have anticipated +that natural selection could have been efficient in so high a degree, +had not the case of these neuter insects convinced me of the fact. I +have, therefore, discussed this case, at some little but wholly +insufficient length, in order to show the power of natural selection, +and likewise because this is by far the most serious special +difficulty, which my theory has encountered. The case, also, is very +interesting, as it proves that with animals, as with plants, any amount +of modification in structure can be effected by the accumulation of +numerous, slight, and as we must call them accidental, variations, +which are in any manner profitable, without exercise or habit having +come into play. For no amount of exercise, or habit, or volition, in +the utterly sterile members of a community could possibly have affected +the structure or instincts of the fertile members, which alone leave +descendants. I am surprised that no one has advanced this demonstrative +case of neuter insects, against the well-known doctrine of Lamarck. + +_Summary_.—I have endeavoured briefly in this chapter to show that the +mental qualities of our domestic animals vary, and that the variations +are inherited. Still more briefly I have attempted to show that +instincts +vary slightly in a state of nature. No one will dispute that instincts +are of the highest importance to each animal. Therefore I can see no +difficulty, under changing conditions of life, in natural selection +accumulating slight modifications of instinct to any extent, in any +useful direction. In some cases habit or use and disuse have probably +come into play. I do not pretend that the facts given in this chapter +strengthen in any great degree my theory; but none of the cases of +difficulty, to the best of my judgment, annihilate it. On the other +hand, the fact that instincts are not always absolutely perfect and are +liable to mistakes;—that no instinct has been produced for the +exclusive good of other animals, but that each animal takes advantage +of the instincts of others;—that the canon in natural history, of +“natura non facit saltum” is applicable to instincts as well as to +corporeal structure, and is plainly explicable on the foregoing views, +but is otherwise inexplicable,—all tend to corroborate the theory of +natural selection. + +This theory is, also, strengthened by some few other facts in regard to +instincts; as by that common case of closely allied, but certainly +distinct, species, when inhabiting distant parts of the world and +living under considerably different conditions of life, yet often +retaining nearly the same instincts. For instance, we can understand on +the principle of inheritance, how it is that the thrush of South +America lines its nest with mud, in the same peculiar manner as does +our British thrush: how it is that the male wrens (Troglodytes) of +North America, build “cock-nests,” to roost in, like the males of our +distinct Kitty-wrens,—a habit wholly unlike that of any other known +bird. Finally, it may not be a logical deduction, but to my imagination +it is far more satisfactory to look at such instincts as the young +cuckoo ejecting its foster-brothers,—ants making slaves,—the larvæ of +ichneumonidæ feeding within the live bodies of caterpillars,—not as +specially endowed or created instincts, but as small consequences of +one general law, leading to the advancement of all organic beings, +namely, multiply, vary, let the strongest live and the weakest die. + +Distinction between the sterility of first crosses and of hybrids. +Sterility various in degree, not universal, affected by close +interbreeding, removed by domestication. Laws governing the sterility +of hybrids. Sterility not a special endowment, but incidental on other +differences. Causes of the sterility of first crosses and of hybrids. +Parallelism between the effects of changed conditions of life and +crossing. Fertility of varieties when crossed and of their mongrel +offspring not universal. Hybrids and mongrels compared independently of +their fertility. Summary. + +The view generally entertained by naturalists is that species, when +intercrossed, have been specially endowed with the quality of +sterility, in order to prevent the confusion of all organic forms. This +view certainly seems at first probable, for species within the same +country could hardly have kept distinct had they been capable of +crossing freely. The importance of the fact that hybrids are very +generally sterile, has, I think, been much underrated by some late +writers. On the theory of natural selection the case is especially +important, inasmuch as the sterility of hybrids could not possibly be +of any advantage to them, and therefore could not have been acquired by +the continued preservation of successive profitable degrees of +sterility. I hope, however, to be able to show that sterility is not a +specially acquired or endowed quality, but is incidental on other +acquired differences. + +In treating this subject, two classes of facts, to a large extent +fundamentally different, have generally been confounded together; +namely, the sterility of two +species when first crossed, and the sterility of the hybrids produced +from them. + +Pure species have of course their organs of reproduction in a perfect +condition, yet when intercrossed they produce either few or no +offspring. Hybrids, on the other hand, have their reproductive organs +functionally impotent, as may be clearly seen in the state of the male +element in both plants and animals; though the organs themselves are +perfect in structure, as far as the microscope reveals. In the first +case the two sexual elements which go to form the embryo are perfect; +in the second case they are either not at all developed, or are +imperfectly developed. This distinction is important, when the cause of +the sterility, which is common to the two cases, has to be considered. +The distinction has probably been slurred over, owing to the sterility +in both cases being looked on as a special endowment, beyond the +province of our reasoning powers. + +The fertility of varieties, that is of the forms known or believed to +have descended from common parents, when intercrossed, and likewise the +fertility of their mongrel offspring, is, on my theory, of equal +importance with the sterility of species; for it seems to make a broad +and clear distinction between varieties and species. + +First, for the sterility of species when crossed and of their hybrid +offspring. It is impossible to study the several memoirs and works of +those two conscientious and admirable observers, Kölreuter and Gärtner, +who almost devoted their lives to this subject, without being deeply +impressed with the high generality of some degree of sterility. +Kölreuter makes the rule universal; but then he cuts the knot, for in +ten cases in which he found two forms, considered by most authors as +distinct species, quite fertile together, he +unhesitatingly ranks them as varieties. Gärtner, also, makes the rule +equally universal; and he disputes the entire fertility of Kölreuter’s +ten cases. But in these and in many other cases, Gärtner is obliged +carefully to count the seeds, in order to show that there is any degree +of sterility. He always compares the maximum number of seeds produced +by two species when crossed and by their hybrid offspring, with the +average number produced by both pure parent-species in a state of +nature. But a serious cause of error seems to me to be here introduced: +a plant to be hybridised must be castrated, and, what is often more +important, must be secluded in order to prevent pollen being brought to +it by insects from other plants. Nearly all the plants experimentised +on by Gärtner were potted, and apparently were kept in a chamber in his +house. That these processes are often injurious to the fertility of a +plant cannot be doubted; for Gärtner gives in his table about a score +of cases of plants which he castrated, and artificially fertilised with +their own pollen, and (excluding all cases such as the Leguminosæ, in +which there is an acknowledged difficulty in the manipulation) half of +these twenty plants had their fertility in some degree impaired. +Moreover, as Gärtner during several years repeatedly crossed the +primrose and cowslip, which we have such good reason to believe to be +varieties, and only once or twice succeeded in getting fertile seed; as +he found the common red and blue pimpernels (Anagallis arvensis and +coerulea), which the best botanists rank as varieties, absolutely +sterile together; and as he came to the same conclusion in several +other analogous cases; it seems to me that we may well be permitted to +doubt whether many other species are really so sterile, when +intercrossed, as Gärtner believes. + +It is certain, on the one hand, that the sterility of various species +when crossed is so different in degree and graduates away so +insensibly, and, on the other hand, that the fertility of pure species +is so easily affected by various circumstances, that for all practical +purposes it is most difficult to say where perfect fertility ends and +sterility begins. I think no better evidence of this can be required +than that the two most experienced observers who have ever lived, +namely, Kölreuter and Gärtner, should have arrived at diametrically +opposite conclusions in regard to the very same species. It is also +most instructive to compare—but I have not space here to enter on +details—the evidence advanced by our best botanists on the question +whether certain doubtful forms should be ranked as species or +varieties, with the evidence from fertility adduced by different +hybridisers, or by the same author, from experiments made during +different years. It can thus be shown that neither sterility nor +fertility affords any clear distinction between species and varieties; +but that the evidence from this source graduates away, and is doubtful +in the same degree as is the evidence derived from other constitutional +and structural differences. + +In regard to the sterility of hybrids in successive generations; though +Gärtner was enabled to rear some hybrids, carefully guarding them from +a cross with either pure parent, for six or seven, and in one case for +ten generations, yet he asserts positively that their fertility never +increased, but generally greatly decreased. I do not doubt that this is +usually the case, and that the fertility often suddenly decreases in +the first few generations. Nevertheless I believe that in all these +experiments the fertility has been diminished by an independent cause, +namely, from close interbreeding. I have collected so large a body of +facts, showing +that close interbreeding lessens fertility, and, on the other hand, +that an occasional cross with a distinct individual or variety +increases fertility, that I cannot doubt the correctness of this almost +universal belief amongst breeders. Hybrids are seldom raised by +experimentalists in great numbers; and as the parent-species, or other +allied hybrids, generally grow in the same garden, the visits of +insects must be carefully prevented during the flowering season: hence +hybrids will generally be fertilised during each generation by their +own individual pollen; and I am convinced that this would be injurious +to their fertility, already lessened by their hybrid origin. I am +strengthened in this conviction by a remarkable statement repeatedly +made by Gärtner, namely, that if even the less fertile hybrids be +artificially fertilised with hybrid pollen of the same kind, their +fertility, notwithstanding the frequent ill effects of manipulation, +sometimes decidedly increases, and goes on increasing. Now, in +artificial fertilisation pollen is as often taken by chance (as I know +from my own experience) from the anthers of another flower, as from the +anthers of the flower itself which is to be fertilised; so that a cross +between two flowers, though probably on the same plant, would be thus +effected. Moreover, whenever complicated experiments are in progress, +so careful an observer as Gärtner would have castrated his hybrids, and +this would have insured in each generation a cross with the pollen from +a distinct flower, either from the same plant or from another plant of +the same hybrid nature. And thus, the strange fact of the increase of +fertility in the successive generations of _artificially fertilised_ +hybrids may, I believe, be accounted for by close interbreeding having +been avoided. + +Now let us turn to the results arrived at by the third most experienced +hybridiser, namely, the Honourable and +Reverend W. Herbert. He is as emphatic in his conclusion that some +hybrids are perfectly fertile—as fertile as the pure parent-species—as +are Kölreuter and Gärtner that some degree of sterility between +distinct species is a universal law of nature. He experimentised on +some of the very same species as did Gärtner. The difference in their +results may, I think, be in part accounted for by Herbert’s great +horticultural skill, and by his having hothouses at his command. Of his +many important statements I will here give only a single one as an +example, namely, that “every ovule in a pod of Crinum capense +fertilised by C. revolutum produced a plant, which (he says) I never +saw to occur in a case of its natural fecundation.” So that we here +have perfect, or even more than commonly perfect, fertility in a first +cross between two distinct species. + +This case of the Crinum leads me to refer to a most singular fact, +namely, that there are individual plants, as with certain species of +Lobelia, and with all the species of the genus Hippeastrum, which can +be far more easily fertilised by the pollen of another and distinct +species, than by their own pollen. For these plants have been found to +yield seed to the pollen of a distinct species, though quite sterile +with their own pollen, notwithstanding that their own pollen was found +to be perfectly good, for it fertilised distinct species. So that +certain individual plants and all the individuals of certain species +can actually be hybridised much more readily than they can be +self-fertilised! For instance, a bulb of Hippeastrum aulicum produced +four flowers; three were fertilised by Herbert with their own pollen, +and the fourth was subsequently fertilised by the pollen of a compound +hybrid descended from three other and distinct species: the result was +that “the ovaries of the three first flowers soon ceased to grow, and +after a +few days perished entirely, whereas the pod impregnated by the pollen +of the hybrid made vigorous growth and rapid progress to maturity, and +bore good seed, which vegetated freely.” In a letter to me, in 1839, +Mr. Herbert told me that he had then tried the experiment during five +years, and he continued to try it during several subsequent years, and +always with the same result. This result has, also, been confirmed by +other observers in the case of Hippeastrum with its sub-genera, and in +the case of some other genera, as Lobelia, Passiflora and Verbascum. +Although the plants in these experiments appeared perfectly healthy, +and although both the ovules and pollen of the same flower were +perfectly good with respect to other species, yet as they were +functionally imperfect in their mutual self-action, we must infer that +the plants were in an unnatural state. Nevertheless these facts show on +what slight and mysterious causes the lesser or greater fertility of +species when crossed, in comparison with the same species when +self-fertilised, sometimes depends. + +The practical experiments of horticulturists, though not made with +scientific precision, deserve some notice. It is notorious in how +complicated a manner the species of Pelargonium, Fuchsia, Calceolaria, +Petunia, Rhododendron, etc., have been crossed, yet many of these +hybrids seed freely. For instance, Herbert asserts that a hybrid from +Calceolaria integrifolia and plantaginea, species most widely +dissimilar in general habit, “reproduced itself as perfectly as if it +had been a natural species from the mountains of Chile.” I have taken +some pains to ascertain the degree of fertility of some of the complex +crosses of Rhododendrons, and I am assured that many of them are +perfectly fertile. Mr. C. Noble, for instance, informs me that he +raises stocks for grafting from a hybrid +between Rhododendron Ponticum and Catawbiense, and that this hybrid +“seeds as freely as it is possible to imagine.” Had hybrids, when +fairly treated, gone on decreasing in fertility in each successive +generation, as Gärtner believes to be the case, the fact would have +been notorious to nurserymen. Horticulturists raise large beds of the +same hybrids, and such alone are fairly treated, for by insect agency +the several individuals of the same hybrid variety are allowed to +freely cross with each other, and the injurious influence of close +interbreeding is thus prevented. Any one may readily convince himself +of the efficiency of insect-agency by examining the flowers of the more +sterile kinds of hybrid rhododendrons, which produce no pollen, for he +will find on their stigmas plenty of pollen brought from other flowers. + +In regard to animals, much fewer experiments have been carefully tried +than with plants. If our systematic arrangements can be trusted, that +is if the genera of animals are as distinct from each other, as are the +genera of plants, then we may infer that animals more widely separated +in the scale of nature can be more easily crossed than in the case of +plants; but the hybrids themselves are, I think, more sterile. I doubt +whether any case of a perfectly fertile hybrid animal can be considered +as thoroughly well authenticated. It should, however, be borne in mind +that, owing to few animals breeding freely under confinement, few +experiments have been fairly tried: for instance, the canary-bird has +been crossed with nine other finches, but as not one of these nine +species breeds freely in confinement, we have no right to expect that +the first crosses between them and the canary, or that their hybrids, +should be perfectly fertile. Again, with respect to the fertility in +successive generations of the more fertile +hybrid animals, I hardly know of an instance in which two families of +the same hybrid have been raised at the same time from different +parents, so as to avoid the ill effects of close interbreeding. On the +contrary, brothers and sisters have usually been crossed in each +successive generation, in opposition to the constantly repeated +admonition of every breeder. And in this case, it is not at all +surprising that the inherent sterility in the hybrids should have gone +on increasing. If we were to act thus, and pair brothers and sisters in +the case of any pure animal, which from any cause had the least +tendency to sterility, the breed would assuredly be lost in a very few +generations. + +Although I do not know of any thoroughly well-authenticated cases of +perfectly fertile hybrid animals, I have some reason to believe that +the hybrids from Cervulus vaginalis and Reevesii, and from Phasianus +colchicus with P. torquatus and with P. versicolor are perfectly +fertile. The hybrids from the common and Chinese geese (A. cygnoides), +species which are so different that they are generally ranked in +distinct genera, have often bred in this country with either pure +parent, and in one single instance they have bred _inter se_. This was +effected by Mr. Eyton, who raised two hybrids from the same parents but +from different hatches; and from these two birds he raised no less than +eight hybrids (grandchildren of the pure geese) from one nest. In +India, however, these cross-bred geese must be far more fertile; for I +am assured by two eminently capable judges, namely Mr. Blyth and Capt. +Hutton, that whole flocks of these crossed geese are kept in various +parts of the country; and as they are kept for profit, where neither +pure parent-species exists, they must certainly be highly fertile. + +A doctrine which originated with Pallas, has been +largely accepted by modern naturalists; namely, that most of our +domestic animals have descended from two or more aboriginal species, +since commingled by intercrossing. On this view, the aboriginal species +must either at first have produced quite fertile hybrids, or the +hybrids must have become in subsequent generations quite fertile under +domestication. This latter alternative seems to me the most probable, +and I am inclined to believe in its truth, although it rests on no +direct evidence. I believe, for instance, that our dogs have descended +from several wild stocks; yet, with perhaps the exception of certain +indigenous domestic dogs of South America, all are quite fertile +together; and analogy makes me greatly doubt, whether the several +aboriginal species would at first have freely bred together and have +produced quite fertile hybrids. So again there is reason to believe +that our European and the humped Indian cattle are quite fertile +together; but from facts communicated to me by Mr. Blyth, I think they +must be considered as distinct species. On this view of the origin of +many of our domestic animals, we must either give up the belief of the +almost universal sterility of distinct species of animals when crossed; +or we must look at sterility, not as an indelible characteristic, but +as one capable of being removed by domestication. + +Finally, looking to all the ascertained facts on the intercrossing of +plants and animals, it may be concluded that some degree of sterility, +both in first crosses and in hybrids,is an extremely general result; +but that it cannot, under our present state of knowledge, be considered +as absolutely universal. + +_Laws governing the Sterility of first Crosses and of Hybrids_.—We will +now consider a little more in detail the +circumstances and rules governing the sterility of first crosses and of +hybrids. Our chief object will be to see whether or not the rules +indicate that species have specially been endowed with this quality, in +order to prevent their crossing and blending together in utter +confusion. The following rules and conclusions are chiefly drawn up +from Gärtner’s admirable work on the hybridisation of plants. I have +taken much pains to ascertain how far the rules apply to animals, and +considering how scanty our knowledge is in regard to hybrid animals, I +have been surprised to find how generally the same rules apply to both +kingdoms. + +It has been already remarked, that the degree of fertility, both of +first crosses and of hybrids, graduates from zero to perfect fertility. +It is surprising in how many curious ways this gradation can be shown +to exist; but only the barest outline of the facts can here be given. +When pollen from a plant of one family is placed on the stigma of a +plant of a distinct family, it exerts no more influence than so much +inorganic dust. From this absolute zero of fertility, the pollen of +different species of the same genus applied to the stigma of some one +species, yields a perfect gradation in the number of seeds produced, up +to nearly complete or even quite complete fertility; and, as we have +seen, in certain abnormal cases, even to an excess of fertility, beyond +that which the plant’s own pollen will produce. So in hybrids +themselves, there are some which never have produced, and probably +never would produce, even with the pollen of either pure parent, a +single fertile seed: but in some of these cases a first trace of +fertility may be detected, by the pollen of one of the pure +parent-species causing the flower of the hybrid to wither earlier than +it otherwise would have done; and the early withering of the flower is +well known to be a sign +of incipient fertilisation. From this extreme degree of sterility we +have self-fertilised hybrids producing a greater and greater number of +seeds up to perfect fertility. + +Hybrids from two species which are very difficult to cross, and which +rarely produce any offspring, are generally very sterile; but the +parallelism between the difficulty of making a first cross, and the +sterility of the hybrids thus produced—two classes of facts which are +generally confounded together—is by no means strict. There are many +cases, in which two pure species can be united with unusual facility, +and produce numerous hybrid-offspring, yet these hybrids are remarkably +sterile. On the other hand, there are species which can be crossed very +rarely, or with extreme difficulty, but the hybrids, when at last +produced, are very fertile. Even within the limits of the same genus, +for instance in Dianthus, these two opposite cases occur. + +The fertility, both of first crosses and of hybrids, is more easily +affected by unfavourable conditions, than is the fertility of pure +species. But the degree of fertility is likewise innately variable; for +it is not always the same when the same two species are crossed under +the same circumstances, but depends in part upon the constitution of +the individuals which happen to have been chosen for the experiment. So +it is with hybrids, for their degree of fertility is often found to +differ greatly in the several individuals raised from seed out of the +same capsule and exposed to exactly the same conditions. + +By the term systematic affinity is meant, the resemblance between +species in structure and in constitution, more especially in the +structure of parts which are of high physiological importance and which +differ little in the allied species. Now the fertility of first crosses +between species, and of the hybrids produced from them, is largely +governed by their systematic affinity. This is clearly shown by hybrids +never having been raised between species ranked by systematists in +distinct families; and on the other hand, by very closely allied +species generally uniting with facility. But the correspondence between +systematic affinity and the facility of crossing is by no means strict. +A multitude of cases could be given of very closely allied species +which will not unite, or only with extreme difficulty; and on the other +hand of very distinct species which unite with the utmost facility. In +the same family there may be a genus, as Dianthus, in which very many +species can most readily be crossed; and another genus, as Silene, in +which the most persevering efforts have failed to produce between +extremely close species a single hybrid. Even within the limits of the +same genus, we meet with this same difference; for instance, the many +species of Nicotiana have been more largely crossed than the species of +almost any other genus; but Gärtner found that N. acuminata, which is +not a particularly distinct species, obstinately failed to fertilise, +or to be fertilised by, no less than eight other species of Nicotiana. +Very many analogous facts could be given. + +No one has been able to point out what kind, or what amount, of +difference in any recognisable character is sufficient to prevent two +species crossing. It can be shown that plants most widely different in +habit and general appearance, and having strongly marked differences in +every part of the flower, even in the pollen, in the fruit, and in the +cotyledons, can be crossed. Annual and perennial plants, deciduous and +evergreen trees, plants inhabiting different stations and fitted for +extremely different climates, can often be crossed with ease. + +By a reciprocal cross between two species, I mean the case, for +instance, of a stallion-horse being first crossed with a female-ass, +and then a male-ass with a mare: these two species may then be said to +have been reciprocally crossed. There is often the widest possible +difference in the facility of making reciprocal crosses. Such cases are +highly important, for they prove that the capacity in any two species +to cross is often completely independent of their systematic affinity, +or of any recognisable difference in their whole organisation. On the +other hand, these cases clearly show that the capacity for crossing is +connected with constitutional differences imperceptible by us, and +confined to the reproductive system. This difference in the result of +reciprocal crosses between the same two species was long ago observed +by Kölreuter. To give an instance: Mirabilis jalappa can easily be +fertilised by the pollen of M. longiflora, and the hybrids thus +produced are sufficiently fertile; but Kölreuter tried more than two +hundred times, during eight following years, to fertilise reciprocally +M. longiflora with the pollen of M. jalappa, and utterly failed. +Several other equally striking cases could be given. Thuret has +observed the same fact with certain sea-weeds or Fuci. Gärtner, +moreover, found that this difference of facility in making reciprocal +crosses is extremely common in a lesser degree. He has observed it even +between forms so closely related (as Matthiola annua and glabra) that +many botanists rank them only as varieties. It is also a remarkable +fact, that hybrids raised from reciprocal crosses, though of course +compounded of the very same two species, the one species having first +been used as the father and then as the mother, generally differ in +fertility in a small, and occasionally in a high degree. + +Several other singular rules could be given from +Gärtner: for instance, some species have a remarkable power of crossing +with other species; other species of the same genus have a remarkable +power of impressing their likeness on their hybrid offspring; but these +two powers do not at all necessarily go together. There are certain +hybrids which instead of having, as is usual, an intermediate character +between their two parents, always closely resemble one of them; and +such hybrids, though externally so like one of their pure +parent-species, are with rare exceptions extremely sterile. So again +amongst hybrids which are usually intermediate in structure between +their parents, exceptional and abnormal individuals sometimes are born, +which closely resemble one of their pure parents; and these hybrids are +almost always utterly sterile, even when the other hybrids raised from +seed from the same capsule have a considerable degree of fertility. +These facts show how completely fertility in the hybrid is independent +of its external resemblance to either pure parent. + +Considering the several rules now given, which govern the fertility of +first crosses and of hybrids, we see that when forms, which must be +considered as good and distinct species, are united, their fertility +graduates from zero to perfect fertility, or even to fertility under +certain conditions in excess. That their fertility, besides being +eminently susceptible to favourable and unfavourable conditions, is +innately variable. That it is by no means always the same in degree in +the first cross and in the hybrids produced from this cross. That the +fertility of hybrids is not related to the degree in which they +resemble in external appearance either parent. And lastly, that the +facility of making a first cross between any two species is not always +governed by their systematic affinity or +degree of resemblance to each other. This latter statement is clearly +proved by reciprocal crosses between the same two species, for +according as the one species or the other is used as the father or the +mother, there is generally some difference, and occasionally the widest +possible difference, in the facility of effecting an union. The +hybrids, moreover, produced from reciprocal crosses often differ in +fertility. + +Now do these complex and singular rules indicate that species have been +endowed with sterility simply to prevent their becoming confounded in +nature? I think not. For why should the sterility be so extremely +different in degree, when various species are crossed, all of which we +must suppose it would be equally important to keep from blending +together? Why should the degree of sterility be innately variable in +the individuals of the same species? Why should some species cross with +facility, and yet produce very sterile hybrids; and other species cross +with extreme difficulty, and yet produce fairly fertile hybrids? Why +should there often be so great a difference in the result of a +reciprocal cross between the same two species? Why, it may even be +asked, has the production of hybrids been permitted? to grant to +species the special power of producing hybrids, and then to stop their +further propagation by different degrees of sterility, not strictly +related to the facility of the first union between their parents, seems +to be a strange arrangement. + +The foregoing rules and facts, on the other hand, appear to me clearly +to indicate that the sterility both of first crosses and of hybrids is +simply incidental or dependent on unknown differences, chiefly in the +reproductive systems, of the species which are crossed. The differences +being of so peculiar and limited a nature, +that, in reciprocal crosses between two species the male sexual element +of the one will often freely act on the female sexual element of the +other, but not in a reversed direction. It will be advisable to explain +a little more fully by an example what I mean by sterility being +incidental on other differences, and not a specially endowed quality. +As the capacity of one plant to be grafted or budded on another is so +entirely unimportant for its welfare in a state of nature, I presume +that no one will suppose that this capacity is a _specially_ endowed +quality, but will admit that it is incidental on differences in the +laws of growth of the two plants. We can sometimes see the reason why +one tree will not take on another, from differences in their rate of +growth, in the hardness of their wood, in the period of the flow or +nature of their sap, etc.; but in a multitude of cases we can assign no +reason whatever. Great diversity in the size of two plants, one being +woody and the other herbaceous, one being evergreen and the other +deciduous, and adaptation to widely different climates, does not always +prevent the two grafting together. As in hybridisation, so with +grafting, the capacity is limited by systematic affinity, for no one +has been able to graft trees together belonging to quite distinct +families; and, on the other hand, closely allied species, and varieties +of the same species, can usually, but not invariably, be grafted with +ease. But this capacity, as in hybridisation, is by no means absolutely +governed by systematic affinity. Although many distinct genera within +the same family have been grafted together, in other cases species of +the same genus will not take on each other. The pear can be grafted far +more readily on the quince, which is ranked as a distinct genus, than +on the apple, which is a member of the same genus. Even different +varieties of the pear take +with different degrees of facility on the quince; so do different +varieties of the apricot and peach on certain varieties of the plum. + +As Gärtner found that there was sometimes an innate difference in +different _individuals_ of the same two species in crossing; so Sagaret +believes this to be the case with different individuals of the same two +species in being grafted together. As in reciprocal crosses, the +facility of effecting an union is often very far from equal, so it +sometimes is in grafting; the common gooseberry, for instance, cannot +be grafted on the currant, whereas the currant will take, though with +difficulty, on the gooseberry. + +We have seen that the sterility of hybrids, which have their +reproductive organs in an imperfect condition, is a very different case +from the difficulty of uniting two pure species, which have their +reproductive organs perfect; yet these two distinct cases run to a +certain extent parallel. Something analogous occurs in grafting; for +Thouin found that three species of Robinia, which seeded freely on +their own roots, and which could be grafted with no great difficulty on +another species, when thus grafted were rendered barren. On the other +hand, certain species of Sorbus, when grafted on other species, yielded +twice as much fruit as when on their own roots. We are reminded by this +latter fact of the extraordinary case of Hippeastrum, Lobelia, etc., +which seeded much more freely when fertilised with the pollen of +distinct species, than when self-fertilised with their own pollen. + +We thus see, that although there is a clear and fundamental difference +between the mere adhesion of grafted stocks, and the union of the male +and female elements in the act of reproduction, yet that there is a +rude degree of parallelism in the results of grafting and +of crossing distinct species. And as we must look at the curious and +complex laws governing the facility with which trees can be grafted on +each other as incidental on unknown differences in their vegetative +systems, so I believe that the still more complex laws governing the +facility of first crosses, are incidental on unknown differences, +chiefly in their reproductive systems. These differences, in both +cases, follow to a certain extent, as might have been expected, +systematic affinity, by which every kind of resemblance and +dissimilarity between organic beings is attempted to be expressed. The +facts by no means seem to me to indicate that the greater or lesser +difficulty of either grafting or crossing together various species has +been a special endowment; although in the case of crossing, the +difficulty is as important for the endurance and stability of specific +forms, as in the case of grafting it is unimportant for their welfare. + +_Causes of the Sterility of first Crosses and of Hybrids_.—We may now +look a little closer at the probable causes of the sterility of first +crosses and of hybrids. These two cases are fundamentally different, +for, as just remarked, in the union of two pure species the male and +female sexual elements are perfect, whereas in hybrids they are +imperfect. Even in first crosses, the greater or lesser difficulty in +effecting a union apparently depends on several distinct causes. There +must sometimes be a physical impossibility in the male element reaching +the ovule, as would be the case with a plant having a pistil too long +for the pollen-tubes to reach the ovarium. It has also been observed +that when pollen of one species is placed on the stigma of a distantly +allied species, though the pollen-tubes protrude, they do not penetrate +the stigmatic surface. Again, the +male element may reach the female element, but be incapable of causing +an embryo to be developed, as seems to have been the case with some of +Thuret’s experiments on Fuci. No explanation can be given of these +facts, any more than why certain trees cannot be grafted on others. +Lastly, an embryo may be developed, and then perish at an early period. +This latter alternative has not been sufficiently attended to; but I +believe, from observations communicated to me by Mr. Hewitt, who has +had great experience in hybridising gallinaceous birds, that the early +death of the embryo is a very frequent cause of sterility in first +crosses. I was at first very unwilling to believe in this view; as +hybrids, when once born, are generally healthy and long-lived, as we +see in the case of the common mule. Hybrids, however, are differently +circumstanced before and after birth: when born and living in a country +where their two parents can live, they are generally placed under +suitable conditions of life. But a hybrid partakes of only half of the +nature and constitution of its mother, and therefore before birth, as +long as it is nourished within its mother’s womb or within the egg or +seed produced by the mother, it may be exposed to conditions in some +degree unsuitable, and consequently be liable to perish at an early +period; more especially as all very young beings seem eminently +sensitive to injurious or unnatural conditions of life. + +In regard to the sterility of hybrids, in which the sexual elements are +imperfectly developed, the case is very different. I have more than +once alluded to a large body of facts, which I have collected, showing +that when animals and plants are removed from their natural conditions, +they are extremely liable to have their reproductive systems seriously +affected. This, in fact, is +the great bar to the domestication of animals. Between the sterility +thus superinduced and that of hybrids, there are many points of +similarity. In both cases the sterility is independent of general +health, and is often accompanied by excess of size or great luxuriance. +In both cases, the sterility occurs in various degrees; in both, the +male element is the most liable to be affected; but sometimes the +female more than the male. In both, the tendency goes to a certain +extent with systematic affinity, for whole groups of animals and plants +are rendered impotent by the same unnatural conditions; and whole +groups of species tend to produce sterile hybrids. On the other hand, +one species in a group will sometimes resist great changes of +conditions with unimpaired fertility; and certain species in a group +will produce unusually fertile hybrids. No one can tell, till he tries, +whether any particular animal will breed under confinement or any plant +seed freely under culture; nor can he tell, till he tries, whether any +two species of a genus will produce more or less sterile hybrids. +Lastly, when organic beings are placed during several generations under +conditions not natural to them, they are extremely liable to vary, +which is due, as I believe, to their reproductive systems having been +specially affected, though in a lesser degree than when sterility +ensues. So it is with hybrids, for hybrids in successive generations +are eminently liable to vary, as every experimentalist has observed. + +Thus we see that when organic beings are placed under new and unnatural +conditions, and when hybrids are produced by the unnatural crossing of +two species, the reproductive system, independently of the general +state of health, is affected by sterility in a very similar manner. In +the one case, the conditions of life have been disturbed, though often +in so slight a degree as to +be inappreciable by us; in the other case, or that of hybrids, the +external conditions have remained the same, but the organisation has +been disturbed by two different structures and constitutions having +been blended into one. For it is scarcely possible that two +organisations should be compounded into one, without some disturbance +occurring in the development, or periodical action, or mutual relation +of the different parts and organs one to another, or to the conditions +of life. When hybrids are able to breed _inter se_, they transmit to +their offspring from generation to generation the same compounded +organisation, and hence we need not be surprised that their sterility, +though in some degree variable, rarely diminishes. + +It must, however, be confessed that we cannot understand, excepting on +vague hypotheses, several facts with respect to the sterility of +hybrids; for instance, the unequal fertility of hybrids produced from +reciprocal crosses; or the increased sterility in those hybrids which +occasionally and exceptionally resemble closely either pure parent. Nor +do I pretend that the foregoing remarks go to the root of the matter: +no explanation is offered why an organism, when placed under unnatural +conditions, is rendered sterile. All that I have attempted to show, is +that in two cases, in some respects allied, sterility is the common +result,—in the one case from the conditions of life having been +disturbed, in the other case from the organisation having been +disturbed by two organisations having been compounded into one. + +It may seem fanciful, but I suspect that a similar parallelism extends +to an allied yet very different class of facts. It is an old and almost +universal belief, founded, I think, on a considerable body of evidence, +that slight changes in the conditions of life are beneficial to all +living things. We see this acted on by +farmers and gardeners in their frequent exchanges of seed, tubers, +etc., from one soil or climate to another, and back again. During the +convalescence of animals, we plainly see that great benefit is derived +from almost any change in the habits of life. Again, both with plants +and animals, there is abundant evidence, that a cross between very +distinct individuals of the same species, that is between members of +different strains or sub-breeds, gives vigour and fertility to the +offspring. I believe, indeed, from the facts alluded to in our fourth +chapter, that a certain amount of crossing is indispensable even with +hermaphrodites; and that close interbreeding continued during several +generations between the nearest relations, especially if these be kept +under the same conditions of life, always induces weakness and +sterility in the progeny. + +Hence it seems that, on the one hand, slight changes in the conditions +of life benefit all organic beings, and on the other hand, that slight +crosses, that is crosses between the males and females of the same +species which have varied and become slightly different, give vigour +and fertility to the offspring. But we have seen that greater changes, +or changes of a particular nature, often render organic beings in some +degree sterile; and that greater crosses, that is crosses between males +and females which have become widely or specifically different, produce +hybrids which are generally sterile in some degree. I cannot persuade +myself that this parallelism is an accident or an illusion. Both series +of facts seem to be connected together by some common but unknown bond, +which is essentially related to the principle of life. + +_Fertility of Varieties when crossed, and of their Mongrel +offspring_.—It may be urged, as a most forcible argument, +that there must be some essential distinction between species and +varieties, and that there must be some error in all the foregoing +remarks, inasmuch as varieties, however much they may differ from each +other in external appearance, cross with perfect facility, and yield +perfectly fertile offspring. I fully admit that this is almost +invariably the case. But if we look to varieties produced under nature, +we are immediately involved in hopeless difficulties; for if two +hitherto reputed varieties be found in any degree sterile together, +they are at once ranked by most naturalists as species. For instance, +the blue and red pimpernel, the primrose and cowslip, which are +considered by many of our best botanists as varieties, are said by +Gärtner not to be quite fertile when crossed, and he consequently ranks +them as undoubted species. If we thus argue in a circle, the fertility +of all varieties produced under nature will assuredly have to be +granted. + +If we turn to varieties, produced, or supposed to have been produced, +under domestication, we are still involved in doubt. For when it is +stated, for instance, that the German Spitz dog unites more easily than +other dogs with foxes, or that certain South American indigenous +domestic dogs do not readily cross with European dogs, the explanation +which will occur to everyone, and probably the true one, is that these +dogs have descended from several aboriginally distinct species. +Nevertheless the perfect fertility of so many domestic varieties, +differing widely from each other in appearance, for instance of the +pigeon or of the cabbage, is a remarkable fact; more especially when we +reflect how many species there are, which, though resembling each other +most closely, are utterly sterile when intercrossed. Several +considerations, however, render the fertility of domestic varieties +less remarkable than +at first appears. It can, in the first place, be clearly shown that +mere external dissimilarity between two species does not determine +their greater or lesser degree of sterility when crossed; and we may +apply the same rule to domestic varieties. In the second place, some +eminent naturalists believe that a long course of domestication tends +to eliminate sterility in the successive generations of hybrids, which +were at first only slightly sterile; and if this be so, we surely ought +not to expect to find sterility both appearing and disappearing under +nearly the same conditions of life. Lastly, and this seems to me by far +the most important consideration, new races of animals and plants are +produced under domestication by man’s methodical and unconscious power +of selection, for his own use and pleasure: he neither wishes to +select, nor could select, slight differences in the reproductive +system, or other constitutional differences correlated with the +reproductive system. He supplies his several varieties with the same +food; treats them in nearly the same manner, and does not wish to alter +their general habits of life. Nature acts uniformly and slowly during +vast periods of time on the whole organisation, in any way which may be +for each creature’s own good; and thus she may, either directly, or +more probably indirectly, through correlation, modify the reproductive +system in the several descendants from any one species. Seeing this +difference in the process of selection, as carried on by man and +nature, we need not be surprised at some difference in the result. + +I have as yet spoken as if the varieties of the same species were +invariably fertile when intercrossed. But it seems to me impossible to +resist the evidence of the existence of a certain amount of sterility +in the few following cases, which I will briefly abstract. The evidence +is at least as good as that from which we believe +in the sterility of a multitude of species. The evidence is, also, +derived from hostile witnesses, who in all other cases consider +fertility and sterility as safe criterions of specific distinction. +Gärtner kept during several years a dwarf kind of maize with yellow +seeds, and a tall variety with red seeds, growing near each other in +his garden; and although these plants have separated sexes, they never +naturally crossed. He then fertilised thirteen flowers of the one with +the pollen of the other; but only a single head produced any seed, and +this one head produced only five grains. Manipulation in this case +could not have been injurious, as the plants have separated sexes. No +one, I believe, has suspected that these varieties of maize are +distinct species; and it is important to notice that the hybrid plants +thus raised were themselves _perfectly_ fertile; so that even Gärtner +did not venture to consider the two varieties as specifically distinct. + +Girou de Buzareingues crossed three varieties of gourd, which like the +maize has separated sexes, and he asserts that their mutual +fertilisation is by so much the less easy as their differences are +greater. How far these experiments may be trusted, I know not; but the +forms experimentised on, are ranked by Sagaret, who mainly founds his +classification by the test of infertility, as varieties. + +The following case is far more remarkable, and seems at first quite +incredible; but it is the result of an astonishing number of +experiments made during many years on nine species of Verbascum, by so +good an observer and so hostile a witness, as Gärtner: namely, that +yellow and white varieties of the same species of Verbascum when +intercrossed produce less seed, than do either coloured varieties when +fertilised with pollen from their own coloured flowers. Moreover, he +asserts that when +yellow and white varieties of one species are crossed with yellow and +white varieties of a _distinct_ species, more seed is produced by the +crosses between the same coloured flowers, than between those which are +differently coloured. Yet these varieties of Verbascum present no other +difference besides the mere colour of the flower; and one variety can +sometimes be raised from the seed of the other. + +From observations which I have made on certain varieties of hollyhock, +I am inclined to suspect that they present analogous facts. + +Kölreuter, whose accuracy has been confirmed by every subsequent +observer, has proved the remarkable fact, that one variety of the +common tobacco is more fertile, when crossed with a widely distinct +species, than are the other varieties. He experimentised on five forms, +which are commonly reputed to be varieties, and which he tested by the +severest trial, namely, by reciprocal crosses, and he found their +mongrel offspring perfectly fertile. But one of these five varieties, +when used either as father or mother, and crossed with the Nicotiana +glutinosa, always yielded hybrids not so sterile as those which were +produced from the four other varieties when crossed with N. glutinosa. +Hence the reproductive system of this one variety must have been in +some manner and in some degree modified. + +From these facts; from the great difficulty of ascertaining the +infertility of varieties in a state of nature, for a supposed variety +if infertile in any degree would generally be ranked as species; from +man selecting only external characters in the production of the most +distinct domestic varieties, and from not wishing or being able to +produce recondite and functional differences in the reproductive +system; from these several considerations and facts, I do not think +that the very general +fertility of varieties can be proved to be of universal occurrence, or +to form a fundamental distinction between varieties and species. The +general fertility of varieties does not seem to me sufficient to +overthrow the view which I have taken with respect to the very general, +but not invariable, sterility of first crosses and of hybrids, namely, +that it is not a special endowment, but is incidental on slowly +acquired modifications, more especially in the reproductive systems of +the forms which are crossed. + +_Hybrids and Mongrels compared, independently of their +fertility_.—Independently of the question of fertility, the offspring +of species when crossed and of varieties when crossed may be compared +in several other respects. Gärtner, whose strong wish was to draw a +marked line of distinction between species and varieties, could find +very few and, as it seems to me, quite unimportant differences between +the so-called hybrid offspring of species, and the so-called mongrel +offspring of varieties. And, on the other hand, they agree most closely +in very many important respects. + +I shall here discuss this subject with extreme brevity. The most +important distinction is, that in the first generation mongrels are +more variable than hybrids; but Gärtner admits that hybrids from +species which have long been cultivated are often variable in the first +generation; and I have myself seen striking instances of this fact. +Gärtner further admits that hybrids between very closely allied species +are more variable than those from very distinct species; and this shows +that the difference in the degree of variability graduates away. When +mongrels and the more fertile hybrids are propagated for several +generations an extreme amount of variability in their offspring is +notorious; +but some few cases both of hybrids and mongrels long retaining +uniformity of character could be given. The variability, however, in +the successive generations of mongrels is, perhaps, greater than in +hybrids. + +This greater variability of mongrels than of hybrids does not seem to +me at all surprising. For the parents of mongrels are varieties, and +mostly domestic varieties (very few experiments having been tried on +natural varieties), and this implies in most cases that there has been +recent variability; and therefore we might expect that such variability +would often continue and be super-added to that arising from the mere +act of crossing. The slight degree of variability in hybrids from the +first cross or in the first generation, in contrast with their extreme +variability in the succeeding generations, is a curious fact and +deserves attention. For it bears on and corroborates the view which I +have taken on the cause of ordinary variability; namely, that it is due +to the reproductive system being eminently sensitive to any change in +the conditions of life, being thus often rendered either impotent or at +least incapable of its proper function of producing offspring identical +with the parent-form. Now hybrids in the first generation are descended +from species (excluding those long cultivated) which have not had their +reproductive systems in any way affected, and they are not variable; +but hybrids themselves have their reproductive systems seriously +affected, and their descendants are highly variable. + +But to return to our comparison of mongrels and hybrids: Gärtner states +that mongrels are more liable than hybrids to revert to either +parent-form; but this, if it be true, is certainly only a difference in +degree. Gärtner further insists that when any two species, although +most closely allied to each other, are +crossed with a third species, the hybrids are widely different from +each other; whereas if two very distinct varieties of one species are +crossed with another species, the hybrids do not differ much. But this +conclusion, as far as I can make out, is founded on a single +experiment; and seems directly opposed to the results of several +experiments made by Kölreuter. + +These alone are the unimportant differences, which Gärtner is able to +point out, between hybrid and mongrel plants. On the other hand, the +resemblance in mongrels and in hybrids to their respective parents, +more especially in hybrids produced from nearly related species, +follows according to Gärtner the same laws. When two species are +crossed, one has sometimes a prepotent power of impressing its likeness +on the hybrid; and so I believe it to be with varieties of plants. With +animals one variety certainly often has this prepotent power over +another variety. Hybrid plants produced from a reciprocal cross, +generally resemble each other closely; and so it is with mongrels from +a reciprocal cross. Both hybrids and mongrels can be reduced to either +pure parent-form, by repeated crosses in successive generations with +either parent. + +These several remarks are apparently applicable to animals; but the +subject is here excessively complicated, partly owing to the existence +of secondary sexual characters; but more especially owing to prepotency +in transmitting likeness running more strongly in one sex than in the +other, both when one species is crossed with another, and when one +variety is crossed with another variety. For instance, I think those +authors are right, who maintain that the ass has a prepotent power over +the horse, so that both the mule and the hinny more resemble the ass +than the horse; but that the prepotency runs more strongly in the +male-ass than in +the female, so that the mule, which is the offspring of the male-ass +and mare, is more like an ass, than is the hinny, which is the +offspring of the female-ass and stallion. + +Much stress has been laid by some authors on the supposed fact, that +mongrel animals alone are born closely like one of their parents; but +it can be shown that this does sometimes occur with hybrids; yet I +grant much less frequently with hybrids than with mongrels. Looking to +the cases which I have collected of cross-bred animals closely +resembling one parent, the resemblances seem chiefly confined to +characters almost monstrous in their nature, and which have suddenly +appeared—such as albinism, melanism, deficiency of tail or horns, or +additional fingers and toes; and do not relate to characters which have +been slowly acquired by selection. Consequently, sudden reversions to +the perfect character of either parent would be more likely to occur +with mongrels, which are descended from varieties often suddenly +produced and semi-monstrous in character, than with hybrids, which are +descended from species slowly and naturally produced. On the whole I +entirely agree with Dr. Prosper Lucas, who, after arranging an enormous +body of facts with respect to animals, comes to the conclusion, that +the laws of resemblance of the child to its parents are the same, +whether the two parents differ much or little from each other, namely +in the union of individuals of the same variety, or of different +varieties, or of distinct species. + +Laying aside the question of fertility and sterility, in all other +respects there seems to be a general and close similarity in the +offspring of crossed species, and of crossed varieties. If we look at +species as having been specially created, and at varieties as having +been produced by secondary laws, this similarity would be an +astonishing fact. But it harmonises perfectly with the view that there +is no essential distinction between species and varieties. + +_Summary of Chapter_.—First crosses between forms sufficiently distinct +to be ranked as species, and their hybrids, are very generally, but not +universally, sterile. The sterility is of all degrees, and is often so +slight that the two most careful experimentalists who have ever lived, +have come to diametrically opposite conclusions in ranking forms by +this test. The sterility is innately variable in individuals of the +same species, and is eminently susceptible of favourable and +unfavourable conditions. The degree of sterility does not strictly +follow systematic affinity, but is governed by several curious and +complex laws. It is generally different, and sometimes widely +different, in reciprocal crosses between the same two species. It is +not always equal in degree in a first cross and in the hybrid produced +from this cross. + +In the same manner as in grafting trees, the capacity of one species or +variety to take on another, is incidental on generally unknown +differences in their vegetative systems, so in crossing, the greater or +less facility of one species to unite with another, is incidental on +unknown differences in their reproductive systems. There is no more +reason to think that species have been specially endowed with various +degrees of sterility to prevent them crossing and blending in nature, +than to think that trees have been specially endowed with various and +somewhat analogous degrees of difficulty in being grafted together in +order to prevent them becoming inarched in our forests. + +The sterility of first crosses between pure species, which have their +reproductive systems perfect, seems +to depend on several circumstances; in some cases largely on the early +death of the embryo. The sterility of hybrids, which have their +reproductive systems imperfect, and which have had this system and +their whole organisation disturbed by being compounded of two distinct +species, seems closely allied to that sterility which so frequently +affects pure species, when their natural conditions of life have been +disturbed. This view is supported by a parallelism of another +kind;—namely, that the crossing of forms only slightly different is +favourable to the vigour and fertility of their offspring; and that +slight changes in the conditions of life are apparently favourable to +the vigour and fertility of all organic beings. It is not surprising +that the degree of difficulty in uniting two species, and the degree of +sterility of their hybrid-offspring should generally correspond, though +due to distinct causes; for both depend on the amount of difference of +some kind between the species which are crossed. Nor is it surprising +that the facility of effecting a first cross, the fertility of the +hybrids produced, and the capacity of being grafted together—though +this latter capacity evidently depends on widely different +circumstances—should all run, to a certain extent, parallel with the +systematic affinity of the forms which are subjected to experiment; for +systematic affinity attempts to express all kinds of resemblance +between all species. + +First crosses between forms known to be varieties, or sufficiently +alike to be considered as varieties, and their mongrel offspring, are +very generally, but not quite universally, fertile. Nor is this nearly +general and perfect fertility surprising, when we remember how liable +we are to argue in a circle with respect to varieties in a state of +nature; and when we remember that the greater number of varieties have +been produced under domestication +by the selection of mere external differences, and not of differences +in the reproductive system. In all other respects, excluding fertility, +there is a close general resemblance between hybrids and mongrels. +Finally, then, the facts briefly given in this chapter do not seem to +me opposed to, but even rather to support the view, that there is no +fundamental distinction between species and varieties. + +On the absence of intermediate varieties at the present day. On the +nature of extinct intermediate varieties; on their number. On the vast +lapse of time, as inferred from the rate of deposition and of +denudation. On the poorness of our palæontological collections. On the +intermittence of geological formations. On the absence of intermediate +varieties in any one formation. On the sudden appearance of groups of +species. On their sudden appearance in the lowest known fossiliferous +strata. + +In the sixth chapter I enumerated the chief objections which might be +justly urged against the views maintained in this volume. Most of them +have now been discussed. One, namely the distinctness of specific +forms, and their not being blended together by innumerable transitional +links, is a very obvious difficulty. I assigned reasons why such links +do not commonly occur at the present day, under the circumstances +apparently most favourable for their presence, namely on an extensive +and continuous area with graduated physical conditions. I endeavoured +to show, that the life of each species depends in a more important +manner on the presence of other already defined organic forms, than on +climate; and, therefore, that the really governing conditions of life +do not graduate away quite insensibly like heat or moisture. I +endeavoured, also, to show that intermediate varieties, from existing +in lesser numbers than the forms which they connect, will generally be +beaten out and exterminated during the course of further modification +and improvement. The main cause, however, of innumerable intermediate +links not now occurring everywhere throughout nature depends +on the very process of natural selection, through which new varieties +continually take the places of and exterminate their parent-forms. But +just in proportion as this process of extermination has acted on an +enormous scale, so must the number of intermediate varieties, which +have formerly existed on the earth, be truly enormous. Why then is not +every geological formation and every stratum full of such intermediate +links? Geology assuredly does not reveal any such finely graduated +organic chain; and this, perhaps, is the most obvious and gravest +objection which can be urged against my theory. The explanation lies, +as I believe, in the extreme imperfection of the geological record. + +In the first place it should always be borne in mind what sort of +intermediate forms must, on my theory, have formerly existed. I have +found it difficult, when looking at any two species, to avoid picturing +to myself, forms _directly_ intermediate between them. But this is a +wholly false view; we should always look for forms intermediate between +each species and a common but unknown progenitor; and the progenitor +will generally have differed in some respects from all its modified +descendants. To give a simple illustration: the fantail and pouter +pigeons have both descended from the rock-pigeon; if we possessed all +the intermediate varieties which have ever existed, we should have an +extremely close series between both and the rock-pigeon; but we should +have no varieties directly intermediate between the fantail and pouter; +none, for instance, combining a tail somewhat expanded with a crop +somewhat enlarged, the characteristic features of these two breeds. +These two breeds, moreover, have become so much modified, that if we +had no historical or indirect evidence regarding their origin, it would +not have been possible to have +determined from a mere comparison of their structure with that of the +rock-pigeon, whether they had descended from this species or from some +other allied species, such as C. oenas. + +So with natural species, if we look to forms very distinct, for +instance to the horse and tapir, we have no reason to suppose that +links ever existed directly intermediate between them, but between each +and an unknown common parent. The common parent will have had in its +whole organisation much general resemblance to the tapir and to the +horse; but in some points of structure may have differed considerably +from both, even perhaps more than they differ from each other. Hence in +all such cases, we should be unable to recognise the parent-form of any +two or more species, even if we closely compared the structure of the +parent with that of its modified descendants, unless at the same time +we had a nearly perfect chain of the intermediate links. + +It is just possible by my theory, that one of two living forms might +have descended from the other; for instance, a horse from a tapir; and +in this case _direct_ intermediate links will have existed between +them. But such a case would imply that one form had remained for a very +long period unaltered, whilst its descendants had undergone a vast +amount of change; and the principle of competition between organism and +organism, between child and parent, will render this a very rare event; +for in all cases the new and improved forms of life will tend to +supplant the old and unimproved forms. + +By the theory of natural selection all living species have been +connected with the parent-species of each genus, by differences not +greater than we see between the varieties of the same species at the +present +day; and these parent-species, now generally extinct, have in their +turn been similarly connected with more ancient species; and so on +backwards, always converging to the common ancestor of each great +class. So that the number of intermediate and transitional links, +between all living and extinct species, must have been inconceivably +great. But assuredly, if this theory be true, such have lived upon this +earth. + +_On the lapse of Time_.—Independently of our not finding fossil remains +of such infinitely numerous connecting links, it may be objected, that +time will not have sufficed for so great an amount of organic change, +all changes having been effected very slowly through natural selection. +It is hardly possible for me even to recall to the reader, who may not +be a practical geologist, the facts leading the mind feebly to +comprehend the lapse of time. He who can read Sir Charles Lyell’s grand +work on the Principles of Geology, which the future historian will +recognise as having produced a revolution in natural science, yet does +not admit how incomprehensibly vast have been the past periods of time, +may at once close this volume. Not that it suffices to study the +Principles of Geology, or to read special treatises by different +observers on separate formations, and to mark how each author attempts +to give an inadequate idea of the duration of each formation or even +each stratum. A man must for years examine for himself great piles of +superimposed strata, and watch the sea at work grinding down old rocks +and making fresh sediment, before he can hope to comprehend anything of +the lapse of time, the monuments of which we see around us. + +It is good to wander along lines of sea-coast, when formed of +moderately hard rocks, and mark the +process of degradation. The tides in most cases reach the cliffs only +for a short time twice a day, and the waves eat into them only when +they are charged with sand or pebbles; for there is reason to believe +that pure water can effect little or nothing in wearing away rock. At +last the base of the cliff is undermined, huge fragments fall down, and +these remaining fixed, have to be worn away, atom by atom, until +reduced in size they can be rolled about by the waves, and then are +more quickly ground into pebbles, sand, or mud. But how often do we see +along the bases of retreating cliffs rounded boulders, all thickly +clothed by marine productions, showing how little they are abraded and +how seldom they are rolled about! Moreover, if we follow for a few +miles any line of rocky cliff, which is undergoing degradation, we find +that it is only here and there, along a short length or round a +promontory, that the cliffs are at the present time suffering. The +appearance of the surface and the vegetation show that elsewhere years +have elapsed since the waters washed their base. + +He who most closely studies the action of the sea on our shores, will, +I believe, be most deeply impressed with the slowness with which rocky +coasts are worn away. The observations on this head by Hugh Miller, and +by that excellent observer Mr. Smith of Jordan Hill, are most +impressive. With the mind thus impressed, let any one examine beds of +conglomerate many thousand feet in thickness, which, though probably +formed at a quicker rate than many other deposits, yet, from being +formed of worn and rounded pebbles, each of which bears the stamp of +time, are good to show how slowly the mass has been accumulated. Let +him remember Lyell’s profound remark, that the thickness and extent of +sedimentary formations +are the result and measure of the degradation which the earth’s crust +has elsewhere suffered. And what an amount of degradation is implied by +the sedimentary deposits of many countries! Professor Ramsay has given +me the maximum thickness, in most cases from actual measurement, in a +few cases from estimate, of each formation in different parts of Great +Britain; and this is the result:— + + Feet + Palæozoic strata (not including igneous beds)...57,154. Secondary + strata................................13,190. Tertiary + strata..................................2,240. + +—making altogether 72,584 feet; that is, very nearly thirteen and +three-quarters British miles. Some of these formations, which are +represented in England by thin beds, are thousands of feet in thickness +on the Continent. Moreover, between each successive formation, we have, +in the opinion of most geologists, enormously long blank periods. So +that the lofty pile of sedimentary rocks in Britain, gives but an +inadequate idea of the time which has elapsed during their +accumulation; yet what time this must have consumed! Good observers +have estimated that sediment is deposited by the great Mississippi +river at the rate of only 600 feet in a hundred thousand years. This +estimate may be quite erroneous; yet, considering over what wide spaces +very fine sediment is transported by the currents of the sea, the +process of accumulation in any one area must be extremely slow. + +But the amount of denudation which the strata have in many places +suffered, independently of the rate of accumulation of the degraded +matter, probably offers the best evidence of the lapse of time. I +remember having been much struck with the evidence of denudation, when +viewing volcanic islands, which have been +worn by the waves and pared all round into perpendicular cliffs of one +or two thousand feet in height; for the gentle slope of the +lava-streams, due to their formerly liquid state, showed at a glance +how far the hard, rocky beds had once extended into the open ocean. The +same story is still more plainly told by faults,—those great cracks +along which the strata have been upheaved on one side, or thrown down +on the other, to the height or depth of thousands of feet; for since +the crust cracked, the surface of the land has been so completely +planed down by the action of the sea, that no trace of these vast +dislocations is externally visible. + +The Craven fault, for instance, extends for upwards of 30 miles, and +along this line the vertical displacement of the strata has varied from +600 to 3000 feet. Professor Ramsay has published an account of a +downthrow in Anglesea of 2300 feet; and he informs me that he fully +believes there is one in Merionethshire of 12,000 feet; yet in these +cases there is nothing on the surface to show such prodigious +movements; the pile of rocks on the one or other side having been +smoothly swept away. The consideration of these facts impresses my mind +almost in the same manner as does the vain endeavour to grapple with +the idea of eternity. + +I am tempted to give one other case, the well-known one of the +denudation of the Weald. Though it must be admitted that the denudation +of the Weald has been a mere trifle, in comparison with that which has +removed masses of our palæozoic strata, in parts ten thousand feet in +thickness, as shown in Professor Ramsay’s masterly memoir on this +subject. Yet it is an admirable lesson to stand on the North Downs and +to look at the distant South Downs; for, remembering that at no great +distance to the west the northern and southern escarpments meet and +close, one can safely picture to +oneself the great dome of rocks which must have covered up the Weald +within so limited a period as since the latter part of the Chalk +formation. The distance from the northern to the southern Downs is +about 22 miles, and the thickness of the several formations is on an +average about 1100 feet, as I am informed by Professor Ramsay. But if, +as some geologists suppose, a range of older rocks underlies the Weald, +on the flanks of which the overlying sedimentary deposits might have +accumulated in thinner masses than elsewhere, the above estimate would +be erroneous; but this source of doubt probably would not greatly +affect the estimate as applied to the western extremity of the +district. If, then, we knew the rate at which the sea commonly wears +away a line of cliff of any given height, we could measure the time +requisite to have denuded the Weald. This, of course, cannot be done; +but we may, in order to form some crude notion on the subject, assume +that the sea would eat into cliffs 500 feet in height at the rate of +one inch in a century. This will at first appear much too small an +allowance; but it is the same as if we were to assume a cliff one yard +in height to be eaten back along a whole line of coast at the rate of +one yard in nearly every twenty-two years. I doubt whether any rock, +even as soft as chalk, would yield at this rate excepting on the most +exposed coasts; though no doubt the degradation of a lofty cliff would +be more rapid from the breakage of the fallen fragments. On the other +hand, I do not believe that any line of coast, ten or twenty miles in +length, ever suffers degradation at the same time along its whole +indented length; and we must remember that almost all strata contain +harder layers or nodules, which from long resisting attrition form a +breakwater at the base. Hence, under ordinary circumstances, I conclude +that for a cliff 500 feet in height, a denudation +of one inch per century for the whole length would be an ample +allowance. At this rate, on the above data, the denudation of the Weald +must have required 306,662,400 years; or say three hundred million +years. + +The action of fresh water on the gently inclined Wealden district, when +upraised, could hardly have been great, but it would somewhat reduce +the above estimate. On the other hand, during oscillations of level, +which we know this area has undergone, the surface may have existed for +millions of years as land, and thus have escaped the action of the sea: +when deeply submerged for perhaps equally long periods, it would, +likewise, have escaped the action of the coast-waves. So that in all +probability a far longer period than 300 million years has elapsed +since the latter part of the Secondary period. + +I have made these few remarks because it is highly important for us to +gain some notion, however imperfect, of the lapse of years. During each +of these years, over the whole world, the land and the water has been +peopled by hosts of living forms. What an infinite number of +generations, which the mind cannot grasp, must have succeeded each +other in the long roll of years! Now turn to our richest geological +museums, and what a paltry display we behold! + +_On the poorness of our Palæontological collections_.—That our +palæontological collections are very imperfect, is admitted by every +one. The remark of that admirable palæontologist, the late Edward +Forbes, should not be forgotten, namely, that numbers of our fossil +species are known and named from single and often broken specimens, or +from a few specimens collected on some one spot. Only a small portion +of the surface of the earth has been geologically explored, and no part +with +sufficient care, as the important discoveries made every year in Europe +prove. No organism wholly soft can be preserved. Shells and bones will +decay and disappear when left on the bottom of the sea, where sediment +is not accumulating. I believe we are continually taking a most +erroneous view, when we tacitly admit to ourselves that sediment is +being deposited over nearly the whole bed of the sea, at a rate +sufficiently quick to embed and preserve fossil remains. Throughout an +enormously large proportion of the ocean, the bright blue tint of the +water bespeaks its purity. The many cases on record of a formation +conformably covered, after an enormous interval of time, by another and +later formation, without the underlying bed having suffered in the +interval any wear and tear, seem explicable only on the view of the +bottom of the sea not rarely lying for ages in an unaltered condition. +The remains which do become embedded, if in sand or gravel, will when +the beds are upraised generally be dissolved by the percolation of +rain-water. I suspect that but few of the very many animals which live +on the beach between high and low watermark are preserved. For +instance, the several species of the Chthamalinæ (a sub-family of +sessile cirripedes) coat the rocks all over the world in infinite +numbers: they are all strictly littoral, with the exception of a single +Mediterranean species, which inhabits deep water and has been found +fossil in Sicily, whereas not one other species has hitherto been found +in any tertiary formation: yet it is now known that the genus +Chthamalus existed during the chalk period. The molluscan genus Chiton +offers a partially analogous case. + +With respect to the terrestrial productions which lived during the +Secondary and Palæozoic periods, it is superfluous to state that our +evidence from fossil +remains is fragmentary in an extreme degree. For instance, not a land +shell is known belonging to either of these vast periods, with one +exception discovered by Sir C. Lyell in the carboniferous strata of +North America. In regard to mammiferous remains, a single glance at the +historical table published in the Supplement to Lyell’s Manual, will +bring home the truth, how accidental and rare is their preservation, +far better than pages of detail. Nor is their rarity surprising, when +we remember how large a proportion of the bones of tertiary mammals +have been discovered either in caves or in lacustrine deposits; and +that not a cave or true lacustrine bed is known belonging to the age of +our secondary or palæozoic formations. + +But the imperfection in the geological record mainly results from +another and more important cause than any of the foregoing; namely, +from the several formations being separated from each other by wide +intervals of time. When we see the formations tabulated in written +works, or when we follow them in nature, it is difficult to avoid +believing that they are closely consecutive. But we know, for instance, +from Sir R. Murchison’s great work on Russia, what wide gaps there are +in that country between the superimposed formations; so it is in North +America, and in many other parts of the world. The most skilful +geologist, if his attention had been exclusively confined to these +large territories, would never have suspected that during the periods +which were blank and barren in his own country, great piles of +sediment, charged with new and peculiar forms of life, had elsewhere +been accumulated. And if in each separate territory, hardly any idea +can be formed of the length of time which has elapsed between the +consecutive formations, we may infer that this could nowhere be +ascertained. The frequent +and great changes in the mineralogical composition of consecutive +formations, generally implying great changes in the geography of the +surrounding lands, whence the sediment has been derived, accords with +the belief of vast intervals of time having elapsed between each +formation. + +But we can, I think, see why the geological formations of each region +are almost invariably intermittent; that is, have not followed each +other in close sequence. Scarcely any fact struck me more when +examining many hundred miles of the South American coasts, which have +been upraised several hundred feet within the recent period, than the +absence of any recent deposits sufficiently extensive to last for even +a short geological period. Along the whole west coast, which is +inhabited by a peculiar marine fauna, tertiary beds are so scantily +developed, that no record of several successive and peculiar marine +faunas will probably be preserved to a distant age. A little reflection +will explain why along the rising coast of the western side of South +America, no extensive formations with recent or tertiary remains can +anywhere be found, though the supply of sediment must for ages have +been great, from the enormous degradation of the coast-rocks and from +muddy streams entering the sea. The explanation, no doubt, is, that the +littoral and sub-littoral deposits are continually worn away, as soon +as they are brought up by the slow and gradual rising of the land +within the grinding action of the coast-waves. + +We may, I think, safely conclude that sediment must be accumulated in +extremely thick, solid, or extensive masses, in order to withstand the +incessant action of the waves, when first upraised and during +subsequent oscillations of level. Such thick and extensive +accumulations of sediment may be formed in two ways; either, +in profound depths of the sea, in which case, judging from the +researches of E. Forbes, we may conclude that the bottom will be +inhabited by extremely few animals, and the mass when upraised will +give a most imperfect record of the forms of life which then existed; +or, sediment may be accumulated to any thickness and extent over a +shallow bottom, if it continue slowly to subside. In this latter case, +as long as the rate of subsidence and supply of sediment nearly balance +each other, the sea will remain shallow and favourable for life, and +thus a fossiliferous formation thick enough, when upraised, to resist +any amount of degradation, may be formed. + +I am convinced that all our ancient formations, which are rich in +fossils, have thus been formed during subsidence. Since publishing my +views on this subject in 1845, I have watched the progress of Geology, +and have been surprised to note how author after author, in treating of +this or that great formation, has come to the conclusion that it was +accumulated during subsidence. I may add, that the only ancient +tertiary formation on the west coast of South America, which has been +bulky enough to resist such degradation as it has as yet suffered, but +which will hardly last to a distant geological age, was certainly +deposited during a downward oscillation of level, and thus gained +considerable thickness. + +All geological facts tell us plainly that each area has undergone +numerous slow oscillations of level, and apparently these oscillations +have affected wide spaces. Consequently formations rich in fossils and +sufficiently thick and extensive to resist subsequent degradation, may +have been formed over wide spaces during periods of subsidence, but +only where the supply of sediment was sufficient to keep the sea +shallow and to embed and +preserve the remains before they had time to decay. On the other hand, +as long as the bed of the sea remained stationary, _thick_ deposits +could not have been accumulated in the shallow parts, which are the +most favourable to life. Still less could this have happened during the +alternate periods of elevation; or, to speak more accurately, the beds +which were then accumulated will have been destroyed by being upraised +and brought within the limits of the coast-action. + +Thus the geological record will almost necessarily be rendered +intermittent. I feel much confidence in the truth of these views, for +they are in strict accordance with the general principles inculcated by +Sir C. Lyell; and E. Forbes independently arrived at a similar +conclusion. + +One remark is here worth a passing notice. During periods of elevation +the area of the land and of the adjoining shoal parts of the sea will +be increased, and new stations will often be formed;—all circumstances +most favourable, as previously explained, for the formation of new +varieties and species; but during such periods there will generally be +a blank in the geological record. On the other hand, during subsidence, +the inhabited area and number of inhabitants will decrease (excepting +the productions on the shores of a continent when first broken up into +an archipelago), and consequently during subsidence, though there will +be much extinction, fewer new varieties or species will be formed; and +it is during these very periods of subsidence, that our great deposits +rich in fossils have been accumulated. Nature may almost be said to +have guarded against the frequent discovery of her transitional or +linking forms. + +From the foregoing considerations it cannot be doubted that the +geological record, viewed as a whole, is extremely imperfect; but if we +confine our attention to any one formation, it becomes more difficult +to understand, +why we do not therein find closely graduated varieties between the +allied species which lived at its commencement and at its close. Some +cases are on record of the same species presenting distinct varieties +in the upper and lower parts of the same formation, but, as they are +rare, they may be here passed over. Although each formation has +indisputably required a vast number of years for its deposition, I can +see several reasons why each should not include a graduated series of +links between the species which then lived; but I can by no means +pretend to assign due proportional weight to the following +considerations. + +Although each formation may mark a very long lapse of years, each +perhaps is short compared with the period requisite to change one +species into another. I am aware that two palæontologists, whose +opinions are worthy of much deference, namely Bronn and Woodward, have +concluded that the average duration of each formation is twice or +thrice as long as the average duration of specific forms. But +insuperable difficulties, as it seems to me, prevent us coming to any +just conclusion on this head. When we see a species first appearing in +the middle of any formation, it would be rash in the extreme to infer +that it had not elsewhere previously existed. So again when we find a +species disappearing before the uppermost layers have been deposited, +it would be equally rash to suppose that it then became wholly extinct. +We forget how small the area of Europe is compared with the rest of the +world; nor have the several stages of the same formation throughout +Europe been correlated with perfect accuracy. + +With marine animals of all kinds, we may safely infer a large amount of +migration during climatal and other changes; and when we see a species +first appearing in any formation, the probability is that it +only then first immigrated into that area. It is well known, for +instance, that several species appeared somewhat earlier in the +palæozoic beds of North America than in those of Europe; time having +apparently been required for their migration from the American to the +European seas. In examining the latest deposits of various quarters of +the world, it has everywhere been noted, that some few still existing +species are common in the deposit, but have become extinct in the +immediately surrounding sea; or, conversely, that some are now abundant +in the neighbouring sea, but are rare or absent in this particular +deposit. It is an excellent lesson to reflect on the ascertained amount +of migration of the inhabitants of Europe during the Glacial period, +which forms only a part of one whole geological period; and likewise to +reflect on the great changes of level, on the inordinately great change +of climate, on the prodigious lapse of time, all included within this +same glacial period. Yet it may be doubted whether in any quarter of +the world, sedimentary deposits, _including fossil remains_, have gone +on accumulating within the same area during the whole of this period. +It is not, for instance, probable that sediment was deposited during +the whole of the glacial period near the mouth of the Mississippi, +within that limit of depth at which marine animals can flourish; for we +know what vast geographical changes occurred in other parts of America +during this space of time. When such beds as were deposited in shallow +water near the mouth of the Mississippi during some part of the glacial +period shall have been upraised, organic remains will probably first +appear and disappear at different levels, owing to the migration of +species and to geographical changes. And in the distant future, a +geologist examining these beds, might be tempted to conclude that the +average duration of life +of the embedded fossils had been less than that of the glacial period, +instead of having been really far greater, that is extending from +before the glacial epoch to the present day. + +In order to get a perfect gradation between two forms in the upper and +lower parts of the same formation, the deposit must have gone on +accumulating for a very long period, in order to have given sufficient +time for the slow process of variation; hence the deposit will +generally have to be a very thick one; and the species undergoing +modification will have had to live on the same area throughout this +whole time. But we have seen that a thick fossiliferous formation can +only be accumulated during a period of subsidence; and to keep the +depth approximately the same, which is necessary in order to enable the +same species to live on the same space, the supply of sediment must +nearly have counterbalanced the amount of subsidence. But this same +movement of subsidence will often tend to sink the area whence the +sediment is derived, and thus diminish the supply whilst the downward +movement continues. In fact, this nearly exact balancing between the +supply of sediment and the amount of subsidence is probably a rare +contingency; for it has been observed by more than one palæontologist, +that very thick deposits are usually barren of organic remains, except +near their upper or lower limits. + +It would seem that each separate formation, like the whole pile of +formations in any country, has generally been intermittent in its +accumulation. When we see, as is so often the case, a formation +composed of beds of different mineralogical composition, we may +reasonably suspect that the process of deposition has been much +interrupted, as a change in the currents of the sea and a supply of +sediment of a different nature will +generally have been due to geographical changes requiring much time. +Nor will the closest inspection of a formation give any idea of the +time which its deposition has consumed. Many instances could be given +of beds only a few feet in thickness, representing formations, +elsewhere thousands of feet in thickness, and which must have required +an enormous period for their accumulation; yet no one ignorant of this +fact would have suspected the vast lapse of time represented by the +thinner formation. Many cases could be given of the lower beds of a +formation having been upraised, denuded, submerged, and then re-covered +by the upper beds of the same formation,—facts, showing what wide, yet +easily overlooked, intervals have occurred in its accumulation. In +other cases we have the plainest evidence in great fossilised trees, +still standing upright as they grew, of many long intervals of time and +changes of level during the process of deposition, which would never +even have been suspected, had not the trees chanced to have been +preserved: thus, Messrs. Lyell and Dawson found carboniferous beds 1400 +feet thick in Nova Scotia, with ancient root-bearing strata, one above +the other, at no less than sixty-eight different levels. Hence, when +the same species occur at the bottom, middle, and top of a formation, +the probability is that they have not lived on the same spot during the +whole period of deposition, but have disappeared and reappeared, +perhaps many times, during the same geological period. So that if such +species were to undergo a considerable amount of modification during +any one geological period, a section would not probably include all the +fine intermediate gradations which must on my theory have existed +between them, but abrupt, though perhaps very slight, changes of form. + +It is all-important to remember that naturalists have +no golden rule by which to distinguish species and varieties; they +grant some little variability to each species, but when they meet with +a somewhat greater amount of difference between any two forms, they +rank both as species, unless they are enabled to connect them together +by close intermediate gradations. And this from the reasons just +assigned we can seldom hope to effect in any one geological section. +Supposing B and C to be two species, and a third, A, to be found in an +underlying bed; even if A were strictly intermediate between B and C, +it would simply be ranked as a third and distinct species, unless at +the same time it could be most closely connected with either one or +both forms by intermediate varieties. Nor should it be forgotten, as +before explained, that A might be the actual progenitor of B and C, and +yet might not at all necessarily be strictly intermediate between them +in all points of structure. So that we might obtain the parent-species +and its several modified descendants from the lower and upper beds of a +formation, and unless we obtained numerous transitional gradations, we +should not recognise their relationship, and should consequently be +compelled to rank them all as distinct species. + +It is notorious on what excessively slight differences many +palæontologists have founded their species; and they do this the more +readily if the specimens come from different sub-stages of the same +formation. Some experienced conchologists are now sinking many of the +very fine species of D’Orbigny and others into the rank of varieties; +and on this view we do find the kind of evidence of change which on my +theory we ought to find. Moreover, if we look to rather wider +intervals, namely, to distinct but consecutive stages of the same great +formation, we find that the embedded fossils, though almost universally +ranked as specifically different, +yet are far more closely allied to each other than are the species +found in more widely separated formations; but to this subject I shall +have to return in the following chapter. + +One other consideration is worth notice: with animals and plants that +can propagate rapidly and are not highly locomotive, there is reason to +suspect, as we have formerly seen, that their varieties are generally +at first local; and that such local varieties do not spread widely and +supplant their parent-forms until they have been modified and perfected +in some considerable degree. According to this view, the chance of +discovering in a formation in any one country all the early stages of +transition between any two forms, is small, for the successive changes +are supposed to have been local or confined to some one spot. Most +marine animals have a wide range; and we have seen that with plants it +is those which have the widest range, that oftenest present varieties; +so that with shells and other marine animals, it is probably those +which have had the widest range, far exceeding the limits of the known +geological formations of Europe, which have oftenest given rise, first +to local varieties and ultimately to new species; and this again would +greatly lessen the chance of our being able to trace the stages of +transition in any one geological formation. + +It should not be forgotten, that at the present day, with perfect +specimens for examination, two forms can seldom be connected by +intermediate varieties and thus proved to be the same species, until +many specimens have been collected from many places; and in the case of +fossil species this could rarely be effected by palæontologists. We +shall, perhaps, best perceive the improbability of our being enabled to +connect species by numerous, fine, intermediate, fossil links, by +asking +ourselves whether, for instance, geologists at some future period will +be able to prove, that our different breeds of cattle, sheep, horses, +and dogs have descended from a single stock or from several aboriginal +stocks; or, again, whether certain sea-shells inhabiting the shores of +North America, which are ranked by some conchologists as distinct +species from their European representatives, and by other conchologists +as only varieties, are really varieties or are, as it is called, +specifically distinct. This could be effected only by the future +geologist discovering in a fossil state numerous intermediate +gradations; and such success seems to me improbable in the highest +degree. + +Geological research, though it has added numerous species to existing +and extinct genera, and has made the intervals between some few groups +less wide than they otherwise would have been, yet has done scarcely +anything in breaking down the distinction between species, by +connecting them together by numerous, fine, intermediate varieties; and +this not having been effected, is probably the gravest and most obvious +of all the many objections which may be urged against my views. Hence +it will be worth while to sum up the foregoing remarks, under an +imaginary illustration. The Malay Archipelago is of about the size of +Europe from the North Cape to the Mediterranean, and from Britain to +Russia; and therefore equals all the geological formations which have +been examined with any accuracy, excepting those of the United States +of America. I fully agree with Mr. Godwin-Austen, that the present +condition of the Malay Archipelago, with its numerous large islands +separated by wide and shallow seas, probably represents the former +state of Europe, when most of our formations were accumulating. The +Malay Archipelago is one of the richest regions of the +whole world in organic beings; yet if all the species were to be +collected which have ever lived there, how imperfectly would they +represent the natural history of the world! + +But we have every reason to believe that the terrestrial productions of +the archipelago would be preserved in an excessively imperfect manner +in the formations which we suppose to be there accumulating. I suspect +that not many of the strictly littoral animals, or of those which lived +on naked submarine rocks, would be embedded; and those embedded in +gravel or sand, would not endure to a distant epoch. Wherever sediment +did not accumulate on the bed of the sea, or where it did not +accumulate at a sufficient rate to protect organic bodies from decay, +no remains could be preserved. + +In our archipelago, I believe that fossiliferous formations could be +formed of sufficient thickness to last to an age, as distant in +futurity as the secondary formations lie in the past, only during +periods of subsidence. These periods of subsidence would be separated +from each other by enormous intervals, during which the area would be +either stationary or rising; whilst rising, each fossiliferous +formation would be destroyed, almost as soon as accumulated, by the +incessant coast-action, as we now see on the shores of South America. +During the periods of subsidence there would probably be much +extinction of life; during the periods of elevation, there would be +much variation, but the geological record would then be least perfect. + +It may be doubted whether the duration of any one great period of +subsidence over the whole or part of the archipelago, together with a +contemporaneous accumulation of sediment, would _exceed_ the average +duration of the same specific forms; and these contingencies are +indispensable for the preservation of all the transitional gradations +between any two or more species. If such gradations were not fully +preserved, transitional varieties would merely appear as so many +distinct species. It is, also, probable that each great period of +subsidence would be interrupted by oscillations of level, and that +slight climatal changes would intervene during such lengthy periods; +and in these cases the inhabitants of the archipelago would have to +migrate, and no closely consecutive record of their modifications could +be preserved in any one formation. + +Very many of the marine inhabitants of the archipelago now range +thousands of miles beyond its confines; and analogy leads me to believe +that it would be chiefly these far-ranging species which would oftenest +produce new varieties; and the varieties would at first generally be +local or confined to one place, but if possessed of any decided +advantage, or when further modified and improved, they would slowly +spread and supplant their parent-forms. When such varieties returned to +their ancient homes, as they would differ from their former state, in a +nearly uniform, though perhaps extremely slight degree, they would, +according to the principles followed by many palæontologists, be ranked +as new and distinct species. + +If then, there be some degree of truth in these remarks, we have no +right to expect to find in our geological formations, an infinite +number of those fine transitional forms, which on my theory assuredly +have connected all the past and present species of the same group into +one long and branching chain of life. We ought only to look for a few +links, some more closely, some more distantly related to each other; +and these links, let them be ever so close, if found in different +stages of the same formation, would, by most palæontologists, +be ranked as distinct species. But I do not pretend that I should ever +have suspected how poor a record of the mutations of life, the best +preserved geological section presented, had not the difficulty of our +not discovering innumerable transitional links between the species +which appeared at the commencement and close of each formation, pressed +so hardly on my theory. + +_On the sudden appearance of whole groups of Allied Species_.—The +abrupt manner in which whole groups of species suddenly appear in +certain formations, has been urged by several palæontologists, for +instance, by Agassiz, Pictet, and by none more forcibly than by +Professor Sedgwick, as a fatal objection to the belief in the +transmutation of species. If numerous species, belonging to the same +genera or families, have really started into life all at once, the fact +would be fatal to the theory of descent with slow modification through +natural selection. For the development of a group of forms, all of +which have descended from some one progenitor, must have been an +extremely slow process; and the progenitors must have lived long ages +before their modified descendants. But we continually over-rate the +perfection of the geological record, and falsely infer, because certain +genera or families have not been found beneath a certain stage, that +they did not exist before that stage. We continually forget how large +the world is, compared with the area over which our geological +formations have been carefully examined; we forget that groups of +species may elsewhere have long existed and have slowly multiplied +before they invaded the ancient archipelagoes of Europe and of the +United States. We do not make due allowance for the enormous intervals +of time, which have +probably elapsed between our consecutive formations,—longer perhaps in +some cases than the time required for the accumulation of each +formation. These intervals will have given time for the multiplication +of species from some one or some few parent-forms; and in the +succeeding formation such species will appear as if suddenly created. + +I may here recall a remark formerly made, namely that it might require +a long succession of ages to adapt an organism to some new and peculiar +line of life, for instance to fly through the air; but that when this +had been effected, and a few species had thus acquired a great +advantage over other organisms, a comparatively short time would be +necessary to produce many divergent forms, which would be able to +spread rapidly and widely throughout the world. + +I will now give a few examples to illustrate these remarks; and to show +how liable we are to error in supposing that whole groups of species +have suddenly been produced. I may recall the well-known fact that in +geological treatises, published not many years ago, the great class of +mammals was always spoken of as having abruptly come in at the +commencement of the tertiary series. And now one of the richest known +accumulations of fossil mammals belongs to the middle of the secondary +series; and one true mammal has been discovered in the new red +sandstone at nearly the commencement of this great series. Cuvier used +to urge that no monkey occurred in any tertiary stratum; but now +extinct species have been discovered in India, South America, and in +Europe even as far back as the eocene stage. The most striking case, +however, is that of the Whale family; as these animals have huge bones, +are marine, and range over the world, the fact of not a single bone of +a whale having been discovered in +any secondary formation, seemed fully to justify the belief that this +great and distinct order had been suddenly produced in the interval +between the latest secondary and earliest tertiary formation. But now +we may read in the Supplement to Lyell’s ‘Manual,’ published in 1858, +clear evidence of the existence of whales in the upper greensand, some +time before the close of the secondary period. + +I may give another instance, which from having passed under my own eyes +has much struck me. In a memoir on Fossil Sessile Cirripedes, I have +stated that, from the number of existing and extinct tertiary species; +from the extraordinary abundance of the individuals of many species all +over the world, from the Arctic regions to the equator, inhabiting +various zones of depths from the upper tidal limits to 50 fathoms; from +the perfect manner in which specimens are preserved in the oldest +tertiary beds; from the ease with which even a fragment of a valve can +be recognised; from all these circumstances, I inferred that had +sessile cirripedes existed during the secondary periods, they would +certainly have been preserved and discovered; and as not one species +had been discovered in beds of this age, I concluded that this great +group had been suddenly developed at the commencement of the tertiary +series. This was a sore trouble to me, adding as I thought one more +instance of the abrupt appearance of a great group of species. But my +work had hardly been published, when a skilful palæontologist, M. +Bosquet, sent me a drawing of a perfect specimen of an unmistakeable +sessile cirripede, which he had himself extracted from the chalk of +Belgium. And, as if to make the case as striking as possible, this +sessile cirripede was a Chthamalus, a very common, large, and +ubiquitous genus, of which not one specimen has as yet been found even +in any tertiary +stratum. Hence we now positively know that sessile cirripedes existed +during the secondary period; and these cirripedes might have been the +progenitors of our many tertiary and existing species. + +The case most frequently insisted on by palæontologists of the +apparently sudden appearance of a whole group of species, is that of +the teleostean fishes, low down in the Chalk period. This group +includes the large majority of existing species. Lately, Professor +Pictet has carried their existence one sub-stage further back; and some +palæontologists believe that certain much older fishes, of which the +affinities are as yet imperfectly known, are really teleostean. +Assuming, however, that the whole of them did appear, as Agassiz +believes, at the commencement of the chalk formation, the fact would +certainly be highly remarkable; but I cannot see that it would be an +insuperable difficulty on my theory, unless it could likewise be shown +that the species of this group appeared suddenly and simultaneously +throughout the world at this same period. It is almost superfluous to +remark that hardly any fossil-fish are known from south of the equator; +and by running through Pictet’s Palæontology it will be seen that very +few species are known from several formations in Europe. Some few +families of fish now have a confined range; the teleostean fish might +formerly have had a similarly confined range, and after having been +largely developed in some one sea, might have spread widely. Nor have +we any right to suppose that the seas of the world have always been so +freely open from south to north as they are at present. Even at this +day, if the Malay Archipelago were converted into land, the tropical +parts of the Indian Ocean would form a large and perfectly enclosed +basin, in which any great group of marine animals might be multiplied; +and +here they would remain confined, until some of the species became +adapted to a cooler climate, and were enabled to double the southern +capes of Africa or Australia, and thus reach other and distant seas. + +From these and similar considerations, but chiefly from our ignorance +of the geology of other countries beyond the confines of Europe and the +United States; and from the revolution in our palæontological ideas on +many points, which the discoveries of even the last dozen years have +effected, it seems to me to be about as rash in us to dogmatize on the +succession of organic beings throughout the world, as it would be for a +naturalist to land for five minutes on some one barren point in +Australia, and then to discuss the number and range of its productions. + +_On the sudden appearance of groups of Allied Species in the lowest +known fossiliferous strata_.—There is another and allied difficulty, +which is much graver. I allude to the manner in which numbers of +species of the same group, suddenly appear in the lowest known +fossiliferous rocks. Most of the arguments which have convinced me that +all the existing species of the same group have descended from one +progenitor, apply with nearly equal force to the earliest known +species. For instance, I cannot doubt that all the Silurian trilobites +have descended from some one crustacean, which must have lived long +before the Silurian age, and which probably differed greatly from any +known animal. Some of the most ancient Silurian animals, as the +Nautilus, Lingula, etc., do not differ much from living species; and it +cannot on my theory be supposed, that these old species were the +progenitors of all the species of the orders to which they belong, for +they do not present characters in any degree intermediate between them. +If, moreover, they had been the progenitors of these orders, they would +almost certainly have been long ago supplanted and exterminated by +their numerous and improved descendants. + +Consequently, if my theory be true, it is indisputable that before the +lowest Silurian stratum was deposited, long periods elapsed, as long +as, or probably far longer than, the whole interval from the Silurian +age to the present day; and that during these vast, yet quite unknown, +periods of time, the world swarmed with living creatures. + +To the question why we do not find records of these vast primordial +periods, I can give no satisfactory answer. Several of the most eminent +geologists, with Sir R. Murchison at their head, are convinced that we +see in the organic remains of the lowest Silurian stratum the dawn of +life on this planet. Other highly competent judges, as Lyell and the +late E. Forbes, dispute this conclusion. We should not forget that only +a small portion of the world is known with accuracy. M. Barrande has +lately added another and lower stage to the Silurian system, abounding +with new and peculiar species. Traces of life have been detected in the +Longmynd beds beneath Barrande’s so-called primordial zone. The +presence of phosphatic nodules and bituminous matter in some of the +lowest azoic rocks, probably indicates the former existence of life at +these periods. But the difficulty of understanding the absence of vast +piles of fossiliferous strata, which on my theory no doubt were +somewhere accumulated before the Silurian epoch, is very great. If +these most ancient beds had been wholly worn away by denudation, or +obliterated by metamorphic action, we ought to find only small remnants +of the formations next succeeding them in age, and these ought to be +very generally in +a metamorphosed condition. But the descriptions which we now possess of +the Silurian deposits over immense territories in Russia and in North +America, do not support the view, that the older a formation is, the +more it has suffered the extremity of denudation and metamorphism. + +The case at present must remain inexplicable; and may be truly urged as +a valid argument against the views here entertained. To show that it +may hereafter receive some explanation, I will give the following +hypothesis. From the nature of the organic remains, which do not appear +to have inhabited profound depths, in the several formations of Europe +and of the United States; and from the amount of sediment, miles in +thickness, of which the formations are composed, we may infer that from +first to last large islands or tracts of land, whence the sediment was +derived, occurred in the neighbourhood of the existing continents of +Europe and North America. But we do not know what was the state of +things in the intervals between the successive formations; whether +Europe and the United States during these intervals existed as dry +land, or as a submarine surface near land, on which sediment was not +deposited, or again as the bed of an open and unfathomable sea. + +Looking to the existing oceans, which are thrice as extensive as the +land, we see them studded with many islands; but not one oceanic island +is as yet known to afford even a remnant of any palæozoic or secondary +formation. Hence we may perhaps infer, that during the palæozoic and +secondary periods, neither continents nor continental islands existed +where our oceans now extend; for had they existed there, palæozoic and +secondary formations would in all probability have been accumulated +from sediment derived from their wear and +tear; and would have been at least partially upheaved by the +oscillations of level, which we may fairly conclude must have +intervened during these enormously long periods. If then we may infer +anything from these facts, we may infer that where our oceans now +extend, oceans have extended from the remotest period of which we have +any record; and on the other hand, that where continents now exist, +large tracts of land have existed, subjected no doubt to great +oscillations of level, since the earliest silurian period. The coloured +map appended to my volume on Coral Reefs, led me to conclude that the +great oceans are still mainly areas of subsidence, the great +archipelagoes still areas of oscillations of level, and the continents +areas of elevation. But have we any right to assume that things have +thus remained from eternity? Our continents seem to have been formed by +a preponderance, during many oscillations of level, of the force of +elevation; but may not the areas of preponderant movement have changed +in the lapse of ages? At a period immeasurably antecedent to the +silurian epoch, continents may have existed where oceans are now spread +out; and clear and open oceans may have existed where our continents +now stand. Nor should we be justified in assuming that if, for +instance, the bed of the Pacific Ocean were now converted into a +continent, we should there find formations older than the silurian +strata, supposing such to have been formerly deposited; for it might +well happen that strata which had subsided some miles nearer to the +centre of the earth, and which had been pressed on by an enormous +weight of superincumbent water, might have undergone far more +metamorphic action than strata which have always remained nearer to the +surface. The immense areas in some parts of the world, for instance in +South America, of bare metamorphic rocks, which +must have been heated under great pressure, have always seemed to me to +require some special explanation; and we may perhaps believe that we +see in these large areas, the many formations long anterior to the +silurian epoch in a completely metamorphosed condition. + +The several difficulties here discussed, namely our not finding in the +successive formations infinitely numerous transitional links between +the many species which now exist or have existed; the sudden manner in +which whole groups of species appear in our European formations; the +almost entire absence, as at present known, of fossiliferous formations +beneath the Silurian strata, are all undoubtedly of the gravest nature. +We see this in the plainest manner by the fact that all the most +eminent palæontologists, namely Cuvier, Owen, Agassiz, Barrande, +Falconer, E. Forbes, etc., and all our greatest geologists, as Lyell, +Murchison, Sedgwick, etc., have unanimously, often vehemently, +maintained the immutability of species. But I have reason to believe +that one great authority, Sir Charles Lyell, from further reflexion +entertains grave doubts on this subject. I feel how rash it is to +differ from these great authorities, to whom, with others, we owe all +our knowledge. Those who think the natural geological record in any +degree perfect, and who do not attach much weight to the facts and +arguments of other kinds given in this volume, will undoubtedly at once +reject my theory. For my part, following out Lyell’s metaphor, I look +at the natural geological record, as a history of the world imperfectly +kept, and written in a changing dialect; of this history we possess the +last volume alone, relating only to two or three countries. Of this +volume, only here and there a short chapter has +been preserved; and of each page, only here and there a few lines. Each +word of the slowly-changing language, in which the history is supposed +to be written, being more or less different in the interrupted +succession of chapters, may represent the apparently abruptly changed +forms of life, entombed in our consecutive, but widely separated +formations. On this view, the difficulties above discussed are greatly +diminished, or even disappear. + +On the slow and successive appearance of new species. On their +different rates of change. Species once lost do not reappear. Groups of +species follow the same general rules in their appearance and +disappearance as do single species. On Extinction. On simultaneous +changes in the forms of life throughout the world. On the affinities of +extinct species to each other and to living species. On the state of +development of ancient forms. On the succession of the same types +within the same areas. Summary of preceding and present chapters. + +Let us now see whether the several facts and rules relating to the +geological succession of organic beings, better accord with the common +view of the immutability of species, or with that of their slow and +gradual modification, through descent and natural selection. + +New species have appeared very slowly, one after another, both on the +land and in the waters. Lyell has shown that it is hardly possible to +resist the evidence on this head in the case of the several tertiary +stages; and every year tends to fill up the blanks between them, and to +make the percentage system of lost and new forms more gradual. In some +of the most recent beds, though undoubtedly of high antiquity if +measured by years, only one or two species are lost forms, and only one +or two are new forms, having here appeared for the first time, either +locally, or, as far as we know, on the face of the earth. If we may +trust the observations of Philippi in Sicily, the successive changes in +the marine inhabitants of that island have been many and most gradual. +The secondary formations are more broken; but, as Bronn has remarked, +neither the appearance +nor disappearance of their many now extinct species has been +simultaneous in each separate formation. + +Species of different genera and classes have not changed at the same +rate, or in the same degree. In the oldest tertiary beds a few living +shells may still be found in the midst of a multitude of extinct forms. +Falconer has given a striking instance of a similar fact, in an +existing crocodile associated with many strange and lost mammals and +reptiles in the sub-Himalayan deposits. The Silurian Lingula differs +but little from the living species of this genus; whereas most of the +other Silurian Molluscs and all the Crustaceans have changed greatly. +The productions of the land seem to change at a quicker rate than those +of the sea, of which a striking instance has lately been observed in +Switzerland. There is some reason to believe that organisms, considered +high in the scale of nature, change more quickly than those that are +low: though there are exceptions to this rule. The amount of organic +change, as Pictet has remarked, does not strictly correspond with the +succession of our geological formations; so that between each two +consecutive formations, the forms of life have seldom changed in +exactly the same degree. Yet if we compare any but the most closely +related formations, all the species will be found to have undergone +some change. When a species has once disappeared from the face of the +earth, we have reason to believe that the same identical form never +reappears. The strongest apparent exception to this latter rule, is +that of the so-called “colonies” of M. Barrande, which intrude for a +period in the midst of an older formation, and then allow the +pre-existing fauna to reappear; but Lyell’s explanation, namely, that +it is a case of temporary migration from a distinct geographical +province, seems to me satisfactory. + +These several facts accord well with my theory. I believe in no fixed +law of development, causing all the inhabitants of a country to change +abruptly, or simultaneously, or to an equal degree. The process of +modification must be extremely slow. The variability of each species is +quite independent of that of all others. Whether such variability be +taken advantage of by natural selection, and whether the variations be +accumulated to a greater or lesser amount, thus causing a greater or +lesser amount of modification in the varying species, depends on many +complex contingencies,—on the variability being of a beneficial nature, +on the power of intercrossing, on the rate of breeding, on the slowly +changing physical conditions of the country, and more especially on the +nature of the other inhabitants with which the varying species comes +into competition. Hence it is by no means surprising that one species +should retain the same identical form much longer than others; or, if +changing, that it should change less. We see the same fact in +geographical distribution; for instance, in the land-shells and +coleopterous insects of Madeira having come to differ considerably from +their nearest allies on the continent of Europe, whereas the marine +shells and birds have remained unaltered. We can perhaps understand the +apparently quicker rate of change in terrestrial and in more highly +organised productions compared with marine and lower productions, by +the more complex relations of the higher beings to their organic and +inorganic conditions of life, as explained in a former chapter. When +many of the inhabitants of a country have become modified and improved, +we can understand, on the principle of competition, and on that of the +many all-important relations of organism to organism, that any form +which does not become in some degree modified and improved, +will be liable to be exterminated. Hence we can see why all the species +in the same region do at last, if we look to wide enough intervals of +time, become modified; for those which do not change will become +extinct. + +In members of the same class the average amount of change, during long +and equal periods of time, may, perhaps, be nearly the same; but as the +accumulation of long-enduring fossiliferous formations depends on great +masses of sediment having been deposited on areas whilst subsiding, our +formations have been almost necessarily accumulated at wide and +irregularly intermittent intervals; consequently the amount of organic +change exhibited by the fossils embedded in consecutive formations is +not equal. Each formation, on this view, does not mark a new and +complete act of creation, but only an occasional scene, taken almost at +hazard, in a slowly changing drama. + +We can clearly understand why a species when once lost should never +reappear, even if the very same conditions of life, organic and +inorganic, should recur. For though the offspring of one species might +be adapted (and no doubt this has occurred in innumerable instances) to +fill the exact place of another species in the economy of nature, and +thus supplant it; yet the two forms—the old and the new—would not be +identically the same; for both would almost certainly inherit different +characters from their distinct progenitors. For instance, it is just +possible, if our fantail-pigeons were all destroyed, that fanciers, by +striving during long ages for the same object, might make a new breed +hardly distinguishable from our present fantail; but if the parent +rock-pigeon were also destroyed, and in nature we have every reason to +believe that the parent-form will generally be supplanted and +exterminated by its improved offspring, it is quite incredible that a +fantail, identical with the existing breed, could be raised from any +other species of pigeon, or even from the other well-established races +of the domestic pigeon, for the newly-formed fantail would be almost +sure to inherit from its new progenitor some slight characteristic +differences. + +Groups of species, that is, genera and families, follow the same +general rules in their appearance and disappearance as do single +species, changing more or less quickly, and in a greater or lesser +degree. A group does not reappear after it has once disappeared; or its +existence, as long as it lasts, is continuous. I am aware that there +are some apparent exceptions to this rule, but the exceptions are +surprisingly few, so few, that E. Forbes, Pictet, and Woodward (though +all strongly opposed to such views as I maintain) admit its truth; and +the rule strictly accords with my theory. For as all the species of the +same group have descended from some one species, it is clear that as +long as any species of the group have appeared in the long succession +of ages, so long must its members have continuously existed, in order +to have generated either new and modified or the same old and +unmodified forms. Species of the genus Lingula, for instance, must have +continuously existed by an unbroken succession of generations, from the +lowest Silurian stratum to the present day. + +We have seen in the last chapter that the species of a group sometimes +falsely appear to have come in abruptly; and I have attempted to give +an explanation of this fact, which if true would have been fatal to my +views. But such cases are certainly exceptional; the general rule being +a gradual increase in number, till the group reaches its maximum, and +then, sooner or later, it gradually decreases. If the +number of the species of a genus, or the number of the genera of a +family, be represented by a vertical line of varying thickness, +crossing the successive geological formations in which the species are +found, the line will sometimes falsely appear to begin at its lower +end, not in a sharp point, but abruptly; it then gradually thickens +upwards, sometimes keeping for a space of equal thickness, and +ultimately thins out in the upper beds, marking the decrease and final +extinction of the species. This gradual increase in number of the +species of a group is strictly conformable with my theory; as the +species of the same genus, and the genera of the same family, can +increase only slowly and progressively; for the process of modification +and the production of a number of allied forms must be slow and +gradual,—one species giving rise first to two or three varieties, these +being slowly converted into species, which in their turn produce by +equally slow steps other species, and so on, like the branching of a +great tree from a single stem, till the group becomes large. + +_On Extinction_.—We have as yet spoken only incidentally of the +disappearance of species and of groups of species. On the theory of +natural selection the extinction of old forms and the production of new +and improved forms are intimately connected together. The old notion of +all the inhabitants of the earth having been swept away at successive +periods by catastrophes, is very generally given up, even by those +geologists, as Elie de Beaumont, Murchison, Barrande, etc., whose +general views would naturally lead them to this conclusion. On the +contrary, we have every reason to believe, from the study of the +tertiary formations, that species and groups of species gradually +disappear, one after another, first from one spot, then from another, +and +finally from the world. Both single species and whole groups of species +last for very unequal periods; some groups, as we have seen, having +endured from the earliest known dawn of life to the present day; some +having disappeared before the close of the palæozoic period. No fixed +law seems to determine the length of time during which any single +species or any single genus endures. There is reason to believe that +the complete extinction of the species of a group is generally a slower +process than their production: if the appearance and disappearance of a +group of species be represented, as before, by a vertical line of +varying thickness, the line is found to taper more gradually at its +upper end, which marks the progress of extermination, than at its lower +end, which marks the first appearance and increase in numbers of the +species. In some cases, however, the extermination of whole groups of +beings, as of ammonites towards the close of the secondary period, has +been wonderfully sudden. + +The whole subject of the extinction of species has been involved in the +most gratuitous mystery. Some authors have even supposed that as the +individual has a definite length of life, so have species a definite +duration. No one I think can have marvelled more at the extinction of +species, than I have done. When I found in La Plata the tooth of a +horse embedded with the remains of Mastodon, Megatherium, Toxodon, and +other extinct monsters, which all co-existed with still living shells +at a very late geological period, I was filled with astonishment; for +seeing that the horse, since its introduction by the Spaniards into +South America, has run wild over the whole country and has increased in +numbers at an unparalleled rate, I asked myself what could so recently +have exterminated the former horse under conditions of life apparently +so favourable. But +how utterly groundless was my astonishment! Professor Owen soon +perceived that the tooth, though so like that of the existing horse, +belonged to an extinct species. Had this horse been still living, but +in some degree rare, no naturalist would have felt the least surprise +at its rarity; for rarity is the attribute of a vast number of species +of all classes, in all countries. If we ask ourselves why this or that +species is rare, we answer that something is unfavourable in its +conditions of life; but what that something is, we can hardly ever +tell. On the supposition of the fossil horse still existing as a rare +species, we might have felt certain from the analogy of all other +mammals, even of the slow-breeding elephant, and from the history of +the naturalisation of the domestic horse in South America, that under +more favourable conditions it would in a very few years have stocked +the whole continent. But we could not have told what the unfavourable +conditions were which checked its increase, whether some one or several +contingencies, and at what period of the horse’s life, and in what +degree, they severally acted. If the conditions had gone on, however +slowly, becoming less and less favourable, we assuredly should not have +perceived the fact, yet the fossil horse would certainly have become +rarer and rarer, and finally extinct;—its place being seized on by some +more successful competitor. + +It is most difficult always to remember that the increase of every +living being is constantly being checked by unperceived injurious +agencies; and that these same unperceived agencies are amply sufficient +to cause rarity, and finally extinction. We see in many cases in the +more recent tertiary formations, that rarity precedes extinction; and +we know that this has been the progress of events with those animals +which have +been exterminated, either locally or wholly, through man’s agency. I +may repeat what I published in 1845, namely, that to admit that species +generally become rare before they become extinct—to feel no surprise at +the rarity of a species, and yet to marvel greatly when it ceases to +exist, is much the same as to admit that sickness in the individual is +the forerunner of death—to feel no surprise at sickness, but when the +sick man dies, to wonder and to suspect that he died by some unknown +deed of violence. + +The theory of natural selection is grounded on the belief that each new +variety, and ultimately each new species, is produced and maintained by +having some advantage over those with which it comes into competition; +and the consequent extinction of less-favoured forms almost inevitably +follows. It is the same with our domestic productions: when a new and +slightly improved variety has been raised, it at first supplants the +less improved varieties in the same neighbourhood; when much improved +it is transported far and near, like our short-horn cattle, and takes +the place of other breeds in other countries. Thus the appearance of +new forms and the disappearance of old forms, both natural and +artificial, are bound together. In certain flourishing groups, the +number of new specific forms which have been produced within a given +time is probably greater than that of the old forms which have been +exterminated; but we know that the number of species has not gone on +indefinitely increasing, at least during the later geological periods, +so that looking to later times we may believe that the production of +new forms has caused the extinction of about the same number of old +forms. + +The competition will generally be most severe, as formerly explained +and illustrated by examples, between the forms which are most like each +other in all respects. +Hence the improved and modified descendants of a species will generally +cause the extermination of the parent-species; and if many new forms +have been developed from any one species, the nearest allies of that +species, _i.e._ the species of the same genus, will be the most liable +to extermination. Thus, as I believe, a number of new species descended +from one species, that is a new genus, comes to supplant an old genus, +belonging to the same family. But it must often have happened that a +new species belonging to some one group will have seized on the place +occupied by a species belonging to a distinct group, and thus caused +its extermination; and if many allied forms be developed from the +successful intruder, many will have to yield their places; and it will +generally be allied forms, which will suffer from some inherited +inferiority in common. But whether it be species belonging to the same +or to a distinct class, which yield their places to other species which +have been modified and improved, a few of the sufferers may often long +be preserved, from being fitted to some peculiar line of life, or from +inhabiting some distant and isolated station, where they have escaped +severe competition. For instance, a single species of Trigonia, a great +genus of shells in the secondary formations, survives in the Australian +seas; and a few members of the great and almost extinct group of Ganoid +fishes still inhabit our fresh waters. Therefore the utter extinction +of a group is generally, as we have seen, a slower process than its +production. + +With respect to the apparently sudden extermination of whole families +or orders, as of Trilobites at the close of the palæozoic period and of +Ammonites at the close of the secondary period, we must remember what +has been already said on the probable wide intervals of time +between our consecutive formations; and in these intervals there may +have been much slow extermination. Moreover, when by sudden immigration +or by unusually rapid development, many species of a new group have +taken possession of a new area, they will have exterminated in a +correspondingly rapid manner many of the old inhabitants; and the forms +which thus yield their places will commonly be allied, for they will +partake of some inferiority in common. + +Thus, as it seems to me, the manner in which single species and whole +groups of species become extinct, accords well with the theory of +natural selection. We need not marvel at extinction; if we must marvel, +let it be at our presumption in imagining for a moment that we +understand the many complex contingencies, on which the existence of +each species depends. If we forget for an instant, that each species +tends to increase inordinately, and that some check is always in +action, yet seldom perceived by us, the whole economy of nature will be +utterly obscured. Whenever we can precisely say why this species is +more abundant in individuals than that; why this species and not +another can be naturalised in a given country; then, and not till then, +we may justly feel surprise why we cannot account for the extinction of +this particular species or group of species. + +_On the Forms of Life changing almost simultaneously throughout the +World_.—Scarcely any palæontological discovery is more striking than +the fact, that the forms of life change almost simultaneously +throughout the world. Thus our European Chalk formation can be +recognised in many distant parts of the world, under the most different +climates, where not a fragment of the mineral chalk itself can be +found; namely, in North +America, in equatorial South America, in Tierra del Fuego, at the Cape +of Good Hope, and in the peninsula of India. For at these distant +points, the organic remains in certain beds present an unmistakeable +degree of resemblance to those of the Chalk. It is not that the same +species are met with; for in some cases not one species is identically +the same, but they belong to the same families, genera, and sections of +genera, and sometimes are similarly characterised in such trifling +points as mere superficial sculpture. Moreover other forms, which are +not found in the Chalk of Europe, but which occur in the formations +either above or below, are similarly absent at these distant points of +the world. In the several successive palæozoic formations of Russia, +Western Europe and North America, a similar parallelism in the forms of +life has been observed by several authors: so it is, according to +Lyell, with the several European and North American tertiary deposits. +Even if the few fossil species which are common to the Old and New +Worlds be kept wholly out of view, the general parallelism in the +successive forms of life, in the stages of the widely separated +palæozoic and tertiary periods, would still be manifest, and the +several formations could be easily correlated. + +These observations, however, relate to the marine inhabitants of +distant parts of the world: we have not sufficient data to judge +whether the productions of the land and of fresh water change at +distant points in the same parallel manner. We may doubt whether they +have thus changed: if the Megatherium, Mylodon, Macrauchenia, and +Toxodon had been brought to Europe from La Plata, without any +information in regard to their geological position, no one would have +suspected that they had coexisted with still living sea-shells; but as +these anomalous monsters coexisted with the Mastodon +and Horse, it might at least have been inferred that they had lived +during one of the latter tertiary stages. + +When the marine forms of life are spoken of as having changed +simultaneously throughout the world, it must not be supposed that this +expression relates to the same thousandth or hundred-thousandth year, +or even that it has a very strict geological sense; for if all the +marine animals which live at the present day in Europe, and all those +that lived in Europe during the pleistocene period (an enormously +remote period as measured by years, including the whole glacial epoch), +were to be compared with those now living in South America or in +Australia, the most skilful naturalist would hardly be able to say +whether the existing or the pleistocene inhabitants of Europe resembled +most closely those of the southern hemisphere. So, again, several +highly competent observers believe that the existing productions of the +United States are more closely related to those which lived in Europe +during certain later tertiary stages, than to those which now live +here; and if this be so, it is evident that fossiliferous beds +deposited at the present day on the shores of North America would +hereafter be liable to be classed with somewhat older European beds. +Nevertheless, looking to a remotely future epoch, there can, I think, +be little doubt that all the more modern _marine_ formations, namely, +the upper pliocene, the pleistocene and strictly modern beds, of +Europe, North and South America, and Australia, from containing fossil +remains in some degree allied, and from not including those forms which +are only found in the older underlying deposits, would be correctly +ranked as simultaneous in a geological sense. + +The fact of the forms of life changing simultaneously, in the above +large sense, at distant parts of the world, has greatly struck those +admirable observers, MM. +de Verneuil and d’Archiac. After referring to the parallelism of the +palæozoic forms of life in various parts of Europe, they add, “If +struck by this strange sequence, we turn our attention to North +America, and there discover a series of analogous phenomena, it will +appear certain that all these modifications of species, their +extinction, and the introduction of new ones, cannot be owing to mere +changes in marine currents or other causes more or less local and +temporary, but depend on general laws which govern the whole animal +kingdom.” M. Barrande has made forcible remarks to precisely the same +effect. It is, indeed, quite futile to look to changes of currents, +climate, or other physical conditions, as the cause of these great +mutations in the forms of life throughout the world, under the most +different climates. We must, as Barrande has remarked, look to some +special law. We shall see this more clearly when we treat of the +present distribution of organic beings, and find how slight is the +relation between the physical conditions of various countries, and the +nature of their inhabitants. + +This great fact of the parallel succession of the forms of life +throughout the world, is explicable on the theory of natural selection. +New species are formed by new varieties arising, which have some +advantage over older forms; and those forms, which are already +dominant, or have some advantage over the other forms in their own +country, would naturally oftenest give rise to new varieties or +incipient species; for these latter must be victorious in a still +higher degree in order to be preserved and to survive. We have distinct +evidence on this head, in the plants which are dominant, that is, which +are commonest in their own homes, and are most widely diffused, having +produced the greatest number of new varieties. It is also natural that +the dominant, +varying, and far-spreading species, which already have invaded to a +certain extent the territories of other species, should be those which +would have the best chance of spreading still further, and of giving +rise in new countries to new varieties and species. The process of +diffusion may often be very slow, being dependent on climatal and +geographical changes, or on strange accidents, but in the long run the +dominant forms will generally succeed in spreading. The diffusion +would, it is probable, be slower with the terrestrial inhabitants of +distinct continents than with the marine inhabitants of the continuous +sea. We might therefore expect to find, as we apparently do find, a +less strict degree of parallel succession in the productions of the +land than of the sea. + +Dominant species spreading from any region might encounter still more +dominant species, and then their triumphant course, or even their +existence, would cease. We know not at all precisely what are all the +conditions most favourable for the multiplication of new and dominant +species; but we can, I think, clearly see that a number of individuals, +from giving a better chance of the appearance of favourable variations, +and that severe competition with many already existing forms, would be +highly favourable, as would be the power of spreading into new +territories. A certain amount of isolation, recurring at long intervals +of time, would probably be also favourable, as before explained. One +quarter of the world may have been most favourable for the production +of new and dominant species on the land, and another for those in the +waters of the sea. If two great regions had been for a long period +favourably circumstanced in an equal degree, whenever their inhabitants +met, the battle would be prolonged and severe; and some from one +birthplace and some from the other might be victorious. But in the +course of time, the +forms dominant in the highest degree, wherever produced, would tend +everywhere to prevail. As they prevailed, they would cause the +extinction of other and inferior forms; and as these inferior forms +would be allied in groups by inheritance, whole groups would tend +slowly to disappear; though here and there a single member might long +be enabled to survive. + +Thus, as it seems to me, the parallel, and, taken in a large sense, +simultaneous, succession of the same forms of life throughout the +world, accords well with the principle of new species having been +formed by dominant species spreading widely and varying; the new +species thus produced being themselves dominant owing to inheritance, +and to having already had some advantage over their parents or over +other species; these again spreading, varying, and producing new +species. The forms which are beaten and which yield their places to the +new and victorious forms, will generally be allied in groups, from +inheriting some inferiority in common; and therefore as new and +improved groups spread throughout the world, old groups will disappear +from the world; and the succession of forms in both ways will +everywhere tend to correspond. + +There is one other remark connected with this subject worth making. I +have given my reasons for believing that all our greater fossiliferous +formations were deposited during periods of subsidence; and that blank +intervals of vast duration occurred during the periods when the bed of +the sea was either stationary or rising, and likewise when sediment was +not thrown down quickly enough to embed and preserve organic remains. +During these long and blank intervals I suppose that the inhabitants of +each region underwent a considerable amount of modification and +extinction, and that there was much migration from +other parts of the world. As we have reason to believe that large areas +are affected by the same movement, it is probable that strictly +contemporaneous formations have often been accumulated over very wide +spaces in the same quarter of the world; but we are far from having any +right to conclude that this has invariably been the case, and that +large areas have invariably been affected by the same movements. When +two formations have been deposited in two regions during nearly, but +not exactly the same period, we should find in both, from the causes +explained in the foregoing paragraphs, the same general succession in +the forms of life; but the species would not exactly correspond; for +there will have been a little more time in the one region than in the +other for modification, extinction, and immigration. + +I suspect that cases of this nature have occurred in Europe. Mr. +Prestwich, in his admirable Memoirs on the eocene deposits of England +and France, is able to draw a close general parallelism between the +successive stages in the two countries; but when he compares certain +stages in England with those in France, although he finds in both a +curious accordance in the numbers of the species belonging to the same +genera, yet the species themselves differ in a manner very difficult to +account for, considering the proximity of the two areas,—unless, +indeed, it be assumed that an isthmus separated two seas inhabited by +distinct, but contemporaneous, faunas. Lyell has made similar +observations on some of the later tertiary formations. Barrande, also, +shows that there is a striking general parallelism in the successive +Silurian deposits of Bohemia and Scandinavia; nevertheless he finds a +surprising amount of difference in the species. If the several +formations in these regions have not been deposited during the same +exact +periods,—a formation in one region often corresponding with a blank +interval in the other,—and if in both regions the species have gone on +slowly changing during the accumulation of the several formations and +during the long intervals of time between them; in this case, the +several formations in the two regions could be arranged in the same +order, in accordance with the general succession of the form of life, +and the order would falsely appear to be strictly parallel; +nevertheless the species would not all be the same in the apparently +corresponding stages in the two regions. + +_On the Affinities of extinct Species to each other, and to living +forms_.—Let us now look to the mutual affinities of extinct and living +species. They all fall into one grand natural system; and this fact is +at once explained on the principle of descent. The more ancient any +form is, the more, as a general rule, it differs from living forms. +But, as Buckland long ago remarked, all fossils can be classed either +in still existing groups, or between them. That the extinct forms of +life help to fill up the wide intervals between existing genera, +families, and orders, cannot be disputed. For if we confine our +attention either to the living or to the extinct alone, the series is +far less perfect than if we combine both into one general system. With +respect to the Vertebrata, whole pages could be filled with striking +illustrations from our great palæontologist, Owen, showing how extinct +animals fall in between existing groups. Cuvier ranked the Ruminants +and Pachyderms, as the two most distinct orders of mammals; but Owen +has discovered so many fossil links, that he has had to alter the whole +classification of these two orders; and has placed certain pachyderms +in the same sub-order with ruminants: for example, he dissolves by fine +gradations the apparently +wide difference between the pig and the camel. In regard to the +Invertebrata, Barrande, and a higher authority could not be named, +asserts that he is every day taught that palæozoic animals, though +belonging to the same orders, families, or genera with those living at +the present day, were not at this early epoch limited in such distinct +groups as they now are. + +Some writers have objected to any extinct species or group of species +being considered as intermediate between living species or groups. If +by this term it is meant that an extinct form is directly intermediate +in all its characters between two living forms, the objection is +probably valid. But I apprehend that in a perfectly natural +classification many fossil species would have to stand between living +species, and some extinct genera between living genera, even between +genera belonging to distinct families. The most common case, especially +with respect to very distinct groups, such as fish and reptiles, seems +to be, that supposing them to be distinguished at the present day from +each other by a dozen characters, the ancient members of the same two +groups would be distinguished by a somewhat lesser number of +characters, so that the two groups, though formerly quite distinct, at +that period made some small approach to each other. + +It is a common belief that the more ancient a form is, by so much the +more it tends to connect by some of its characters groups now widely +separated from each other. This remark no doubt must be restricted to +those groups which have undergone much change in the course of +geological ages; and it would be difficult to prove the truth of the +proposition, for every now and then even a living animal, as the +Lepidosiren, is discovered having affinities directed towards very +distinct groups. Yet if we compare the older Reptiles and +Batrachians, the older Fish, the older Cephalopods, and the eocene +Mammals, with the more recent members of the same classes, we must +admit that there is some truth in the remark. + +Let us see how far these several facts and inferences accord with the +theory of descent with modification. As the subject is somewhat +complex, I must request the reader to turn to the diagram in the fourth +chapter. We may suppose that the numbered letters represent genera, and +the dotted lines diverging from them the species in each genus. The +diagram is much too simple, too few genera and too few species being +given, but this is unimportant for us. The horizontal lines may +represent successive geological formations, and all the forms beneath +the uppermost line may be considered as extinct. The three existing +genera, _a_14, _q_14, _p_14, will form a small family; _b_14 and _f_14 +a closely allied family or sub-family; and _o_14, _e_14, _m_14, a third +family. These three families, together with the many extinct genera on +the several lines of descent diverging from the parent-form A, will +form an order; for all will have inherited something in common from +their ancient and common progenitor. On the principle of the continued +tendency to divergence of character, which was formerly illustrated by +this diagram, the more recent any form is, the more it will generally +differ from its ancient progenitor. Hence we can understand the rule +that the most ancient fossils differ most from existing forms. We must +not, however, assume that divergence of character is a necessary +contingency; it depends solely on the descendants from a species being +thus enabled to seize on many and different places in the economy of +nature. Therefore it is quite possible, as we have seen in the case of +some Silurian forms, that a species might go on being slightly +modified in relation to its slightly altered conditions of life, and +yet retain throughout a vast period the same general characteristics. +This is represented in the diagram by the letter F14. + +All the many forms, extinct and recent, descended from A, make, as +before remarked, one order; and this order, from the continued effects +of extinction and divergence of character, has become divided into +several sub-families and families, some of which are supposed to have +perished at different periods, and some to have endured to the present +day. + +By looking at the diagram we can see that if many of the extinct forms, +supposed to be embedded in the successive formations, were discovered +at several points low down in the series, the three existing families +on the uppermost line would be rendered less distinct from each other. +If, for instance, the genera _a_1, _a_5, _a_10, _f_8, _m_3, _m_6, _m_9 +were disinterred, these three families would be so closely linked +together that they probably would have to be united into one great +family, in nearly the same manner as has occurred with ruminants and +pachyderms. Yet he who objected to call the extinct genera, which thus +linked the living genera of three families together, intermediate in +character, would be justified, as they are intermediate, not directly, +but only by a long and circuitous course through many widely different +forms. If many extinct forms were to be discovered above one of the +middle horizontal lines or geological formations—for instance, above +Number VI.—but none from beneath this line, then only the two families +on the left hand (namely, _a_14, etc., and _b_14, etc.) would have to +be united into one family; and the two other families (namely, _a_14 to +_f_14 now including five genera, and _o_14 to _m_14) would yet remain +distinct. These two families, however, would be less distinct from each +other than they were before the +discovery of the fossils. If, for instance, we suppose the existing +genera of the two families to differ from each other by a dozen +characters, in this case the genera, at the early period marked VI., +would differ by a lesser number of characters; for at this early stage +of descent they have not diverged in character from the common +progenitor of the order, nearly so much as they subsequently diverged. +Thus it comes that ancient and extinct genera are often in some slight +degree intermediate in character between their modified descendants, or +between their collateral relations. + +In nature the case will be far more complicated than is represented in +the diagram; for the groups will have been more numerous, they will +have endured for extremely unequal lengths of time, and will have been +modified in various degrees. As we possess only the last volume of the +geological record, and that in a very broken condition, we have no +right to expect, except in very rare cases, to fill up wide intervals +in the natural system, and thus unite distinct families or orders. All +that we have a right to expect, is that those groups, which have within +known geological periods undergone much modification, should in the +older formations make some slight approach to each other; so that the +older members should differ less from each other in some of their +characters than do the existing members of the same groups; and this by +the concurrent evidence of our best palæontologists seems frequently to +be the case. + +Thus, on the theory of descent with modification, the main facts with +respect to the mutual affinities of the extinct forms of life to each +other and to living forms, seem to me explained in a satisfactory +manner. And they are wholly inexplicable on any other view. + +On this same theory, it is evident that the fauna of any great period +in the earth’s history will be intermediate +in general character between that which preceded and that which +succeeded it. Thus, the species which lived at the sixth great stage of +descent in the diagram are the modified offspring of those which lived +at the fifth stage, and are the parents of those which became still +more modified at the seventh stage; hence they could hardly fail to be +nearly intermediate in character between the forms of life above and +below. We must, however, allow for the entire extinction of some +preceding forms, and for the coming in of quite new forms by +immigration, and for a large amount of modification, during the long +and blank intervals between the successive formations. Subject to these +allowances, the fauna of each geological period undoubtedly is +intermediate in character, between the preceding and succeeding faunas. +I need give only one instance, namely, the manner in which the fossils +of the Devonian system, when this system was first discovered, were at +once recognised by palæontologists as intermediate in character between +those of the overlying carboniferous, and underlying Silurian system. +But each fauna is not necessarily exactly intermediate, as unequal +intervals of time have elapsed between consecutive formations. + +It is no real objection to the truth of the statement, that the fauna +of each period as a whole is nearly intermediate in character between +the preceding and succeeding faunas, that certain genera offer +exceptions to the rule. For instance, mastodons and elephants, when +arranged by Dr. Falconer in two series, first according to their mutual +affinities and then according to their periods of existence, do not +accord in arrangement. The species extreme in character are not the +oldest, or the most recent; nor are those which are intermediate in +character, intermediate in age. But +supposing for an instant, in this and other such cases, that the record +of the first appearance and disappearance of the species was perfect, +we have no reason to believe that forms successively produced +necessarily endure for corresponding lengths of time: a very ancient +form might occasionally last much longer than a form elsewhere +subsequently produced, especially in the case of terrestrial +productions inhabiting separated districts. To compare small things +with great: if the principal living and extinct races of the domestic +pigeon were arranged as well as they could be in serial affinity, this +arrangement would not closely accord with the order in time of their +production, and still less with the order of their disappearance; for +the parent rock-pigeon now lives; and many varieties between the +rock-pigeon and the carrier have become extinct; and carriers which are +extreme in the important character of length of beak originated earlier +than short-beaked tumblers, which are at the opposite end of the series +in this same respect. + +Closely connected with the statement, that the organic remains from an +intermediate formation are in some degree intermediate in character, is +the fact, insisted on by all palæontologists, that fossils from two +consecutive formations are far more closely related to each other, than +are the fossils from two remote formations. Pictet gives as a +well-known instance, the general resemblance of the organic remains +from the several stages of the chalk formation, though the species are +distinct in each stage. This fact alone, from its generality, seems to +have shaken Professor Pictet in his firm belief in the immutability of +species. He who is acquainted with the distribution of existing species +over the globe, will not attempt to account for the close resemblance +of the distinct species in closely consecutive +formations, by the physical conditions of the ancient areas having +remained nearly the same. Let it be remembered that the forms of life, +at least those inhabiting the sea, have changed almost simultaneously +throughout the world, and therefore under the most different climates +and conditions. Consider the prodigious vicissitudes of climate during +the pleistocene period, which includes the whole glacial period, and +note how little the specific forms of the inhabitants of the sea have +been affected. + +On the theory of descent, the full meaning of the fact of fossil +remains from closely consecutive formations, though ranked as distinct +species, being closely related, is obvious. As the accumulation of each +formation has often been interrupted, and as long blank intervals have +intervened between successive formations, we ought not to expect to +find, as I attempted to show in the last chapter, in any one or two +formations all the intermediate varieties between the species which +appeared at the commencement and close of these periods; but we ought +to find after intervals, very long as measured by years, but only +moderately long as measured geologically, closely allied forms, or, as +they have been called by some authors, representative species; and +these we assuredly do find. We find, in short, such evidence of the +slow and scarcely sensible mutation of specific forms, as we have a +just right to expect to find. + +_On the state of Development of Ancient Forms_.—There has been much +discussion whether recent forms are more highly developed than ancient. +I will not here enter on this subject, for naturalists have not as yet +defined to each other’s satisfaction what is meant by high and low +forms. But in one particular sense the +more recent forms must, on my theory, be higher than the more ancient; +for each new species is formed by having had some advantage in the +struggle for life over other and preceding forms. If under a nearly +similar climate, the eocene inhabitants of one quarter of the world +were put into competition with the existing inhabitants of the same or +some other quarter, the eocene fauna or flora would certainly be beaten +and exterminated; as would a secondary fauna by an eocene, and a +palæozoic fauna by a secondary fauna. I do not doubt that this process +of improvement has affected in a marked and sensible manner the +organisation of the more recent and victorious forms of life, in +comparison with the ancient and beaten forms; but I can see no way of +testing this sort of progress. Crustaceans, for instance, not the +highest in their own class, may have beaten the highest molluscs. From +the extraordinary manner in which European productions have recently +spread over New Zealand, and have seized on places which must have been +previously occupied, we may believe, if all the animals and plants of +Great Britain were set free in New Zealand, that in the course of time +a multitude of British forms would become thoroughly naturalized there, +and would exterminate many of the natives. On the other hand, from what +we see now occurring in New Zealand, and from hardly a single +inhabitant of the southern hemisphere having become wild in any part of +Europe, we may doubt, if all the productions of New Zealand were set +free in Great Britain, whether any considerable number would be enabled +to seize on places now occupied by our native plants and animals. Under +this point of view, the productions of Great Britain may be said to be +higher than those of New Zealand. Yet the most skilful naturalist from +an examination of the species +of the two countries could not have foreseen this result. + +Agassiz insists that ancient animals resemble to a certain extent the +embryos of recent animals of the same classes; or that the geological +succession of extinct forms is in some degree parallel to the +embryological development of recent forms. I must follow Pictet and +Huxley in thinking that the truth of this doctrine is very far from +proved. Yet I fully expect to see it hereafter confirmed, at least in +regard to subordinate groups, which have branched off from each other +within comparatively recent times. For this doctrine of Agassiz accords +well with the theory of natural selection. In a future chapter I shall +attempt to show that the adult differs from its embryo, owing to +variations supervening at a not early age, and being inherited at a +corresponding age. This process, whilst it leaves the embryo almost +unaltered, continually adds, in the course of successive generations, +more and more difference to the adult. + +Thus the embryo comes to be left as a sort of picture, preserved by +nature, of the ancient and less modified condition of each animal. This +view may be true, and yet it may never be capable of full proof. +Seeing, for instance, that the oldest known mammals, reptiles, and fish +strictly belong to their own proper classes, though some of these old +forms are in a slight degree less distinct from each other than are the +typical members of the same groups at the present day, it would be vain +to look for animals having the common embryological character of the +Vertebrata, until beds far beneath the lowest Silurian strata are +discovered—a discovery of which the chance is very small. + +_On the Succession of the same Types within the same +areas, during the later tertiary periods_.—Mr. Clift many years ago +showed that the fossil mammals from the Australian caves were closely +allied to the living marsupials of that continent. In South America, a +similar relationship is manifest, even to an uneducated eye, in the +gigantic pieces of armour like those of the armadillo, found in several +parts of La Plata; and Professor Owen has shown in the most striking +manner that most of the fossil mammals, buried there in such numbers, +are related to South American types. This relationship is even more +clearly seen in the wonderful collection of fossil bones made by MM. +Lund and Clausen in the caves of Brazil. I was so much impressed with +these facts that I strongly insisted, in 1839 and 1845, on this “law of +the succession of types,”—on “this wonderful relationship in the same +continent between the dead and the living.” Professor Owen has +subsequently extended the same generalisation to the mammals of the Old +World. We see the same law in this author’s restorations of the extinct +and gigantic birds of New Zealand. We see it also in the birds of the +caves of Brazil. Mr. Woodward has shown that the same law holds good +with sea-shells, but from the wide distribution of most genera of +molluscs, it is not well displayed by them. Other cases could be added, +as the relation between the extinct and living land-shells of Madeira; +and between the extinct and living brackish-water shells of the +Aralo-Caspian Sea. + +Now what does this remarkable law of the succession of the same types +within the same areas mean? He would be a bold man, who after comparing +the present climate of Australia and of parts of South America under +the same latitude, would attempt to account, on the one hand, by +dissimilar physical conditions for the dissimilarity of the inhabitants +of these two continents, +and, on the other hand, by similarity of conditions, for the uniformity +of the same types in each during the later tertiary periods. Nor can it +be pretended that it is an immutable law that marsupials should have +been chiefly or solely produced in Australia; or that Edentata and +other American types should have been solely produced in South America. +For we know that Europe in ancient times was peopled by numerous +marsupials; and I have shown in the publications above alluded to, that +in America the law of distribution of terrestrial mammals was formerly +different from what it now is. North America formerly partook strongly +of the present character of the southern half of the continent; and the +southern half was formerly more closely allied, than it is at present, +to the northern half. In a similar manner we know from Falconer and +Cautley’s discoveries, that northern India was formerly more closely +related in its mammals to Africa than it is at the present time. +Analogous facts could be given in relation to the distribution of +marine animals. + +On the theory of descent with modification, the great law of the long +enduring, but not immutable, succession of the same types within the +same areas, is at once explained; for the inhabitants of each quarter +of the world will obviously tend to leave in that quarter, during the +next succeeding period of time, closely allied though in some degree +modified descendants. If the inhabitants of one continent formerly +differed greatly from those of another continent, so will their +modified descendants still differ in nearly the same manner and degree. +But after very long intervals of time and after great geographical +changes, permitting much inter-migration, the feebler will yield to the +more dominant forms, and there will be nothing immutable in the laws of +past and present distribution. + +It may be asked in ridicule, whether I suppose that the megatherium and +other allied huge monsters have left behind them in South America the +sloth, armadillo, and anteater, as their degenerate descendants. This +cannot for an instant be admitted. These huge animals have become +wholly extinct, and have left no progeny. But in the caves of Brazil, +there are many extinct species which are closely allied in size and in +other characters to the species still living in South America; and some +of these fossils may be the actual progenitors of living species. It +must not be forgotten that, on my theory, all the species of the same +genus have descended from some one species; so that if six genera, each +having eight species, be found in one geological formation, and in the +next succeeding formation there be six other allied or representative +genera with the same number of species, then we may conclude that only +one species of each of the six older genera has left modified +descendants, constituting the six new genera. The other seven species +of the old genera have all died out and have left no progeny. Or, which +would probably be a far commoner case, two or three species of two or +three alone of the six older genera will have been the parents of the +six new genera; the other old species and the other whole genera having +become utterly extinct. In failing orders, with the genera and species +decreasing in numbers, as apparently is the case of the Edentata of +South America, still fewer genera and species will have left modified +blood-descendants. + +_Summary of the preceding and present Chapters_.—I have attempted to +show that the geological record is extremely imperfect; that only a +small portion of the globe has been geologically explored with care; +that only +certain classes of organic beings have been largely preserved in a +fossil state; that the number both of specimens and of species, +preserved in our museums, is absolutely as nothing compared with the +incalculable number of generations which must have passed away even +during a single formation; that, owing to subsidence being necessary +for the accumulation of fossiliferous deposits thick enough to resist +future degradation, enormous intervals of time have elapsed between the +successive formations; that there has probably been more extinction +during the periods of subsidence, and more variation during the periods +of elevation, and during the latter the record will have been least +perfectly kept; that each single formation has not been continuously +deposited; that the duration of each formation is, perhaps, short +compared with the average duration of specific forms; that migration +has played an important part in the first appearance of new forms in +any one area and formation; that widely ranging species are those which +have varied most, and have oftenest given rise to new species; and that +varieties have at first often been local. All these causes taken +conjointly, must have tended to make the geological record extremely +imperfect, and will to a large extent explain why we do not find +interminable varieties, connecting together all the extinct and +existing forms of life by the finest graduated steps. + +He who rejects these views on the nature of the geological record, will +rightly reject my whole theory. For he may ask in vain where are the +numberless transitional links which must formerly have connected the +closely allied or representative species, found in the several stages +of the same great formation. He may disbelieve in the enormous +intervals of time which have elapsed between our consecutive +formations; he +may overlook how important a part migration must have played, when the +formations of any one great region alone, as that of Europe, are +considered; he may urge the apparent, but often falsely apparent, +sudden coming in of whole groups of species. He may ask where are the +remains of those infinitely numerous organisms which must have existed +long before the first bed of the Silurian system was deposited: I can +answer this latter question only hypothetically, by saying that as far +as we can see, where our oceans now extend they have for an enormous +period extended, and where our oscillating continents now stand they +have stood ever since the Silurian epoch; but that long before that +period, the world may have presented a wholly different aspect; and +that the older continents, formed of formations older than any known to +us, may now all be in a metamorphosed condition, or may lie buried +under the ocean. + +Passing from these difficulties, all the other great leading facts in +palæontology seem to me simply to follow on the theory of descent with +modification through natural selection. We can thus understand how it +is that new species come in slowly and successively; how species of +different classes do not necessarily change together, or at the same +rate, or in the same degree; yet in the long run that all undergo +modification to some extent. The extinction of old forms is the almost +inevitable consequence of the production of new forms. We can +understand why when a species has once disappeared it never reappears. +Groups of species increase in numbers slowly, and endure for unequal +periods of time; for the process of modification is necessarily slow, +and depends on many complex contingencies. The dominant species of the +larger dominant groups tend to leave many modified +descendants, and thus new sub-groups and groups are formed. As these +are formed, the species of the less vigorous groups, from their +inferiority inherited from a common progenitor, tend to become extinct +together, and to leave no modified offspring on the face of the earth. +But the utter extinction of a whole group of species may often be a +very slow process, from the survival of a few descendants, lingering in +protected and isolated situations. When a group has once wholly +disappeared, it does not reappear; for the link of generation has been +broken. + +We can understand how the spreading of the dominant forms of life, +which are those that oftenest vary, will in the long run tend to people +the world with allied, but modified, descendants; and these will +generally succeed in taking the places of those groups of species which +are their inferiors in the struggle for existence. Hence, after long +intervals of time, the productions of the world will appear to have +changed simultaneously. + +We can understand how it is that all the forms of life, ancient and +recent, make together one grand system; for all are connected by +generation. We can understand, from the continued tendency to +divergence of character, why the more ancient a form is, the more it +generally differs from those now living. Why ancient and extinct forms +often tend to fill up gaps between existing forms, sometimes blending +two groups previously classed as distinct into one; but more commonly +only bringing them a little closer together. The more ancient a form +is, the more often, apparently, it displays characters in some degree +intermediate between groups now distinct; for the more ancient a form +is, the more nearly it will be related to, and consequently resemble, +the common progenitor of groups, since become +widely divergent. Extinct forms are seldom directly intermediate +between existing forms; but are intermediate only by a long and +circuitous course through many extinct and very different forms. We can +clearly see why the organic remains of closely consecutive formations +are more closely allied to each other, than are those of remote +formations; for the forms are more closely linked together by +generation: we can clearly see why the remains of an intermediate +formation are intermediate in character. + +The inhabitants of each successive period in the world’s history have +beaten their predecessors in the race for life, and are, in so far, +higher in the scale of nature; and this may account for that vague yet +ill-defined sentiment, felt by many palæontologists, that organisation +on the whole has progressed. If it should hereafter be proved that +ancient animals resemble to a certain extent the embryos of more recent +animals of the same class, the fact will be intelligible. The +succession of the same types of structure within the same areas during +the later geological periods ceases to be mysterious, and is simply +explained by inheritance. + +If then the geological record be as imperfect as I believe it to be, +and it may at least be asserted that the record cannot be proved to be +much more perfect, the main objections to the theory of natural +selection are greatly diminished or disappear. On the other hand, all +the chief laws of palæontology plainly proclaim, as it seems to me, +that species have been produced by ordinary generation: old forms +having been supplanted by new and improved forms of life, produced by +the laws of variation still acting round us, and preserved by Natural +Selection. + +Present distribution cannot be accounted for by differences in physical +conditions. Importance of barriers. Affinity of the productions of the +same continent. Centres of creation. Means of dispersal, by changes of +climate and of the level of the land, and by occasional means. +Dispersal during the Glacial period co-extensive with the world. + +In considering the distribution of organic beings over the face of the +globe, the first great fact which strikes us is, that neither the +similarity nor the dissimilarity of the inhabitants of various regions +can be accounted for by their climatal and other physical conditions. +Of late, almost every author who has studied the subject has come to +this conclusion. The case of America alone would almost suffice to +prove its truth: for if we exclude the northern parts where the +circumpolar land is almost continuous, all authors agree that one of +the most fundamental divisions in geographical distribution is that +between the New and Old Worlds; yet if we travel over the vast American +continent, from the central parts of the United States to its extreme +southern point, we meet with the most diversified conditions; the most +humid districts, arid deserts, lofty mountains, grassy plains, forests, +marshes, lakes, and great rivers, under almost every temperature. There +is hardly a climate or condition in the Old World which cannot be +paralleled in the New—at least as closely as the same species generally +require; for it is a most rare case to find a group of organisms +confined to any small spot, having conditions peculiar in only a slight +degree; for instance, small areas in the Old World could be pointed out +hotter than any in the New World, yet these are not inhabited by a +peculiar fauna or flora. Notwithstanding this parallelism in the +conditions of the Old and New Worlds, how widely different are their +living productions! + +In the southern hemisphere, if we compare large tracts of land in +Australia, South Africa, and western South America, between latitudes +25° and 35°, we shall find parts extremely similar in all their +conditions, yet it would not be possible to point out three faunas and +floras more utterly dissimilar. Or again we may compare the productions +of South America south of lat. 35° with those north of 25°, which +consequently inhabit a considerably different climate, and they will be +found incomparably more closely related to each other, than they are to +the productions of Australia or Africa under nearly the same climate. +Analogous facts could be given with respect to the inhabitants of the +sea. + +A second great fact which strikes us in our general review is, that +barriers of any kind, or obstacles to free migration, are related in a +close and important manner to the differences between the productions +of various regions. We see this in the great difference of nearly all +the terrestrial productions of the New and Old Worlds, excepting in the +northern parts, where the land almost joins, and where, under a +slightly different climate, there might have been free migration for +the northern temperate forms, as there now is for the strictly arctic +productions. We see the same fact in the great difference between the +inhabitants of Australia, Africa, and South America under the same +latitude: for these countries are almost as much isolated from each +other as is possible. On each continent, also, we see the same fact; +for on the opposite sides of +lofty and continuous mountain-ranges, and of great deserts, and +sometimes even of large rivers, we find different productions; though +as mountain chains, deserts, etc., are not as impassable, or likely to +have endured so long as the oceans separating continents, the +differences are very inferior in degree to those characteristic of +distinct continents. + +Turning to the sea, we find the same law. No two marine faunas are more +distinct, with hardly a fish, shell, or crab in common, than those of +the eastern and western shores of South and Central America; yet these +great faunas are separated only by the narrow, but impassable, isthmus +of Panama. Westward of the shores of America, a wide space of open +ocean extends, with not an island as a halting-place for emigrants; +here we have a barrier of another kind, and as soon as this is passed +we meet in the eastern islands of the Pacific, with another and totally +distinct fauna. So that here three marine faunas range far northward +and southward, in parallel lines not far from each other, under +corresponding climates; but from being separated from each other by +impassable barriers, either of land or open sea, they are wholly +distinct. On the other hand, proceeding still further westward from the +eastern islands of the tropical parts of the Pacific, we encounter no +impassable barriers, and we have innumerable islands as halting-places, +until after travelling over a hemisphere we come to the shores of +Africa; and over this vast space we meet with no well-defined and +distinct marine faunas. Although hardly one shell, crab or fish is +common to the above-named three approximate faunas of Eastern and +Western America and the eastern Pacific islands, yet many fish range +from the Pacific into the Indian Ocean, and many shells are common to +the eastern islands of the Pacific +and the eastern shores of Africa, on almost exactly opposite meridians +of longitude. + +A third great fact, partly included in the foregoing statements, is the +affinity of the productions of the same continent or sea, though the +species themselves are distinct at different points and stations. It is +a law of the widest generality, and every continent offers innumerable +instances. Nevertheless the naturalist in travelling, for instance, +from north to south never fails to be struck by the manner in which +successive groups of beings, specifically distinct, yet clearly +related, replace each other. He hears from closely allied, yet distinct +kinds of birds, notes nearly similar, and sees their nests similarly +constructed, but not quite alike, with eggs coloured in nearly the same +manner. The plains near the Straits of Magellan are inhabited by one +species of Rhea (American ostrich), and northward the plains of La +Plata by another species of the same genus; and not by a true ostrich +or emeu, like those found in Africa and Australia under the same +latitude. On these same plains of La Plata, we see the agouti and +bizcacha, animals having nearly the same habits as our hares and +rabbits and belonging to the same order of Rodents, but they plainly +display an American type of structure. We ascend the lofty peaks of the +Cordillera and we find an alpine species of bizcacha; we look to the +waters, and we do not find the beaver or musk-rat, but the coypu and +capybara, rodents of the American type. Innumerable other instances +could be given. If we look to the islands off the American shore, +however much they may differ in geological structure, the inhabitants, +though they may be all peculiar species, are essentially American. We +may look back to past ages, as shown in the last chapter, and we find +American types then prevalent on +the American continent and in the American seas. We see in these facts +some deep organic bond, prevailing throughout space and time, over the +same areas of land and water, and independent of their physical +conditions. The naturalist must feel little curiosity, who is not led +to inquire what this bond is. + +This bond, on my theory, is simply inheritance, that cause which alone, +as far as we positively know, produces organisms quite like, or, as we +see in the case of varieties nearly like each other. The dissimilarity +of the inhabitants of different regions may be attributed to +modification through natural selection, and in a quite subordinate +degree to the direct influence of different physical conditions. The +degree of dissimilarity will depend on the migration of the more +dominant forms of life from one region into another having been +effected with more or less ease, at periods more or less remote;—on the +nature and number of the former immigrants;—and on their action and +reaction, in their mutual struggles for life;—the relation of organism +to organism being, as I have already often remarked, the most important +of all relations. Thus the high importance of barriers comes into play +by checking migration; as does time for the slow process of +modification through natural selection. Widely-ranging species, +abounding in individuals, which have already triumphed over many +competitors in their own widely-extended homes will have the best +chance of seizing on new places, when they spread into new countries. +In their new homes they will be exposed to new conditions, and will +frequently undergo further modification and improvement; and thus they +will become still further victorious, and will produce groups of +modified descendants. On this principle of inheritance with +modification, we can understand how it is that sections of genera, +whole genera, +and even families are confined to the same areas, as is so commonly and +notoriously the case. + +I believe, as was remarked in the last chapter, in no law of necessary +development. As the variability of each species is an independent +property, and will be taken advantage of by natural selection, only so +far as it profits the individual in its complex struggle for life, so +the degree of modification in different species will be no uniform +quantity. If, for instance, a number of species, which stand in direct +competition with each other, migrate in a body into a new and +afterwards isolated country, they will be little liable to +modification; for neither migration nor isolation in themselves can do +anything. These principles come into play only by bringing organisms +into new relations with each other, and in a lesser degree with the +surrounding physical conditions. As we have seen in the last chapter +that some forms have retained nearly the same character from an +enormously remote geological period, so certain species have migrated +over vast spaces, and have not become greatly modified. + +On these views, it is obvious, that the several species of the same +genus, though inhabiting the most distant quarters of the world, must +originally have proceeded from the same source, as they have descended +from the same progenitor. In the case of those species, which have +undergone during whole geological periods but little modification, +there is not much difficulty in believing that they may have migrated +from the same region; for during the vast geographical and climatal +changes which will have supervened since ancient times, almost any +amount of migration is possible. But in many other cases, in which we +have reason to believe that the species of a genus have been produced +within comparatively recent times, there is great difficulty on this +head. It +is also obvious that the individuals of the same species, though now +inhabiting distant and isolated regions, must have proceeded from one +spot, where their parents were first produced: for, as explained in the +last chapter, it is incredible that individuals identically the same +should ever have been produced through natural selection from parents +specifically distinct. + +We are thus brought to the question which has been largely discussed by +naturalists, namely, whether species have been created at one or more +points of the earth’s surface. Undoubtedly there are very many cases of +extreme difficulty, in understanding how the same species could +possibly have migrated from some one point to the several distant and +isolated points, where now found. Nevertheless the simplicity of the +view that each species was first produced within a single region +captivates the mind. He who rejects it, rejects the _vera causa_ of +ordinary generation with subsequent migration, and calls in the agency +of a miracle. It is universally admitted, that in most cases the area +inhabited by a species is continuous; and when a plant or animal +inhabits two points so distant from each other, or with an interval of +such a nature, that the space could not be easily passed over by +migration, the fact is given as something remarkable and exceptional. +The capacity of migrating across the sea is more distinctly limited in +terrestrial mammals, than perhaps in any other organic beings; and, +accordingly, we find no inexplicable cases of the same mammal +inhabiting distant points of the world. No geologist will feel any +difficulty in such cases as Great Britain having been formerly united +to Europe, and consequently possessing the same quadrupeds. But if the +same species can be produced at two separate points, why do we not find +a single mammal common to Europe and Australia or South America? The +conditions of life are +nearly the same, so that a multitude of European animals and plants +have become naturalised in America and Australia; and some of the +aboriginal plants are identically the same at these distant points of +the northern and southern hemispheres? The answer, as I believe, is, +that mammals have not been able to migrate, whereas some plants, from +their varied means of dispersal, have migrated across the vast and +broken interspace. The great and striking influence which barriers of +every kind have had on distribution, is intelligible only on the view +that the great majority of species have been produced on one side +alone, and have not been able to migrate to the other side. Some few +families, many sub-families, very many genera, and a still greater +number of sections of genera are confined to a single region; and it +has been observed by several naturalists, that the most natural genera, +or those genera in which the species are most closely related to each +other, are generally local, or confined to one area. What a strange +anomaly it would be, if, when coming one step lower in the series, to +the individuals of the same species, a directly opposite rule +prevailed; and species were not local, but had been produced in two or +more distinct areas! + +Hence it seems to me, as it has to many other naturalists, that the +view of each species having been produced in one area alone, and having +subsequently migrated from that area as far as its powers of migration +and subsistence under past and present conditions permitted, is the +most probable. Undoubtedly many cases occur, in which we cannot explain +how the same species could have passed from one point to the other. But +the geographical and climatal changes, which have certainly occurred +within recent geological times, must have interrupted or rendered +discontinuous the formerly continuous range of many species. So that we +are reduced to consider whether the exceptions to +continuity of range are so numerous and of so grave a nature, that we +ought to give up the belief, rendered probable by general +considerations, that each species has been produced within one area, +and has migrated thence as far as it could. It would be hopelessly +tedious to discuss all the exceptional cases of the same species, now +living at distant and separated points; nor do I for a moment pretend +that any explanation could be offered of many such cases. But after +some preliminary remarks, I will discuss a few of the most striking +classes of facts; namely, the existence of the same species on the +summits of distant mountain-ranges, and at distant points in the arctic +and antarctic regions; and secondly (in the following chapter), the +wide distribution of freshwater productions; and thirdly, the +occurrence of the same terrestrial species on islands and on the +mainland, though separated by hundreds of miles of open sea. If the +existence of the same species at distant and isolated points of the +earth’s surface, can in many instances be explained on the view of each +species having migrated from a single birthplace; then, considering our +ignorance with respect to former climatal and geographical changes and +various occasional means of transport, the belief that this has been +the universal law, seems to me incomparably the safest. + +In discussing this subject, we shall be enabled at the same time to +consider a point equally important for us, namely, whether the several +distinct species of a genus, which on my theory have all descended from +a common progenitor, can have migrated (undergoing modification during +some part of their migration) from the area inhabited by their +progenitor. If it can be shown to be almost invariably the case, that a +region, of which most of its inhabitants are closely related to, or +belong to the same genera with the species of a second region, +has probably received at some former period immigrants from this other +region, my theory will be strengthened; for we can clearly understand, +on the principle of modification, why the inhabitants of a region +should be related to those of another region, whence it has been +stocked. A volcanic island, for instance, upheaved and formed at the +distance of a few hundreds of miles from a continent, would probably +receive from it in the course of time a few colonists, and their +descendants, though modified, would still be plainly related by +inheritance to the inhabitants of the continent. Cases of this nature +are common, and are, as we shall hereafter more fully see, inexplicable +on the theory of independent creation. This view of the relation of +species in one region to those in another, does not differ much (by +substituting the word variety for species) from that lately advanced in +an ingenious paper by Mr. Wallace, in which he concludes, that “every +species has come into existence coincident both in space and time with +a pre-existing closely allied species.” And I now know from +correspondence, that this coincidence he attributes to generation with +modification. + +The previous remarks on “single and multiple centres of creation” do +not directly bear on another allied question,—namely whether all the +individuals of the same species have descended from +a single pair, or single hermaphrodite, or whether, as some authors +suppose, from many individuals simultaneously created. With those +organic beings which never intercross (if such exist), the species, on +my theory, must have descended from a succession of improved varieties, +which will never have blended with other individuals or varieties, but +will have supplanted each other; so that, at each successive stage of +modification and improvement, all the individuals of each variety will +have descended from a single parent. But in the majority of cases, +namely, with all organisms which habitually unite for each birth, or +which often intercross, I believe that during the slow process of +modification the individuals of the species will have been kept nearly +uniform by intercrossing; so that many individuals will have gone on +simultaneously changing, and the whole amount of modification will not +have been due, at each stage, to descent from a single parent. To +illustrate what I mean: our English racehorses differ slightly from the +horses of every other breed; but they do not owe their difference and +superiority to descent from any single pair, but to continued care in +selecting and training many individuals during many generations. + +Before discussing the three classes of facts, which I have selected as +presenting the greatest amount of difficulty on the theory of “single +centres of creation,” I must say a few words on the means of dispersal. + +_Means of Dispersal_.—Sir C. Lyell and other authors have ably treated +this subject. I can give here only the briefest abstract of the more +important facts. Change of climate must have had a powerful influence +on migration: a region when its climate was different may have been a +high road for migration, but now be impassable; I shall, however, +presently have to discuss this branch of the subject in some detail. +Changes of level in the land must also have been highly influential: a +narrow isthmus now separates two marine faunas; submerge it, or let it +formerly have been submerged, and the two faunas will now blend or may +formerly have blended: where the sea now extends, land may at a former +period have connected islands or possibly even continents together, and +thus have allowed terrestrial productions to pass from one to the +other. +No geologist will dispute that great mutations of level have occurred +within the period of existing organisms. Edward Forbes insisted that +all the islands in the Atlantic must recently have been connected with +Europe or Africa, and Europe likewise with America. Other authors have +thus hypothetically bridged over every ocean, and have united almost +every island to some mainland. If indeed the arguments used by Forbes +are to be trusted, it must be admitted that scarcely a single island +exists which has not recently been united to some continent. This view +cuts the Gordian knot of the dispersal of the same species to the most +distant points, and removes many a difficulty: but to the best of my +judgment we are not authorized in admitting such enormous geographical +changes within the period of existing species. It seems to me that we +have abundant evidence of great oscillations of level in our +continents; but not of such vast changes in their position and +extension, as to have united them within the recent period to each +other and to the several intervening oceanic islands. I freely admit +the former existence of many islands, now buried beneath the sea, which +may have served as halting places for plants and for many animals +during their migration. In the coral-producing oceans such sunken +islands are now marked, as I believe, by rings of coral or atolls +standing over them. Whenever it is fully admitted, as I believe it will +some day be, that each species has proceeded from a single birthplace, +and when in the course of time we know something definite about the +means of distribution, we shall be enabled to speculate with security +on the former extension of the land. But I do not believe that it will +ever be proved that within the recent period continents which are now +quite separate, have been continuously, or almost continuously, united +with each other, and with the many existing oceanic islands. Several +facts in distribution,—such as the great difference in the marine +faunas on the opposite sides of almost every continent,—the close +relation of the tertiary inhabitants of several lands and even seas to +their present inhabitants,—a certain degree of relation (as we shall +hereafter see) between the distribution of mammals and the depth of the +sea,—these and other such facts seem to me opposed to the admission of +such prodigious geographical revolutions within the recent period, as +are necessitated on the view advanced by Forbes and admitted by his +many followers. The nature and relative proportions of the inhabitants +of oceanic islands likewise seem to me opposed to the belief of their +former continuity with continents. Nor does their almost universally +volcanic composition favour the admission that they are the wrecks of +sunken continents;—if they had originally existed as mountain-ranges on +the land, some at least of the islands would have been formed, like +other mountain-summits, of granite, metamorphic schists, old +fossiliferous or other such rocks, instead of consisting of mere piles +of volcanic matter. + +I must now say a few words on what are called accidental means, but +which more properly might be called occasional means of distribution. I +shall here confine myself to plants. In botanical works, this or that +plant is stated to be ill adapted for wide dissemination; but for +transport across the sea, the greater or less facilities may be said to +be almost wholly unknown. Until I tried, with Mr. Berkeley’s aid, a few +experiments, it was not even known how far seeds could resist the +injurious action of sea-water. To my surprise I found that out of 87 +kinds, 64 germinated after an immersion of 28 days, and a few survived +an immersion of 137 days. +For convenience sake I chiefly tried small seeds, without the capsule +or fruit; and as all of these sank in a few days, they could not be +floated across wide spaces of the sea, whether or not they were injured +by the salt-water. Afterwards I tried some larger fruits, capsules, +etc., and some of these floated for a long time. It is well known what +a difference there is in the buoyancy of green and seasoned timber; and +it occurred to me that floods might wash down plants or branches, and +that these might be dried on the banks, and then by a fresh rise in the +stream be washed into the sea. Hence I was led to dry stems and +branches of 94 plants with ripe fruit, and to place them on sea water. +The majority sank quickly, but some which whilst green floated for a +very short time, when dried floated much longer; for instance, ripe +hazel-nuts sank immediately, but when dried, they floated for 90 days +and afterwards when planted they germinated; an asparagus plant with +ripe berries floated for 23 days, when dried it floated for 85 days, +and the seeds afterwards germinated: the ripe seeds of Helosciadium +sank in two days, when dried they floated for above 90 days, and +afterwards germinated. Altogether out of the 94 dried plants, 18 +floated for above 28 days, and some of the 18 floated for a very much +longer period. So that as 64/87 seeds germinated after an immersion of +28 days; and as 18/94 plants with ripe fruit (but not all the same +species as in the foregoing experiment) floated, after being dried, for +above 28 days, as far as we may infer anything from these scanty facts, +we may conclude that the seeds of 14/100 plants of any country might be +floated by sea-currents during 28 days, and would retain their power of +germination. In Johnston’s Physical Atlas, the average rate of the +several Atlantic currents is 33 miles per diem (some currents running +at the rate of 60 miles +per diem); on this average, the seeds of 14/100 plants belonging to one +country might be floated across 924 miles of sea to another country; +and when stranded, if blown to a favourable spot by an inland gale, +they would germinate. + +Subsequently to my experiments, M. Martens tried similar ones, but in a +much better manner, for he placed the seeds in a box in the actual sea, +so that they were alternately wet and exposed to the air like really +floating plants. He tried 98 seeds, mostly different from mine; but he +chose many large fruits and likewise seeds from plants which live near +the sea; and this would have favoured the average length of their +flotation and of their resistance to the injurious action of the +salt-water. On the other hand he did not previously dry the plants or +branches with the fruit; and this, as we have seen, would have caused +some of them to have floated much longer. The result was that 18/98 of +his seeds floated for 42 days, and were then capable of germination. +But I do not doubt that plants exposed to the waves would float for a +less time than those protected from violent movement as in our +experiments. Therefore it would perhaps be safer to assume that the +seeds of about 10/100 plants of a flora, after having been dried, could +be floated across a space of sea 900 miles in width, and would then +germinate. The fact of the larger fruits often floating longer than the +small, is interesting; as plants with large seeds or fruit could hardly +be transported by any other means; and Alph. de Candolle has shown that +such plants generally have restricted ranges. + +But seeds may be occasionally transported in another manner. Drift +timber is thrown up on most islands, even on those in the midst of the +widest oceans; and the natives of the coral-islands in the Pacific, +procure +stones for their tools, solely from the roots of drifted trees, these +stones being a valuable royal tax. I find on examination, that when +irregularly shaped stones are embedded in the roots of trees, small +parcels of earth are very frequently enclosed in their interstices and +behind them,—so perfectly that not a particle could be washed away in +the longest transport: out of one small portion of earth thus +_completely_ enclosed by wood in an oak about 50 years old, three +dicotyledonous plants germinated: I am certain of the accuracy of this +observation. Again, I can show that the carcasses of birds, when +floating on the sea, sometimes escape being immediately devoured; and +seeds of many kinds in the crops of floating birds long retain their +vitality: peas and vetches, for instance, are killed by even a few +days’ immersion in sea-water; but some taken out of the crop of a +pigeon, which had floated on artificial salt-water for 30 days, to my +surprise nearly all germinated. + +Living birds can hardly fail to be highly effective agents in the +transportation of seeds. I could give many facts showing how frequently +birds of many kinds are blown by gales to vast distances across the +ocean. We may I think safely assume that under such circumstances their +rate of flight would often be 35 miles an hour; and some authors have +given a far higher estimate. I have never seen an instance of +nutritious seeds passing through the intestines of a bird; but hard +seeds of fruit will pass uninjured through even the digestive organs of +a turkey. In the course of two months, I picked up in my garden 12 +kinds of seeds, out of the excrement of small birds, and these seemed +perfect, and some of them, which I tried, germinated. But the following +fact is more important: the crops of birds do not secrete gastric +juice, and do not in the +least injure, as I know by trial, the germination of seeds; now after a +bird has found and devoured a large supply of food, it is positively +asserted that all the grains do not pass into the gizzard for 12 or +even 18 hours. A bird in this interval might easily be blown to the +distance of 500 miles, and hawks are known to look out for tired birds, +and the contents of their torn crops might thus readily get scattered. +Mr. Brent informs me that a friend of his had to give up flying +carrier-pigeons from France to England, as the hawks on the English +coast destroyed so many on their arrival. Some hawks and owls bolt +their prey whole, and after an interval of from twelve to twenty hours, +disgorge pellets, which, as I know from experiments made in the +Zoological Gardens, include seeds capable of germination. Some seeds of +the oat, wheat, millet, canary, hemp, clover, and beet germinated after +having been from twelve to twenty-one hours in the stomachs of +different birds of prey; and two seeds of beet grew after having been +thus retained for two days and fourteen hours. Freshwater fish, I find, +eat seeds of many land and water plants: fish are frequently devoured +by birds, and thus the seeds might be transported from place to place. +I forced many kinds of seeds into the stomachs of dead fish, and then +gave their bodies to fishing-eagles, storks, and pelicans; these birds +after an interval of many hours, either rejected the seeds in pellets +or passed them in their excrement; and several of these seeds retained +their power of germination. Certain seeds, however, were always killed +by this process. + +Although the beaks and feet of birds are generally quite clean, I can +show that earth sometimes adheres to them: in one instance I removed +twenty-two grains of dry argillaceous earth from one foot of a +partridge, and in this earth there was a pebble quite as large as +the seed of a vetch. Thus seeds might occasionally be transported to +great distances; for many facts could be given showing that soil almost +everywhere is charged with seeds. Reflect for a moment on the millions +of quails which annually cross the Mediterranean; and can we doubt that +the earth adhering to their feet would sometimes include a few minute +seeds? But I shall presently have to recur to this subject. + +As icebergs are known to be sometimes loaded with earth and stones, and +have even carried brushwood, bones, and the nest of a land-bird, I can +hardly doubt that they must occasionally have transported seeds from +one part to another of the arctic and antarctic regions, as suggested +by Lyell; and during the Glacial period from one part of the now +temperate regions to another. In the Azores, from the large number of +the species of plants common to Europe, in comparison with the plants +of other oceanic islands nearer to the mainland, and (as remarked by +Mr. H. C. Watson) from the somewhat northern character of the flora in +comparison with the latitude, I suspected that these islands had been +partly stocked by ice-borne seeds, during the Glacial epoch. At my +request Sir C. Lyell wrote to M. Hartung to inquire whether he had +observed erratic boulders on these islands, and he answered that he had +found large fragments of granite and other rocks, which do not occur in +the archipelago. Hence we may safely infer that icebergs formerly +landed their rocky burthens on the shores of these mid-ocean islands, +and it is at least possible that they may have brought thither the +seeds of northern plants. + +Considering that the several above means of transport, and that several +other means, which without doubt remain to be discovered, have been in +action year after year, for centuries and tens of thousands of +years, it would I think be a marvellous fact if many plants had not +thus become widely transported. These means of transport are sometimes +called accidental, but this is not strictly correct: the currents of +the sea are not accidental, nor is the direction of prevalent gales of +wind. It should be observed that scarcely any means of transport would +carry seeds for very great distances; for seeds do not retain their +vitality when exposed for a great length of time to the action of +seawater; nor could they be long carried in the crops or intestines of +birds. These means, however, would suffice for occasional transport +across tracts of sea some hundred miles in breadth, or from island to +island, or from a continent to a neighbouring island, but not from one +distant continent to another. The floras of distant continents would +not by such means become mingled in any great degree; but would remain +as distinct as we now see them to be. The currents, from their course, +would never bring seeds from North America to Britain, though they +might and do bring seeds from the West Indies to our western shores, +where, if not killed by so long an immersion in salt-water, they could +not endure our climate. Almost every year, one or two land-birds are +blown across the whole Atlantic Ocean, from North America to the +western shores of Ireland and England; but seeds could be transported +by these wanderers only by one means, namely, in dirt sticking to their +feet, which is in itself a rare accident. Even in this case, how small +would the chance be of a seed falling on favourable soil, and coming to +maturity! But it would be a great error to argue that because a +well-stocked island, like Great Britain, has not, as far as is known +(and it would be very difficult to prove this), received within the +last few centuries, through occasional means +of transport, immigrants from Europe or any other continent, that a +poorly-stocked island, though standing more remote from the mainland, +would not receive colonists by similar means. I do not doubt that out +of twenty seeds or animals transported to an island, even if far less +well-stocked than Britain, scarcely more than one would be so well +fitted to its new home, as to become naturalised. But this, as it seems +to me, is no valid argument against what would be effected by +occasional means of transport, during the long lapse of geological +time, whilst an island was being upheaved and formed, and before it had +become fully stocked with inhabitants. On almost bare land, with few or +no destructive insects or birds living there, nearly every seed, which +chanced to arrive, would be sure to germinate and survive. + +_Dispersal during the Glacial period_.—The identity of many plants and +animals, on mountain-summits, separated from each other by hundreds of +miles of lowlands, where the Alpine species could not possibly exist, +is one of the most striking cases known of the same species living at +distant points, without the apparent possibility of their having +migrated from one to the other. It is indeed a remarkable fact to see +so many of the same plants living on the snowy regions of the Alps or +Pyrenees, and in the extreme northern parts of Europe; but it is far +more remarkable, that the plants on the White Mountains, in the United +States of America, are all the same with those of Labrador, and nearly +all the same, as we hear from Asa Gray, with those on the loftiest +mountains of Europe. Even as long ago as 1747, such facts led Gmelin to +conclude that the same species must have been independently created at +several distinct points; and we might have remained +in this same belief, had not Agassiz and others called vivid attention +to the Glacial period, which, as we shall immediately see, affords a +simple explanation of these facts. We have evidence of almost every +conceivable kind, organic and inorganic, that within a very recent +geological period, central Europe and North America suffered under an +Arctic climate. The ruins of a house burnt by fire do not tell their +tale more plainly, than do the mountains of Scotland and Wales, with +their scored flanks, polished surfaces, and perched boulders, of the +icy streams with which their valleys were lately filled. So greatly has +the climate of Europe changed, that in Northern Italy, gigantic +moraines, left by old glaciers, are now clothed by the vine and maize. +Throughout a large part of the United States, erratic boulders, and +rocks scored by drifted icebergs and coast-ice, plainly reveal a former +cold period. + +The former influence of the glacial climate on the distribution of the +inhabitants of Europe, as explained with remarkable clearness by Edward +Forbes, is substantially as follows. But we shall follow the changes +more readily, by supposing a new glacial period to come slowly on, and +then pass away, as formerly occurred. As the cold came on, and as each +more southern zone became fitted for arctic beings and ill-fitted for +their former more temperate inhabitants, the latter would be supplanted +and arctic productions would take their places. The inhabitants of the +more temperate regions would at the same time travel southward, unless +they were stopped by barriers, in which case they would perish. The +mountains would become covered with snow and ice, and their former +Alpine inhabitants would descend to the plains. By the time that the +cold had reached its maximum, we should have a uniform arctic fauna and +flora, covering the central parts of Europe, as far +south as the Alps and Pyrenees, and even stretching into Spain. The now +temperate regions of the United States would likewise be covered by +arctic plants and animals, and these would be nearly the same with +those of Europe; for the present circumpolar inhabitants, which we +suppose to have everywhere travelled southward, are remarkably uniform +round the world. We may suppose that the Glacial period came on a +little earlier or later in North America than in Europe, so will the +southern migration there have been a little earlier or later; but this +will make no difference in the final result. + +As the warmth returned, the arctic forms would retreat northward, +closely followed up in their retreat by the productions of the more +temperate regions. And as the snow melted from the bases of the +mountains, the arctic forms would seize on the cleared and thawed +ground, always ascending higher and higher, as the warmth increased, +whilst their brethren were pursuing their northern journey. Hence, when +the warmth had fully returned, the same arctic species, which had +lately lived in a body together on the lowlands of the Old and New +Worlds, would be left isolated on distant mountain-summits (having been +exterminated on all lesser heights) and in the arctic regions of both +hemispheres. + +Thus we can understand the identity of many plants at points so +immensely remote as on the mountains of the United States and of +Europe. We can thus also understand the fact that the Alpine plants of +each mountain-range are more especially related to the arctic forms +living due north or nearly due north of them: for the migration as the +cold came on, and the re-migration on the returning warmth, will +generally have been due south and north. The Alpine plants, for +example, of Scotland, as remarked by Mr. H. C. Watson, +and those of the Pyrenees, as remarked by Ramond, are more especially +allied to the plants of northern Scandinavia; those of the United +States to Labrador; those of the mountains of Siberia to the arctic +regions of that country. These views, grounded as they are on the +perfectly well-ascertained occurrence of a former Glacial period, seem +to me to explain in so satisfactory a manner the present distribution +of the Alpine and Arctic productions of Europe and America, that when +in other regions we find the same species on distant mountain-summits, +we may almost conclude without other evidence, that a colder climate +permitted their former migration across the low intervening tracts, +since become too warm for their existence. + +If the climate, since the Glacial period, has ever been in any degree +warmer than at present (as some geologists in the United States believe +to have been the case, chiefly from the distribution of the fossil +Gnathodon), then the arctic and temperate productions will at a very +late period have marched a little further north, and subsequently have +retreated to their present homes; but I have met with no satisfactory +evidence with respect to this intercalated slightly warmer period, +since the Glacial period. + +The arctic forms, during their long southern migration and re-migration +northward, will have been exposed to nearly the same climate, and, as +is especially to be noticed, they will have kept in a body together; +consequently their mutual relations will not have been much disturbed, +and, in accordance with the principles inculcated in this volume, they +will not have been liable to much modification. But with our Alpine +productions, left isolated from the moment of the returning warmth, +first at the bases and ultimately on the summits of the mountains, the +case will have been somewhat different; +for it is not likely that all the same arctic species will have been +left on mountain ranges distant from each other, and have survived +there ever since; they will, also, in all probability have become +mingled with ancient Alpine species, which must have existed on the +mountains before the commencement of the Glacial epoch, and which +during its coldest period will have been temporarily driven down to the +plains; they will, also, have been exposed to somewhat different +climatal influences. Their mutual relations will thus have been in some +degree disturbed; consequently they will have been liable to +modification; and this we find has been the case; for if we compare the +present Alpine plants and animals of the several great European +mountain-ranges, though very many of the species are identically the +same, some present varieties, some are ranked as doubtful forms, and +some few are distinct yet closely allied or representative species. + +In illustrating what, as I believe, actually took place during the +Glacial period, I assumed that at its commencement the arctic +productions were as uniform round the polar regions as they are at the +present day. But the foregoing remarks on distribution apply not only +to strictly arctic forms, but also to many sub-arctic and to some few +northern temperate forms, for some of these are the same on the lower +mountains and on the plains of North America and Europe; and it may be +reasonably asked how I account for the necessary degree of uniformity +of the sub-arctic and northern temperate forms round the world, at the +commencement of the Glacial period. At the present day, the sub-arctic +and northern temperate productions of the Old and New Worlds are +separated from each other by the Atlantic Ocean and by the extreme +northern part of the Pacific. During the Glacial period, when the +inhabitants +of the Old and New Worlds lived further southwards than at present, +they must have been still more completely separated by wider spaces of +ocean. I believe the above difficulty may be surmounted by looking to +still earlier changes of climate of an opposite nature. We have good +reason to believe that during the newer Pliocene period, before the +Glacial epoch, and whilst the majority of the inhabitants of the world +were specifically the same as now, the climate was warmer than at the +present day. Hence we may suppose that the organisms now living under +the climate of latitude 60°, during the Pliocene period lived further +north under the Polar Circle, in latitude 66°-67°; and that the +strictly arctic productions then lived on the broken land still nearer +to the pole. Now if we look at a globe, we shall see that under the +Polar Circle there is almost continuous land from western Europe, +through Siberia, to eastern America. And to this continuity of the +circumpolar land, and to the consequent freedom for intermigration +under a more favourable climate, I attribute the necessary amount of +uniformity in the sub-arctic and northern temperate productions of the +Old and New Worlds, at a period anterior to the Glacial epoch. + +Believing, from reasons before alluded to, that our continents have +long remained in nearly the same relative position, though subjected to +large, but partial oscillations of level, I am strongly inclined to +extend the above view, and to infer that during some earlier and still +warmer period, such as the older Pliocene period, a large number of the +same plants and animals inhabited the almost continuous circumpolar +land; and that these plants and animals, both in the Old and New +Worlds, began slowly to migrate southwards as the climate became less +warm, long before the commencement +of the Glacial period. We now see, as I believe, their descendants, +mostly in a modified condition, in the central parts of Europe and the +United States. On this view we can understand the relationship, with +very little identity, between the productions of North America and +Europe,—a relationship which is most remarkable, considering the +distance of the two areas, and their separation by the Atlantic Ocean. +We can further understand the singular fact remarked on by several +observers, that the productions of Europe and America during the later +tertiary stages were more closely related to each other than they are +at the present time; for during these warmer periods the northern parts +of the Old and New Worlds will have been almost continuously united by +land, serving as a bridge, since rendered impassable by cold, for the +inter-migration of their inhabitants. + +During the slowly decreasing warmth of the Pliocene period, as soon as +the species in common, which inhabited the New and Old Worlds, migrated +south of the Polar Circle, they must have been completely cut off from +each other. This separation, as far as the more temperate productions +are concerned, took place long ages ago. And as the plants and animals +migrated southward, they will have become mingled in the one great +region with the native American productions, and have had to compete +with them; and in the other great region, with those of the Old World. +Consequently we have here everything favourable for much +modification,—for far more modification than with the Alpine +productions, left isolated, within a much more recent period, on the +several mountain-ranges and on the arctic lands of the two Worlds. +Hence it has come, that when we compare the now living productions of +the temperate regions of the New and Old Worlds, we find very few +identical +species (though Asa Gray has lately shown that more plants are +identical than was formerly supposed), but we find in every great class +many forms, which some naturalists rank as geographical races, and +others as distinct species; and a host of closely allied or +representative forms which are ranked by all naturalists as +specifically distinct. + +As on the land, so in the waters of the sea, a slow southern migration +of a marine fauna, which during the Pliocene or even a somewhat earlier +period, was nearly uniform along the continuous shores of the Polar +Circle, will account, on the theory of modification, for many closely +allied forms now living in areas completely sundered. Thus, I think, we +can understand the presence of many existing and tertiary +representative forms on the eastern and western shores of temperate +North America; and the still more striking case of many closely allied +crustaceans (as described in Dana’s admirable work), of some fish and +other marine animals, in the Mediterranean and in the seas of +Japan,—areas now separated by a continent and by nearly a hemisphere of +equatorial ocean. + +These cases of relationship, without identity, of the inhabitants of +seas now disjoined, and likewise of the past and present inhabitants of +the temperate lands of North America and Europe, are inexplicable on +the theory of creation. We cannot say that they have been created +alike, in correspondence with the nearly similar physical conditions of +the areas; for if we compare, for instance, certain parts of South +America with the southern continents of the Old World, we see countries +closely corresponding in all their physical conditions, but with their +inhabitants utterly dissimilar. + +But we must return to our more immediate subject, the Glacial period. I +am convinced that Forbes’s view +may be largely extended. In Europe we have the plainest evidence of the +cold period, from the western shores of Britain to the Oural range, and +southward to the Pyrenees. We may infer, from the frozen mammals and +nature of the mountain vegetation, that Siberia was similarly affected. +Along the Himalaya, at points 900 miles apart, glaciers have left the +marks of their former low descent; and in Sikkim, Dr. Hooker saw maize +growing on gigantic ancient moraines. South of the equator, we have +some direct evidence of former glacial action in New Zealand; and the +same plants, found on widely separated mountains in this island, tell +the same story. If one account which has been published can be trusted, +we have direct evidence of glacial action in the south-eastern corner +of Australia. + +Looking to America; in the northern half, ice-borne fragments of rock +have been observed on the eastern side as far south as lat. 36°-37°, +and on the shores of the Pacific, where the climate is now so +different, as far south as lat. 46 deg; erratic boulders have, also, +been noticed on the Rocky Mountains. In the Cordillera of Equatorial +South America, glaciers once extended far below their present level. In +central Chile I was astonished at the structure of a vast mound of +detritus, about 800 feet in height, crossing a valley of the Andes; and +this I now feel convinced was a gigantic moraine, left far below any +existing glacier. Further south on both sides of the continent, from +lat. 41° to the southernmost extremity, we have the clearest evidence +of former glacial action, in huge boulders transported far from their +parent source. + +We do not know that the Glacial epoch was strictly simultaneous at +these several far distant points on opposite sides of the world. But we +have good evidence in almost every case, that the epoch was included +within +the latest geological period. We have, also, excellent evidence, that +it endured for an enormous time, as measured by years, at each point. +The cold may have come on, or have ceased, earlier at one point of the +globe than at another, but seeing that it endured for long at each, and +that it was contemporaneous in a geological sense, it seems to me +probable that it was, during a part at least of the period, actually +simultaneous throughout the world. Without some distinct evidence to +the contrary, we may at least admit as probable that the glacial action +was simultaneous on the eastern and western sides of North America, in +the Cordillera under the equator and under the warmer temperate zones, +and on both sides of the southern extremity of the continent. If this +be admitted, it is difficult to avoid believing that the temperature of +the whole world was at this period simultaneously cooler. But it would +suffice for my purpose, if the temperature was at the same time lower +along certain broad belts of longitude. + +On this view of the whole world, or at least of broad longitudinal +belts, having been simultaneously colder from pole to pole, much light +can be thrown on the present distribution of identical and allied +species. In America, Dr. Hooker has shown that between forty and fifty +of the flowering plants of Tierra del Fuego, forming no inconsiderable +part of its scanty flora, are common to Europe, enormously remote as +these two points are; and there are many closely allied species. On the +lofty mountains of equatorial America a host of peculiar species +belonging to European genera occur. On the highest mountains of Brazil, +some few European genera were found by Gardner, which do not exist in +the wide intervening hot countries. So on the Silla of Caraccas the +illustrious Humboldt long ago found species belonging +to genera characteristic of the Cordillera. On the mountains of +Abyssinia, several European forms and some few representatives of the +peculiar flora of the Cape of Good Hope occur. At the Cape of Good Hope +a very few European species, believed not to have been introduced by +man, and on the mountains, some few representative European forms are +found, which have not been discovered in the intertropical parts of +Africa. On the Himalaya, and on the isolated mountain-ranges of the +peninsula of India, on the heights of Ceylon, and on the volcanic cones +of Java, many plants occur, either identically the same or representing +each other, and at the same time representing plants of Europe, not +found in the intervening hot lowlands. A list of the genera collected +on the loftier peaks of Java raises a picture of a collection made on a +hill in Europe! Still more striking is the fact that southern +Australian forms are clearly represented by plants growing on the +summits of the mountains of Borneo. Some of these Australian forms, as +I hear from Dr. Hooker, extend along the heights of the peninsula of +Malacca, and are thinly scattered, on the one hand over India and on +the other as far north as Japan. + +On the southern mountains of Australia, Dr. F. Müller has discovered +several European species; other species, not introduced by man, occur +on the lowlands; and a long list can be given, as I am informed by Dr. +Hooker, of European genera, found in Australia, but not in the +intermediate torrid regions. In the admirable ‘Introduction to the +Flora of New Zealand,’ by Dr. Hooker, analogous and striking facts are +given in regard to the plants of that large island. Hence we see that +throughout the world, the plants growing on the more lofty mountains, +and on the temperate lowlands of the northern and southern hemispheres, +are sometimes +identically the same; but they are much oftener specifically distinct, +though related to each other in a most remarkable manner. + +This brief abstract applies to plants alone: some strictly analogous +facts could be given on the distribution of terrestrial animals. In +marine productions, similar cases occur; as an example, I may quote a +remark by the highest authority, Professor Dana, that “it is certainly +a wonderful fact that New Zealand should have a closer resemblance in +its crustacea to Great Britain, its antipode, than to any other part of +the world.” Sir J. Richardson, also, speaks of the reappearance on the +shores of New Zealand, Tasmania, etc., of northern forms of fish. Dr. +Hooker informs me that twenty-five species of Algæ are common to New +Zealand and to Europe, but have not been found in the intermediate +tropical seas. + +It should be observed that the northern species and forms found in the +southern parts of the southern hemisphere, and on the mountain-ranges +of the intertropical regions, are not arctic, but belong to the +northern temperate zones. As Mr. H. C. Watson has recently remarked, +“In receding from polar towards equatorial latitudes, the Alpine or +mountain floras really become less and less arctic.” Many of the forms +living on the mountains of the warmer regions of the earth and in the +southern hemisphere are of doubtful value, being ranked by some +naturalists as specifically distinct, by others as varieties; but some +are certainly identical, and many, though closely related to northern +forms, must be ranked as distinct species. + +Now let us see what light can be thrown on the foregoing facts, on the +belief, supported as it is by a large body of geological evidence, that +the whole world, or a large part of it, was during the Glacial period +simultaneously much +colder than at present. The Glacial period, as measured by years, must +have been very long; and when we remember over what vast spaces some +naturalised plants and animals have spread within a few centuries, this +period will have been ample for any amount of migration. As the cold +came slowly on, all the tropical plants and other productions will have +retreated from both sides towards the equator, followed in the rear by +the temperate productions, and these by the arctic; but with the latter +we are not now concerned. The tropical plants probably suffered much +extinction; how much no one can say; perhaps formerly the tropics +supported as many species as we see at the present day crowded together +at the Cape of Good Hope, and in parts of temperate Australia. As we +know that many tropical plants and animals can withstand a considerable +amount of cold, many might have escaped extermination during a moderate +fall of temperature, more especially by escaping into the warmest +spots. But the great fact to bear in mind is, that all tropical +productions will have suffered to a certain extent. On the other hand, +the temperate productions, after migrating nearer to the equator, +though they will have been placed under somewhat new conditions, will +have suffered less. And it is certain that many temperate plants, if +protected from the inroads of competitors, can withstand a much warmer +climate than their own. Hence, it seems to me possible, bearing in mind +that the tropical productions were in a suffering state and could not +have presented a firm front against intruders, that a certain number of +the more vigorous and dominant temperate forms might have penetrated +the native ranks and have reached or even crossed the equator. The +invasion would, of course, have been greatly favoured by high land, and +perhaps +by a dry climate; for Dr. Falconer informs me that it is the damp with +the heat of the tropics which is so destructive to perennial plants +from a temperate climate. On the other hand, the most humid and hottest +districts will have afforded an asylum to the tropical natives. The +mountain-ranges north-west of the Himalaya, and the long line of the +Cordillera, seem to have afforded two great lines of invasion: and it +is a striking fact, lately communicated to me by Dr. Hooker, that all +the flowering plants, about forty-six in number, common to Tierra del +Fuego and to Europe still exist in North America, which must have lain +on the line of march. But I do not doubt that some temperate +productions entered and crossed even the _lowlands_ of the tropics at +the period when the cold was most intense,—when arctic forms had +migrated some twenty-five degrees of latitude from their native country +and covered the land at the foot of the Pyrenees. At this period of +extreme cold, I believe that the climate under the equator at the level +of the sea was about the same with that now felt there at the height of +six or seven thousand feet. During this the coldest period, I suppose +that large spaces of the tropical lowlands were clothed with a mingled +tropical and temperate vegetation, like that now growing with strange +luxuriance at the base of the Himalaya, as graphically described by +Hooker. + +Thus, as I believe, a considerable number of plants, a few terrestrial +animals, and some marine productions, migrated during the Glacial +period from the northern and southern temperate zones into the +intertropical regions, and some even crossed the equator. As the warmth +returned, these temperate forms would naturally ascend the higher +mountains, being exterminated on the lowlands; those which had not +reached the equator, would re-migrate northward or southward towards +their former +homes; but the forms, chiefly northern, which had crossed the equator, +would travel still further from their homes into the more temperate +latitudes of the opposite hemisphere. Although we have reason to +believe from geological evidence that the whole body of arctic shells +underwent scarcely any modification during their long southern +migration and re-migration northward, the case may have been wholly +different with those intruding forms which settled themselves on the +intertropical mountains, and in the southern hemisphere. These being +surrounded by strangers will have had to compete with many new forms of +life; and it is probable that selected modifications in their +structure, habits, and constitutions will have profited them. Thus many +of these wanderers, though still plainly related by inheritance to +their brethren of the northern or southern hemispheres, now exist in +their new homes as well-marked varieties or as distinct species. + +It is a remarkable fact, strongly insisted on by Hooker in regard to +America, and by Alph. de Candolle in regard to Australia, that many +more identical plants and allied forms have apparently migrated from +the north to the south, than in a reversed direction. We see, however, +a few southern vegetable forms on the mountains of Borneo and +Abyssinia. I suspect that this preponderant migration from north to +south is due to the greater extent of land in the north, and to the +northern forms having existed in their own homes in greater numbers, +and having consequently been advanced through natural selection and +competition to a higher stage of perfection or dominating power, than +the southern forms. And thus, when they became commingled during the +Glacial period, the northern forms were enabled to beat the less +powerful southern forms. Just in the same manner as we see at the +present day, +that very many European productions cover the ground in La Plata, and +in a lesser degree in Australia, and have to a certain extent beaten +the natives; whereas extremely few southern forms have become +naturalised in any part of Europe, though hides, wool, and other +objects likely to carry seeds have been largely imported into Europe +during the last two or three centuries from La Plata, and during the +last thirty or forty years from Australia. Something of the same kind +must have occurred on the intertropical mountains: no doubt before the +Glacial period they were stocked with endemic Alpine forms; but these +have almost everywhere largely yielded to the more dominant forms, +generated in the larger areas and more efficient workshops of the +north. In many islands the native productions are nearly equalled or +even outnumbered by the naturalised; and if the natives have not been +actually exterminated, their numbers have been greatly reduced, and +this is the first stage towards extinction. A mountain is an island on +the land; and the intertropical mountains before the Glacial period +must have been completely isolated; and I believe that the productions +of these islands on the land yielded to those produced within the +larger areas of the north, just in the same way as the productions of +real islands have everywhere lately yielded to continental forms, +naturalised by man’s agency. + +I am far from supposing that all difficulties are removed on the view +here given in regard to the range and affinities of the allied species +which live in the northern and southern temperate zones and on the +mountains of the intertropical regions. Very many difficulties remain +to be solved. I do not pretend to indicate the exact lines and means of +migration, or the reason why certain species and not others have +migrated; +why certain species have been modified and have given rise to new +groups of forms, and others have remained unaltered. We cannot hope to +explain such facts, until we can say why one species and not another +becomes naturalised by man’s agency in a foreign land; why one ranges +twice or thrice as far, and is twice or thrice as common, as another +species within their own homes. + +I have said that many difficulties remain to be solved: some of the +most remarkable are stated with admirable clearness by Dr. Hooker in +his botanical works on the antarctic regions. These cannot be here +discussed. I will only say that as far as regards the occurrence of +identical species at points so enormously remote as Kerguelen Land, New +Zealand, and Fuegia, I believe that towards the close of the Glacial +period, icebergs, as suggested by Lyell, have been largely concerned in +their dispersal. But the existence of several quite distinct species, +belonging to genera exclusively confined to the south, at these and +other distant points of the southern hemisphere, is, on my theory of +descent with modification, a far more remarkable case of difficulty. +For some of these species are so distinct, that we cannot suppose that +there has been time since the commencement of the Glacial period for +their migration, and for their subsequent modification to the necessary +degree. The facts seem to me to indicate that peculiar and very +distinct species have migrated in radiating lines from some common +centre; and I am inclined to look in the southern, as in the northern +hemisphere, to a former and warmer period, before the commencement of +the Glacial period, when the antarctic lands, now covered with ice, +supported a highly peculiar and isolated flora. I suspect that before +this flora was exterminated by the Glacial epoch, a few forms were +widely dispersed to various points of the southern hemisphere by +occasional means of transport, and by the aid, as halting-places, of +existing and now sunken islands, and perhaps at the commencement of the +Glacial period, by icebergs. By these means, as I believe, the southern +shores of America, Australia, New Zealand have become slightly tinted +by the same peculiar forms of vegetable life. + +Sir C. Lyell in a striking passage has speculated, in language almost +identical with mine, on the effects of great alternations of climate on +geographical distribution. I believe that the world has recently felt +one of his great cycles of change; and that on this view, combined with +modification through natural selection, a multitude of facts in the +present distribution both of the same and of allied forms of life can +be explained. The living waters may be said to have flowed during one +short period from the north and from the south, and to have crossed at +the equator; but to have flowed with greater force from the north so as +to have freely inundated the south. As the tide leaves its drift in +horizontal lines, though rising higher on the shores where the tide +rises highest, so have the living waters left their living drift on our +mountain-summits, in a line gently rising from the arctic lowlands to a +great height under the equator. The various beings thus left stranded +may be compared with savage races of man, driven up and surviving in +the mountain-fastnesses of almost every land, which serve as a record, +full of interest to us, of the former inhabitants of the surrounding +lowlands. + +CHAPTER XII. +GEOGRAPHICAL DISTRIBUTION—_continued_. + +Distribution of fresh-water productions. On the inhabitants of oceanic +islands. Absence of Batrachians and of terrestrial Mammals. On the +relation of the inhabitants of islands to those of the nearest +mainland. On colonisation from the nearest source with subsequent +modification. Summary of the last and present chapters. + +As lakes and river-systems are separated from each other by barriers of +land, it might have been thought that fresh-water productions would not +have ranged widely within the same country, and as the sea is +apparently a still more impassable barrier, that they never would have +extended to distant countries. But the case is exactly the reverse. Not +only have many fresh-water species, belonging to quite different +classes, an enormous range, but allied species prevail in a remarkable +manner throughout the world. I well remember, when first collecting in +the fresh waters of Brazil, feeling much surprise at the similarity of +the fresh-water insects, shells, etc., and at the dissimilarity of the +surrounding terrestrial beings, compared with those of Britain. + +But this power in fresh-water productions of ranging widely, though so +unexpected, can, I think, in most cases be explained by their having +become fitted, in a manner highly useful to them, for short and +frequent migrations from pond to pond, or from stream to stream; and +liability to wide dispersal would follow from this capacity as an +almost necessary consequence. We can here consider only a few cases. In +regard to +fish, I believe that the same species never occur in the fresh waters +of distant continents. But on the same continent the species often +range widely and almost capriciously; for two river-systems will have +some fish in common and some different. A few facts seem to favour the +possibility of their occasional transport by accidental means; like +that of the live fish not rarely dropped by whirlwinds in India, and +the vitality of their ova when removed from the water. But I am +inclined to attribute the dispersal of fresh-water fish mainly to +slight changes within the recent period in the level of the land, +having caused rivers to flow into each other. Instances, also, could be +given of this having occurred during floods, without any change of +level. We have evidence in the loess of the Rhine of considerable +changes of level in the land within a very recent geological period, +and when the surface was peopled by existing land and fresh-water +shells. The wide difference of the fish on opposite sides of continuous +mountain-ranges, which from an early period must have parted +river-systems and completely prevented their inosculation, seems to +lead to this same conclusion. With respect to allied fresh-water fish +occurring at very distant points of the world, no doubt there are many +cases which cannot at present be explained: but some fresh-water fish +belong to very ancient forms, and in such cases there will have been +ample time for great geographical changes, and consequently time and +means for much migration. In the second place, salt-water fish can with +care be slowly accustomed to live in fresh water; and, according to +Valenciennes, there is hardly a single group of fishes confined +exclusively to fresh water, so that we may imagine that a marine member +of a fresh-water group might travel far along the shores of the sea, +and subsequently +become modified and adapted to the fresh waters of a distant land. + +Some species of fresh-water shells have a very wide range, and allied +species, which, on my theory, are descended from a common parent and +must have proceeded from a single source, prevail throughout the world. +Their distribution at first perplexed me much, as their ova are not +likely to be transported by birds, and they are immediately killed by +sea water, as are the adults. I could not even understand how some +naturalised species have rapidly spread throughout the same country. +But two facts, which I have observed—and no doubt many others remain to +be observed—throw some light on this subject. When a duck suddenly +emerges from a pond covered with duck-weed, I have twice seen these +little plants adhering to its back; and it has happened to me, in +removing a little duck-weed from one aquarium to another, that I have +quite unintentionally stocked the one with fresh-water shells from the +other. But another agency is perhaps more effectual: I suspended a +duck’s feet, which might represent those of a bird sleeping in a +natural pond, in an aquarium, where many ova of fresh-water shells were +hatching; and I found that numbers of the extremely minute and just +hatched shells crawled on the feet, and clung to them so firmly that +when taken out of the water they could not be jarred off, though at a +somewhat more advanced age they would voluntarily drop off. These just +hatched molluscs, though aquatic in their nature, survived on the +duck’s feet, in damp air, from twelve to twenty hours; and in this +length of time a duck or heron might fly at least six or seven hundred +miles, and would be sure to alight on a pool or rivulet, if blown +across sea to an oceanic island or to any other distant point. Sir +Charles Lyell also +informs me that a Dyticus has been caught with an Ancylus (a +fresh-water shell like a limpet) firmly adhering to it; and a +water-beetle of the same family, a Colymbetes, once flew on board the +‘Beagle,’ when forty-five miles distant from the nearest land: how much +farther it might have flown with a favouring gale no one can tell. + +With respect to plants, it has long been known what enormous ranges +many fresh-water and even marsh-species have, both over continents and +to the most remote oceanic islands. This is strikingly shown, as +remarked by Alph. de Candolle, in large groups of terrestrial plants, +which have only a very few aquatic members; for these latter seem +immediately to acquire, as if in consequence, a very wide range. I +think favourable means of dispersal explain this fact. I have before +mentioned that earth occasionally, though rarely, adheres in some +quantity to the feet and beaks of birds. Wading birds, which frequent +the muddy edges of ponds, if suddenly flushed, would be the most likely +to have muddy feet. Birds of this order I can show are the greatest +wanderers, and are occasionally found on the most remote and barren +islands in the open ocean; they would not be likely to alight on the +surface of the sea, so that the dirt would not be washed off their +feet; when making land, they would be sure to fly to their natural +fresh-water haunts. I do not believe that botanists are aware how +charged the mud of ponds is with seeds: I have tried several little +experiments, but will here give only the most striking case: I took in +February three table-spoonfuls of mud from three different points, +beneath water, on the edge of a little pond; this mud when dry weighed +only 6 3/4 ounces; I kept it covered up in my study for six months, +pulling up and counting each plant as it grew; the plants were +of many kinds, and were altogether 537 in number; and yet the viscid +mud was all contained in a breakfast cup! Considering these facts, I +think it would be an inexplicable circumstance if water-birds did not +transport the seeds of fresh-water plants to vast distances, and if +consequently the range of these plants was not very great. The same +agency may have come into play with the eggs of some of the smaller +fresh-water animals. + +Other and unknown agencies probably have also played a part. I have +stated that fresh-water fish eat some kinds of seeds, though they +reject many other kinds after having swallowed them; even small fish +swallow seeds of moderate size, as of the yellow water-lily and +Potamogeton. Herons and other birds, century after century, have gone +on daily devouring fish; they then take flight and go to other waters, +or are blown across the sea; and we have seen that seeds retain their +power of germination, when rejected in pellets or in excrement, many +hours afterwards. When I saw the great size of the seeds of that fine +water-lily, the Nelumbium, and remembered Alph. de Candolle’s remarks +on this plant, I thought that its distribution must remain quite +inexplicable; but Audubon states that he found the seeds of the great +southern water-lily (probably, according to Dr. Hooker, the Nelumbium +luteum) in a heron’s stomach; although I do not know the fact, yet +analogy makes me believe that a heron flying to another pond and +getting a hearty meal of fish, would probably reject from its stomach a +pellet containing the seeds of the Nelumbium undigested; or the seeds +might be dropped by the bird whilst feeding its young, in the same way +as fish are known sometimes to be dropped. + +In considering these several means of distribution, +it should be remembered that when a pond or stream is first formed, for +instance, on a rising islet, it will be unoccupied; and a single seed +or egg will have a good chance of succeeding. Although there will +always be a struggle for life between the individuals of the species, +however few, already occupying any pond, yet as the number of kinds is +small, compared with those on the land, the competition will probably +be less severe between aquatic than between terrestrial species; +consequently an intruder from the waters of a foreign country, would +have a better chance of seizing on a place, than in the case of +terrestrial colonists. We should, also, remember that some, perhaps +many, fresh-water productions are low in the scale of nature, and that +we have reason to believe that such low beings change or become +modified less quickly than the high; and this will give longer time +than the average for the migration of the same aquatic species. We +should not forget the probability of many species having formerly +ranged as continuously as fresh-water productions ever can range, over +immense areas, and having subsequently become extinct in intermediate +regions. But the wide distribution of fresh-water plants and of the +lower animals, whether retaining the same identical form or in some +degree modified, I believe mainly depends on the wide dispersal of +their seeds and eggs by animals, more especially by fresh-water birds, +which have large powers of flight, and naturally travel from one to +another and often distant piece of water. Nature, like a careful +gardener, thus takes her seeds from a bed of a particular nature, and +drops them in another equally well fitted for them. + +_On the Inhabitants of Oceanic Islands_.—We now come to the last of the +three classes of facts, which I +have selected as presenting the greatest amount of difficulty, on the +view that all the individuals both of the same and of allied species +have descended from a single parent; and therefore have all proceeded +from a common birthplace, notwithstanding that in the course of time +they have come to inhabit distant points of the globe. I have already +stated that I cannot honestly admit Forbes’s view on continental +extensions, which, if legitimately followed out, would lead to the +belief that within the recent period all existing islands have been +nearly or quite joined to some continent. This view would remove many +difficulties, but it would not, I think, explain all the facts in +regard to insular productions. In the following remarks I shall not +confine myself to the mere question of dispersal; but shall consider +some other facts, which bear on the truth of the two theories of +independent creation and of descent with modification. + +The species of all kinds which inhabit oceanic islands are few in +number compared with those on equal continental areas: Alph. de +Candolle admits this for plants, and Wollaston for insects. If we look +to the large size and varied stations of New Zealand, extending over +780 miles of latitude, and compare its flowering plants, only 750 in +number, with those on an equal area at the Cape of Good Hope or in +Australia, we must, I think, admit that something quite independently +of any difference in physical conditions has caused so great a +difference in number. Even the uniform county of Cambridge has 847 +plants, and the little island of Anglesea 764, but a few ferns and a +few introduced plants are included in these numbers, and the comparison +in some other respects is not quite fair. We have evidence that the +barren island of Ascension aboriginally possessed under half-a-dozen +flowering +plants; yet many have become naturalised on it, as they have on New +Zealand and on every other oceanic island which can be named. In St. +Helena there is reason to believe that the naturalised plants and +animals have nearly or quite exterminated many native productions. He +who admits the doctrine of the creation of each separate species, will +have to admit, that a sufficient number of the best adapted plants and +animals have not been created on oceanic islands; for man has +unintentionally stocked them from various sources far more fully and +perfectly than has nature. + +Although in oceanic islands the number of kinds of inhabitants is +scanty, the proportion of endemic species (_i.e._ those found nowhere +else in the world) is often extremely large. If we compare, for +instance, the number of the endemic land-shells in Madeira, or of the +endemic birds in the Galapagos Archipelago, with the number found on +any continent, and then compare the area of the islands with that of +the continent, we shall see that this is true. This fact might have +been expected on my theory, for, as already explained, species +occasionally arriving after long intervals in a new and isolated +district, and having to compete with new associates, will be eminently +liable to modification, and will often produce groups of modified +descendants. But it by no means follows, that, because in an island +nearly all the species of one class are peculiar, those of another +class, or of another section of the same class, are peculiar; and this +difference seems to depend on the species which do not become modified +having immigrated with facility and in a body, so that their mutual +relations have not been much disturbed. Thus in the Galapagos Islands +nearly every land-bird, but only two out of the eleven marine birds, +are peculiar; and it is obvious that +marine birds could arrive at these islands more easily than land-birds. +Bermuda, on the other hand, which lies at about the same distance from +North America as the Galapagos Islands do from South America, and which +has a very peculiar soil, does not possess one endemic land bird; and +we know from Mr. J. M. Jones’s admirable account of Bermuda, that very +many North American birds, during their great annual migrations, visit +either periodically or occasionally this island. Madeira does not +possess one peculiar bird, and many European and African birds are +almost every year blown there, as I am informed by Mr. E. V. Harcourt. +So that these two islands of Bermuda and Madeira have been stocked by +birds, which for long ages have struggled together in their former +homes, and have become mutually adapted to each other; and when settled +in their new homes, each kind will have been kept by the others to +their proper places and habits, and will consequently have been little +liable to modification. Madeira, again, is inhabited by a wonderful +number of peculiar land-shells, whereas not one species of sea-shell is +confined to its shores: now, though we do not know how seashells are +dispersed, yet we can see that their eggs or larvæ, perhaps attached to +seaweed or floating timber, or to the feet of wading-birds, might be +transported far more easily than land-shells, across three or four +hundred miles of open sea. The different orders of insects in Madeira +apparently present analogous facts. + +Oceanic islands are sometimes deficient in certain classes, and their +places are apparently occupied by the other inhabitants; in the +Galapagos Islands reptiles, and in New Zealand gigantic wingless birds, +take the place of mammals. In the plants of the Galapagos Islands, Dr. +Hooker has shown that the proportional numbers of the different orders +are very different from +what they are elsewhere. Such cases are generally accounted for by the +physical conditions of the islands; but this explanation seems to me +not a little doubtful. Facility of immigration, I believe, has been at +least as important as the nature of the conditions. + +Many remarkable little facts could be given with respect to the +inhabitants of remote islands. For instance, in certain islands not +tenanted by mammals, some of the endemic plants have beautifully hooked +seeds; yet few relations are more striking than the adaptation of +hooked seeds for transportal by the wool and fur of quadrupeds. This +case presents no difficulty on my view, for a hooked seed might be +transported to an island by some other means; and the plant then +becoming slightly modified, but still retaining its hooked seeds, would +form an endemic species, having as useless an appendage as any +rudimentary organ,—for instance, as the shrivelled wings under the +soldered elytra of many insular beetles. Again, islands often possess +trees or bushes belonging to orders which elsewhere include only +herbaceous species; now trees, as Alph. de Candolle has shown, +generally have, whatever the cause may be, confined ranges. Hence trees +would be little likely to reach distant oceanic islands; and an +herbaceous plant, though it would have no chance of successfully +competing in stature with a fully developed tree, when established on +an island and having to compete with herbaceous plants alone, might +readily gain an advantage by growing taller and taller and overtopping +the other plants. If so, natural selection would often tend to add to +the stature of herbaceous plants when growing on an island, to whatever +order they belonged, and thus convert them first into bushes and +ultimately into trees. + +With respect to the absence of whole orders on +oceanic islands, Bory St. Vincent long ago remarked that Batrachians +(frogs, toads, newts) have never been found on any of the many islands +with which the great oceans are studded. I have taken pains to verify +this assertion, and I have found it strictly true. I have, however, +been assured that a frog exists on the mountains of the great island of +New Zealand; but I suspect that this exception (if the information be +correct) may be explained through glacial agency. This general absence +of frogs, toads, and newts on so many oceanic islands cannot be +accounted for by their physical conditions; indeed it seems that +islands are peculiarly well fitted for these animals; for frogs have +been introduced into Madeira, the Azores, and Mauritius, and have +multiplied so as to become a nuisance. But as these animals and their +spawn are known to be immediately killed by sea-water, on my view we +can see that there would be great difficulty in their transportal +across the sea, and therefore why they do not exist on any oceanic +island. But why, on the theory of creation, they should not have been +created there, it would be very difficult to explain. + +Mammals offer another and similar case. I have carefully searched the +oldest voyages, but have not finished my search; as yet I have not +found a single instance, free from doubt, of a terrestrial mammal +(excluding domesticated animals kept by the natives) inhabiting an +island situated above 300 miles from a continent or great continental +island; and many islands situated at a much less distance are equally +barren. The Falkland Islands, which are inhabited by a wolf-like fox, +come nearest to an exception; but this group cannot be considered as +oceanic, as it lies on a bank connected with the mainland; moreover, +icebergs formerly brought boulders to its western shores, and they may +have formerly transported foxes, as so frequently now happens in the +arctic regions. Yet it cannot be said that small islands will not +support small mammals, for they occur in many parts of the world on +very small islands, if close to a continent; and hardly an island can +be named on which our smaller quadrupeds have not become naturalised +and greatly multiplied. It cannot be said, on the ordinary view of +creation, that there has not been time for the creation of mammals; +many volcanic islands are sufficiently ancient, as shown by the +stupendous degradation which they have suffered and by their tertiary +strata: there has also been time for the production of endemic species +belonging to other classes; and on continents it is thought that +mammals appear and disappear at a quicker rate than other and lower +animals. Though terrestrial mammals do not occur on oceanic islands, +ærial mammals do occur on almost every island. New Zealand possesses +two bats found nowhere else in the world: Norfolk Island, the Viti +Archipelago, the Bonin Islands, the Caroline and Marianne +Archipelagoes, and Mauritius, all possess their peculiar bats. Why, it +may be asked, has the supposed creative force produced bats and no +other mammals on remote islands? On my view this question can easily be +answered; for no terrestrial mammal can be transported across a wide +space of sea, but bats can fly across. Bats have been seen wandering by +day far over the Atlantic Ocean; and two North American species either +regularly or occasionally visit Bermuda, at the distance of 600 miles +from the mainland. I hear from Mr. Tomes, who has specially studied +this family, that many of the same species have enormous ranges, and +are found on continents and on far distant islands. Hence we have only +to suppose that such wandering species have been modified +through natural selection in their new homes in relation to their new +position, and we can understand the presence of endemic bats on +islands, with the absence of all terrestrial mammals. + +Besides the absence of terrestrial mammals in relation to the +remoteness of islands from continents, there is also a relation, to a +certain extent independent of distance, between the depth of the sea +separating an island from the neighbouring mainland, and the presence +in both of the same mammiferous species or of allied species in a more +or less modified condition. Mr. Windsor Earl has made some striking +observations on this head in regard to the great Malay Archipelago, +which is traversed near Celebes by a space of deep ocean; and this +space separates two widely distinct mammalian faunas. On either side +the islands are situated on moderately deep submarine banks, and they +are inhabited by closely allied or identical quadrupeds. No doubt some +few anomalies occur in this great archipelago, and there is much +difficulty in forming a judgment in some cases owing to the probable +naturalisation of certain mammals through man’s agency; but we shall +soon have much light thrown on the natural history of this archipelago +by the admirable zeal and researches of Mr. Wallace. I have not as yet +had time to follow up this subject in all other quarters of the world; +but as far as I have gone, the relation generally holds good. We see +Britain separated by a shallow channel from Europe, and the mammals are +the same on both sides; we meet with analogous facts on many islands +separated by similar channels from Australia. The West Indian Islands +stand on a deeply submerged bank, nearly 1000 fathoms in depth, and +here we find American forms, but the species and even the genera are +distinct. As the amount of modification in all cases depends to +a certain degree on the lapse of time, and as during changes of level +it is obvious that islands separated by shallow channels are more +likely to have been continuously united within a recent period to the +mainland than islands separated by deeper channels, we can understand +the frequent relation between the depth of the sea and the degree of +affinity of the mammalian inhabitants of islands with those of a +neighbouring continent,—an inexplicable relation on the view of +independent acts of creation. + +All the foregoing remarks on the inhabitants of oceanic +islands,—namely, the scarcity of kinds—the richness in endemic forms in +particular classes or sections of classes,—the absence of whole groups, +as of batrachians, and of terrestrial mammals notwithstanding the +presence of ærial bats,—the singular proportions of certain orders of +plants,—herbaceous forms having been developed into trees, etc.,—seem +to me to accord better with the view of occasional means of transport +having been largely efficient in the long course of time, than with the +view of all our oceanic islands having been formerly connected by +continuous land with the nearest continent; for on this latter view the +migration would probably have been more complete; and if modification +be admitted, all the forms of life would have been more equally +modified, in accordance with the paramount importance of the relation +of organism to organism. + +I do not deny that there are many and grave difficulties in +understanding how several of the inhabitants of the more remote +islands, whether still retaining the same specific form or modified +since their arrival, could have reached their present homes. But the +probability of many islands having existed as halting-places, of which +not a wreck now remains, must not be overlooked. +I will here give a single instance of one of the cases of difficulty. +Almost all oceanic islands, even the most isolated and smallest, are +inhabited by land-shells, generally by endemic species, but sometimes +by species found elsewhere. Dr. Aug. A. Gould has given several +interesting cases in regard to the land-shells of the islands of the +Pacific. Now it is notorious that land-shells are very easily killed by +salt; their eggs, at least such as I have tried, sink in sea-water and +are killed by it. Yet there must be, on my view, some unknown, but +highly efficient means for their transportal. Would the just-hatched +young occasionally crawl on and adhere to the feet of birds roosting on +the ground, and thus get transported? It occurred to me that +land-shells, when hybernating and having a membranous diaphragm over +the mouth of the shell, might be floated in chinks of drifted timber +across moderately wide arms of the sea. And I found that several +species did in this state withstand uninjured an immersion in sea-water +during seven days: one of these shells was the Helix pomatia, and after +it had again hybernated I put it in sea-water for twenty days, and it +perfectly recovered. As this species has a thick calcareous operculum, +I removed it, and when it had formed a new membranous one, I immersed +it for fourteen days in sea-water, and it recovered and crawled away: +but more experiments are wanted on this head. + +The most striking and important fact for us in regard to the +inhabitants of islands, is their affinity to those of the nearest +mainland, without being actually the same species. Numerous instances +could be given of this fact. I will give only one, that of the +Galapagos Archipelago, situated under the equator, between 500 and 600 +miles from the shores of South America. Here +almost every product of the land and water bears the unmistakeable +stamp of the American continent. There are twenty-six land birds, and +twenty-five of these are ranked by Mr. Gould as distinct species, +supposed to have been created here; yet the close affinity of most of +these birds to American species in every character, in their habits, +gestures, and tones of voice, was manifest. So it is with the other +animals, and with nearly all the plants, as shown by Dr. Hooker in his +admirable memoir on the Flora of this archipelago. The naturalist, +looking at the inhabitants of these volcanic islands in the Pacific, +distant several hundred miles from the continent, yet feels that he is +standing on American land. Why should this be so? why should the +species which are supposed to have been created in the Galapagos +Archipelago, and nowhere else, bear so plain a stamp of affinity to +those created in America? There is nothing in the conditions of life, +in the geological nature of the islands, in their height or climate, or +in the proportions in which the several classes are associated +together, which resembles closely the conditions of the South American +coast: in fact there is a considerable dissimilarity in all these +respects. On the other hand, there is a considerable degree of +resemblance in the volcanic nature of the soil, in climate, height, and +size of the islands, between the Galapagos and Cape de Verde +Archipelagos: but what an entire and absolute difference in their +inhabitants! The inhabitants of the Cape de Verde Islands are related +to those of Africa, like those of the Galapagos to America. I believe +this grand fact can receive no sort of explanation on the ordinary view +of independent creation; whereas on the view here maintained, it is +obvious that the Galapagos Islands would be likely to receive +colonists, whether by occasional means of transport or +by formerly continuous land, from America; and the Cape de Verde +Islands from Africa; and that such colonists would be liable to +modification;—the principle of inheritance still betraying their +original birthplace. + +Many analogous facts could be given: indeed it is an almost universal +rule that the endemic productions of islands are related to those of +the nearest continent, or of other near islands. The exceptions are +few, and most of them can be explained. Thus the plants of Kerguelen +Land, though standing nearer to Africa than to America, are related, +and that very closely, as we know from Dr. Hooker’s account, to those +of America: but on the view that this island has been mainly stocked by +seeds brought with earth and stones on icebergs, drifted by the +prevailing currents, this anomaly disappears. New Zealand in its +endemic plants is much more closely related to Australia, the nearest +mainland, than to any other region: and this is what might have been +expected; but it is also plainly related to South America, which, +although the next nearest continent, is so enormously remote, that the +fact becomes an anomaly. But this difficulty almost disappears on the +view that both New Zealand, South America, and other southern lands +were long ago partially stocked from a nearly intermediate though +distant point, namely from the antarctic islands, when they were +clothed with vegetation, before the commencement of the Glacial period. +The affinity, which, though feeble, I am assured by Dr. Hooker is real, +between the flora of the south-western corner of Australia and of the +Cape of Good Hope, is a far more remarkable case, and is at present +inexplicable: but this affinity is confined to the plants, and will, I +do not doubt, be some day explained. + +The law which causes the inhabitants of an archipelago, +though specifically distinct, to be closely allied to those of the +nearest continent, we sometimes see displayed on a small scale, yet in +a most interesting manner, within the limits of the same archipelago. +Thus the several islands of the Galapagos Archipelago are tenanted, as +I have elsewhere shown, in a quite marvellous manner, by very closely +related species; so that the inhabitants of each separate island, +though mostly distinct, are related in an incomparably closer degree to +each other than to the inhabitants of any other part of the world. And +this is just what might have been expected on my view, for the islands +are situated so near each other that they would almost certainly +receive immigrants from the same original source, or from each other. +But this dissimilarity between the endemic inhabitants of the islands +may be used as an argument against my views; for it may be asked, how +has it happened in the several islands situated within sight of each +other, having the same geological nature, the same height, climate, +etc., that many of the immigrants should have been differently +modified, though only in a small degree. This long appeared to me a +great difficulty: but it arises in chief part from the deeply-seated +error of considering the physical conditions of a country as the most +important for its inhabitants; whereas it cannot, I think, be disputed +that the nature of the other inhabitants, with which each has to +compete, is at least as important, and generally a far more important +element of success. Now if we look to those inhabitants of the +Galapagos Archipelago which are found in other parts of the world +(laying on one side for the moment the endemic species, which cannot be +here fairly included, as we are considering how they have come to be +modified since their arrival), we find a considerable amount +of difference in the several islands. This difference might indeed have +been expected on the view of the islands having been stocked by +occasional means of transport—a seed, for instance, of one plant having +been brought to one island, and that of another plant to another +island. Hence when in former times an immigrant settled on any one or +more of the islands, or when it subsequently spread from one island to +another, it would undoubtedly be exposed to different conditions of +life in the different islands, for it would have to compete with +different sets of organisms: a plant, for instance, would find the +best-fitted ground more perfectly occupied by distinct plants in one +island than in another, and it would be exposed to the attacks of +somewhat different enemies. If then it varied, natural selection would +probably favour different varieties in the different islands. Some +species, however, might spread and yet retain the same character +throughout the group, just as we see on continents some species +spreading widely and remaining the same. + +The really surprising fact in this case of the Galapagos Archipelago, +and in a lesser degree in some analogous instances, is that the new +species formed in the separate islands have not quickly spread to the +other islands. But the islands, though in sight of each other, are +separated by deep arms of the sea, in most cases wider than the British +Channel, and there is no reason to suppose that they have at any former +period been continuously united. The currents of the sea are rapid and +sweep across the archipelago, and gales of wind are extraordinarily +rare; so that the islands are far more effectually separated from each +other than they appear to be on a map. Nevertheless a good many +species, both those found in other parts of the world and those +confined to the archipelago, are common to +the several islands, and we may infer from certain facts that these +have probably spread from some one island to the others. But we often +take, I think, an erroneous view of the probability of closely allied +species invading each other’s territory, when put into free +intercommunication. Undoubtedly if one species has any advantage +whatever over another, it will in a very brief time wholly or in part +supplant it; but if both are equally well fitted for their own places +in nature, both probably will hold their own places and keep separate +for almost any length of time. Being familiar with the fact that many +species, naturalised through man’s agency, have spread with astonishing +rapidity over new countries, we are apt to infer that most species +would thus spread; but we should remember that the forms which become +naturalised in new countries are not generally closely allied to the +aboriginal inhabitants, but are very distinct species, belonging in a +large proportion of cases, as shown by Alph. de Candolle, to distinct +genera. In the Galapagos Archipelago, many even of the birds, though so +well adapted for flying from island to island, are distinct on each; +thus there are three closely-allied species of mocking-thrush, each +confined to its own island. Now let us suppose the mocking-thrush of +Chatham Island to be blown to Charles Island, which has its own +mocking-thrush: why should it succeed in establishing itself there? We +may safely infer that Charles Island is well stocked with its own +species, for annually more eggs are laid there than can possibly be +reared; and we may infer that the mocking-thrush peculiar to Charles +Island is at least as well fitted for its home as is the species +peculiar to Chatham Island. Sir C. Lyell and Mr. Wollaston have +communicated to me a remarkable fact bearing on this subject; namely, +that Madeira and the adjoining islet of +Porto Santo possess many distinct but representative land-shells, some +of which live in crevices of stone; and although large quantities of +stone are annually transported from Porto Santo to Madeira, yet this +latter island has not become colonised by the Porto Santo species: +nevertheless both islands have been colonised by some European +land-shells, which no doubt had some advantage over the indigenous +species. From these considerations I think we need not greatly marvel +at the endemic and representative species, which inhabit the several +islands of the Galapagos Archipelago, not having universally spread +from island to island. In many other instances, as in the several +districts of the same continent, pre-occupation has probably played an +important part in checking the commingling of species under the same +conditions of life. Thus, the south-east and south-west corners of +Australia have nearly the same physical conditions, and are united by +continuous land, yet they are inhabited by a vast number of distinct +mammals, birds, and plants. + +The principle which determines the general character of the fauna and +flora of oceanic islands, namely, that the inhabitants, when not +identically the same, yet are plainly related to the inhabitants of +that region whence colonists could most readily have been derived,—the +colonists having been subsequently modified and better fitted to their +new homes,—is of the widest application throughout nature. We see this +on every mountain, in every lake and marsh. For Alpine species, +excepting in so far as the same forms, chiefly of plants, have spread +widely throughout the world during the recent Glacial epoch, are +related to those of the surrounding lowlands;—thus we have in South +America, Alpine humming-birds, Alpine rodents, Alpine plants, etc., all +of strictly American forms, and it is obvious +that a mountain, as it became slowly upheaved, would naturally be +colonised from the surrounding lowlands. So it is with the inhabitants +of lakes and marshes, excepting in so far as great facility of +transport has given the same general forms to the whole world. We see +this same principle in the blind animals inhabiting the caves of +America and of Europe. Other analogous facts could be given. And it +will, I believe, be universally found to be true, that wherever in two +regions, let them be ever so distant, many closely allied or +representative species occur, there will likewise be found some +identical species, showing, in accordance with the foregoing view, that +at some former period there has been intercommunication or migration +between the two regions. And wherever many closely-allied species +occur, there will be found many forms which some naturalists rank as +distinct species, and some as varieties; these doubtful forms showing +us the steps in the process of modification. + +This relation between the power and extent of migration of a species, +either at the present time or at some former period under different +physical conditions, and the existence at remote points of the world of +other species allied to it, is shown in another and more general way. +Mr. Gould remarked to me long ago, that in those genera of birds which +range over the world, many of the species have very wide ranges. I can +hardly doubt that this rule is generally true, though it would be +difficult to prove it. Amongst mammals, we see it strikingly displayed +in Bats, and in a lesser degree in the Felidæ and Canidæ. We see it, if +we compare the distribution of butterflies and beetles. So it is with +most fresh-water productions, in which so many genera range over the +world, and many individual species have enormous ranges. It is not +meant that in world-ranging +genera all the species have a wide range, or even that they have on an +_average_ a wide range; but only that some of the species range very +widely; for the facility with which widely-ranging species vary and +give rise to new forms will largely determine their average range. For +instance, two varieties of the same species inhabit America and Europe, +and the species thus has an immense range; but, if the variation had +been a little greater, the two varieties would have been ranked as +distinct species, and the common range would have been greatly reduced. +Still less is it meant, that a species which apparently has the +capacity of crossing barriers and ranging widely, as in the case of +certain powerfully-winged birds, will necessarily range widely; for we +should never forget that to range widely implies not only the power of +crossing barriers, but the more important power of being victorious in +distant lands in the struggle for life with foreign associates. But on +the view of all the species of a genus having descended from a single +parent, though now distributed to the most remote points of the world, +we ought to find, and I believe as a general rule we do find, that some +at least of the species range very widely; for it is necessary that the +unmodified parent should range widely, undergoing modification during +its diffusion, and should place itself under diverse conditions +favourable for the conversion of its offspring, firstly into new +varieties and ultimately into new species. + +In considering the wide distribution of certain genera, we should bear +in mind that some are extremely ancient, and must have branched off +from a common parent at a remote epoch; so that in such cases there +will have been ample time for great climatal and geographical changes +and for accidents of transport; and consequently for the migration of +some of the species into all +quarters of the world, where they may have become slightly modified in +relation to their new conditions. There is, also, some reason to +believe from geological evidence that organisms low in the scale within +each great class, generally change at a slower rate than the higher +forms; and consequently the lower forms will have had a better chance +of ranging widely and of still retaining the same specific character. +This fact, together with the seeds and eggs of many low forms being +very minute and better fitted for distant transportation, probably +accounts for a law which has long been observed, and which has lately +been admirably discussed by Alph. de Candolle in regard to plants, +namely, that the lower any group of organisms is, the more widely it is +apt to range. + +The relations just discussed,—namely, low and slowly-changing organisms +ranging more widely than the high,—some of the species of +widely-ranging genera themselves ranging widely,—such facts, as alpine, +lacustrine, and marsh productions being related (with the exceptions +before specified) to those on the surrounding low lands and dry lands, +though these stations are so different—the very close relation of the +distinct species which inhabit the islets of the same archipelago,—and +especially the striking relation of the inhabitants of each whole +archipelago or island to those of the nearest mainland,—are, I think, +utterly inexplicable on the ordinary view of the independent creation +of each species, but are explicable on the view of colonisation from +the nearest and readiest source, together with the subsequent +modification and better adaptation of the colonists to their new homes. + +_Summary of last and present Chapters_.—In these chapters I have +endeavoured to show, that if we make due allowance for our ignorance of +the full effects of all +the changes of climate and of the level of the land, which have +certainly occurred within the recent period, and of other similar +changes which may have occurred within the same period; if we remember +how profoundly ignorant we are with respect to the many and curious +means of occasional transport,—a subject which has hardly ever been +properly experimentised on; if we bear in mind how often a species may +have ranged continuously over a wide area, and then have become extinct +in the intermediate tracts, I think the difficulties in believing that +all the individuals of the same species, wherever located, have +descended from the same parents, are not insuperable. And we are led to +this conclusion, which has been arrived at by many naturalists under +the designation of single centres of creation, by some general +considerations, more especially from the importance of barriers and +from the analogical distribution of sub-genera, genera, and families. + +With respect to the distinct species of the same genus, which on my +theory must have spread from one parent-source; if we make the same +allowances as before for our ignorance, and remember that some forms of +life change most slowly, enormous periods of time being thus granted +for their migration, I do not think that the difficulties are +insuperable; though they often are in this case, and in that of the +individuals of the same species, extremely grave. + +As exemplifying the effects of climatal changes on distribution, I have +attempted to show how important has been the influence of the modern +Glacial period, which I am fully convinced simultaneously affected the +whole world, or at least great meridional belts. As showing how +diversified are the means of occasional transport, I have discussed at +some little length the means of dispersal of fresh-water productions. + +If the difficulties be not insuperable in admitting that in the long +course of time the individuals of the same species, and likewise of +allied species, have proceeded from some one source; then I think all +the grand leading facts of geographical distribution are explicable on +the theory of migration (generally of the more dominant forms of life), +together with subsequent modification and the multiplication of new +forms. We can thus understand the high importance of barriers, whether +of land or water, which separate our several zoological and botanical +provinces. We can thus understand the localisation of sub-genera, +genera, and families; and how it is that under different latitudes, for +instance in South America, the inhabitants of the plains and mountains, +of the forests, marshes, and deserts, are in so mysterious a manner +linked together by affinity, and are likewise linked to the extinct +beings which formerly inhabited the same continent. Bearing in mind +that the mutual relations of organism to organism are of the highest +importance, we can see why two areas having nearly the same physical +conditions should often be inhabited by very different forms of life; +for according to the length of time which has elapsed since new +inhabitants entered one region; according to the nature of the +communication which allowed certain forms and not others to enter, +either in greater or lesser numbers; according or not, as those which +entered happened to come in more or less direct competition with each +other and with the aborigines; and according as the immigrants were +capable of varying more or less rapidly, there would ensue in different +regions, independently of their physical conditions, infinitely +diversified conditions of life,—there would be an almost endless amount +of organic action and reaction,—and we should find, as we do find, some +groups of beings greatly, and some only slightly modified,—some +developed +in great force, some existing in scanty numbers—in the different great +geographical provinces of the world. + +On these same principles, we can understand, as I have endeavoured to +show, why oceanic islands should have few inhabitants, but of these a +great number should be endemic or peculiar; and why, in relation to the +means of migration, one group of beings, even within the same class, +should have all its species endemic, and another group should have all +its species common to other quarters of the world. We can see why whole +groups of organisms, as batrachians and terrestrial mammals, should be +absent from oceanic islands, whilst the most isolated islands possess +their own peculiar species of ærial mammals or bats. We can see why +there should be some relation between the presence of mammals, in a +more or less modified condition, and the depth of the sea between an +island and the mainland. We can clearly see why all the inhabitants of +an archipelago, though specifically distinct on the several islets, +should be closely related to each other, and likewise be related, but +less closely, to those of the nearest continent or other source whence +immigrants were probably derived. We can see why in two areas, however +distant from each other, there should be a correlation, in the presence +of identical species, of varieties, of doubtful species, and of +distinct but representative species. + +As the late Edward Forbes often insisted, there is a striking +parallelism in the laws of life throughout time and space: the laws +governing the succession of forms in past times being nearly the same +with those governing at the present time the differences in different +areas. We see this in many facts. The endurance of each species and +group of species is continuous in time; for the exceptions to the rule +are so few, that they may +fairly be attributed to our not having as yet discovered in an +intermediate deposit the forms which are therein absent, but which +occur above and below: so in space, it certainly is the general rule +that the area inhabited by a single species, or by a group of species, +is continuous; and the exceptions, which are not rare, may, as I have +attempted to show, be accounted for by migration at some former period +under different conditions or by occasional means of transport, and by +the species having become extinct in the intermediate tracts. Both in +time and space, species and groups of species have their points of +maximum development. Groups of species, belonging either to a certain +period of time, or to a certain area, are often characterised by +trifling characters in common, as of sculpture or colour. In looking to +the long succession of ages, as in now looking to distant provinces +throughout the world, we find that some organisms differ little, whilst +others belonging to a different class, or to a different order, or even +only to a different family of the same order, differ greatly. In both +time and space the lower members of each class generally change less +than the higher; but there are in both cases marked exceptions to the +rule. On my theory these several relations throughout time and space +are intelligible; for whether we look to the forms of life which have +changed during successive ages within the same quarter of the world, or +to those which have changed after having migrated into distant +quarters, in both cases the forms within each class have been connected +by the same bond of ordinary generation; and the more nearly any two +forms are related in blood, the nearer they will generally stand to +each other in time and space; in both cases the laws of variation have +been the same, and modifications have been accumulated by the same +power of natural selection. + +CLASSIFICATION, groups subordinate to groups. Natural system. Rules and +difficulties in classification, explained on the theory of descent with +modification. Classification of varieties. Descent always used in +classification. Analogical or adaptive characters. Affinities, general, +complex and radiating. Extinction separates and defines groups. +MORPHOLOGY, between members of the same class, between parts of the +same individual. EMBRYOLOGY, laws of, explained by variations not +supervening at an early age, and being inherited at a corresponding +age. RUDIMENTARY ORGANS; their origin explained. Summary. + +From the first dawn of life, all organic beings are found to resemble +each other in descending degrees, so that they can be classed in groups +under groups. This classification is evidently not arbitrary like the +grouping of the stars in constellations. The existence of groups would +have been of simple signification, if one group had been exclusively +fitted to inhabit the land, and another the water; one to feed on +flesh, another on vegetable matter, and so on; but the case is widely +different in nature; for it is notorious how commonly members of even +the same subgroup have different habits. In our second and fourth +chapters, on Variation and on Natural Selection, I have attempted to +show that it is the widely ranging, the much diffused and common, that +is the dominant species belonging to the larger genera, which vary +most. The varieties, or incipient species, thus produced ultimately +become converted, as I believe, into new and distinct species; and +these, on the principle of inheritance, tend to produce other new and +dominant +species. Consequently the groups which are now large, and which +generally include many dominant species, tend to go on increasing +indefinitely in size. I further attempted to show that from the varying +descendants of each species trying to occupy as many and as different +places as possible in the economy of nature, there is a constant +tendency in their characters to diverge. This conclusion was supported +by looking at the great diversity of the forms of life which, in any +small area, come into the closest competition, and by looking to +certain facts in naturalisation. + +I attempted also to show that there is a constant tendency in the forms +which are increasing in number and diverging in character, to supplant +and exterminate the less divergent, the less improved, and preceding +forms. I request the reader to turn to the diagram illustrating the +action, as formerly explained, of these several principles; and he will +see that the inevitable result is that the modified descendants +proceeding from one progenitor become broken up into groups subordinate +to groups. In the diagram each letter on the uppermost line may +represent a genus including several species; and all the genera on this +line form together one class, for all have descended from one ancient +but unseen parent, and, consequently, have inherited something in +common. But the three genera on the left hand have, on this same +principle, much in common, and form a sub-family, distinct from that +including the next two genera on the right hand, which diverged from a +common parent at the fifth stage of descent. These five genera have +also much, though less, in common; and they form a family distinct from +that including the three genera still further to the right hand, which +diverged at a still earlier period. And all these genera, descended +from (A), form an order distinct from the +genera descended from (I). So that we here have many species descended +from a single progenitor grouped into genera; and the genera are +included in, or subordinate to, sub-families, families, and orders, all +united into one class. Thus, the grand fact in natural history of the +subordination of group under group, which, from its familiarity, does +not always sufficiently strike us, is in my judgment fully explained. + +Naturalists try to arrange the species, genera, and families in each +class, on what is called the Natural System. But what is meant by this +system? Some authors look at it merely as a scheme for arranging +together those living objects which are most alike, and for separating +those which are most unlike; or as an artificial means for enunciating, +as briefly as possible, general propositions,—that is, by one sentence +to give the characters common, for instance, to all mammals, by another +those common to all carnivora, by another those common to the +dog-genus, and then by adding a single sentence, a full description is +given of each kind of dog. The ingenuity and utility of this system are +indisputable. But many naturalists think that something more is meant +by the Natural System; they believe that it reveals the plan of the +Creator; but unless it be specified whether order in time or space, or +what else is meant by the plan of the Creator, it seems to me that +nothing is thus added to our knowledge. Such expressions as that famous +one of Linnæus, and which we often meet with in a more or less +concealed form, that the characters do not make the genus, but that the +genus gives the characters, seem to imply that something more is +included in our classification, than mere resemblance. I believe that +something more is included; and that propinquity of descent,—the only +known cause of the similarity of organic beings,—is the bond, hidden as +it is by various degrees of modification, +which is partially revealed to us by our classifications. + +Let us now consider the rules followed in classification, and the +difficulties which are encountered on the view that classification +either gives some unknown plan of creation, or is simply a scheme for +enunciating general propositions and of placing together the forms most +like each other. It might have been thought (and was in ancient times +thought) that those parts of the structure which determined the habits +of life, and the general place of each being in the economy of nature, +would be of very high importance in classification. Nothing can be more +false. No one regards the external similarity of a mouse to a shrew, of +a dugong to a whale, of a whale to a fish, as of any importance. These +resemblances, though so intimately connected with the whole life of the +being, are ranked as merely “adaptive or analogical characters;” but to +the consideration of these resemblances we shall have to recur. It may +even be given as a general rule, that the less any part of the +organisation is concerned with special habits, the more important it +becomes for classification. As an instance: Owen, in speaking of the +dugong, says, “The generative organs being those which are most +remotely related to the habits and food of an animal, I have always +regarded as affording very clear indications of its true affinities. We +are least likely in the modifications of these organs to mistake a +merely adaptive for an essential character.” So with plants, how +remarkable it is that the organs of vegetation, on which their whole +life depends, are of little signification, excepting in the first main +divisions; whereas the organs of reproduction, with their product the +seed, are of paramount importance! + +We must not, therefore, in classifying, trust to resemblances in parts +of the organisation, however important +they may be for the welfare of the being in relation to the outer +world. Perhaps from this cause it has partly arisen, that almost all +naturalists lay the greatest stress on resemblances in organs of high +vital or physiological importance. No doubt this view of the +classificatory importance of organs which are important is generally, +but by no means always, true. But their importance for classification, +I believe, depends on their greater constancy throughout large groups +of species; and this constancy depends on such organs having generally +been subjected to less change in the adaptation of the species to their +conditions of life. That the mere physiological importance of an organ +does not determine its classificatory value, is almost shown by the one +fact, that in allied groups, in which the same organ, as we have every +reason to suppose, has nearly the same physiological value, its +classificatory value is widely different. No naturalist can have worked +at any group without being struck with this fact; and it has been most +fully acknowledged in the writings of almost every author. It will +suffice to quote the highest authority, Robert Brown, who in speaking +of certain organs in the Proteaceæ, says their generic importance, +“like that of all their parts, not only in this but, as I apprehend, in +every natural family, is very unequal, and in some cases seems to be +entirely lost.” Again in another work he says, the genera of the +Connaraceæ “differ in having one or more ovaria, in the existence or +absence of albumen, in the imbricate or valvular æstivation. Any one of +these characters singly is frequently of more than generic importance, +though here even when all taken together they appear insufficient to +separate Cnestis from Connarus.” To give an example amongst insects, in +one great division of the Hymenoptera, the antennæ, as Westwood has +remarked, are most constant in structure; +in another division they differ much, and the differences are of quite +subordinate value in classification; yet no one probably will say that +the antennæ in these two divisions of the same order are of unequal +physiological importance. Any number of instances could be given of the +varying importance for classification of the same important organ +within the same group of beings. + +Again, no one will say that rudimentary or atrophied organs are of high +physiological or vital importance; yet, undoubtedly, organs in this +condition are often of high value in classification. No one will +dispute that the rudimentary teeth in the upper jaws of young +ruminants, and certain rudimentary bones of the leg, are highly +serviceable in exhibiting the close affinity between Ruminants and +Pachyderms. Robert Brown has strongly insisted on the fact that the +rudimentary florets are of the highest importance in the classification +of the Grasses. + +Numerous instances could be given of characters derived from parts +which must be considered of very trifling physiological importance, but +which are universally admitted as highly serviceable in the definition +of whole groups. For instance, whether or not there is an open passage +from the nostrils to the mouth, the only character, according to Owen, +which absolutely distinguishes fishes and reptiles—the inflection of +the angle of the jaws in Marsupials—the manner in which the wings of +insects are folded—mere colour in certain Algæ—mere pubescence on parts +of the flower in grasses—the nature of the dermal covering, as hair or +feathers, in the Vertebrata. If the Ornithorhynchus had been covered +with feathers instead of hair, this external and trifling character +would, I think, have been considered by naturalists as important an aid +in determining the degree of affinity of this strange creature to +birds and reptiles, as an approach in structure in any one internal and +important organ. + +The importance, for classification, of trifling characters, mainly +depends on their being correlated with several other characters of more +or less importance. The value indeed of an aggregate of characters is +very evident in natural history. Hence, as has often been remarked, a +species may depart from its allies in several characters, both of high +physiological importance and of almost universal prevalence, and yet +leave us in no doubt where it should be ranked. Hence, also, it has +been found, that a classification founded on any single character, +however important that may be, has always failed; for no part of the +organisation is universally constant. The importance of an aggregate of +characters, even when none are important, alone explains, I think, that +saying of Linnæus, that the characters do not give the genus, but the +genus gives the characters; for this saying seems founded on an +appreciation of many trifling points of resemblance, too slight to be +defined. Certain plants, belonging to the Malpighiaceæ, bear perfect +and degraded flowers; in the latter, as A. de Jussieu has remarked, +“the greater number of the characters proper to the species, to the +genus, to the family, to the class, disappear, and thus laugh at our +classification.” But when Aspicarpa produced in France, during several +years, only degraded flowers, departing so wonderfully in a number of +the most important points of structure from the proper type of the +order, yet M. Richard sagaciously saw, as Jussieu observes, that this +genus should still be retained amongst the Malpighiaceæ. This case +seems to me well to illustrate the spirit with which our +classifications are sometimes necessarily founded. + +Practically when naturalists are at work, they do +not trouble themselves about the physiological value of the characters +which they use in defining a group, or in allocating any particular +species. If they find a character nearly uniform, and common to a great +number of forms, and not common to others, they use it as one of high +value; if common to some lesser number, they use it as of subordinate +value. This principle has been broadly confessed by some naturalists to +be the true one; and by none more clearly than by that excellent +botanist, Aug. St. Hilaire. If certain characters are always found +correlated with others, though no apparent bond of connexion can be +discovered between them, especial value is set on them. As in most +groups of animals, important organs, such as those for propelling the +blood, or for ærating it, or those for propagating the race, are found +nearly uniform, they are considered as highly serviceable in +classification; but in some groups of animals all these, the most +important vital organs, are found to offer characters of quite +subordinate value. + +We can see why characters derived from the embryo should be of equal +importance with those derived from the adult, for our classifications +of course include all ages of each species. But it is by no means +obvious, on the ordinary view, why the structure of the embryo should +be more important for this purpose than that of the adult, which alone +plays its full part in the economy of nature. Yet it has been strongly +urged by those great naturalists, Milne Edwards and Agassiz, that +embryonic characters are the most important of any in the +classification of animals; and this doctrine has very generally been +admitted as true. The same fact holds good with flowering plants, of +which the two main divisions have been founded on characters derived +from the embryo,—on the number and position of the embryonic +leaves or cotyledons, and on the mode of development of the plumule and +radicle. In our discussion on embryology, we shall see why such +characters are so valuable, on the view of classification tacitly +including the idea of descent. + +Our classifications are often plainly influenced by chains of +affinities. Nothing can be easier than to define a number of characters +common to all birds; but in the case of crustaceans, such definition +has hitherto been found impossible. There are crustaceans at the +opposite ends of the series, which have hardly a character in common; +yet the species at both ends, from being plainly allied to others, and +these to others, and so onwards, can be recognised as unequivocally +belonging to this, and to no other class of the Articulata. + +Geographical distribution has often been used, though perhaps not quite +logically, in classification, more especially in very large groups of +closely allied forms. Temminck insists on the utility or even necessity +of this practice in certain groups of birds; and it has been followed +by several entomologists and botanists. + +Finally, with respect to the comparative value of the various groups of +species, such as orders, sub-orders, families, sub-families, and +genera, they seem to be, at least at present, almost arbitrary. Several +of the best botanists, such as Mr. Bentham and others, have strongly +insisted on their arbitrary value. Instances could be given amongst +plants and insects, of a group of forms, first ranked by practised +naturalists as only a genus, and then raised to the rank of a +sub-family or family; and this has been done, not because further +research has detected important structural differences, at first +overlooked, but because numerous allied species, with slightly +different grades of difference, have been subsequently discovered. + +All the foregoing rules and aids and difficulties in classification are +explained, if I do not greatly deceive myself, on the view that the +natural system is founded on descent with modification; that the +characters which naturalists consider as showing true affinity between +any two or more species, are those which have been inherited from a +common parent, and, in so far, all true classification is genealogical; +that community of descent is the hidden bond which naturalists have +been unconsciously seeking, and not some unknown plan of creation, or +the enunciation of general propositions, and the mere putting together +and separating objects more or less alike. + +But I must explain my meaning more fully. I believe that the +_arrangement_ of the groups within each class, in due subordination and +relation to the other groups, must be strictly genealogical in order to +be natural; but that the _amount_ of difference in the several branches +or groups, though allied in the same degree in blood to their common +progenitor, may differ greatly, being due to the different degrees of +modification which they have undergone; and this is expressed by the +forms being ranked under different genera, families, sections, or +orders. The reader will best understand what is meant, if he will take +the trouble of referring to the diagram in the fourth chapter. We will +suppose the letters A to L to represent allied genera, which lived +during the Silurian epoch, and these have descended from a species +which existed at an unknown anterior period. Species of three of these +genera (A, F, and I) have transmitted modified descendants to the +present day, represented by the fifteen genera (_a_14 to _z_14) on the +uppermost horizontal line. Now all these modified descendants from a +single species, are represented as related in blood or descent to the +same +degree; they may metaphorically be called cousins to the same millionth +degree; yet they differ widely and in different degrees from each +other. The forms descended from A, now broken up into two or three +families, constitute a distinct order from those descended from I, also +broken up into two families. Nor can the existing species, descended +from A, be ranked in the same genus with the parent A; or those from I, +with the parent I. But the existing genus F14 may be supposed to have +been but slightly modified; and it will then rank with the parent-genus +F; just as some few still living organic beings belong to Silurian +genera. So that the amount or value of the differences between organic +beings all related to each other in the same degree in blood, has come +to be widely different. Nevertheless their genealogical _arrangement_ +remains strictly true, not only at the present time, but at each +successive period of descent. All the modified descendants from A will +have inherited something in common from their common parent, as will +all the descendants from I; so will it be with each subordinate branch +of descendants, at each successive period. If, however, we choose to +suppose that any of the descendants of A or of I have been so much +modified as to have more or less completely lost traces of their +parentage, in this case, their places in a natural classification will +have been more or less completely lost,—as sometimes seems to have +occurred with existing organisms. All the descendants of the genus F, +along its whole line of descent, are supposed to have been but little +modified, and they yet form a single genus. But this genus, though much +isolated, will still occupy its proper intermediate position; for F +originally was intermediate in character between A and I, and the +several genera descended from these two genera will +have inherited to a certain extent their characters. This natural +arrangement is shown, as far as is possible on paper, in the diagram, +but in much too simple a manner. If a branching diagram had not been +used, and only the names of the groups had been written in a linear +series, it would have been still less possible to have given a natural +arrangement; and it is notoriously not possible to represent in a +series, on a flat surface, the affinities which we discover in nature +amongst the beings of the same group. Thus, on the view which I hold, +the natural system is genealogical in its arrangement, like a pedigree; +but the degrees of modification which the different groups have +undergone, have to be expressed by ranking them under different +so-called genera, sub-families, families, sections, orders, and +classes. + +It may be worth while to illustrate this view of classification, by +taking the case of languages. If we possessed a perfect pedigree of +mankind, a genealogical arrangement of the races of man would afford +the best classification of the various languages now spoken throughout +the world; and if all extinct languages, and all intermediate and +slowly changing dialects, had to be included, such an arrangement +would, I think, be the only possible one. Yet it might be that some +very ancient language had altered little, and had given rise to few new +languages, whilst others (owing to the spreading and subsequent +isolation and states of civilisation of the several races, descended +from a common race) had altered much, and had given rise to many new +languages and dialects. The various degrees of difference in the +languages from the same stock, would have to be expressed by groups +subordinate to groups; but the proper or even only possible arrangement +would still be genealogical; and this would be strictly natural, as +it would connect together all languages, extinct and modern, by the +closest affinities, and would give the filiation and origin of each +tongue. + +In confirmation of this view, let us glance at the classification of +varieties, which are believed or known to have descended from one +species. These are grouped under species, with sub-varieties under +varieties; and with our domestic productions, several other grades of +difference are requisite, as we have seen with pigeons. The origin of +the existence of groups subordinate to groups, is the same with +varieties as with species, namely, closeness of descent with various +degrees of modification. Nearly the same rules are followed in +classifying varieties, as with species. Authors have insisted on the +necessity of classing varieties on a natural instead of an artificial +system; we are cautioned, for instance, not to class two varieties of +the pine-apple together, merely because their fruit, though the most +important part, happens to be nearly identical; no one puts the swedish +and common turnips together, though the esculent and thickened stems +are so similar. Whatever part is found to be most constant, is used in +classing varieties: thus the great agriculturist Marshall says the +horns are very useful for this purpose with cattle, because they are +less variable than the shape or colour of the body, etc.; whereas with +sheep the horns are much less serviceable, because less constant. In +classing varieties, I apprehend if we had a real pedigree, a +genealogical classification would be universally preferred; and it has +been attempted by some authors. For we might feel sure, whether there +had been more or less modification, the principle of inheritance would +keep the forms together which were allied in the greatest number of +points. In tumbler pigeons, though some sub-varieties differ from the +others +in the important character of having a longer beak, yet all are kept +together from having the common habit of tumbling; but the short-faced +breed has nearly or quite lost this habit; nevertheless, without any +reasoning or thinking on the subject, these tumblers are kept in the +same group, because allied in blood and alike in some other respects. +If it could be proved that the Hottentot had descended from the Negro, +I think he would be classed under the Negro group, however much he +might differ in colour and other important characters from negroes. + +With species in a state of nature, every naturalist has in fact brought +descent into his classification; for he includes in his lowest grade, +or that of a species, the two sexes; and how enormously these sometimes +differ in the most important characters, is known to every naturalist: +scarcely a single fact can be predicated in common of the males and +hermaphrodites of certain cirripedes, when adult, and yet no one dreams +of separating them. The naturalist includes as one species the several +larval stages of the same individual, however much they may differ from +each other and from the adult; as he likewise includes the so-called +alternate generations of Steenstrup, which can only in a technical +sense be considered as the same individual. He includes monsters; he +includes varieties, not solely because they closely resemble the +parent-form, but because they are descended from it. He who believes +that the cowslip is descended from the primrose, or conversely, ranks +them together as a single species, and gives a single definition. As +soon as three Orchidean forms (Monochanthus, Myanthus, and Catasetum), +which had previously been ranked as three distinct genera, were known +to be sometimes produced on the same spike, they were immediately +included as a single species. +But it may be asked, what ought we to do, if it could be proved that +one species of kangaroo had been produced, by a long course of +modification, from a bear? Ought we to rank this one species with +bears, and what should we do with the other species? The supposition is +of course preposterous; and I might answer by the _argumentum ad +hominem_, and ask what should be done if a perfect kangaroo were seen +to come out of the womb of a bear? According to all analogy, it would +be ranked with bears; but then assuredly all the other species of the +kangaroo family would have to be classed under the bear genus. The +whole case is preposterous; for where there has been close descent in +common, there will certainly be close resemblance or affinity. + +As descent has universally been used in classing together the +individuals of the same species, though the males and females and larvæ +are sometimes extremely different; and as it has been used in classing +varieties which have undergone a certain, and sometimes a considerable +amount of modification, may not this same element of descent have been +unconsciously used in grouping species under genera, and genera under +higher groups, though in these cases the modification has been greater +in degree, and has taken a longer time to complete? I believe it has +thus been unconsciously used; and only thus can I understand the +several rules and guides which have been followed by our best +systematists. We have no written pedigrees; we have to make out +community of descent by resemblances of any kind. Therefore we choose +those characters which, as far as we can judge, are the least likely to +have been modified in relation to the conditions of life to which each +species has been recently exposed. Rudimentary structures on this view +are as good as, or even sometimes better than, other parts of the +organisation. We +care not how trifling a character may be—let it be the mere inflection +of the angle of the jaw, the manner in which an insect’s wing is +folded, whether the skin be covered by hair or feathers—if it prevail +throughout many and different species, especially those having very +different habits of life, it assumes high value; for we can account for +its presence in so many forms with such different habits, only by its +inheritance from a common parent. We may err in this respect in regard +to single points of structure, but when several characters, let them be +ever so trifling, occur together throughout a large group of beings +having different habits, we may feel almost sure, on the theory of +descent, that these characters have been inherited from a common +ancestor. And we know that such correlated or aggregated characters +have especial value in classification. + +We can understand why a species or a group of species may depart, in +several of its most important characteristics, from its allies, and yet +be safely classed with them. This may be safely done, and is often +done, as long as a sufficient number of characters, let them be ever so +unimportant, betrays the hidden bond of community of descent. Let two +forms have not a single character in common, yet if these extreme forms +are connected together by a chain of intermediate groups, we may at +once infer their community of descent, and we put them all into the +same class. As we find organs of high physiological importance—those +which serve to preserve life under the most diverse conditions of +existence—are generally the most constant, we attach especial value to +them; but if these same organs, in another group or section of a group, +are found to differ much, we at once value them less in our +classification. We shall hereafter, I think, clearly see why +embryological characters are of such high classificatory importance. +Geographical distribution may sometimes be brought usefully into play +in classing large and widely-distributed genera, because all the +species of the same genus, inhabiting any distinct and isolated region, +have in all probability descended from the same parents. + +We can understand, on these views, the very important distinction +between real affinities and analogical or adaptive resemblances. +Lamarck first called attention to this distinction, and he has been +ably followed by Macleay and others. The resemblance, in the shape of +the body and in the fin-like anterior limbs, between the dugong, which +is a pachydermatous animal, and the whale, and between both these +mammals and fishes, is analogical. Amongst insects there are +innumerable instances: thus Linnæus, misled by external appearances, +actually classed an homopterous insect as a moth. We see something of +the same kind even in our domestic varieties, as in the thickened stems +of the common and swedish turnip. The resemblance of the greyhound and +racehorse is hardly more fanciful than the analogies which have been +drawn by some authors between very distinct animals. On my view of +characters being of real importance for classification, only in so far +as they reveal descent, we can clearly understand why analogical or +adaptive character, although of the utmost importance to the welfare of +the being, are almost valueless to the systematist. For animals, +belonging to two most distinct lines of descent, may readily become +adapted to similar conditions, and thus assume a close external +resemblance; but such resemblances will not reveal—will rather tend to +conceal their blood-relationship to their proper lines of descent. We +can also understand the apparent paradox, that the very same characters +are analogical when one class or order is compared with another, but +give true affinities when the members of +the same class or order are compared one with another: thus the shape +of the body and fin-like limbs are only analogical when whales are +compared with fishes, being adaptations in both classes for swimming +through the water; but the shape of the body and fin-like limbs serve +as characters exhibiting true affinity between the several members of +the whale family; for these cetaceans agree in so many characters, +great and small, that we cannot doubt that they have inherited their +general shape of body and structure of limbs from a common ancestor. So +it is with fishes. + +As members of distinct classes have often been adapted by successive +slight modifications to live under nearly similar circumstances,—to +inhabit for instance the three elements of land, air, and water,—we can +perhaps understand how it is that a numerical parallelism has sometimes +been observed between the sub-groups in distinct classes. A naturalist, +struck by a parallelism of this nature in any one class, by arbitrarily +raising or sinking the value of the groups in other classes (and all +our experience shows that this valuation has hitherto been arbitrary), +could easily extend the parallelism over a wide range; and thus the +septenary, quinary, quaternary, and ternary classifications have +probably arisen. + +As the modified descendants of dominant species, belonging to the +larger genera, tend to inherit the advantages, which made the groups to +which they belong large and their parents dominant, they are almost +sure to spread widely, and to seize on more and more places in the +economy of nature. The larger and more dominant groups thus tend to go +on increasing in size; and they consequently supplant many smaller and +feebler groups. Thus we can account for the fact that all organisms, +recent and extinct, are included under a few great +orders, under still fewer classes, and all in one great natural system. +As showing how few the higher groups are in number, and how widely +spread they are throughout the world, the fact is striking, that the +discovery of Australia has not added a single insect belonging to a new +order; and that in the vegetable kingdom, as I learn from Dr. Hooker, +it has added only two or three orders of small size. + +In the chapter on geological succession I attempted to show, on the +principle of each group having generally diverged much in character +during the long-continued process of modification, how it is that the +more ancient forms of life often present characters in some slight +degree intermediate between existing groups. A few old and intermediate +parent-forms having occasionally transmitted to the present day +descendants but little modified, will give to us our so-called osculant +or aberrant groups. The more aberrant any form is, the greater must be +the number of connecting forms which on my theory have been +exterminated and utterly lost. And we have some evidence of aberrant +forms having suffered severely from extinction, for they are generally +represented by extremely few species; and such species as do occur are +generally very distinct from each other, which again implies +extinction. The genera Ornithorhynchus and Lepidosiren, for example, +would not have been less aberrant had each been represented by a dozen +species instead of by a single one; but such richness in species, as I +find after some investigation, does not commonly fall to the lot of +aberrant genera. We can, I think, account for this fact only by looking +at aberrant forms as failing groups conquered by more successful +competitors, with a few members preserved by some unusual coincidence +of favourable circumstances. + +Mr. Waterhouse has remarked that, when a member +belonging to one group of animals exhibits an affinity to a quite +distinct group, this affinity in most cases is general and not special: +thus, according to Mr. Waterhouse, of all Rodents, the bizcacha is most +nearly related to Marsupials; but in the points in which it approaches +this order, its relations are general, and not to any one marsupial +species more than to another. As the points of affinity of the bizcacha +to Marsupials are believed to be real and not merely adaptive, they are +due on my theory to inheritance in common. Therefore we must suppose +either that all Rodents, including the bizcacha, branched off from some +very ancient Marsupial, which will have had a character in some degree +intermediate with respect to all existing Marsupials; or that both +Rodents and Marsupials branched off from a common progenitor, and that +both groups have since undergone much modification in divergent +directions. On either view we may suppose that the bizcacha has +retained, by inheritance, more of the character of its ancient +progenitor than have other Rodents; and therefore it will not be +specially related to any one existing Marsupial, but indirectly to all +or nearly all Marsupials, from having partially retained the character +of their common progenitor, or of an early member of the group. On the +other hand, of all Marsupials, as Mr. Waterhouse has remarked, the +phascolomys resembles most nearly, not any one species, but the general +order of Rodents. In this case, however, it may be strongly suspected +that the resemblance is only analogical, owing to the phascolomys +having become adapted to habits like those of a Rodent. The elder De +Candolle has made nearly similar observations on the general nature of +the affinities of distinct orders of plants. + +On the principle of the multiplication and gradual divergence in +character of the species descended from +a common parent, together with their retention by inheritance of some +characters in common, we can understand the excessively complex and +radiating affinities by which all the members of the same family or +higher group are connected together. For the common parent of a whole +family of species, now broken up by extinction into distinct groups and +sub-groups, will have transmitted some of its characters, modified in +various ways and degrees, to all; and the several species will +consequently be related to each other by circuitous lines of affinity +of various lengths (as may be seen in the diagram so often referred +to), mounting up through many predecessors. As it is difficult to show +the blood-relationship between the numerous kindred of any ancient and +noble family, even by the aid of a genealogical tree, and almost +impossible to do this without this aid, we can understand the +extraordinary difficulty which naturalists have experienced in +describing, without the aid of a diagram, the various affinities which +they perceive between the many living and extinct members of the same +great natural class. + +Extinction, as we have seen in the fourth chapter, has played an +important part in defining and widening the intervals between the +several groups in each class. We may thus account even for the +distinctness of whole classes from each other—for instance, of birds +from all other vertebrate animals—by the belief that many ancient forms +of life have been utterly lost, through which the early progenitors of +birds were formerly connected with the early progenitors of the other +vertebrate classes. There has been less entire extinction of the forms +of life which once connected fishes with batrachians. There has been +still less in some other classes, as in that of the Crustacea, for here +the most wonderfully diverse forms are still tied +together by a long, but broken, chain of affinities. Extinction has +only separated groups: it has by no means made them; for if every form +which has ever lived on this earth were suddenly to reappear, though it +would be quite impossible to give definitions by which each group could +be distinguished from other groups, as all would blend together by +steps as fine as those between the finest existing varieties, +nevertheless a natural classification, or at least a natural +arrangement, would be possible. We shall see this by turning to the +diagram: the letters, A to L, may represent eleven Silurian genera, +some of which have produced large groups of modified descendants. Every +intermediate link between these eleven genera and their primordial +parent, and every intermediate link in each branch and sub-branch of +their descendants, may be supposed to be still alive; and the links to +be as fine as those between the finest varieties. In this case it would +be quite impossible to give any definition by which the several members +of the several groups could be distinguished from their more immediate +parents; or these parents from their ancient and unknown progenitor. +Yet the natural arrangement in the diagram would still hold good; and, +on the principle of inheritance, all the forms descended from A, or +from I, would have something in common. In a tree we can specify this +or that branch, though at the actual fork the two unite and blend +together. We could not, as I have said, define the several groups; but +we could pick out types, or forms, representing most of the characters +of each group, whether large or small, and thus give a general idea of +the value of the differences between them. This is what we should be +driven to, if we were ever to succeed in collecting all the forms in +any class which have lived throughout all time and space. We shall +certainly never succeed in making +so perfect a collection: nevertheless, in certain classes, we are +tending in this direction; and Milne Edwards has lately insisted, in an +able paper, on the high importance of looking to types, whether or not +we can separate and define the groups to which such types belong. + +Finally, we have seen that natural selection, which results from the +struggle for existence, and which almost inevitably induces extinction +and divergence of character in the many descendants from one dominant +parent-species, explains that great and universal feature in the +affinities of all organic beings, namely, their subordination in group +under group. We use the element of descent in classing the individuals +of both sexes and of all ages, although having few characters in +common, under one species; we use descent in classing acknowledged +varieties, however different they may be from their parent; and I +believe this element of descent is the hidden bond of connexion which +naturalists have sought under the term of the Natural System. On this +idea of the natural system being, in so far as it has been perfected, +genealogical in its arrangement, with the grades of difference between +the descendants from a common parent, expressed by the terms genera, +families, orders, etc., we can understand the rules which we are +compelled to follow in our classification. We can understand why we +value certain resemblances far more than others; why we are permitted +to use rudimentary and useless organs, or others of trifling +physiological importance; why, in comparing one group with a distinct +group, we summarily reject analogical or adaptive characters, and yet +use these same characters within the limits of the same group. We can +clearly see how it is that all living and extinct forms can be grouped +together in one great system; and how the several members of each class +are connected together by the most complex and radiating +lines of affinities. We shall never, probably, disentangle the +inextricable web of affinities between the members of any one class; +but when we have a distinct object in view, and do not look to some +unknown plan of creation, we may hope to make sure but slow progress. + +_Morphology_.—We have seen that the members of the same class, +independently of their habits of life, resemble each other in the +general plan of their organisation. This resemblance is often expressed +by the term “unity of type;” or by saying that the several parts and +organs in the different species of the class are homologous. The whole +subject is included under the general name of Morphology. This is the +most interesting department of natural history, and may be said to be +its very soul. What can be more curious than that the hand of a man, +formed for grasping, that of a mole for digging, the leg of the horse, +the paddle of the porpoise, and the wing of the bat, should all be +constructed on the same pattern, and should include the same bones, in +the same relative positions? Geoffroy St. Hilaire has insisted strongly +on the high importance of relative connexion in homologous organs: the +parts may change to almost any extent in form and size, and yet they +always remain connected together in the same order. We never find, for +instance, the bones of the arm and forearm, or of the thigh and leg, +transposed. Hence the same names can be given to the homologous bones +in widely different animals. We see the same great law in the +construction of the mouths of insects: what can be more different than +the immensely long spiral proboscis of a sphinx-moth, the curious +folded one of a bee or bug, and the great jaws of a beetle?—yet all +these organs, serving for such different +purposes, are formed by infinitely numerous modifications of an upper +lip, mandibles, and two pairs of maxillæ. Analogous laws govern the +construction of the mouths and limbs of crustaceans. So it is with the +flowers of plants. + +Nothing can be more hopeless than to attempt to explain this similarity +of pattern in members of the same class, by utility or by the doctrine +of final causes. The hopelessness of the attempt has been expressly +admitted by Owen in his most interesting work on the ‘Nature of Limbs.’ +On the ordinary view of the independent creation of each being, we can +only say that so it is;—that it has so pleased the Creator to construct +each animal and plant. + +The explanation is manifest on the theory of the natural selection of +successive slight modifications,—each modification being profitable in +some way to the modified form, but often affecting by correlation of +growth other parts of the organisation. In changes of this nature, +there will be little or no tendency to modify the original pattern, or +to transpose parts. The bones of a limb might be shortened and widened +to any extent, and become gradually enveloped in thick membrane, so as +to serve as a fin; or a webbed foot might have all its bones, or +certain bones, lengthened to any extent, and the membrane connecting +them increased to any extent, so as to serve as a wing: yet in all this +great amount of modification there will be no tendency to alter the +framework of bones or the relative connexion of the several parts. If +we suppose that the ancient progenitor, the archetype as it may be +called, of all mammals, had its limbs constructed on the existing +general pattern, for whatever purpose they served, we can at once +perceive the plain signification of the homologous construction of the +limbs throughout the whole class. So with the mouths of insects, we +have only to +suppose that their common progenitor had an upper lip, mandibles, and +two pair of maxillæ, these parts being perhaps very simple in form; and +then natural selection will account for the infinite diversity in +structure and function of the mouths of insects. Nevertheless, it is +conceivable that the general pattern of an organ might become so much +obscured as to be finally lost, by the atrophy and ultimately by the +complete abortion of certain parts, by the soldering together of other +parts, and by the doubling or multiplication of others,—variations +which we know to be within the limits of possibility. In the paddles of +the extinct gigantic sea-lizards, and in the mouths of certain +suctorial crustaceans, the general pattern seems to have been thus to a +certain extent obscured. + +There is another and equally curious branch of the present subject; +namely, the comparison not of the same part in different members of a +class, but of the different parts or organs in the same individual. +Most physiologists believe that the bones of the skull are homologous +with—that is correspond in number and in relative connexion with—the +elemental parts of a certain number of vertebræ. The anterior and +posterior limbs in each member of the vertebrate and articulate classes +are plainly homologous. We see the same law in comparing the +wonderfully complex jaws and legs in crustaceans. It is familiar to +almost every one, that in a flower the relative position of the sepals, +petals, stamens, and pistils, as well as their intimate structure, are +intelligible on the view that they consist of metamorphosed leaves, +arranged in a spire. In monstrous plants, we often get direct evidence +of the possibility of one organ being transformed into another; and we +can actually see in embryonic crustaceans and in many other animals, +and in flowers, that organs, which when mature +become extremely different, are at an early stage of growth exactly +alike. + +How inexplicable are these facts on the ordinary view of creation! Why +should the brain be enclosed in a box composed of such numerous and +such extraordinarily shaped pieces of bone? As Owen has remarked, the +benefit derived from the yielding of the separate pieces in the act of +parturition of mammals, will by no means explain the same construction +in the skulls of birds. Why should similar bones have been created in +the formation of the wing and leg of a bat, used as they are for such +totally different purposes? Why should one crustacean, which has an +extremely complex mouth formed of many parts, consequently always have +fewer legs; or conversely, those with many legs have simpler mouths? +Why should the sepals, petals, stamens, and pistils in any individual +flower, though fitted for such widely different purposes, be all +constructed on the same pattern? + +On the theory of natural selection, we can satisfactorily answer these +questions. In the vertebrata, we see a series of internal vertebræ +bearing certain processes and appendages; in the articulata, we see the +body divided into a series of segments, bearing external appendages; +and in flowering plants, we see a series of successive spiral whorls of +leaves. An indefinite repetition of the same part or organ is the +common characteristic (as Owen has observed) of all low or +little-modified forms; therefore we may readily believe that the +unknown progenitor of the vertebrata possessed many vertebræ; the +unknown progenitor of the articulata, many segments; and the unknown +progenitor of flowering plants, many spiral whorls of leaves. We have +formerly seen that parts many times repeated are eminently liable to +vary in number and structure; consequently it is quite probable that +natural selection, during a long-continued course of modification, +should have seized on a certain number of the primordially similar +elements, many times repeated, and have adapted them to the most +diverse purposes. And as the whole amount of modification will have +been effected by slight successive steps, we need not wonder at +discovering in such parts or organs, a certain degree of fundamental +resemblance, retained by the strong principle of inheritance. + +In the great class of molluscs, though we can homologise the parts of +one species with those of another and distinct species, we can indicate +but few serial homologies; that is, we are seldom enabled to say that +one part or organ is homologous with another in the same individual. +And we can understand this fact; for in molluscs, even in the lowest +members of the class, we do not find nearly so much indefinite +repetition of any one part, as we find in the other great classes of +the animal and vegetable kingdoms. + +Naturalists frequently speak of the skull as formed of metamorphosed +vertebræ: the jaws of crabs as metamorphosed legs; the stamens and +pistils of flowers as metamorphosed leaves; but it would in these cases +probably be more correct, as Professor Huxley has remarked, to speak of +both skull and vertebræ, both jaws and legs, etc.,—as having been +metamorphosed, not one from the other, but from some common element. +Naturalists, however, use such language only in a metaphorical sense: +they are far from meaning that during a long course of descent, +primordial organs of any kind—vertebræ in the one case and legs in the +other—have actually been modified into skulls or jaws. Yet so strong is +the appearance of a modification of this nature having occurred, that +naturalists can hardly avoid employing language having this plain +signification. On my view +these terms may be used literally; and the wonderful fact of the jaws, +for instance, of a crab retaining numerous characters, which they would +probably have retained through inheritance, if they had really been +metamorphosed during a long course of descent from true legs, or from +some simple appendage, is explained. + +_Embryology_.—It has already been casually remarked that certain organs +in the individual, which when mature become widely different and serve +for different purposes, are in the embryo exactly alike. The embryos, +also, of distinct animals within the same class are often strikingly +similar: a better proof of this cannot be given, than a circumstance +mentioned by Agassiz, namely, that having forgotten to ticket the +embryo of some vertebrate animal, he cannot now tell whether it be that +of a mammal, bird, or reptile. The vermiform larvæ of moths, flies, +beetles, etc., resemble each other much more closely than do the mature +insects; but in the case of larvæ, the embryos are active, and have +been adapted for special lines of life. A trace of the law of embryonic +resemblance, sometimes lasts till a rather late age: thus birds of the +same genus, and of closely allied genera, often resemble each other in +their first and second plumage; as we see in the spotted feathers in +the thrush group. In the cat tribe, most of the species are striped or +spotted in lines; and stripes can be plainly distinguished in the whelp +of the lion. We occasionally though rarely see something of this kind +in plants: thus the embryonic leaves of the ulex or furze, and the +first leaves of the phyllodineous acaceas, are pinnate or divided like +the ordinary leaves of the leguminosæ. + +The points of structure, in which the embryos of widely different +animals of the same class resemble each other, often have no direct +relation to their conditions +of existence. We cannot, for instance, suppose that in the embryos of +the vertebrata the peculiar loop-like course of the arteries near the +branchial slits are related to similar conditions,—in the young mammal +which is nourished in the womb of its mother, in the egg of the bird +which is hatched in a nest, and in the spawn of a frog under water. We +have no more reason to believe in such a relation, than we have to +believe that the same bones in the hand of a man, wing of a bat, and +fin of a porpoise, are related to similar conditions of life. No one +will suppose that the stripes on the whelp of a lion, or the spots on +the young blackbird, are of any use to these animals, or are related to +the conditions to which they are exposed. + +The case, however, is different when an animal during any part of its +embryonic career is active, and has to provide for itself. The period +of activity may come on earlier or later in life; but whenever it comes +on, the adaptation of the larva to its conditions of life is just as +perfect and as beautiful as in the adult animal. From such special +adaptations, the similarity of the larvæ or active embryos of allied +animals is sometimes much obscured; and cases could be given of the +larvæ of two species, or of two groups of species, differing quite as +much, or even more, from each other than do their adult parents. In +most cases, however, the larvæ, though active, still obey more or less +closely the law of common embryonic resemblance. Cirripedes afford a +good instance of this: even the illustrious Cuvier did not perceive +that a barnacle was, as it certainly is, a crustacean; but a glance at +the larva shows this to be the case in an unmistakeable manner. So +again the two main divisions of cirripedes, the pedunculated and +sessile, which differ widely in external appearance, have larvæ in all +their several stages barely distinguishable. + +The embryo in the course of development generally rises in +organisation: I use this expression, though I am aware that it is +hardly possible to define clearly what is meant by the organisation +being higher or lower. But no one probably will dispute that the +butterfly is higher than the caterpillar. In some cases, however, the +mature animal is generally considered as lower in the scale than the +larva, as with certain parasitic crustaceans. To refer once again to +cirripedes: the larvæ in the first stage have three pairs of legs, a +very simple single eye, and a probosciformed mouth, with which they +feed largely, for they increase much in size. In the second stage, +answering to the chrysalis stage of butterflies, they have six pairs of +beautifully constructed natatory legs, a pair of magnificent compound +eyes, and extremely complex antennæ; but they have a closed and +imperfect mouth, and cannot feed: their function at this stage is, to +search by their well-developed organs of sense, and to reach by their +active powers of swimming, a proper place on which to become attached +and to undergo their final metamorphosis. When this is completed they +are fixed for life: their legs are now converted into prehensile +organs; they again obtain a well-constructed mouth; but they have no +antennæ, and their two eyes are now reconverted into a minute, single, +and very simple eye-spot. In this last and complete state, cirripedes +may be considered as either more highly or more lowly organised than +they were in the larval condition. But in some genera the larvæ become +developed either into hermaphrodites having the ordinary structure, or +into what I have called complemental males: and in the latter, the +development has assuredly been retrograde; for the male is a mere sack, +which lives for a short time, and is destitute of mouth, stomach, or +other organ of importance, excepting for reproduction. + +The present book is intended, as far as possible, to give an exact +insight into the theory of Relativity to those readers who, from a +general scientific and philosophical point of view, are interested in +the theory, but who are not conversant with the mathematical apparatus +of theoretical physics. The work presumes a standard of education +corresponding to that of a university matriculation examination, and, +despite the shortness of the book, a fair amount of patience and force +of will on the part of the reader. The author has spared himself no +pains in his endeavour to present the main ideas in the simplest and +most intelligible form, and on the whole, in the sequence and +connection in which they actually originated. In the interest of +clearness, it appeared to me inevitable that I should repeat myself +frequently, without paying the slightest attention to the elegance of +the presentation. I adhered scrupulously to the precept of that +brilliant theoretical physicist L. Boltzmann, according to whom matters +of elegance ought to be left to the tailor and to the cobbler. I make +no pretence of having withheld from the reader difficulties which are +inherent to the subject. On the other hand, I have purposely treated +the empirical physical foundations of the theory in a “step-motherly” +fashion, so that readers unfamiliar with physics may not feel like the +wanderer who was unable to see the forest for the trees. May the book +bring some one a few happy hours of suggestive thought! + +December, 1916 + +In your schooldays most of you who read this book made acquaintance +with the noble building of Euclid’s geometry, and you remember—perhaps +with more respect than love—the magnificent structure, on the lofty +staircase of which you were chased about for uncounted hours by +conscientious teachers. By reason of our past experience, you would +certainly regard everyone with disdain who should pronounce even the +most out-of-the-way proposition of this science to be untrue. But +perhaps this feeling of proud certainty would leave you immediately if +some one were to ask you: “What, then, do you mean by the assertion +that these propositions are true?” Let us proceed to give this question +a little consideration. + +Geometry sets out from certain conceptions such as “plane,” “point,” +and “straight line,” with which we are able to associate more or less +definite ideas, and from certain simple propositions (axioms) which, in +virtue of these ideas, we are inclined to accept as “true.” Then, on +the basis of a logical process, the justification of which we feel +ourselves compelled to admit, all remaining propositions are shown to +follow from those axioms, _i.e._ they are proven. A proposition is then +correct (“true”) when it has been derived in the recognised manner from +the axioms. The question of “truth” of the individual geometrical +propositions is thus reduced to one of the “truth” of the axioms. Now +it has long been known that the last question is not only unanswerable +by the methods of geometry, but that it is in itself entirely without +meaning. We cannot ask whether it is true that only one straight line +goes through two points. We can only say that Euclidean geometry deals +with things called “straight lines,” to each of which is ascribed the +property of being uniquely determined by two points situated on it. The +concept “true” does not tally with the assertions of pure geometry, +because by the word “true” we are eventually in the habit of +designating always the correspondence with a “real” object; geometry, +however, is not concerned with the relation of the ideas involved in it +to objects of experience, but only with the logical connection of these +ideas among themselves. + +It is not difficult to understand why, in spite of this, we feel +constrained to call the propositions of geometry “true.” Geometrical +ideas correspond to more or less exact objects in nature, and these +last are undoubtedly the exclusive cause of the genesis of those ideas. +Geometry ought to refrain from such a course, in order to give to its +structure the largest possible logical unity. The practice, for +example, of seeing in a “distance” two marked positions on a +practically rigid body is something which is lodged deeply in our habit +of thought. We are accustomed further to regard three points as being +situated on a straight line, if their apparent positions can be made to +coincide for observation with one eye, under suitable choice of our +place of observation. + +If, in pursuance of our habit of thought, we now supplement the +propositions of Euclidean geometry by the single proposition that two +points on a practically rigid body always correspond to the same +distance (line-interval), independently of any changes in position to +which we may subject the body, the propositions of Euclidean geometry +then resolve themselves into propositions on the possible relative +position of practically rigid bodies.[1] Geometry which has been +supplemented in this way is then to be treated as a branch of physics. +We can now legitimately ask as to the “truth” of geometrical +propositions interpreted in this way, since we are justified in asking +whether these propositions are satisfied for those real things we have +associated with the geometrical ideas. In less exact terms we can +express this by saying that by the “truth” of a geometrical proposition +in this sense we understand its validity for a construction with rule +and compasses. + + [1] It follows that a natural object is associated also with a + straight line. Three points _A, B_ and _C_ on a rigid body thus lie in + a straight line when the points _A_ and _C_ being given, _B_ is chosen + such that the sum of the distances _AB_ and _BC_ is as short as + possible. This incomplete suggestion will suffice for the present + purpose. + +Of course the conviction of the “truth” of geometrical propositions in +this sense is founded exclusively on rather incomplete experience. For +the present we shall assume the “truth” of the geometrical +propositions, then at a later stage (in the general theory of +relativity) we shall see that this “truth” is limited, and we shall +consider the extent of its limitation. + +On the basis of the physical interpretation of distance which has been +indicated, we are also in a position to establish the distance between +two points on a rigid body by means of measurements. For this purpose +we require a “distance” (rod _S_) which is to be used once and for all, +and which we employ as a standard measure. If, now, _A_ and _B_ are two +points on a rigid body, we can construct the line joining them +according to the rules of geometry; then, starting from _A_, we can +mark off the distance _S_ time after time until we reach _B_. The +number of these operations required is the numerical measure of the +distance _AB_. This is the basis of all measurement of length.[2] + + [2] Here we have assumed that there is nothing left over _i.e._ that + the measurement gives a whole number. This difficulty is got over by + the use of divided measuring-rods, the introduction of which does not + demand any fundamentally new method. + +Every description of the scene of an event or of the position of an +object in space is based on the specification of the point on a rigid +body (body of reference) with which that event or object coincides. +This applies not only to scientific description, but also to everyday +life. If I analyse the place specification “Trafalgar Square, +London”[3] I arrive at the following result. The earth is the rigid +body to which the specification of place refers; “Trafalgar Square, +London” is a well-defined point, to which a name has been assigned, and +with which the event coincides in space.[4] + +I have chosen this as being more familiar to the English reader than +the “Potzdammer Platz, Berlin,” which is referred to in the original. +(R. W. L.) + + [4] It is not necessary here to investigate further the significance + of the expression “coincidence in space.” This conception is + sufficiently obvious to ensure that differences of opinion are + scarcely likely to arise as to its applicability in practice. + +This primitive method of place specification deals only with places on +the surface of rigid bodies, and is dependent on the existence of +points on this surface which are distinguishable from each other. But +we can free ourselves from both of these limitations without altering +the nature of our specification of position. If, for instance, a cloud +is hovering over Trafalgar Square, then we can determine its position +relative to the surface of the earth by erecting a pole perpendicularly +on the Square, so that it reaches the cloud. The length of the pole +measured with the standard measuring-rod, combined with the +specification of the position of the foot of the pole, supplies us with +a complete place specification. On the basis of this illustration, we +are able to see the manner in which a refinement of the conception of +position has been developed. + +(_a_) We imagine the rigid body, to which the place specification is +referred, supplemented in such a manner that the object whose position +we require is reached by the completed rigid body. + +(_b_) In locating the position of the object, we make use of a number +(here the length of the pole measured with the measuring-rod) instead +of designated points of reference. + +(_c_) We speak of the height of the cloud even when the pole which +reaches the cloud has not been erected. By means of optical +observations of the cloud from different positions on the ground, and +taking into account the properties of the propagation of light, we +determine the length of the pole we should have required in order to +reach the cloud. + +From this consideration we see that it will be advantageous if, in the +description of position, it should be possible by means of numerical +measures to make ourselves independent of the existence of marked +positions (possessing names) on the rigid body of reference. In the +physics of measurement this is attained by the application of the +Cartesian system of co-ordinates. + +This consists of three plane surfaces perpendicular to each other and +rigidly attached to a rigid body. Referred to a system of co-ordinates, +the scene of any event will be determined (for the main part) by the +specification of the lengths of the three perpendiculars or +co-ordinates (_x, y, z_) which can be dropped from the scene of the +event to those three plane surfaces. The lengths of these three +perpendiculars can be determined by a series of manipulations with +rigid measuring-rods performed according to the rules and methods laid +down by Euclidean geometry. + +In practice, the rigid surfaces which constitute the system of +co-ordinates are generally not available; furthermore, the magnitudes +of the co-ordinates are not actually determined by constructions with +rigid rods, but by indirect means. If the results of physics and +astronomy are to maintain their clearness, the physical meaning of +specifications of position must always be sought in accordance with the +above considerations.[5] + + [5] A refinement and modification of these views does not become + necessary until we come to deal with the general theory of relativity, + treated in the second part of this book. + +We thus obtain the following result: Every description of events in +space involves the use of a rigid body to which such events have to be +referred. The resulting relationship takes for granted that the laws of +Euclidean geometry hold for “distances;” the “distance” being +represented physically by means of the convention of two marks on a +rigid body. + +The purpose of mechanics is to describe how bodies change their +position in space with “time.” I should load my conscience with grave +sins against the sacred spirit of lucidity were I to formulate the aims +of mechanics in this way, without serious reflection and detailed +explanations. Let us proceed to disclose these sins. + +It is not clear what is to be understood here by “position” and +“space.” I stand at the window of a railway carriage which is +travelling uniformly, and drop a stone on the embankment, without +throwing it. Then, disregarding the influence of the air resistance, I +see the stone descend in a straight line. A pedestrian who observes the +misdeed from the footpath notices that the stone falls to earth in a +parabolic curve. I now ask: Do the “positions” traversed by the stone +lie “in reality” on a straight line or on a parabola? Moreover, what is +meant here by motion “in space”? From the considerations of the +previous section the answer is self-evident. In the first place we +entirely shun the vague word “space,” of which, we must honestly +acknowledge, we cannot form the slightest conception, and we replace it +by “motion relative to a practically rigid body of reference.” The +positions relative to the body of reference (railway carriage or +embankment) have already been defined in detail in the preceding +section. If instead of “body of reference” we insert “system of +co-ordinates,” which is a useful idea for mathematical description, we +are in a position to say: The stone traverses a straight line relative +to a system of co-ordinates rigidly attached to the carriage, but +relative to a system of co-ordinates rigidly attached to the ground +(embankment) it describes a parabola. With the aid of this example it +is clearly seen that there is no such thing as an independently +existing trajectory (lit. “path-curve”[6], but only a trajectory +relative to a particular body of reference. + + [6] That is, a curve along which the body moves. + +In order to have a _complete_ description of the motion, we must +specify how the body alters its position _with time; i.e._ for every +point on the trajectory it must be stated at what time the body is +situated there. These data must be supplemented by such a definition of +time that, in virtue of this definition, these time-values can be +regarded essentially as magnitudes (results of measurements) capable of +observation. If we take our stand on the ground of classical mechanics, +we can satisfy this requirement for our illustration in the following +manner. We imagine two clocks of identical construction; the man at the +railway-carriage window is holding one of them, and the man on the +footpath the other. Each of the observers determines the position on +his own reference-body occupied by the stone at each tick of the clock +he is holding in his hand. In this connection we have not taken account +of the inaccuracy involved by the finiteness of the velocity of +propagation of light. With this and with a second difficulty prevailing +here we shall have to deal in detail later. + +As is well known, the fundamental law of the mechanics of +Galilei-Newton, which is known as the _law of inertia_, can be stated +thus: A body removed sufficiently far from other bodies continues in a +state of rest or of uniform motion in a straight line. This law not +only says something about the motion of the bodies, but it also +indicates the reference-bodies or systems of coordinates, permissible +in mechanics, which can be used in mechanical description. The visible +fixed stars are bodies for which the law of inertia certainly holds to +a high degree of approximation. Now if we use a system of co-ordinates +which is rigidly attached to the earth, then, relative to this system, +every fixed star describes a circle of immense radius in the course of +an astronomical day, a result which is opposed to the statement of the +law of inertia. So that if we adhere to this law we must refer these +motions only to systems of coordinates relative to which the fixed +stars do not move in a circle. A system of co-ordinates of which the +state of motion is such that the law of inertia holds relative to it is +called a “Galileian system of co-ordinates.” The laws of the mechanics +of Galilei-Newton can be regarded as valid only for a Galileian system +of co-ordinates. + +In order to attain the greatest possible clearness, let us return to +our example of the railway carriage supposed to be travelling +uniformly. We call its motion a uniform translation (“uniform” because +it is of constant velocity and direction, “translation” because +although the carriage changes its position relative to the embankment +yet it does not rotate in so doing). Let us imagine a raven flying +through the air in such a manner that its motion, as observed from the +embankment, is uniform and in a straight line. If we were to observe +the flying raven from the moving railway carriage. we should find that +the motion of the raven would be one of different velocity and +direction, but that it would still be uniform and in a straight line. +Expressed in an abstract manner we may say: If a mass _m_ is moving +uniformly in a straight line with respect to a co-ordinate system _K_, +then it will also be moving uniformly and in a straight line relative +to a second co-ordinate system _K′_ provided that the latter is +executing a uniform translatory motion with respect to _K_. In +accordance with the discussion contained in the preceding section, it +follows that: + +If _K_ is a Galileian co-ordinate system. then every other co-ordinate +system _K′_ is a Galileian one, when, in relation to _K_, it is in a +condition of uniform motion of translation. Relative to _K′_ the +mechanical laws of Galilei-Newton hold good exactly as they do with +respect to _K_. + +We advance a step farther in our generalisation when we express the +tenet thus: If, relative to _K_, _K′_ is a uniformly moving co-ordinate +system devoid of rotation, then natural phenomena run their course with +respect to _K′_ according to exactly the same general laws as with +respect to _K_. This statement is called the _principle of relativity_ +(in the restricted sense). + +As long as one was convinced that all natural phenomena were capable of +representation with the help of classical mechanics, there was no need +to doubt the validity of this principle of relativity. But in view of +the more recent development of electrodynamics and optics it became +more and more evident that classical mechanics affords an insufficient +foundation for the physical description of all natural phenomena. At +this juncture the question of the validity of the principle of +relativity became ripe for discussion, and it did not appear impossible +that the answer to this question might be in the negative. + +Nevertheless, there are two general facts which at the outset speak +very much in favour of the validity of the principle of relativity. +Even though classical mechanics does not supply us with a sufficiently +broad basis for the theoretical presentation of all physical phenomena, +still we must grant it a considerable measure of “truth,” since it +supplies us with the actual motions of the heavenly bodies with a +delicacy of detail little short of wonderful. The principle of +relativity must therefore apply with great accuracy in the domain of +_mechanics_. But that a principle of such broad generality should hold +with such exactness in one domain of phenomena, and yet should be +invalid for another, is _a priori_ not very probable. + +We now proceed to the second argument, to which, moreover, we shall +return later. If the principle of relativity (in the restricted sense) +does not hold, then the Galileian co-ordinate systems _K, K′, K″_, +etc., which are moving uniformly relative to each other, will not be +_equivalent_ for the description of natural phenomena. In this case we +should be constrained to believe that natural laws are capable of being +formulated in a particularly simple manner, and of course only on +condition that, from amongst all possible Galileian co-ordinate +systems, we should have chosen _one_ (_K0_) of a particular state of +motion as our body of reference. We should then be justified (because +of its merits for the description of natural phenomena) in calling this +system “absolutely at rest,” and all other Galileian systems _K_ “in +motion.” If, for instance, our embankment were the system _K0_ then our +railway carriage would be a system _K_, relative to which less simple +laws would hold than with respect to _K0_. This diminished simplicity +would be due to the fact that the carriage _K_ would be in motion +(_i.e._ “really”)with respect to _K0_. In the general laws of nature +which have been formulated with reference to _K_, the magnitude and +direction of the velocity of the carriage would necessarily play a +part. We should expect, for instance, that the note emitted by an +organpipe placed with its axis parallel to the direction of travel +would be different from that emitted if the axis of the pipe were +placed perpendicular to this direction. + +Now in virtue of its motion in an orbit round the sun, our earth is +comparable with a railway carriage travelling with a velocity of about +30 kilometres per second. If the principle of relativity were not valid +we should therefore expect that the direction of motion of the earth at +any moment would enter into the laws of nature, and also that physical +systems in their behaviour would be dependent on the orientation in +space with respect to the earth. For owing to the alteration in +direction of the velocity of revolution of the earth in the course of a +year, the earth cannot be at rest relative to the hypothetical system +_K0_ throughout the whole year. However, the most careful observations +have never revealed such anisotropic properties in terrestrial physical +space, _i.e._ a physical non-equivalence of different directions. This +is very powerful argument in favour of the principle of relativity. + +Let us suppose our old friend the railway carriage to be travelling +along the rails with a constant velocity _v_, and that a man traverses +the length of the carriage in the direction of travel with a velocity +_w_. How quickly or, in other words, with what velocity _W_ does the +man advance relative to the embankment during the process? The only +possible answer seems to result from the following consideration: If +the man were to stand still for a second, he would advance relative to +the embankment through a distance _v_ equal numerically to the velocity +of the carriage. As a consequence of his walking, however, he traverses +an additional distance w relative to the carriage, and hence also +relative to the embankment, in this second, the distance w being +numerically equal to the velocity with which he is walking. Thus in +total he covers the distance _W = v + w_ relative to the embankment in +the second considered. We shall see later that this result, which +expresses the theorem of the addition of velocities employed in +classical mechanics, cannot be maintained; in other words, the law that +we have just written down does not hold in reality. For the time being, +however, we shall assume its correctness. + +There is hardly a simpler law in physics than that according to which +light is propagated in empty space. Every child at school knows, or +believes he knows, that this propagation takes place in straight lines +with a velocity _c_ = 300,000 km./sec. At all events we know with great +exactness that this velocity is the same for all colours, because if +this were not the case, the minimum of emission would not be observed +simultaneously for different colours during the eclipse of a fixed star +by its dark neighbour. By means of similar considerations based on +observations of double stars, the Dutch astronomer De Sitter was also +able to show that the velocity of propagation of light cannot depend on +the velocity of motion of the body emitting the light. The assumption +that this velocity of propagation is dependent on the direction “in +space” is in itself improbable. + +In short, let us assume that the simple law of the constancy of the +velocity of light _c_ (in vacuum) is justifiably believed by the child +at school. Who would imagine that this simple law has plunged the +conscientiously thoughtful physicist into the greatest intellectual +difficulties? Let us consider how these difficulties arise. + +Of course we must refer the process of the propagation of light (and +indeed every other process) to a rigid reference-body (co-ordinate +system). As such a system let us again choose our embankment. We shall +imagine the air above it to have been removed. If a ray of light be +sent along the embankment, we see from the above that the tip of the +ray will be transmitted with the velocity _c_ relative to the +embankment. Now let us suppose that our railway carriage is again +travelling along the railway lines with the velocity _v_, and that its +direction is the same as that of the ray of light, but its velocity of +course much less. Let us inquire about the velocity of propagation of +the ray of light relative to the carriage. It is obvious that we can +here apply the consideration of the previous section, since the ray of +light plays the part of the man walking along relatively to the +carriage. The velocity _W_ of the man relative to the embankment is +here replaced by the velocity of light relative to the embankment. _w_ +is the required velocity of light with respect to the carriage, and we +have + +_w = c – v._ + +The velocity of propagation ot a ray of light relative to the carriage +thus comes out smaller than _c_. + +But this result comes into conflict with the principle of relativity +set forth in Section V. For, like every other general law of nature, +the law of the transmission of light _in vacuo_ [in vacuum] must, +according to the principle of relativity, be the same for the railway +carriage as reference-body as when the rails are the body of reference. +But, from our above consideration, this would appear to be impossible. +If every ray of light is propagated relative to the embankment with the +velocity _c_, then for this reason it would appear that another law of +propagation of light must necessarily hold with respect to the +carriage—a result contradictory to the principle of relativity. + +In view of this dilemma there appears to be nothing else for it than to +abandon either the principle of relativity or the simple law of the +propagation of light _in vacuo_. Those of you who have carefully +followed the preceding discussion are almost sure to expect that we +should retain the principle of relativity, which appeals so +convincingly to the intellect because it is so natural and simple. The +law of the propagation of light _in vacuo_ would then have to be +replaced by a more complicated law conformable to the principle of +relativity. The development of theoretical physics shows, however, that +we cannot pursue this course. The epoch-making theoretical +investigations of H. A. Lorentz on the electrodynamical and optical +phenomena connected with moving bodies show that experience in this +domain leads conclusively to a theory of electromagnetic phenomena, of +which the law of the constancy of the velocity of light in vacuo is a +necessary consequence. Prominent theoretical physicists were therefore +more inclined to reject the principle of relativity, in spite of the +fact that no empirical data had been found which were contradictory to +this principle. + +At this juncture the theory of relativity entered the arena. As a +result of an analysis of the physical conceptions of time and space, it +became evident that _in reality there is not the least incompatibilitiy +between the principle of relativity and the law of propagation of +light_, and that by systematically holding fast to both these laws a +logically rigid theory could be arrived at. This theory has been called +the _special theory of relativity_ to distinguish it from the extended +theory, with which we shall deal later. In the following pages we shall +present the fundamental ideas of the special theory of relativity. + +Lightning has struck the rails on our railway embankment at two places +_A_ and _B_ far distant from each other. I make the additional +assertion that these two lightning flashes occurred simultaneously. If +I ask you whether there is sense in this statement, you will answer my +question with a decided “Yes.” But if I now approach you with the +request to explain to me the sense of the statement more precisely, you +find after some consideration that the answer to this question is not +so easy as it appears at first sight. + +After some time perhaps the following answer would occur to you: “The +significance of the statement is clear in itself and needs no further +explanation; of course it would require some consideration if I were to +be commissioned to determine by observations whether in the actual case +the two events took place simultaneously or not.” I cannot be satisfied +with this answer for the following reason. Supposing that as a result +of ingenious considerations an able meteorologist were to discover that +the lightning must always strike the places _A_ and _B_ simultaneously, +then we should be faced with the task of testing whether or not this +theoretical result is in accordance with the reality. We encounter the +same difficulty with all physical statements in which the conception +“simultaneous” plays a part. The concept does not exist for the +physicist until he has the possibility of discovering whether or not it +is fulfilled in an actual case. We thus require a definition of +simultaneity such that this definition supplies us with the method by +means of which, in the present case, he can decide by experiment +whether or not both the lightning strokes occurred simultaneously. As +long as this requirement is not satisfied, I allow myself to be +deceived as a physicist (and of course the same applies if I am not a +physicist), when I imagine that I am able to attach a meaning to the +statement of simultaneity. (I would ask the reader not to proceed +farther until he is fully convinced on this point.) + +After thinking the matter over for some time you then offer the +following suggestion with which to test simultaneity. By measuring +along the rails, the connecting line _AB_ should be measured up and an +observer placed at the mid-point M of the distance _AB_. This observer +should be supplied with an arrangement (_e.g._ two mirrors inclined at +90°) which allows him visually to observe both places _A_ and _B_ at +the same time. If the observer perceives the two flashes of lightning +at the same time, then they are simultaneous. + +I am very pleased with this suggestion, but for all that I cannot +regard the matter as quite settled, because I feel constrained to raise +the following objection: “Your definition would certainly be right, if +only I knew that the light by means of which the observer at _M_ +perceives the lightning flashes travels along the length _A_ → _M_ with +the same velocity as along the length _B_ → _M_. But an examination of +this supposition would only be possible if we already had at our +disposal the means of measuring time. It would thus appear as though we +were moving here in a logical circle.” + +After further consideration you cast a somewhat disdainful glance at +me—and rightly so—and you declare: “I maintain my previous definition +nevertheless, because in reality it assumes absolutely nothing about +light. There is only _one_ demand to be made of the definition of +simultaneity, namely, that in every real case it must supply us with an +empirical decision as to whether or not the conception that has to be +defined is fulfilled. That my definition satisfies this demand is +indisputable. That light requires the same time to traverse the path +_A_ → _M_ as for the path _B_ → _M_ is in reality neither a +_supposition nor a hypothesis_ about the physical nature of light, but +a _stipulation_ which I can make of my own freewill in order to arrive +at a definition of simultaneity.” + +It is clear that this definition can be used to give an exact meaning +not only to _two_ events, but to as many events as we care to choose, +and independently of the positions of the scenes of the events with +respect to the body of reference[7] (here the railway embankment). We +are thus led also to a definition of “time” in physics. For this +purpose we suppose that clocks of identical construction are placed at +the points _A, B_ and _C_ of the railway line (co-ordinate system) and +that they are set in such a manner that the positions of their pointers +are simultaneously (in the above sense) the same. Under these +conditions we understand by the “time” of an event the reading +(position of the hands) of that one of these clocks which is in the +immediate vicinity (in space) of the event. In this manner a time-value +is associated with every event which is essentially capable of +observation. + + [7] We suppose further that, when three events _A, B_ and _C_ occur in + different places in such a manner that, if _A_ is simultaneous with + _B_, and _B_ is simultaneous with _C_ (simultaneous in the sense of + the above definition), then the criterion for the simultaneity of the + pair of events _A, C_ is also satisfied. This assumption is a physical + hypothesis about the law of propagation of light; it must certainly be + fulfilled if we are to maintain the law of the constancy of the + velocity of light _in vacuo_. + +This stipulation contains a further physical hypothesis, the validity +of which will hardly be doubted without empirical evidence to the +contrary. It has been assumed that all these clocks _go at the same +rate_ if they are of identical construction. Stated more exactly: When +two clocks arranged at rest in different places of a reference-body are +set in such a manner that a _particular_ position of the pointers of +the one clock is _simultaneous_ (in the above sense) with the _same_ +position, of the pointers of the other clock, then identical “settings” +are always simultaneous (in the sense of the above definition). + +Up to now our considerations have been referred to a particular body of +reference, which we have styled a “railway embankment.” We suppose a +very long train travelling along the rails with the constant velocity v +and in the direction indicated in Fig 1. People travelling in this +train will with a vantage view the train as a rigid reference-body +(co-ordinate system); they regard all events in reference to the train. +Then every event which takes place along the line also takes place at a +particular point of the train. Also the definition of simultaneity can +be given relative to the train in exactly the same way as with respect +to the embankment. As a natural consequence, however, the following +question arises: + +image001 + +Are two events (_e.g._ the two strokes of lightning _A_ and _B_) which +are simultaneous _with reference to the railway embankment_ also +simultaneous _relatively to the train?_ We shall show directly that the +answer must be in the negative. + +When we say that the lightning strokes _A_ and _B_ are simultaneous +with respect to be embankment, we mean: the rays of light emitted at +the places _A_ and _B_, where the lightning occurs, meet each other at +the mid-point _M_ of the length _A_ → _B_ of the embankment. But the +events _A_ and _B_ also correspond to positions _A_ and _B_ on the +train. Let _M′_ be the mid-point of the distance _A_ → _B_ on the +travelling train. Just when the flashes (as judged from the embankment) +of lightning occur, this point _M′_ naturally coincides with the point +_M_ but it moves towards the right in the diagram with the velocity v +of the train. If an observer sitting in the position _M′_ in the train +did not possess this velocity, then he would remain permanently at M, +and the light rays emitted by the flashes of lightning _A_ and _B_ +would reach him simultaneously, _i.e._ they would meet just where he is +situated. Now in reality (considered with reference to the railway +embankment) he is hastening towards the beam of light coming from _B_, +whilst he is riding on ahead of the beam of light coming from _A_. +Hence the observer will see the beam of light emitted from _B_ earlier +than he will see that emitted from _A_. Observers who take the railway +train as their reference-body must therefore come to the conclusion +that the lightning flash _B_ took place earlier than the lightning +flash _A_. We thus arrive at the important result: + +Events which are simultaneous with reference to the embankment are not +simultaneous with respect to the train, and _vice versa_ (relativity of +simultaneity). Every reference-body (co-ordinate system) has its own +particular time; unless we are told the reference-body to which the +statement of time refers, there is no meaning in a statement of the +time of an event. + +Now before the advent of the theory of relativity it had always tacitly +been assumed in physics that the statement of time had an absolute +significance, _i.e._ that it is independent of the state of motion of +the body of reference. But we have just seen that this assumption is +incompatible with the most natural definition of simultaneity; if we +discard this assumption, then the conflict between the law of the +propagation of light _in vacuo_ and the principle of relativity +(developed in Section VII) disappears. + +We were led to that conflict by the considerations of Section VI, which +are now no longer tenable. In that section we concluded that the man in +the carriage, who traverses the distance _w per second_ relative to the +carriage, traverses the same distance also with respect to the +embankment _in each second_ of time. But, according to the foregoing +considerations, the time required by a particular occurrence with +respect to the carriage must not be considered equal to the duration of +the same occurrence as judged from the embankment (as reference-body). +Hence it cannot be contended that the man in walking travels the +distance _w_ relative to the railway line in a time which is equal to +one second as judged from the embankment. + +Moreover, the considerations of Section VI are based on yet a second +assumption, which, in the light of a strict consideration, appears to +be arbitrary, although it was always tacitly made even before the +introduction of the theory of relativity. + +Let us consider two particular points on the train [8] travelling along +the embankment with the velocity _v_, and inquire as to their distance +apart. We already know that it is necessary to have a body of reference +for the measurement of a distance, with respect to which body the +distance can be measured up. It is the simplest plan to use the train +itself as reference-body (co-ordinate system). An observer in the train +measures the interval by marking off his measuring-rod in a straight +line (_e.g._ along the floor of the carriage) as many times as is +necessary to take him from the one marked point to the other. Then the +number which tells us how often the rod has to be laid down is the +required distance. + + [8] _e.g._ the middle of the first and of the hundredth carriage. + +It is a different matter when the distance has to be judged from the +railway line. Here the following method suggests itself. If we call +_A′_ and _B′_ the two points on the train whose distance apart is +required, then both of these points are moving with the velocity v +along the embankment. In the first place we require to determine the +points _A_ and _B_ of the embankment which are just being passed by the +two points _A′_ and _B′_ at a particular time t—judged from the +embankment. These points _A_ and _B_ of the embankment can be +determined by applying the definition of time given in Section VIII. +The distance between these points A and B is then measured by repeated +application of the measuring-rod along the embankment. + +_A priori_ it is by no means certain that this last measurement will +supply us with the same result as the first. Thus the length of the +train as measured from the embankment may be different from that +obtained by measuring in the train itself. This circumstance leads us +to a second objection which must be raised against the apparently +obvious consideration of Section VI. Namely, if the man in the carriage +covers the distance _w_ in a unit of time—_measured from the +train_,—then this distance—_as measured from the embankment_ is not +necessarily also equal to _w_. + +The results of the last three sections show that the apparent +incompatibility of the law of propagation of light with the principle +of relativity (Section VII) has been derived by means of a +consideration which borrowed two unjustifiable hypotheses from +classical mechanics; these are as follows: + +(1) The time-interval (time) between two events is independent of the +condition of motion of the body of reference. + +(2) The space-interval (distance) between two points of a rigid body is +independent of the condition of motion of the body of reference. + +If we drop these hypotheses, then the dilemma of Section VII +disappears, because the theorem of the addition of velocities derived +in Section VI becomes invalid. The possibility presents itself that the +law of the propagation of light _in vacuo_ may be compatible with the +principle of relativity, and the question arises: How have we to modify +the considerations of Section VI in order to remove the apparent +disagreement between these two fundamental results of experience? This +question leads to a general one. In the discussion of Section VI we +have to do with places and times relative both to the train and to the +embankment. How are we to find the place and time of an event in +relation to the train, when we know the place and time of the event +with respect to the railway embankment? Is there a thinkable answer to +this question of such a nature that the law of transmission of light +_in vacuo_ does not contradict the principle of relativity? In other +words: Can we conceive of a relation between place and time of the +individual events relative to both reference-bodies, such that every +ray of light possesses the velocity of transmission _c_ relative to the +embankment and relative to the train? This question leads to a quite +definite positive answer, and to a perfectly definite transformation +law for the space-time magnitudes of an event when changing over from +one body of reference to another. + +Before we deal with this, we shall introduce the following incidental +consideration. Up to the present we have only considered events taking +place along the embankment, which had mathematically to assume the +function of a straight line. In the manner indicated in Section II we +can imagine this reference-body supplemented laterally and in a +vertical direction by means of a framework of rods, so that an event +which takes place anywhere can be localised with reference to this +framework. Similarly, we can imagine the train travelling with the +velocity _v_ to be continued across the whole of space, so that every +event, no matter how far off it may be, could also be localised with +respect to the second framework. Without committing any fundamental +error, we can disregard the fact that in reality these frameworks would +continually interfere with each other, owing to the impenetrability of +solid bodies. In every such framework we imagine three surfaces +perpendicular to each other marked out, and designated as “co-ordinate +planes” (“co-ordinate system”). A co-ordinate system _K_ then +corresponds to the embankment, and a co-ordinate system _K′_ to the +train. An event, wherever it may have taken place, would be fixed in +space with respect to _K_ by the three perpendiculars _x, y, z_ on the +co-ordinate planes, and with regard to time by a time value _t_. +Relative to _K′, the same event_ would be fixed in respect of space and +time by corresponding values _x′, y′, z′, t′_, which of course are not +identical with _x, y, z, t_. It has already been set forth in detail +how these magnitudes are to be regarded as results of physical +measurements. + +image002 + +Obviously our problem can be exactly formulated in the following +manner. What are the values _x′, y′, z′, t′_, of an event with respect +to _K′_, when the magnitudes _x, y, z, t_, of the same event with +respect to _K_ are given? The relations must be so chosen that the law +of the transmission of light in vacuo is satisfied for one and the same +ray of light (and of course for every ray) with respect to _K_ and +_K′_. For the relative orientation in space of the co-ordinate systems +indicated in the diagram (Fig. 2), this problem is solved by means of +the equations: + +image003 + +_y′_ = _y_ + +_z′_ = _z_ + +image004 + +This system of equations is known as the “Lorentz transformation.”[9] + + [9] A simple derivation of the Lorentz transformation is given in + Appendix I. + +If in place of the law of transmission of light we had taken as our +basis the tacit assumptions of the older mechanics as to the absolute +character of times and lengths, then instead of the above we should +have obtained the following equations: + +_x′_ = _x_ – _vt_ + +_y′_ = _y_ + +_z′_ = _z_ + +_t′_ = _t_ + +This system of equations is often termed the “Galilei transformation.” +The Galilei transformation can be obtained from the Lorentz +transformation by substituting an infinitely large value for the +velocity of light _c_ in the latter transformation. + +Aided by the following illustration, we can readily see that, in +accordance with the Lorentz transformation, the law of the transmission +of light _in vacuo_ is satisfied both for the reference-body _K_ and +for the reference-body _K′_. A light-signal is sent along the positive +_x_-axis, and this light-stimulus advances in accordance with the +equation + +_x_ = _ct_, + +_i.e._ with the velocity _c_. According to the equations of the Lorentz +transformation, this simple relation between _x_ and _t_ involves a +relation between _x′_ and _t′_. In point of fact, if we substitute for +_x_ the value _ct_ in the first and fourth equations of the Lorentz +transformation, we obtain: + +image005 + +from which, by division, the expression + +_x′_ = _ct′_ + +immediately follows. If referred to the system _K′_, the propagation of +light takes place according to this equation. We thus see that the +velocity of transmission relative to the reference-body _K′_ is also +equal to _c_. The same result is obtained for rays of light advancing +in any other direction whatsoever. Of cause this is not surprising, +since the equations of the Lorentz transformation were derived +conformably to this point of view. + +Place a metre-rod in the _x′_-axis of _K′_ in such a manner that one +end (the beginning) coincides with the point _x′_ = 0 whilst the other +end (the end of the rod) coincides with the point _x′_ = 1. What is the +length of the metre-rod relatively to the system _K_? In order to learn +this, we need only ask where the beginning of the rod and the end of +the rod lie with respect to _K_ at a particular time _t_ of the system +_K_. By means of the first equation of the Lorentz transformation the +values of these two points at the time _t_ = 0 can be shown to be + +image006 + +the distance between the points being + +image007 + +But the metre-rod is moving with the velocity _v_ relative to _K_. It +therefore follows that the length of a rigid metre-rod moving in the +direction of its length with a velocity _v_ is + +image008 + +of a metre. The rigid rod is thus shorter when in motion than when at +rest, and the more quickly it is moving, the shorter is the rod. For +the velocity _v_ = _c_ we should have + +image009 + +and for still greater velocities the square-root becomes imaginary. +From this we conclude that in the theory of relativity the velocity _c_ +plays the part of a limiting velocity, which can neither be reached nor +exceeded by any real body. + +Of course this feature of the velocity _c_ as a limiting velocity also +clearly follows from the equations of the Lorentz transformation, for +these became meaningless if we choose values of _v_ greater than _c_. + +If, on the contrary, we had considered a metre-rod at rest in the +_x_-axis with respect to _K_, then we should have found that the length +of the rod as judged from _K′_ would have been + +image010 + +this is quite in accordance with the principle of relativity which +forms the basis of our considerations. + +_A priori_ it is quite clear that we must be able to learn something +about the physical behaviour of measuring-rods and clocks from the +equations of transformation, for the magnitudes _z, y, x, t_, are +nothing more nor less than the results of measurements obtainable by +means of measuring-rods and clocks. If we had based our considerations +on the Galileian transformation we should not have obtained a +contraction of the rod as a consequence of its motion. + +Let us now consider a seconds-clock which is permanently situated at +the origin (_x′_ = 0) of _K′_. _t′_ = 0 and _t′_ = 1 are two successive +ticks of this clock. The first and fourth equations of the Lorentz +transformation give for these two ticks: + +_t_ = 0 + +and + +image011 + +As judged from _K_, the clock is moving with the velocity _v_; as +judged from this reference-body, the time which elapses between two +strokes of the clock is not one second, but + +image012 + +seconds, _i.e._ a somewhat larger time. As a consequence of its motion +the clock goes more slowly than when at rest. Here also the velocity +_c_ plays the part of an unattainable limiting velocity. + +Now in practice we can move clocks and measuring-rods only with +velocities that are small compared with the velocity of light; hence we +shall hardly be able to compare the results of the previous section +directly with the reality. But, on the other hand, these results must +strike you as being very singular, and for that reason I shall now draw +another conclusion from the theory, one which can easily be derived +from the foregoing considerations, and which has been most elegantly +confirmed by experiment. + +In Section VI we derived the theorem of the addition of velocities in +one direction in the form which also results from the hypotheses of +classical mechanics. This theorem can also be deduced readily from the +Galilei transformation (Section XI). In place of the man walking inside +the carriage, we introduce a point moving relatively to the co-ordinate +system _K′_ in accordance with the equation + +_x′_ = _wt′_ + +By means of the first and fourth equations of the Galilei +transformation we can express _x′_ and _t′_ in terms of _x_ and _t_, +and we then obtain + +_x_ = (_v_ + _w_)_t_ + +This equation expresses nothing else than the law of motion of the +point with reference to the system _K_ (of the man with reference to +the embankment). We denote this velocity by the symbol _W_, and we then +obtain, as in Section VI, + +_W_ = _v_ + _w_ . . . . . . . (A). + +But we can carry out this consideration just as well on the basis of +the theory of relativity. In the equation + +_x′_ = _wt′_ + +we must then express _x′_ and _t′_ in terms of _x_ and _t_, making use +of the first and fourth equations of the _Lorentz transformation_. +Instead of the equation (A) we then obtain the equation + +image013 + +which corresponds to the theorem of addition for velocities in one +direction according to the theory of relativity. The question now +arises as to which of these two theorems is the better in accord with +experience. On this point we are enlightened by a most important +experiment which the brilliant physicist Fizeau performed more than +half a century ago, and which has been repeated since then by some of +the best experimental physicists, so that there can be no doubt about +its result. The experiment is concerned with the following question. +Light travels in a motionless liquid with a particular velocity _w_. +How quickly does it travel in the direction of the arrow in the tube +_T_ (see the accompanying diagram, Fig. 3) when the liquid above +mentioned is flowing through the tube with a velocity _v_? + +image014 + +In accordance with the principle of relativity we shall certainly have +to take for granted that the propagation of light always takes place +with the same velocity _w with respect to the liquid_, whether the +latter is in motion with reference to other bodies or not. The velocity +of light relative to the liquid and the velocity of the latter relative +to the tube are thus known, and we require the velocity of light +relative to the tube. + +It is clear that we have the problem of Section VI again before us. The +tube plays the part of the railway embankment or of the co-ordinate +system _K_, the liquid plays the part of the carriage or of the +co-ordinate system _K′_, and finally, the light plays the part of the +man walking along the carriage, or of the moving point in the present +section. If we denote the velocity of the light relative to the tube by +_W_, then this is given by the equation (A) or (B), according as the +Galilei transformation or the Lorentz transformation corresponds to the +facts. Experiment[10] decides in favour of equation (B) derived from +the theory of relativity, and the agreement is, indeed, very exact. +According to recent and most excellent measurements by Zeeman, the +influence of the velocity of flow _v_ on the propagation of light is +represented by formula (B) to within one per cent. + + [10] Fizeau found + +image015 + +where + +image016 + +is the index of refraction of the liquid. On the other hand, owing to +the smallness of + +image017 + +as compared with 1, we can replace (B) in the first place by + +image018 + +or to the same order of approximation by + +image019 + +which agrees with Fizeau’s result. + +Nevertheless we must now draw attention to the fact that a theory of +this phenomenon was given by H. A. Lorentz long before the statement of +the theory of relativity. This theory was of a purely electrodynamical +nature, and was obtained by the use of particular hypotheses as to the +electromagnetic structure of matter. This circumstance, however, does +not in the least diminish the conclusiveness of the experiment as a +crucial test in favour of the theory of relativity, for the +electrodynamics of Maxwell-Lorentz, on which the original theory was +based, in no way opposes the theory of relativity. Rather has the +latter been developed trom electrodynamics as an astoundingly simple +combination and generalisation of the hypotheses, formerly independent +of each other, on which electrodynamics was built. + +Our train of thought in the foregoing pages can be epitomised in the +following manner. Experience has led to the conviction that, on the one +hand, the principle of relativity holds true and that on the other hand +the velocity of transmission of light _in vacuo_ has to be considered +equal to a constant _c_. By uniting these two postulates we obtained +the law of transformation for the rectangular co-ordinates _x, y, z_ +and the time _t_ of the events which constitute the processes of +nature. In this connection we did not obtain the Galilei +transformation, but, differing from classical mechanics, the _Lorentz +transformation_. + +The law of transmission of light, the acceptance of which is justified +by our actual knowledge, played an important part in this process of +thought. Once in possession of the Lorentz transformation, however, we +can combine this with the principle of relativity, and sum up the +theory thus: + +Every general law of nature must be so constituted that it is +transformed into a law of exactly the same form when, instead of the +space-time variables _x, y, z, t_ of the original coordinate system +_K_, we introduce new space-time variables _x′, y′, z′, t′_ of a +co-ordinate system _K′_. In this connection the relation between the +ordinary and the accented magnitudes is given by the Lorentz +transformation. Or in brief: General laws of nature are co-variant with +respect to Lorentz transformations. + +This is a definite mathematical condition that the theory of relativity +demands of a natural law, and in virtue of this, the theory becomes a +valuable heuristic aid in the search for general laws of nature. If a +general law of nature were to be found which did not satisfy this +condition, then at least one of the two fundamental assumptions of the +theory would have been disproved. Let us now examine what general +results the latter theory has hitherto evinced. + +It is clear from our previous considerations that the (special) theory +of relativity has grown out of electrodynamics and optics. In these +fields it has not appreciably altered the predictions of theory, but it +has considerably simplified the theoretical structure, _i.e._ the +derivation of laws, and—what is incomparably more important—it has +considerably reduced the number of independent hypotheses forming the +basis of theory. The special theory of relativity has rendered the +Maxwell-Lorentz theory so plausible, that the latter would have been +generally accepted by physicists even if experiment had decided less +unequivocally in its favour. + +Classical mechanics required to be modified before it could come into +line with the demands of the special theory of relativity. For the main +part, however, this modification affects only the laws for rapid +motions, in which the velocities of matter _v_ are not very small as +compared with the velocity of light. We have experience of such rapid +motions only in the case of electrons and ions; for other motions the +variations from the laws of classical mechanics are too small to make +themselves evident in practice. We shall not consider the motion of +stars until we come to speak of the general theory of relativity. In +accordance with the theory of relativity the kinetic energy of a +material point of mass _m_ is no longer given by the well-known +expression + +image020 + +but by the expression + +image021 + +This expression approaches infinity as the velocity _v_ approaches the +velocity of light _c_. The velocity must therefore always remain less +than _c_, however great may be the energies used to produce the +acceleration. If we develop the expression for the kinetic energy in +the form of a series, we obtain + +image022 + +When + +image023 + +is small compared with unity, the third of these terms is always small +in comparison with the second, which last is alone considered in +classical mechanics. The first term _mc_2 does not contain the +velocity, and requires no consideration if we are only dealing with the +question as to how the energy of a point-mass; depends on the velocity. +We shall speak of its essential significance later. + +The most important result of a general character to which the special +theory of relativity has led is concerned with the conception of mass. +Before the advent of relativity, physics recognised two conservation +laws of fundamental importance, namely, the law of the conservation of +energy and the law of the conservation of mass these two fundamental +laws appeared to be quite independent of each other. By means of the +theory of relativity they have been united into one law. We shall now +briefly consider how this unification came about, and what meaning is +to be attached to it. + +The principle of relativity requires that the law of the conservation +of energy should hold not only with reference to a co-ordinate system +_K_, but also with respect to every co-ordinate system _K′_ which is in +a state of uniform motion of translation relative to _K_, or, briefly, +relative to every “Galileian” system of co-ordinates. In contrast to +classical mechanics; the Lorentz transformation is the deciding factor +in the transition from one such system to another. + +By means of comparatively simple considerations we are led to draw the +following conclusion from these premises, in conjunction with the +fundamental equations of the electrodynamics of Maxwell: A body moving +with the velocity _v_, which absorbs[11] an amount of energy _E_0 in +the form of radiation without suffering an alteration in velocity in +the process, has, as a consequence, its energy increased by an amount + +image024 + + [11] _E_0 is the energy taken up, as judged from a co-ordinate system + moving with the body. + +In consideration of the expression given above for the kinetic energy +of the body, the required energy of the body comes out to be + +image025 + +Thus the body has the same energy as a body of mass + +image026 + +moving with the velocity _v_. Hence we can say: If a body takes up an +amount of energy _E_0, then its inertial mass increases by an amount + +image027 + +the inertial mass of a body is not a constant but varies according to +the change in the energy of the body. The inertial mass of a system of +bodies can even be regarded as a measure of its energy. The law of the +conservation of the mass of a system becomes identical with the law of +the conservation of energy, and is only valid provided that the system +neither takes up nor sends out energy. Writing the expression for the +energy in the form + +image028 + +we see that the term _mc_2, which has hitherto attracted our attention, +is nothing else than the energy possessed by the body[12] before it +absorbed the energy _E_0. + + [12] As judged from a co-ordinate system moving with the body. + +A direct comparison of this relation with experiment is not possible at +the present time (1920; see[Note], p. 48), owing to the fact that the +changes in energy _E_0 to which we can subject a system are not large +enough to make themselves perceptible as a change in the inertial mass +of the system. + +image027 + +is too small in comparison with the mass _m_, which was present before +the alteration of the energy. It is owing to this circumstance that +classical mechanics was able to establish successfully the conservation +of mass as a law of independent validity. + + [Note] The equation E = mc2 has been thoroughly proved time and again + since this time. + +Let me add a final remark of a fundamental nature. The success of the +Faraday-Maxwell interpretation of electromagnetic action at a distance +resulted in physicists becoming convinced that there are no such things +as instantaneous actions at a distance (not involving an intermediary +medium) of the type of Newton’s law of gravitation. + +According to the theory of relativity, action at a distance with the +velocity of light always takes the place of instantaneous action at a +distance or of action at a distance with an infinite velocity of +transmission. This is connected with the fact that the velocity _c_ +plays a fundamental role in this theory. In Part II we shall see in +what way this result becomes modified in the general theory of +relativity. + +To what extent is the special theory of relativity supported by +experience? This question is not easily answered for the reason already +mentioned in connection with the fundamental experiment of Fizeau. The +special theory of relativity has crystallised out from the +Maxwell-Lorentz theory of electromagnetic phenomena. Thus all facts of +experience which support the electromagnetic theory also support the +theory of relativity. As being of particular importance, I mention here +the fact that the theory of relativity enables us to predict the +effects produced on the light reaching us from the fixed stars. These +results are obtained in an exceedingly simple manner, and the effects +indicated, which are due to the relative motion of the earth with +reference to those fixed stars are found to be in accord with +experience. We refer to the yearly movement of the apparent position of +the fixed stars resulting from the motion of the earth round the sun +(aberration), and to the influence of the radial components of the +relative motions of the fixed stars with respect to the earth on the +colour of the light reaching us from them. The latter effect manifests +itself in a slight displacement of the spectral lines of the light +transmitted to us from a fixed star, as compared with the position of +the same spectral lines when they are produced by a terrestrial source +of light (Doppler principle). The experimental arguments in favour of +the Maxwell-Lorentz theory, which are at the same time arguments in +favour of the theory of relativity, are too numerous to be set forth +here. In reality they limit the theoretical possibilities to such an +extent, that no other theory than that of Maxwell and Lorentz has been +able to hold its own when tested by experience. + +But there are two classes of experimental facts hitherto obtained which +can be represented in the Maxwell-Lorentz theory only by the +introduction of an auxiliary hypothesis, which in itself—_i.e._ without +making use of the theory of relativity—appears extraneous. + +It is known that cathode rays and the so-called β-rays emitted by +radioactive substances consist of negatively electrified particles +(electrons) of very small inertia and large velocity. By examining the +deflection of these rays under the influence of electric and magnetic +fields, we can study the law of motion of these particles very exactly. + +In the theoretical treatment of these electrons, we are faced with the +difficulty that electrodynamic theory of itself is unable to give an +account of their nature. For since electrical masses of one sign repel +each other, the negative electrical masses constituting the electron +would necessarily be scattered under the influence of their mutual +repulsions, unless there are forces of another kind operating between +them, the nature of which has hitherto remained obscure to us.[13] If +we now assume that the relative distances between the electrical masses +constituting the electron remain unchanged during the motion of the +electron (rigid connection in the sense of classical mechanics), we +arrive at a law of motion of the electron which does not agree with +experience. Guided by purely formal points of view, H. A. Lorentz was +the first to introduce the hypothesis that the form of the electron +experiences a contraction in the direction of motion in consequence of +that motion. the contracted length being proportional to the expression + +image029 + +This, hypothesis, which is not justifiable by any electrodynamical +facts, supplies us then with that particular law of motion which has +been confirmed with great precision in recent years. + + [13] The general theory of relativity renders it likely that the + electrical masses of an electron are held together by gravitational + forces. + +The theory of relativity leads to the same law of motion, without +requiring any special hypothesis whatsoever as to the structure and the +behaviour of the electron. We arrived at a similar conclusion in +Section XIII in connection with the experiment of Fizeau, the result of +which is foretold by the theory of relativity without the necessity of +drawing on hypotheses as to the physical nature of the liquid. + +The second class of facts to which we have alluded has reference to the +question whether or not the motion of the earth in space can be made +perceptible in terrestrial experiments. We have already remarked in +Section V that all attempts of this nature led to a negative result. +Before the theory of relativity was put forward, it was difficult to +become reconciled to this negative result, for reasons now to be +discussed. The inherited prejudices about time and space did not allow +any doubt to arise as to the prime importance of the Galileian +transformation for changing over from one body of reference to another. +Now assuming that the Maxwell-Lorentz equations hold for a +reference-body _K_, we then find that they do not hold for a +reference-body _K′_ moving uniformly with respect to _K_, if we assume +that the relations of the Galileian transformation exist between the +co-ordinates of _K_ and _K′_. It thus appears that, of all Galileian +co-ordinate systems, one (_K_) corresponding to a particular state of +motion is physically unique. This result was interpreted physically by +regarding _K_ as at rest with respect to a hypothetical æther of space. +On the other hand, all coordinate systems _K′_ moving relatively to _K_ +were to be regarded as in motion with respect to the æther. To this +motion of _K′_ against the æther (“æther-drift” relative to _K′_) were +attributed the more complicated laws which were supposed to hold +relative to _K′_. Strictly speaking, such an æther-drift ought also to +be assumed relative to the earth, and for a long time the efforts of +physicists were devoted to attempts to detect the existence of an +æther-drift at the earth’s surface. + +In one of the most notable of these attempts Michelson devised a method +which appears as though it must be decisive. Imagine two mirrors so +arranged on a rigid body that the reflecting surfaces face each other. +A ray of light requires a perfectly definite time _T_ to pass from one +mirror to the other and back again, if the whole system be at rest with +respect to the æther. It is found by calculation, however, that a +slightly different time _T′_ is required for this process, if the body, +together with the mirrors, be moving relatively to the æther. And yet +another point: it is shown by calculation that for a given velocity _v_ +with reference to the æther, this time _T′_ is different when the body +is moving perpendicularly to the planes of the mirrors from that +resulting when the motion is parallel to these planes. Although the +estimated difference between these two times is exceedingly small, +Michelson and Morley performed an experiment involving interference in +which this difference should have been clearly detectable. But the +experiment gave a negative result—a fact very perplexing to physicists. +Lorentz and FitzGerald rescued the theory from this difficulty by +assuming that the motion of the body relative to the æther produces a +contraction of the body in the direction of motion, the amount of +contraction being just sufficient to compensate for the difference in +time mentioned above. Comparison with the discussion in Section XII +shows that also from the standpoint of the theory of relativity this +solution of the difficulty was the right one. But on the basis of the +theory of relativity the method of interpretation is incomparably more +satisfactory. According to this theory there is no such thing as a +“specially favoured” (unique) co-ordinate system to occasion the +introduction of the æther-idea, and hence there can be no æther-drift, +nor any experiment with which to demonstrate it. Here the contraction +of moving bodies follows from the two fundamental principles of the +theory, without the introduction of particular hypotheses; and as the +prime factor involved in this contraction we find, not the motion in +itself, to which we cannot attach any meaning, but the motion with +respect to the body of reference chosen in the particular case in +point. Thus for a co-ordinate system moving with the earth the mirror +system of Michelson and Morley is not shortened, but it _is_ shortened +for a co-ordinate system which is at rest relatively to the sun. + +The non-mathematician is seized by a mysterious shuddering when he +hears of “four-dimensional” things, by a feeling not unlike that +awakened by thoughts of the occult. And yet there is no more +common-place statement than that the world in which we live is a +four-dimensional space-time continuum. + +Space is a three-dimensional continuum. By this we mean that it is +possible to describe the position of a point (at rest) by means of +three numbers (co-ordinates) _x, y, z_, and that there is an indefinite +number of points in the neighbourhood of this one, the position of +which can be described by co-ordinates such as _x1, y1, z1_, which may +be as near as we choose to the respective values of the co-ordinates +_x, y, z_, of the first point. In virtue of the latter property we +speak of a “continuum,” and owing to the fact that there are three +co-ordinates we speak of it as being “three-dimensional.” + +Similarly, the world of physical phenomena which was briefly called +“world” by Minkowski is naturally four dimensional in the space-time +sense. For it is composed of individual events, each of which is +described by four numbers, namely, three space co-ordinates _x, y, z_, +and a time co-ordinate, the time value _t_. The “world” is in this +sense also a continuum; for to every event there are as many +“neighbouring” events (realised or at least thinkable) as we care to +choose, the co-ordinates _x1, y1, z1, t1_ of which differ by an +indefinitely small amount from those of the event _x, y, z, t_ +originally considered. That we have not been accustomed to regard the +world in this sense as a four-dimensional continuum is due to the fact +that in physics, before the advent of the theory of relativity, time +played a different and more independent rôle, as compared with the +space coordinates. It is for this reason that we have been in the habit +of treating time as an independent continuum. As a matter of fact, +according to classical mechanics, time is absolute, _i.e._ it is +independent of the position and the condition of motion of the system +of co-ordinates. We see this expressed in the last equation of the +Galileian transformation (_t′_ = _t_). + +The four-dimensional mode of consideration of the “world” is natural on +the theory of relativity, since according to this theory time is robbed +of its independence. This is shown by the fourth equation of the +Lorentz transformation: + +image030 + +Moreover, according to this equation the time difference Δ_t′_ of two +events with respect to _K′_ does not in general vanish, even when the +time difference Δ_t_ of the same events with reference to _K_ vanishes. +Pure “space-distance” of two events with respect to _K_ results in +“time-distance ” of the same events with respect to _K_. But the +discovery of Minkowski, which was of importance for the formal +development of the theory of relativity, does not lie here. It is to be +found rather in the fact of his recognition that the four-dimensional +space-time continuum of the theory of relativity, in its most essential +formal properties, shows a pronounced relationship to the +three-dimensional continuum of Euclidean geometrical space.[14] In +order to give due prominence to this relationship, however, we must +replace the usual time co-ordinate t by an imaginary magnitude + +image031 + +proportional to it. Under these conditions, the natural laws satisfying +the demands of the (special) theory of relativity assume mathematical +forms, in which the time co-ordinate plays exactly the same role as the +three space co-ordinates. Formally, these four co-ordinates correspond +exactly to the three space co-ordinates in Euclidean geometry. It must +be clear even to the non-mathematician that, as a consequence of this +purely formal addition to our knowledge, the theory perforce gained +clearness in no mean measure. + + [14] Cf. the somewhat more detailed discussion in Appendix II. + +These inadequate remarks can give the reader only a vague notion of the +important idea contributed by Minkowski. Without it the general theory +of relativity, of which the fundamental ideas are developed in the +following pages, would perhaps have got no farther than its long +clothes. Minkowski’s work is doubtless difficult of access to anyone +inexperienced in mathematics, but since it is not necessary to have a +very exact grasp of this work in order to understand the fundamental +ideas of either the special or the general theory of relativity, I +shall leave it here at present, and revert to it only towards the end +of Part II. + +The basal principle, which was the pivot of all our previous +considerations, was the _special_ principle of relativity, _i.e._ the +principle of the physical relativity of all _uniform_ motion. Let as +once more analyse its meaning carefully. + +It was at all times clear that, from the point of view of the idea it +conveys to us, every motion must be considered only as a relative +motion. Returning to the illustration we have frequently used of the +embankment and the railway carriage, we can express the fact of the +motion here taking place in the following two forms, both of which are +equally justifiable: + +(_a_) The carriage is in motion relative to the embankment, + +(_b_) The embankment is in motion relative to the carriage. + +In (_a_) the embankment, in (_b_) the carriage, serves as the body of +reference in our statement of the motion taking place. If it is simply +a question of detecting or of describing the motion involved, it is in +principle immaterial to what reference-body we refer the motion. As +already mentioned, this is self-evident, but it must not be confused +with the much more comprehensive statement called “the principle of +relativity,” which we have taken as the basis of our investigations. + +The principle we have made use of not only maintains that we may +equally well choose the carriage or the embankment as our +reference-body for the description of any event (for this, too, is +self-evident). Our principle rather asserts what follows: If we +formulate the general laws of nature as they are obtained from +experience, by making use of + +(_a_) the embankment as reference-body, + +(_b_) the railway carriage as reference-body, + +then these general laws of nature (_e.g._ the laws of mechanics or the +law of the propagation of light _in vacuo_) have exactly the same form +in both cases. This can also be expressed as follows: For the physical +description of natural processes, neither of the reference bodies _K, +K′_ is unique (lit. “specially marked out”) as compared with the other. +Unlike the first, this latter statement need not of necessity hold _a +priori;_ it is not contained in the conceptions of “motion” and +“reference-body” and derivable from them; only _experience_ can decide +as to its correctness or incorrectness. + +Up to the present, however, we have by no means maintained the +equivalence of _all_ bodies of reference _K_ in connection with the +formulation of natural laws. Our course was more on the following +Iines. In the first place, we started out from the assumption that +there exists a reference-body _K_, whose condition of motion is such +that the Galileian law holds with respect to it: A particle left to +itself and sufficiently far removed from all other particles moves +uniformly in a straight line. With reference to K (Galileian +reference-body) the laws of nature were to be as simple as possible. +But in addition to K, all bodies of reference _K′_ should be given +preference in this sense, and they should be exactly equivalent to _K_ +for the formulation of natural laws, provided that they are in a state +of _uniform rectilinear and non-rotary motion_ with respect to _K_; all +these bodies of reference are to be regarded as Galileian +reference-bodies. The validity of the principle of relativity was +assumed only for these reference-bodies, but not for others (_e.g._ +those possessing motion of a different kind). In this sense we speak of +the _special_ principle of relativity, or special theory of relativity. + +In contrast to this we wish to understand by the “general principle of +relativity” the following statement: All bodies of reference _K, K′_, +etc., are equivalent for the description of natural phenomena +(formulation of the general laws of nature), whatever may be their +state of motion. But before proceeding farther, it ought to be pointed +out that this formulation must be replaced later by a more abstract +one, for reasons which will become evident at a later stage. + +Since the introduction of the special principle of relativity has been +justified, every intellect which strives after generalisation must feel +the temptation to venture the step towards the general principle of +relativity. But a simple and apparently quite reliable consideration +seems to suggest that, for the present at any rate, there is little +hope of success in such an attempt; Let us imagine ourselves +transferred to our old friend the railway carriage, which is travelling +at a uniform rate. As long as it is moving uniformly, the occupant of +the carriage is not sensible of its motion, and it is for this reason +that he can without reluctance interpret the facts of the case as +indicating that the carriage is at rest, but the embankment in motion. +Moreover, according to the special principle of relativity, this +interpretation is quite justified also from a physical point of view. +If the motion of the carriage is now changed into a non-uniform motion, +as for instance by a powerful application of the brakes, then the +occupant of the carriage experiences a correspondingly powerful jerk +forwards. The retarded motion is manifested in the mechanical behaviour +of bodies relative to the person in the railway carriage. The +mechanical behaviour is different from that of the case previously +considered, and for this reason it would appear to be impossible that +the same mechanical laws hold relatively to the non-uniformly moving +carriage, as hold with reference to the carriage when at rest or in +uniform motion. At all events it is clear that the Galileian law does +not hold with respect to the non-uniformly moving carriage. Because of +this, we feel compelled at the present juncture to grant a kind of +absolute physical reality to non-uniform motion, in opposition to the +general principle of relativity. But in what follows we shall soon see +that this conclusion cannot be maintained. + +“If we pick up a stone and then let it go, why does it fall to the +ground?” The usual answer to this question is: “Because it is attracted +by the earth.” Modern physics formulates the answer rather differently +for the following reason. As a result of the more careful study of +electromagnetic phenomena, we have come to regard action at a distance +as a process impossible without the intervention of some intermediary +medium. If, for instance, a magnet attracts a piece of iron, we cannot +be content to regard this as meaning that the magnet acts directly on +the iron through the intermediate empty space, but we are constrained +to imagine—after the manner of Faraday—that the magnet always calls +into being something physically real in the space around it, that +something being what we call a “magnetic field.” In its turn this +magnetic field operates on the piece of iron, so that the latter +strives to move towards the magnet. We shall not discuss here the +justification for this incidental conception, which is indeed a +somewhat arbitrary one. We shall only mention that with its aid +electromagnetic phenomena can be theoretically represented much more +satisfactorily than without it, and this applies particularly to the +transmission of electromagnetic waves. The effects of gravitation also +are regarded in an analogous manner. + +The action of the earth on the stone takes place indirectly. The earth +produces in its surrounding a gravitational field, which acts on the +stone and produces its motion of fall. As we know from experience, the +intensity of the action on a body dimishes according to a quite +definite law, as we proceed farther and farther away from the earth. +From our point of view this means: The law governing the properties of +the gravitational field in space must be a perfectly definite one, in +order correctly to represent the diminution of gravitational action +with the distance from operative bodies. It is something like this: The +body (_e.g._ the earth) produces a field in its immediate neighbourhood +directly; the intensity and direction of the field at points farther +removed from the body are thence determined by the law which governs +the properties in space of the gravitational fields themselves. + +In contrast to electric and magnetic fields, the gravitational field +exhibits a most remarkable property, which is of fundamental importance +for what follows. Bodies which are moving under the sole influence of a +gravitational field receive an acceleration, _which does not in the +least depend either on the material or on the physical state of the +body._ For instance, a piece of lead and a piece of wood fall in +exactly the same manner in a gravitational field (_in vacuo_), when +they start off from rest or with the same initial velocity. This law, +which holds most accurately, can be expressed in a different form in +the light of the following consideration. + +According to Newton’s law of motion, we have + +(Force) = (inertial mass) x (acceleration), + +where the “inertial mass” is a characteristic constant of the +accelerated body. If now gravitation is the cause of the acceleration, +we then have + +(Force) = (gravitational mass) x (intensity of the gravitational +field), + +where the “gravitational mass” is likewise a characteristic constant +for the body. From these two relations follows: + +image032 + +If now, as we find from experience, the acceleration is to be +independent of the nature and the condition of the body and always the +same for a given gravitational field, then the ratio of the +gravitational to the inertial mass must likewise be the same for all +bodies. By a suitable choice of units we can thus make this ratio equal +to unity. We then have the following law: The _gravitational_ mass of a +body is equal to its _inertial_ mass. + +It is true that this important law had hitherto been recorded in +mechanics, but it had not been _interpreted_. A satisfactory +interpretation can be obtained only if we recognise the following fact: +_The same_ quality of a body manifests itself according to +circumstances as “inertia” or as “weight” (lit. “heaviness”). In the +following section we shall show to what extent this is actually the +case, and how this question is connected with the general postulate of +relativity. + +We imagine a large portion of empty space, so far removed from stars +and other appreciable masses, that we have before us approximately the +conditions required by the fundamental law of Galilei. It is then +possible to choose a Galileian reference-body for this part of space +(world), relative to which points at rest remain at rest and points in +motion continue permanently in uniform rectilinear motion. As +reference-body let us imagine a spacious chest resembling a room with +an observer inside who is equipped with apparatus. Gravitation +naturally does not exist for this observer. He must fasten himself with +strings to the floor, otherwise the slightest impact against the floor +will cause him to rise slowly towards the ceiling of the room. + +To the middle of the lid of the chest is fixed externally a hook with +rope attached, and now a “being” (what kind of a being is immaterial to +us) begins pulling at this with a constant force. The chest together +with the observer then begin to move “upwards” with a uniformly +accelerated motion. In course of time their velocity will reach +unheard-of values—provided that we are viewing all this from another +reference-body which is not being pulled with a rope. + +But how does the man in the chest regard the Process? The acceleration +of the chest will be transmitted to him by the reaction of the floor of +the chest. He must therefore take up this pressure by means of his legs +if he does not wish to be laid out full length on the floor. He is then +standing in the chest in exactly the same way as anyone stands in a +room of a home on our earth. If he releases a body which he previously +had in his land, the accelertion of the chest will no longer be +transmitted to this body, and for this reason the body will approach +the floor of the chest with an accelerated relative motion. The +observer will further convince himself _that the acceleration of the +body towards the floor of the chest is always of the same magnitude, +whatever kind of body he may happen to use for the experiment._ + +Relying on his knowledge of the gravitational field (as it was +discussed in the preceding section), the man in the chest will thus +come to the conclusion that he and the chest are in a gravitational +field which is constant with regard to time. Of course he will be +puzzled for a moment as to why the chest does not fall in this +gravitational field. just then, however, he discovers the hook in the +middle of the lid of the chest and the rope which is attached to it, +and he consequently comes to the conclusion that the chest is suspended +at rest in the gravitational field. + +Ought we to smile at the man and say that he errs in his conclusion? I +do not believe we ought to if we wish to remain consistent; we must +rather admit that his mode of grasping the situation violates neither +reason nor known mechanical laws. Even though it is being accelerated +with respect to the “Galileian space” first considered, we can +nevertheless regard the chest as being at rest. We have thus good +grounds for extending the principle of relativity to include bodies of +reference which are accelerated with respect to each other, and as a +result we have gained a powerful argument for a generalised postulate +of relativity. + +We must note carefully that the possibility of this mode of +interpretation rests on the fundamental property of the gravitational +field of giving all bodies the same acceleration, or, what comes to the +same thing, on the law of the equality of inertial and gravitational +mass. If this natural law did not exist, the man in the accelerated +chest would not be able to interpret the behaviour of the bodies around +him on the supposition of a gravitational field, and he would not be +justified on the grounds of experience in supposing his reference-body +to be “at rest.” + +Suppose that the man in the chest fixes a rope to the inner side of the +lid, and that he attaches a body to the free end of the rope. The +result of this will be to stretch the rope so that it will hang +“vertically” downwards. If we ask for an opinion of the cause of +tension in the rope, the man in the chest will say: “The suspended body +experiences a downward force in the gravitational field, and this is +neutralised by the tension of the rope; what determines the magnitude +of the tension of the rope is the _gravitational mass_ of the suspended +body.” On the other hand, an observer who is poised freely in space +will interpret the condition of things thus: “The rope must perforce +take part in the accelerated motion of the chest, and it transmits this +motion to the body attached to it. The tension of the rope is just +large enough to effect the acceleration of the body. That which +determines the magnitude of the tension of the rope is the _inertial +mass_ of the body.” Guided by this example, we see that our extension +of the principle of relativity implies the _necessity_ of the law of +the equality of inertial and gravitational mass. Thus we have obtained +a physical interpretation of this law. + +From our consideration of the accelerated chest we see that a general +theory of relativity must yield important results on the laws of +gravitation. In point of fact, the systematic pursuit of the general +idea of relativity has supplied the laws satisfied by the gravitational +field. Before proceeding farther, however, I must warn the reader +against a misconception suggested by these considerations. A +gravitational field exists for the man in the chest, despite the fact +that there was no such field for the co-ordinate system first chosen. +Now we might easily suppose that the existence of a gravitational field +is always only an _apparent_ one. We might also think that, regardless +of the kind of gravitational field which may be present, we could +always choose another reference-body such that _no_ gravitational field +exists with reference to it. This is by no means true for all +gravitational fields, but only for those of quite special form. It is, +for instance, impossible to choose a body of reference such that, as +judged from it, the gravitational field of the earth (in its entirety) +vanishes. + +We can now appreciate why that argument is not convincing, which we +brought forward against the general principle of relativity at the end +of Section XVIII. It is certainly true that the observer in the railway +carriage experiences a jerk forwards as a result of the application of +the brake, and that he recognises, in this the non-uniformity of motion +(retardation) of the carriage. But he is compelled by nobody to refer +this jerk to a “real” acceleration (retardation) of the carriage. He +might also interpret his experience thus: “My body of reference (the +carriage) remains permanently at rest. With reference to it, however, +there exists (during the period of application of the brakes) a +gravitational field which is directed forwards and which is variable +with respect to time. Under the influence of this field, the embankment +together with the earth moves non-uniformly in such a manner that their +original velocity in the backwards direction is continuously reduced.” + +We have already stated several times that classical mechanics starts +out from the following law: Material particles sufficiently far removed +from other material particles continue to move uniformly in a straight +line or continue in a state of rest. We have also repeatedly emphasised +that this fundamental law can only be valid for bodies of reference _K_ +which possess certain unique states of motion, and which are in uniform +translational motion relative to each other. Relative to other +reference-bodies _K_ the law is not valid. Both in classical mechanics +and in the special theory of relativity we therefore differentiate +between reference-bodies _K_ relative to which the recognised “laws of +nature” can be said to hold, and reference-bodies _K_ relative to which +these laws do not hold. + +But no person whose mode of thought is logical can rest satisfied with +this condition of things. He asks: “How does it come that certain +reference-bodies (or their states of motion) are given priority over +other reference-bodies (or their states of motion)? _What is the reason +for this preference?_” In order to show clearly what I mean by this +question, I shall make use of a comparison. + +I am standing in front of a gas range. Standing alongside of each other +on the range are two pans so much alike that one may be mistaken for +the other. Both are half full of water. I notice that steam is being +emitted continuously from the one pan, but not from the other. I am +surprised at this, even if I have never seen either a gas range or a +pan before. But if I now notice a luminous something of bluish colour +under the first pan but not under the other, I cease to be astonished, +even if I have never before seen a gas flame. For I can only say that +this bluish something will cause the emission of the steam, or at least +_possibly_ it may do so. If, however, I notice the bluish something in +neither case, and if I observe that the one continuously emits steam +whilst the other does not, then I shall remain astonished and +dissatisfied until I have discovered some circumstance to which I can +attribute the different behaviour of the two pans. + +Analogously, I seek in vain for a real something in classical mechanics +(or in the special theory of relativity) to which I can attribute the +different behaviour of bodies considered with respect to the reference +systems _K_ and _K′_.[15] Newton saw this objection and attempted to +invalidate it, but without success. But E. Mach recognised it most +clearly of all, and because of this objection he claimed that mechanics +must be placed on a new basis. It can only be got rid of by means of a +physics which is conformable to the general principle of relativity, +since the equations of such a theory hold for every body of reference, +whatever may be its state of motion. + + [15] The objection is of importance more especially when the state of + motion of the reference-body is of such a nature that it does not + require any external agency for its maintenance, _e.g._ in the case + when the reference-body is rotating uniformly. + +The considerations of Section XX show that the general principle of +relativity puts us in a position to derive properties of the +gravitational field in a purely theoretical manner. Let us suppose, for +instance, that we know the space-time “course” for any natural process +whatsoever, as regards the manner in which it takes place in the +Galileian domain relative to a Galileian body of reference _K_. By +means of purely theoretical operations (_i.e._ simply by calculation) +we are then able to find how this known natural process appears, as +seen from a reference-body _K′_ which is accelerated relatively to _K_. +But since a gravitational field exists with respect to this new body of +reference _K′_, our consideration also teaches us how the gravitational +field influences the process studied. + +For example, we learn that a body which is in a state of uniform +rectilinear motion with respect to _K_ (in accordance with the law of +Galilei) is executing an accelerated and in general curvilinear motion +with respect to the accelerated reference-body _K′_ (chest). This +acceleration or curvature corresponds to the influence on the moving +body of the gravitational field prevailing relatively to _K_. It is +known that a gravitational field influences the movement of bodies in +this way, so that our consideration supplies us with nothing +essentially new. + +However, we obtain a new result of fundamental importance when we carry +out the analogous consideration for a ray of light. With respect to the +Galileian reference-body _K_, such a ray of light is transmitted +rectilinearly with the velocity _c_. It can easily be shown that the +path of the same ray of light is no longer a straight line when we +consider it with reference to the accelerated chest (reference-body +_K′_). From this we conclude, _that, in general, rays of light are +propagated curvilinearly in gravitational fields._ In two respects this +result is of great importance. + +In the first place, it can be compared with the reality. Although a +detailed examination of the question shows that the curvature of light +rays required by the general theory of relativity is only exceedingly +small for the gravitational fields at our disposal in practice, its +estimated magnitude for light rays passing the sun at grazing incidence +is nevertheless 1.7 seconds of arc. This ought to manifest itself in +the following way. As seen from the earth, certain fixed stars appear +to be in the neighbourhood of the sun, and are thus capable of +observation during a total eclipse of the sun. At such times, these +stars ought to appear to be displaced outwards from the sun by an +amount indicated above, as compared with their apparent position in the +sky when the sun is situated at another part of the heavens. The +examination of the correctness or otherwise of this deduction is a +problem of the greatest importance, the early solution of which is to +be expected of astronomers.[16] + + [16] By means of the star photographs of two expeditions equipped by a + Joint Committee of the Royal and Royal Astronomical Societies, the + existence of the deflection of light demanded by theory was first + confirmed during the solar eclipse of 29th May, 1919. (Cf. Appendix + III.) + +In the second place our result shows that, according to the general +theory of relativity, the law of the constancy of the velocity of light +in vacuo, which constitutes one of the two fundamental assumptions in +the special theory of relativity and to which we have already +frequently referred, cannot claim any unlimited validity. A curvature +of rays of light can only take place when the velocity of propagation +of light varies with position. Now we might think that as a consequence +of this, the special theory of relativity and with it the whole theory +of relativity would be laid in the dust. But in reality this is not the +case. We can only conclude that the special theory of relativity cannot +claim an unlimited domain of validity; its results hold only so long as +we are able to disregard the influences of gravitational fields on the +phenomena (_e.g._ of light). + +Since it has often been contended by opponents of the theory of +relativity that the special theory of relativity is overthrown by the +general theory of relativity, it is perhaps advisable to make the facts +of the case clearer by means of an appropriate comparison. Before the +development of electrodynamics the laws of electrostatics were looked +upon as the laws of electricity. At the present time we know that +electric fields can be derived correctly from electrostatic +considerations only for the case, which is never strictly realised, in +which the electrical masses are quite at rest relatively to each other, +and to the co-ordinate system. Should we be justified in saying that +for this reason electrostatics is overthrown by the field-equations of +Maxwell in electrodynamics? Not in the least. Electrostatics is +contained in electrodynamics as a limiting case; the laws of the latter +lead directly to those of the former for the case in which the fields +are invariable with regard to time. No fairer destiny could be allotted +to any physical theory, than that it should of itself point out the way +to the introduction of a more comprehensive theory, in which it lives +on as a limiting case. + +In the example of the transmission of light just dealt with, we have +seen that the general theory of relativity enables us to derive +theoretically the influence of a gravitational field on the course of +natural processes, the laws of which are already known when a +gravitational field is absent. But the most attractive problem, to the +solution of which the general theory of relativity supplies the key, +concerns the investigation of the laws satisfied by the gravitational +field itself. Let us consider this for a moment. + +We are acquainted with space-time domains which behave (approximately) +in a “Galileian” fashion under suitable choice of reference-body, +_i.e._ domains in which gravitational fields are absent. If we now +refer such a domain to a reference-body _K′_ possessing any kind of +motion, then relative to _K′_ there exists a gravitational field which +is variable with respect to space and time.[17] The character of this +field will of course depend on the motion chosen for _K′._ According to +the general theory of relativity, the general law of the gravitational +field must be satisfied for all gravitational fields obtainable in this +way. Even though by no means all gravitationial fields can be produced +in this way, yet we may entertain the hope that the general law of +gravitation will be derivable from such gravitational fields of a +special kind. This hope has been realised in the most beautiful manner. +But between the clear vision of this goal and its actual realisation it +was necessary to surmount a serious difficulty, and as this lies deep +at the root of things, I dare not withhold it from the reader. We +require to extend our ideas of the space-time continuum still farther. + + [17] This follows from a generalisation of the discussion in Section + XX. + +Hitherto I have purposely refrained from speaking about the physical +interpretation of space- and time-data in the case of the general +theory of relativity. As a consequence, I am guilty of a certain +slovenliness of treatment, which, as we know from the special theory of +relativity, is far from being unimportant and pardonable. It is now +high time that we remedy this defect; but I would mention at the +outset, that this matter lays no small claims on the patience and on +the power of abstraction of the reader. + +We start off again from quite special cases, which we have frequently +used before. Let us consider a space time domain in which no +gravitational field exists relative to a reference-body _K_ whose state +of motion has been suitably chosen. _K_ is then a Galileian +reference-body as regards the domain considered, and the results of the +special theory of relativity hold relative to _K_. Let us suppose the +same domain referred to a second body of reference _K′_, which is +rotating uniformly with respect to _K_. In order to fix our ideas, we +shall imagine _K′_ to be in the form of a plane circular disc, which +rotates uniformly in its own plane about its centre. An observer who is +sitting eccentrically on the disc _K′_ is sensible of a force which +acts outwards in a radial direction, and which would be interpreted as +an effect of inertia (centrifugal force) by an observer who was at rest +with respect to the original reference-body _K_. But the observer on +the disc may regard his disc as a reference-body which is “at rest”; on +the basis of the general principle of relativity he is justified in +doing this. The force acting on himself, and in fact on all other +bodies which are at rest relative to the disc, he regards as the effect +of a gravitational field. Nevertheless, the space-distribution of this +gravitational field is of a kind that would not be possible on Newton’s +theory of gravitation.[18] But since the observer believes in the +general theory of relativity, this does not disturb him; he is quite in +the right when he believes that a general law of gravitation can be +formulated—a law which not only explains the motion of the stars +correctly, but also the field of force experienced by himself. + + [18] The field disappears at the centre of the disc and increases + proportionally to the distance from the centre as we proceed outwards. + +The observer performs experiments on his circular disc with clocks and +measuring-rods. In doing so, it is his intention to arrive at exact +definitions for the signification of time- and space-data with +reference to the circular disc _K′_, these definitions being based on +his observations. What will be his experience in this enterprise? + +To start with, he places one of two identically constructed clocks at +the centre of the circular disc, and the other on the edge of the disc, +so that they are at rest relative to it. We now ask ourselves whether +both clocks go at the same rate from the standpoint of the non-rotating +Galileian reference-body _K_. As judged from this body, the clock at +the centre of the disc has no velocity, whereas the clock at the edge +of the disc is in motion relative to _K_ in consequence of the +rotation. According to a result obtained in Section XII, it follows +that the latter clock goes at a rate permanently slower than that of +the clock at the centre of the circular disc, _i.e._ as observed from +_K_. It is obvious that the same effect would be noted by an observer +whom we will imagine sitting alongside his clock at the centre of the +circular disc. Thus on our circular disc, or, to make the case more +general, in every gravitational field, a clock will go more quickly or +less quickly, according to the position in which the clock is situated +(at rest). For this reason it is not possible to obtain a reasonable +definition of time with the aid of clocks which are arranged at rest +with respect to the body of reference. A similar difficulty presents +itself when we attempt to apply our earlier definition of simultaneity +in such a case, but I do not wish to go any farther into this question. + +Moreover, at this stage the definition of the space co-ordinates also +presents insurmountable difficulties. If the observer applies his +standard measuring-rod (a rod which is short as compared with the +radius of the disc) tangentially to the edge of the disc, then, as +judged from the Galileian system, the length of this rod will be less +than 1, since, according to Section XII, moving bodies suffer a +shortening in the direction of the motion. On the other hand, the +measuring-rod will not experience a shortening in length, as judged +from _K_, if it is applied to the disc in the direction of the radius. +If, then, the observer first measures the circumference of the disc +with his measuring-rod and then the diameter of the disc, on dividing +the one by the other, he will not obtain as quotient the familiar +number π = 3.14 . . ., but a larger number,[19] whereas of course, for +a disc which is at rest with respect to _K_, this operation would yield +π exactly. This proves that the propositions of Euclidean geometry +cannot hold exactly on the rotating disc, nor in general in a +gravitational field, at least if we attribute the length 1 to the rod +in all positions and in every orientation. Hence the idea of a straight +line also loses its meaning. We are therefore not in a position to +define exactly the co-ordinates _x, y, z_ relative to the disc by means +of the method used in discussing the special theory, and as long as the +co-ordinates and times of events have not been defined, we cannot +assign an exact meaning to the natural laws in which these occur. + + [19] Throughout this consideration we have to use the Galileian + (non-rotating) system _K_ as reference-body, since we may only assume + the validity of the results of the special theory of relativity + relative to _K_ (relative to _K′_ a gravitational field prevails). + +Thus all our previous conclusions based on general relativity would +appear to be called in question. In reality we must make a subtle +detour in order to be able to apply the postulate of general relativity +exactly. I shall prepare the reader for this in the following +paragraphs. + +The surface of a marble table is spread out in front of me. I can get +from any one point on this table to any other point by passing +continuously from one point to a “neighbouring” one, and repeating this +process a (large) number of times, or, in other words, by going from +point to point without executing “jumps.” I am sure the reader will +appreciate with sufficient clearness what I mean here by “neighbouring” +and by “jumps” (if he is not too pedantic). We express this property of +the surface by describing the latter as a continuum. + +Let us now imagine that a large number of little rods of equal length +have been made, their lengths being small compared with the dimensions +of the marble slab. When I say they are of equal length, I mean that +one can be laid on any other without the ends overlapping. We next lay +four of these little rods on the marble slab so that they constitute a +quadrilateral figure (a square), the diagonals of which are equally +long. To ensure the equality of the diagonals, we make use of a little +testing-rod. To this square we add similar ones, each of which has one +rod in common with the first. We proceed in like manner with each of +these squares until finally the whole marble slab is laid out with +squares. The arrangement is such, that each side of a square belongs to +two squares and each corner to four squares. + +It is a veritable wonder that we can carry out this business without +getting into the greatest difficulties. We only need to think of the +following. If at any moment three squares meet at a corner, then two +sides of the fourth square are already laid, and, as a consequence, the +arrangement of the remaining two sides of the square is already +completely determined. But I am now no longer able to adjust the +quadrilateral so that its diagonals may be equal. If they are equal of +their own accord, then this is an especial favour of the marble slab +and of the little rods, about which I can only be thankfully surprised. +We must experience many such surprises if the construction is to be +successful. + +If everything has really gone smoothly, then I say that the points of +the marble slab constitute a Euclidean continuum with respect to the +little rod, which has been used as a “distance” (line-interval). By +choosing one corner of a square as “origin” I can characterise every +other corner of a square with reference to this origin by means of two +numbers. I only need state how many rods I must pass over when, +starting from the origin, I proceed towards the “right” and then +“upwards,” in order to arrive at the corner of the square under +consideration. These two numbers are then the “Cartesian co-ordinates” +of this corner with reference to the “Cartesian co-ordinate system” +which is determined by the arrangement of little rods. + +By making use of the following modification of this abstract +experiment, we recognise that there must also be cases in which the +experiment would be unsuccessful. We shall suppose that the rods +“expand” by in amount proportional to the increase of temperature. We +heat the central part of the marble slab, but not the periphery, in +which case two of our little rods can still be brought into coincidence +at every position on the table. But our construction of squares must +necessarily come into disorder during the heating, because the little +rods on the central region of the table expand, whereas those on the +outer part do not. + +With reference to our little rods—defined as unit lengths—the marble +slab is no longer a Euclidean continuum, and we are also no longer in +the position of defining Cartesian co-ordinates directly with their +aid, since the above construction can no longer be carried out. But +since there are other things which are not influenced in a similar +manner to the little rods (or perhaps not at all) by the temperature of +the table, it is possible quite naturally to maintain the point of view +that the marble slab is a “Euclidean continuum.” This can be done in a +satisfactory manner by making a more subtle stipulation about the +measurement or the comparison of lengths. + +But if rods of every kind (_i.e._ of every material) were to behave _in +the same way_ as regards the influence of temperature when they are on +the variably heated marble slab, and if we had no other means of +detecting the effect of temperature than the geometrical behaviour of +our rods in experiments analogous to the one described above, then our +best plan would be to assign the distance one to two points on the +slab, provided that the ends of one of our rods could be made to +coincide with these two points; for how else should we define the +distance without our proceeding being in the highest measure grossly +arbitrary? The method of Cartesian coordinates must then be discarded, +and replaced by another which does not assume the validity of Euclidean +geometry for rigid bodies.[20] The reader will notice that the +situation depicted here corresponds to the one brought about by the +general postulate of relativity (Section XXIII). + + [20] Mathematicians have been confronted with our problem in the + following form. If we are given a surface (_e.g._ an ellipsoid) in + Euclidean three-dimensional space, then there exists for this surface + a two-dimensional geometry, just as much as for a plane surface. Gauss + undertook the task of treating this two-dimensional geometry from + first principles, without making use of the fact that the surface + belongs to a Euclidean continuum of three dimensions. If we imagine + constructions to be made with rigid rods _in the surface_ (similar to + that above with the marble slab), we should find that different laws + hold for these from those resulting on the basis of Euclidean plane + geometry. The surface is not a Euclidean continuum with respect to the + rods, and we cannot define Cartesian co-ordinates _in the surface_. + Gauss indicated the principles according to which we can treat the + geometrical relationships in the surface, and thus pointed out the way + to the method of Riemann of treating multi-dimensional, non-Euclidean + _continuum_. Thus it is that mathematicians long ago solved the formal + problems to which we are led by the general postulate of relativity. + +image033 + +According to Gauss, this combined analytical and geometrical mode of +handling the problem can be arrived at in the following way. We imagine +a system of arbitrary curves (see Fig. 4) drawn on the surface of the +table. These we designate as _u_-curves, and we indicate each of them +by means of a number. The Curves _u_ = 1, _u_ = 2 and _u_ = 3 are drawn +in the diagram. Between the curves _u_ = 1 and _u_ = 2 we must imagine +an infinitely large number to be drawn, all of which correspond to real +numbers lying between 1 and 2. We have then a system of _u_-curves, and +this “infinitely dense” system covers the whole surface of the table. +These _u_-curves must not intersect each other, and through each point +of the surface one and only one curve must pass. Thus a perfectly +definite value of _u_ belongs to every point on the surface of the +marble slab. In like manner we imagine a system of _v_-curves drawn on +the surface. These satisfy the same conditions as the _u_-curves, they +are provided with numbers in a corresponding manner, and they may +likewise be of arbitrary shape. It follows that a value of _u_ and a +value of _v_ belong to every point on the surface of the table. We call +these two numbers the co-ordinates of the surface of the table +(Gaussian co-ordinates). For example, the point _P_ in the diagram has +the Gaussian co-ordinates _u_ = 3, _v_ = 1. Two neighbouring points _P_ +and _P′_ on the surface then correspond to the co-ordinates + +_P_: _u, v_ + +_P′_: _u_ + _du, v_ + _dv_, + +where _du_ and _dv_ signify very small numbers. In a similar manner we +may indicate the distance (line-interval) between _P_ and _P′_, as +measured with a little rod, by means of the very small number _ds_. +Then according to Gauss we have + +_ds_2 = _g_11_du_2 + 2_g_12_du dv_ + _g_22_dv_2, + +where _g_11, _g_12, _g_22, are magnitudes which depend in a perfectly +definite way on _u_ and _v_. The magnitudes _g_11, _g_12 and _g_22, +determine the behaviour of the rods relative to the _u_-curves and +_v_-curves, and thus also relative to the surface of the table. For the +case in which the points of the surface considered form a Euclidean +continuum with reference to the measuring-rods, but only in this case, +it is possible to draw the _u_-curves and _v_-curves and to attach +numbers to them, in such a manner, that we simply have: + +_ds_2 = _du_2 + _dv_2 + +Under these conditions, the _u_-curves and _v_-curves are straight +lines in the sense of Euclidean geometry, and they are perpendicular to +each other. Here the Gaussian coordinates are simply Cartesian ones. It +is clear that Gauss co-ordinates are nothing more than an association +of two sets of numbers with the points of the surface considered, of +such a nature that numerical values differing very slightly from each +other are associated with neighbouring points “in space.” + +So far, these considerations hold for a continuum of two dimensions. +But the Gaussian method can be applied also to a continuum of three, +four or more dimensions. If, for instance, a continuum of four +dimensions be supposed available, we may represent it in the following +way. With every point of the continuum, we associate arbitrarily four +numbers, _x_1, _x_2, _x_3, _x_4, which are known as “co-ordinates.” +Adjacent points correspond to adjacent values of the coordinates. If a +distance _ds_ is associated with the adjacent points _P_ and _P′_, this +distance being measurable and well defined from a physical point of +view, then the following formula holds: + +_ds_2 = _g_11_dx_12 + 2_g_12_dx_1_dx_2 . . . . + _g_44_dx_42, + +where the magnitudes _g_11, etc., have values which vary with the +position in the continuum. Only when the continuum is a Euclidean one +is it possible to associate the co-ordinates _x_1 . . _x_4. with the +points of the continuum so that we have simply + +_ds_2 = _dx_12 + _dx_22 + _dx_32 + _dx_42. + +In this case relations hold in the four-dimensional continuum which are +analogous to those holding in our three-dimensional measurements. + +However, the Gauss treatment for _ds_2 which we have given above is not +always possible. It is only possible when sufficiently small regions of +the continuum under consideration may be regarded as Euclidean +continua. For example, this obviously holds in the case of the marble +slab of the table and local variation of temperature. The temperature +is practically constant for a small part of the slab, and thus the +geometrical behaviour of the rods is _almost_ as it ought to be +according to the rules of Euclidean geometry. Hence the imperfections +of the construction of squares in the previous section do not show +themselves clearly until this construction is extended over a +considerable portion of the surface of the table. + +We can sum this up as follows: Gauss invented a method for the +mathematical treatment of continua in general, in which +“size-relations” (“distances” between neighbouring points) are defined. +To every point of a continuum are assigned as many numbers (Gaussian +coordinates) as the continuum has dimensions. This is done in such a +way, that only one meaning can be attached to the assignment, and that +numbers (Gaussian coordinates) which differ by an indefinitely small +amount are assigned to adjacent points. The Gaussian coordinate system +is a logical generalisation of the Cartesian co-ordinate system. It is +also applicable to non-Euclidean continua, but only when, with respect +to the defined “size” or “distance,” small parts of the continuum under +consideration behave more nearly like a Euclidean system, the smaller +the part of the continuum under our notice. + +We are now in a position to formulate more exactly the idea of +Minkowski, which was only vaguely indicated in Section XVII. In +accordance with the special theory of relativity, certain co-ordinate +systems are given preference for the description of the +four-dimensional, space-time continuum. We called these “Galileian +co-ordinate systems.” For these systems, the four co-ordinates _x, y, +z, t_, which determine an event or—in other words—a point of the +four-dimensional continuum, are defined physically in a simple manner, +as set forth in detail in the first part of this book. For the +transition from one Galileian system to another, which is moving +uniformly with reference to the first, the equations of the Lorentz +transformation are valid. These last form the basis for the derivation +of deductions from the special theory of relativity, and in themselves +they are nothing more than the expression of the universal validity of +the law of transmission of light for all Galileian systems of +reference. + +Minkowski found that the Lorentz transformations satisfy the following +simple conditions. Let us consider two neighbouring events, the +relative position of which in the four-dimensional continuum is given +with respect to a Galileian reference-body _K_ by the space co-ordinate +differences _dx, dy, dz_ and the time-difference _dt_. With reference +to a second Galileian system we shall suppose that the corresponding +differences for these two events are _dx′, dy′, dz′, dt′_. Then these +magnitudes always fulfill the condition.[21] + + [21] Cf. Appendixes I and II. The relations which are derived there + for the co-ordinates themselves are valid also for co-ordinate + _differences_, and thus also for co-ordinate differentials + (indefinitely small differences). + +_dx_2 + _dy_2 + _dz_2 – _c_2_dt_2 = _dx′_2 + _dy′_2 + _dz′_2 – +_c_2_dt′_2. + +The validity of the Lorentz transformation follows from this condition. +We can express this as follows: The magnitude + +_ds_2 = _dx_2 + _dy_2 + _dz_2 – _c_2 _dt_2, + +which belongs to two adjacent points of the four-dimensional space-time +continuum, has the same value for all selected (Galileian) +reference-bodies. If we replace _x, y, z_, + +image034 + +by _x_1, _x_2, _x_3, _x_4, we also obtain the result that + +_ds_2 = _dx_12 + _dx_22 + _dx_32 + _dx_42. + +is independent of the choice of the body of reference. We call the +magnitude _ds_ the “distance” apart of the two events or +four-dimensional points. + +Thus, if we choose as time-variable the imaginary variable + +image035 + +instead of the real quantity _t_, we can regard the space-time +contintium—accordance with the special theory of relativity—as a +“Euclidean” four-dimensional continuum, a result which follows from the +considerations of the preceding section. + +In the first part of this book we were able to make use of space-time +co-ordinates which allowed of a simple and direct physical +interpretation, and which, according to Section XXVI, can be regarded +as four-dimensional Cartesian co-ordinates. This was possible on the +basis of the law of the constancy of the velocity of light. But +according to Section XXI the general theory of relativity cannot retain +this law. On the contrary, we arrived at the result that according to +this latter theory the velocity of light must always depend on the +co-ordinates when a gravitational field is present. In connection with +a specific illustration in Section XXIII, we found that the presence of +a gravitational field invalidates the definition of the coordinates and +the time, which led us to our objective in the special theory of +relativity. + +In view of the resuIts of these considerations we are led to the +conviction that, according to the general principle of relativity, the +space-time continuum cannot be regarded as a Euclidean one, but that +here we have the general case, corresponding to the marble slab with +local variations of temperature, and with which we made acquaintance as +an example of a two-dimensional continuum. Just as it was there +impossible to construct a Cartesian co-ordinate system from equal rods, +so here it is impossible to build up a system (reference-body) from +rigid bodies and clocks, which shall be of such a nature that +measuring-rods and clocks, arranged rigidly with respect to one +another, shall indicate position and time directly. Such was the +essence of the difficulty with which we were confronted in Section +XXIII. + +But the considerations of Sections XXV and XXVI show us the way to +surmount this difficulty. We refer the four-dimensional space-time +continuum in an arbitrary manner to Gauss co-ordinates. We assign to +every point of the continuum (event) four numbers, _x_1, _x_2, _x_3, +_x_4 (co-ordinates), which have not the least direct physical +significance, but only serve the purpose of numbering the points of the +continuum in a definite but arbitrary manner. This arrangement does not +even need to be of such a kind that we must regard _x_1, _x_2, _x_3, as +“space” co-ordinates and _x_4, as a “time” co-ordinate. + +The reader may think that such a description of the world would be +quite inadequate. What does it mean to assign to an event the +particular co-ordinates _x_1, _x_2, _x_3, _x_4, if in themselves these +co-ordinates have no significance? More careful consideration shows, +however, that this anxiety is unfounded. Let us consider, for instance, +a material point with any kind of motion. If this point had only a +momentary existence without duration, then it would to described in +space-time by a single system of values _x_1, _x_2, _x_3, _x_4. Thus +its permanent existence must be characterised by an infinitely large +number of such systems of values, the co-ordinate values of which are +so close together as to give continuity; corresponding to the material +point, we thus have a (uni-dimensional) line in the four-dimensional +continuum. In the same way, any such lines in our continuum correspond +to many points in motion. The only statements having regard to these +points which can claim a physical existence are in reality the +statements about their encounters. In our mathematical treatment, such +an encounter is expressed in the fact that the two lines which +represent the motions of the points in question have a particular +system of co-ordinate values, _x_1, _x_2, _x_3, _x_4, in common. After +mature consideration the reader will doubtless admit that in reality +such encounters constitute the only actual evidence of a time-space +nature with which we meet in physical statements. + +When we were describing the motion of a material point relative to a +body of reference, we stated nothing more than the encounters of this +point with particular points of the reference-body. We can also +determine the corresponding values of the time by the observation of +encounters of the body with clocks, in conjunction with the observation +of the encounter of the hands of clocks with particular points on the +dials. It is just the same in the case of space-measurements by means +of measuring-rods, as a little consideration will show. + +The following statements hold generally: Every physical description +resolves itself into a number of statements, each of which refers to +the space-time coincidence of two events _A_ and _B_. In terms of +Gaussian co-ordinates, every such statement is expressed by the +agreement of their four co-ordinates _x_1, _x_2, _x_3, _x_4. Thus in +reality, the description of the time-space continuum by means of Gauss +co-ordinates completely replaces the description with the aid of a body +of reference, without suffering from the defects of the latter mode of +description; it is not tied down to the Euclidean character of the +continuum which has to be represented. + +We are now in a position to replace the provisional formulation of the +general principle of relativity given in Section XVIII by an exact +formulation. The form there used, “All bodies of reference _K, K′_, +etc., are equivalent for the description of natural phenomena +(formulation of the general laws of nature), whatever may be their +state of motion,” cannot be maintained, because the use of rigid +reference-bodies, in the sense of the method followed in the special +theory of relativity, is in general not possible in space-time +description. The Gauss co-ordinate system has to take the place of the +body of reference. The following statement corresponds to the +fundamental idea of the general principle of relativity: “_All Gaussian +co-ordinate systems are essentially equivalent for the formulation of +the general laws of nature._” + +We can state this general principle of relativity in still another +form, which renders it yet more clearly intelligible than it is when in +the form of the natural extension of the special principle of +relativity. According to the special theory of relativity, the +equations which express the general laws of nature pass over into +equations of the same form when, by making use of the Lorentz +transformation, we replace the space-time variables _x, y, z, t_, of a +(Galileian) reference-body _K_ by the space-time variables _x′, y′, z′, +t′_, of a new reference-body _K′_. According to the general theory of +relativity, on the other hand, by application of _arbitrary +substitutions_ of the Gauss variables _x_1, _x_2, _x_3, _x_4, the +equations must pass over into equations of the same form; for every +transformation (not only the Lorentz transformation) corresponds to the +transition of one Gauss co-ordinate system into another. + +If we desire to adhere to our “old-time” three-dimensional view of +things, then we can characterise the development which is being +undergone by the fundamental idea of the general theory of relativity +as follows: The special theory of relativity has reference to Galileian +domains, _i.e._ to those in which no gravitational field exists. In +this connection a Galileian reference-body serves as body of reference, +_i.e._ a rigid body the state of motion of which is so chosen that the +Galileian law of the uniform rectilinear motion of “isolated” material +points holds relatively to it. + +Certain considerations suggest that we should refer the same Galileian +domains to _non-Galileian_ reference-bodies also. A gravitational field +of a special kind is then present with respect to these bodies (cf. +Sections XX and XXIII). + +In gravitational fields there are no such things as rigid bodies with +Euclidean properties; thus the fictitious rigid body of reference is of +no avail in the general theory of relativity. The motion of clocks is +also influenced by gravitational fields, and in such a way that a +physical definition of time which is made directly with the aid of +clocks has by no means the same degree of plausibility as in the +special theory of relativity. + +For this reason non-rigid reference-bodies are used, which are as a +whole not only moving in any way whatsoever, but which also suffer +alterations in form _ad lib._ during their motion. Clocks, for which +the law of motion is of any kind, however irregular, serve for the +definition of time. We have to imagine each of these clocks fixed at a +point on the non-rigid reference-body. These clocks satisfy only the +one condition, that the “readings” which are observed simultaneously on +adjacent clocks (in space) differ from each other by an indefinitely +small amount. This non-rigid reference-body, which might appropriately +be termed a “reference-mollusc”, is in the main equivalent to a +Gaussian four-dimensional co-ordinate system chosen arbitrarily. That +which gives the “mollusc” a certain comprehensibility as compared with +the Gauss co-ordinate system is the (really unjustified) formal +retention of the separate existence of the + +space co-ordinates as opposed to the time co-ordinate. Every point on +the mollusc is treated as a space-point, and every material point which +is at rest relatively to it as at rest, so long as the mollusc is +considered as reference-body. The general principle of relativity +requires that all these molluscs can be used as reference-bodies with +equal right and equal success in the formulation of the general laws of +nature; the laws themselves must be quite independent of the choice of +mollusc. + +The great power possessed by the general principle of relativity lies +in the comprehensive limitation which is imposed on the laws of nature +in consequence of what we have seen above. + +If the reader has followed all our previous considerations, he will +have no further difficulty in understanding the methods leading to the +solution of the problem of gravitation. + +We start off on a consideration of a Galileian domain, _i.e._ a domain +in which there is no gravitational field relative to the Galileian +reference-body _K_. The behaviour of measuring-rods and clocks with +reference to _K_ is known from the special theory of relativity, +likewise the behaviour of “isolated” material points; the latter move +uniformly and in straight lines. + +Now let us refer this domain to a random Gauss coordinate system or to +a “mollusc” as reference-body _K′_. Then with respect to _K′_ there is +a gravitational field _G_ (of a particular kind). We learn the +behaviour of measuring-rods and clocks and also of freely-moving +material points with reference to _K′_ simply by mathematical +transformation. We interpret this behaviour as the behaviour of +measuring-rods, clocks and material points under the influence of the +gravitational field _G_. Hereupon we introduce a hypothesis: that the +influence of the gravitational field on measuring-rods, clocks and +freely-moving material points continues to take place according to the +same laws, even in the case where the prevailing gravitational field is +_not_ derivable from the Galileian special case, simply by means of a +transformation of co-ordinates. + +The next step is to investigate the space-time behaviour of the +gravitational field _G_, which was derived from the Galileian special +case simply by transformation of the coordinates. This behaviour is +formulated in a law, which is always valid, no matter how the +reference-body (mollusc) used in the description may be chosen. + +This law is not yet the _general_ law of the gravitational field, since +the gravitational field under consideration is of a special kind. In +order to find out the general law-of-field of gravitation we still +require to obtain a generalisation of the law as found above. This can +be obtained without caprice, however, by taking into consideration the +following demands: + +(_a_) The required generalisation must likewise satisfy the general +postulate of relativity. + +(_b_) If there is any matter in the domain under consideration, only +its inertial mass, and thus according to Section XV only its energy is +of importance for its effect in exciting a field. + +(_c_) Gravitational field and matter together must satisfy the law of +the conservation of energy (and of impulse). + +Finally, the general principle of relativity permits us to determine +the influence of the gravitational field on the course of all those +processes which take place according to known laws when a gravitational +field is absent _i.e._ which have already been fitted into the frame of +the special theory of relativity. In this connection we proceed in +principle according to the method which has already been explained for +measuring-rods, clocks and freely moving material points. + +The theory of gravitation derived in this way from the general +postulate of relativity excels not only in its beauty; nor in removing +the defect attaching to classical mechanics which was brought to light +in Section XXI; nor in interpreting the empirical law of the equality +of inertial and gravitational mass; but it has also already explained a +result of observation in astronomy, against which classical mechanics +is powerless. + +If we confine the application of the theory to the case where the +gravitational fields can be regarded as being weak, and in which all +masses move with respect to the coordinate system with velocities which +are small compared with the velocity of light, we then obtain as a +first approximation the Newtonian theory. Thus the latter theory is +obtained here without any particular assumption, whereas Newton had to +introduce the hypothesis that the force of attraction between mutually +attracting material points is inversely proportional to the square of +the distance between them. If we increase the accuracy of the +calculation, deviations from the theory of Newton make their +appearance, practically all of which must nevertheless escape the test +of observation owing to their smallness. + +We must draw attention here to one of these deviations. According to +Newton’s theory, a planet moves round the sun in an ellipse, which +would permanently maintain its position with respect to the fixed +stars, if we could disregard the motion of the fixed stars themselves +and the action of the other planets under consideration. Thus, if we +correct the observed motion of the planets for these two influences, +and if Newton’s theory be strictly correct, we ought to obtain for the +orbit of the planet an ellipse, which is fixed with reference to the +fixed stars. This deduction, which can be tested with great accuracy, +has been confirmed for all the planets save one, with the precision +that is capable of being obtained by the delicacy of observation +attainable at the present time. The sole exception is Mercury, the +planet which lies nearest the sun. Since the time of Leverrier, it has +been known that the ellipse corresponding to the orbit of Mercury, +after it has been corrected for the influences mentioned above, is not +stationary with respect to the fixed stars, but that it rotates +exceedingly slowly in the plane of the orbit and in the sense of the +orbital motion. The value obtained for this rotary movement of the +orbital ellipse was 43 seconds of arc per century, an amount ensured to +be correct to within a few seconds of arc. This effect can be explained +by means of classical mechanics only on the assumption of hypotheses +which have little probability, and which were devised solely for this +purponse. + +On the basis of the general theory of relativity, it is found that the +ellipse of every planet round the sun must necessarily rotate in the +manner indicated above; that for all the planets, with the exception of +Mercury, this rotation is too small to be detected with the delicacy of +observation possible at the present time; but that in the case of +Mercury it must amount to 43 seconds of arc per century, a result which +is strictly in agreement with observation. + +Apart from this one, it has hitherto been possible to make only two +deductions from the theory which admit of being tested by observation, +to wit, the curvature of light rays by the gravitational field of the +sun,[22] and a displacement of the spectral lines of light reaching us +from large stars, as compared with the corresponding lines for light +produced in an analogous manner terrestrially (_i.e._ by the same kind +of atom).[23] These two deductions from the theory have both been +confirmed. + + [22] First observed by Eddington and others in 1919. (Cf. Appendix + III). + + [23] Established by Adams in 1924. (Cf. p. 132) + +Part from the difficulty discussed in Section XXI, there is a second +fundamental difficulty attending classical celestial mechanics, which, +to the best of my knowledge, was first discussed in detail by the +astronomer Seeliger. If we ponder over the question as to how the +universe, considered as a whole, is to be regarded, the first answer +that suggests itself to us is surely this: As regards space (and time) +the universe is infinite. There are stars everywhere, so that the +density of matter, although very variable in detail, is nevertheless on +the average everywhere the same. In other words: However far we might +travel through space, we should find everywhere an attenuated swarm of +fixed stars of approrimately the same kind and density. + +This view is not in harmony with the theory of Newton. The latter +theory rather requires that the universe should have a kind of centre +in which the density of the stars is a maximum, and that as we proceed +outwards from this centre the group-density of the stars should +diminish, until finally, at great distances, it is succeeded by an +infinite region of emptiness. The stellar universe ought to be a finite +island in the infinite ocean of space.[24] + + [24] _Proof_—According to the theory of Newton, the number of “lines + of force” which come from infinity and terminate in a mass m is + proportional to the mass _m_. If, on the average, the mass density ρ0 + is constant throughout the universe, then a sphere of volume _V_ will + enclose the average mass ρ0_V_. Thus the number of lines of force + passing through the surface _F_ of the sphere into its interior is + proportional to ρ0_V_. For unit area of the surface of the sphere the + number of lines of force which enters the sphere is thus proportional + to ρ0_V/F_ or to ρ0_R_. Hence the intensity of the field at the + surface would ultimately become infinite with increasing radius _R_ of + the sphere, which is impossible. + +This conception is in itself not very satisfactory. It is still less +satisfactory because it leads to the result that the light emitted by +the stars and also individual stars of the stellar system are +perpetually passing out into infinite space, never to return, and +without ever again coming into interaction with other objects of +nature. Such a finite material universe would be destined to become +gradually but systematically impoverished. + +In order to escape this dilemma, Seeliger suggested a modification of +Newton’s law, in which he assumes that for great distances the force of +attraction between two masses diminishes more rapidly than would result +from the inverse square law. In this way it is possible for the mean +density of matter to be constant everywhere, even to infinity, without +infinitely large gravitational fields being produced. We thus free +ourselves from the distasteful conception that the material universe +ought to possess something of the nature of a centre. Of course we +purchase our emancipation from the fundamental difficulties mentioned, +at the cost of a modification and complication of Newton’s law which +has neither empirical nor theoretical foundation. We can imagine +innumerable laws which would serve the same purpose, without our being +able to state a reason why one of them is to be preferred to the +others; for any one of these laws would be founded just as little on +more general theoretical principles as is the law of Newton. + +But speculations on the structure of the universe also move in quite +another direction. The development of non-Euclidean geometry led to the +recognition of the fact, that we can cast doubt on the _infiniteness_ +of our space without coming into conflict with the laws of thought or +with experience (Riemann, Helmholtz). These questions have already been +treated in detail and with unsurpassable lucidity by Helmholtz and +Poincaré, whereas I can only touch on them briefly here. + +In the first place, we imagine an existence in two dimensional space. +Flat beings with flat implements, and in particular flat rigid +measuring-rods, are free to move in a _plane_. For them nothing exists +outside of this plane: that which they observe to happen to themselves +and to their flat “things” is the all-inclusive reality of their plane. +In particular, the constructions of plane Euclidean geometry can be +carried out by means of the rods _e.g._ the lattice construction, +considered in Section XXIV. In contrast to ours, the universe of these +beings is two-dimensional; but, like ours, it extends to infinity. In +their universe there is room for an infinite number of identical +squares made up of rods, _i.e._ its volume (surface) is infinite. If +these beings say their universe is “plane,” there is sense in the +statement, because they mean that they can perform the constructions of +plane Euclidean geometry with their rods. In this connection the +individual rods always represent the same distance, independently of +their position. + +Let us consider now a second two-dimensional existence, but this time +on a spherical surface instead of on a plane. The flat beings with +their measuring-rods and other objects fit exactly on this surface and +they are unable to leave it. Their whole universe of observation +extends exclusively over the surface of the sphere. Are these beings +able to regard the geometry of their universe as being plane geometry +and their rods withal as the realisation of “distance”? They cannot do +this. For if they attempt to realise a straight line, they will obtain +a curve, which we “three-dimensional beings” designate as a great +circle, _i.e._ a self-contained line of definite finite length, which +can be measured up by means of a measuring-rod. Similarly, this +universe has a finite area that can be compared with the area, of a +square constructed with rods. The great charm resulting from this +consideration lies in the recognition of the fact that _the universe of +these beings is finite and yet has no limits._ + +But the spherical-surface beings do not need to go on a world-tour in +order to perceive that they are not living in a Euclidean universe. +They can convince themselves of this on every part of their “world,” +provided they do not use too small a piece of it. Starting from a +point, they draw “straight lines” (arcs of circles as judged in three +dimensional space) of equal length in all directions. They will call +the line joining the free ends of these lines a “circle.” For a plane +surface, the ratio of the circumference of a circle to its diameter, +both lengths being measured with the same rod, is, according to +Euclidean geometry of the plane, equal to a constant value π, which is +independent of the diameter of the circle. On their spherical surface +our flat beings would find for this ratio the value + +image036 + +_i.e._ a smaller value than π, the difference being the more +considerable, the greater is the radius of the circle in comparison +with the radius _R_ of the “world-sphere.” By means of this relation +the spherical beings can determine the radius of their universe +(“world”), even when only a relatively small part of their worldsphere +is available for their measurements. But if this part is very small +indeed, they will no longer be able to demonstrate that they are on a +spherical “world” and not on a Euclidean plane, for a small part of a +spherical surface differs only slightly from a piece of a plane of the +same size. + +Thus if the spherical surface beings are living on a planet of which +the solar system occupies only a negligibly small part of the spherical +universe, they have no means of determining whether they are living in +a finite or in an infinite universe, because the “piece of universe” to +which they have access is in both cases practically plane, or +Euclidean. It follows directly from this discussion, that for our +sphere-beings the circumference of a circle first increases with the +radius until the “circumference of the universe” is reached, and that +it thenceforward gradually decreases to zero for still further +increasing values of the radius. During this process the area of the +circle continues to increase more and more, until finally it becomes +equal to the total area of the whole “world-sphere.” + +Perhaps the reader will wonder why we have placed our “beings” on a +sphere rather than on another closed surface. But this choice has its +justification in the fact that, of all closed surfaces, the sphere is +unique in possessing the property that all points on it are equivalent. +I admit that the ratio of the circumference _c_ of a circle to its +radius _r_ depends on _r_, but for a given value of _r_ it is the same +for all points of the “worldsphere”; in other words, the “world-sphere” +is a “surface of constant curvature.” + +To this two-dimensional sphere-universe there is a three-dimensional +analogy, namely, the three-dimensional spherical space which was +discovered by Riemann. its points are likewise all equivalent. It +possesses a finite volume, which is determined by its “radius” +(2π2_R_3). Is it possible to imagine a spherical space? To imagine a +space means nothing else than that we imagine an epitome of our “space” +experience, _i.e._ of experience that we can have in the movement of +“rigid” bodies. In this sense we _can_ imagine a spherical space. + +Suppose we draw lines or stretch strings in all directions from a +point, and mark off from each of these the distance _r_ with a +measuring-rod. All the free end-points of these lengths lie on a +spherical surface. We can specially measure up the area (_F_) of this +surface by means of a square made up of measuring-rods. If the universe +is Euclidean, then _F_ = 4π_r_2; if it is spherical, then _F_ is always +less than 4π_r_2. With increasing values of _r, F_ increases from zero +up to a maximum value which is determined by the “world-radius,” but +for still further increasing values of _r_, the area gradually +diminishes to zero. At first, the straight lines which radiate from the +starting point diverge farther and farther from one another, but later +they approach each other, and finally they run together again at a +“counter-point” to the starting point. Under such conditions they have +traversed the whole spherical space. It is easily seen that the +three-dimensional spherical space is quite analogous to the +two-dimensional spherical surface. It is finite (_i.e._ of finite +volume), and has no bounds. + +It may be mentioned that there is yet another kind of curved space: +“elliptical space.” It can be regarded as a curved space in which the +two “counter-points” are identical (indistinguishable from each other). +An elliptical universe can thus be considered to some extent as a +curved universe possessing central symmetry. + +It follows from what has been said, that closed spaces without limits +are conceivable. From amongst these, the spherical space (and the +elliptical) excels in its simplicity, since all points on it are +equivalent. As a result of this discussion, a most interesting question +arises for astronomers and physicists, and that is whether the universe +in which we live is infinite, or whether it is finite in the manner of +the spherical universe. Our experience is far from being sufficient to +enable us to answer this question. But the general theory of relativity +permits of our answering it with a moderate degree of certainty, and in +this connection the difficulty mentioned in Section XXX finds its +solution. + +According to the general theory of relativity, the geometrical +properties of space are not independent, but they are determined by +matter. Thus we can draw conclusions about the geometrical structure of +the universe only if we base our considerations on the state of the +matter as being something that is known. We know from experience that, +for a suitably chosen co-ordinate system, the velocities of the stars +are small as compared with the velocity of transmission of light. We +can thus as a rough approximation arrive at a conclusion as to the +nature of the universe as a whole, if we treat the matter as being at +rest. + +We already know from our previous discussion that the behaviour of +measuring-rods and clocks is influenced by gravitational fields, _i.e._ +by the distribution of matter. This in itself is sufficient to exclude +the possibility of the exact validity of Euclidean geometry in our +universe. But it is conceivable that our universe differs only slightly +from a Euclidean one, and this notion seems all the more probable, +since calculations show that the metrics of surrounding space is +influenced only to an exceedingly small extent by masses even of the +magnitude of our sun. We might imagine that, as regards geometry, our +universe behaves analogously to a surface which is irregularly curved +in its individual parts, but which nowhere departs appreciably from a +plane: something like the rippled surface of a lake. Such a universe +might fittingly be called a quasi-Euclidean universe. As regards its +space it would be infinite. But calculation shows that in a +quasi-Euclidean universe the average density of matter would +necessarily be _nil_. Thus such a universe could not be inhabited by +matter everywhere; it would present to us that unsatisfactory picture +which we portrayed in Section XXX. + +If we are to have in the universe an average density of matter which +differs from zero, however small may be that difference, then the +universe cannot be quasi-Euclidean. On the contrary, the results of +calculation indicate that if matter be distributed uniformly, the +universe would necessarily be spherical (or elliptical). Since in +reality the detailed distribution of matter is not uniform, the real +universe will deviate in individual parts from the spherical, _i.e._ +the universe will be quasi-spherical. But it will be necessarily +finite. In fact, the theory supplies us with a simple connection[25] +between the space-expanse of the universe and the average density of +matter in it. + + [25] For the radius _R_ of the universe we obtain the equation + +image037 + +The use of the C.G.S. system in this equation gives 2/k = 1.08 x 1027; +ρ is the average density of the matter and _k_ is a constant connected +with the Newtonian constant of gravitation. + +For the relative orientation of the co-ordinate systems indicated in +Fig. 2, the _x_-axes of both systems permanently coincide. In the +present case we can divide the problem into parts by considering first +only events which are localised on the _x_-axis. Any such event is +represented with respect to the co-ordinate system _K_ by the abscissa +_x_ and the time _t_, and with respect to the system _K′_ by the +abscissa _x′_ and the time _t′_. We require to find _x′_ and _t′_ when +_x_ and _t_ are given. + +A light-signal, which is proceeding along the positive axis of _x_, is +transmitted according to the equation + +_x_ = _ct_ + +or + +_x_ – _ct_ = 0 . . . . . (1). + +Since the same light-signal has to be transmitted relative to _K′_ with +the velocity _c_, the propagation relative to the system _K′_ will be +represented by the analogous formula + +_x′_ – _ct′_ = 0 . . . . . (2) + +Those space-time points (events) which satisfy (1) must also satisfy +(2). Obviously this will be the case when the relation + +(_x′_ – _ct′_) = λ(_x_ – _ct_) . . . (3). + +is fulfilled in general, where λ indicates a constant; for, according +to (3), the disappearance of (_x_ – _ct_) involves the disappearance of +(_x′_ – _ct′_). + +If we apply quite similar considerations to light rays which are being +transmitted along the negative _x_-axis, we obtain the condition + +(_x′_ + _ct′_) = (_x + ct_) . . . (4). + +By adding (or subtracting) equations (3) and (4), and introducing for +convenience the constants _a_ and _b_ in place of the constants λ and μ +where + +image038 + +and + +image039 + +we obtain the equations + +image040 + +We should thus have the solution of our problem, if the constants _a_ +and _b_ were known. These result from the following discussion. + +For the origin of _K′_ we have permanently _x′_ = 0, and hence +according to the first of the equations (5) + +image041 + +If we call _v_ the velocity with which the origin of _K′_ is moving +relative to _K_, we then have + +image042 + +The same value _v_ can be obtained from equations (5), if we calculate +the velocity of another point of _K′_ relative to _K_, or the velocity +(directed towards the negative _x_-axis) of a point of _K_ with respect +to _K′_. In short, we can designate _v_ as the relative velocity of the +two systems. + +Furthermore, the principle of relativity teaches us that, as judged +from K, the length of a unit measuring-rod which is at rest with +reference to _K′_ must be exactly the same as the length, as judged +from _K′_, of a unit measuring-rod which is at rest relative to _K_. In +order to see how the points of the _x′_-axis appear as viewed from _K_, +we only require to take a “snapshot” of _K′_ from _K_; this means that +we have to insert a particular value of _t_ (time of _K_), _e.g._ _t_ = +0. For this value of _t_ we then obtain from the first of the equations +(5) + +_x′_ = _ax_ + +Two points of the _x′_-axis which are separated by the distance Δ_x′_ = +1 when measured in the _K′_ system are thus separated in our +instantaneous photograph by the distance + +image043 + +But if the snapshot be taken from _K′_(_t′_ = 0), and if we eliminate +_t_ from the equations (5), taking into account the expression (6), we +obtain + +image044 + +From this we conclude that two points on the _x_-axis separated by the +distance 1 (relative to _K_) will be represented on our snapshot by the +distance + +image045 + +But from what has been said, the two snapshots must be identical; hence +Δ_x_ in (7) must be equal to Δ_x′_ in (7_a_), so that we obtain + +image046 + +The equations (6) and (7_b_) determine the constants _a_ and _b_. By +inserting the values of these constants in (5), we obtain the first and +the fourth of the equations given in Section XI. + +image047 + +Thus we have obtained the Lorentz transformation for events on the +_x_-axis. It satisfies the condition + +_x′_2 – _c_2_t′_2 = _x_2 – _c_2_t_2 . . . . . . (8a). + +The extension of this result, to include events which take place +outside the _x_-axis, is obtained by retaining equations (8) and +supplementing them by the relations + +image048 + +In this way we satisfy the postulate of the constancy of the velocity +of light _in vacuo_ for rays of light of arbitrary direction, both for +the system _K_ and for the system _K′_. This may be shown in the +following manner. + +We suppose a light-signal sent out from the origin of _K_ at the time +_t_ = 0. It will be propagated according to the equation + +image049 + +or, if we square this equation, according to the equation + +_x_2 + _y_2 + _z_2 – _c_2_t_2 = 0 . . . . . (10). + +It is required by the law of propagation of light, in conjunction with +the postulate of relativity, that the transmission of the signal in +question should take place—as judged from _K′_—in accordance with the +corresponding formula + +_r′_ = _ct′_ + +or, + +_x′_2 + _y′_2 + _z′_2 – _c_2_t′_2 = 0 . . . . . . (10_a_). + +In order that equation (10_a_) may be a consequence of equation (10), +we must have + +_x′_2 + _y′_2 + _z′_2 – _c_2_t′_2 = σ (_x_2 + _y_2 + _z_2 – _c_2_t_2) +(11). + +Since equation (8_a_) must hold for points on the _x_-axis, we thus +have σ = 1. It is easily seen that the Lorentz transformation really +satisfies equation (11) for σ = 1; for (11) is a consequence of (8_a_) +and (9), and hence also of (8) and (9). We have thus derived the +Lorentz transformation. + +The Lorentz transformation represented by (8) and (9) still requires to +be generalised. Obviously it is immaterial whether the axes of _K′_ be +chosen so that they are spatially parallel to those of _K_. It is also +not essential that the velocity of translation of _K′_ with respect to +_K_ should be in the direction of the _x_-axis. A simple consideration +shows that we are able to construct the Lorentz transformation in this +general sense from two kinds of transformations, viz. from Lorentz +transformations in the special sense and from purely spatial +transformations. which corresponds to the replacement of the +rectangular co-ordinate system by a new system with its axes pointing +in other directions. + +Mathematically, we can characterise the generalised Lorentz +transformation thus: + +It expresses _x′, y′, x′, t′_, in terms of linear homogeneous functions +of _x, y, x, t_, of such a kind that the relation + +_x′_2 + _y′_2 + _z′_2 – _c_2_t′_2 = _x_2 + _y_2 + _z_2 – _c_2_t_2 +(11_a_). + +is satisficd identically. That is to say: If we substitute their +expressions in _x, y, x, t_, in place of _x′, y′, x′, t′_, on the +left-hand side, then the left-hand side of (11_a_) agrees with the +right-hand side. + +We can characterise the Lorentz transformation still more simply if we +introduce the imaginary + +image031 + +in place of _t_, as time-variable. If, in accordance with this, we +insert + +image050 + +and similarly for the accented system _K′_, then the condition which is +identically satisfied by the transformation can be expressed thus: + +_x_1′2 + _x_2′2 + _x_3′2 + _x_4′2 = _x_12 + _x_22 + _x_32 + _x_42 (12). + +That is, by the afore-mentioned choice of “coordinates,” (11_a_) [see +the end of Appendix II] is transformed into this equation. + +We see from (12) that the imaginary time co-ordinate _x_4, enters into +the condition of transformation in exactly the same way as the space +co-ordinates _x_1, _x_2, _x_3. It is due to this fact that, according +to the theory of relativity, the “time” _x_4, enters into natural laws +in the same form as the space co ordinates _x_1, _x_2, _x_3. + +A four-dimensional continuum described by the “co-ordinates” _x_1, +_x_2, _x_3, _x_4, was called “world” by Minkowski, who also termed a +point-event a “world-point.” From a “happening” in three-dimensional +space, physics becomes, as it were, an “existence” in the +four-dimensional “world.” diff --git a/data/science-v2/validation.txt b/data/science-v2/validation.txt new file mode 100644 index 0000000..ba5b025 --- /dev/null +++ b/data/science-v2/validation.txt @@ -0,0 +1,1738 @@ +We are so much accustomed to see differences in structure between the +embryo and the adult, and likewise a close similarity in the embryos of +widely different animals within the same class, that we might be led to +look at these facts as necessarily contingent in some manner on growth. +But there is no obvious reason why, for instance, the wing of a bat, or +the fin of a porpoise, should not have been sketched out with all the +parts in proper proportion, as soon as any structure became visible in +the embryo. And in some whole groups of animals and in certain members +of other groups, the embryo does not at any period differ widely from +the adult: thus Owen has remarked in regard to cuttle-fish, “there is +no metamorphosis; the cephalopodic character is manifested long before +the parts of the embryo are completed;” and again in spiders, “there is +nothing worthy to be called a metamorphosis.” The larvæ of insects, +whether adapted to the most diverse and active habits, or quite +inactive, being fed by their parents or placed in the midst of proper +nutriment, yet nearly all pass through a similar worm-like stage of +development; but in some few cases, as in that of Aphis, if we look to +the admirable drawings by Professor Huxley of the development of this +insect, we see no trace of the vermiform stage. + +How, then, can we explain these several facts in embryology,—namely the +very general, but not universal difference in structure between the +embryo and the adult;—of parts in the same individual embryo, which +ultimately become very unlike and serve for diverse purposes, being at +this early period of growth alike;—of embryos of different species +within the same class, generally, but not universally, resembling each +other;—of the structure of the embryo not being closely related to its +conditions of existence, except when the +embryo becomes at any period of life active and has to provide for +itself;—of the embryo apparently having sometimes a higher organisation +than the mature animal, into which it is developed. I believe that all +these facts can be explained, as follows, on the view of descent with +modification. + +It is commonly assumed, perhaps from monstrosities often affecting the +embryo at a very early period, that slight variations necessarily +appear at an equally early period. But we have little evidence on this +head—indeed the evidence rather points the other way; for it is +notorious that breeders of cattle, horses, and various fancy animals, +cannot positively tell, until some time after the animal has been born, +what its merits or form will ultimately turn out. We see this plainly +in our own children; we cannot always tell whether the child will be +tall or short, or what its precise features will be. The question is +not, at what period of life any variation has been caused, but at what +period it is fully displayed. The cause may have acted, and I believe +generally has acted, even before the embryo is formed; and the +variation may be due to the male and female sexual elements having been +affected by the conditions to which either parent, or their ancestors, +have been exposed. Nevertheless an effect thus caused at a very early +period, even before the formation of the embryo, may appear late in +life; as when an hereditary disease, which appears in old age alone, +has been communicated to the offspring from the reproductive element of +one parent. Or again, as when the horns of cross-bred cattle have been +affected by the shape of the horns of either parent. For the welfare of +a very young animal, as long as it remains in its mother’s womb, or in +the egg, or as long as it is nourished and protected by its parent, it +must be quite unimportant whether most of its characters are fully +acquired a little earlier or later in life. It would not signify, for +instance, to a bird which obtained its food best by having a long beak, +whether or not it assumed a beak of this particular length, as long as +it was fed by its parents. Hence, I conclude, that it is quite +possible, that each of the many successive modifications, by which each +species has acquired its present structure, may have supervened at a +not very early period of life; and some direct evidence from our +domestic animals supports this view. But in other cases it is quite +possible that each successive modification, or most of them, may have +appeared at an extremely early period. + +I have stated in the first chapter, that there is some evidence to +render it probable, that at whatever age any variation first appears in +the parent, it tends to reappear at a corresponding age in the +offspring. Certain variations can only appear at corresponding ages, +for instance, peculiarities in the caterpillar, cocoon, or imago states +of the silk-moth; or, again, in the horns of almost full-grown cattle. +But further than this, variations which, for all that we can see, might +have appeared earlier or later in life, tend to appear at a +corresponding age in the offspring and parent. I am far from meaning +that this is invariably the case; and I could give a good many cases of +variations (taking the word in the largest sense) which have supervened +at an earlier age in the child than in the parent. + +These two principles, if their truth be admitted, will, I believe, +explain all the above specified leading facts in embryology. But first +let us look at a few analogous cases in domestic varieties. Some +authors who have written on Dogs, maintain that the greyhound and +bulldog, though appearing so different, are really varieties most +closely allied, and have probably descended from +the same wild stock; hence I was curious to see how far their puppies +differed from each other: I was told by breeders that they differed +just as much as their parents, and this, judging by the eye, seemed +almost to be the case; but on actually measuring the old dogs and their +six-days old puppies, I found that the puppies had not nearly acquired +their full amount of proportional difference. So, again, I was told +that the foals of cart and race-horses differed as much as the +full-grown animals; and this surprised me greatly, as I think it +probable that the difference between these two breeds has been wholly +caused by selection under domestication; but having had careful +measurements made of the dam and of a three-days old colt of a race and +heavy cart-horse, I find that the colts have by no means acquired their +full amount of proportional difference. + +As the evidence appears to me conclusive, that the several domestic +breeds of Pigeon have descended from one wild species, I compared young +pigeons of various breeds, within twelve hours after being hatched; I +carefully measured the proportions (but will not here give details) of +the beak, width of mouth, length of nostril and of eyelid, size of feet +and length of leg, in the wild stock, in pouters, fantails, runts, +barbs, dragons, carriers, and tumblers. Now some of these birds, when +mature, differ so extraordinarily in length and form of beak, that they +would, I cannot doubt, be ranked in distinct genera, had they been +natural productions. But when the nestling birds of these several +breeds were placed in a row, though most of them could be distinguished +from each other, yet their proportional differences in the above +specified several points were incomparably less than in the full-grown +birds. Some characteristic points of difference—for instance, that of +the width of mouth—could hardly be detected in the young. +But there was one remarkable exception to this rule, for the young of +the short-faced tumbler differed from the young of the wild rock-pigeon +and of the other breeds, in all its proportions, almost exactly as much +as in the adult state. + +The two principles above given seem to me to explain these facts in +regard to the later embryonic stages of our domestic varieties. +Fanciers select their horses, dogs, and pigeons, for breeding, when +they are nearly grown up: they are indifferent whether the desired +qualities and structures have been acquired earlier or later in life, +if the full-grown animal possesses them. And the cases just given, more +especially that of pigeons, seem to show that the characteristic +differences which give value to each breed, and which have been +accumulated by man’s selection, have not generally first appeared at an +early period of life, and have been inherited by the offspring at a +corresponding not early period. But the case of the short-faced +tumbler, which when twelve hours old had acquired its proper +proportions, proves that this is not the universal rule; for here the +characteristic differences must either have appeared at an earlier +period than usual, or, if not so, the differences must have been +inherited, not at the corresponding, but at an earlier age. + +Now let us apply these facts and the above two principles—which latter, +though not proved true, can be shown to be in some degree probable—to +species in a state of nature. Let us take a genus of birds, descended +on my theory from some one parent-species, and of which the several new +species have become modified through natural selection in accordance +with their diverse habits. Then, from the many slight successive steps +of variation having supervened at a rather late age, and having been +inherited at a corresponding +age, the young of the new species of our supposed genus will manifestly +tend to resemble each other much more closely than do the adults, just +as we have seen in the case of pigeons. We may extend this view to +whole families or even classes. The fore-limbs, for instance, which +served as legs in the parent-species, may become, by a long course of +modification, adapted in one descendant to act as hands, in another as +paddles, in another as wings; and on the above two principles—namely of +each successive modification supervening at a rather late age, and +being inherited at a corresponding late age—the fore-limbs in the +embryos of the several descendants of the parent-species will still +resemble each other closely, for they will not have been modified. But +in each individual new species, the embryonic fore-limbs will differ +greatly from the fore-limbs in the mature animal; the limbs in the +latter having undergone much modification at a rather late period of +life, and having thus been converted into hands, or paddles, or wings. +Whatever influence long-continued exercise or use on the one hand, and +disuse on the other, may have in modifying an organ, such influence +will mainly affect the mature animal, which has come to its full powers +of activity and has to gain its own living; and the effects thus +produced will be inherited at a corresponding mature age. Whereas the +young will remain unmodified, or be modified in a lesser degree, by the +effects of use and disuse. + +In certain cases the successive steps of variation might supervene, +from causes of which we are wholly ignorant, at a very early period of +life, or each step might be inherited at an earlier period than that at +which it first appeared. In either case (as with the short-faced +tumbler) the young or embryo would closely +resemble the mature parent-form. We have seen that this is the rule of +development in certain whole groups of animals, as with cuttle-fish and +spiders, and with a few members of the great class of insects, as with +Aphis. With respect to the final cause of the young in these cases not +undergoing any metamorphosis, or closely resembling their parents from +their earliest age, we can see that this would result from the two +following contingencies; firstly, from the young, during a course of +modification carried on for many generations, having to provide for +their own wants at a very early stage of development, and secondly, +from their following exactly the same habits of life with their +parents; for in this case, it would be indispensable for the existence +of the species, that the child should be modified at a very early age +in the same manner with its parents, in accordance with their similar +habits. Some further explanation, however, of the embryo not undergoing +any metamorphosis is perhaps requisite. If, on the other hand, it +profited the young to follow habits of life in any degree different +from those of their parent, and consequently to be constructed in a +slightly different manner, then, on the principle of inheritance at +corresponding ages, the active young or larvæ might easily be rendered +by natural selection different to any conceivable extent from their +parents. Such differences might, also, become correlated with +successive stages of development; so that the larvæ, in the first +stage, might differ greatly from the larvæ in the second stage, as we +have seen to be the case with cirripedes. The adult might become fitted +for sites or habits, in which organs of locomotion or of the senses, +etc., would be useless; and in this case the final metamorphosis would +be said to be retrograde. + +As all the organic beings, extinct and recent, which +have ever lived on this earth have to be classed together, and as all +have been connected by the finest gradations, the best, or indeed, if +our collections were nearly perfect, the only possible arrangement, +would be genealogical. Descent being on my view the hidden bond of +connexion which naturalists have been seeking under the term of the +natural system. On this view we can understand how it is that, in the +eyes of most naturalists, the structure of the embryo is even more +important for classification than that of the adult. For the embryo is +the animal in its less modified state; and in so far it reveals the +structure of its progenitor. In two groups of animal, however much they +may at present differ from each other in structure and habits, if they +pass through the same or similar embryonic stages, we may feel assured +that they have both descended from the same or nearly similar parents, +and are therefore in that degree closely related. Thus, community in +embryonic structure reveals community of descent. It will reveal this +community of descent, however much the structure of the adult may have +been modified and obscured; we have seen, for instance, that cirripedes +can at once be recognised by their larvæ as belonging to the great +class of crustaceans. As the embryonic state of each species and group +of species partially shows us the structure of their less modified +ancient progenitors, we can clearly see why ancient and extinct forms +of life should resemble the embryos of their descendants,—our existing +species. Agassiz believes this to be a law of nature; but I am bound to +confess that I only hope to see the law hereafter proved true. It can +be proved true in those cases alone in which the ancient state, now +supposed to be represented in many embryos, has not been obliterated, +either by the successive variations in a long course of modification +having supervened +at a very early age, or by the variations having been inherited at an +earlier period than that at which they first appeared. It should also +be borne in mind, that the supposed law of resemblance of ancient forms +of life to the embryonic stages of recent forms, may be true, but yet, +owing to the geological record not extending far enough back in time, +may remain for a long period, or for ever, incapable of demonstration. + +Thus, as it seems to me, the leading facts in embryology, which are +second in importance to none in natural history, are explained on the +principle of slight modifications not appearing, in the many +descendants from some one ancient progenitor, at a very early period in +the life of each, though perhaps caused at the earliest, and being +inherited at a corresponding not early period. Embryology rises greatly +in interest, when we thus look at the embryo as a picture, more or less +obscured, of the common parent-form of each great class of animals. + +_Rudimentary, atrophied, or aborted organs_.—Organs or parts in this +strange condition, bearing the stamp of inutility, are extremely common +throughout nature. For instance, rudimentary mammæ are very general in +the males of mammals: I presume that the “bastard-wing” in birds may be +safely considered as a digit in a rudimentary state: in very many +snakes one lobe of the lungs is rudimentary; in other snakes there are +rudiments of the pelvis and hind limbs. Some of the cases of +rudimentary organs are extremely curious; for instance, the presence of +teeth in foetal whales, which when grown up have not a tooth in their +heads; and the presence of teeth, which never cut through the gums, in +the upper jaws of our unborn calves. It has even been stated on good +authority that rudiments of teeth can be detected +in the beaks of certain embryonic birds. Nothing can be plainer than +that wings are formed for flight, yet in how many insects do we see +wings so reduced in size as to be utterly incapable of flight, and not +rarely lying under wing-cases, firmly soldered together! + +The meaning of rudimentary organs is often quite unmistakeable: for +instance there are beetles of the same genus (and even of the same +species) resembling each other most closely in all respects, one of +which will have full-sized wings, and another mere rudiments of +membrane; and here it is impossible to doubt, that the rudiments +represent wings. Rudimentary organs sometimes retain their +potentiality, and are merely not developed: this seems to be the case +with the mammæ of male mammals, for many instances are on record of +these organs having become well developed in full-grown males, and +having secreted milk. So again there are normally four developed and +two rudimentary teats in the udders of the genus Bos, but in our +domestic cows the two sometimes become developed and give milk. In +individual plants of the same species the petals sometimes occur as +mere rudiments, and sometimes in a well-developed state. In plants with +separated sexes, the male flowers often have a rudiment of a pistil; +and Kölreuter found that by crossing such male plants with an +hermaphrodite species, the rudiment of the pistil in the hybrid +offspring was much increased in size; and this shows that the rudiment +and the perfect pistil are essentially alike in nature. + +An organ serving for two purposes, may become rudimentary or utterly +aborted for one, even the more important purpose; and remain perfectly +efficient for the other. Thus in plants, the office of the pistil is to +allow the pollen-tubes to reach the ovules protected in the ovarium at +its base. The pistil consists of a stigma +supported on the style; but in some Compositæ, the male florets, which +of course cannot be fecundated, have a pistil, which is in a +rudimentary state, for it is not crowned with a stigma; but the style +remains well developed, and is clothed with hairs as in other +compositæ, for the purpose of brushing the pollen out of the +surrounding anthers. Again, an organ may become rudimentary for its +proper purpose, and be used for a distinct object: in certain fish the +swim-bladder seems to be rudimentary for its proper function of giving +buoyancy, but has become converted into a nascent breathing organ or +lung. Other similar instances could be given. + +Rudimentary organs in the individuals of the same species are very +liable to vary in degree of development and in other respects. +Moreover, in closely allied species, the degree to which the same organ +has been rendered rudimentary occasionally differs much. This latter +fact is well exemplified in the state of the wings of the female moths +in certain groups. Rudimentary organs may be utterly aborted; and this +implies, that we find in an animal or plant no trace of an organ, which +analogy would lead us to expect to find, and which is occasionally +found in monstrous individuals of the species. Thus in the snapdragon +(antirrhinum) we generally do not find a rudiment of a fifth stamen; +but this may sometimes be seen. In tracing the homologies of the same +part in different members of a class, nothing is more common, or more +necessary, than the use and discovery of rudiments. This is well shown +in the drawings given by Owen of the bones of the leg of the horse, ox, +and rhinoceros. + +It is an important fact that rudimentary organs, such as teeth in the +upper jaws of whales and ruminants, can often be detected in the +embryo, but afterwards wholly disappear. It is also, I believe, a +universal +rule, that a rudimentary part or organ is of greater size relatively to +the adjoining parts in the embryo, than in the adult; so that the organ +at this early age is less rudimentary, or even cannot be said to be in +any degree rudimentary. Hence, also, a rudimentary organ in the adult, +is often said to have retained its embryonic condition. + +I have now given the leading facts with respect to rudimentary organs. +In reflecting on them, every one must be struck with astonishment: for +the same reasoning power which tells us plainly that most parts and +organs are exquisitely adapted for certain purposes, tells us with +equal plainness that these rudimentary or atrophied organs, are +imperfect and useless. In works on natural history rudimentary organs +are generally said to have been created “for the sake of symmetry,” or +in order “to complete the scheme of nature;” but this seems to me no +explanation, merely a restatement of the fact. Would it be thought +sufficient to say that because planets revolve in elliptic courses +round the sun, satellites follow the same course round the planets, for +the sake of symmetry, and to complete the scheme of nature? An eminent +physiologist accounts for the presence of rudimentary organs, by +supposing that they serve to excrete matter in excess, or injurious to +the system; but can we suppose that the minute papilla, which often +represents the pistil in male flowers, and which is formed merely of +cellular tissue, can thus act? Can we suppose that the formation of +rudimentary teeth which are subsequently absorbed, can be of any +service to the rapidly growing embryonic calf by the excretion of +precious phosphate of lime? When a man’s fingers have been amputated, +imperfect nails sometimes appear on the stumps: I could as soon believe +that these vestiges of nails have appeared, not from unknown laws +of growth, but in order to excrete horny matter, as that the +rudimentary nails on the fin of the manatee were formed for this +purpose. + +On my view of descent with modification, the origin of rudimentary +organs is simple. We have plenty of cases of rudimentary organs in our +domestic productions,—as the stump of a tail in tailless breeds,—the +vestige of an ear in earless breeds,—the reappearance of minute +dangling horns in hornless breeds of cattle, more especially, according +to Youatt, in young animals,—and the state of the whole flower in the +cauliflower. We often see rudiments of various parts in monsters. But I +doubt whether any of these cases throw light on the origin of +rudimentary organs in a state of nature, further than by showing that +rudiments can be produced; for I doubt whether species under nature +ever undergo abrupt changes. I believe that disuse has been the main +agency; that it has led in successive generations to the gradual +reduction of various organs, until they have become rudimentary,—as in +the case of the eyes of animals inhabiting dark caverns, and of the +wings of birds inhabiting oceanic islands, which have seldom been +forced to take flight, and have ultimately lost the power of flying. +Again, an organ useful under certain conditions, might become injurious +under others, as with the wings of beetles living on small and exposed +islands; and in this case natural selection would continue slowly to +reduce the organ, until it was rendered harmless and rudimentary. + +Any change in function, which can be effected by insensibly small +steps, is within the power of natural selection; so that an organ +rendered, during changed habits of life, useless or injurious for one +purpose, might easily be modified and used for another purpose. Or an +organ might be retained for one alone of its +former functions. An organ, when rendered useless, may well be +variable, for its variations cannot be checked by natural selection. At +whatever period of life disuse or selection reduces an organ, and this +will generally be when the being has come to maturity and to its full +powers of action, the principle of inheritance at corresponding ages +will reproduce the organ in its reduced state at the same age, and +consequently will seldom affect or reduce it in the embryo. Thus we can +understand the greater relative size of rudimentary organs in the +embryo, and their lesser relative size in the adult. But if each step +of the process of reduction were to be inherited, not at the +corresponding age, but at an extremely early period of life (as we have +good reason to believe to be possible) the rudimentary part would tend +to be wholly lost, and we should have a case of complete abortion. The +principle, also, of economy, explained in a former chapter, by which +the materials forming any part or structure, if not useful to the +possessor, will be saved as far as is possible, will probably often +come into play; and this will tend to cause the entire obliteration of +a rudimentary organ. + +As the presence of rudimentary organs is thus due to the tendency in +every part of the organisation, which has long existed, to be +inherited—we can understand, on the genealogical view of +classification, how it is that systematists have found rudimentary +parts as useful as, or even sometimes more useful than, parts of high +physiological importance. Rudimentary organs may be compared with the +letters in a word, still retained in the spelling, but become useless +in the pronunciation, but which serve as a clue in seeking for its +derivation. On the view of descent with modification, we may conclude +that the existence of organs in a rudimentary, imperfect, and useless +condition, or quite aborted, far +from presenting a strange difficulty, as they assuredly do on the +ordinary doctrine of creation, might even have been anticipated, and +can be accounted for by the laws of inheritance. + +_Summary_.—In this chapter I have attempted to show, that the +subordination of group to group in all organisms throughout all time; +that the nature of the relationship, by which all living and extinct +beings are united by complex, radiating, and circuitous lines of +affinities into one grand system; the rules followed and the +difficulties encountered by naturalists in their classifications; the +value set upon characters, if constant and prevalent, whether of high +vital importance, or of the most trifling importance, or, as in +rudimentary organs, of no importance; the wide opposition in value +between analogical or adaptive characters, and characters of true +affinity; and other such rules;—all naturally follow on the view of the +common parentage of those forms which are considered by naturalists as +allied, together with their modification through natural selection, +with its contingencies of extinction and divergence of character. In +considering this view of classification, it should be borne in mind +that the element of descent has been universally used in ranking +together the sexes, ages, and acknowledged varieties of the same +species, however different they may be in structure. If we extend the +use of this element of descent,—the only certainly known cause of +similarity in organic beings,—we shall understand what is meant by the +natural system: it is genealogical in its attempted arrangement, with +the grades of acquired difference marked by the terms varieties, +species, genera, families, orders, and classes. + +On this same view of descent with modification, all the great facts in +Morphology become intelligible,—whether +we look to the same pattern displayed in the homologous organs, to +whatever purpose applied, of the different species of a class; or to +the homologous parts constructed on the same pattern in each individual +animal and plant. + +On the principle of successive slight variations, not necessarily or +generally supervening at a very early period of life, and being +inherited at a corresponding period, we can understand the great +leading facts in Embryology; namely, the resemblance in an individual +embryo of the homologous parts, which when matured will become widely +different from each other in structure and function; and the +resemblance in different species of a class of the homologous parts or +organs, though fitted in the adult members for purposes as different as +possible. Larvæ are active embryos, which have become specially +modified in relation to their habits of life, through the principle of +modifications being inherited at corresponding ages. On this same +principle—and bearing in mind, that when organs are reduced in size, +either from disuse or selection, it will generally be at that period of +life when the being has to provide for its own wants, and bearing in +mind how strong is the principle of inheritance—the occurrence of +rudimentary organs and their final abortion, present to us no +inexplicable difficulties; on the contrary, their presence might have +been even anticipated. The importance of embryological characters and +of rudimentary organs in classification is intelligible, on the view +that an arrangement is only so far natural as it is genealogical. + +Finally, the several classes of facts which have been considered in +this chapter, seem to me to proclaim so plainly, that the innumerable +species, genera, and families of organic beings, with which this world +is +peopled, have all descended, each within its own class or group, from +common parents, and have all been modified in the course of descent, +that I should without hesitation adopt this view, even if it were +unsupported by other facts or arguments. + +Recapitulation of the difficulties on the theory of Natural Selection. +Recapitulation of the general and special circumstances in its favour. +Causes of the general belief in the immutability of species. How far +the theory of natural selection may be extended. Effects of its +adoption on the study of Natural history. Concluding remarks. + +As this whole volume is one long argument, it may be convenient to the +reader to have the leading facts and inferences briefly recapitulated. + +That many and grave objections may be advanced against the theory of +descent with modification through natural selection, I do not deny. I +have endeavoured to give to them their full force. Nothing at first can +appear more difficult to believe than that the more complex organs and +instincts should have been perfected, not by means superior to, though +analogous with, human reason, but by the accumulation of innumerable +slight variations, each good for the individual possessor. +Nevertheless, this difficulty, though appearing to our imagination +insuperably great, cannot be considered real if we admit the following +propositions, namely,—that gradations in the perfection of any organ or +instinct, which we may consider, either do now exist or could have +existed, each good of its kind,—that all organs and instincts are, in +ever so slight a degree, variable,—and, lastly, that there is a +struggle for existence leading to the preservation of each profitable +deviation of structure or instinct. The truth of these propositions +cannot, I think, be disputed. + +It is, no doubt, extremely difficult even to conjecture by what +gradations many structures have been perfected, more especially amongst +broken and failing groups of organic beings; but we see so many strange +gradations in nature, as is proclaimed by the canon, “Natura non facit +saltum,” that we ought to be extremely cautious in saying that any +organ or instinct, or any whole being, could not have arrived at its +present state by many graduated steps. There are, it must be admitted, +cases of special difficulty on the theory of natural selection; and one +of the most curious of these is the existence of two or three defined +castes of workers or sterile females in the same community of ants; but +I have attempted to show how this difficulty can be mastered. + +With respect to the almost universal sterility of species when first +crossed, which forms so remarkable a contrast with the almost universal +fertility of varieties when crossed, I must refer the reader to the +recapitulation of the facts given at the end of the eighth chapter, +which seem to me conclusively to show that this sterility is no more a +special endowment than is the incapacity of two trees to be grafted +together, but that it is incidental on constitutional differences in +the reproductive systems of the intercrossed species. We see the truth +of this conclusion in the vast difference in the result, when the same +two species are crossed reciprocally; that is, when one species is +first used as the father and then as the mother. + +The fertility of varieties when intercrossed and of their mongrel +offspring cannot be considered as universal; nor is their very general +fertility surprising when we remember that it is not likely that either +their constitutions or their reproductive systems should have been +profoundly modified. Moreover, most of the +varieties which have been experimentised on have been produced under +domestication; and as domestication apparently tends to eliminate +sterility, we ought not to expect it also to produce sterility. + +The sterility of hybrids is a very different case from that of first +crosses, for their reproductive organs are more or less functionally +impotent; whereas in first crosses the organs on both sides are in a +perfect condition. As we continually see that organisms of all kinds +are rendered in some degree sterile from their constitutions having +been disturbed by slightly different and new conditions of life, we +need not feel surprise at hybrids being in some degree sterile, for +their constitutions can hardly fail to have been disturbed from being +compounded of two distinct organisations. This parallelism is supported +by another parallel, but directly opposite, class of facts; namely, +that the vigour and fertility of all organic beings are increased by +slight changes in their conditions of life, and that the offspring of +slightly modified forms or varieties acquire from being crossed +increased vigour and fertility. So that, on the one hand, considerable +changes in the conditions of life and crosses between greatly modified +forms, lessen fertility; and on the other hand, lesser changes in the +conditions of life and crosses between less modified forms, increase +fertility. + +Turning to geographical distribution, the difficulties encountered on +the theory of descent with modification are grave enough. All the +individuals of the same species, and all the species of the same genus, +or even higher group, must have descended from common parents; and +therefore, in however distant and isolated parts of the world they are +now found, they must in the course of successive generations have +passed from some one part to the others. We are often wholly unable +even to conjecture how this could have been effected. Yet, as we have +reason to believe that some species have retained the same specific +form for very long periods, enormously long as measured by years, too +much stress ought not to be laid on the occasional wide diffusion of +the same species; for during very long periods of time there will +always be a good chance for wide migration by many means. A broken or +interrupted range may often be accounted for by the extinction of the +species in the intermediate regions. It cannot be denied that we are as +yet very ignorant of the full extent of the various climatal and +geographical changes which have affected the earth during modern +periods; and such changes will obviously have greatly facilitated +migration. As an example, I have attempted to show how potent has been +the influence of the Glacial period on the distribution both of the +same and of representative species throughout the world. We are as yet +profoundly ignorant of the many occasional means of transport. With +respect to distinct species of the same genus inhabiting very distant +and isolated regions, as the process of modification has necessarily +been slow, all the means of migration will have been possible during a +very long period; and consequently the difficulty of the wide diffusion +of species of the same genus is in some degree lessened. + +As on the theory of natural selection an interminable number of +intermediate forms must have existed, linking together all the species +in each group by gradations as fine as our present varieties, it may be +asked, Why do we not see these linking forms all around us? Why are not +all organic beings blended together in an inextricable chaos? With +respect to existing forms, we should remember that we have no right to +expect (excepting in rare cases) to discover _directly_ connecting +links between them, but only between each and some extinct and +supplanted form. Even on a wide area, which has during a long period +remained continuous, and of which the climate and other conditions of +life change insensibly in going from a district occupied by one species +into another district occupied by a closely allied species, we have no +just right to expect often to find intermediate varieties in the +intermediate zone. For we have reason to believe that only a few +species are undergoing change at any one period; and all changes are +slowly effected. I have also shown that the intermediate varieties +which will at first probably exist in the intermediate zones, will be +liable to be supplanted by the allied forms on either hand; and the +latter, from existing in greater numbers, will generally be modified +and improved at a quicker rate than the intermediate varieties, which +exist in lesser numbers; so that the intermediate varieties will, in +the long run, be supplanted and exterminated. + +On this doctrine of the extermination of an infinitude of connecting +links, between the living and extinct inhabitants of the world, and at +each successive period between the extinct and still older species, why +is not every geological formation charged with such links? Why does not +every collection of fossil remains afford plain evidence of the +gradation and mutation of the forms of life? We meet with no such +evidence, and this is the most obvious and forcible of the many +objections which may be urged against my theory. Why, again, do whole +groups of allied species appear, though certainly they often falsely +appear, to have come in suddenly on the several geological stages? Why +do we not find great piles of strata beneath the Silurian system, +stored with the remains of the progenitors of the Silurian groups of +fossils? For certainly on my theory such +strata must somewhere have been deposited at these ancient and utterly +unknown epochs in the world’s history. + +I can answer these questions and grave objections only on the +supposition that the geological record is far more imperfect than most +geologists believe. It cannot be objected that there has not been time +sufficient for any amount of organic change; for the lapse of time has +been so great as to be utterly inappreciable by the human intellect. +The number of specimens in all our museums is absolutely as nothing +compared with the countless generations of countless species which +certainly have existed. We should not be able to recognise a species as +the parent of any one or more species if we were to examine them ever +so closely, unless we likewise possessed many of the intermediate links +between their past or parent and present states; and these many links +we could hardly ever expect to discover, owing to the imperfection of +the geological record. Numerous existing doubtful forms could be named +which are probably varieties; but who will pretend that in future ages +so many fossil links will be discovered, that naturalists will be able +to decide, on the common view, whether or not these doubtful forms are +varieties? As long as most of the links between any two species are +unknown, if any one link or intermediate variety be discovered, it will +simply be classed as another and distinct species. Only a small portion +of the world has been geologically explored. Only organic beings of +certain classes can be preserved in a fossil condition, at least in any +great number. Widely ranging species vary most, and varieties are often +at first local,—both causes rendering the discovery of intermediate +links less likely. Local varieties will not spread into other and +distant regions until they are considerably modified and improved; +and when they do spread, if discovered in a geological formation, they +will appear as if suddenly created there, and will be simply classed as +new species. Most formations have been intermittent in their +accumulation; and their duration, I am inclined to believe, has been +shorter than the average duration of specific forms. Successive +formations are separated from each other by enormous blank intervals of +time; for fossiliferous formations, thick enough to resist future +degradation, can be accumulated only where much sediment is deposited +on the subsiding bed of the sea. During the alternate periods of +elevation and of stationary level the record will be blank. During +these latter periods there will probably be more variability in the +forms of life; during periods of subsidence, more extinction. + +With respect to the absence of fossiliferous formations beneath the +lowest Silurian strata, I can only recur to the hypothesis given in the +ninth chapter. That the geological record is imperfect all will admit; +but that it is imperfect to the degree which I require, few will be +inclined to admit. If we look to long enough intervals of time, geology +plainly declares that all species have changed; and they have changed +in the manner which my theory requires, for they have changed slowly +and in a graduated manner. We clearly see this in the fossil remains +from consecutive formations invariably being much more closely related +to each other, than are the fossils from formations distant from each +other in time. + +Such is the sum of the several chief objections and difficulties which +may justly be urged against my theory; and I have now briefly +recapitulated the answers and explanations which can be given to them. +I have felt these difficulties far too heavily during many years to +doubt their weight. But it deserves especial notice that the more +important objections relate to questions on which we are confessedly +ignorant; nor do we know how ignorant we are. We do not know all the +possible transitional gradations between the simplest and the most +perfect organs; it cannot be pretended that we know all the varied +means of Distribution during the long lapse of years, or that we know +how imperfect the Geological Record is. Grave as these several +difficulties are, in my judgment they do not overthrow the theory of +descent with modification. + +Now let us turn to the other side of the argument. Under domestication +we see much variability. This seems to be mainly due to the +reproductive system being eminently susceptible to changes in the +conditions of life; so that this system, when not rendered impotent, +fails to reproduce offspring exactly like the parent-form. Variability +is governed by many complex laws,—by correlation of growth, by use and +disuse, and by the direct action of the physical conditions of life. +There is much difficulty in ascertaining how much modification our +domestic productions have undergone; but we may safely infer that the +amount has been large, and that modifications can be inherited for long +periods. As long as the conditions of life remain the same, we have +reason to believe that a modification, which has already been inherited +for many generations, may continue to be inherited for an almost +infinite number of generations. On the other hand we have evidence that +variability, when it has once come into play, does not wholly cease; +for new varieties are still occasionally produced by our most anciently +domesticated productions. + +Man does not actually produce variability; he only +unintentionally exposes organic beings to new conditions of life, and +then nature acts on the organisation, and causes variability. But man +can and does select the variations given to him by nature, and thus +accumulate them in any desired manner. He thus adapts animals and +plants for his own benefit or pleasure. He may do this methodically, or +he may do it unconsciously by preserving the individuals most useful to +him at the time, without any thought of altering the breed. It is +certain that he can largely influence the character of a breed by +selecting, in each successive generation, individual differences so +slight as to be quite inappreciable by an uneducated eye. This process +of selection has been the great agency in the production of the most +distinct and useful domestic breeds. That many of the breeds produced +by man have to a large extent the character of natural species, is +shown by the inextricable doubts whether very many of them are +varieties or aboriginal species. + +There is no obvious reason why the principles which have acted so +efficiently under domestication should not have acted under nature. In +the preservation of favoured individuals and races, during the +constantly-recurrent Struggle for Existence, we see the most powerful +and ever-acting means of selection. The struggle for existence +inevitably follows from the high geometrical ratio of increase which is +common to all organic beings. This high rate of increase is proved by +calculation, by the effects of a succession of peculiar seasons, and by +the results of naturalisation, as explained in the third chapter. More +individuals are born than can possibly survive. A grain in the balance +will determine which individual shall live and which shall die,—which +variety or species shall increase in number, and which shall decrease, +or finally become extinct. As the individuals +of the same species come in all respects into the closest competition +with each other, the struggle will generally be most severe between +them; it will be almost equally severe between the varieties of the +same species, and next in severity between the species of the same +genus. But the struggle will often be very severe between beings most +remote in the scale of nature. The slightest advantage in one being, at +any age or during any season, over those with which it comes into +competition, or better adaptation in however slight a degree to the +surrounding physical conditions, will turn the balance. + +With animals having separated sexes there will in most cases be a +struggle between the males for possession of the females. The most +vigorous individuals, or those which have most successfully struggled +with their conditions of life, will generally leave most progeny. But +success will often depend on having special weapons or means of +defence, or on the charms of the males; and the slightest advantage +will lead to victory. + +As geology plainly proclaims that each land has undergone great +physical changes, we might have expected that organic beings would have +varied under nature, in the same way as they generally have varied +under the changed conditions of domestication. And if there be any +variability under nature, it would be an unaccountable fact if natural +selection had not come into play. It has often been asserted, but the +assertion is quite incapable of proof, that the amount of variation +under nature is a strictly limited quantity. Man, though acting on +external characters alone and often capriciously, can produce within a +short period a great result by adding up mere individual differences in +his domestic productions; and every one admits that there are at least +individual differences in species under nature. But, besides such +differences, all naturalists +have admitted the existence of varieties, which they think sufficiently +distinct to be worthy of record in systematic works. No one can draw +any clear distinction between individual differences and slight +varieties; or between more plainly marked varieties and sub-species, +and species. Let it be observed how naturalists differ in the rank +which they assign to the many representative forms in Europe and North +America. + +If then we have under nature variability and a powerful agent always +ready to act and select, why should we doubt that variations in any way +useful to beings, under their excessively complex relations of life, +would be preserved, accumulated, and inherited? Why, if man can by +patience select variations most useful to himself, should nature fail +in selecting variations useful, under changing conditions of life, to +her living products? What limit can be put to this power, acting during +long ages and rigidly scrutinising the whole constitution, structure, +and habits of each creature,—favouring the good and rejecting the bad? +I can see no limit to this power, in slowly and beautifully adapting +each form to the most complex relations of life. The theory of natural +selection, even if we looked no further than this, seems to me to be in +itself probable. I have already recapitulated, as fairly as I could, +the opposed difficulties and objections: now let us turn to the special +facts and arguments in favour of the theory. + +On the view that species are only strongly marked and permanent +varieties, and that each species first existed as a variety, we can see +why it is that no line of demarcation can be drawn between species, +commonly supposed to have been produced by special acts of creation, +and varieties which are acknowledged to have been produced by secondary +laws. On this same view we can understand how it is that in each region +where many species of a genus have been produced, and where they now +flourish, these same species should present many varieties; for where +the manufactory of species has been active, we might expect, as a +general rule, to find it still in action; and this is the case if +varieties be incipient species. Moreover, the species of the larger +genera, which afford the greater number of varieties or incipient +species, retain to a certain degree the character of varieties; for +they differ from each other by a less amount of difference than do the +species of smaller genera. The closely allied species also of the +larger genera apparently have restricted ranges, and they are clustered +in little groups round other species—in which respects they resemble +varieties. These are strange relations on the view of each species +having been independently created, but are intelligible if all species +first existed as varieties. + +As each species tends by its geometrical ratio of reproduction to +increase inordinately in number; and as the modified descendants of +each species will be enabled to increase by so much the more as they +become more diversified in habits and structure, so as to be enabled to +seize on many and widely different places in the economy of nature, +there will be a constant tendency in natural selection to preserve the +most divergent offspring of any one species. Hence during a +long-continued course of modification, the slight differences, +characteristic of varieties of the same species, tend to be augmented +into the greater differences characteristic of species of the same +genus. New and improved varieties will inevitably supplant and +exterminate the older, less improved and intermediate varieties; and +thus species are rendered to a large extent defined and distinct +objects. Dominant species belonging to the larger groups tend to give +birth to new and dominant +forms; so that each large group tends to become still larger, and at +the same time more divergent in character. But as all groups cannot +thus succeed in increasing in size, for the world would not hold them, +the more dominant groups beat the less dominant. This tendency in the +large groups to go on increasing in size and diverging in character, +together with the almost inevitable contingency of much extinction, +explains the arrangement of all the forms of life, in groups +subordinate to groups, all within a few great classes, which we now see +everywhere around us, and which has prevailed throughout all time. This +grand fact of the grouping of all organic beings seems to me utterly +inexplicable on the theory of creation. + +As natural selection acts solely by accumulating slight, successive, +favourable variations, it can produce no great or sudden modification; +it can act only by very short and slow steps. Hence the canon of +“Natura non facit saltum,” which every fresh addition to our knowledge +tends to make more strictly correct, is on this theory simply +intelligible. We can plainly see why nature is prodigal in variety, +though niggard in innovation. But why this should be a law of nature if +each species has been independently created, no man can explain. + +Many other facts are, as it seems to me, explicable on this theory. How +strange it is that a bird, under the form of woodpecker, should have +been created to prey on insects on the ground; that upland geese, which +never or rarely swim, should have been created with webbed feet; that a +thrush should have been created to dive and feed on sub-aquatic +insects; and that a petrel should have been created with habits and +structure fitting it for the life of an auk or grebe! and so on in +endless other cases. But on the view of each +species constantly trying to increase in number, with natural selection +always ready to adapt the slowly varying descendants of each to any +unoccupied or ill-occupied place in nature, these facts cease to be +strange, or perhaps might even have been anticipated. + +As natural selection acts by competition, it adapts the inhabitants of +each country only in relation to the degree of perfection of their +associates; so that we need feel no surprise at the inhabitants of any +one country, although on the ordinary view supposed to have been +specially created and adapted for that country, being beaten and +supplanted by the naturalised productions from another land. Nor ought +we to marvel if all the contrivances in nature be not, as far as we can +judge, absolutely perfect; and if some of them be abhorrent to our +ideas of fitness. We need not marvel at the sting of the bee causing +the bee’s own death; at drones being produced in such vast numbers for +one single act, and being then slaughtered by their sterile sisters; at +the astonishing waste of pollen by our fir-trees; at the instinctive +hatred of the queen bee for her own fertile daughters; at ichneumonidæ +feeding within the live bodies of caterpillars; and at other such +cases. The wonder indeed is, on the theory of natural selection, that +more cases of the want of absolute perfection have not been observed. + +The complex and little known laws governing variation are the same, as +far as we can see, with the laws which have governed the production of +so-called specific forms. In both cases physical conditions seem to +have produced but little direct effect; yet when varieties enter any +zone, they occasionally assume some of the characters of the species +proper to that zone. In both varieties and species, use and disuse seem +to have produced some effect; for it is difficult to resist this +conclusion +when we look, for instance, at the logger-headed duck, which has wings +incapable of flight, in nearly the same condition as in the domestic +duck; or when we look at the burrowing tucutucu, which is occasionally +blind, and then at certain moles, which are habitually blind and have +their eyes covered with skin; or when we look at the blind animals +inhabiting the dark caves of America and Europe. In both varieties and +species correlation of growth seems to have played a most important +part, so that when one part has been modified other parts are +necessarily modified. In both varieties and species reversions to +long-lost characters occur. How inexplicable on the theory of creation +is the occasional appearance of stripes on the shoulder and legs of the +several species of the horse-genus and in their hybrids! How simply is +this fact explained if we believe that these species have descended +from a striped progenitor, in the same manner as the several domestic +breeds of pigeon have descended from the blue and barred rock-pigeon! + +On the ordinary view of each species having been independently created, +why should the specific characters, or those by which the species of +the same genus differ from each other, be more variable than the +generic characters in which they all agree? Why, for instance, should +the colour of a flower be more likely to vary in any one species of a +genus, if the other species, supposed to have been created +independently, have differently coloured flowers, than if all the +species of the genus have the same coloured flowers? If species are +only well-marked varieties, of which the characters have become in a +high degree permanent, we can understand this fact; for they have +already varied since they branched off from a common progenitor in +certain characters, by which they have come to be specifically distinct +from each other; +and therefore these same characters would be more likely still to be +variable than the generic characters which have been inherited without +change for an enormous period. It is inexplicable on the theory of +creation why a part developed in a very unusual manner in any one +species of a genus, and therefore, as we may naturally infer, of great +importance to the species, should be eminently liable to variation; +but, on my view, this part has undergone, since the several species +branched off from a common progenitor, an unusual amount of variability +and modification, and therefore we might expect this part generally to +be still variable. But a part may be developed in the most unusual +manner, like the wing of a bat, and yet not be more variable than any +other structure, if the part be common to many subordinate forms, that +is, if it has been inherited for a very long period; for in this case +it will have been rendered constant by long-continued natural +selection. + +Glancing at instincts, marvellous as some are, they offer no greater +difficulty than does corporeal structure on the theory of the natural +selection of successive, slight, but profitable modifications. We can +thus understand why nature moves by graduated steps in endowing +different animals of the same class with their several instincts. I +have attempted to show how much light the principle of gradation throws +on the admirable architectural powers of the hive-bee. Habit no doubt +sometimes comes into play in modifying instincts; but it certainly is +not indispensable, as we see, in the case of neuter insects, which +leave no progeny to inherit the effects of long-continued habit. On the +view of all the species of the same genus having descended from a +common parent, and having inherited much in common, we can understand +how it is that allied species, when placed under considerably different +conditions of life, +yet should follow nearly the same instincts; why the thrush of South +America, for instance, lines her nest with mud like our British +species. On the view of instincts having been slowly acquired through +natural selection we need not marvel at some instincts being apparently +not perfect and liable to mistakes, and at many instincts causing other +animals to suffer. + +If species be only well-marked and permanent varieties, we can at once +see why their crossed offspring should follow the same complex laws in +their degrees and kinds of resemblance to their parents,—in being +absorbed into each other by successive crosses, and in other such +points,—as do the crossed offspring of acknowledged varieties. On the +other hand, these would be strange facts if species have been +independently created, and varieties have been produced by secondary +laws. + +If we admit that the geological record is imperfect in an extreme +degree, then such facts as the record gives, support the theory of +descent with modification. New species have come on the stage slowly +and at successive intervals; and the amount of change, after equal +intervals of time, is widely different in different groups. The +extinction of species and of whole groups of species, which has played +so conspicuous a part in the history of the organic world, almost +inevitably follows on the principle of natural selection; for old forms +will be supplanted by new and improved forms. Neither single species +nor groups of species reappear when the chain of ordinary generation +has once been broken. The gradual diffusion of dominant forms, with the +slow modification of their descendants, causes the forms of life, after +long intervals of time, to appear as if they had changed simultaneously +throughout the world. The fact of the fossil remains of each formation +being in some degree intermediate in character between the +fossils in the formations above and below, is simply explained by their +intermediate position in the chain of descent. The grand fact that all +extinct organic beings belong to the same system with recent beings, +falling either into the same or into intermediate groups, follows from +the living and the extinct being the offspring of common parents. As +the groups which have descended from an ancient progenitor have +generally diverged in character, the progenitor with its early +descendants will often be intermediate in character in comparison with +its later descendants; and thus we can see why the more ancient a +fossil is, the oftener it stands in some degree intermediate between +existing and allied groups. Recent forms are generally looked at as +being, in some vague sense, higher than ancient and extinct forms; and +they are in so far higher as the later and more improved forms have +conquered the older and less improved organic beings in the struggle +for life. Lastly, the law of the long endurance of allied forms on the +same continent,—of marsupials in Australia, of edentata in America, and +other such cases,—is intelligible, for within a confined country, the +recent and the extinct will naturally be allied by descent. + +Looking to geographical distribution, if we admit that there has been +during the long course of ages much migration from one part of the +world to another, owing to former climatal and geographical changes and +to the many occasional and unknown means of dispersal, then we can +understand, on the theory of descent with modification, most of the +great leading facts in Distribution. We can see why there should be so +striking a parallelism in the distribution of organic beings throughout +space, and in their geological succession throughout time; for in both +cases the beings have been connected by the bond of ordinary +generation, and the means of +modification have been the same. We see the full meaning of the +wonderful fact, which must have struck every traveller, namely, that on +the same continent, under the most diverse conditions, under heat and +cold, on mountain and lowland, on deserts and marshes, most of the +inhabitants within each great class are plainly related; for they will +generally be descendants of the same progenitors and early colonists. +On this same principle of former migration, combined in most cases with +modification, we can understand, by the aid of the Glacial period, the +identity of some few plants, and the close alliance of many others, on +the most distant mountains, under the most different climates; and +likewise the close alliance of some of the inhabitants of the sea in +the northern and southern temperate zones, though separated by the +whole intertropical ocean. Although two areas may present the same +physical conditions of life, we need feel no surprise at their +inhabitants being widely different, if they have been for a long period +completely separated from each other; for as the relation of organism +to organism is the most important of all relations, and as the two +areas will have received colonists from some third source or from each +other, at various periods and in different proportions, the course of +modification in the two areas will inevitably be different. + +On this view of migration, with subsequent modification, we can see why +oceanic islands should be inhabited by few species, but of these, that +many should be peculiar. We can clearly see why those animals which +cannot cross wide spaces of ocean, as frogs and terrestrial mammals, +should not inhabit oceanic islands; and why, on the other hand, new and +peculiar species of bats, which can traverse the ocean, should so often +be found on islands far distant from any continent. Such facts +as the presence of peculiar species of bats, and the absence of all +other mammals, on oceanic islands, are utterly inexplicable on the +theory of independent acts of creation. + +The existence of closely allied or representative species in any two +areas, implies, on the theory of descent with modification, that the +same parents formerly inhabited both areas; and we almost invariably +find that wherever many closely allied species inhabit two areas, some +identical species common to both still exist. Wherever many closely +allied yet distinct species occur, many doubtful forms and varieties of +the same species likewise occur. It is a rule of high generality that +the inhabitants of each area are related to the inhabitants of the +nearest source whence immigrants might have been derived. We see this +in nearly all the plants and animals of the Galapagos archipelago, of +Juan Fernandez, and of the other American islands being related in the +most striking manner to the plants and animals of the neighbouring +American mainland; and those of the Cape de Verde archipelago and other +African islands to the African mainland. It must be admitted that these +facts receive no explanation on the theory of creation. + +The fact, as we have seen, that all past and present organic beings +constitute one grand natural system, with group subordinate to group, +and with extinct groups often falling in between recent groups, is +intelligible on the theory of natural selection with its contingencies +of extinction and divergence of character. On these same principles we +see how it is, that the mutual affinities of the species and genera +within each class are so complex and circuitous. We see why certain +characters are far more serviceable than others for classification;—why +adaptive characters, though of paramount importance to the being, are +of hardly any +importance in classification; why characters derived from rudimentary +parts, though of no service to the being, are often of high +classificatory value; and why embryological characters are the most +valuable of all. The real affinities of all organic beings are due to +inheritance or community of descent. The natural system is a +genealogical arrangement, in which we have to discover the lines of +descent by the most permanent characters, however slight their vital +importance may be. + +The framework of bones being the same in the hand of a man, wing of a +bat, fin of the porpoise, and leg of the horse,—the same number of +vertebræ forming the neck of the giraffe and of the elephant,—and +innumerable other such facts, at once explain themselves on the theory +of descent with slow and slight successive modifications. The +similarity of pattern in the wing and leg of a bat, though used for +such different purpose,—in the jaws and legs of a crab,—in the petals, +stamens, and pistils of a flower, is likewise intelligible on the view +of the gradual modification of parts or organs, which were alike in the +early progenitor of each class. On the principle of successive +variations not always supervening at an early age, and being inherited +at a corresponding not early period of life, we can clearly see why the +embryos of mammals, birds, reptiles, and fishes should be so closely +alike, and should be so unlike the adult forms. We may cease marvelling +at the embryo of an air-breathing mammal or bird having branchial slits +and arteries running in loops, like those in a fish which has to +breathe the air dissolved in water, by the aid of well-developed +branchiæ. + +Disuse, aided sometimes by natural selection, will often tend to reduce +an organ, when it has become useless by changed habits or under changed +conditions +of life; and we can clearly understand on this view the meaning of +rudimentary organs. But disuse and selection will generally act on each +creature, when it has come to maturity and has to play its full part in +the struggle for existence, and will thus have little power of acting +on an organ during early life; hence the organ will not be much reduced +or rendered rudimentary at this early age. The calf, for instance, has +inherited teeth, which never cut through the gums of the upper jaw, +from an early progenitor having well-developed teeth; and we may +believe, that the teeth in the mature animal were reduced, during +successive generations, by disuse or by the tongue and palate having +been fitted by natural selection to browse without their aid; whereas +in the calf, the teeth have been left untouched by selection or disuse, +and on the principle of inheritance at corresponding ages have been +inherited from a remote period to the present day. On the view of each +organic being and each separate organ having been specially created, +how utterly inexplicable it is that parts, like the teeth in the +embryonic calf or like the shrivelled wings under the soldered +wing-covers of some beetles, should thus so frequently bear the plain +stamp of inutility! Nature may be said to have taken pains to reveal, +by rudimentary organs and by homologous structures, her scheme of +modification, which it seems that we wilfully will not understand. + +I have now recapitulated the chief facts and considerations which have +thoroughly convinced me that species have changed, and are still slowly +changing by the preservation and accumulation of successive slight +favourable variations. Why, it may be asked, have all the most eminent +living naturalists and geologists rejected this view of the mutability +of species? It cannot be +asserted that organic beings in a state of nature are subject to no +variation; it cannot be proved that the amount of variation in the +course of long ages is a limited quantity; no clear distinction has +been, or can be, drawn between species and well-marked varieties. It +cannot be maintained that species when intercrossed are invariably +sterile, and varieties invariably fertile; or that sterility is a +special endowment and sign of creation. The belief that species were +immutable productions was almost unavoidable as long as the history of +the world was thought to be of short duration; and now that we have +acquired some idea of the lapse of time, we are too apt to assume, +without proof, that the geological record is so perfect that it would +have afforded us plain evidence of the mutation of species, if they had +undergone mutation. + +But the chief cause of our natural unwillingness to admit that one +species has given birth to other and distinct species, is that we are +always slow in admitting any great change of which we do not see the +intermediate steps. The difficulty is the same as that felt by so many +geologists, when Lyell first insisted that long lines of inland cliffs +had been formed, and great valleys excavated, by the slow action of the +coast-waves. The mind cannot possibly grasp the full meaning of the +term of a hundred million years; it cannot add up and perceive the full +effects of many slight variations, accumulated during an almost +infinite number of generations. + +Although I am fully convinced of the truth of the views given in this +volume under the form of an abstract, I by no means expect to convince +experienced naturalists whose minds are stocked with a multitude of +facts all viewed, during a long course of years, from a point of view +directly opposite to mine. It is so easy +to hide our ignorance under such expressions as the “plan of creation,” +“unity of design,” etc., and to think that we give an explanation when +we only restate a fact. Any one whose disposition leads him to attach +more weight to unexplained difficulties than to the explanation of a +certain number of facts will certainly reject my theory. A few +naturalists, endowed with much flexibility of mind, and who have +already begun to doubt on the immutability of species, may be +influenced by this volume; but I look with confidence to the future, to +young and rising naturalists, who will be able to view both sides of +the question with impartiality. Whoever is led to believe that species +are mutable will do good service by conscientiously expressing his +conviction; for only thus can the load of prejudice by which this +subject is overwhelmed be removed. + +Several eminent naturalists have of late published their belief that a +multitude of reputed species in each genus are not real species; but +that other species are real, that is, have been independently created. +This seems to me a strange conclusion to arrive at. They admit that a +multitude of forms, which till lately they themselves thought were +special creations, and which are still thus looked at by the majority +of naturalists, and which consequently have every external +characteristic feature of true species,—they admit that these have been +produced by variation, but they refuse to extend the same view to other +and very slightly different forms. Nevertheless they do not pretend +that they can define, or even conjecture, which are the created forms +of life, and which are those produced by secondary laws. They admit +variation as a _vera causa_ in one case, they arbitrarily reject it in +another, without assigning any distinction in the two cases. The day +will come when this will be given as a curious illustration of +the blindness of preconceived opinion. These authors seem no more +startled at a miraculous act of creation than at an ordinary birth. But +do they really believe that at innumerable periods in the earth’s +history certain elemental atoms have been commanded suddenly to flash +into living tissues? Do they believe that at each supposed act of +creation one individual or many were produced? Were all the infinitely +numerous kinds of animals and plants created as eggs or seed, or as +full grown? and in the case of mammals, were they created bearing the +false marks of nourishment from the mother’s womb? Although naturalists +very properly demand a full explanation of every difficulty from those +who believe in the mutability of species, on their own side they ignore +the whole subject of the first appearance of species in what they +consider reverent silence. + +It may be asked how far I extend the doctrine of the modification of +species. The question is difficult to answer, because the more distinct +the forms are which we may consider, by so much the arguments fall away +in force. But some arguments of the greatest weight extend very far. +All the members of whole classes can be connected together by chains of +affinities, and all can be classified on the same principle, in groups +subordinate to groups. Fossil remains sometimes tend to fill up very +wide intervals between existing orders. Organs in a rudimentary +condition plainly show that an early progenitor had the organ in a +fully developed state; and this in some instances necessarily implies +an enormous amount of modification in the descendants. Throughout whole +classes various structures are formed on the same pattern, and at an +embryonic age the species closely resemble each other. Therefore I +cannot doubt that the theory of descent with modification +embraces all the members of the same class. I believe that animals have +descended from at most only four or five progenitors, and plants from +an equal or lesser number. + +Analogy would lead me one step further, namely, to the belief that all +animals and plants have descended from some one prototype. But analogy +may be a deceitful guide. Nevertheless all living things have much in +common, in their chemical composition, their germinal vesicles, their +cellular structure, and their laws of growth and reproduction. We see +this even in so trifling a circumstance as that the same poison often +similarly affects plants and animals; or that the poison secreted by +the gall-fly produces monstrous growths on the wild rose or oak-tree. +Therefore I should infer from analogy that probably all the organic +beings which have ever lived on this earth have descended from some one +primordial form, into which life was first breathed. + +When the views entertained in this volume on the origin of species, or +when analogous views are generally admitted, we can dimly foresee that +there will be a considerable revolution in natural history. +Systematists will be able to pursue their labours as at present; but +they will not be incessantly haunted by the shadowy doubt whether this +or that form be in essence a species. This I feel sure, and I speak +after experience, will be no slight relief. The endless disputes +whether or not some fifty species of British brambles are true species +will cease. Systematists will have only to decide (not that this will +be easy) whether any form be sufficiently constant and distinct from +other forms, to be capable of definition; and if definable, whether the +differences be sufficiently important to deserve a specific name. This +latter point will become a far more essential consideration +than it is at present; for differences, however slight, between any two +forms, if not blended by intermediate gradations, are looked at by most +naturalists as sufficient to raise both forms to the rank of species. +Hereafter we shall be compelled to acknowledge that the only +distinction between species and well-marked varieties is, that the +latter are known, or believed, to be connected at the present day by +intermediate gradations, whereas species were formerly thus connected. +Hence, without quite rejecting the consideration of the present +existence of intermediate gradations between any two forms, we shall be +led to weigh more carefully and to value higher the actual amount of +difference between them. It is quite possible that forms now generally +acknowledged to be merely varieties may hereafter be thought worthy of +specific names, as with the primrose and cowslip; and in this case +scientific and common language will come into accordance. In short, we +shall have to treat species in the same manner as those naturalists +treat genera, who admit that genera are merely artificial combinations +made for convenience. This may not be a cheering prospect; but we shall +at least be freed from the vain search for the undiscovered and +undiscoverable essence of the term species. + +The other and more general departments of natural history will rise +greatly in interest. The terms used by naturalists of affinity, +relationship, community of type, paternity, morphology, adaptive +characters, rudimentary and aborted organs, etc., will cease to be +metaphorical, and will have a plain signification. When we no longer +look at an organic being as a savage looks at a ship, as at something +wholly beyond his comprehension; when we regard every production of +nature as one which has had a history; when we contemplate every +complex structure +and instinct as the summing up of many contrivances, each useful to the +possessor, nearly in the same way as when we look at any great +mechanical invention as the summing up of the labour, the experience, +the reason, and even the blunders of numerous workmen; when we thus +view each organic being, how far more interesting, I speak from +experience, will the study of natural history become! + +A grand and almost untrodden field of inquiry will be opened, on the +causes and laws of variation, on correlation of growth, on the effects +of use and disuse, on the direct action of external conditions, and so +forth. The study of domestic productions will rise immensely in value. +A new variety raised by man will be a far more important and +interesting subject for study than one more species added to the +infinitude of already recorded species. Our classifications will come +to be, as far as they can be so made, genealogies; and will then truly +give what may be called the plan of creation. The rules for classifying +will no doubt become simpler when we have a definite object in view. We +possess no pedigrees or armorial bearings; and we have to discover and +trace the many diverging lines of descent in our natural genealogies, +by characters of any kind which have long been inherited. Rudimentary +organs will speak infallibly with respect to the nature of long-lost +structures. Species and groups of species, which are called aberrant, +and which may fancifully be called living fossils, will aid us in +forming a picture of the ancient forms of life. Embryology will reveal +to us the structure, in some degree obscured, of the prototypes of each +great class. + +When we can feel assured that all the individuals of the same species, +and all the closely allied species of most genera, have within a not +very remote period descended +from one parent, and have migrated from some one birthplace; and when +we better know the many means of migration, then, by the light which +geology now throws, and will continue to throw, on former changes of +climate and of the level of the land, we shall surely be enabled to +trace in an admirable manner the former migrations of the inhabitants +of the whole world. Even at present, by comparing the differences of +the inhabitants of the sea on the opposite sides of a continent, and +the nature of the various inhabitants of that continent in relation to +their apparent means of immigration, some light can be thrown on +ancient geography. + +The noble science of Geology loses glory from the extreme imperfection +of the record. The crust of the earth with its embedded remains must +not be looked at as a well-filled museum, but as a poor collection made +at hazard and at rare intervals. The accumulation of each great +fossiliferous formation will be recognised as having depended on an +unusual concurrence of circumstances, and the blank intervals between +the successive stages as having been of vast duration. But we shall be +able to gauge with some security the duration of these intervals by a +comparison of the preceding and succeeding organic forms. We must be +cautious in attempting to correlate as strictly contemporaneous two +formations, which include few identical species, by the general +succession of their forms of life. As species are produced and +exterminated by slowly acting and still existing causes, and not by +miraculous acts of creation and by catastrophes; and as the most +important of all causes of organic change is one which is almost +independent of altered and perhaps suddenly altered physical +conditions, namely, the mutual relation of organism to organism,—the +improvement of one being entailing the improvement or the extermination +of +others; it follows, that the amount of organic change in the fossils of +consecutive formations probably serves as a fair measure of the lapse +of actual time. A number of species, however, keeping in a body might +remain for a long period unchanged, whilst within this same period, +several of these species, by migrating into new countries and coming +into competition with foreign associates, might become modified; so +that we must not overrate the accuracy of organic change as a measure +of time. During early periods of the earth’s history, when the forms of +life were probably fewer and simpler, the rate of change was probably +slower; and at the first dawn of life, when very few forms of the +simplest structure existed, the rate of change may have been slow in an +extreme degree. The whole history of the world, as at present known, +although of a length quite incomprehensible by us, will hereafter be +recognised as a mere fragment of time, compared with the ages which +have elapsed since the first creature, the progenitor of innumerable +extinct and living descendants, was created. + +In the distant future I see open fields for far more important +researches. Psychology will be based on a new foundation, that of the +necessary acquirement of each mental power and capacity by gradation. +Light will be thrown on the origin of man and his history. + +Authors of the highest eminence seem to be fully satisfied with the +view that each species has been independently created. To my mind it +accords better with what we know of the laws impressed on matter by the +Creator, that the production and extinction of the past and present +inhabitants of the world should have been due to secondary causes, like +those determining the birth and death of the individual. When I view +all beings not as special creations, but as the lineal descendants of +some few beings which lived long before the +first bed of the Silurian system was deposited, they seem to me to +become ennobled. Judging from the past, we may safely infer that not +one living species will transmit its unaltered likeness to a distant +futurity. And of the species now living very few will transmit progeny +of any kind to a far distant futurity; for the manner in which all +organic beings are grouped, shows that the greater number of species of +each genus, and all the species of many genera, have left no +descendants, but have become utterly extinct. We can so far take a +prophetic glance into futurity as to foretel that it will be the common +and widely-spread species, belonging to the larger and dominant groups, +which will ultimately prevail and procreate new and dominant species. +As all the living forms of life are the lineal descendants of those +which lived long before the Silurian epoch, we may feel certain that +the ordinary succession by generation has never once been broken, and +that no cataclysm has desolated the whole world. Hence we may look with +some confidence to a secure future of equally inappreciable length. And +as natural selection works solely by and for the good of each being, +all corporeal and mental endowments will tend to progress towards +perfection. + +It is interesting to contemplate an entangled bank, clothed with many +plants of many kinds, with birds singing on the bushes, with various +insects flitting about, and with worms crawling through the damp earth, +and to reflect that these elaborately constructed forms, so different +from each other, and dependent on each other in so complex a manner, +have all been produced by laws acting around us. These laws, taken in +the largest sense, being Growth with Reproduction; Inheritance which is +almost implied by reproduction; Variability from the indirect and +direct action of the external conditions +of life, and from use and disuse; a Ratio of Increase so high as to +lead to a Struggle for Life, and as a consequence to Natural Selection, +entailing Divergence of Character and the Extinction of less-improved +forms. Thus, from the war of nature, from famine and death, the most +exalted object which we are capable of conceiving, namely, the +production of the higher animals, directly follows. There is grandeur +in this view of life, with its several powers, having been originally +breathed into a few forms or into one; and that, whilst this planet has +gone cycling on according to the fixed law of gravity, from so simple a +beginning endless forms most beautiful and most wonderful have been, +and are being, evolved. + + +This four-dimensional “world” bears a close similarity to the +three-dimensional “space” of (Euclidean) analytical geometry. If we +introduce into the latter a new Cartesian co-ordinate system (_x′_1, +_x′_2, _x′_3) with the same origin, then _x′_1, _x′_2, _x′_3, are +linear homogeneous functions of _x_1, _x_2, _x_3 which identically +satisfy the equation + +_x_1′2 + _x_2′2 + _x_3′2 = _x_12 + _x_22 + _x_32 + +The analogy with (12) is a complete one. We can regard Minkowski’s +“world” in a formal manner as a four-dimensional Euclidean space (with +an imaginary time coordinate); the Lorentz transformation corresponds +to a “rotation” of the co-ordinate system in the four-dimensional +“world.” + +From a systematic theoretical point of view, we may imagine the process +of evolution of an empirical science to be a continuous process of +induction. Theories are evolved and are expressed in short compass as +statements of a large number of individual observations in the form of +empirical laws, from which the general laws can be ascertained by +comparison. Regarded in this way, the development of a science bears +some resemblance to the compilation of a classified catalogue. It is, +as it were, a purely empirical enterprise. + +But this point of view by no means embraces the whole of the actual +process; for it slurs over the important part played by intuition and +deductive thought in the development of an exact science. As soon as a +science has emerged from its initial stages, theoretical advances are +no longer achieved merely by a process of arrangement. Guided by +empirical data, the investigator rather develops a system of thought +which, in general, is built up logically from a small number of +fundamental assumptions, the so-called axioms. We call such a system of +thought a _theory_. The theory finds the justification for its +existence in the fact that it correlates a large number of single +observations, and it is just here that the “truth” of the theory lies. + +Corresponding to the same complex of empirical data, there may be +several theories, which differ from one another to a considerable +extent. But as regards the deductions from the theories which are +capable of being tested, the agreement between the theories may be so +complete that it becomes difficult to find any deductions in which the +two theories differ from each other. As an example, a case of general +interest is available in the province of biology, in the Darwinian +theory of the development of species by selection in the struggle for +existence, and in the theory of development which is based on the +hypothesis of the hereditary transmission of acquired characters. + +We have another instance of far-reaching agreement between the +deductions from two theories in Newtonian mechanics on the one hand, +and the general theory of relativity on the other. This agreement goes +so far, that up to the present we have been able to find only a few +deductions from the general theory of relativity which are capable of +investigation, and to which the physics of pre-relativity days does not +also lead, and this despite the profound difference in the fundamental +assumptions of the two theories. In what follows, we shall again +consider these important deductions, and we shall also discuss the +empirical evidence appertaining to them which has hitherto been +obtained. + +(_a_) Motion of the Perihelion of Mercury + +According to Newtonian mechanics and Newton’s law of gravitation, a +planet which is revolving round the sun would describe an ellipse round +the latter, or, more correctly, round the common centre of gravity of +the sun and the planet. In such a system, the sun, or the common centre +of gravity, lies in one of the foci of the orbital ellipse in such a +manner that, in the course of a planet-year, the distance sun-planet +grows from a minimum to a maximum, and then decreases again to a +minimum. If instead of Newton’s law we insert a somewhat different law +of attraction into the calculation, we find that, according to this new +law, the motion would still take place in such a manner that the +distance sun-planet exhibits periodic variations; but in this case the +angle described by the line joining sun and planet during such a period +(from perihelion—closest proximity to the sun—to perihelion) would +differ from 360°. The line of the orbit would not then be a closed one +but in the course of time it would fill up an annular part of the +orbital plane, viz. between the circle of least and the circle of +greatest distance of the planet from the sun. + +According also to the general theory of relativity, which differs of +course from the theory of Newton, a small variation from the +Newton-Kepler motion of a planet in its orbit should take place, and in +such away, that the angle described by the radius sun-planet between +one perhelion and the next should exceed that corresponding to one +complete revolution by an amount given by + +image051 + +(_N.B._—One complete revolution corresponds to the angle 2π in the +absolute angular measure customary in physics, and the above expression +given the amount by which the radius sun-planet exceeds this angle +during the interval between one perihelion and the next.) In this +expression _a_ represents the major semi-axis of the ellipse, _e_ its +eccentricity, _c_ the velocity of light, and _T_ the period of +revolution of the planet. Our result may also be stated as follows: +According to the general theory of relativity, the major axis of the +ellipse rotates round the sun in the same sense as the orbital motion +of the planet. Theory requires that this rotation should amount to 43 +seconds of arc per century for the planet Mercury, but for the other +Planets of our solar system its magnitude should be so small that it +would necessarily escape detection.[26] + + [26] Especially since the next planet Venus has an orbit that is + almost an exact circle, which makes it more difficult to locate the + perihelion with precision. + +In point of fact, astronomers have found that the theory of Newton does +not suffice to calculate the observed motion of Mercury with an +exactness corresponding to that of the delicacy of observation +attainable at the present time. After taking account of all the +disturbing influences exerted on Mercury by the remaining planets, it +was found (Leverrier: 1859; and Newcomb: 1895) that an unexplained +perihelial movement of the orbit of Mercury remained over, the amount +of which does not differ sensibly from the above mentioned +43 seconds +of arc per century. The uncertainty of the empirical result amounts to +a few seconds only. + +(_b_) Deflection of Light by a Gravitational Field + +image052 + +In Section XXII it has been already mentioned that according to the +general theory of relativity, a ray of light will experience a +curvature of its path when passing through a gravitational field, this +curvature being similar to that experienced by the path of a body which +is projected through a gravitational field. As a result of this theory, +we should expect that a ray of light which is passing close to a +heavenly body would be deviated towards the latter. For a ray of light +which passes the sun at a distance of Δ sun-radii from its centre, the +angle of deflection (α) should amount to + +image053 + +It may be added that, according to the theory, half of this deflection +is produced by the Newtonian field of attraction of the sun, and the +other half by the geometrical modification (“curvature”) of space +caused by the sun. + +This result admits of an experimental test by means of the photographic +registration of stars during a total eclipse of the sun. The only +reason why we must wait for a total eclipse is because at every other +time the atmosphere is so strongly illuminated by the light from the +sun that the stars situated near the sun’s disc are invisible. The +predicted effect can be seen clearly from the accompanying diagram. If +the sun (_S_) were not present, a star which is practically infinitely +distant would be seen in the direction _D_1, as observed front the +earth. But as a consequence of the deflection of light from the star by +the sun, the star will be seen in the direction _D_2, _i.e._ at a +somewhat greater distance from the centre of the sun than corresponds +to its real position. + +In practice, the question is tested in the following way. The stars in +the neighbourhood of the sun are photographed during a solar eclipse. + +In addition, a second photograph of the same stars is taken when the +sun is situated at another position in the sky, _i.e._ a few months +earlier or later. As compared with the standard photograph, the +positions of the stars on the eclipse-photograph ought to appear +displaced radially outwards (away from the centre of the sun) by an +amount corresponding to the angle _a_. + +We are indebted to the [British] Royal Society and to the Royal +Astronomical Society for the investigation of this important deduction. +Undaunted by the [first world] war and by difficulties of both a +material and a psychological nature aroused by the war, these societies +equipped two expeditions—to Sobral (Brazil), and to the island of +Principe (West Africa)—and sent several of Britain’s most celebrated +astronomers (Eddington, Cottingham, Crommelin, Davidson), in order to +obtain photographs of the solar eclipse of 29th May, 1919. The relative +discrepancies to be expected between the stellar photographs obtained +during the eclipse and the comparison photographs amounted to a few +hundredths of a millimetre only. Thus great accuracy was necessary in +making the adjustments required for the taking of the photographs, and +in their subsequent measurement. + +The results of the measurements confirmed the theory in a thoroughly +satisfactory manner. The rectangular components of the observed and of +the calculated deviations of the stars (in seconds of arc) are set +forth in the following table of results: + +image054 + +(_c_) Displacement of Spectral Lines Towards the Red + +In Section XXIII it has been shown that in a system _K′_ which is in +rotation with regard to a Galileian system _K_, clocks of identical +construction, and which are considered at rest with respect to the +rotating reference-body, go at rates which are dependent on the +positions of the clocks. We shall now examine this dependence +quantitatively. A clock, which is situated at a distance r from the +centre of the disc, has a velocity relative to _K_ which is given by + +_v_ = ω_r_, + +where ω represents the angular velocity of rotation of the disc _K′_ +with respect to _K_. If _v_0, represents the number of ticks of the +clock per unit time (“rate” of the clock) relative to _K_ when the +clock is at rest, then the “rate” of the clock (_v_) when it is moving +relative to _K_ with a velocity _v_, but at rest with respect to the +disc, will, in accordance with Section XII, be given by + +image055 + +or with sufficient accuracy by + +image056 + +This expression may also be stated in the following form: + +image057 + +If we represent the difference of potential of the centrifugal force +between the position of the clock and the centre of the disc by φ, +_i.e._ the work, considered negatively, which must be performed on the +unit of mass against the centrifugal force in order to transport it +from the position of the clock on the rotating disc to the centre of +the disc, then we have + +image058 + +From this it follows that + +image059 + +In the first place, we see from this expression that two clocks of +identical construction will go at different rates when situated at +different distances from the centre of the disc. This result is also +valid from the standpoint of an observer who is rotating with the disc. + +Now, as judged from the disc, the latter is in a gravitational field of +potential φ, hence the result we have obtained will hold quite +generally for gravitational fields. Furthermore, we can regard an atom +which is emitting spectral lines as a clock, so that the following +statement will hold: + +_An atom absorbs or emits light of a frequency which is dependent on +the potential of the gravitational field in which it is situated._ + +The frequency of an atom situated on the surface of a heavenly body +will be somewhat less than the frequency of an atom of the same element +which is situated in free space (or on the surface of a smaller +celestial body). + +Now φ = – _K (M/r)_, where _K_ is Newton’s constant of gravitation, and +_M_ is the mass of the heavenly body. Thus a displacement towards the +red ought to take place for spectral lines produced at the surface of +stars as compared with the spectral lines of the same element produced +at the surface of the earth, the amount of this displacement being + +image060 + +For the sun, the displacement towards the red predicted by theory +amounts to about two millionths of the wave-length. A trustworthy +calculation is not possible in the case of the stars, because in +general neither the mass _M_ nor the radius _r_ are known. + +It is an open question whether or not this effect exists, and at the +present time (1920) astronomers are working with great zeal towards the +solution. Owing to the smallness of the effect in the case of the sun, +it is difficult to form an opinion as to its existence. Whereas Grebe +and Bachem (Bonn), as a result of their own measurements and those of +Evershed and Schwarzschild on the cyanogen bands, have placed the +existence of the effect almost beyond doubt, while other investigators, +particularly St. John, have been led to the opposite opinion in +consequence of their measurements. + +Mean displacements of lines towards the less refrangible end of the +spectrum are certainly revealed by statistical investigations of the +fixed stars; but up to the present the examination of the available +data does not allow of any definite decision being arrived at, as to +whether or not these displacements are to be referred in reality to the +effect of gravitation. The results of observation have been collected +together, and discussed in detail from the standpoint of the question +which has been engaging our attention here, in a paper by E. Freundlich +entitled “Zur Prüfung der allgemeinen Relativitäts-Theorie” (_Die +Naturwissenschaften_, 1919, No. 35, p. 520: Julius Springer, Berlin). + +At all events, a definite decision will be reached during the next few +years. If the displacement of spectral lines towards the red by the +gravitational potential does not exist, then the general theory of +relativity will be untenable. On the other hand, if the cause of the +displacement of spectral lines be definitely traced to the +gravitational potential, then the study of this displacement will +furnish us with important information as to the mass of the heavenly +bodies.[27] + + [27] The displacement of spectral lines towards the red end of the + spectrum was definitely established by Adams in 1924, by observations + on the dense companion of Sirius, for which the effect is about thirty + times greater than for the Sun. R.W.L.—translator + +Since the publication of the first edition of this little book, our +knowledge about the structure of space in the large (“cosmological +problem”) has had an important development, which ought to be mentioned +even in a popular presentation of the subject. + +My original considerations on the subject were based on two hypotheses: + +(1) There exists an average density of matter in the whole of space +which is everywhere the same and different from zero. + +(2) The magnitude (“radius”) of space is independent of time. + +Both these hypotheses proved to be consistent, according to the general +theory of relativity, but only after a hypothetical term was added to +the field equations, a term which was not required by the theory as +such nor did it seem natural from a theoretical point of view +(“cosmological term of the field equations”). + +Hypothesis (2) appeared unavoidable to me at the time, since I thought +that one would get into bottomless speculations if one departed from +it. + +However, already in the ’twenties, the Russian mathematician Friedman +showed that a different hypothesis was natural from a purely +theoretical point of view. He realized that it was possible to preserve +hypothesis (1) without introducing the less natural cosmological term +into the field equations of gravitation, if one was ready to drop +hypothesis (2). Namely, the original field equations admit a solution +in which the “world radius” depends on time (expanding space). In that +sense one can say, according to Friedman, that the theory demands an +expansion of space. + +A few years later Hubble showed, by a special investigation of the +extra-galactic nebulae (“milky ways”), that the spectral lines emitted +showed a red shift which increased regularly with the distance of the +nebulae. This can be interpreted in regard to our present knowledge +only in the sense of Doppler’s principle, as an expansive motion of the +system of stars in the large—as required, according to Friedman, by the +field equations of gravitation. Hubble’s discovery can, therefore, be +considered to some extent as a confirmation of the theory. + +There does arise, however, a strange difficulty. The interpretation of +the galactic line-shift discovered by Hubble as an expansion (which can +hardly be doubted from a theoretical point of view), leads to an origin +of this expansion which lies “only” about 109 years ago, while physical +astronomy makes it appear likely that the development of individual +stars and systems of stars takes considerably longer. It is in no way +known how this incongruity is to be overcome. + +I further want to remark that the theory of expanding space, together +with the empirical data of astronomy, permit no decision to be reached +about the finite or infinite character of (three-dimensional) space, +while the original “static” hypothesis of space yielded the closure +(finiteness) of space. diff --git a/docs/EXPERIMENT_RESULTS.md b/docs/EXPERIMENT_RESULTS.md new file mode 100644 index 0000000..e95654d --- /dev/null +++ b/docs/EXPERIMENT_RESULTS.md @@ -0,0 +1,581 @@ +# Experiment results + +Every number in a table below is generated by `scripts/build_experiment_report.py` +from JSON records in `experiments/records/`. Sections between +`BEGIN GENERATED` / `END GENERATED` markers are overwritten on rebuild; prose +outside them is hand-written and cites those tables. + +## Protocols and comparability + +| Protocol | Corpus | Splits | Used in | +| --- | --- | --- | --- | +| **science-v1** (legacy, unchanged) | `data/science/`: Darwin + Einstein, Gutenberg boilerplate removed, nothing else | train / validation (validation also selects checkpoints and is reported) | Part A, published September 2026 results | +| **science-v2** | `data/science-v2/`: same two books with front/back matter and non-prose removed by deterministic rules; Faraday as an untouched test book | train / validation (selection) / test (final evaluation only) | Part B | + +**Perplexities are never compared across protocols.** v2 removes the Darwin index +and other non-prose text and uses a different validation set, so a lower v2 number +is not an improvement over v1. Timing experiments (02, 03 and the training-step +benchmark) use random weights and synthetic tokens and do not depend on either corpus. + +Hardware for every record: one Apple M4 MacBook (16 GB), CPU with 4 torch threads +and MPS; Python 3.12.14, torch 2.14.1. No CUDA hardware was available; CUDA code +paths are untested. + +## Summary + +### Part A · systems experiments and science-v1 + +1. **The three attention implementations compute the same function.** In float32, + legacy MHA, SDPA and the explicit reference give identical held-out perplexity to + four decimals, 100% top-1 agreement with a float64 reference run and identical greedy + text, on CPU and MPS, for three trained seeds. Gradients agree with the float64 + reference and pass `gradcheck` under every backend. The "float64 oracle" is not + float64 throughout: attention, matmuls and the loss run in float64, but RMSNorm variance and RoPE angles are computed in float32 even for float64 inputs (then cast back), so it bounds + errors at roughly float32 precision in those two operations. +2. **Inference: SDPA is faster at every shape and the gap grows with length** + (reference/SDPA forward 1.2–1.3× at L=128 → 3.4–3.9× at L=2048 on CPU; 2.0–3.1× → + 3.9–5.4× on MPS) **and avoids the L² score matrix** (B=8, L=2048: 223 vs 1,765 MiB on + CPU fp32, 180 vs 1,608 MiB on MPS fp32, 90 vs 1,950 MiB on MPS fp16). +3. **The legacy MHA path is close to the reference on CPU at inference** (1,439 ms vs + 321 ms for SDPA at B=8, L=2048) and 1.1–1.6× slower than SDPA on MPS. +4. **KV cache.** On CPU the cache is faster at every measured shape: 2.2–15× over + uncached SDPA at batch 1 and 3.7–23× at batch 8 (up to 63× over the uncached legacy MHA + path). On MPS at batch 1 with prompts of 1–16 tokens it is *slower* (0.88× fp32, + 0.89–0.91× fp16); it pays off from 128 tokens (1.1×) and reaches 3.6× at 1,024 (fp32). + Past the context window every step rebuilds the cache, so caching gives 0.94–1.00× + on MPS. These values come from the October 22 re-measurement that stores raw + per-repeat samples; the first KV records (summaries only) are superseded, and their + speedups agree within ±0.5× at every point. Cached and + uncached greedy tokens are identical in every fp32 configuration; in fp16 they + differ in 2 of 11 configurations (near-tie argmax flips with random weights). + Cache memory grows by exactly 6,144 bytes per position per sequence (fp32), and + each concatenating decode step transiently allocates about one full cache copy. +5. **Training with SDPA vs the legacy MHA path (science-v1, 3 seeds).** Validation + perplexity is indistinguishable: 19.21 ± 1.75 (MHA) vs 19.04 ± 1.37 (SDPA); + paired differences +0.10, +0.02, −0.65. Both backends agree to seven decimals + while the model sits on its early plateau and diverge only when it leaves it. + In the controlled step benchmark, SDPA steps are 5–28% faster than the legacy + MHA path on CPU (8% at L=128, 5% at 256, 8% at 512, 28% at 1,024) with similar memory; on MPS all three backends are + within ~5% in time and memory. CPU evaluation of the full validation split takes + ~16% less time with SDPA (1.55 vs 1.85 s mean). Full-run tokens/s were confounded by + run order (the first run of the session was the fastest) and are not used for claims. +6. **The published 17.58 perplexity is the best of three seeds** (17.58 / 18.99 / 21.07). + +### Part B · science-v2 + +7. **Cleaned corpus with an untouched test book.** Darwin's index, both books' front + matter, Einstein's symbol glossary, Faraday's notes, headings, bracketed editorial + paragraphs and numeric tables are removed by documented rules before splitting. + Zero exact and zero near duplicates (≥ 50% shared word 8-grams) across + train/validation/test. +8. **Test perplexity (Faraday, never used for training or selection): + 15.66 ± 2.58 (MHA) and 15.60 ± 2.63 (SDPA)**, 0.1–0.4 above validation in all + six runs. Seeds dominate the spread (13.8–18.6); the backend does not + (paired test differences −0.10, 0.00, −0.08). + +### Decision on SDPA + +Correctness (forward, gradients, causality, cache equivalence) and performance are +verified. SDPA matches the legacy path's quality on two corpora, trains as fast or +faster, evaluates faster, and is much faster with far less memory at long-sequence +inference. **From the first release, SDPA is the default** for training, evaluation and +generation. The legacy path is the explicit backend `mha` (named `auto` in records and +checkpoints written before the release; that name is still accepted), and every v1 +reproduction command selects it explicitly. With `--attention-backend mha`, v1 training +on CPU remains bitwise identical to the published runs. The explicit `reference` +backend remains selectable but did not produce any published training result. + +## Method + +- **Matched comparisons.** Each timing record uses one set of seeded weights with the + published architecture (vocab 256, width 192, 4 layers, 6 heads); only the position + table grows to fit long sequences. `set_attention_backend` switches attention math + without touching weights. Inputs are seeded; dtype and device are fixed per record. +- **Timing.** Warmups, then repeated trials with backend/mode order rotated each + repeat; devices are synchronized around every timed region; decode steps are timed + individually. Medians, p95, IQR and CV are computed from raw samples stored in each + current record, and `scripts/validate_records.py` recomputes them to check. The three + historical KV-cache records stored summaries only; they are superseded and their + samples cannot be recovered. +- **Memory.** One fresh process per point (`provenance.measure_peak_memory`). CPU: host + peak-RSS delta. MPS: live allocator bytes sampled after every aten op (forward and + backward). CUDA would use the device allocator peak; it is untested and no CUDA + memory is reported. Earlier MPS records using + driver pool size (~1 GiB granularity) are kept but superseded. +- **Quality.** Oracle = reference backend in float64 on CPU: attention, softmax, matmuls + and loss in float64, but RMSNorm variance and RoPE angles are computed in float32 even for float64 inputs. Perplexity uses the repository's + non-overlapping 128-token chunking, with loss computed in fp32. +- **Provenance.** Every table joins evaluations to training runs by checkpoint SHA-256 + and resolved model identity, checks seeds, backends, protocols and dataset hashes, + and refuses to build on any conflict or on model identities that are only + metadata-consistent. KV-cache speedups are derived from the timing samples, never + read from stored ratio fields. See [RECORD_SCHEMA.md](RECORD_SCHEMA.md). +- **Training comparison.** Identical command, data, seeds and hardware (CPU, fp32); + only `--attention-backend` differs; backend order alternates per seed. Default-path + checkpoints are bitwise identical to the original v1 runs. +- **v2 protocol.** Hyperparameters are the v1 values (nothing tuned on v2). The best + checkpoint per run is chosen on validation loss; the test book is scored once per + selected checkpoint after all training (`experiments/v2_protocol.sh`). + +## Limitations + +- One laptop, one session per record; no repeat across reboots or thermal states. +- All models are trained for 1,000 steps and leave a near-unigram plateau only around + step 750–1,000, so they are under-trained and seed-sensitive. Three seeds give a + spread estimate, not a confidence interval. +- Latency uses random weights; quality comparisons are limited to the 128-token + trained context. +- MPS conclusions are for a tiny model where launch overhead dominates. +- The v2 test book is a single 1861 popular-science lecture series; it measures + transfer to one new author and topic, not general language ability. +- No CUDA, `torch.compile` or FlashAttention-specific kernels. + +## Reproduce + +```bash +# Inference and cache benchmarks (corpus-independent) +python scripts/benchmark_attention.py --device cpu --dtype float32 +python scripts/benchmark_attention.py --device mps --dtype float32 +python scripts/benchmark_attention.py --device mps --dtype float16 +python scripts/benchmark_kv_scaling.py --device cpu +python scripts/benchmark_kv_scaling.py --device mps +python scripts/benchmark_kv_scaling.py --device mps --dtype float16 +python scripts/benchmark_training_step.py --device cpu +python scripts/benchmark_training_step.py --device mps + +# science-v1: matched MHA vs SDPA training, then inference quality per backend +# (the published records used run directories named auto-s4N; the script now writes mha-s4N) +PYTHON=python ./experiments/sdpa_v1.sh +python scripts/compare_attention_quality.py artifacts/runs/sdpa-v1/mha-s4{2,3,4}/best_checkpoint.pt + +# science-v2: rebuild corpus (verified against committed files), train, select, test once +PYTHON=python ./experiments/v2_protocol.sh + +python scripts/build_experiment_report.py +``` + +Each script writes a new timestamped record and never overwrites an existing one. The +report uses the latest record per name or device/dtype; superseded records stay on disk. + +![SDPA speedup](figures/attention_speedup.svg) + +![Attention memory](figures/attention_memory.svg) + +![Decode latency](figures/attention_decode.svg) + +![KV cache speedup](figures/kv_cache_speedup.svg) + +## Environment + + +| Record | Device | Dtype | Processor | torch | Python | Threads | Commit | Dirty | +| --- | --- | --- | --- | --- | --- | ---: | --- | --- | +| `20261022T033005Z_attention_cpu_float32_615767d.json` | cpu | float32 | Apple M4 | 2.14.1 | 3.12.14 | 4 | `615767d` | False | +| `20261022T034036Z_attention_mps_float32_0502118.json` | mps | float32 | Apple M4 | 2.14.1 | 3.12.14 | 4 | `0502118` | False | +| `20261022T034304Z_attention_mps_float16_0502118.json` | mps | float16 | Apple M4 | 2.14.1 | 3.12.14 | 4 | `0502118` | False | +| `20261022T041752Z_attention_quality_48e95f8.json` | cpu,mps | mixed | Apple M4 | 2.14.1 | 3.12.14 | 4 | `48e95f8` | False | +| `20261022T053124Z_kv_cpu_float32_57be26d.json` | cpu | float32 | Apple M4 | 2.14.1 | 3.12.14 | 4 | `57be26d` | False | +| `20261022T053346Z_kv_mps_float32_57be26d.json` | mps | float32 | Apple M4 | 2.14.1 | 3.12.14 | 4 | `57be26d` | False | +| `20261022T053547Z_kv_mps_float16_57be26d.json` | mps | float16 | Apple M4 | 2.14.1 | 3.12.14 | 4 | `57be26d` | False | + + +# Part A · Systems experiments and corpus science-v1 (legacy) + +## Experiment 02 · Attention: reference vs PyTorch SDPA + +### Forward pass latency and memory + + +**cpu/float32** · warmup 3, 10 repeats · median ms (p95) · peak extra MiB per forward + +| B | L | MHA (legacy v1) | SDPA | Reference | Ref / SDPA | MHA MiB | SDPA MiB | Ref MiB | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 1 | 128 | 1.86 (2.06) | 1.42 (2.40) | 1.78 (2.04) | 1.25× | 4.1 | 4.0 | 4.0 | +| 1 | 256 | 4.18 (4.81) | 3.00 (3.45) | 3.72 (3.84) | 1.24× | 8.9 | 5.2 | 7.3 | +| 1 | 512 | 10.49 (11.15) | 5.79 (6.23) | 8.33 (8.81) | 1.44× | 24.1 | 11.5 | 23.4 | +| 1 | 1024 | 41.98 (55.30) | 15.52 (29.28) | 38.46 (46.94) | 2.48× | 75.7 | 21.6 | 66.4 | +| 1 | 2048 | 192.72 (227.30) | 45.93 (77.64) | 156.18 (189.24) | 3.40× | 255.4 | 37.1 | 234.8 | +| 4 | 128 | 5.36 (5.94) | 4.24 (4.71) | 5.03 (5.39) | 1.19× | 11.5 | 11.4 | 11.4 | +| 4 | 256 | 20.53 (28.74) | 14.00 (18.88) | 20.06 (29.97) | 1.43× | 34.7 | 19.0 | 31.3 | +| 4 | 512 | 52.62 (54.63) | 31.23 (32.66) | 51.83 (54.97) | 1.66× | 87.3 | 35.0 | 82.2 | +| 4 | 1024 | 191.96 (203.43) | 61.17 (80.83) | 181.63 (197.34) | 2.97× | 252.5 | 37.6 | 228.5 | +| 4 | 2048 | 726.96 (1103.65) | 174.91 (301.39) | 615.28 (732.49) | 3.52× | 898.9 | 135.0 | 900.7 | +| 8 | 128 | 10.68 (11.83) | 8.69 (9.87) | 10.25 (10.74) | 1.18× | 20.7 | 18.2 | 18.2 | +| 8 | 256 | 34.89 (36.53) | 25.44 (26.03) | 35.29 (44.47) | 1.39× | 39.2 | 34.8 | 34.3 | +| 8 | 512 | 111.00 (196.13) | 68.26 (118.38) | 98.43 (227.01) | 1.44× | 131.9 | 35.2 | 131.3 | +| 8 | 1024 | 431.37 (448.51) | 140.66 (169.55) | 377.14 (466.85) | 2.68× | 582.2 | 134.2 | 517.1 | +| 8 | 2048 | 1439.33 (1525.57) | 321.03 (336.56) | 1251.08 (1522.71) | 3.90× | 1834.7 | 222.8 | 1764.6 | + +**mps/float32** · warmup 3, 10 repeats · median ms (p95) · peak extra MiB per forward + +| B | L | MHA (legacy v1) | SDPA | Reference | Ref / SDPA | MHA MiB | SDPA MiB | Ref MiB | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 1 | 128 | 1.88 (1.98) | 1.68 (1.82) | 5.17 (5.42) | 3.08× | 1.4 | 1.4 | 1.4 | +| 1 | 256 | 2.77 (3.19) | 2.34 (2.92) | 6.53 (8.16) | 2.80× | 2.8 | 2.8 | 4.1 | +| 1 | 512 | 4.09 (4.12) | 3.51 (5.95) | 9.52 (16.60) | 2.71× | 5.6 | 5.6 | 14.6 | +| 1 | 1024 | 8.03 (8.26) | 6.17 (8.02) | 19.37 (24.79) | 3.14× | 11.3 | 11.3 | 52.8 | +| 1 | 2048 | 20.55 (23.01) | 13.02 (15.56) | 55.65 (61.46) | 4.27× | 28.0 | 22.5 | 204.0 | +| 4 | 128 | 3.76 (4.44) | 3.24 (3.34) | 7.73 (9.14) | 2.39× | 5.6 | 5.6 | 5.6 | +| 4 | 256 | 6.88 (13.39) | 5.54 (6.05) | 11.93 (13.06) | 2.15× | 11.3 | 11.3 | 16.5 | +| 4 | 512 | 12.78 (12.97) | 10.59 (11.10) | 23.73 (24.14) | 2.24× | 22.5 | 22.5 | 57.0 | +| 4 | 1024 | 28.17 (28.78) | 21.75 (22.22) | 61.56 (64.84) | 2.83× | 45.0 | 45.0 | 210.0 | +| 4 | 2048 | 71.88 (74.93) | 49.24 (53.41) | 195.01 (197.49) | 3.96× | 90.0 | 90.0 | 804.0 | +| 8 | 128 | 6.12 (6.87) | 5.37 (6.94) | 10.53 (11.89) | 1.96× | 11.3 | 11.3 | 11.3 | +| 8 | 256 | 11.86 (13.97) | 10.22 (10.79) | 18.91 (19.42) | 1.85× | 22.5 | 22.5 | 33.0 | +| 8 | 512 | 24.36 (25.03) | 20.32 (21.72) | 45.50 (50.84) | 2.24× | 45.0 | 45.0 | 114.0 | +| 8 | 1024 | 55.41 (56.97) | 43.55 (45.83) | 115.97 (117.24) | 2.66× | 90.0 | 90.0 | 420.0 | +| 8 | 2048 | 139.77 (140.99) | 98.72 (102.43) | 386.74 (396.31) | 3.92× | 180.0 | 180.0 | 1608.0 | + +**mps/float16** · warmup 3, 10 repeats · median ms (p95) · peak extra MiB per forward + +| B | L | MHA (legacy v1) | SDPA | Reference | Ref / SDPA | MHA MiB | SDPA MiB | Ref MiB | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 1 | 128 | 2.02 (2.23) | 1.88 (2.27) | 5.54 (5.74) | 2.95× | 0.7 | 0.7 | 1.2 | +| 1 | 256 | 2.58 (2.88) | 2.21 (2.41) | 6.48 (6.92) | 2.94× | 1.4 | 1.4 | 4.2 | +| 1 | 512 | 3.96 (4.67) | 3.28 (3.62) | 9.89 (11.19) | 3.01× | 2.8 | 2.8 | 15.9 | +| 1 | 1024 | 7.27 (7.82) | 5.60 (6.33) | 20.70 (20.90) | 3.70× | 6.1 | 5.6 | 61.9 | +| 1 | 2048 | 17.45 (17.55) | 11.23 (11.40) | 60.15 (64.73) | 5.36× | 14.0 | 12.0 | 243.8 | +| 4 | 128 | 3.64 (5.11) | 3.24 (3.95) | 7.23 (8.94) | 2.23× | 2.8 | 2.8 | 4.7 | +| 4 | 256 | 5.75 (5.99) | 5.04 (5.99) | 11.77 (12.29) | 2.33× | 5.6 | 5.6 | 16.9 | +| 4 | 512 | 10.65 (12.21) | 8.97 (9.03) | 24.18 (26.01) | 2.70× | 11.3 | 12.0 | 63.8 | +| 4 | 1024 | 23.85 (24.58) | 18.68 (19.80) | 67.64 (71.91) | 3.62× | 23.0 | 22.5 | 247.5 | +| 4 | 2048 | 60.74 (62.22) | 42.51 (49.76) | 216.36 (221.87) | 5.09× | 45.0 | 45.0 | 975.0 | +| 8 | 128 | 5.63 (7.31) | 4.91 (6.26) | 10.16 (10.56) | 2.07× | 5.6 | 5.6 | 9.4 | +| 8 | 256 | 9.84 (12.22) | 8.76 (9.09) | 18.46 (18.86) | 2.11× | 11.3 | 12.0 | 33.8 | +| 8 | 512 | 21.83 (27.02) | 18.96 (23.19) | 51.48 (58.01) | 2.71× | 22.5 | 22.5 | 127.5 | +| 8 | 1024 | 46.50 (47.40) | 36.64 (39.90) | 125.23 (129.83) | 3.42× | 45.0 | 45.0 | 495.0 | +| 8 | 2048 | 118.43 (121.10) | 83.26 (88.71) | 440.90 (458.74) | 5.30× | 90.0 | 90.0 | 1950.0 | + + +### Prefill, decode and generation throughput + + +**cpu/float32** · 32 greedy decode steps after an L-token prefill · median (p95) + +| B | L | SDPA prefill ms | Ref prefill ms | SDPA decode ms/step | Ref decode ms/step | SDPA gen tok/s | Ref gen tok/s | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 1 | 128 | 1.44 (1.98) | 1.78 (2.15) | 0.36 (0.45) | 0.42 (0.47) | 2443 | 2058 | +| 1 | 256 | 2.99 (3.47) | 3.59 (3.91) | 0.46 (0.53) | 0.54 (0.62) | 1773 | 1537 | +| 1 | 512 | 5.75 (6.07) | 8.36 (8.65) | 0.51 (0.63) | 0.57 (0.69) | 1415 | 1190 | +| 1 | 1024 | 15.65 (19.65) | 38.94 (50.15) | 0.84 (1.04) | 0.98 (1.26) | 730 | 453 | +| 1 | 2048 | 45.73 (80.18) | 147.24 (176.31) | 1.35 (2.88) | 1.66 (2.43) | 358 | 160 | +| 4 | 128 | 4.30 (4.92) | 5.07 (5.43) | 0.74 (0.91) | 0.97 (1.10) | 4436 | 3467 | +| 4 | 256 | 12.93 (28.47) | 19.27 (37.01) | 1.24 (2.20) | 1.66 (2.31) | 2387 | 1696 | +| 4 | 512 | 32.20 (33.98) | 52.16 (55.70) | 1.42 (1.67) | 1.83 (2.15) | 1640 | 1151 | +| 4 | 1024 | 60.95 (77.18) | 155.43 (181.36) | 1.89 (2.59) | 2.83 (3.73) | 1037 | 511 | +| 4 | 2048 | 175.27 (288.85) | 576.90 (635.33) | 3.47 (9.16) | 5.74 (7.29) | 450 | 165 | +| 8 | 128 | 8.80 (10.38) | 10.36 (11.45) | 1.05 (1.22) | 1.45 (1.62) | 5930 | 4453 | +| 8 | 256 | 25.86 (26.94) | 35.85 (38.53) | 1.47 (1.68) | 1.99 (2.31) | 3464 | 2564 | +| 8 | 512 | 61.36 (96.22) | 102.72 (201.61) | 2.13 (3.36) | 3.20 (8.11) | 1957 | 1256 | +| 8 | 1024 | 150.42 (171.68) | 345.12 (433.23) | 3.59 (5.13) | 6.16 (8.85) | 947 | 450 | +| 8 | 2048 | 360.67 (441.24) | 1263.21 (1348.73) | 7.71 (10.18) | 10.57 (13.46) | 430 | 158 | + +**mps/float32** · 32 greedy decode steps after an L-token prefill · median (p95) + +| B | L | SDPA prefill ms | Ref prefill ms | SDPA decode ms/step | Ref decode ms/step | SDPA gen tok/s | Ref gen tok/s | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 1 | 128 | 1.67 (2.31) | 2.21 (2.46) | 1.64 (1.79) | 4.10 (4.99) | 587 | 237 | +| 1 | 256 | 2.28 (2.50) | 3.43 (3.99) | 1.70 (2.80) | 4.30 (5.23) | 555 | 223 | +| 1 | 512 | 3.53 (5.77) | 5.86 (7.84) | 1.76 (1.93) | 4.26 (4.79) | 530 | 225 | +| 1 | 1024 | 6.17 (7.43) | 16.25 (19.09) | 1.84 (2.13) | 4.36 (5.11) | 486 | 203 | +| 1 | 2048 | 13.07 (14.32) | 52.20 (55.90) | 2.08 (3.26) | 4.72 (6.22) | 392 | 155 | +| 4 | 128 | 3.31 (4.71) | 4.33 (4.54) | 1.91 (2.86) | 4.37 (4.97) | 1968 | 883 | +| 4 | 256 | 5.61 (6.35) | 8.96 (10.08) | 2.07 (2.81) | 4.74 (6.54) | 1771 | 804 | +| 4 | 512 | 10.57 (10.76) | 20.36 (20.64) | 2.34 (2.48) | 4.81 (5.28) | 1495 | 729 | +| 4 | 1024 | 21.51 (22.44) | 59.15 (61.19) | 2.79 (4.02) | 5.53 (9.26) | 1177 | 505 | +| 4 | 2048 | 49.23 (53.32) | 193.99 (197.43) | 4.19 (6.50) | 7.02 (11.78) | 715 | 287 | +| 8 | 128 | 5.36 (6.36) | 7.19 (7.35) | 2.18 (2.45) | 4.68 (5.19) | 3382 | 1619 | +| 8 | 256 | 10.31 (11.25) | 15.86 (16.26) | 2.49 (2.68) | 4.96 (5.63) | 2822 | 1444 | +| 8 | 512 | 20.33 (22.47) | 40.45 (43.12) | 3.08 (4.80) | 5.66 (8.66) | 2197 | 1065 | +| 8 | 1024 | 42.99 (46.05) | 113.80 (115.24) | 4.58 (6.74) | 7.26 (14.64) | 1363 | 675 | +| 8 | 2048 | 97.37 (102.13) | 383.13 (399.19) | 6.47 (8.87) | 9.62 (14.22) | 836 | 356 | + +**mps/float16** · 32 greedy decode steps after an L-token prefill · median (p95) + +| B | L | SDPA prefill ms | Ref prefill ms | SDPA decode ms/step | Ref decode ms/step | SDPA gen tok/s | Ref gen tok/s | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 1 | 128 | 1.82 (2.06) | 2.33 (2.62) | 1.57 (1.84) | 4.15 (4.58) | 602 | 237 | +| 1 | 256 | 2.23 (2.36) | 3.25 (3.43) | 1.61 (1.92) | 4.17 (4.81) | 588 | 233 | +| 1 | 512 | 3.28 (3.76) | 6.42 (7.30) | 1.71 (1.86) | 4.25 (4.68) | 550 | 223 | +| 1 | 1024 | 5.60 (6.02) | 17.49 (18.14) | 1.78 (1.97) | 4.30 (4.84) | 510 | 205 | +| 1 | 2048 | 11.20 (11.26) | 57.02 (58.05) | 1.90 (2.25) | 4.41 (5.38) | 442 | 160 | +| 4 | 128 | 3.11 (3.62) | 4.30 (4.87) | 1.86 (2.03) | 4.36 (4.92) | 2031 | 880 | +| 4 | 256 | 5.04 (5.53) | 8.37 (8.78) | 1.99 (2.14) | 4.50 (5.11) | 1854 | 837 | +| 4 | 512 | 9.02 (9.13) | 20.87 (21.49) | 2.28 (2.48) | 4.75 (5.40) | 1556 | 727 | +| 4 | 1024 | 18.73 (20.24) | 63.57 (65.75) | 2.67 (5.06) | 5.23 (7.07) | 1254 | 541 | +| 4 | 2048 | 42.40 (45.02) | 214.22 (216.86) | 3.50 (5.57) | 6.63 (11.13) | 809 | 281 | +| 8 | 128 | 4.98 (5.90) | 6.99 (7.46) | 2.02 (2.14) | 4.49 (5.03) | 3658 | 1672 | +| 8 | 256 | 8.72 (8.98) | 15.25 (15.68) | 2.36 (2.65) | 4.82 (5.32) | 3032 | 1498 | +| 8 | 512 | 18.98 (21.46) | 44.86 (56.61) | 2.90 (4.86) | 5.60 (8.95) | 2193 | 1079 | +| 8 | 1024 | 36.92 (40.58) | 124.45 (128.06) | 3.90 (6.53) | 6.71 (10.62) | 1631 | 708 | +| 8 | 2048 | 84.88 (87.50) | 430.49 (446.11) | 6.06 (8.56) | 9.05 (15.89) | 923 | 343 | + + +### Timing variability + + +| Target | Phase | Median CV | Worst CV | Worst config | +| --- | --- | ---: | ---: | --- | +| cpu/float32 | forward_ms | 0.074 | 0.493 | reference B=8 L=512 | +| cpu/float32 | prefill_ms | 0.099 | 0.537 | sdpa B=4 L=256 | +| cpu/float32 | decode_ms_per_step | 0.130 | 0.935 | reference B=8 L=1024 | +| mps/float32 | forward_ms | 0.040 | 0.454 | mha B=4 L=256 | +| mps/float32 | prefill_ms | 0.040 | 0.266 | sdpa B=1 L=512 | +| mps/float32 | decode_ms_per_step | 0.182 | 0.474 | reference B=4 L=256 | +| mps/float16 | forward_ms | 0.048 | 0.165 | mha B=4 L=128 | +| mps/float16 | prefill_ms | 0.033 | 0.139 | reference B=8 L=512 | +| mps/float16 | decode_ms_per_step | 0.166 | 0.339 | reference B=8 L=1024 | + + +### Numerical correctness and output quality (trained checkpoints) + + +Oracle: cpu float64, reference backend (RMSNorm variance and RoPE angles still run in float32); 112,111 held-out targets; greedy: 8 prompts × 64 tokens. + +| Seed | Target | Backend | max abs Δlogit | mean abs Δlogit | top-1 agree | Perplexity | Greedy identical | +| ---: | --- | --- | ---: | ---: | ---: | ---: | ---: | +| 42 | cpu/float32 | MHA (legacy v1) | 3.60e-05 | 1.04e-06 | 100.000% | 17.5827 | 8/8 | +| 42 | cpu/float32 | SDPA | 3.40e-05 | 1.04e-06 | 100.000% | 17.5827 | 8/8 | +| 42 | cpu/float32 | Reference | 3.71e-05 | 1.04e-06 | 100.000% | 17.5827 | 8/8 | +| 42 | mps/float32 | MHA (legacy v1) | 3.31e-05 | 1.03e-06 | 100.000% | 17.5827 | 8/8 | +| 42 | mps/float32 | SDPA | 3.31e-05 | 1.03e-06 | 100.000% | 17.5827 | 8/8 | +| 42 | mps/float32 | Reference | 4.16e-05 | 1.03e-06 | 100.000% | 17.5827 | 8/8 | +| 42 | mps/float16 | MHA (legacy v1) | 7.36e-02 | 2.18e-03 | 99.880% | 17.5826 | 6/8 | +| 42 | mps/float16 | SDPA | 7.36e-02 | 2.18e-03 | 99.880% | 17.5826 | 6/8 | +| 42 | mps/float16 | Reference | 8.14e-02 | 2.19e-03 | 99.880% | 17.5825 | 6/8 | +| 42 | mps/bfloat16 | MHA (legacy v1) | 6.00e-01 | 1.75e-02 | 98.786% | 17.5863 | 4/8 | +| 42 | mps/bfloat16 | SDPA | 6.00e-01 | 1.75e-02 | 98.786% | 17.5863 | 4/8 | +| 42 | mps/bfloat16 | Reference | 6.00e-01 | 1.75e-02 | 98.755% | 17.5858 | 3/8 | +| 43 | cpu/float32 | MHA (legacy v1) | 1.82e-05 | 7.85e-07 | 100.000% | 18.9894 | 8/8 | +| 43 | cpu/float32 | SDPA | 1.70e-05 | 7.86e-07 | 100.000% | 18.9894 | 8/8 | +| 43 | cpu/float32 | Reference | 2.00e-05 | 7.87e-07 | 100.000% | 18.9894 | 8/8 | +| 43 | mps/float32 | MHA (legacy v1) | 1.60e-05 | 7.80e-07 | 100.000% | 18.9894 | 8/8 | +| 43 | mps/float32 | SDPA | 1.60e-05 | 7.80e-07 | 100.000% | 18.9894 | 8/8 | +| 43 | mps/float32 | Reference | 1.84e-05 | 7.80e-07 | 100.000% | 18.9894 | 8/8 | +| 43 | mps/float16 | MHA (legacy v1) | 4.85e-02 | 1.78e-03 | 99.851% | 18.9889 | 6/8 | +| 43 | mps/float16 | SDPA | 4.85e-02 | 1.78e-03 | 99.851% | 18.9889 | 6/8 | +| 43 | mps/float16 | Reference | 4.85e-02 | 1.78e-03 | 99.855% | 18.9888 | 6/8 | +| 43 | mps/bfloat16 | MHA (legacy v1) | 4.58e-01 | 1.43e-02 | 98.708% | 18.9908 | 3/8 | +| 43 | mps/bfloat16 | SDPA | 4.58e-01 | 1.43e-02 | 98.708% | 18.9908 | 3/8 | +| 43 | mps/bfloat16 | Reference | 4.29e-01 | 1.43e-02 | 98.701% | 18.9894 | 4/8 | +| 44 | cpu/float32 | MHA (legacy v1) | 1.22e-05 | 7.66e-07 | 100.000% | 21.0677 | 8/8 | +| 44 | cpu/float32 | SDPA | 9.53e-06 | 7.67e-07 | 100.000% | 21.0677 | 8/8 | +| 44 | cpu/float32 | Reference | 9.77e-06 | 7.67e-07 | 100.000% | 21.0677 | 8/8 | +| 44 | mps/float32 | MHA (legacy v1) | 1.19e-05 | 7.67e-07 | 100.000% | 21.0677 | 8/8 | +| 44 | mps/float32 | SDPA | 1.19e-05 | 7.67e-07 | 100.000% | 21.0677 | 8/8 | +| 44 | mps/float32 | Reference | 1.24e-05 | 7.67e-07 | 100.000% | 21.0677 | 8/8 | +| 44 | mps/float16 | MHA (legacy v1) | 3.33e-02 | 1.56e-03 | 99.804% | 21.0675 | 8/8 | +| 44 | mps/float16 | SDPA | 3.33e-02 | 1.56e-03 | 99.804% | 21.0675 | 8/8 | +| 44 | mps/float16 | Reference | 3.33e-02 | 1.57e-03 | 99.802% | 21.0674 | 7/8 | +| 44 | mps/bfloat16 | MHA (legacy v1) | 2.88e-01 | 1.24e-02 | 98.422% | 21.0724 | 5/8 | +| 44 | mps/bfloat16 | SDPA | 2.88e-01 | 1.24e-02 | 98.422% | 21.0724 | 5/8 | +| 44 | mps/bfloat16 | Reference | 2.75e-01 | 1.24e-02 | 98.416% | 21.0718 | 6/8 | + + +## Seed variability of the published configuration (science-v1) + + +| Seed | Held-out perplexity | Bits per byte | +| ---: | ---: | ---: | +| 42 | 17.583 | 4.1361 | +| 43 | 18.989 | 4.2471 | +| 44 | 21.068 | 4.3970 | +| **mean ± sd (n=3)** | **19.213 ± 1.753** | | + + +## Experiment 03 · KV cache: cached vs uncached decoding (corpus-independent) + + +**cpu/float32** · 64 greedy tokens · median total ms over 5 repeats · step 0 is the prefill + +| B | Prompt | Cached | Uncached (SDPA) | Uncached (MHA) | Cache speedup vs SDPA | vs MHA | Cached decode ms/step | Tokens identical | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | :---: | +| 1 | 1 | 21.6 | 46.9 | 54.2 | 2.17× | 2.50× | 0.33 | yes | +| 1 | 16 | 22.0 | 57.7 | 65.0 | 2.62× | 2.96× | 0.33 | yes | +| 1 | 128 | 26.7 | 117.5 | 151.3 | 4.41× | 5.67× | 0.37 | yes | +| 1 | 512 | 39.5 | 385.1 | 770.3 | 9.76× | 19.52× | 0.52 | yes | +| 1 | 1024 | 65.8 | 1017.1 | 2459.4 | 15.46× | 37.39× | 0.81 | yes | +| 8 | 1 | 39.0 | 144.6 | 169.6 | 3.71× | 4.35× | 0.64 | yes | +| 8 | 16 | 43.0 | 196.6 | 226.8 | 4.57× | 5.27× | 0.69 | yes | +| 8 | 128 | 66.6 | 640.3 | 835.9 | 9.62× | 12.55× | 0.93 | yes | +| 8 | 512 | 157.1 | 3045.3 | 5610.6 | 19.38× | 35.70× | 1.65 | yes | +| 8 | 1024 | 382.7 | 8932.2 | 24046.7 | 23.34× | 62.83× | 3.71 | yes | + +Past the window (context 128, prompt 32, 224 tokens): 128 of 224 cached steps rebuild the cache; cached 231.6 ms vs uncached SDPA 321.4 ms (1.39×); tokens identical: yes. + +**mps/float32** · 64 greedy tokens · median total ms over 5 repeats · step 0 is the prefill + +| B | Prompt | Cached | Uncached (SDPA) | Uncached (MHA) | Cache speedup vs SDPA | vs MHA | Cached decode ms/step | Tokens identical | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | :---: | +| 1 | 1 | 106.3 | 93.8 | 100.6 | 0.88× | 0.95× | 1.68 | yes | +| 1 | 16 | 107.7 | 94.9 | 102.5 | 0.88× | 0.95× | 1.69 | yes | +| 1 | 128 | 109.7 | 123.1 | 132.4 | 1.12× | 1.21× | 1.72 | yes | +| 1 | 512 | 124.9 | 247.6 | 295.3 | 1.98× | 2.36× | 1.81 | yes | +| 1 | 1024 | 158.7 | 564.6 | 679.3 | 3.56× | 4.28× | 2.33 | yes | +| 8 | 1 | 129.2 | 169.9 | 184.0 | 1.32× | 1.42× | 2.02 | yes | +| 8 | 16 | 135.7 | 196.8 | 219.9 | 1.45× | 1.62× | 2.11 | yes | +| 8 | 128 | 242.6 | 524.5 | 667.3 | 2.16× | 2.75× | 3.34 | yes | +| 8 | 512 | 339.8 | 2054.9 | 2456.5 | 6.05× | 7.23× | 4.51 | yes | +| 8 | 1024 | 446.6 | 4391.7 | 5630.7 | 9.83× | 12.61× | 5.61 | yes | + +Past the window (context 128, prompt 32, 224 tokens): 128 of 224 cached steps rebuild the cache; cached 408.9 ms vs uncached SDPA 383.6 ms (0.94×); tokens identical: yes. + +**mps/float16** · 64 greedy tokens · median total ms over 5 repeats · step 0 is the prefill + +| B | Prompt | Cached | Uncached (SDPA) | Uncached (MHA) | Cache speedup vs SDPA | vs MHA | Cached decode ms/step | Tokens identical | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | :---: | +| 1 | 1 | 101.3 | 89.8 | 97.1 | 0.89× | 0.96× | 1.56 | yes | +| 1 | 16 | 101.1 | 92.5 | 100.9 | 0.91× | 1.00× | 1.56 | yes | +| 1 | 128 | 102.5 | 123.0 | 132.8 | 1.20× | 1.30× | 1.58 | **no** | +| 1 | 512 | 118.3 | 229.8 | 266.0 | 1.94× | 2.25× | 1.74 | yes | +| 1 | 1024 | 151.4 | 512.7 | 596.1 | 3.39× | 3.94× | 2.28 | yes | +| 8 | 1 | 139.3 | 157.1 | 180.5 | 1.13× | 1.30× | 2.11 | **no** | +| 8 | 16 | 126.5 | 184.6 | 211.1 | 1.46× | 1.67× | 1.98 | yes | +| 8 | 128 | 207.5 | 523.9 | 566.8 | 2.53× | 2.73× | 3.05 | yes | +| 8 | 512 | 334.9 | 1645.2 | 2027.3 | 4.91× | 6.05× | 4.58 | yes | +| 8 | 1024 | 406.0 | 3532.7 | 4605.7 | 8.70× | 11.34× | 5.38 | yes | + +Past the window (context 128, prompt 32, 224 tokens): 128 of 224 cached steps rebuild the cache; cached 376.7 ms vs uncached SDPA 376.8 ms (1.00×); tokens identical: yes. + + +### Cache memory growth + + +| Target | B | Cached positions | Cache bytes | Bytes per position per sequence | Live-allocator peak during step (MPS) | +| --- | ---: | ---: | ---: | ---: | ---: | +| cpu/float32 | 1 | 129 | 0.76 MiB | 6144 | – | +| cpu/float32 | 1 | 191 | 1.12 MiB | 6144 | – | +| cpu/float32 | 8 | 129 | 6.05 MiB | 6144 | – | +| cpu/float32 | 8 | 191 | 8.95 MiB | 6144 | – | +| mps/float32 | 1 | 129 | 0.76 MiB | 6144 | 0.77 MiB | +| mps/float32 | 1 | 191 | 1.12 MiB | 6144 | 1.13 MiB | +| mps/float32 | 8 | 129 | 6.05 MiB | 6144 | 6.14 MiB | +| mps/float32 | 8 | 191 | 8.95 MiB | 6144 | 11.44 MiB | +| mps/float16 | 1 | 129 | 0.38 MiB | 3072 | 0.39 MiB | +| mps/float16 | 1 | 191 | 0.56 MiB | 3072 | 0.57 MiB | +| mps/float16 | 8 | 129 | 3.02 MiB | 3072 | 3.07 MiB | +| mps/float16 | 8 | 191 | 4.48 MiB | 3072 | 4.52 MiB | + + +## Experiment 04 · Matched training: legacy MHA vs SDPA (science-v1 corpus) + +### Controlled training-step benchmark + + +**cpu/float32** · batch 8 · forward + backward + AdamW · 10 repeats after 3 warmups · median (p95) + +| L | Backend | Step ms | Tokens/s | First-step extra memory MiB | max abs loss difference vs MHA | +| ---: | --- | ---: | ---: | ---: | ---: | +| 128 | MHA (legacy v1) | 27.8 (28.5) | 36867 | 107.8 | 0.0e+00 | +| 128 | SDPA | 25.6 (27.1) | 39931 | 105.7 | 4.8e-07 | +| 128 | Reference | 27.7 (29.2) | 37029 | 119.5 | 4.8e-07 | +| 256 | MHA (legacy v1) | 54.6 (55.8) | 37542 | 211.1 | 0.0e+00 | +| 256 | SDPA | 51.9 (52.7) | 39476 | 206.9 | 0.0e+00 | +| 256 | Reference | 60.9 (61.9) | 33613 | 283.7 | 0.0e+00 | +| 512 | MHA (legacy v1) | 132.0 (135.1) | 31034 | 415.0 | 0.0e+00 | +| 512 | SDPA | 122.0 (134.0) | 33588 | 405.8 | 4.8e-07 | +| 512 | Reference | 168.2 (170.2) | 24356 | 684.1 | 9.5e-07 | +| 1024 | MHA (legacy v1) | 353.1 (367.3) | 23200 | 800.7 | 0.0e+00 | +| 1024 | SDPA | 276.1 (288.4) | 29670 | 768.2 | 4.8e-07 | +| 1024 | Reference | 531.4 (543.8) | 15416 | 1918.8 | 4.8e-07 | + +**mps/float32** · batch 8 · forward + backward + AdamW · 10 repeats after 3 warmups · median (p95) + +| L | Backend | Step ms | Tokens/s | First-step extra memory MiB | max abs loss difference vs MHA | +| ---: | --- | ---: | ---: | ---: | ---: | +| 128 | MHA (legacy v1) | 26.6 (27.0) | 38498 | 105.0 | 0.0e+00 | +| 128 | SDPA | 25.6 (25.9) | 39940 | 98.5 | 4.8e-07 | +| 128 | Reference | 25.1 (25.4) | 40733 | 98.6 | 4.8e-07 | +| 256 | MHA (legacy v1) | 51.9 (57.9) | 39498 | 232.3 | 0.0e+00 | +| 256 | SDPA | 49.8 (55.7) | 41119 | 220.8 | 0.0e+00 | +| 256 | Reference | 50.3 (51.8) | 40717 | 221.1 | 0.0e+00 | +| 512 | MHA (legacy v1) | 117.3 (117.7) | 34915 | 567.1 | 0.0e+00 | +| 512 | SDPA | 112.3 (115.7) | 36487 | 549.1 | 4.8e-07 | +| 512 | Reference | 115.3 (124.4) | 35520 | 550.1 | 4.8e-07 | +| 1024 | MHA (legacy v1) | 306.5 (308.2) | 26732 | 1708.3 | 0.0e+00 | +| 1024 | SDPA | 296.8 (298.8) | 27604 | 1672.3 | 4.8e-07 | +| 1024 | Reference | 309.9 (311.9) | 26432 | 1676.3 | 0.0e+00 | + + +### Full training runs (1,000 steps, 3 seeds, CPU) + + +| Seed | Backend | Selected step | Val loss | Val perplexity | Train tok/s | Train s | Peak RSS MiB | Eval s (CPU) | Eval s (MPS) | +| ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 42 | MHA (legacy v1) | 1000 | 2.8669 | 17.583 | 36842 | 111.2 | 444 | 1.71 | 2.34 | +| 42 | SDPA | 1000 | 2.8726 | 17.683 | 30232 | 135.5 | 448 | 1.53 | 2.15 | +| 43 | MHA (legacy v1) | 1000 | 2.9439 | 18.989 | 29278 | 139.9 | 449 | 1.93 | 2.12 | +| 43 | SDPA | 1000 | 2.9449 | 19.009 | 31231 | 131.2 | 445 | 1.57 | 2.11 | +| 44 | MHA (legacy v1) | 1000 | 3.0477 | 21.068 | 28831 | 142.1 | 451 | 1.90 | 2.17 | +| 44 | SDPA | 1000 | 3.0163 | 20.416 | 31579 | 129.7 | 442 | 1.56 | 2.21 | + +| Backend | n | Val perplexity mean ± sd | Train tok/s mean ± sd | Peak RSS MiB mean ± sd | CPU eval s mean ± sd | +| --- | ---: | ---: | ---: | ---: | ---: | +| MHA (legacy v1) | 3 | 19.213 ± 1.753 | 31650.275 ± 4501.336 | 447.990 ± 3.342 | 1.847 ± 0.118 | +| SDPA | 3 | 19.036 ± 1.366 | 31013.909 ± 698.998 | 445.031 ± 3.414 | 1.550 ± 0.020 | + +| Seed | SDPA − MHA val perplexity | SDPA / MHA train tok/s | +| ---: | ---: | ---: | +| 42 | +0.101 | 0.821× | +| 43 | +0.020 | 1.067× | +| 44 | -0.652 | 1.095× | + + +# Part B · Corpus protocol science-v2 + +Results in this part use a different corpus and preprocessing protocol from Part A. +**Perplexities are not comparable across the two parts.** + +**Test-set status: published.** The Faraday test file was evaluated exactly once, by +`experiments/v2_protocol.sh` after all training (record +`evaluation/20261022T043652Z_v2_final_cpu_897db37.json`, six selected checkpoints, one +score each). No training command references it, and no hyperparameter, checkpoint or +code change was made after seeing its results. Its cleaning rules (start/end markers, +bracketed editorial paragraphs, numeric tables) were designed while viewing the book's +raw text, but never using any model output on it. Now that the results are published, +this test set must not guide further tuning under the same untouched-test claim; a new +experiment that changes the model or protocol in response needs a new held-out book. +One book also cannot establish cross-author generalization in general. + +## Corpus science-v2 + + +| Source | Role | Body SHA-256 | Main-text paragraphs | Dropped (rule: count) | Kept | Train paragraphs | +| --- | --- | --- | ---: | --- | ---: | ---: | +| On the Origin of Species (1859) (Charles Darwin) | train_validation | `9ff49617f2a0…` | 711 | no_lowercase_heading: 15 | 696 | 623 | +| Relativity: The Special and General Theory (Lawson translation) (Albert Einstein) | train_validation | `384e707c77e5…` | 542 | no_lowercase_heading: 79 | 463 | 404 | +| The Chemical History of a Candle (Michael Faraday) | test | `5580d9bfa409…` | 217 | bracketed_editorial: 40, no_lowercase_heading: 15, numeric_list_or_table: 2 | 160 | – | + +| File | Bytes | SHA-256 | +| --- | ---: | --- | +| `train.txt` | 969,506 | `65182db3ac268ec9…` | +| `validation.txt` | 105,666 | `6457fe8eaf9011ef…` | +| `test.txt` | 211,849 | `9e98ba26c3517318…` | +| `darwin-1859-validation.txt` | 87,711 | `dea22adfd1f580ec…` | +| `einstein-1924-validation.txt` | 17,953 | `c6728bc6890bc714…` | + +| Check | Paragraphs | Exact duplicates | Near duplicates (≥50% word 8-grams) | Mean 8-gram overlap | +| --- | ---: | ---: | ---: | ---: | +| darwin-1859_validation_vs_train | 73 | 0 | 0 | 0.00624 | +| einstein-1924_validation_vs_train | 59 | 0 | 0 | 0.00976 | +| test_vs_train | 160 | 0 | 0 | 0.00002 | +| test_vs_validation | 160 | 0 | 0 | 0.00002 | + + +## Results under science-v2 + + +| Seed | Backend | Selected step (on validation) | Val perplexity | Val bits/byte | **Test perplexity** | **Test bits/byte** | Darwin val ppl | Einstein val ppl | +| ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 42 | MHA (legacy v1) | 1000 | 14.241 | 3.8319 | **14.472** | **3.8552** | 13.903 | 16.015 | +| 42 | SDPA | 1000 | 14.140 | 3.8217 | **14.372** | **3.8451** | 13.802 | 15.908 | +| 43 | MHA (legacy v1) | 1000 | 18.497 | 4.2092 | **18.623** | **4.2190** | 18.009 | 21.065 | +| 43 | SDPA | 1000 | 18.497 | 4.2092 | **18.623** | **4.2190** | 18.009 | 21.065 | +| 44 | MHA (legacy v1) | 1000 | 13.501 | 3.7550 | **13.886** | **3.7955** | 13.151 | 15.316 | +| 44 | SDPA | 1000 | 13.482 | 3.7530 | **13.802** | **3.7868** | 13.134 | 15.286 | + +| Backend | n | Val perplexity mean ± sd | Test perplexity mean ± sd | +| --- | ---: | ---: | ---: | +| MHA (legacy v1) | 3 | 15.413 ± 2.697 | 15.660 ± 2.583 | +| SDPA | 3 | 15.373 ± 2.726 | 15.599 ± 2.634 | + diff --git a/docs/RECORD_SCHEMA.md b/docs/RECORD_SCHEMA.md new file mode 100644 index 0000000..9820a64 --- /dev/null +++ b/docs/RECORD_SCHEMA.md @@ -0,0 +1,130 @@ +# Experiment records: schema, identity and provenance rules + +Records live in `experiments/records//__.json`. +They are append-only: a record is never edited or deleted. The rules below are +implemented in `src/minilm_lab/records.py` and enforced by +`python scripts/validate_records.py` (exit status 1 on any problem) and by +`scripts/build_experiment_report.py`, which refuses to build from invalid records. + +## Record versions + +| `record_version` | Written | Adds | +| ---: | --- | --- | +| 1 | before commit `cd6a11d` | environment, settings, results; checkpoint file SHA-256 | +| 2 | from commit `cd6a11d` | training: `best_/final_model_identity_sha256`, explicit `run_summary.hyperparameters`; evaluation and quality rows: `model_identity_sha256`; KV-cache benchmarks: raw samples | + +Every record has `environment` (Python, torch, platform, processor, device, threads, +git commit and dirty flag, command), `settings` and `results`. A record whose git tree +was dirty must be listed in `SUPERSEDED.json`. + +## Two kinds of identity + +- **Artifact identity**: the SHA-256 of a checkpoint *file*. It identifies one file. Checkpoints embed elapsed train/eval seconds, so retraining the same model produces a different file hash. +- **Model identity** (`minilm-model-identity/v1`, `src/minilm_lab/identity.py`): SHA-256 of a versioned canonical stream: + 1. `minilm-model-identity/v1\n` + 2. canonical JSON (sorted keys, `,`/`:` separators, ASCII, no NaN) of every `MiniLMConfig` field plus the tokenizer identifier, then `\n` + 3. for each state-dict tensor in ascending name order: canonical JSON header `{"dtype", "name", "nbytes", "shape"}`, then `\n`, then the elements in row-major order, each element little-endian, in the tensor's own dtype (no casting). + + Excluded: elapsed time, metrics, optimizer and RNG state, file paths, timestamps, seeds and the attention backend. Defined for byte-tokenizer checkpoints only. Legacy checkpoints are hashed from their tensors; configs missing newer fields take their defaults. + +Historical training records (version 1) have no model identity. `scripts/attest_model_identity.py` verifies that a run's checkpoint files are byte-identical to the recorded file hashes and then records their model identity in `identity/` records. It never infers an identity for a missing file. + +## Model-identity resolution and evidence levels + +For each current training record, the best and final checkpoints are resolved separately (`records.resolve_identity`). All available sources are collected: + +| Source | Where it comes from | Independent of the records? | +| --- | --- | --- | +| `declared` | the training record's `best_/final_model_identity_sha256` (schema v2) | no | +| `attested` | any current `identity/` attestation row with the same checkpoint SHA-256 | no (same mutable record set) | +| `baseline` | the pinned audited baseline entry for that checkpoint SHA-256 | anchored by a pin in code (see below) | +| `artifact` | the checkpoint file: its bytes must hash to the recorded SHA-256, then the identity is recomputed from its tensors and config | yes | + +Every available source must give the same fingerprint. A disagreement is a validation failure and leaves the checkpoint with no fingerprint. If a checkpoint file exists at the recorded location but its bytes do not match the recorded SHA-256, validation fails; it does not fall back to other sources. + +| Evidence level | Meaning | +| --- | --- | +| `artifact_verified` | the artifact was present, its bytes matched, and the recomputed identity agrees with every other source | +| `trusted_baseline_verified` | no artifact; the pinned audited baseline has an entry and every other source agrees with it | +| `metadata_consistent_only` | only declared and/or attested values, mutually consistent. **Not verification**: a consistent forgery of both is undetectable at this level | +| `unverified` | no fingerprint, or sources disagree | + +`validate_records.py` prints the count per level. `--strict`, used by `scripts/verify_release.sh` and by the report builder, requires every current training checkpoint to be `artifact_verified` or `trusted_baseline_verified`. `--require-artifacts` requires `artifact_verified` for all of them. + +### Audited baseline and trust assumptions + +`experiments/baselines/model_identity_baseline.json` maps each historical checkpoint SHA-256 to its model identity. `scripts/build_identity_baseline.py` produced it on 2026-10-22 from the 24 checkpoint files of the 12 current training runs. It hashed each file, required the result to equal the training record's SHA-256, and recomputed the identity from the tensors; it does not read attestations or declared values. Its output is deterministic: `build_identity_baseline.py --check` regenerates it byte-for-byte wherever the artifacts exist. Its SHA-256 is pinned as `BASELINE_SHA256` in `src/minilm_lab/records.py`. A baseline whose bytes do not match the pin is ignored and reported. + +What this does and does not establish: + +- It binds historical model identities to a specific, reviewable file whose change requires a code change, so attestations and declared values can no longer silently redefine them. +- It is **not** a cryptographic signature and gives no authenticity against someone who can change both the baseline and the pin. Trust rests on the repository's review history and on the maintainer's statement that the files were checked against the original artifacts, which are not distributed. +- Anyone holding the original artifacts can re-establish `artifact_verified` with `validate_records.py --require-artifacts` and check the baseline with `build_identity_baseline.py --check`. +- New training runs (schema v2) declare their identity, but without their artifact or a baseline update they are only `metadata_consistent_only` and fail `--strict`. + +## Experiment identity, duplicates and selection (training) + +A training record's experiment identity is: + +`(protocol, canonical backend, seed, config digest, dataset digest)` + +- **canonical backend**: `auto` (pre-release name) and a missing field both mean `mha`. Canonicalization happens before any grouping. Original labels stay in the records. +- **config digest**: SHA-256 of canonical JSON of the full model config, tokenizer and hyperparameters (steps, batch size, gradient accumulation, eval interval, learning rate, optimizer, device). +- **dataset digest**: SHA-256 of canonical JSON of the training and validation file hashes. + +Records are first grouped by **experiment slot** `(protocol, canonical backend, seed, dataset digest)`. Backend canonicalization and model-identity resolution happen before comparison. Selection, applied in order: + +1. Records listed in `SUPERSEDED.json` are excluded. They stay on disk with a reason and a replacement. +2. Records in the same slot are **duplicates of one experiment** only if all of the following hold: + - the configuration digest is the same; + - per-step train/validation losses and the selected step are identical; + - checkpoint identity agrees: the same checkpoint file with consistent resolved fingerprints, or different files whose resolved fingerprints are both established and equal. + + The seed is then counted once, and the earliest record represents the group. They are identical, so this choice cannot change any result. +3. Any other combination is a **conflict**: no record of that slot is selected, and validation and reporting fail until an explicit supersession resolves it. Record order or file names never pick between conflicting records. + +Every current training record must also name a known protocol. Its training and validation hashes must match the files on disk and the protocol manifest. Its `manifest_path` and `manifest_sha256` (when present) must match the protocol manifest. Its `training_code_commit` must exist in the repository. + +**Legacy rule (version 1).** Hyperparameters are not stored explicitly; they are parsed from the recorded `training_command`. Absent flags take `train_tiny.py`'s defaults (`steps 100, batch size 8, accumulation 1, eval interval 25, device auto`), unchanged since `1fb8f37` and guarded by a test. + +## Evaluation provenance + +Every row of a current evaluation, attention-quality or legacy record must resolve to **exactly one experiment identity** through its checkpoint file SHA-256. Several current training records may share that checkpoint only if they are replicates of one identity. The row must then agree with the training record on: + +- protocol (evaluation `settings.protocol` = training `settings.protocol`); +- seed and canonical backend; +- selected step (`selected_step` = training `best_step`); +- corpus hash (`corpus_sha256` = the hash stored in the checkpoint); +- model identity, when the row and the training record (or an attestation) both have one; +- validation split (the evaluation's `validation` file = the file training selected on). + +Every evaluation split must lie inside the protocol's data directory, match the file on disk, and match the protocol manifest wherever the manifest lists the file. Rows that map to the same experiment identity twice are rejected. Reports join training and evaluation through these rules only, never by seed and backend alone, and fail if a selected training identity lacks an evaluation row. + +| Protocol | Manifest | Data directory | +| --- | --- | --- | +| `science-v1` | `data/science/manifest.json` (train/validation hashes) | `data/science/` | +| `science-v2` | `data/science-v2/manifest.json` (all split files) | `data/science-v2/` | + +## Benchmarks and raw samples + +Current benchmark records must contain the raw measurements behind every summary: + +- attention: `samples_seconds` (forward, prefill, per-step decode); +- training step: `step_seconds_samples`; +- KV cache: `raw.step_seconds`, one list of per-step seconds per measured repeat. + +The validator recomputes median, p95, IQR, mean, CV, min and max from the raw values (relative tolerance 1e-12). Every raw timing must be finite and positive. + +**Derived KV-cache ratios.** `speedup_cached_vs_uncached_sdpa`, `speedup_cached_vs_uncached_mha` and the pre-release `speedup_cached_vs_uncached_default` are defined as: + +`median over repeats of the uncached mode's total time (ms)` ÷ `median over repeats of the cached mode's total time (ms)` + +A repeat's total is the sum of its per-step seconds × 1000, prefill included. The ratio is unitless, and > 1 means the cache is faster. The validator derives each ratio from the raw samples (`records.kv_speedups`) and requires any stored ratio to be finite, positive and equal within relative tolerance 1e-12. The report renders only the derived values, never stored ratio fields. The legacy single-prompt KV benchmark (`legacy/`) is checked the same way: medians from its samples, `speedup` = uncached ÷ cached, tokens/s = tokens ÷ median. + +**Legacy rule for summary-only KV records.** The three superseded historical KV records have no raw samples. Their summaries are checked for finite, positive times, and their stored ratios must equal the ratio of their stored medians. Their summaries themselves cannot be reconstructed or independently verified. Units and the synchronization strategy are recorded in `settings`. The three KV-cache records from 2026-10-22 03:55–04:00 UTC predate raw samples. They are superseded, kept unchanged, and their samples cannot be recovered. + +Peak memory: CPU uses the host peak-RSS delta, MPS uses live allocator bytes sampled after every op, and CUDA uses the device allocator peak (`torch.cuda.max_memory_allocated`). Host RSS is never reported for an accelerator. The CUDA path is untested. + +## Supersession + +`SUPERSEDED.json` maps a record key to `{reason, superseded_by}`. Both records must exist and be the same kind of record: same seed and canonical backend for training, same device and dtype for benchmarks, same name for evaluations. Entries may not supersede themselves or form cycles, and every chain must end at a current record. Superseded records are never used by reports, but stay on disk as the audit trail. diff --git a/docs/RELEASE_CHECKLIST.md b/docs/RELEASE_CHECKLIST.md new file mode 100644 index 0000000..3d4f823 --- /dev/null +++ b/docs/RELEASE_CHECKLIST.md @@ -0,0 +1,153 @@ +# Release checklist · first research-grade release candidate + +Branch `experiments/sdpa-training-v2`, stacked on `experiments/attention-kv-audit`, which +branches from `origin/main` at `1fb8f37`. Not pushed, merged, tagged for release or +published. The local tag `sdpa-v1-frozen` (→ `827b605`) marks the frozen SDPA-on-v1 +results. Verified on 2026-10-22 on an Apple M4 (16 GB), macOS 27.0.1, Python 3.12.14, +torch 2.14.1, in the development worktree and in a fresh clone installed from +`requirements/release-lock.txt`. + +## Release blockers in `ad71757`: status + +Three defects were reproduced on the rejected commit `ad71757` (probe script and logs: +corruption probes over a copy of `experiments/records`) and fixed in `a8e300d`–`0c84b30`. + +| Defect | Before (`ad71757`) | After | +| --- | --- | --- | +| 1. Attested / declared model fingerprints not bound to artifacts | all-zero attested fingerprint, a declared fingerprint contradicting the attestation, and another checkpoint's fingerprint all passed (exit 0, report built) | identity resolved from declared, attested, pinned-baseline and artifact sources; any disagreement or mismatched checkpoint bytes fails; evidence levels reported; `--strict` requires artifact or pinned-baseline verification | +| 2. Duplicate detection ignored resolved identities | auto + mha duplicate with a contradictory fingerprint, and with a different checkpoint hash, passed | grouping by (protocol, canonical backend, seed, dataset) before comparing configuration, outcomes and resolved checkpoint identity; conflicts select nothing and fail; supersession entries validated | +| 3. Stored KV speedups trusted | speedup 999 and NaN passed and 999.00× was rendered; zero timings crashed the validator | speedups derived from timings and rendered only from derivation; stored ratios must match (rel 1e-12); non-finite or non-positive timings rejected without crashing | + +| Probe | Before | After | +| --- | --- | --- | +| unmodified records | exit 0, seeds [42, 43, 44] | exit 0, seeds [42, 43, 44] | +| attested fingerprint = 64 zeros (seed 43) | exit 0 | exit 1 (sources disagree) | +| declared fingerprint ≠ attested | exit 0 | exit 1 | +| attestation carries another checkpoint's fingerprint | exit 0 | exit 1 | +| legitimate auto + mha duplicate | exit 0, counted once | exit 0, counted once | +| duplicate with contradictory fingerprint | exit 0 | exit 1 | +| duplicate with conflicting checkpoint hash | exit 0 | exit 1 | +| duplicate with contradictory losses | exit 1 | exit 1 | +| invalid supersession reference | exit 1 | exit 1 | +| stored speedup = 999 | exit 0, 999.00× rendered | exit 1, never rendered | +| stored speedup = NaN | exit 0 | exit 1 | +| median altered, raw kept / raw altered, summary kept | exit 1 / exit 1 | exit 1 / exit 1 | +| all cached timings = 0 | crash (ZeroDivisionError) | exit 1 | + +### Model-identity evidence + +| Environment | Evidence levels (24 checkpoints) | `--strict` | `--require-artifacts` | +| --- | --- | ---: | ---: | +| development worktree (original checkpoints present) | artifact_verified = 24 | exit 0 | exit 0 | +| fresh clone (no checkpoints) | trusted_baseline_verified = 24 | exit 0 | exit 1 | +| fresh clone with `--no-baseline` | metadata_consistent_only = 24 | exit 1 | exit 1 | + +Trust assumption: without the checkpoints, verification rests on the audited baseline +`experiments/baselines/model_identity_baseline.json`, built from the 24 original +checkpoint files after checking their bytes. Its SHA-256 is pinned in +`src/minilm_lab/records.py`, which makes it an integrity anchor in reviewed code. It is +not a cryptographic signature. Independent check: the README v1 retrain in the fresh +clone produced a checkpoint whose file hash differs but whose model identity equals the +baseline entry. `build_identity_baseline.py --check` regenerates the baseline +byte-for-byte where the artifacts exist. Rules: [RECORD_SCHEMA.md](RECORD_SCHEMA.md). + +## Independent audit of `c256f20`: findings and status + +| Finding | Status | Fix | Regression tests | +| --- | --- | --- | --- | +| 1. Checkpoint file hashes change on identical retraining | **Resolved** | deterministic model identity `minilm-model-identity/v1` (`src/minilm_lab/identity.py`); file SHA-256 kept for artifact identity; 24 historical checkpoints attested | `tests/test_model_identity.py` (14), attestation tests in `tests/test_record_provenance.py` | +| 2. Legacy `auto` / `mha` duplicates inflate seed counts | **Resolved** | experiment identity after backend canonicalization; replicate / conflict / supersession rules (`src/minilm_lab/records.py`) | `tests/test_record_provenance.py` | +| 3. Evaluation provenance not validated; report joined on seed/backend | **Resolved** | evaluation rows resolve to one experiment identity by checkpoint SHA-256 with seed, backend, protocol, step, corpus, model identity, split, commit and manifest checks; report fails closed | `tests/test_record_provenance.py` | +| 4. KV-cache records lack raw samples | **Resolved for current records** | raw per-repeat samples persisted, summaries derived and re-checked; historical records superseded, not edited | `tests/test_record_provenance.py` | +| A. float64 oracle overstated | Resolved | qualified everywhere (RMSNorm variance and RoPE angles run in float32) | documentation | +| B. CUDA memory could be host RSS | Resolved | `provenance.measure_peak_memory` (CUDA: device allocator peak) | `tests/test_memory_measurement.py` (CUDA stubbed) | +| C. Stale INT8 / corpus statements | Resolved | annotated in `implementation_audit.md`; updated in `research_report.md` | `tests/test_published_claims.py` | +| D. Exit codes through pipelines | Resolved | validator exits 1 on failure; `scripts/verify_release.sh` uses `set -euo pipefail` | `test_validator_failure_propagates_through_exit_codes_and_pipelines` | +| E. No unrelated reformatting | Respected | no formatter run on legacy files | — | + +Detailed rules: [RECORD_SCHEMA.md](RECORD_SCHEMA.md). + +## Validation commands and results + +| Command | Exit | Result | +| --- | ---: | --- | +| `./scripts/verify_release.sh` (development worktree) | 0 | lint clean; 186 passed, 0 failed, 0 skipped; strict validation of 41 records (artifact_verified = 24); report regenerated with no documentation drift | +| `./scripts/verify_release.sh` (fresh clone of `0c84b30`, lock-file install, no checkpoints) | 0 | lint clean; 184 passed, 2 skipped (they need the original checkpoint files); strict validation (trusted_baseline_verified = 24); no drift | +| `python scripts/build_identity_baseline.py --check` (development worktree) | 0 | baseline regenerated byte-for-byte from the checkpoint files | +| `python -m ruff check .` | 0 | clean | +| `python -m pytest -q` | 0 | 186 passed (development worktree) | +| `python scripts/validate_records.py --strict` | 0 | `validated 41 records (strict): OK` | +| `python scripts/build_experiment_report.py && git diff --exit-code -- docs/EXPERIMENT_RESULTS.md docs/figures` | 0 | no drift | +| README v1 training and `evaluate.py --attention-backend mha` (fresh clone) | 0 | four checkpoint losses bitwise equal to the published record; perplexity 17.582732137354572 (published value) | +| Regenerated v1 seed-42 checkpoint vs attested original | — | file SHA-256 **differs**; stored model identity matches its tensors; model identity **equals** the attested original | +| `python scripts/prepare_science_corpus.py --output-dir …` (network) | 0 | v1 splits verified against published hashes | +| `python scripts/prepare_science_corpus_v2.py --output-dir …` (network) | 0 | all five v2 files byte-identical to the committed corpus | + +Auditor reproductions, rerun on the fixed code (`validate_records.py --records-dir `; report via `records.protocol_rows`): + +| Scenario | Validator exit | Report | +| --- | ---: | --- | +| unmodified records | 0 | builds; v1 MHA seeds [42, 43, 44] | +| evaluation checkpoint SHA-256 = 64 zeros | 1 | refused | +| evaluation rows swapped to the wrong training runs | 1 | refused | +| evaluation protocol changed | 1 | refused | +| evaluation corpus (test split) hash changed | 1 | refused | +| invalid training commit | 1 | refused | +| invalid manifest hash | 1 | refused | +| legacy `auto` + canonical `mha` seed-42 replicate | 0 | builds; seeds [42, 43, 44] (not inflated) | +| contradictory duplicate seed-42 record | 1 | refused | +| KV summary not derivable from raw samples | 1 | refused | + +Before the fix, on `c256f20`: the six corruption scenarios added no validator errors, and the replicate scenario produced MHA seeds [42, 42, 43, 44]. + +## Published-number changes + +Only the KV-cache timings changed, because the replacement records are new measurements. Every other generated section has numbers identical to `c256f20`. KV-cache memory is identical too. + +| Claim | Before (historical record) | Now (record with raw samples) | +| --- | --- | --- | +| MPS fp32, batch 1, 1–16-token prompts | 0.87–0.88× | 0.88× | +| CPU cache speedup vs uncached SDPA | batch 1: 2.2–16×; batch 8: 3.8–23× | batch 1: 2.2–15×; batch 8: 3.7–23× | +| CPU vs uncached legacy MHA | up to 62× | up to 63× | +| MPS fp32 at 1,024 tokens, batch 1 | 3.4× | 3.6× | +| MPS past the context window | ~1.0× | 0.94–1.00× | +| README headline "2.2–23× on CPU" | unchanged | unchanged | + +Corrected statements: a regenerated checkpoint does not reproduce its file hash (the README previously implied it could be checked against it). The float64 oracle is qualified. The research report's INT8 and corpus-regeneration statements are updated. + +## Historical data compatibility + +- No record was edited or deleted. 41 records: 14 superseded (listed with reasons and replacements in `experiments/records/SUPERSEDED.json`), 27 current. +- Historical training and evaluation records (schema v1) validate under the documented legacy rules: hyperparameters are parsed from the recorded command, and nothing missing is fabricated. Their model identities come from attestations of byte-verified checkpoint files. +- Original backend labels (`auto`) remain in records; canonicalization happens only when reading. +- Legacy checkpoints (no `model_identity`, no `attention_backend`, older config fields) load and hash. +- The v1 corpus, v1 manifest, published record `docs/data/science_run.json` and tag `sdpa-v1-frozen` are unchanged; the original `MiniLM-Lab` worktree is untouched. + +## Reproducibility statement + +- Bitwise reproduction of losses and model identity is verified on the same machine type, CPU, Python 3.12.14 and torch 2.14.1. No cross-device or cross-version bitwise reproducibility is claimed. +- Checkpoint **files** are not reproducible byte-for-byte (they embed elapsed seconds). Use the model identity to check a regenerated model. +- Timings are environment-dependent measurements with medians, p95 and CV derived from stored raw samples; they are not expected to reproduce exactly. + +## Test-set status + +The Faraday test set was evaluated exactly once, after all training (`v2_final_cpu`). Its results are published and must not guide further tuning under the same untouched-test claim. The v2 cleaning rules were designed while viewing its raw text (structure only, never model output). One book does not establish broad cross-author generalization. + +## Outstanding limitations (not blocking) + +| Item | Status | +| --- | --- | +| Historical KV raw samples | lost for the three superseded records; current results use re-measurements | +| Model identity scope | byte-tokenizer checkpoints only (BPE checkpoints raise) | +| Baseline trust | historical identities in fresh clones rest on the pinned audited baseline (integrity anchor in reviewed code, not a signature); the original checkpoints are not distributed | +| New training runs | schema-v2 runs declare their identity but are only metadata-consistent until their artifacts are available or the baseline is extended (strict validation then fails) | +| Artifact-dependent tests | two tests skip where the original checkpoint files are absent | +| Float64 oracle | RMSNorm variance and RoPE angles still float32 (documented, not changed) | +| CUDA | untested; CUDA memory path tested with stubbed counters only | +| MPS training determinism | not verified; all published training ran on CPU | +| Not re-verified | BPE tokenizer training path, LoRA integration, RoPE ablation | +| Code formatting | `ruff format` is not a gate; legacy files not reformatted | +| Local paths in historical records | five records contain absolute local paths; append-only, left as written | +| Report-drift test | fails by design while `docs/` has uncommitted edits | +| Older artifacts | PDF paper and September figures describe single-seed v1 only | +| Release actions | push, merge, release tag and public release await approval | diff --git a/docs/figures/architecture-dark.svg b/docs/figures/architecture-dark.svg new file mode 100644 index 0000000..2818179 --- /dev/null +++ b/docs/figures/architecture-dark.svg @@ -0,0 +1,75 @@ + +MiniLM architecture +Decoder-only Transformer: byte token embedding plus learned position embedding, four pre-norm decoder blocks (RMSNorm, causal self-attention, residual add, RMSNorm, SwiGLU MLP, residual add), final RMSNorm and an LM head tied to the token embedding. + + +next-byte logits (B, T, 256) + + +Linear (LM head) +192 → 256, no bias + + +RMSNorm + +Decoder block × 4 + + + + +SwiGLU MLP +192 → 768 → 192 + + +RMSNorm + + + + + + +Causal self-attention +6 heads, d_head = 32 + + +RMSNorm + + + + + + +Token embedding +256 × 192 + + +Learned position embedding +128 × 192 · added unless RoPE + +input bytes (B, T), T ≤ 128 + + +shared weights (tied) + + +Causal self-attention + + +out_proj · Linear 192 → 192 + +causal attention (selectable backend) + +RoPE on Q, K (positional = rope) + +in_proj · Linear 192 → 576 → Q, K, V + + + + + + +SwiGLU MLP +down( SiLU(gate(n)) ⊙ up(n) ) +gate, up: 192 → 768 · down: 768 → 192 + + diff --git a/docs/figures/architecture-light.svg b/docs/figures/architecture-light.svg new file mode 100644 index 0000000..5cfcd48 --- /dev/null +++ b/docs/figures/architecture-light.svg @@ -0,0 +1,75 @@ + +MiniLM architecture +Decoder-only Transformer: byte token embedding plus learned position embedding, four pre-norm decoder blocks (RMSNorm, causal self-attention, residual add, RMSNorm, SwiGLU MLP, residual add), final RMSNorm and an LM head tied to the token embedding. + + +next-byte logits (B, T, 256) + + +Linear (LM head) +192 → 256, no bias + + +RMSNorm + +Decoder block × 4 + + + + +SwiGLU MLP +192 → 768 → 192 + + +RMSNorm + + + + + + +Causal self-attention +6 heads, d_head = 32 + + +RMSNorm + + + + + + +Token embedding +256 × 192 + + +Learned position embedding +128 × 192 · added unless RoPE + +input bytes (B, T), T ≤ 128 + + +shared weights (tied) + + +Causal self-attention + + +out_proj · Linear 192 → 192 + +causal attention (selectable backend) + +RoPE on Q, K (positional = rope) + +in_proj · Linear 192 → 576 → Q, K, V + + + + + + +SwiGLU MLP +down( SiLU(gate(n)) ⊙ up(n) ) +gate, up: 192 → 768 · down: 768 → 192 + + diff --git a/docs/figures/attention-backends-dark.svg b/docs/figures/attention-backends-dark.svg new file mode 100644 index 0000000..b8a7462 --- /dev/null +++ b/docs/figures/attention-backends-dark.svg @@ -0,0 +1,42 @@ + +Attention backends +Three interchangeable attention kernels sharing the same weights, and which kernel actually runs for each call path and selected backend. + + + +SDPA +F.scaled_dot_product_attention +is_causal, or a bottom-right mask + +nn.MultiheadAttention +explicit boolean causal mask +projections inside the module + +Reference +softmax(QKᵀ / √d) · V +T × T scores, softmax ≥ float32 + +Shared weights: in_proj 192 → 576 · out_proj 192 → 192 · 6 heads × 32 +Kernel that runs, by call path (rows) and selected backend (columns) + +call path +sdpa (default) +mha (legacy v1) +reference +forward, learned positions +SDPA +nn.MultiheadAttention +Reference +forward, RoPE +SDPA +SDPA * +Reference +forward_cached (KV cache) +SDPA +SDPA * +Reference +* RoPE and the KV cache are implemented only on the explicit Q, K, V path, so mha dispatches to SDPA there. +The backend is chosen at run time (set_attention_backend, --attention-backend). It changes no weights +and is not part of the model identity. + + diff --git a/docs/figures/attention-backends-light.svg b/docs/figures/attention-backends-light.svg new file mode 100644 index 0000000..c2bd87d --- /dev/null +++ b/docs/figures/attention-backends-light.svg @@ -0,0 +1,42 @@ + +Attention backends +Three interchangeable attention kernels sharing the same weights, and which kernel actually runs for each call path and selected backend. + + + +SDPA +F.scaled_dot_product_attention +is_causal, or a bottom-right mask + +nn.MultiheadAttention +explicit boolean causal mask +projections inside the module + +Reference +softmax(QKᵀ / √d) · V +T × T scores, softmax ≥ float32 + +Shared weights: in_proj 192 → 576 · out_proj 192 → 192 · 6 heads × 32 +Kernel that runs, by call path (rows) and selected backend (columns) + +call path +sdpa (default) +mha (legacy v1) +reference +forward, learned positions +SDPA +nn.MultiheadAttention +Reference +forward, RoPE +SDPA +SDPA * +Reference +forward_cached (KV cache) +SDPA +SDPA * +Reference +* RoPE and the KV cache are implemented only on the explicit Q, K, V path, so mha dispatches to SDPA there. +The backend is chosen at run time (set_attention_backend, --attention-backend). It changes no weights +and is not part of the model identity. + + diff --git a/docs/figures/attention_decode.svg b/docs/figures/attention_decode.svg new file mode 100644 index 0000000..0116fbd --- /dev/null +++ b/docs/figures/attention_decode.svg @@ -0,0 +1,11 @@ + +Single-token decode against a KV cache.Median time per generated token at batch 1 as the cached context grows. 32 greedy steps per trial, each step synchronized and timed · source: experiments/records/attention +EXPERIMENT 02 / DECODESingle-token decode against a KV cache.Median time per generated token at batch 1 as the cached context grows.SDPAReferencecpu/float32 · batch 10.5112825651210242048sequence length (tokens)decode ms per token (median)cpu/float32 SDPA B=1 L=128: 0.36 ms/stepcpu/float32 SDPA B=1 L=256: 0.46 ms/stepcpu/float32 SDPA B=1 L=512: 0.51 ms/stepcpu/float32 SDPA B=1 L=1024: 0.84 ms/stepcpu/float32 SDPA B=1 L=2048: 1.35 ms/stepcpu/float32 Reference B=1 L=128: 0.42 ms/stepcpu/float32 Reference B=1 L=256: 0.54 ms/stepcpu/float32 Reference B=1 L=512: 0.57 ms/stepcpu/float32 Reference B=1 L=1024: 0.98 ms/stepcpu/float32 Reference B=1 L=2048: 1.66 ms/stepmps/float32 · batch 1123512825651210242048sequence length (tokens)decode ms per token (median)mps/float32 SDPA B=1 L=128: 1.64 ms/stepmps/float32 SDPA B=1 L=256: 1.70 ms/stepmps/float32 SDPA B=1 L=512: 1.76 ms/stepmps/float32 SDPA B=1 L=1024: 1.84 ms/stepmps/float32 SDPA B=1 L=2048: 2.08 ms/stepmps/float32 Reference B=1 L=128: 4.10 ms/stepmps/float32 Reference B=1 L=256: 4.30 ms/stepmps/float32 Reference B=1 L=512: 4.26 ms/stepmps/float32 Reference B=1 L=1024: 4.36 ms/stepmps/float32 Reference B=1 L=2048: 4.72 ms/stepmps/float16 · batch 1123512825651210242048sequence length (tokens)decode ms per token (median)mps/float16 SDPA B=1 L=128: 1.57 ms/stepmps/float16 SDPA B=1 L=256: 1.61 ms/stepmps/float16 SDPA B=1 L=512: 1.71 ms/stepmps/float16 SDPA B=1 L=1024: 1.78 ms/stepmps/float16 SDPA B=1 L=2048: 1.90 ms/stepmps/float16 Reference B=1 L=128: 4.15 ms/stepmps/float16 Reference B=1 L=256: 4.17 ms/stepmps/float16 Reference B=1 L=512: 4.25 ms/stepmps/float16 Reference B=1 L=1024: 4.30 ms/stepmps/float16 Reference B=1 L=2048: 4.41 ms/step32 greedy steps per trial, each step synchronized and timed · source: experiments/records/attention + \ No newline at end of file diff --git a/docs/figures/attention_memory.svg b/docs/figures/attention_memory.svg new file mode 100644 index 0000000..8e559a4 --- /dev/null +++ b/docs/figures/attention_memory.svg @@ -0,0 +1,11 @@ + +Explicit attention pays for the full score matrix.Peak extra memory during one forward pass, fresh process per point. Log scales on both axes. CPU: process peak-RSS delta. MPS: live allocator bytes sampled after every op. Dashed legacy MHA coincides with SDPA on MPS. +EXPERIMENT 02 / MEMORYExplicit attention pays for the full score matrix.Peak extra memory during one forward pass, fresh process per point. Log scales on both axes.MHA (legacy v1)SDPAReferencecpu/float32 · batch 820501002005001000200012825651210242048sequence length (tokens)peak extra memory (MiB)cpu/float32 SDPA B=8 L=128: 18.2 MiBcpu/float32 SDPA B=8 L=256: 34.8 MiBcpu/float32 SDPA B=8 L=512: 35.2 MiBcpu/float32 SDPA B=8 L=1024: 134.2 MiBcpu/float32 SDPA B=8 L=2048: 222.8 MiBcpu/float32 Reference B=8 L=128: 18.2 MiBcpu/float32 Reference B=8 L=256: 34.3 MiBcpu/float32 Reference B=8 L=512: 131.3 MiBcpu/float32 Reference B=8 L=1024: 517.1 MiBcpu/float32 Reference B=8 L=2048: 1764.6 MiBcpu/float32 MHA (legacy v1) B=8 L=128: 20.7 MiBcpu/float32 MHA (legacy v1) B=8 L=256: 39.2 MiBcpu/float32 MHA (legacy v1) B=8 L=512: 131.9 MiBcpu/float32 MHA (legacy v1) B=8 L=1024: 582.2 MiBcpu/float32 MHA (legacy v1) B=8 L=2048: 1834.7 MiBmps/float32 · batch 8102050100200500100012825651210242048sequence length (tokens)peak extra memory (MiB)mps/float32 SDPA B=8 L=128: 11.3 MiBmps/float32 SDPA B=8 L=256: 22.5 MiBmps/float32 SDPA B=8 L=512: 45.0 MiBmps/float32 SDPA B=8 L=1024: 90.0 MiBmps/float32 SDPA B=8 L=2048: 180.0 MiBmps/float32 Reference B=8 L=128: 11.3 MiBmps/float32 Reference B=8 L=256: 33.0 MiBmps/float32 Reference B=8 L=512: 114.0 MiBmps/float32 Reference B=8 L=1024: 420.0 MiBmps/float32 Reference B=8 L=2048: 1608.0 MiBmps/float32 MHA (legacy v1) B=8 L=128: 11.3 MiBmps/float32 MHA (legacy v1) B=8 L=256: 22.5 MiBmps/float32 MHA (legacy v1) B=8 L=512: 45.0 MiBmps/float32 MHA (legacy v1) B=8 L=1024: 90.0 MiBmps/float32 MHA (legacy v1) B=8 L=2048: 180.0 MiBmps/float16 · batch 81020501002005001000200012825651210242048sequence length (tokens)peak extra memory (MiB)mps/float16 SDPA B=8 L=128: 5.6 MiBmps/float16 SDPA B=8 L=256: 12.0 MiBmps/float16 SDPA B=8 L=512: 22.5 MiBmps/float16 SDPA B=8 L=1024: 45.0 MiBmps/float16 SDPA B=8 L=2048: 90.0 MiBmps/float16 Reference B=8 L=128: 9.4 MiBmps/float16 Reference B=8 L=256: 33.8 MiBmps/float16 Reference B=8 L=512: 127.5 MiBmps/float16 Reference B=8 L=1024: 495.0 MiBmps/float16 Reference B=8 L=2048: 1950.0 MiBmps/float16 MHA (legacy v1) B=8 L=128: 5.6 MiBmps/float16 MHA (legacy v1) B=8 L=256: 11.3 MiBmps/float16 MHA (legacy v1) B=8 L=512: 22.5 MiBmps/float16 MHA (legacy v1) B=8 L=1024: 45.0 MiBmps/float16 MHA (legacy v1) B=8 L=2048: 90.0 MiBCPU: process peak-RSS delta. MPS: live allocator bytes sampled after every op. Dashed legacy MHA coincides with SDPA on MPS. + \ No newline at end of file diff --git a/docs/figures/attention_speedup.svg b/docs/figures/attention_speedup.svg new file mode 100644 index 0000000..88f23ea --- /dev/null +++ b/docs/figures/attention_speedup.svg @@ -0,0 +1,11 @@ + +How much faster is fused SDPA than explicit attention?Full-sequence forward time, reference divided by SDPA (median). Above the dashed line, SDPA is faster. Same seeded weights, inputs, dtype and device per panel · 10 repeats after 3 warmups · source: experiments/records/attention +EXPERIMENT 02 / ATTENTIONHow much faster is fused SDPA than explicit attention?Full-sequence forward time, reference divided by SDPA (median). Above the dashed line, SDPA is faster.batch 1batch 4batch 8cpu/float321×2×3×5×12825651210242048sequence length (tokens)reference ÷ SDPA forward timecpu/float32 B=1 L=128: reference/SDPA = 1.25xcpu/float32 B=1 L=256: reference/SDPA = 1.24xcpu/float32 B=1 L=512: reference/SDPA = 1.44xcpu/float32 B=1 L=1024: reference/SDPA = 2.48xcpu/float32 B=1 L=2048: reference/SDPA = 3.40xcpu/float32 B=4 L=128: reference/SDPA = 1.19xcpu/float32 B=4 L=256: reference/SDPA = 1.43xcpu/float32 B=4 L=512: reference/SDPA = 1.66xcpu/float32 B=4 L=1024: reference/SDPA = 2.97xcpu/float32 B=4 L=2048: reference/SDPA = 3.52xcpu/float32 B=8 L=128: reference/SDPA = 1.18xcpu/float32 B=8 L=256: reference/SDPA = 1.39xcpu/float32 B=8 L=512: reference/SDPA = 1.44xcpu/float32 B=8 L=1024: reference/SDPA = 2.68xcpu/float32 B=8 L=2048: reference/SDPA = 3.90xmps/float321×2×3×5×12825651210242048sequence length (tokens)reference ÷ SDPA forward timemps/float32 B=1 L=128: reference/SDPA = 3.08xmps/float32 B=1 L=256: reference/SDPA = 2.80xmps/float32 B=1 L=512: reference/SDPA = 2.71xmps/float32 B=1 L=1024: reference/SDPA = 3.14xmps/float32 B=1 L=2048: reference/SDPA = 4.27xmps/float32 B=4 L=128: reference/SDPA = 2.39xmps/float32 B=4 L=256: reference/SDPA = 2.15xmps/float32 B=4 L=512: reference/SDPA = 2.24xmps/float32 B=4 L=1024: reference/SDPA = 2.83xmps/float32 B=4 L=2048: reference/SDPA = 3.96xmps/float32 B=8 L=128: reference/SDPA = 1.96xmps/float32 B=8 L=256: reference/SDPA = 1.85xmps/float32 B=8 L=512: reference/SDPA = 2.24xmps/float32 B=8 L=1024: reference/SDPA = 2.66xmps/float32 B=8 L=2048: reference/SDPA = 3.92xmps/float161×2×3×5×12825651210242048sequence length (tokens)reference ÷ SDPA forward timemps/float16 B=1 L=128: reference/SDPA = 2.95xmps/float16 B=1 L=256: reference/SDPA = 2.94xmps/float16 B=1 L=512: reference/SDPA = 3.01xmps/float16 B=1 L=1024: reference/SDPA = 3.70xmps/float16 B=1 L=2048: reference/SDPA = 5.36xmps/float16 B=4 L=128: reference/SDPA = 2.23xmps/float16 B=4 L=256: reference/SDPA = 2.33xmps/float16 B=4 L=512: reference/SDPA = 2.70xmps/float16 B=4 L=1024: reference/SDPA = 3.62xmps/float16 B=4 L=2048: reference/SDPA = 5.09xmps/float16 B=8 L=128: reference/SDPA = 2.07xmps/float16 B=8 L=256: reference/SDPA = 2.11xmps/float16 B=8 L=512: reference/SDPA = 2.71xmps/float16 B=8 L=1024: reference/SDPA = 3.42xmps/float16 B=8 L=2048: reference/SDPA = 5.30xSame seeded weights, inputs, dtype and device per panel · 10 repeats after 3 warmups · source: experiments/records/attention + \ No newline at end of file diff --git a/docs/figures/kv-cache-dark.svg b/docs/figures/kv-cache-dark.svg new file mode 100644 index 0000000..80b3d9e --- /dev/null +++ b/docs/figures/kv-cache-dark.svg @@ -0,0 +1,35 @@ + +KV-cache inference +Cached decoding: one prefill pass builds per-layer keys and values, then each step processes one new token and appends its key and value. Uncached decoding recomputes the full window every step. + + +Cached · forward_cached + +Prefill +prompt of P tokens +one pass over all P positions + + +KV cache, per layer +K, V: (B, 6, P, 32) + + +Decode step +one new token (position P) +attends to the cache and itself + +append new k, v (torch.cat): cache grows by one position per token +Beyond 128 tokens the cache is rebuilt from the last 128 tokens at every step (sliding window). +Uncached · forward + +Every step: full forward over the last ≤ 128 tokens +keys and values recomputed for every position + + +Logits at last position +→ next token + +append token, repeat +Cache size: 2 × 4 layers × 192 values per position and sequence (fp32: 6,144 bytes). + + diff --git a/docs/figures/kv-cache-light.svg b/docs/figures/kv-cache-light.svg new file mode 100644 index 0000000..b8cd0e6 --- /dev/null +++ b/docs/figures/kv-cache-light.svg @@ -0,0 +1,35 @@ + +KV-cache inference +Cached decoding: one prefill pass builds per-layer keys and values, then each step processes one new token and appends its key and value. Uncached decoding recomputes the full window every step. + + +Cached · forward_cached + +Prefill +prompt of P tokens +one pass over all P positions + + +KV cache, per layer +K, V: (B, 6, P, 32) + + +Decode step +one new token (position P) +attends to the cache and itself + +append new k, v (torch.cat): cache grows by one position per token +Beyond 128 tokens the cache is rebuilt from the last 128 tokens at every step (sliding window). +Uncached · forward + +Every step: full forward over the last ≤ 128 tokens +keys and values recomputed for every position + + +Logits at last position +→ next token + +append token, repeat +Cache size: 2 × 4 layers × 192 values per position and sequence (fp32: 6,144 bytes). + + diff --git a/docs/figures/kv_cache_speedup.svg b/docs/figures/kv_cache_speedup.svg new file mode 100644 index 0000000..f305a05 --- /dev/null +++ b/docs/figures/kv_cache_speedup.svg @@ -0,0 +1,11 @@ + +When does the KV cache pay off?Time to generate 64 tokens, uncached (SDPA) divided by cached. Below the dashed line, caching is slower. Same weights, prompts, dtype and attention kernel · 5 repeats · source: experiments/records/kv_cache +EXPERIMENT 03 / KV CACHEWhen does the KV cache pay off?Time to generate 64 tokens, uncached (SDPA) divided by cached. Below the dashed line, caching is slower.batch 1batch 8cpu/float321×2×5×10×20×1161285121024prompt length (tokens)uncached ÷ cached total timecpu/float32 B=1 prompt=1: cached is 2.17x uncached SDPAcpu/float32 B=1 prompt=16: cached is 2.62x uncached SDPAcpu/float32 B=1 prompt=128: cached is 4.41x uncached SDPAcpu/float32 B=1 prompt=512: cached is 9.76x uncached SDPAcpu/float32 B=1 prompt=1024: cached is 15.46x uncached SDPAcpu/float32 B=8 prompt=1: cached is 3.71x uncached SDPAcpu/float32 B=8 prompt=16: cached is 4.57x uncached SDPAcpu/float32 B=8 prompt=128: cached is 9.62x uncached SDPAcpu/float32 B=8 prompt=512: cached is 19.38x uncached SDPAcpu/float32 B=8 prompt=1024: cached is 23.34x uncached SDPAmps/float321×2×5×10×1161285121024prompt length (tokens)uncached ÷ cached total timemps/float32 B=1 prompt=1: cached is 0.88x uncached SDPAmps/float32 B=1 prompt=16: cached is 0.88x uncached SDPAmps/float32 B=1 prompt=128: cached is 1.12x uncached SDPAmps/float32 B=1 prompt=512: cached is 1.98x uncached SDPAmps/float32 B=1 prompt=1024: cached is 3.56x uncached SDPAmps/float32 B=8 prompt=1: cached is 1.32x uncached SDPAmps/float32 B=8 prompt=16: cached is 1.45x uncached SDPAmps/float32 B=8 prompt=128: cached is 2.16x uncached SDPAmps/float32 B=8 prompt=512: cached is 6.05x uncached SDPAmps/float32 B=8 prompt=1024: cached is 9.83x uncached SDPAmps/float161×2×5×10×1161285121024prompt length (tokens)uncached ÷ cached total timemps/float16 B=1 prompt=1: cached is 0.89x uncached SDPAmps/float16 B=1 prompt=16: cached is 0.91x uncached SDPAmps/float16 B=1 prompt=128: cached is 1.20x uncached SDPAmps/float16 B=1 prompt=512: cached is 1.94x uncached SDPAmps/float16 B=1 prompt=1024: cached is 3.39x uncached SDPAmps/float16 B=8 prompt=1: cached is 1.13x uncached SDPAmps/float16 B=8 prompt=16: cached is 1.46x uncached SDPAmps/float16 B=8 prompt=128: cached is 2.53x uncached SDPAmps/float16 B=8 prompt=512: cached is 4.91x uncached SDPAmps/float16 B=8 prompt=1024: cached is 8.70x uncached SDPASame weights, prompts, dtype and attention kernel · 5 repeats · source: experiments/records/kv_cache + \ No newline at end of file diff --git a/docs/figures/reproducibility-dark.svg b/docs/figures/reproducibility-dark.svg new file mode 100644 index 0000000..3e6de73 --- /dev/null +++ b/docs/figures/reproducibility-dark.svg @@ -0,0 +1,72 @@ + +Reproducibility and verification +Checkpoint files stay local. The repository holds data manifests, training and evaluation records, self-asserted identity attestations, and an audited baseline whose hash is pinned in code. Strict validation accepts a model identity only if it is verified from the checkpoint file or from the pinned baseline. + + + +LOCAL ONLY · not distributed + +REPOSITORY · versioned, reviewed + +VALIDATION + +train_tiny.py +seeded training run + + +Checkpoint files +best_checkpoint.pt +tiny_checkpoint.pt + +Corpus files and manifests +SHA-256 per split + +Training record (record_training_run.py) +command · seed · data hashes +checkpoint SHA-256 · fingerprint + +Evaluation record +joined by checkpoint SHA-256 + +Identity attestations +self-asserted: compared, never trusted + +Audited baseline (build_identity_baseline.py) +checkpoint SHA-256 → model fingerprint + +BASELINE_SHA256 pin +constant in records.py (reviewed code) + +hash of + + +bytes = recorded +SHA-256, then +fingerprint from +tensors + + + + + + + +validate_records.py +--strict + +Evidence per checkpoint +artifact_verified · pass +trusted_baseline_verified · pass +metadata_consistent_only · fail +unverified · fail + + +Report builder +fails closed on any problem + +baseline trusted only if +its hash equals the pin + +if present: rehash bytes, recompute fingerprint + + diff --git a/docs/figures/reproducibility-light.svg b/docs/figures/reproducibility-light.svg new file mode 100644 index 0000000..b77e672 --- /dev/null +++ b/docs/figures/reproducibility-light.svg @@ -0,0 +1,72 @@ + +Reproducibility and verification +Checkpoint files stay local. The repository holds data manifests, training and evaluation records, self-asserted identity attestations, and an audited baseline whose hash is pinned in code. Strict validation accepts a model identity only if it is verified from the checkpoint file or from the pinned baseline. + + + +LOCAL ONLY · not distributed + +REPOSITORY · versioned, reviewed + +VALIDATION + +train_tiny.py +seeded training run + + +Checkpoint files +best_checkpoint.pt +tiny_checkpoint.pt + +Corpus files and manifests +SHA-256 per split + +Training record (record_training_run.py) +command · seed · data hashes +checkpoint SHA-256 · fingerprint + +Evaluation record +joined by checkpoint SHA-256 + +Identity attestations +self-asserted: compared, never trusted + +Audited baseline (build_identity_baseline.py) +checkpoint SHA-256 → model fingerprint + +BASELINE_SHA256 pin +constant in records.py (reviewed code) + +hash of + + +bytes = recorded +SHA-256, then +fingerprint from +tensors + + + + + + + +validate_records.py +--strict + +Evidence per checkpoint +artifact_verified · pass +trusted_baseline_verified · pass +metadata_consistent_only · fail +unverified · fail + + +Report builder +fails closed on any problem + +baseline trusted only if +its hash equals the pin + +if present: rehash bytes, recompute fingerprint + + diff --git a/docs/figures/src/make_readme_diagrams.py b/docs/figures/src/make_readme_diagrams.py new file mode 100644 index 0000000..769742a --- /dev/null +++ b/docs/figures/src/make_readme_diagrams.py @@ -0,0 +1,268 @@ +"""Generate the README diagrams as monochrome, transparent SVGs (light and dark). + +Every component, dimension and dispatch rule is taken from src/minilm_lab +(model.py, kv_cache.py, records.py). Run from the repository root: + + python docs/figures/src/make_readme_diagrams.py +""" + +from html import escape +from pathlib import Path + +OUT = Path(__file__).resolve().parents[1] +FONT = "Helvetica Neue, Helvetica, Arial, sans-serif" +THEMES = {"light": "#000000", "dark": "#ffffff"} +STROKE = 1.2 + + +class Figure: + def __init__(self, width: int, height: int, color: str, title: str, description: str) -> None: + self.width, self.height, self.color = width, height, color + self.title, self.description = title, description + self.parts: list[str] = [] + + # -------------------------------------------------------------- primitives + def text(self, x, y, s, size=13, anchor="middle", weight=400, muted=False, rotate=None) -> None: + opacity = ' fill-opacity="0.68"' if muted else "" + transform = f' transform="rotate({rotate} {x} {y})"' if rotate else "" + self.parts.append( + f'{escape(s)}' + ) + + def rect(self, x, y, w, h, dashed=False, radius=3) -> None: + dash = ' stroke-dasharray="4 3"' if dashed else "" + self.parts.append(f'') + + def box(self, x, y, w, h, title, *details, dashed=False, size=13, weight=500) -> None: + """Box with a title line and muted detail lines, vertically centred.""" + self.rect(x, y, w, h, dashed) + lines = [(title, size, weight, False)] + [(d, size - 2, 400, True) for d in details] + step = 15 + top = y + h / 2 - step * (len(lines) - 1) / 2 + for index, (s, font, line_weight, muted) in enumerate(lines): + self.text(x + w / 2, top + index * step, s, font, weight=line_weight, muted=muted) + + def path(self, points, arrow=True, dashed=False, dotted=False) -> None: + d = "M " + " L ".join(f"{px} {py}" for px, py in points) + dash = ' stroke-dasharray="4 3"' if dashed else ' stroke-dasharray="1.5 3"' if dotted else "" + marker = ' marker-end="url(#arrow)"' if arrow else "" + self.parts.append(f'') + + def plus(self, cx, cy, r=9) -> None: + self.parts.append(f'') + self.parts.append(f'') + + def render(self) -> str: + c = self.color + return ( + f'\n' + f'{escape(self.title)}\n{escape(self.description)}\n' + f'\n' + f'\n' + + "\n".join(p.replace("\n\n" + ) + + +# ================================================================== architecture + + +def architecture(color: str) -> Figure: + f = Figure(720, 712, color, "MiniLM architecture", + "Decoder-only Transformer: byte token embedding plus learned position embedding, four pre-norm decoder blocks " + "(RMSNorm, causal self-attention, residual add, RMSNorm, SwiGLU MLP, residual add), final RMSNorm and an LM head " + "tied to the token embedding.") + cx, x0, w = 210, 95, 230 + # output and head + f.text(cx, 18, "next-byte logits (B, T, 256)", 12, muted=True) + f.path([(cx, 52), (cx, 30)]) + f.box(x0, 52, w, 40, "Linear (LM head)", "192 → 256, no bias") + f.path([(cx, 120), (cx, 92)]) + f.box(x0, 120, w, 30, "RMSNorm") + # decoder block group + f.rect(52, 168, 298, 372, dashed=True) + f.text(64, 184, "Decoder block × 4", 12, anchor="start", weight=500) + f.plus(cx, 210) + f.path([(cx, 201), (cx, 150)]) + f.box(x0, 238, w, 42, "SwiGLU MLP", "192 → 768 → 192") + f.path([(cx, 238), (cx, 219)]) + f.box(x0, 304, w, 28, "RMSNorm") + f.path([(cx, 304), (cx, 280)]) + f.path([(cx, 363), (cx, 332)]) + f.path([(cx, 352), (78, 352), (78, 210), (201, 210)]) # residual 2 + f.plus(cx, 372) + f.box(x0, 400, w, 42, "Causal self-attention", "6 heads, d_head = 32") + f.path([(cx, 400), (cx, 381)]) + f.box(x0, 466, w, 28, "RMSNorm") + f.path([(cx, 466), (cx, 442)]) + f.path([(cx, 573), (cx, 494)]) + f.path([(cx, 515), (78, 515), (78, 372), (201, 372)]) # residual 1 + # embeddings + f.plus(cx, 582) + f.box(x0, 612, w, 40, "Token embedding", "256 × 192") + f.path([(cx, 612), (cx, 591)]) + f.box(400, 562, 260, 40, "Learned position embedding", "128 × 192 · added unless RoPE") + f.path([(400, 582), (219, 582)]) + f.text(cx, 690, "input bytes (B, T), T ≤ 128", 12, muted=True) + f.path([(cx, 680), (cx, 652)]) + # weight tying + f.path([(x0, 632), (30, 632), (30, 72), (x0, 72)], arrow=False, dashed=True) + f.text(20, 352, "shared weights (tied)", 11, muted=True, rotate=-90) + # attention detail + f.path([(x0 + w, 421), (400, 421)], arrow=False, dotted=True) + f.rect(400, 312, 300, 226, radius=4) + f.text(412, 328, "Causal self-attention", 12, anchor="start", weight=500) + f.path([(550, 352), (550, 340)]) + f.box(420, 352, 260, 26, "out_proj · Linear 192 → 192", size=12, weight=400) + f.box(420, 394, 260, 26, "causal attention (selectable backend)", size=12, weight=400) + f.box(420, 438, 260, 26, "RoPE on Q, K (positional = rope)", dashed=True, size=12, weight=400) + f.box(420, 482, 260, 26, "in_proj · Linear 192 → 576 → Q, K, V", size=12, weight=400) + f.path([(550, 526), (550, 508)]) + f.path([(550, 482), (550, 464)]) + f.path([(550, 438), (550, 420)]) + f.path([(550, 394), (550, 378)]) + # MLP detail + f.path([(x0 + w, 259), (400, 259)], arrow=False, dotted=True) + f.rect(400, 214, 300, 90, radius=4) + f.text(412, 230, "SwiGLU MLP", 12, anchor="start", weight=500) + f.text(550, 258, "down( SiLU(gate(n)) ⊙ up(n) )", 13) + f.text(550, 282, "gate, up: 192 → 768 · down: 768 → 192", 11, muted=True) + return f + + +# ================================================================== attention backends + + +def attention_backends(color: str) -> Figure: + f = Figure(760, 356, color, "Attention backends", + "Three interchangeable attention kernels sharing the same weights, and which kernel actually runs for each " + "call path and selected backend.") + columns = [(20, "SDPA", "F.scaled_dot_product_attention", "is_causal, or a bottom-right mask"), + (270, "nn.MultiheadAttention", "explicit boolean causal mask", "projections inside the module"), + (520, "Reference", "softmax(QKᵀ / √d) · V", "T × T scores, softmax ≥ float32")] + for x, title, a, b in columns: + f.box(x, 8, 220, 62, title, a, b) + f.rect(20, 84, 720, 30) + f.text(380, 99, "Shared weights: in_proj 192 → 576 · out_proj 192 → 192 · 6 heads × 32", 12) + # dispatch table + f.text(20, 144, "Kernel that runs, by call path (rows) and selected backend (columns)", 12, anchor="start", weight=500) + top, row, cols = 160, 30, [20, 250, 410, 580, 740] + headers = ["call path", "sdpa (default)", "mha (legacy v1)", "reference"] + rows = [("forward, learned positions", "SDPA", "nn.MultiheadAttention", "Reference"), + ("forward, RoPE", "SDPA", "SDPA *", "Reference"), + ("forward_cached (KV cache)", "SDPA", "SDPA *", "Reference")] + f.parts.append(f'') + for index, header in enumerate(headers): + anchor_x = cols[index] + 8 if index == 0 else (cols[index] + cols[index + 1]) / 2 + f.text(anchor_x, top + row / 2, header, 12, anchor="start" if index == 0 else "middle", weight=500) + for r, values in enumerate(rows): + y = top + row * (r + 1) + row / 2 + for index, value in enumerate(values): + anchor_x = cols[index] + 8 if index == 0 else (cols[index] + cols[index + 1]) / 2 + f.text(anchor_x, y, value, 12, anchor="start" if index == 0 else "middle", muted=index == 0) + f.text(20, 304, "* RoPE and the KV cache are implemented only on the explicit Q, K, V path, so mha dispatches to SDPA there.", + 11, anchor="start", muted=True) + f.text(20, 324, "The backend is chosen at run time (set_attention_backend, --attention-backend). It changes no weights", + 11, anchor="start", muted=True) + f.text(20, 342, "and is not part of the model identity.", 11, anchor="start", muted=True) + return f + + +# ================================================================== KV cache + + +def kv_cache(color: str) -> Figure: + f = Figure(760, 348, color, "KV-cache inference", + "Cached decoding: one prefill pass builds per-layer keys and values, then each step processes one new token " + "and appends its key and value. Uncached decoding recomputes the full window every step.") + f.text(20, 16, "Cached · forward_cached", 12, anchor="start", weight=500) + f.box(20, 32, 190, 58, "Prefill", "prompt of P tokens", "one pass over all P positions") + f.path([(210, 61), (262, 61)]) + f.box(262, 32, 210, 58, "KV cache, per layer", "K, V: (B, 6, P, 32)") + f.path([(472, 61), (524, 61)]) + f.box(524, 32, 216, 58, "Decode step", "one new token (position P)", "attends to the cache and itself") + f.path([(632, 90), (632, 112), (367, 112), (367, 90)]) + f.text(500, 126, "append new k, v (torch.cat): cache grows by one position per token", 11, muted=True) + f.text(20, 152, "Beyond 128 tokens the cache is rebuilt from the last 128 tokens at every step (sliding window).", + 11, anchor="start", muted=True) + f.text(20, 196, "Uncached · forward", 12, anchor="start", weight=500) + f.box(20, 212, 452, 58, "Every step: full forward over the last ≤ 128 tokens", "keys and values recomputed for every position") + f.path([(472, 241), (524, 241)]) + f.box(524, 212, 216, 58, "Logits at last position", "→ next token") + f.path([(632, 270), (632, 292), (246, 292), (246, 270)]) + f.text(439, 306, "append token, repeat", 11, muted=True) + f.text(20, 336, "Cache size: 2 × 4 layers × 192 values per position and sequence (fp32: 6,144 bytes).", + 11, anchor="start", muted=True) + return f + + +# ================================================================== reproducibility + + +def reproducibility(color: str) -> Figure: + f = Figure(884, 470, color, "Reproducibility and verification", + "Checkpoint files stay local. The repository holds data manifests, training and evaluation records, " + "self-asserted identity attestations, and an audited baseline whose hash is pinned in code. Strict validation " + "accepts a model identity only if it is verified from the checkpoint file or from the pinned baseline.") + # trust zones + f.rect(12, 12, 220, 446, dashed=True) + f.text(24, 30, "LOCAL ONLY · not distributed", 10.5, anchor="start", weight=500) + f.rect(252, 12, 340, 446, dashed=True) + f.text(264, 30, "REPOSITORY · versioned, reviewed", 10.5, anchor="start", weight=500) + f.rect(612, 12, 260, 446, dashed=True) + f.text(624, 30, "VALIDATION", 10.5, anchor="start", weight=500) + # local + f.box(30, 52, 184, 44, "train_tiny.py", "seeded training run") + f.path([(122, 96), (122, 124)]) + f.box(30, 124, 184, 58, "Checkpoint files", "best_checkpoint.pt", "tiny_checkpoint.pt") + # repository + f.box(270, 52, 304, 40, "Corpus files and manifests", "SHA-256 per split") + f.box(270, 108, 304, 58, "Training record (record_training_run.py)", "command · seed · data hashes", + "checkpoint SHA-256 · fingerprint") + f.box(270, 182, 304, 40, "Evaluation record", "joined by checkpoint SHA-256") + f.box(270, 238, 304, 40, "Identity attestations", "self-asserted: compared, never trusted") + f.box(270, 300, 304, 40, "Audited baseline (build_identity_baseline.py)", "checkpoint SHA-256 → model fingerprint") + f.box(270, 376, 304, 40, "BASELINE_SHA256 pin", "constant in records.py (reviewed code)") + f.path([(422, 340), (422, 376)], arrow=False, dashed=True) + f.text(430, 358, "hash of", 10.5, anchor="start", muted=True) + # checkpoint files -> training record, -> baseline + f.path([(214, 150), (270, 150)]) + f.path([(122, 182), (122, 320), (270, 320)]) + f.text(130, 226, "bytes = recorded", 10.5, anchor="start", muted=True) + f.text(130, 242, "SHA-256, then", 10.5, anchor="start", muted=True) + f.text(130, 258, "fingerprint from", 10.5, anchor="start", muted=True) + f.text(130, 274, "tensors", 10.5, anchor="start", muted=True) + # records bus -> validator + for y in (74, 137, 202, 258, 320): + f.path([(574, y), (602, y)], arrow=False) + f.path([(602, 320), (602, 74), (630, 74)]) + # validation + f.box(630, 52, 204, 44, "validate_records.py", "--strict") + f.box(630, 128, 204, 96, "Evidence per checkpoint", "artifact_verified · pass", "trusted_baseline_verified · pass", + "metadata_consistent_only · fail", "unverified · fail") + f.path([(732, 96), (732, 128)]) + f.box(630, 256, 204, 44, "Report builder", "fails closed on any problem") + f.path([(732, 224), (732, 256)]) + f.text(732, 326, "baseline trusted only if", 10.5, muted=True) + f.text(732, 342, "its hash equals the pin", 10.5, muted=True) + # checkpoint files -> validator, if present (crosses the trust boundary) + f.path([(60, 182), (60, 440), (852, 440), (852, 74), (834, 74)], dashed=True) + f.text(422, 430, "if present: rehash bytes, recompute fingerprint", 10.5, muted=True) + return f + + +def main() -> None: + for name, build in (("architecture", architecture), ("attention-backends", attention_backends), + ("kv-cache", kv_cache), ("reproducibility", reproducibility)): + for theme, color in THEMES.items(): + (OUT / f"{name}-{theme}.svg").write_text(build(color).render(), encoding="utf-8") + print("wrote 8 SVG files to", OUT) + + +if __name__ == "__main__": + main() diff --git a/docs/implementation_audit.md b/docs/implementation_audit.md index 5cf41f6..9789704 100644 --- a/docs/implementation_audit.md +++ b/docs/implementation_audit.md @@ -1,3 +1,94 @@ +# Implementation audit · October 2026 + +Audit of `1fb8f37` on branch `experiments/attention-kv-audit`, Apple M4 (16 GB), +macOS 27.0.1, Python 3.12.14, torch 2.14.1. Every "verified" row cites the test +or record that verifies it. Baseline before any change: 16/16 tests passed, ruff clean. + +## Verified + +| Claim | Evidence | +| --- | --- | +| Published run reproduces bit-for-bit | Retraining seed 42 on CPU gives identical train and validation loss at all four checkpoints (torch 2.14.1 vs published 2.14.0) | +| Attention changes leave the default path unchanged | Seed 42 retrained on the modified code gives a bitwise-identical checkpoint | +| Attention tensor shapes | `test_cached_backends_match_full_forward_in_chunks`, `test_cache_grows_by_one_position_per_decoded_token`: cache is `(batch, heads, positions, head_width)` | +| Causal masking | `test_future_tokens_do_not_change_past_logits`: perturbing tokens ≥ t leaves logits < t unchanged, for all three backends × learned/RoPE | +| Cache mask alignment | `test_causal_mask_is_bottom_right_aligned`; chunked prefill (1, 8, 1, 22 tokens) matches the full forward | +| All backends compute the same function | `test_backends_match_float64_oracle` (< 1e-4 max logit error); trained checkpoints: identical perplexity to 4 decimals in fp32 on CPU and MPS **Update (release audit):** the float64 oracle still computes RMSNorm variance and RoPE angles in float32. | +| RoPE correctness | matches complex-number rotation exactly; scores depend only on m − n; norm preserved | +| KV cache equivalence | cached and uncached greedy decoding are token-identical for single-token prompts, prompts at the context limit, batch 4, and 40 tokens past the window (sliding refresh) | +| Batch independence | each row of a batched cached decode equals decoding that row alone | +| Gradient accumulation | 2 × 4-sequence microbatches with `loss / 2` reproduce the 8-sequence gradient (`test_gradient_accumulation_matches_full_batch`) | +| Checkpoint resume | 3 + 3 resumed steps give bitwise-identical weights to 6 uninterrupted steps | +| Train/validation separation | separate files whose hashes match the manifest; 1 of 318 validation paragraphs (> 40 chars) also appears in train; 0.04% of 64-character validation windows occur verbatim in train | +| Evaluation counting | every held-out target scored once (existing test) | + +## Defects found and fixed + +| Defect | Evidence | Fix | +| --- | --- | --- | +| RoPE positions and angles computed in the activation dtype | bf16 max error 0.38 at 128 positions and 7.3 at 2048, versus a 0.008–0.015 rounding floor; fp16 2.8 at 2048 | Angles in float32, cast after `cos`/`sin`; `test_rope_half_precision_error_is_at_rounding_level` | +| `compare_attention_quality.py` (new in this branch) cast MPS tensors to float64 | crash on MPS | Move to CPU before upcasting | +| MPS memory measured via driver pool size (new in this branch) | values quantized to ~1 GiB heaps | Per-op sampling of live allocator bytes; first MPS records kept but superseded | + +## Findings not changed (need a decision or are out of scope) + +| Finding | Consequence | +| --- | --- | +| Default attention (`nn.MultiheadAttention` with an explicit boolean mask) is used by training, evaluation and uncached generation | On CPU at B=8, L=2048 it is 4.5× slower and uses 8× more memory than SDPA with `is_causal` (see EXPERIMENT_RESULTS.md). Switching the default changes training numerics slightly, so it should be its own experiment **Update:** SDPA is now a selectable training backend; matched experiments in EXPERIMENT_RESULTS.md (Experiment 04, Part B) support making it the default, which is left as an explicit decision | +| 26% of all validation bytes (31.5% of Darwin's) are the book's alphabetical index | Held-out perplexity mixes prose with index entries; per-source numbers are affected most **Update:** unchanged in legacy science-v1; removed in science-v2 | +| The validation split both selects the best checkpoint and is reported | No untouched test split. The best checkpoint was the final step in all three seeds, so selection bias is currently nil, but the protocol allows it **Update:** science-v2 adds an untouched test book (Faraday), evaluated once per selected checkpoint | +| Training plateaus near unigram entropy (≈3.27 nats) until ~750 of 1,000 steps in every seed | The published checkpoint is captured as it leaves the plateau. Seeds 42/43/44 give perplexity 17.58 / 18.99 / 21.07, so the single-seed headline is optimistic | +| Evaluation uses non-overlapping 128-token chunks | Early tokens in each chunk see little context. Perplexity is pessimistic relative to sliding-window evaluation (correctly counted, but not comparable to sliding-window numbers) | +| Resumed runs log an extra evaluation at each segment boundary | Weights are identical; `metrics` gains a row, and `best_checkpoint.pt` could be chosen at a step an uninterrupted run never evaluates | +| RoPE path ignores attention dropout; RoPE models still allocate the learned position table | Inert at the published `dropout=0`; inflates RoPE parameter counts | +| `benchmark_generation.py` has no warmup and one sample; it and `generate.py` load checkpoints with `weights_only=False` | Its single timing is not a benchmark; unsafe deserialization of untrusted checkpoints | + +## Unverified assumptions + +- Upstream Gutenberg sources were not re-downloaded. The local split files match the manifest hashes, but the raw-source hashes were not re-checked. **Update:** re-downloaded on 2026-10-22; the Darwin raw hash changed, but both v1 splits regenerate byte-identically. +- MPS training determinism and resume on MPS (the RNG state saved is CPU-only; batch sampling uses the CPU RNG, dropout would use the device RNG). +- The BPE tokenizer path through training/evaluation, LoRA integration into the model, and the published INT8 numbers were not re-measured in this pass. **Update:** the INT8 losses and stored bytes were re-verified exactly in October (`experiments/records/legacy/`); BPE and LoRA remain unverified. +- CUDA code paths (`synchronize`, `max_memory_allocated`) are untested: no CUDA hardware. +- Benchmark timings assume a quiet machine. Thermal state and background load were not controlled beyond warmups, rotated backend order and repeat statistics. + +## Independent release audit · resolution (October 2026) + +An independent audit of `c256f20` raised four medium-severity findings. Each was +reproduced before fixing; regression tests cover every reproduction. + +| Finding | Reproduction on `c256f20` | Resolution | +| --- | --- | --- | +| 1. Checkpoint file hashes are not reproducible | v1 seed 42 retrained in a fresh clone: identical tensors, config and losses; different file SHA-256 (elapsed/train/eval seconds in metadata) | Deterministic model identity `minilm-model-identity/v1` (`src/minilm_lab/identity.py`), stored in new checkpoints and schema-v2 records; file SHA-256 kept as artifact identity; 24 historical checkpoints attested after verifying their file hashes. The regenerated seed-42 file has a different hash and the same identity | +| 2. Legacy `auto` and `mha` could double-count a seed | an `mha` re-record of seed 42 made the v1 MHA aggregate use seeds [42, 42, 43, 44] | Experiment identity (protocol, canonical backend, seed, config digest, dataset digest) computed after canonicalization; replicates with identical outcomes count once; conflicting identities fail validation and reporting unless explicitly superseded | +| 3. Evaluation provenance was not validated | zeroed checkpoint hashes, rows swapped across seeds, a changed protocol and duplicate records all passed validation; the report joined on seed/backend | Evaluation, quality and legacy rows must resolve to exactly one experiment identity by checkpoint SHA-256 with matching seed, backend, protocol, selected step, corpus hash and model identity; splits must match files and protocol manifests; training commit and manifest references must exist. The report builder uses the same join and fails closed | +| 4. KV-cache records lacked raw samples | records kept only summaries (`count: 5`) | Raw per-repeat, per-step seconds persisted; summaries derived from them and re-checked by the validator; three historical KV records superseded (not edited) by same-settings re-measurements | + +Additional corrections: the float64 oracle claim is qualified (RMSNorm variance and +RoPE angles run in float32); CUDA peak memory now uses the device allocator, never host +RSS (`provenance.measure_peak_memory`); stale INT8 and corpus-regeneration statements are +annotated above; validator failures propagate as exit status 1, including through +`pipefail` pipelines. Rules: [RECORD_SCHEMA.md](RECORD_SCHEMA.md). + +## Addendum · SDPA training and corpus v2 (October 2026) + +| Finding | Evidence | Resolution | +| --- | --- | --- | +| `reference_causal_attention` downcast float64 scores to float32 (`.float()`) | `torch.autograd.gradcheck` failed for the reference backend only | Promote to at least float32, never below the input dtype; float64 reference now equals SDPA to 1e-12. The quality record that used the float64 "oracle" is superseded; re-running with the corrected oracle changed fp32 logit errors by < 1e-6 (e.g. 3.65e-5 → 3.60e-5) and no conclusion | +| SDPA and reference paths skipped attention dropout in training | code inspection | Apply `nn.MultiheadAttention`'s dropout probability in training mode; inert at the published `dropout=0` | +| Upstream Darwin file changed (`ededa9c0…` → `362142cb…`) | fresh download | The extracted v1 split regenerates byte-identically from the new file; only the Gutenberg header/licence changed. `prepare_science_corpus.py` (v1) still pins the old raw hash and therefore refuses to run; it is left unchanged as a legacy script. v2 pins the hash of the extracted body instead **Update:** the v1 script now verifies regenerated split hashes and runs again. | +| One training record was written while unrelated documentation edits were uncommitted | record `git.dirty = true` | Re-recorded from a clean tree; the dirty record is kept and marked superseded | + +**Release note.** From the first release the default backend is `sdpa`. The path called +"default" or `auto` in the statements above is now the explicit backend `mha`; `auto` is +accepted as a legacy alias and checkpoints without a backend field are treated as `mha`. + +Verified after the changes: + +- Default-backend (`auto`) training on v1 is bitwise identical to the original runs for seeds 42, 43 and 44. +- Per-backend gradients agree with a float64 oracle to a relative 1e-4 for learned and RoPE models; `gradcheck` passes for the attention layer under every backend. (The oracle's RMSNorm variance and RoPE angles run in float32.) +- Training resumes refuse a backend change; checkpoints record the backend. +- v2 corpus: body hashes verified on a fresh download; zero exact and near duplicates across train/validation/test; legacy v1 files still match their original manifest (all enforced by `tests/test_corpus_data.py`). + # Implementation audit · September 2026 ## Corrected defects diff --git a/docs/research_report.md b/docs/research_report.md index bc5cf47..2d3d41e 100644 --- a/docs/research_report.md +++ b/docs/research_report.md @@ -1,24 +1,115 @@ # MiniLM Lab: science prose systems study **Author:** Anvit Devadiga -**Run:** one seed, 1,000 optimizer updates, CPU -**Status:** reproducible baseline, not a broad-language benchmark +**Status:** reproducible small-model study, not a broad-language benchmark +**Protocols:** science-v1 (legacy, September 2026) and science-v2 (October 2026). Perplexities are not comparable across protocols. + +## October 2026 update + +All numbers below come from JSON records in `experiments/records/`; the full +generated tables, method and limitations are in [EXPERIMENT_RESULTS.md](EXPERIMENT_RESULTS.md), +and the evidence for every correctness claim is in the [implementation audit](implementation_audit.md). +Hardware: Apple M4 (16 GB), CPU and MPS, torch 2.14.1. CUDA was not available. + +### Follow-ups on science-v1 (corpus unchanged) + +- **Reproduction.** The published seed-42 run reproduces bit-for-bit. Seeds 42/43/44 + give held-out perplexity 17.58 / 18.99 / 21.07 (19.21 ± 1.75); the published value + is the best of three. Every seed sits near 3.27 nats until roughly step 750. +- **Attention implementations.** The legacy `nn.MultiheadAttention` path (`mha`), PyTorch + SDPA and an explicit reference compute the same function (identical fp32 perplexity, + 100% top-1 agreement with a float64 reference run whose RMSNorm variance and RoPE + angles are still computed in float32; gradients verified with `gradcheck`). + At inference SDPA is up to 3.9× (CPU) and 5.4× (MPS) faster than the reference at + 2,048 tokens and needs 8–22× less memory. +- **Matched training, legacy MHA vs SDPA (3 seeds).** Validation perplexity + 19.21 ± 1.75 vs 19.04 ± 1.37 with no consistent paired direction. A controlled + training-step benchmark shows SDPA 5–28% faster on CPU (L=128–1,024) and within + ~5% of the other backends on MPS. +- **KV cache.** 2.2–23× faster than recomputation on CPU, but slower than + recomputation on MPS for batch-1 prompts under 128 tokens, and no faster once + generation passes the context window (the cache is rebuilt every step). +- The September CPU-only cache speedup (3.58×) and INT8 numbers below are retained + unchanged. In October both were re-run with the legacy `mha` backend + (`experiments/records/legacy/`): the INT8 validation losses and stored byte counts + reproduce exactly, while timings differ (cache speedup 3.40× vs 3.58×; INT8 32-pass + latency 55.3/65.7 ms vs 55.6/65.9 ms), so the timings are measurements, not constants. +- **Corpus regeneration.** Project Gutenberg changed the Darwin file's header after + September; `prepare_science_corpus.py` now verifies the regenerated v1 splits against + the published hashes (byte-identical as of October 22) instead of the raw-file hash. + +### Protocol science-v2 + +**Data.** The same two books, cleaned before splitting by deterministic rules +(exact start/end heading markers per book; headings, fully bracketed editorial +paragraphs and multi-line numeric lists/tables removed; displayed equations kept), +plus a third book by a different author, Faraday's *The Chemical History of a Candle* +(Gutenberg 14474), held out entirely as a test set. Extracted book bodies are pinned by +SHA-256 in `scripts/prepare_science_corpus_v2.py`; split hashes, dropped-paragraph counts +and contamination checks are in [the v2 manifest](../data/science-v2/manifest.json). +No validation or test paragraph is an exact or ≥ 50% word-8-gram duplicate of training +(or validation) text. Train 969,506 bytes; validation 105,666; test 211,849. + +**Protocol.** v1 hyperparameters unchanged (nothing tuned on v2); best checkpoint +chosen on validation; test scored once per selected checkpoint after all training. + +| Backend | Seeds | Validation perplexity | Test perplexity (Faraday) | +| --- | ---: | ---: | ---: | +| Legacy MHA | 42, 43, 44 | 15.41 ± 2.70 | 15.66 ± 2.58 | +| SDPA | 42, 43, 44 | 15.37 ± 2.73 | 15.60 ± 2.63 | + +Test perplexity is 0.1–0.4 above validation in every run: the model transfers to an +unseen author and topic at close to its in-domain level, at this (low) level of +quality. Seeds, not backends, dominate the spread. **These numbers must not be +compared with the v1 perplexities above or below**: v2 removes non-prose text and +uses a different validation set. + +The Faraday test set was evaluated once, after all training, and is now published: +it must not guide further tuning under the same untouched-test claim, and one book +does not establish broad cross-author generalization. + +From the first release SDPA is the default backend; every v1 reproduction command +passes `--attention-backend mha` (recorded as `auto` before the release). + +### Reproduce (October 2026) + +```bash +PYTHON=python ./experiments/sdpa_v1.sh # science-v1 matched training, 3 seeds x 2 backends +PYTHON=python ./experiments/v2_protocol.sh # build/verify science-v2, train, select, test once +python scripts/benchmark_attention.py --device cpu # and --device mps [--dtype float16] +python scripts/benchmark_kv_scaling.py --device cpu # and --device mps [--dtype float16] +python scripts/benchmark_training_step.py --device cpu # and --device mps +python scripts/build_experiment_report.py +``` + +### Remaining limitations + +Under-trained models (1,000 steps) whose quality depends strongly on when each seed +leaves its early plateau; three seeds; one laptop; a single test book; no CUDA. +The PDF paper is generated from the September science-v1 record only and does not +include this update. + +## Legacy baseline · science-v1 (September 2026, single seed) + +The section below is the original report, unchanged. Its "next experiments" list is +partly addressed above (multiple seeds, unseen-book test, M4 MPS measurements); the +RoPE ablation remains open. -## Question +### Question Can a compact decoder-only model learn held-out scientific prose while exposing measurable inference and storage trade-offs on laptop-class hardware? -## Data and split +### Data and split Two Project Gutenberg public-domain editions are used: [Darwin, *On the Origin of Species*](https://www.gutenberg.org/ebooks/1228) and [Einstein, *Relativity: The Special and General Theory*](https://www.gutenberg.org/ebooks/30155). `prepare_science_corpus.py` removes the Gutenberg boilerplate, takes approximately the first 90% of each book's bytes at a paragraph boundary for training, and holds out the remainder. The exact downloaded and prepared SHA-256 hashes are in [the manifest](../data/science/manifest.json). The prepared corpus contains 1,008,977 training bytes and 112,112 validation bytes. This is a within-document holdout. It tests continuation on unseen passages from the same books, not transfer to unseen authors, topics or factual question answering. The corpus is modest and may contain repeated phrases across splits. -## Model and protocol +### Model and protocol Four decoder blocks, width 192, six attention heads, context 128, byte vocabulary 256, learned positions, RMSNorm, SwiGLU MLP, tied embeddings. AdamW with learning rate 3e-4, microbatch size eight, four microbatches per optimizer step, seed 42. Checkpoints are selected by deterministic full-split validation at steps 250, 500, 750 and 1,000. The evaluator scores every next token once in nonoverlapping 128-token chunks, with the first token of each chunk as context. This is a fixed-window metric, not a rolling-context metric. -## Results +### Results | Measure | Value | | --- | ---: | @@ -40,7 +131,7 @@ Darwin contributes most of the aggregate held-out bytes. Each per-book evaluatio The decode benchmark uses a fixed 21-token prompt, one warm-up and seven timed runs per mode, the same sampling seed, temperature, and no top-k filtering. It ran on CPU. The [raw record](data/science_run.json) includes every timing sample and training checkpoint. The [learning](figures/learning.svg), [speed](figures/speed.svg), and [compression](figures/tradeoff.svg) figures are generated from that record. -### Portable INT8 reference +#### Portable INT8 reference The old dynamic INT8 path failed on this Mac with `NoQEngine`. A portable per-output-channel INT8 reference now quantizes MLP linear weights and dequantizes during each forward pass. Attention projections, embeddings and the tied LM head remain FP32. @@ -52,7 +143,7 @@ The old dynamic INT8 path failed on this Mac with `NoQEngine`. A portable per-ou The format saves 53.6% of stored tensor bytes in memory. It is slower because dequantization happens at runtime. Actual checkpoint file size, peak memory, and accelerated INT8 kernels were not measured. -## Threats to validity +### Threats to validity - Only one seed and one model shape were run. Do not treat the difference between methods as statistically established. - The science corpus is 1 MB. Perplexity 17.58 does not imply scientific understanding or reliable factual responses. @@ -60,6 +151,6 @@ The format saves 53.6% of stored tensor bytes in memory. It is slower because de - The training curves have four validation points. They show checkpoints, not a continuous trajectory. - The BPE tokenizer and LoRA wrapper remain educational components without a matched end-to-end comparison in this study. -## Reproduce and extend +### Reproduce and extend Run the commands in [README](../README.md). `scripts/publish_science_results.py` checks the local artifacts and writes the tracked JSON record. `scripts/build_report_assets.py` renders the repository figures from that record. The next useful experiments are matched-seed RoPE and learned-position ablations, multi-seed runs, a genuinely unseen-book test, and an M4 MPS benchmark. diff --git a/experiments/baselines/model_identity_baseline.json b/experiments/baselines/model_identity_baseline.json new file mode 100644 index 0000000..d0ac672 --- /dev/null +++ b/experiments/baselines/model_identity_baseline.json @@ -0,0 +1,126 @@ +{ + "entries": { + "02c866b578364f3fcbddb8b7e0f53d5088c0f540ab5ff006b4f821579544c599": { + "kind": "final", + "model_identity_sha256": "cc7a5dd2af26be2c8c5bd1a550820a4fe44b01684895202a272a75581a885bd6", + "training_record": "training/20261022T040834Z_training_science-v1_sdpa_seed43_48e95f8.json" + }, + "0a1357240562d1851b8db80c85e3ce4b4bfd0b00a4c6d9c41ddead928d90c916": { + "kind": "best", + "model_identity_sha256": "cc7a5dd2af26be2c8c5bd1a550820a4fe44b01684895202a272a75581a885bd6", + "training_record": "training/20261022T040834Z_training_science-v1_sdpa_seed43_48e95f8.json" + }, + "1cb3fbca57f10b72d33c945325dfc9ed1335cca02fc2fe07150fae5ba40a4b8e": { + "kind": "best", + "model_identity_sha256": "7b25516b2cf08abfd68d2a998e1b91963b1b5c0290a09727a14744e6fd7840af", + "training_record": "training/20261022T042403Z_training_science-v2_auto_seed42_897db37.json" + }, + "23f0aa813704e0cfc02d0780386fe76d1ee26f70aaa674ce83a849a85fb93546": { + "kind": "best", + "model_identity_sha256": "c0c6f2995c5b97d331dc976f01a9af2f6cbea0bed3de442bf0d43986242b1425", + "training_record": "training/20261022T041654Z_training_science-v1_auto_seed42_48e95f8.json" + }, + "2488847e2e18787df5cbd5dd3000f2405527961878cb1c4a0ee510b7f561c038": { + "kind": "best", + "model_identity_sha256": "95142ee6b38e36487d76a0d3bf7b021d9a0800efa956b70db9da79b85f9ceab0", + "training_record": "training/20261022T043559Z_training_science-v2_sdpa_seed44_897db37.json" + }, + "264b42c669549c77acc92d05c122beedd9ec87b942a077b073fd7758fb3abef9": { + "kind": "final", + "model_identity_sha256": "cecb5f2ebbcae98441896f1da50d5306ee8e49ef063a2a3217b22e089a935625", + "training_record": "training/20261022T042630Z_training_science-v2_sdpa_seed42_897db37.json" + }, + "3451e8577c0a5cd8f5b5cadd7aec3f7fd2d9b72fde24db01e6520f9b083de4a4": { + "kind": "best", + "model_identity_sha256": "d25ea987563add6fa1a8f53f032a82f4c3d5893c233b6d9cc2802fd1cefae0cc", + "training_record": "training/20261022T043119Z_training_science-v2_sdpa_seed43_897db37.json" + }, + "46117ebc02af5b2fe2a5cf6b6301435955f8a865aeef7f30a992e5c33694543c": { + "kind": "final", + "model_identity_sha256": "242d813f6a59b1b50abeddc211983de28e0fc76ea0dc323ca611a9f9acec4578", + "training_record": "training/20261022T042859Z_training_science-v2_auto_seed43_897db37.json" + }, + "4c52fc4c8b32fae5ca82295eb33f4c9331ad699b3b87d3c6d533ff043412b242": { + "kind": "final", + "model_identity_sha256": "292224766ca70f7501a2ad3b1dbd7b593a7f32ff025ec8ead35c81914bdde2e9", + "training_record": "training/20261022T040614Z_training_science-v1_sdpa_seed42_48e95f8.json" + }, + "59dd70bd83b42f3f4075d7f44fbbe162a908c10d8f07c0a831cc13d080c53340": { + "kind": "final", + "model_identity_sha256": "d25ea987563add6fa1a8f53f032a82f4c3d5893c233b6d9cc2802fd1cefae0cc", + "training_record": "training/20261022T043119Z_training_science-v2_sdpa_seed43_897db37.json" + }, + "5de158ff5c205dcf1e06ecce60ad189878d8f0899a7c81e9b4b19145e7628244": { + "kind": "final", + "model_identity_sha256": "22694540faf7514d97d05b0408f1d7a659d9380bae3822137aa814935a82afbd", + "training_record": "training/20261022T041553Z_training_science-v1_sdpa_seed44_48e95f8.json" + }, + "5f25f43bcdd4a01e7bab23978fa15fdf85e650008b93d8a39b7960437885c687": { + "kind": "final", + "model_identity_sha256": "95142ee6b38e36487d76a0d3bf7b021d9a0800efa956b70db9da79b85f9ceab0", + "training_record": "training/20261022T043559Z_training_science-v2_sdpa_seed44_897db37.json" + }, + "6e8b0471016093200fa1a7045e41b4dcb1be07bb0bb742444bd2c19eaa9f2cc2": { + "kind": "best", + "model_identity_sha256": "cecb5f2ebbcae98441896f1da50d5306ee8e49ef063a2a3217b22e089a935625", + "training_record": "training/20261022T042630Z_training_science-v2_sdpa_seed42_897db37.json" + }, + "ad66af927334d8991901682c91a0c1ca539707681a0ba890d7145101ba3c041c": { + "kind": "best", + "model_identity_sha256": "242d813f6a59b1b50abeddc211983de28e0fc76ea0dc323ca611a9f9acec4578", + "training_record": "training/20261022T042859Z_training_science-v2_auto_seed43_897db37.json" + }, + "b1af615e4dc2908a234f45c3c8c48c481f7c2b15294e76abd588f17da9b8763d": { + "kind": "final", + "model_identity_sha256": "c0c6f2995c5b97d331dc976f01a9af2f6cbea0bed3de442bf0d43986242b1425", + "training_record": "training/20261022T041654Z_training_science-v1_auto_seed42_48e95f8.json" + }, + "becaa1b609e62d9591a2cce4cae60e0d158c501db10a7e2edcc5209a1a4ed5ef": { + "kind": "final", + "model_identity_sha256": "f2546d9a7967994e554618d99394bc4bdd5d718a3d4ec7cee8c99686f59314e8", + "training_record": "training/20261022T041335Z_training_science-v1_auto_seed44_48e95f8.json" + }, + "c9a217ad9a778171738c115cd23a3b271957fb570f7990e616d5444e7baed976": { + "kind": "final", + "model_identity_sha256": "c609dd197aa7bfa285b3a5fe16eb834cbeab4369e5ff512d5ea27ea6c93ff7d6", + "training_record": "training/20261022T041103Z_training_science-v1_auto_seed43_48e95f8.json" + }, + "cb77533a909fb6f40f08c3a50854a28765ab214aa87a25a8bb8bc5566211cf40": { + "kind": "best", + "model_identity_sha256": "6e32612238ffcfd87920db587df899a4a021e5206f50e55dc5f74041a8961062", + "training_record": "training/20261022T043349Z_training_science-v2_auto_seed44_897db37.json" + }, + "d4fe3800f75cd1f3b7ef68bfa463dc48f503585e5cd038da8f22b7f67014f794": { + "kind": "best", + "model_identity_sha256": "f2546d9a7967994e554618d99394bc4bdd5d718a3d4ec7cee8c99686f59314e8", + "training_record": "training/20261022T041335Z_training_science-v1_auto_seed44_48e95f8.json" + }, + "de734545979c6a4b9a5d6a21bb0299c8c91ac69c32effbfcaca23a86b7cd1fa0": { + "kind": "best", + "model_identity_sha256": "292224766ca70f7501a2ad3b1dbd7b593a7f32ff025ec8ead35c81914bdde2e9", + "training_record": "training/20261022T040614Z_training_science-v1_sdpa_seed42_48e95f8.json" + }, + "e2285dfc79e7aa1d430cac7720bc5997139210fef876055c23da375aab24b04c": { + "kind": "best", + "model_identity_sha256": "c609dd197aa7bfa285b3a5fe16eb834cbeab4369e5ff512d5ea27ea6c93ff7d6", + "training_record": "training/20261022T041103Z_training_science-v1_auto_seed43_48e95f8.json" + }, + "e7544665b394924f64facd47c731cd2950caa35d0ced5aaa66c77cdeb4034e47": { + "kind": "best", + "model_identity_sha256": "22694540faf7514d97d05b0408f1d7a659d9380bae3822137aa814935a82afbd", + "training_record": "training/20261022T041553Z_training_science-v1_sdpa_seed44_48e95f8.json" + }, + "eeef0fd313c7d9d023674a38ac94c05423bd02c5490937cf05d3bdabadb1b219": { + "kind": "final", + "model_identity_sha256": "6e32612238ffcfd87920db587df899a4a021e5206f50e55dc5f74041a8961062", + "training_record": "training/20261022T043349Z_training_science-v2_auto_seed44_897db37.json" + }, + "f4e18218a9da9670cd647c4baf264ddada9ca50fdb86234721e59292ecaae880": { + "kind": "final", + "model_identity_sha256": "7b25516b2cf08abfd68d2a998e1b91963b1b5c0290a09727a14744e6fd7840af", + "training_record": "training/20261022T042403Z_training_science-v2_auto_seed42_897db37.json" + } + }, + "method": "checkpoint bytes hashed and required to equal the training record's SHA-256; model identity recomputed from the checkpoint tensors and configuration", + "scheme": "minilm-model-identity/v1" +} diff --git a/experiments/records/SUPERSEDED.json b/experiments/records/SUPERSEDED.json new file mode 100644 index 0000000..eaedfd5 --- /dev/null +++ b/experiments/records/SUPERSEDED.json @@ -0,0 +1,61 @@ +{ + "policy": "Records are never deleted or edited. A superseded record is kept for history, is not used by docs/EXPERIMENT_RESULTS.md, and names its replacement.", + "records": { + "attention/20261022T033255Z_attention_mps_float32_615767d.json": { + "reason": "MPS memory measured as driver pool size, which moves in ~1 GiB heaps; timing values were valid but the record is replaced as a whole", + "superseded_by": "attention/20261022T034036Z_attention_mps_float32_0502118.json" + }, + "attention/20261022T033548Z_attention_mps_float16_615767d.json": { + "reason": "MPS memory measured as driver pool size, which moves in ~1 GiB heaps; timing values were valid but the record is replaced as a whole", + "superseded_by": "attention/20261022T034304Z_attention_mps_float16_0502118.json" + }, + "attention_quality/20261022T033711Z_attention_quality_615767d.json": { + "reason": "written with an uncommitted fix to compare_attention_quality.py (git.dirty = true)", + "superseded_by": "attention_quality/20261022T041752Z_attention_quality_48e95f8.json" + }, + "attention_quality/20261022T034349Z_attention_quality_0502118.json": { + "reason": "the float64 oracle ran its softmax in float32 (reference_causal_attention downcast); rerun changed fp32 logit errors by < 1e-6 and no conclusion", + "superseded_by": "attention_quality/20261022T041752Z_attention_quality_48e95f8.json" + }, + "training/20261022T034420Z_training_seed42_0502118.json": { + "reason": "recorded while unrelated documentation edits were uncommitted (git.dirty = true); training ran at commit 1fb8f37", + "superseded_by": "training/20261022T041654Z_training_science-v1_auto_seed42_48e95f8.json" + }, + "training/20261022T034421Z_training_seed43_0502118.json": { + "reason": "recorded with git.dirty = true; training used 1fb8f37 code plus model.py edits later committed as 274722f. The clean replacement's checkpoint is bitwise identical", + "superseded_by": "training/20261022T041103Z_training_science-v1_auto_seed43_48e95f8.json" + }, + "training/20261022T034421Z_training_seed44_0502118.json": { + "reason": "recorded with git.dirty = true; training used 1fb8f37 code plus model.py edits later committed as 274722f. The clean replacement's checkpoint is bitwise identical", + "superseded_by": "training/20261022T041335Z_training_science-v1_auto_seed44_48e95f8.json" + }, + "training/20261022T040350Z_training_science-v1_auto_seed42_48e95f8.json": { + "reason": "recorded while unrelated documentation edits were uncommitted (git.dirty = true)", + "superseded_by": "training/20261022T041654Z_training_science-v1_auto_seed42_48e95f8.json" + }, + "legacy/20261022T045414Z_legacy_kv_cache_cpu_13530f4.json": { + "reason": "recorded while README/doc edits were uncommitted (git.dirty = true); code was committed", + "superseded_by": "legacy/20261022T045428Z_legacy_kv_cache_cpu_13530f4.json" + }, + "legacy/20261022T045419Z_legacy_int8_cpu_13530f4.json": { + "reason": "recorded while README/doc edits were uncommitted (git.dirty = true); code was committed", + "superseded_by": "legacy/20261022T045433Z_legacy_int8_cpu_13530f4.json" + }, + "kv_cache/20261022T035538Z_kv_cpu_float32_93507c4.json": { + "reason": "historical measurement without raw per-repeat timing samples (summaries only; audit finding 4). Its values are kept unchanged but cannot be re-derived; replaced by a re-measurement with the same settings that stores raw samples", + "superseded_by": "kv_cache/20261022T053124Z_kv_cpu_float32_57be26d.json" + }, + "kv_cache/20261022T035808Z_kv_mps_float32_93507c4.json": { + "reason": "historical measurement without raw per-repeat timing samples (summaries only; audit finding 4). Its values are kept unchanged but cannot be re-derived; replaced by a re-measurement with the same settings that stores raw samples", + "superseded_by": "kv_cache/20261022T053346Z_kv_mps_float32_57be26d.json" + }, + "kv_cache/20261022T040019Z_kv_mps_float16_93507c4.json": { + "reason": "historical measurement without raw per-repeat timing samples (summaries only; audit finding 4). Its values are kept unchanged but cannot be re-derived; replaced by a re-measurement with the same settings that stores raw samples", + "superseded_by": "kv_cache/20261022T053547Z_kv_mps_float16_57be26d.json" + }, + "identity/20261022T053900Z_model_identity_attestation_935a229.json": { + "reason": "recorded while a regenerated report was uncommitted (git.dirty = true); identical attestation re-recorded from a clean tree", + "superseded_by": "identity/20261022T053914Z_model_identity_attestation_935a229.json" + } + } +} diff --git a/experiments/records/attention/20261022T033005Z_attention_cpu_float32_615767d.json b/experiments/records/attention/20261022T033005Z_attention_cpu_float32_615767d.json new file mode 100644 index 0000000..e6e0b94 --- /dev/null +++ b/experiments/records/attention/20261022T033005Z_attention_cpu_float32_615767d.json @@ -0,0 +1,14061 @@ +{ + "record_version": 1, + "created_utc": "20261022T033005Z", + "experiment": "attention_backends", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "cpu", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "615767d83416226055484fb1f7336bdf4a40eaa0", + "branch": "experiments/attention-kv-audit", + "dirty": false + }, + "command": "python scripts/benchmark_attention.py --device cpu --dtype float32" + }, + "settings": { + "dtype": "float32", + "position": "learned", + "backends": [ + "auto", + "sdpa", + "reference" + ], + "seq_lens": [ + 128, + 256, + 512, + 1024, + 2048 + ], + "batch_sizes": [ + 1, + 4, + 8 + ], + "decode_tokens": 32, + "warmup": 3, + "repeats": 10, + "seed": 0, + "weights": "seeded random initialization (timing does not depend on trained values)", + "model_config": { + "vocab_size": 256, + "context_length": 2080, + "embedding_dim": 192, + "num_layers": 4, + "num_heads": 6, + "dropout": 0.0, + "positional_encoding": "learned", + "norm": "rmsnorm", + "activation": "swiglu", + "rope_theta": 10000.0 + }, + "inference_mode": true + }, + "results": [ + { + "backend": "auto", + "batch": 1, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.0018277079943800345 + }, + { + "forward": 0.0018588750099297613 + }, + { + "forward": 0.0018565829959698021 + }, + { + "forward": 0.0018587079975986853 + }, + { + "forward": 0.0018698749918257818 + }, + { + "forward": 0.0017938340024556965 + }, + { + "forward": 0.002208125006291084 + }, + { + "forward": 0.0018717499915510416 + }, + { + "forward": 0.001820208999561146 + }, + { + "forward": 0.0017870000010589138 + } + ], + "forward_ms": { + "count": 10, + "median": 1.8576454967842437, + "p95": 2.0567562496580645, + "min": 1.7870000010589138, + "max": 2.208125006291084, + "iqr": 0.04504124808590859, + "mean": 1.8752666990621947, + "cv": 0.06444441723358571 + }, + "memory": { + "peak_extra_bytes": 4325376, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "sdpa", + "batch": 1, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.00141291601175908, + "prefill": 0.0014619580033468083, + "decode_steps": [ + 0.00040795799577608705, + 0.0003597080067265779, + 0.0003590830019675195, + 0.00034925001091323793, + 0.0003468750073807314, + 0.00037229200825095177, + 0.0003741670079762116, + 0.00037670900928787887, + 0.0003776250086957589, + 0.0003632500011008233, + 0.0003696670028148219, + 0.00036570901283994317, + 0.0003518330049701035, + 0.00036366700078360736, + 0.0003595000016503036, + 0.00035258400021120906, + 0.0003587910032365471, + 0.00035254200338386, + 0.0003547919914126396, + 0.0003571249981177971, + 0.0003590829874156043, + 0.0003627499972935766, + 0.00036016599915456027, + 0.0003612909931689501, + 0.00036370899761095643, + 0.00035945899435319006, + 0.000356332995579578, + 0.0003570420085452497, + 0.00036741700023412704, + 0.00036041699058841914, + 0.0003592080029193312, + 0.0003643750096671283 + ] + }, + { + "forward": 0.0014795830065850168, + "prefill": 0.001424624992068857, + "decode_steps": [ + 0.00039362499956041574, + 0.0003831670037470758, + 0.00035595799272414297, + 0.0003630410064943135, + 0.00035758299054577947, + 0.0003595000016503036, + 0.00036129201180301607, + 0.00035262499295640737, + 0.000385499995900318, + 0.00042495899833738804, + 0.0005548330082092434, + 0.0004534579929895699, + 0.0003831249923678115, + 0.0003670840087579563, + 0.0003967500088037923, + 0.0003697500069392845, + 0.00036025000736117363, + 0.0003600839991122484, + 0.0003550830006133765, + 0.0003503749903757125, + 0.0003578750038286671, + 0.0003574169968487695, + 0.0003635000030044466, + 0.00041095800406765193, + 0.00037150000571273267, + 0.0003815829986706376, + 0.00036833299964200705, + 0.0003572919958969578, + 0.00036025000736117363, + 0.0003728339943336323, + 0.00036891600757371634, + 0.0003614579909481108 + ] + }, + { + "forward": 0.001421584005584009, + "prefill": 0.0014378750056494027, + "decode_steps": [ + 0.0004473329900065437, + 0.00041074999899137765, + 0.00037812499795109034, + 0.0003667919954750687, + 0.00037275000067893416, + 0.0003678339999169111, + 0.0003511660033836961, + 0.00035545899299904704, + 0.00036175000423099846, + 0.00035512499744072556, + 0.0003667090059025213, + 0.0003564159997040406, + 0.00035849999403581023, + 0.0003567500098142773, + 0.0003587499959394336, + 0.00036670800182037055, + 0.0003615000023273751, + 0.0003553749993443489, + 0.0003596660098992288, + 0.00036012500640936196, + 0.0003660829970613122, + 0.0003571660054149106, + 0.00035841700446326286, + 0.0003648749989224598, + 0.00035016699985135347, + 0.0003622499934863299, + 0.00035695800033863634, + 0.0003529169916873798, + 0.00036520800495054573, + 0.00036204200296197087, + 0.0003708750009536743, + 0.0003606669924920425 + ] + }, + { + "forward": 0.0014207499916665256, + "prefill": 0.0014393329911399633, + "decode_steps": [ + 0.0004353339900262654, + 0.0003757090016733855, + 0.0003693329927045852, + 0.0003647090052254498, + 0.00037162500666454434, + 0.00036108300264459103, + 0.0003541249898262322, + 0.0003480830055195838, + 0.00034449998929630965, + 0.00035358298919163644, + 0.000368666005670093, + 0.00035141699481755495, + 0.00035258301068097353, + 0.0003552919952198863, + 0.0003564169892342761, + 0.00036800000816583633, + 0.00036758300848305225, + 0.0003549999964889139, + 0.00035924999974668026, + 0.0003646669938461855, + 0.00035195901000406593, + 0.00036633299896493554, + 0.0003549999964889139, + 0.0003538750024745241, + 0.0003566659870557487, + 0.00036083400482311845, + 0.0003575830050976947, + 0.0003642920055426657, + 0.0003517500008456409, + 0.0003633330052252859, + 0.0003627909900387749, + 0.00035845799720846117 + ] + }, + { + "forward": 0.0014172919909469783, + "prefill": 0.0016041250055423006, + "decode_steps": [ + 0.0005328340048436075, + 0.0003969160025008023, + 0.0003669159923447296, + 0.0003541670012054965, + 0.00036950000503566116, + 0.0003613330045482144, + 0.00035533300251699984, + 0.0003512499970383942, + 0.00035204099549446255, + 0.00035087500873487443, + 0.0005025419959565625, + 0.00047975000052247196, + 0.0003794170042965561, + 0.00036837501102127135, + 0.00035695800033863634, + 0.00035599998955149204, + 0.000364125007763505, + 0.00035158298851456493, + 0.00038041699735913426, + 0.00037362499278970063, + 0.0003541250043781474, + 0.0003540420002536848, + 0.0003583329962566495, + 0.00035479200596455485, + 0.000355791999027133, + 0.000352167000528425, + 0.0003591249987948686, + 0.000355416996171698, + 0.00037554200389422476, + 0.00035820800985675305, + 0.0003607499966165051, + 0.00035645799653138965 + ] + }, + { + "forward": 0.0015122920012800023, + "prefill": 0.0014380829961737618, + "decode_steps": [ + 0.0003975829895352945, + 0.00036008299503009766, + 0.0003514160052873194, + 0.00035000000207219273, + 0.00035204099549446255, + 0.00035720798769034445, + 0.0003512090042931959, + 0.00037320799310691655, + 0.0003775000077439472, + 0.0003533749986672774, + 0.0003615000023273751, + 0.0003599170013330877, + 0.0003557500021997839, + 0.00035600000410340726, + 0.0003563750069588423, + 0.0003510000096866861, + 0.0003558750031515956, + 0.00035887499689124525, + 0.0003580830089049414, + 0.0003547919914126396, + 0.00036016599915456027, + 0.0003599170013330877, + 0.00036433299828786403, + 0.000363041996024549, + 0.0003647499979706481, + 0.00035841698991134763, + 0.00037104199873283505, + 0.0003734170022653416, + 0.00036108399217482656, + 0.00036387500585988164, + 0.0003580000047804788, + 0.0003655829932540655 + ] + }, + { + "forward": 0.003077417000895366, + "prefill": 0.0022789170034229755, + "decode_steps": [ + 0.0006550000107381493, + 0.0005320829950505868, + 0.0004620830004569143, + 0.00043937499867752194, + 0.0004513329913606867, + 0.00043400000140536577, + 0.00045937499089632183, + 0.000444875011453405, + 0.00045249999675434083, + 0.0004461250064196065, + 0.00045941700227558613, + 0.0004342920001363382, + 0.000464208991616033, + 0.0006477090064436197, + 0.0004760409938171506, + 0.000528458011103794, + 0.00042424999992363155, + 0.00044350000098347664, + 0.00045033400238025934, + 0.00043941701005678624, + 0.0004314580000936985, + 0.00042062500142492354, + 0.00043737499800045043, + 0.000428542000008747, + 0.0003922500036424026, + 0.00041625001176726073, + 0.0004321250016801059, + 0.00043662500684149563, + 0.0004183749988442287, + 0.00041787500958889723, + 0.00044629200419876724, + 0.0004376249999040738 + ] + }, + { + "forward": 0.001571208005771041, + "prefill": 0.0014236669958336279, + "decode_steps": [ + 0.0004116670024814084, + 0.00036029198963660747, + 0.0003582500066841021, + 0.00036550000368151814, + 0.00035945899435319006, + 0.0003581670025596395, + 0.0003475419944152236, + 0.0003482920001260936, + 0.0003529999958118424, + 0.0003524170024320483, + 0.0003517079894663766, + 0.0003515410062391311, + 0.0003442499873926863, + 0.0003588330000638962, + 0.00035283299803268164, + 0.0003468329960014671, + 0.00035841700446326286, + 0.000356542004738003, + 0.000352917006239295, + 0.00035341699549462646, + 0.0003526250075083226, + 0.00035433399898465723, + 0.0003577089955797419, + 0.00034954099101014435, + 0.00036445900332182646, + 0.00036245900264475495, + 0.0003544999926816672, + 0.0003725830028997734, + 0.000353833005647175, + 0.000361416008672677, + 0.00035945800482295454, + 0.0003552089910954237 + ] + }, + { + "forward": 0.001415124992490746, + "prefill": 0.0014184579922584817, + "decode_steps": [ + 0.00039645799552090466, + 0.0003595830057747662, + 0.0003500840102788061, + 0.0003525000065565109, + 0.0003503339976305142, + 0.0003517500008456409, + 0.0003546670050127432, + 0.00035304199263919145, + 0.00036033301148563623, + 0.00038312500691972673, + 0.0003505420027067885, + 0.00035095900238957256, + 0.0003545420040609315, + 0.0003507090004859492, + 0.0003465000045252964, + 0.000346583008649759, + 0.0003531660040607676, + 0.0003499169979477301, + 0.00035591699997894466, + 0.00035487501008901745, + 0.00035120800021104515, + 0.00034649998997338116, + 0.0003504169872030616, + 0.0003479579900158569, + 0.00035600000410340726, + 0.00035741600731853396, + 0.0003537500015227124, + 0.00034737499663606286, + 0.00035191699862480164, + 0.00035333300183992833, + 0.0003499160084174946, + 0.00035866600228473544 + ] + }, + { + "forward": 0.0014027089928276837, + "prefill": 0.0014212499954737723, + "decode_steps": [ + 0.0004181669937679544, + 0.00040083400381263345, + 0.0004622089909389615, + 0.00042629199742805213, + 0.0003822090075118467, + 0.00038225000025704503, + 0.0004042090004077181, + 0.0003808750043390319, + 0.00037445800262503326, + 0.00039762500091455877, + 0.00041845900705084205, + 0.00036454200744628906, + 0.0003602920041885227, + 0.0003802499995799735, + 0.0003553340065991506, + 0.00035525001294445246, + 0.000359957994078286, + 0.0003557500021997839, + 0.00035645799653138965, + 0.0003505829954519868, + 0.0003514580021146685, + 0.0003521659964462742, + 0.00036354099574964494, + 0.0004141249955864623, + 0.00040433299727737904, + 0.00045812499593012035, + 0.0004785000055562705, + 0.0004116670024814084, + 0.0004038749902974814, + 0.0003697079955600202, + 0.0003640420036390424, + 0.0003637080080807209 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 1.4211669986252673, + "p95": 2.399622953089418, + "min": 1.4027089928276837, + "max": 3.077417000895366, + "iqr": 0.08844801050145179, + "mean": 1.6130875999806449, + "cv": 0.32076419454488203 + }, + "prefill_ms": { + "count": 10, + "median": 1.4379790009115823, + "p95": 1.9752606043766712, + "min": 1.4184579922584817, + "max": 2.2789170034229755, + "iqr": 0.03239525540266186, + "mean": 1.534829098090995, + "cv": 0.17409648767574332 + }, + "decode_ms_per_step": { + "count": 320, + "median": 0.36025000736117363, + "p95": 0.4536913431365975, + "min": 0.3442499873926863, + "max": 0.6550000107381493, + "iqr": 0.019239760149503127, + "mean": 0.37632684425261687, + "cv": 0.11029720821633276 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 2746.8717793724973, + "p95": 2809.3959775157923, + "min": 2182.8411815580266, + "max": 2819.372813151892, + "iqr": 120.86108305352673, + "mean": 2670.930565269744, + "cv": 0.070168819259352 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 2442.818205247726, + "p95": 2497.8238352124763, + "min": 1889.1639259463407, + "max": 2506.167512220104, + "iqr": 106.57583302698595, + "mean": 2371.601870918251, + "cv": 0.07625285189964733 + }, + "memory": { + "peak_extra_bytes": 4177920, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "reference", + "batch": 1, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.0017766669916454703, + "prefill": 0.0017495829961262643, + "decode_steps": [ + 0.00046683299297001213, + 0.0004500420036492869, + 0.0004318749997764826, + 0.0004202499985694885, + 0.0004329999937908724, + 0.00040783399890642613, + 0.00042904200381599367, + 0.0004509169957600534, + 0.0004484590026549995, + 0.00044070898729842156, + 0.00042554100218694657, + 0.00045979200513102114, + 0.000417333998484537, + 0.00041920899820979685, + 0.0004220000118948519, + 0.0004218340036459267, + 0.0004189160099485889, + 0.0004236249951645732, + 0.0004055419994983822, + 0.0004252500075381249, + 0.0004259999986970797, + 0.00043812500371132046, + 0.000447375001385808, + 0.00044308300130069256, + 0.0004324160108808428, + 0.0004286659968784079, + 0.00043083299533464015, + 0.00044512499880511314, + 0.00042820899398066103, + 0.0004303330060793087, + 0.0004404170031193644, + 0.00043924999772571027 + ] + }, + { + "forward": 0.0018292919994564727, + "prefill": 0.0017653749964665622, + "decode_steps": [ + 0.00046979199396446347, + 0.0004207500023767352, + 0.0004300839937059209, + 0.0004269170021871105, + 0.0004132920003030449, + 0.0004035419988213107, + 0.00042304200178477913, + 0.00041695800609886646, + 0.0004160000098636374, + 0.0004141670069657266, + 0.00041379198955837637, + 0.0004154170019319281, + 0.0004223750001983717, + 0.00041649999911896884, + 0.0004464999947231263, + 0.00045162500464357436, + 0.000435957990703173, + 0.00042387501162011176, + 0.00043566699605435133, + 0.0004385000065667555, + 0.0004378750018076971, + 0.00045533300726674497, + 0.0004298329877201468, + 0.000414624999393709, + 0.000432166998507455, + 0.00046541700430680066, + 0.0004337499995017424, + 0.0004329999937908724, + 0.00044433301081880927, + 0.0004360409948276356, + 0.00042033300269395113, + 0.00043304100108798593 + ] + }, + { + "forward": 0.0017672089888947085, + "prefill": 0.0018599169998196885, + "decode_steps": [ + 0.0004931250005029142, + 0.0004326250054873526, + 0.0004275839892216027, + 0.00043029199878219515, + 0.0004480829957174137, + 0.00045429100282490253, + 0.00043070899846497923, + 0.0004415840085130185, + 0.0004380830068839714, + 0.00040316600643564016, + 0.0004224169970257208, + 0.00041200000850949436, + 0.00042829199810512364, + 0.0004345420020399615, + 0.00041708300705067813, + 0.0004260419955244288, + 0.00042220800241921097, + 0.00043654200271703303, + 0.000429292005719617, + 0.0004210410115774721, + 0.0004254590021446347, + 0.00042204199417028576, + 0.0004430410044733435, + 0.0004389590030768886, + 0.0004342500033089891, + 0.0004242080030962825, + 0.00042804099211934954, + 0.00040870900556910783, + 0.0004251669888617471, + 0.0004270419885870069, + 0.00044879200868308544, + 0.0004241249989718199 + ] + }, + { + "forward": 0.0017823750094976276, + "prefill": 0.001746499998262152, + "decode_steps": [ + 0.0004726250044768676, + 0.00042391700844746083, + 0.0004092500021215528, + 0.00040495798748452216, + 0.00041595900256652385, + 0.0004243340081302449, + 0.000421667005866766, + 0.00040875001286622137, + 0.0004251669888617471, + 0.00042041699634864926, + 0.00041758400038816035, + 0.00041700000292621553, + 0.0004249579942552373, + 0.0004166250000707805, + 0.0004180410032859072, + 0.00042433300404809415, + 0.00042491700151003897, + 0.0004245000018272549, + 0.0004092919989489019, + 0.00042416699579916894, + 0.0004116670024814084, + 0.000422583005274646, + 0.00043220800580456853, + 0.00046816600661259145, + 0.0004639169928850606, + 0.0004191250045550987, + 0.00042645800567697734, + 0.000418249997892417, + 0.00042312500590924174, + 0.0004182919947197661, + 0.00042004199349321425, + 0.0004147500003455207 + ] + }, + { + "forward": 0.00179220900463406, + "prefill": 0.0017591659998288378, + "decode_steps": [ + 0.0004792909894604236, + 0.0004293340025469661, + 0.00041849999979604036, + 0.000415000002249144, + 0.0004215420049149543, + 0.00044291699305176735, + 0.000443582990556024, + 0.00043679200462065637, + 0.0004267080075806007, + 0.00043620800715871155, + 0.00043283299601171166, + 0.0004414999857544899, + 0.0004480829957174137, + 0.00043200000072829425, + 0.00041891699947882444, + 0.00042666700028348714, + 0.0004117920034332201, + 0.00042758299969136715, + 0.0004412920097820461, + 0.0004666659951908514, + 0.0004402920021675527, + 0.0004282080044504255, + 0.00042554098763503134, + 0.00042120899888686836, + 0.0004195420042378828, + 0.00043162499787285924, + 0.00042954200762324035, + 0.00041441699431743473, + 0.0004239160043653101, + 0.0004312499950174242, + 0.000417499992181547, + 0.0004242919967509806 + ] + }, + { + "forward": 0.0017688340012682602, + "prefill": 0.0017538329993840307, + "decode_steps": [ + 0.00047412500134669244, + 0.00041812501149252057, + 0.00040741699922364205, + 0.0004093750030733645, + 0.00041120799141936004, + 0.00041454100573901087, + 0.00041308399522677064, + 0.00047808299132157117, + 0.0004249159974278882, + 0.00042308300908189267, + 0.00042445899453014135, + 0.00042091700015589595, + 0.0004382920014904812, + 0.0004348340007709339, + 0.0004285829927539453, + 0.00041979200614150614, + 0.0004012080025859177, + 0.0004123750113649294, + 0.00041774999408517033, + 0.0004244169977027923, + 0.0004122499958612025, + 0.0004168330051470548, + 0.00042895801016129553, + 0.00042004199349321425, + 0.000417874995036982, + 0.0004134579940000549, + 0.0004227089957566932, + 0.00041929200233425945, + 0.00041200000850949436, + 0.0004180830001132563, + 0.00041920899820979685, + 0.0004239999980200082 + ] + }, + { + "forward": 0.0022104579984443262, + "prefill": 0.002346791996387765, + "decode_steps": [ + 0.0006992499984335154, + 0.000570459000300616, + 0.00047862500650808215, + 0.0004288339987397194, + 0.0004428329993970692, + 0.0004216660017846152, + 0.0004212079948047176, + 0.0004422089987201616, + 0.00042045798909384757, + 0.0004242080030962825, + 0.0004169169988017529, + 0.0004392080008983612, + 0.000433833003626205, + 0.0004287920019123703, + 0.00042387501162011176, + 0.0004315000114729628, + 0.00042391700844746083, + 0.000420958996983245, + 0.00039820799429435283, + 0.0004112919996259734, + 0.0004303339956095442, + 0.0004716669936897233, + 0.00045162499009165913, + 0.00044920798973180354, + 0.0004451660061022267, + 0.00044175000221002847, + 0.0004453330038813874, + 0.0004634159995475784, + 0.00043233299220446497, + 0.0004324999899836257, + 0.0004261249996488914, + 0.00042566699266899377 + ] + }, + { + "forward": 0.0017730840045260265, + "prefill": 0.0019191669998690486, + "decode_steps": [ + 0.0005214999982854351, + 0.0004352910036686808, + 0.0004063340020366013, + 0.0004099590005353093, + 0.00041491599404253066, + 0.0004009590047644451, + 0.00040791700303088874, + 0.0004226249875500798, + 0.0004009160038549453, + 0.00040874999831430614, + 0.00041237499681301415, + 0.00041283298924099654, + 0.0004149590095039457, + 0.00041641700954642147, + 0.00042241599294357, + 0.00042912500794045627, + 0.00042308299452997744, + 0.00039766699774190784, + 0.0004101670056115836, + 0.000408500010962598, + 0.0004276249965187162, + 0.00041649999911896884, + 0.00041725000482983887, + 0.00042058300459757447, + 0.00043195900798309594, + 0.00041633400542195886, + 0.00041362499177921563, + 0.00042541700531728566, + 0.0004278330015949905, + 0.00042679099715314806, + 0.00040695900679565966, + 0.00042374999611638486 + ] + }, + { + "forward": 0.0017767079989425838, + "prefill": 0.001792750001186505, + "decode_steps": [ + 0.00048387500282842666, + 0.0004299589927541092, + 0.00042262500210199505, + 0.0004247089964337647, + 0.000436875008745119, + 0.00043125000956933945, + 0.0004212079948047176, + 0.00042383300024084747, + 0.00041533299372531474, + 0.00040066600195132196, + 0.0004149169981246814, + 0.0004202499985694885, + 0.0004159170057391748, + 0.00040433299727737904, + 0.0004184580029686913, + 0.00041237499681301415, + 0.0004232500068610534, + 0.00040454200643580407, + 0.0003964999923482537, + 0.0004014999867649749, + 0.0004020829946966842, + 0.00042374999611638486, + 0.00041037499613594264, + 0.0004179579991614446, + 0.0004147080035181716, + 0.00041508399590384215, + 0.0004137920041102916, + 0.00041008299740497023, + 0.0004225000011501834, + 0.00041274999966844916, + 0.0004457500035641715, + 0.0004437500028871 + ] + }, + { + "forward": 0.001820749996113591, + "prefill": 0.0017943749990081415, + "decode_steps": [ + 0.00048787498963065445, + 0.0004370420065242797, + 0.0004257499967934564, + 0.0004150829918216914, + 0.0004088329878868535, + 0.0004159170057391748, + 0.0004075420001754537, + 0.00042291700083296746, + 0.0004149160085944459, + 0.0004110409936401993, + 0.00040604099922347814, + 0.00041370801045559347, + 0.00041987501026596874, + 0.0004165419959463179, + 0.0004044579982291907, + 0.0004151250032009557, + 0.00041483298991806805, + 0.00044641600106842816, + 0.00044054200407117605, + 0.00043158400512766093, + 0.00044629200419876724, + 0.0004402920021675527, + 0.00045020900142844766, + 0.00045279199548531324, + 0.00045591600064653903, + 0.0004479589988477528, + 0.00042766600381582975, + 0.00043191699660383165, + 0.0004323330067563802, + 0.00046062500041443855, + 0.0004939170030411333, + 0.0007049160049064085 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 1.7795415042201057, + "p95": 2.0389332988997917, + "min": 1.7672089888947085, + "max": 2.210457998444326, + "iqr": 0.03963499693782069, + "mean": 1.8297585993423127, + "cv": 0.07403010675720109 + }, + "prefill_ms": { + "count": 10, + "median": 1.7790624988265336, + "p95": 2.154360747954342, + "min": 1.746499998262152, + "max": 2.346791996387765, + "iqr": 0.08836525012156926, + "mean": 1.8487457986338995, + "cv": 0.09932958182168612 + }, + "decode_ms_per_step": { + "count": 320, + "median": 0.42424999992363155, + "p95": 0.4668996436521411, + "min": 0.3964999923482537, + "max": 0.7049160049064085, + "iqr": 0.018416754755889997, + "mean": 0.429908106525545, + "cv": 0.0673612011704873 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 2321.4220744404934, + "p95": 2382.1992923262815, + "min": 2238.799447881447, + "max": 2382.4884073657595, + "iqr": 52.00017954073337, + "mean": 2326.9543130220227, + "cv": 0.020347557047382486 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 2057.906989792077, + "p95": 2098.4969391915506, + "min": 1923.0576265898085, + "max": 2101.432469012023, + "iqr": 42.80120967984749, + "mean": 2051.7876070543825, + "cv": 0.02571127849611151 + }, + "memory": { + "peak_extra_bytes": 4161536, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "auto", + "batch": 1, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.00417354199453257 + }, + { + "forward": 0.004216625005938113 + }, + { + "forward": 0.004342750005889684 + }, + { + "forward": 0.004187416998320259 + }, + { + "forward": 0.004175667010713369 + }, + { + "forward": 0.004618833991116844 + }, + { + "forward": 0.004141625002375804 + }, + { + "forward": 0.0041371670085936785 + }, + { + "forward": 0.004179958996246569 + }, + { + "forward": 0.0049742499977583066 + } + ], + "forward_ms": { + "count": 10, + "median": 4.183687997283414, + "p95": 4.8143127947696485, + "min": 4.1371670085936785, + "max": 4.974249997758307, + "iqr": 0.1371455073240213, + "mean": 4.31478360114852, + "cv": 0.06337606754944948 + }, + "memory": { + "peak_extra_bytes": 9371648, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "sdpa", + "batch": 1, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.0029251249943627045, + "prefill": 0.0029639580025104806, + "decode_steps": [ + 0.0005514579970622435, + 0.0004663339932449162, + 0.0004612079937942326, + 0.00046595800085924566, + 0.0004522499948507175, + 0.00045770799624733627, + 0.00044729099317919463, + 0.00045933299406897277, + 0.00044800000614486635, + 0.000454790992080234, + 0.0004581250104820356, + 0.00045095800305716693, + 0.0004623749991878867, + 0.0004954159958288074, + 0.00046970898984000087, + 0.0004470409912755713, + 0.00046020798617973924, + 0.0004618330131052062, + 0.000451540996436961, + 0.0004482079966692254, + 0.0004505829856498167, + 0.0004474580055102706, + 0.00046395800018217415, + 0.0004554170009214431, + 0.00044754099508281797, + 0.0004501249932218343, + 0.0004509170103119686, + 0.0004555000050459057, + 0.0004455419984878972, + 0.00044266700570005924, + 0.0004534170002443716, + 0.00046875000407453626 + ] + }, + { + "forward": 0.0029372910066740587, + "prefill": 0.002949750007246621, + "decode_steps": [ + 0.0006144169892650098, + 0.0004943750100210309, + 0.00047887499385979027, + 0.000483750001876615, + 0.0004686250031227246, + 0.00047170800098683685, + 0.0004607500013662502, + 0.00047937499766703695, + 0.00048720800259616226, + 0.0004542920069070533, + 0.00045370901352725923, + 0.0004530000005615875, + 0.00045170799421612173, + 0.0004548330034594983, + 0.0004486249963520095, + 0.0004469170089578256, + 0.00044800000614486635, + 0.0004547499993350357, + 0.0004524169926298782, + 0.00046270800521597266, + 0.00044641600106842816, + 0.00045858300291001797, + 0.00045691699779126793, + 0.0004571669996948913, + 0.0004674170049838722, + 0.0004530000005615875, + 0.0004530419973889366, + 0.00044795799476560205, + 0.0004514579923124984, + 0.00044983399857301265, + 0.0004794999986188486, + 0.0004999999946448952 + ] + }, + { + "forward": 0.0030188340024324134, + "prefill": 0.0029933749901829287, + "decode_steps": [ + 0.0005854170012753457, + 0.0005024999991292134, + 0.0004748329956782982, + 0.00046920799650251865, + 0.000468166996142827, + 0.0004658339894376695, + 0.000454249995527789, + 0.0004571250028675422, + 0.000452291002147831, + 0.0004533750034170225, + 0.0004537920030998066, + 0.00044687501213047653, + 0.000454084001830779, + 0.0004551250021904707, + 0.0005110000056447461, + 0.0005156249972060323, + 0.0006017499981680885, + 0.0005337920010788366, + 0.00047212500066962093, + 0.00047783298941794783, + 0.0009166669915430248, + 0.000627791989245452, + 0.0005959580012131482, + 0.0005600000004051253, + 0.0005158330022823066, + 0.00045700000191573054, + 0.00045229199167806655, + 0.00047033299051690847, + 0.0004612909979186952, + 0.0004602919943863526, + 0.000457499991171062, + 0.00045616700663231313 + ] + }, + { + "forward": 0.0030167079967213795, + "prefill": 0.003075040993280709, + "decode_steps": [ + 0.0005627500067930669, + 0.0004750420048367232, + 0.0004547920107143, + 0.000458958005765453, + 0.00044824999349657446, + 0.00045441699330694973, + 0.00045083300210535526, + 0.0004517920024227351, + 0.0004599170060828328, + 0.0004653750074794516, + 0.0004538339999271557, + 0.0004530000005615875, + 0.0004557499923976138, + 0.00044995799544267356, + 0.0004522499948507175, + 0.00048349999997299165, + 0.00047899999481160194, + 0.00047316699055954814, + 0.0004555830091703683, + 0.00046141700295265764, + 0.00045429199235513806, + 0.00044800000614486635, + 0.0004507919948082417, + 0.00045325000246521086, + 0.0004592500044964254, + 0.00045395900087896734, + 0.0005216669960645959, + 0.0005283750069793314, + 0.00047112499305512756, + 0.0005200409941608086, + 0.0005107089964440092, + 0.0004650409973692149 + ] + }, + { + "forward": 0.003384625000762753, + "prefill": 0.003405542011023499, + "decode_steps": [ + 0.0006270410085562617, + 0.00048262499331030995, + 0.00046537499292753637, + 0.00048225000500679016, + 0.00045641699398402125, + 0.00045729200064670295, + 0.0004676669923355803, + 0.00045820800005458295, + 0.00045154200051911175, + 0.0004604170098900795, + 0.0004673750081565231, + 0.0004575830098474398, + 0.0004579999949783087, + 0.00046112500422168523, + 0.0004655840020859614, + 0.0004647080058930442, + 0.0005162089946679771, + 0.00048620899906381965, + 0.0004740830045193434, + 0.00047120799717959017, + 0.0004870830016443506, + 0.0004738749994430691, + 0.00048062499263323843, + 0.0004667080065701157, + 0.0004682079888880253, + 0.00047908299893606454, + 0.0004697499971371144, + 0.0005338339979061857, + 0.00047812500270083547, + 0.0005013330082874745, + 0.0004784170014318079, + 0.0004879580083070323 + ] + }, + { + "forward": 0.003060415998334065, + "prefill": 0.003128999989712611, + "decode_steps": [ + 0.0006407079927157611, + 0.00047745899064466357, + 0.0004654580116039142, + 0.00046291699982248247, + 0.0004691659996751696, + 0.00047524999536108226, + 0.0004628750029951334, + 0.00046024999755900353, + 0.0004561660025501624, + 0.00046799999836366624, + 0.000451375002739951, + 0.00045104199671186507, + 0.00045200000749900937, + 0.00047487500705756247, + 0.0004547499993350357, + 0.000461667004856281, + 0.0004559579974738881, + 0.0004616250080289319, + 0.00046216699411161244, + 0.00045233400305733085, + 0.0004635420045815408, + 0.00047154199273791164, + 0.0004531250015133992, + 0.00045325000246521086, + 0.00044137499935459346, + 0.0004543329996522516, + 0.0004504999960772693, + 0.0004485829995246604, + 0.00044741699821315706, + 0.00045341698569245636, + 0.00044354201236274093, + 0.0004508339916355908 + ] + }, + { + "forward": 0.002989915999933146, + "prefill": 0.002957791992230341, + "decode_steps": [ + 0.0005329999985406175, + 0.000472707994049415, + 0.00045416600187309086, + 0.0004505830002017319, + 0.00045458300155587494, + 0.0004596250073518604, + 0.0004539159999694675, + 0.00045083300210535526, + 0.0004535829939413816, + 0.00045020900142844766, + 0.0004587500006891787, + 0.0004457500035641715, + 0.00046799999836366624, + 0.0004481249925447628, + 0.0004517499910434708, + 0.0004479580093175173, + 0.0004515839973464608, + 0.0004512080049607903, + 0.0004487080004764721, + 0.00044925000111106783, + 0.0004547499993350357, + 0.0004469580017030239, + 0.0005040419928263873, + 0.00046379200648516417, + 0.000450999999884516, + 0.0004525419935816899, + 0.0004491669969866052, + 0.00046170900168363005, + 0.00045941700227558613, + 0.0004682909930124879, + 0.0004540000081760809, + 0.0004524169926298782 + ] + }, + { + "forward": 0.0029125409928383306, + "prefill": 0.0029416670004138723, + "decode_steps": [ + 0.0005282499914756045, + 0.0004669170011766255, + 0.00045570901420433074, + 0.00045537500409409404, + 0.00045029200555291027, + 0.0004539579967968166, + 0.00045950000640004873, + 0.0004469580017030239, + 0.00045437499647960067, + 0.00045754099846817553, + 0.0004534579929895699, + 0.00044904199603479356, + 0.00045437499647960067, + 0.00045679199683945626, + 0.00045050001062918454, + 0.00045791600132361054, + 0.0004628750029951334, + 0.00045708300604019314, + 0.00046354100049939007, + 0.00045162499009165913, + 0.000451375002739951, + 0.000454415989224799, + 0.00044991698814556, + 0.0004514169995673001, + 0.00044912500015925616, + 0.0004555420018732548, + 0.0004534170002443716, + 0.0004512080049607903, + 0.00045083298755344003, + 0.00045037499512545764, + 0.0004490420105867088, + 0.0004464169905986637 + ] + }, + { + "forward": 0.0029284170013852417, + "prefill": 0.002981957994052209, + "decode_steps": [ + 0.0005452500045066699, + 0.00047591600741725415, + 0.00045487500028684735, + 0.0004556659987429157, + 0.0004565840063150972, + 0.0004580829991027713, + 0.0005249579990049824, + 0.000490999998874031, + 0.0004682080034399405, + 0.00045641699398402125, + 0.000453874992672354, + 0.00044837500900030136, + 0.0004502079973462969, + 0.00045029200555291027, + 0.0004472500004339963, + 0.00044925000111106783, + 0.0004518329951679334, + 0.00045012500777374953, + 0.0004541249945759773, + 0.000456583991763182, + 0.00045316699834074825, + 0.00048587500350549817, + 0.0005764169909525663, + 0.0004979999939678237, + 0.0005020419921493158, + 0.0005197079881327227, + 0.0005029169988119975, + 0.0004771249950863421, + 0.0004974170005880296, + 0.00052375000086613, + 0.0005441249959403649, + 0.0005140830035088584 + ] + }, + { + "forward": 0.0035098340013064444, + "prefill": 0.003517959004966542, + "decode_steps": [ + 0.0006748330051777884, + 0.0005266250082058832, + 0.0005272499984130263, + 0.0004751670057885349, + 0.0004826250078622252, + 0.0004682920116465539, + 0.000500916998134926, + 0.0004788329970324412, + 0.0005076669913250953, + 0.00047737499698996544, + 0.0004920829960610718, + 0.0004616249934770167, + 0.00046066599315963686, + 0.0004697499971371144, + 0.0005078340036561713, + 0.00048008399608079344, + 0.000472542000352405, + 0.0005199160077609122, + 0.0004943329986417666, + 0.0004747909988509491, + 0.00047879100020509213, + 0.0004767920036101714, + 0.0004737919953186065, + 0.00047541600361000746, + 0.00046983400534372777, + 0.0004706249892478809, + 0.00047291599912568927, + 0.0005054590001236647, + 0.0004951669980073348, + 0.0004934169992338866, + 0.0004749999934574589, + 0.000493708997964859 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 3.0033119983272627, + "p95": 3.4534899510617834, + "min": 2.9125409928383306, + "max": 3.5098340013064444, + "iqr": 0.11938499665120617, + "mean": 3.0683706994750537, + "cv": 0.06765276112526458 + }, + "prefill_ms": { + "count": 10, + "median": 2.987666492117569, + "p95": 3.4673713576921728, + "min": 2.9416670004138723, + "max": 3.517959004966542, + "iqr": 0.15617674580425955, + "mean": 3.0916041985619813, + "cv": 0.06650620631561788 + }, + "decode_ms_per_step": { + "count": 320, + "median": 0.45945850433781743, + "p95": 0.5337941009202041, + "min": 0.44137499935459346, + "max": 0.9166669915430248, + "iqr": 0.02362426312174648, + "mean": 0.4731179714326572, + "cv": 0.08840721377650176 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 2134.961560329716, + "p95": 2186.9845885146606, + "min": 1983.127802438825, + "max": 2191.9060336411835, + "iqr": 90.5334824620304, + "mean": 2115.7002293245737, + "cv": 0.03254465173709856 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1773.4856096376875, + "p95": 1820.1549754683745, + "min": 1662.157631586388, + "max": 1824.3147308802024, + "iqr": 89.77776021452883, + "mean": 1757.0253759679622, + "cv": 0.03355145081467758 + }, + "memory": { + "peak_extra_bytes": 5472256, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "reference", + "batch": 1, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.003635249988292344, + "prefill": 0.00351595800020732, + "decode_steps": [ + 0.0006149160035420209, + 0.000538249994860962, + 0.000541832996532321, + 0.0005356670008040965, + 0.0005329170089680701, + 0.0005384590040193871, + 0.0005240830068942159, + 0.000526624993653968, + 0.0005537919932976365, + 0.0005492499913088977, + 0.0005352089938241988, + 0.0005296249873936176, + 0.0005329580017132685, + 0.0005226250068517402, + 0.0005337079928722233, + 0.0005364999960875139, + 0.0005215419951127842, + 0.000538042004336603, + 0.0005502500134753063, + 0.0005462920089485124, + 0.000528500007931143, + 0.000528125005075708, + 0.0005295829905662686, + 0.0005449999880511314, + 0.0005317920004017651, + 0.0005395829939516261, + 0.0005235830030869693, + 0.0005341670039342716, + 0.0005438749940367416, + 0.0005245419888524339, + 0.0005524580046767369, + 0.000526999996509403 + ] + }, + { + "forward": 0.0037775829987367615, + "prefill": 0.003616417001467198, + "decode_steps": [ + 0.0006219579954631627, + 0.0005286249943310395, + 0.0006440410070354119, + 0.00057533300423529, + 0.0007042920042295009, + 0.0005962499999441206, + 0.0005473749915836379, + 0.0005362499941838905, + 0.0005291249981382862, + 0.0005347079859348014, + 0.000548875003005378, + 0.0005240000027697533, + 0.0005369999998947605, + 0.0005576249968726188, + 0.000530083998455666, + 0.0005213749973336235, + 0.0005408750002970919, + 0.0005269579996820539, + 0.0005864160048076883, + 0.0005983329901937395, + 0.0005650829989463091, + 0.0005508750036824495, + 0.000550583004951477, + 0.0005821249942528084, + 0.0005404580006143078, + 0.0005327499966369942, + 0.0005897499941056594, + 0.000559791995328851, + 0.0005327499966369942, + 0.000537832995178178, + 0.0005141250003362074, + 0.0005444169946713373 + ] + }, + { + "forward": 0.003629332990385592, + "prefill": 0.003586499995435588, + "decode_steps": [ + 0.0006196670001372695, + 0.0005502919957507402, + 0.0005429999873740599, + 0.0005829589936183766, + 0.0005969579942757264, + 0.0005297919997246936, + 0.0005299579934217036, + 0.0005174999969312921, + 0.0005307910032570362, + 0.0005409999866969883, + 0.0005275000003166497, + 0.0005265000072540715, + 0.0005339589988579974, + 0.0005253749986877665, + 0.0005246249929768965, + 0.0005237080040387809, + 0.0005444999987957999, + 0.0006798330141464248, + 0.000622375009697862, + 0.0006147499952930957, + 0.0006630409916397184, + 0.0006207919941516593, + 0.0005810840084450319, + 0.000610957999015227, + 0.0005810000002384186, + 0.0005706660012947395, + 0.0005324999947333708, + 0.000533667000127025, + 0.0005321659991750494, + 0.0005347080004867166, + 0.0005188329960219562, + 0.0005694169958587736 + ] + }, + { + "forward": 0.0037654170009773225, + "prefill": 0.004123499995330349, + "decode_steps": [ + 0.0007694999949308112, + 0.0006369159964378923, + 0.0006502909964183345, + 0.0006022919988026842, + 0.0005458750092657283, + 0.0005727079987991601, + 0.0005917499947827309, + 0.0005539579869946465, + 0.0005691250116797164, + 0.0005690839898306876, + 0.0005813750030938536, + 0.000577291997615248, + 0.000644083003862761, + 0.0005709160031983629, + 0.0005387919954955578, + 0.0005282920028548688, + 0.0005411669990280643, + 0.000549625008716248, + 0.0005707079981220886, + 0.0005626250058412552, + 0.0005958339897915721, + 0.0005362500087358057, + 0.0005400829977588728, + 0.0005336249887477607, + 0.0005581659934250638, + 0.0005404160037869588, + 0.000545625007362105, + 0.0005310419946908951, + 0.0005305419908836484, + 0.0005295830051181838, + 0.0005304160004016012, + 0.0005494170036399737 + ] + }, + { + "forward": 0.003849707994959317, + "prefill": 0.003656750006484799, + "decode_steps": [ + 0.0006394590018317103, + 0.0005383750103646889, + 0.0005352080042939633, + 0.0005392919993028045, + 0.000525874987943098, + 0.0005247500084806234, + 0.0005408750002970919, + 0.0005399170040618628, + 0.0005248329980531707, + 0.0005276250012684613, + 0.0005322919896570966, + 0.0005248329980531707, + 0.0005360420036595315, + 0.0005212499963818118, + 0.0005464169953484088, + 0.0005275000003166497, + 0.000545082992175594, + 0.0005398330104071647, + 0.0005218750011408702, + 0.0005127500044181943, + 0.0005322920042090118, + 0.0005332080036168918, + 0.0005274589930195361, + 0.0005357909976737574, + 0.0005524169973796234, + 0.0005299999902490526, + 0.0005368749989429489, + 0.0005309170082909986, + 0.000538624997716397, + 0.0005484579887706786, + 0.0005283750069793314, + 0.0005426249990705401 + ] + }, + { + "forward": 0.0035419580090092495, + "prefill": 0.0035578340030042455, + "decode_steps": [ + 0.000599791994318366, + 0.0005428750009741634, + 0.0005240830068942159, + 0.0005282079946482554, + 0.0005139589920872822, + 0.0005300829943735152, + 0.0005198330036364496, + 0.000535333005245775, + 0.0005254580028122291, + 0.0005387910059653223, + 0.0005185409972909838, + 0.0005297500028973445, + 0.0005264999927021563, + 0.000530958001036197, + 0.0005200000014156103, + 0.0005236660072114319, + 0.0005220419989200309, + 0.0005215410055825487, + 0.0005155419930815697, + 0.0005271249974612147, + 0.0005422499962151051, + 0.0005670000100508332, + 0.0005522919964278117, + 0.0005449580057756975, + 0.0005311660061124712, + 0.0005258329911157489, + 0.0005252909904811531, + 0.0005327079998096451, + 0.0005264999927021563, + 0.0005202500033192337, + 0.0005263750063022599, + 0.0005283339996822178 + ] + }, + { + "forward": 0.0036697920004371554, + "prefill": 0.003539792000083253, + "decode_steps": [ + 0.0006170419947011396, + 0.0005479170067701489, + 0.0005277919990476221, + 0.0005214169941609725, + 0.0005364579992601648, + 0.0005204589979257435, + 0.0005170840013306588, + 0.0005896249931538478, + 0.0005321669887052849, + 0.0005361250077839941, + 0.0005293750000419095, + 0.000531540994415991, + 0.0005357079935492948, + 0.0005304169899318367, + 0.0005453339981613681, + 0.0005305419908836484, + 0.0005371669976739213, + 0.0005311249988153577, + 0.0005269590037642047, + 0.000533874990651384, + 0.0005221659957896918, + 0.0005214999982854351, + 0.0005250829999567941, + 0.0005233339907135814, + 0.0005259999888949096, + 0.000563749999855645, + 0.0005365840042941272, + 0.0005288750107865781, + 0.0005302079953253269, + 0.000540083012310788, + 0.0005210829986026511, + 0.0005320419877534732 + ] + }, + { + "forward": 0.0036257920000934973, + "prefill": 0.0035368329990888014, + "decode_steps": [ + 0.0005904589925194159, + 0.0005591660010395572, + 0.0005222920008236542, + 0.0005350420105969533, + 0.0005430409946711734, + 0.0005327499966369942, + 0.0005402079987106845, + 0.0005292910063872114, + 0.0005240409955149516, + 0.0005274170107441023, + 0.000531209007021971, + 0.0005272499984130263, + 0.0005463749985210598, + 0.0005312499997671694, + 0.0005213330005062744, + 0.0005179580039111897, + 0.000545292001334019, + 0.0005183329922147095, + 0.0005375829932745546, + 0.0005364579992601648, + 0.0005221250030444935, + 0.0005123750015627593, + 0.0005295000009937212, + 0.0005360419891076162, + 0.0005257919983705506, + 0.0005337500042514876, + 0.0005277079908410087, + 0.0005315000016707927, + 0.0005355000030249357, + 0.0005205839988775551, + 0.0005271250120131299, + 0.0005404159892350435 + ] + }, + { + "forward": 0.003803416999289766, + "prefill": 0.0035987500014016405, + "decode_steps": [ + 0.0006113750132499263, + 0.0006483339966507629, + 0.0006621250067837536, + 0.0006375420052791014, + 0.0005615410045720637, + 0.0005422919930424541, + 0.0005424169939942658, + 0.0005249159876257181, + 0.0005214580014580861, + 0.0005649999948218465, + 0.0005319999909261242, + 0.0005599580035777763, + 0.000541082990821451, + 0.0005244159983703867, + 0.0005213750118855387, + 0.0005214170087128878, + 0.0005424590053735301, + 0.0005439160013338551, + 0.0005517910030903295, + 0.0005492499913088977, + 0.0005738750041928142, + 0.0005548339977394789, + 0.0005401670059654862, + 0.0005651250103255734, + 0.0005508329923031852, + 0.0005582500016316772, + 0.0006070830131648108, + 0.0005400829977588728, + 0.0005543340084841475, + 0.0005348330014385283, + 0.0005354580061975867, + 0.0005739159969380125 + ] + }, + { + "forward": 0.0038172079948708415, + "prefill": 0.0036179590097162873, + "decode_steps": [ + 0.0006182500073919073, + 0.0005507500027306378, + 0.0005318339972291142, + 0.0005287079984555021, + 0.0005319589981809258, + 0.0005402499955380335, + 0.000530249992152676, + 0.0005358750058803707, + 0.0005154999962542206, + 0.0005181670130696148, + 0.0005355419998522848, + 0.000529416007339023, + 0.0005307079991325736, + 0.000539708009455353, + 0.0005332079890649766, + 0.0005290000117383897, + 0.0005252080009086058, + 0.0005324999947333708, + 0.0005340409989003092, + 0.0005223329935688525, + 0.0005360829964047298, + 0.0005280409968690947, + 0.0005332080036168918, + 0.0005252919945633039, + 0.0005307089886628091, + 0.0005333750013960525, + 0.0005772909935330972, + 0.0005795409961137921, + 0.0005517079989658669, + 0.0005299170006765053, + 0.0005238329904386774, + 0.000520166999194771 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 3.717604500707239, + "p95": 3.835082994919503, + "min": 3.5419580090092495, + "max": 3.849707994959317, + "iqr": 0.16614625928923488, + "mean": 3.7115457977051847, + "cv": 0.027883566935858833 + }, + "prefill_ms": { + "count": 10, + "median": 3.5926249984186143, + "p95": 3.913462500349851, + "min": 3.51595800020732, + "max": 4.123499995330349, + "iqr": 0.07327100684051402, + "mean": 3.635029301221948, + "cv": 0.048716455827873394 + }, + "decode_ms_per_step": { + "count": 320, + "median": 0.5352709995349869, + "p95": 0.6183208570291754, + "min": 0.5123750015627593, + "max": 0.7694999949308112, + "iqr": 0.02201026291004382, + "mean": 0.546387593021791, + "cv": 0.06076308591693818 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1860.8383914237863, + "p95": 1878.3786435307622, + "min": 1753.677092854688, + "max": 1880.4683803936905, + "iqr": 76.69035372041253, + "mean": 1831.3648177811872, + "cv": 0.02639206104808694 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1537.0511046247218, + "p95": 1554.5773651080588, + "min": 1430.4314380188205, + "max": 1555.2952920431358, + "iqr": 57.86101402653935, + "mean": 1516.2466843731781, + "cv": 0.027472068875141253 + }, + "memory": { + "peak_extra_bytes": 7651328, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "auto", + "batch": 1, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.010288042001775466 + }, + { + "forward": 0.010224917001323774 + }, + { + "forward": 0.010208499996224418 + }, + { + "forward": 0.010464124992722645 + }, + { + "forward": 0.010531209001783282 + }, + { + "forward": 0.011072082997998223 + }, + { + "forward": 0.011128917001769878 + }, + { + "forward": 0.010345000002416782 + }, + { + "forward": 0.01052508300927002 + }, + { + "forward": 0.011160499998368323 + } + ], + "forward_ms": { + "count": 10, + "median": 10.494604000996333, + "p95": 11.146287649899023, + "min": 10.208499996224418, + "max": 11.160499998368323, + "iqr": 0.6345829970086925, + "mean": 10.594837600365281, + "cv": 0.03589436510008904 + }, + "memory": { + "peak_extra_bytes": 25264128, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "sdpa", + "batch": 1, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.005947458994342014, + "prefill": 0.005910209001740441, + "decode_steps": [ + 0.000696083006914705, + 0.0005388330027926713, + 0.0005227500078035519, + 0.0005239999882178381, + 0.0005149999924469739, + 0.0005098749970784411, + 0.0005114999949000776, + 0.0005034999921917915, + 0.0005060830008005723, + 0.0005082080024294555, + 0.0005002919933758676, + 0.000528125005075708, + 0.0005104999872855842, + 0.0005574579990934581, + 0.0005147079937160015, + 0.0005064169963588938, + 0.0005107079923618585, + 0.0005065000004833564, + 0.0005088329926365986, + 0.0005185000045457855, + 0.0005230829992797226, + 0.0005552920047193766, + 0.0005180000007385388, + 0.0005143330054124817, + 0.0005117079999763519, + 0.0005117499968037009, + 0.0005082909920020029, + 0.0005182080058148131, + 0.0005139999993843958, + 0.0005102919967612252, + 0.000502583003253676, + 0.0005146670009708032 + ] + }, + { + "forward": 0.006164667007396929, + "prefill": 0.0060452089965110645, + "decode_steps": [ + 0.0007080420036800206, + 0.0005352500011213124, + 0.0005207500071264803, + 0.0005152500089025125, + 0.0005127909971633926, + 0.0005498749960679561, + 0.0005580830038525164, + 0.0005479579995153472, + 0.0005488329916261137, + 0.0005276250012684613, + 0.0005330409912858158, + 0.000537832995178178, + 0.0005133749946253374, + 0.0005246250075288117, + 0.000507333010318689, + 0.0005490409967023879, + 0.0005137499974807724, + 0.0005180410080356523, + 0.0005087500030640513, + 0.0005157080013304949, + 0.0005038750095991418, + 0.0005067919992143288, + 0.0005119580018799752, + 0.0005165829934412614, + 0.0005053329950897023, + 0.0005043330020271242, + 0.0005029169988119975, + 0.0005113340012030676, + 0.0005026250000810251, + 0.0005205420020502061, + 0.0005040840042056516, + 0.0005108749901410192 + ] + }, + { + "forward": 0.005814083997393027, + "prefill": 0.0060824999964097515, + "decode_steps": [ + 0.0006974169955356047, + 0.000626042005023919, + 0.0006775419897167012, + 0.000627959001576528, + 0.0005437499930849299, + 0.0005416250060079619, + 0.000516542000696063, + 0.0005102920113131404, + 0.0005115000094519928, + 0.0005035000067437068, + 0.000507333010318689, + 0.0005369590071495622, + 0.0005535829986911267, + 0.0005250419926596805, + 0.0005090830090921372, + 0.0005112910002935678, + 0.0005119170091347769, + 0.0005074159998912364, + 0.0005115840031066909, + 0.0005067919992143288, + 0.0005118330009281635, + 0.0005059169925516471, + 0.0005079580005258322, + 0.0005076250090496615, + 0.0005052500055171549, + 0.0005039999959990382, + 0.0005358339985832572, + 0.0005426249990705401, + 0.0005457500083139166, + 0.0005348749982658774, + 0.0005287919921102002, + 0.0005315829912433401 + ] + }, + { + "forward": 0.005804458007332869, + "prefill": 0.005553209004574455, + "decode_steps": [ + 0.0007280419958988205, + 0.0005456249928101897, + 0.0005419999943114817, + 0.0005255830037640408, + 0.0004947080014972016, + 0.0005036670045228675, + 0.0004966669948771596, + 0.0005387080018408597, + 0.0005233330011833459, + 0.0005076670058770105, + 0.000502958006109111, + 0.0004959580110153183, + 0.0004939159989589825, + 0.0004942919913446531, + 0.0004988339933333918, + 0.0005114999949000776, + 0.0005425419949460775, + 0.000533667000127025, + 0.0005433329934021458, + 0.0005242499901214615, + 0.0005424580012913793, + 0.0005382089875638485, + 0.0005356250039767474, + 0.0005414589977590367, + 0.0005493329954333603, + 0.0005828749999636784, + 0.0005367919948184863, + 0.0005087919998914003, + 0.0004990830057067797, + 0.0005036249931436032, + 0.0005002919933758676, + 0.0005085420125396922 + ] + }, + { + "forward": 0.0057297919993288815, + "prefill": 0.005750625001383014, + "decode_steps": [ + 0.0008760420023463666, + 0.0005801670049550012, + 0.0005676670116372406, + 0.0005246249929768965, + 0.0005129589990247041, + 0.0004996250063413754, + 0.0005027500010328367, + 0.0005101249989820644, + 0.0005058749957242981, + 0.0005771660071332008, + 0.0005084169970359653, + 0.0004948749992763624, + 0.000494042003992945, + 0.0004934579919790849, + 0.0005021249962737784, + 0.0004944579995935783, + 0.0005104169977130368, + 0.0004979579971404746, + 0.0005039590032538399, + 0.0004987499996786937, + 0.0004898329934803769, + 0.0004916660109302029, + 0.0004980829980922863, + 0.0005010830063838512, + 0.000532291000126861, + 0.0005508749891305342, + 0.0005467500013764948, + 0.0005349579878384247, + 0.0005335000023478642, + 0.0005399580113589764, + 0.0005463749985210598, + 0.0005035829963162541 + ] + }, + { + "forward": 0.00628537499869708, + "prefill": 0.005746333001297899, + "decode_steps": [ + 0.000716499998816289, + 0.0005494170036399737, + 0.0005374579923227429, + 0.0005047089944127947, + 0.0005171670054551214, + 0.0005324579979060218, + 0.0006061249878257513, + 0.000531915997271426, + 0.000505667005199939, + 0.0005208749935263768, + 0.0005212909891270101, + 0.0005139999993843958, + 0.0005002910038456321, + 0.0004961249942425638, + 0.0005043329874752089, + 0.0004983749968232587, + 0.0005059159884694964, + 0.0005075420049251989, + 0.0005594999965978786, + 0.0006259590008994564, + 0.000601625011768192, + 0.0005177080020075664, + 0.000500916998134926, + 0.0005063330027041957, + 0.0004958329955115914, + 0.0005236249999143183, + 0.0004984580009477213, + 0.0005002920079277828, + 0.0004982919926987961, + 0.0004932909941999242, + 0.0004919170023640618, + 0.0004994999908376485 + ] + }, + { + "forward": 0.005489125003805384, + "prefill": 0.005772208998678252, + "decode_steps": [ + 0.0007072500011418015, + 0.0005318329931469634, + 0.0005126659962115809, + 0.000498958004754968, + 0.0004893750010523945, + 0.0005044170102337375, + 0.0005097499961266294, + 0.0005996660038363189, + 0.0006058329890947789, + 0.0006125000072643161, + 0.0005139169952599332, + 0.0005154579994268715, + 0.0005040410032961518, + 0.000497667002491653, + 0.0004919589991914108, + 0.0004980829980922863, + 0.0004962090024491772, + 0.0004924999957438558, + 0.00048312499711755663, + 0.0004944579995935783, + 0.0004955000040354207, + 0.000522666989127174, + 0.0005055000074207783, + 0.0004994580085622147, + 0.0004958330100635067, + 0.0005487919988809153, + 0.0005245840002316982, + 0.0005045410071033984, + 0.0004995000053895637, + 0.0005002910038456321, + 0.0004947919951518998, + 0.0004938750062137842 + ] + }, + { + "forward": 0.005629541992675513, + "prefill": 0.0055919169972185045, + "decode_steps": [ + 0.0007115000043995678, + 0.0005343750090105459, + 0.0005080830014776438, + 0.0004990000015823171, + 0.0004884590016445145, + 0.000490624996018596, + 0.0004962090024491772, + 0.0005011670000385493, + 0.0006098330050008371, + 0.0006723330006934702, + 0.000654249990475364, + 0.0005632079992210492, + 0.0005214170087128878, + 0.0005090409977128729, + 0.000498583001899533, + 0.000516875006724149, + 0.0005331249994924292, + 0.0005014170019421726, + 0.000555959006305784, + 0.0005296670133247972, + 0.0005029159947298467, + 0.0005064169963588938, + 0.0005010000022593886, + 0.0005035840003984049, + 0.0004948330024490133, + 0.0004909580020466819, + 0.0004952500021317974, + 0.0004990000015823171, + 0.0004892909928457811, + 0.0005020829994464293, + 0.0004952919989591464, + 0.0004999580123694614 + ] + }, + { + "forward": 0.005778958002338186, + "prefill": 0.005564459002926014, + "decode_steps": [ + 0.0006923330074641854, + 0.0005424999981187284, + 0.0005668750090990216, + 0.0005451250035548583, + 0.0005250000103842467, + 0.000527374999364838, + 0.0005197920108912513, + 0.000524500006577, + 0.0005433749902294949, + 0.0004997079959139228, + 0.000604125001700595, + 0.0005439999949885532, + 0.0005099999980302528, + 0.0005010829918319359, + 0.0004964169929735363, + 0.0004979579971404746, + 0.0004937920020893216, + 0.0005087910103611648, + 0.0005032079934608191, + 0.0005017920047976077, + 0.0004987499996786937, + 0.0005049590108683333, + 0.0005019590025767684, + 0.0004948749992763624, + 0.000509625009726733, + 0.0005185420013731346, + 0.0004917499900329858, + 0.0004970839945599437, + 0.0005007500003557652, + 0.0006085419881856069, + 0.000584208988584578, + 0.0006084589986130595 + ] + }, + { + "forward": 0.0055280410015257075, + "prefill": 0.0055012499942677096, + "decode_steps": [ + 0.0008721250051166862, + 0.0006484579935204238, + 0.0005383749958127737, + 0.0005245419888524339, + 0.0005084580043330789, + 0.0005143339949427173, + 0.0004946670087520033, + 0.0004995000053895637, + 0.0010064160014735535, + 0.0006864999886602163, + 0.0006030830118106678, + 0.0006225420074770227, + 0.0005515420052688569, + 0.0005040420073783025, + 0.0005022090044803917, + 0.0004957920027663931, + 0.0004947499983245507, + 0.0004955420008627698, + 0.0005001669924240559, + 0.0004938329948345199, + 0.0005182500026421621, + 0.0005472500051837415, + 0.0005503749998752028, + 0.0005562079895753413, + 0.0005440420063678175, + 0.0005289999971864745, + 0.0005282499914756045, + 0.0005213330005062744, + 0.0006104159983806312, + 0.0005813329917145893, + 0.000541291999979876, + 0.0005252499977359548 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 5.791708004835527, + "p95": 6.231056402612012, + "min": 5.489125003805384, + "max": 6.28537499869708, + "iqr": 0.25951075076591223, + "mean": 5.817150100483559, + "cv": 0.04411216873887646 + }, + "prefill_ms": { + "count": 10, + "median": 5.748479001340456, + "p95": 6.065719046455342, + "min": 5.5012499942677096, + "max": 6.0824999964097515, + "iqr": 0.30438549947575666, + "mean": 5.7517919995007105, + "cv": 0.03590583935154828 + }, + "decode_ms_per_step": { + "count": 320, + "median": 0.5117289983900264, + "p95": 0.628983951173723, + "min": 0.48312499711755663, + "max": 1.0064160014735535, + "iqr": 0.036009740142617375, + "mean": 0.5313952999586036, + "cv": 0.10980444431894651 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1894.738604284662, + "p95": 1919.6460327607308, + "min": 1766.991668644447, + "max": 1922.4994763797145, + "iqr": 11.443501018218967, + "mean": 1882.8064391525998, + "cv": 0.023360757165799526 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1414.6763847244179, + "p95": 1426.6067557062584, + "min": 1355.293263739906, + "max": 1427.4748837733061, + "iqr": 22.065751785044768, + "mean": 1406.5714518238317, + "cv": 0.017067200028911742 + }, + "memory": { + "peak_extra_bytes": 12042240, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "reference", + "batch": 1, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.008326417009811848, + "prefill": 0.008399833008297719, + "decode_steps": [ + 0.000839917003759183, + 0.0006290830060606822, + 0.0006082500040065497, + 0.0005937499954598024, + 0.0005887919978704304, + 0.0005884159909328446, + 0.00059270799101796, + 0.0005911659973207861, + 0.0005863749975105748, + 0.0006108749948907644, + 0.0005986250034766272, + 0.0005959999980404973, + 0.0005954590014880523, + 0.0005789999995613471, + 0.0006017920095473528, + 0.0005911670014029369, + 0.0005910840118303895, + 0.0005886669969186187, + 0.0005850000015925616, + 0.0005863749975105748, + 0.0005844999977853149, + 0.0005958339897915721, + 0.0005847499996889383, + 0.0006129169923951849, + 0.0005941249983152375, + 0.0005914159992244095, + 0.0005870419990969822, + 0.00060299999313429, + 0.0005919579998590052, + 0.0006092909898143262, + 0.0006819580012233928, + 0.000637250006548129 + ] + }, + { + "forward": 0.008338416009792127, + "prefill": 0.00845399999525398, + "decode_steps": [ + 0.0008333330042660236, + 0.0006112909904913977, + 0.000600292012677528, + 0.0005987920012557879, + 0.0005857920041307807, + 0.0005924159922869876, + 0.0005901249969610944, + 0.0005869579908903688, + 0.0005908330058446154, + 0.0005827499990118667, + 0.0005927499878453091, + 0.0006077919970266521, + 0.0006275829946389422, + 0.0005938340036664158, + 0.0005865410057595, + 0.0005890419997740537, + 0.0006557500018971041, + 0.0005939169932389632, + 0.0005959160043857992, + 0.0005842920072609559, + 0.0005844160041306168, + 0.0005938749964116141, + 0.0005949589976808056, + 0.0005941249983152375, + 0.000590166004258208, + 0.0006000839930493385, + 0.0005914590001339093, + 0.0005874170019524172, + 0.000596332989516668, + 0.0005929579929215834, + 0.000607750000199303, + 0.0005912089982302859 + ] + }, + { + "forward": 0.008202500001061708, + "prefill": 0.008343082998180762, + "decode_steps": [ + 0.0008226250065490603, + 0.0006349169998429716, + 0.0006015419930918142, + 0.0006119580066297203, + 0.0005926250014454126, + 0.0005914170033065602, + 0.0005922499985899776, + 0.0005931250052526593, + 0.0005791669973405078, + 0.0005774169985670596, + 0.0005922499985899776, + 0.0005825420084875077, + 0.0005907500017201528, + 0.000586000009207055, + 0.000613874988630414, + 0.0005963750008959323, + 0.0005947080062469468, + 0.0006067079957574606, + 0.0005926659941906109, + 0.0005882090044906363, + 0.0005950000049779192, + 0.0005956669920124114, + 0.0005976669926894829, + 0.0006005420000292361, + 0.0005933329957770184, + 0.0006040000007487833, + 0.0005945419979980215, + 0.0005969589983578771, + 0.0006115839933045208, + 0.0005998749984428287, + 0.0005933329957770184, + 0.0006020420114509761 + ] + }, + { + "forward": 0.008312000005389564, + "prefill": 0.00822558299114462, + "decode_steps": [ + 0.0007711249927524477, + 0.0005441660032374784, + 0.0005323329969542101, + 0.0005180830048630014, + 0.0005171249940758571, + 0.0005144580063642934, + 0.0005141250003362074, + 0.0005328340048436075, + 0.0005179579893592745, + 0.0005176669947104529, + 0.0005190840020077303, + 0.0005203750042710453, + 0.0005289999971864745, + 0.0005331249994924292, + 0.0005207500071264803, + 0.0005098749970784411, + 0.0005175410042284057, + 0.0005182500026421621, + 0.0005146250041434541, + 0.000509625009726733, + 0.0005144169990671799, + 0.0005222079926170409, + 0.0005228750087553635, + 0.0005401249945862219, + 0.0005286250088829547, + 0.0005298750038491562, + 0.0005251660040812567, + 0.0005169159994693473, + 0.0005592499946942553, + 0.0005596660048468038, + 0.0005597909912467003, + 0.0005629170045722276 + ] + }, + { + "forward": 0.008687000008649193, + "prefill": 0.008212957996875048, + "decode_steps": [ + 0.0007889160042395815, + 0.0005928330065216869, + 0.000654957999358885, + 0.000673625007038936, + 0.0005789589922642335, + 0.0005878330121049657, + 0.0005382920062402263, + 0.0005252919945633039, + 0.0005168749921722338, + 0.0005217919970164075, + 0.0005135840037837625, + 0.0005152919911779463, + 0.0005157090054126456, + 0.000516875006724149, + 0.0005311249988153577, + 0.0005626670026686043, + 0.0005271249974612147, + 0.0005205409979680553, + 0.0005202920001465827, + 0.0005150829965714365, + 0.0005282499914756045, + 0.0005316250026226044, + 0.0005890419997740537, + 0.0006773329951101914, + 0.0006103339983383194, + 0.0005572919908445328, + 0.0005606250051641837, + 0.0005790419963886961, + 0.0005900830001337454, + 0.0005998749984428287, + 0.0007382499898085371, + 0.0007491669966839254 + ] + }, + { + "forward": 0.008705500003998168, + "prefill": 0.008761583012528718, + "decode_steps": [ + 0.0008089159964583814, + 0.0005990000063320622, + 0.0005982500006211922, + 0.000586042006034404, + 0.0005663329939125106, + 0.0005820840015076101, + 0.0005907919985475019, + 0.0006876249972265214, + 0.0006754160131094977, + 0.0006272079917835072, + 0.0006036669947206974, + 0.0005966250027995557, + 0.0006022919988026842, + 0.0006045829941285774, + 0.0005989580095047131, + 0.0006120839971117675, + 0.0006292909965850413, + 0.0006041669985279441, + 0.0006763749988749623, + 0.0006797919922973961, + 0.0006180829950608313, + 0.0006528749945573509, + 0.0006980409962125123, + 0.0006534170097438619, + 0.0006225000106496736, + 0.0006060000014258549, + 0.0008108749898383394, + 0.0008626659982837737, + 0.0007132920000003651, + 0.0006322080007521436, + 0.0006208749982761219, + 0.0006166249950183555 + ] + }, + { + "forward": 0.008161417004885152, + "prefill": 0.008077666003373452, + "decode_steps": [ + 0.0007761249871691689, + 0.0005431660101749003, + 0.0005248329980531707, + 0.0005295409937389195, + 0.00052375000086613, + 0.0005259579920675606, + 0.0005202499887673184, + 0.0005394580075517297, + 0.0005236669967416674, + 0.0005234580021351576, + 0.0005198339931666851, + 0.0005189170042285696, + 0.0005224169872235507, + 0.0005199169972911477, + 0.0005114999949000776, + 0.0005104169977130368, + 0.0005116250104038045, + 0.0005134999955771491, + 0.0005317920004017651, + 0.0005507919995579869, + 0.0005539999983739108, + 0.0005611249944195151, + 0.0005672080005751923, + 0.0005613340035779402, + 0.0005722920031985268, + 0.000558833999093622, + 0.000556125000002794, + 0.0005683339986717328, + 0.0005549999914364889, + 0.0005557080003200099, + 0.0005507500027306378, + 0.0005522920109797269 + ] + }, + { + "forward": 0.00889441701292526, + "prefill": 0.008516500005498528, + "decode_steps": [ + 0.0008201669988920912, + 0.0005956249951850623, + 0.0005843749968335032, + 0.0005670829996233806, + 0.0005598330026259646, + 0.0005656249995809048, + 0.0005700000037904829, + 0.0005501249979715794, + 0.0005610000080196187, + 0.000569333991734311, + 0.0005848330038134009, + 0.0005747499963035807, + 0.0005965829914202914, + 0.0005724160000681877, + 0.0005665419885190204, + 0.0005634580011246726, + 0.000570792006328702, + 0.0005582089943345636, + 0.0005648329970426857, + 0.000563749999855645, + 0.0005640419985866174, + 0.0005665839998982847, + 0.0005666250071953982, + 0.0005750830023316666, + 0.0005716250016121194, + 0.0005858749937033281, + 0.0005784579989267513, + 0.0005717499880120158, + 0.0005772079894086346, + 0.0005739170010201633, + 0.0005680419999407604, + 0.0005614999972749501 + ] + }, + { + "forward": 0.007812374999048188, + "prefill": 0.008044541988056153, + "decode_steps": [ + 0.0008329169941134751, + 0.0005472909979289398, + 0.0005332909931894392, + 0.0005515829980140552, + 0.0005077920068288222, + 0.0005274169961921871, + 0.0005216669960645959, + 0.0005659159942297265, + 0.00056487500842195, + 0.0005549579946091399, + 0.0005558330012718216, + 0.00055141698976513, + 0.0005534169904422015, + 0.0005472920020110905, + 0.0005542500002775341, + 0.0005507919995579869, + 0.0005551669892156497, + 0.0005507500027306378, + 0.0005525829910766333, + 0.0005508330068551004, + 0.000557082996238023, + 0.0005575410032179207, + 0.0005801249935757369, + 0.0005685410142177716, + 0.0005572090012719855, + 0.0005562500009546056, + 0.0005560409917961806, + 0.0005555410025408491, + 0.0005659579910570756, + 0.0005552499933401123, + 0.0006201249925652519, + 0.0006218339985935017 + ] + }, + { + "forward": 0.008404624997638166, + "prefill": 0.008379874998354353, + "decode_steps": [ + 0.0007827920053387061, + 0.0005615840054815635, + 0.0005351669969968498, + 0.0005323750083334744, + 0.0005890419997740537, + 0.000550916010979563, + 0.0005350419960450381, + 0.0005376669869292527, + 0.000582000007852912, + 0.0005314999871188775, + 0.0005286249943310395, + 0.0005155420076334849, + 0.0005332920118235052, + 0.0005166670016478747, + 0.0005196249985601753, + 0.0005234160053078085, + 0.0005150830111233518, + 0.0005132079968461767, + 0.0005130420031491667, + 0.0005081249983049929, + 0.000511959005962126, + 0.0005198340077186003, + 0.0005169579962966964, + 0.0005195420089876279, + 0.0005160830041859299, + 0.0005224579945206642, + 0.0005142500012880191, + 0.0005575000104727224, + 0.0005347910046111792, + 0.0005524159932974726, + 0.0005676659930031747, + 0.0005551250069402158 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 8.332416509801988, + "p95": 8.809404358908068, + "min": 7.812374999048188, + "max": 8.89441701292526, + "iqr": 0.3865312537527643, + "mean": 8.384466705319937, + "cv": 0.03720831931033514 + }, + "prefill_ms": { + "count": 10, + "median": 8.361478998267557, + "p95": 8.651295659365132, + "min": 8.044541988056153, + "max": 8.761583012528718, + "iqr": 0.22434400307247415, + "mean": 8.341562299756333, + "cv": 0.025661421497699153 + }, + "decode_ms_per_step": { + "count": 320, + "median": 0.574333498661872, + "p95": 0.688145797175821, + "min": 0.5077920068288222, + "max": 0.8626659982837737, + "iqr": 0.060801256040576845, + "mean": 0.5803601619390975, + "cv": 0.10653852317518404 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1727.4594764514632, + "p95": 1860.1314064533346, + "min": 1550.6162092123984, + "max": 1869.8191013553287, + "iqr": 161.6411128879522, + "mean": 1728.6620745454122, + "cv": 0.059648941806422176 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1190.1116362518385, + "p95": 1258.5740159576637, + "min": 1088.4894287111276, + "max": 1262.848595870753, + "iqr": 87.20333362516203, + "mean": 1191.4108347968838, + "cv": 0.04691165723244365 + }, + "memory": { + "peak_extra_bytes": 24559616, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "auto", + "batch": 1, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.039600500007509254 + }, + { + "forward": 0.03833154200401623 + }, + { + "forward": 0.038057458004914224 + }, + { + "forward": 0.04058479100058321 + }, + { + "forward": 0.03956108300189953 + }, + { + "forward": 0.0433708330092486 + }, + { + "forward": 0.04397912499553058 + }, + { + "forward": 0.059873750011320226 + }, + { + "forward": 0.04971108300378546 + }, + { + "forward": 0.04827783300424926 + } + ], + "forward_ms": { + "count": 10, + "median": 41.977812004915904, + "p95": 55.30054985792957, + "min": 38.057458004914224, + "max": 59.873750011320226, + "iqr": 7.632218748767627, + "mean": 44.13479980430566, + "cv": 0.15501744766635195 + }, + "memory": { + "peak_extra_bytes": 79347712, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "sdpa", + "batch": 1, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.014822291996097192, + "prefill": 0.014571082996553741, + "decode_steps": [ + 0.001072250001016073, + 0.0008472920017084107, + 0.0008132090006256476, + 0.0007951250008773059, + 0.0007706670003244653, + 0.0007654169894522056, + 0.000750000006519258, + 0.0007782089960528538, + 0.000764292009989731, + 0.0007861250051064417, + 0.0008085419976850972, + 0.0008448750013485551, + 0.0008219169976655394, + 0.0008198330033337697, + 0.0007941670046420768, + 0.0007903749938122928, + 0.0007949160062707961, + 0.0008068750030361116, + 0.0008020420063985512, + 0.000783125011366792, + 0.00079629200627096, + 0.0008418749930569902, + 0.00082654099969659, + 0.0008042080007726327, + 0.0007918330084066838, + 0.0007828749949112535, + 0.0006880829896545038, + 0.0006792080093873665, + 0.000717791001079604, + 0.0007191669865278527, + 0.0007232500065583736, + 0.000778041998273693 + ] + }, + { + "forward": 0.014298125010100193, + "prefill": 0.014652750003733672, + "decode_steps": [ + 0.001011125001241453, + 0.0008092079951893538, + 0.0007592500041937456, + 0.0007721669971942902, + 0.0007222499989438802, + 0.0008257080044131726, + 0.0008112089999485761, + 0.0007650829938938841, + 0.0007375830027740449, + 0.0007472500001313165, + 0.000746000005165115, + 0.0007508330018026754, + 0.00074562500230968, + 0.0007498749910155311, + 0.0007580000092275441, + 0.0007714999956078827, + 0.0007818339945515618, + 0.0007966249977471307, + 0.0007909159903647378, + 0.0007517909980379045, + 0.0007606660074088722, + 0.0007757919956929982, + 0.0008009169978322461, + 0.0007659169932594523, + 0.0008626670023659244, + 0.0007904999947641045, + 0.0008254999993368983, + 0.000839917003759183, + 0.0007900409982539713, + 0.0007833750132704154, + 0.000795583997387439, + 0.0007870839908719063 + ] + }, + { + "forward": 0.014378083011251874, + "prefill": 0.014019499998539686, + "decode_steps": [ + 0.0010748750064522028, + 0.0008380829967791215, + 0.0007876250019762665, + 0.0008184159960364923, + 0.000907583991647698, + 0.0008491670014336705, + 0.0008525830053258687, + 0.0007497500046156347, + 0.0007680000126129016, + 0.0007697910041315481, + 0.0008306250092573464, + 0.0007801250030752271, + 0.0008388330024899915, + 0.00083666700811591, + 0.0007730840006843209, + 0.000799749992438592, + 0.0008377079939236864, + 0.0008139580022543669, + 0.0008394159958697855, + 0.0008437910000793636, + 0.0008456659998046234, + 0.0008163329912349582, + 0.0008428750006714836, + 0.0008384170068893582, + 0.0008322919893544167, + 0.000868709001224488, + 0.0008482089906465262, + 0.0007981670059962198, + 0.0007325000042328611, + 0.0007373330008704215, + 0.0008081660052994266, + 0.000795917003415525 + ] + }, + { + "forward": 0.014864707991364412, + "prefill": 0.01455341698601842, + "decode_steps": [ + 0.0010517500049900264, + 0.0008754169975873083, + 0.000818000000435859, + 0.0007788329967297614, + 0.0007505420071538538, + 0.0007721250003669411, + 0.0007783330074744299, + 0.0007556249911431223, + 0.0007775839912937954, + 0.0007756669947411865, + 0.0007860839978093281, + 0.00081516700447537, + 0.0007653329957975075, + 0.0008049590105656534, + 0.0008307920070365071, + 0.0007671669882256538, + 0.0007961250084917992, + 0.0008017910004127771, + 0.0007690420025028288, + 0.0008015839994186535, + 0.0007999580120667815, + 0.0007806659996276721, + 0.0007820839964551851, + 0.000760082999477163, + 0.0007725420000497252, + 0.0008163329912349582, + 0.0007723750022705644, + 0.0007590840104967356, + 0.0007956659974297509, + 0.0008527080062776804, + 0.0008123339939629659, + 0.000781249997089617 + ] + }, + { + "forward": 0.025945125002181157, + "prefill": 0.02122041599068325, + "decode_steps": [ + 0.0015378749958472326, + 0.000975500006461516, + 0.0009250420116586611, + 0.000972416004515253, + 0.0009663750097388402, + 0.000909583002794534, + 0.001008250008453615, + 0.0009306659922003746, + 0.0007935410103527829, + 0.000921208004001528, + 0.0008650420058984309, + 0.0008969999908003956, + 0.0009038340067490935, + 0.0009589590044924989, + 0.0008849580044625327, + 0.0008465839928248897, + 0.0008666249923408031, + 0.000840292006614618, + 0.0008834159962134436, + 0.0009889579960145056, + 0.0010002089984482154, + 0.0009315829956904054, + 0.0008941669948399067, + 0.0008958339894888923, + 0.0009754999919096008, + 0.0008752910071052611, + 0.0008480000105919316, + 0.0008527499885531142, + 0.0008389170106966048, + 0.000858915998833254, + 0.0008621669985586777, + 0.0008732090063858777 + ] + }, + { + "forward": 0.0320007089903811, + "prefill": 0.01696691699908115, + "decode_steps": [ + 0.0014737919991603121, + 0.0010011669946834445, + 0.0009179999906336889, + 0.0009305419953307137, + 0.0008330419950652868, + 0.0010090829891851172, + 0.001039334005326964, + 0.0010659170075086877, + 0.0008867920114425942, + 0.0008723330101929605, + 0.0009118329908233136, + 0.0008646250062156469, + 0.0009602919890312478, + 0.0009472919919062406, + 0.0008803750097285956, + 0.0009682500094641, + 0.0009540419996483251, + 0.0009403750009369105, + 0.0009332909976365045, + 0.0009055420086951926, + 0.0008489580068271607, + 0.0008020840032259002, + 0.0009556670120218769, + 0.0010142499959329143, + 0.000992457993561402, + 0.0012800410040654242, + 0.0009209999989252537, + 0.0008994579984573647, + 0.0007654999935766682, + 0.0008079999970505014, + 0.000935500007472001, + 0.0009955420100595802 + ] + }, + { + "forward": 0.016173166994121857, + "prefill": 0.016087916999822482, + "decode_steps": [ + 0.0012113749980926514, + 0.000945792009588331, + 0.00108712499786634, + 0.001142374996561557, + 0.0009396250097779557, + 0.0008846249984344468, + 0.0009182909998344257, + 0.0009042909950949252, + 0.0008611660014139488, + 0.0009082079923246056, + 0.0008680419996380806, + 0.0008592499943915755, + 0.0008389589929720387, + 0.0008892499899957329, + 0.0008362079970538616, + 0.0007819579914212227, + 0.0009037090057972819, + 0.0008496249938616529, + 0.0008291249978356063, + 0.0008758330077398568, + 0.0008303329959744588, + 0.0008658750011818483, + 0.0008828330028336495, + 0.0009668749989941716, + 0.0010699170088628307, + 0.0008182919991668314, + 0.0008866250136634335, + 0.0008763749938225374, + 0.0008655830024508759, + 0.0008746669918764383, + 0.0009335000067949295, + 0.0009017499978654087 + ] + }, + { + "forward": 0.01593545799551066, + "prefill": 0.017019708000589162, + "decode_steps": [ + 0.0013246660091681406, + 0.0010764589969767258, + 0.0010262080031679943, + 0.0010397500009275973, + 0.000953249997110106, + 0.0009371249907417223, + 0.0009560000034980476, + 0.0009717500070109963, + 0.0009101670002564788, + 0.0009085000056074932, + 0.0009558330057188869, + 0.0009572910057613626, + 0.0008888340089470148, + 0.0009374999935971573, + 0.000923707993933931, + 0.0009542920015519485, + 0.0009563750063534826, + 0.0009103339980356395, + 0.0009099160088226199, + 0.0009891249937936664, + 0.0008761670032981783, + 0.000966959007200785, + 0.0009512079996056855, + 0.0009480000007897615, + 0.0009019169956445694, + 0.000924250009120442, + 0.0010164999985136092, + 0.0009850830101640895, + 0.0009626249957364053, + 0.0010325420007575303, + 0.0009708750003483146, + 0.0009156250016530976 + ] + }, + { + "forward": 0.016494667012011632, + "prefill": 0.015208541008178145, + "decode_steps": [ + 0.0011976249952567741, + 0.0009259579965146258, + 0.0009600419871276245, + 0.000951916998019442, + 0.0008592919912189245, + 0.0008459999953629449, + 0.0008357920014532283, + 0.0008260829927166924, + 0.0008480840042466298, + 0.0008184579928638414, + 0.0008277920132968575, + 0.0008404589898418635, + 0.0008247920050052926, + 0.0008235830027842894, + 0.0008294580038636923, + 0.000823291004053317, + 0.0008273329876828939, + 0.0008395829936489463, + 0.0008217079885071144, + 0.0008451250032521784, + 0.0008393750031245872, + 0.000818041997263208, + 0.0009106669895118102, + 0.0008308329997817054, + 0.0008278750028694049, + 0.0008794999885139987, + 0.0008775409951340407, + 0.0008458340016659349, + 0.0008315829909406602, + 0.0008242909971158952, + 0.0008735419978620484, + 0.0008956250094342977 + ] + }, + { + "forward": 0.015098625008249655, + "prefill": 0.017720458999974653, + "decode_steps": [ + 0.0012220420030644163, + 0.0009023329912452027, + 0.0008705419895704836, + 0.0008445419953204691, + 0.0008333750010933727, + 0.0008263749914476648, + 0.000840292006614618, + 0.000825084003736265, + 0.0008227499929489568, + 0.0008416250057052821, + 0.0008190410007955506, + 0.0008542080031475052, + 0.0008285420044558123, + 0.0008233749977080151, + 0.0008202919998439029, + 0.0008170410001184791, + 0.0008229580125771463, + 0.0008236669964389876, + 0.0008168750064214692, + 0.0008304579969262704, + 0.0008371669973712415, + 0.0008233340049628168, + 0.0008454170019831508, + 0.0008145419997163117, + 0.0008277090091723949, + 0.0009239159990102053, + 0.0008537920075468719, + 0.0008389579888898879, + 0.0008218330040108413, + 0.0008169579959940165, + 0.0008402499952353537, + 0.0008423749968642369 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 15.517041501880158, + "p95": 29.27569619569112, + "min": 14.298125010100193, + "max": 32.0007089903811, + "iqr": 1.5813960126251914, + "mean": 18.001095901126973, + "cv": 0.3334048703919425 + }, + "prefill_ms": { + "count": 10, + "median": 15.648229004000314, + "p95": 19.645435344864378, + "min": 14.019499998539686, + "max": 21.22041599068325, + "iqr": 2.415010501863435, + "mean": 16.202070798317436, + "cv": 0.1337833540133545 + }, + "decode_ms_per_step": { + "count": 320, + "median": 0.8402710009249859, + "p95": 1.0393548051069956, + "min": 0.6792080093873665, + "max": 1.5378749958472326, + "iqr": 0.10879150067921728, + "mean": 0.8669178564559843, + "cv": 0.12199880673025669 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1167.9905024359778, + "p95": 1266.776461926482, + "min": 1034.2988344144055, + "max": 1270.7594450788683, + "iqr": 156.27197104817174, + "mean": 1159.812284523018, + "cv": 0.07727232171280791 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 729.9308148131845, + "p95": 802.4632089624561, + "min": 629.8830325857172, + "max": 803.3229053997505, + "iqr": 113.92641095102101, + "mean": 733.3705897455326, + "cv": 0.08742795129886531 + }, + "memory": { + "peak_extra_bytes": 22609920, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "reference", + "batch": 1, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.03584962499735411, + "prefill": 0.03557387499313336, + "decode_steps": [ + 0.0011434170010033995, + 0.0009352920023957267, + 0.0008957079990068451, + 0.0008931250049499795, + 0.0009020419965963811, + 0.0008760420023463666, + 0.0009164160001091659, + 0.0008863749972078949, + 0.0008870409947121516, + 0.0008704999927431345, + 0.0008243330084951594, + 0.0008404160034842789, + 0.0008791250002104789, + 0.0009227500122506171, + 0.0008731249981792644, + 0.0009045410115504637, + 0.0008971250063041225, + 0.0009052079985849559, + 0.0008917910017771646, + 0.0009072499960893765, + 0.0008709579997230321, + 0.0008795000030659139, + 0.0009205000096699223, + 0.0008931250049499795, + 0.0008815000037429854, + 0.0009005839965539053, + 0.0008600000001024455, + 0.0008603330061305314, + 0.0008859579975251108, + 0.0008125830063363537, + 0.0008618749998277053, + 0.000904333996004425 + ] + }, + { + "forward": 0.035295124995172955, + "prefill": 0.03478362500027288, + "decode_steps": [ + 0.0011081249976996332, + 0.0008827499987091869, + 0.0008338750049006194, + 0.0008382910018553957, + 0.0008148329943651333, + 0.0008385830005863681, + 0.0008337079925695434, + 0.0008286249940283597, + 0.0008615420083515346, + 0.0008555409876862541, + 0.0008440000092377886, + 0.0008507079910486937, + 0.0008640840096632019, + 0.0008519999973941594, + 0.0008547089964849874, + 0.0007951659936225042, + 0.0008073339995462447, + 0.0008216670103138313, + 0.0007977920031407848, + 0.0008133329974953085, + 0.0008463330013910308, + 0.0008481249969918281, + 0.0008613749960204586, + 0.0008420829981332645, + 0.000824499991722405, + 0.0008443330007139593, + 0.0008515840017935261, + 0.0008497079979861155, + 0.0008120419952319935, + 0.0008344170055352151, + 0.0009463329915888608, + 0.0008659159939270467 + ] + }, + { + "forward": 0.035916791995987296, + "prefill": 0.0362083329964662, + "decode_steps": [ + 0.001174167002318427, + 0.0009642090008128434, + 0.0009458330023335293, + 0.0008929579926189035, + 0.0009057499992195517, + 0.0009862919978331774, + 0.001113958001951687, + 0.001006791993859224, + 0.000927707995288074, + 0.0008636669954285026, + 0.0009183329966617748, + 0.0008589170029154047, + 0.000846957991598174, + 0.000917958008358255, + 0.0009047079947777092, + 0.0008479579992126673, + 0.0008978330006357282, + 0.0009017499978654087, + 0.000895666002179496, + 0.0008349169947905466, + 0.0008198750001611188, + 0.0008019590022740886, + 0.0008319160115206614, + 0.0008480420074192807, + 0.0009322080004494637, + 0.0008666250068927184, + 0.0008238330046879128, + 0.0008361250074813142, + 0.0008693330019013956, + 0.0008112079958664253, + 0.0007272910006577149, + 0.0007391660037683323 + ] + }, + { + "forward": 0.037501917002373375, + "prefill": 0.0374584159872029, + "decode_steps": [ + 0.0011941249977098778, + 0.00102508399868384, + 0.0009618330077501014, + 0.0009458749991608784, + 0.0009126249933615327, + 0.0009158329921774566, + 0.0009058750001713634, + 0.0009188330004690215, + 0.0009524170018266886, + 0.0008932909986469895, + 0.000948874992900528, + 0.0009540409955661744, + 0.0009410000056959689, + 0.0008747919928282499, + 0.0008574999956181273, + 0.0008637499995529652, + 0.000971125002251938, + 0.0009660829964559525, + 0.0009100830066017807, + 0.0008775000023888424, + 0.0009251250012312084, + 0.0009885830077109858, + 0.0009869579953374341, + 0.0009659999923314899, + 0.0012724169937428087, + 0.00106470899481792, + 0.001105957999243401, + 0.0009050830121850595, + 0.0008628340001450852, + 0.0008682500047143549, + 0.0008867500000633299, + 0.0009830420021899045 + ] + }, + { + "forward": 0.038223667012061924, + "prefill": 0.03897950000828132, + "decode_steps": [ + 0.0013887080131098628, + 0.0010830840037669986, + 0.0009967079968191683, + 0.0009805830050026998, + 0.0009086250065593049, + 0.0008402499952353537, + 0.0009022080048453063, + 0.0009072079992620274, + 0.0009120830072788522, + 0.0008637499995529652, + 0.0009284999978262931, + 0.0009909170039463788, + 0.0009766250004759058, + 0.0010054579906864092, + 0.001011542000924237, + 0.0009767500014277175, + 0.0010034169972641394, + 0.0009629170090192929, + 0.00090470899885986, + 0.0009408750047441572, + 0.0009356669906992465, + 0.000921042010304518, + 0.0009373329958179966, + 0.0009233339951606467, + 0.0009397920075571164, + 0.0008975000091595575, + 0.0009504589979769662, + 0.0008793750021141022, + 0.0009728329896461219, + 0.0009290000016335398, + 0.0009255420009139925, + 0.0009634579910198227 + ] + }, + { + "forward": 0.03872599999886006, + "prefill": 0.039566999999806285, + "decode_steps": [ + 0.0012975830031791702, + 0.0010427500092191622, + 0.001229458997840993, + 0.001152708995505236, + 0.0012193750008009374, + 0.00124004200915806, + 0.0011505419970490038, + 0.0010335840051993728, + 0.0009359579998999834, + 0.0009881250007310882, + 0.0011380000069038942, + 0.001093041995773092, + 0.0010364589979872108, + 0.000993207999272272, + 0.0009942920005414635, + 0.0010122079984284937, + 0.0011115829984191805, + 0.001232583003002219, + 0.0011822920059785247, + 0.0011805829999502748, + 0.0010881250054808334, + 0.0010568750003585592, + 0.0010906250099651515, + 0.0009863340092124417, + 0.0010916660103248432, + 0.001206332992296666, + 0.001304083998547867, + 0.0010757499985629693, + 0.0010633330093696713, + 0.0010249170009046793, + 0.0012579589965753257, + 0.0011435839987825602 + ] + }, + { + "forward": 0.038703999991412275, + "prefill": 0.03890841698739678, + "decode_steps": [ + 0.0014831250009592623, + 0.0011150409991387278, + 0.0010881659982260317, + 0.0010867910023080185, + 0.0010725419997470453, + 0.0010651249904185534, + 0.0010512500011827797, + 0.0011043330014217645, + 0.0010665419977158308, + 0.0010830419923877344, + 0.001056250010151416, + 0.0010396669968031347, + 0.00100637499417644, + 0.001008250008453615, + 0.000948499990045093, + 0.0008862919930834323, + 0.0009748330048751086, + 0.0009862909937510267, + 0.0010184159909840673, + 0.000991334003629163, + 0.0010456659947521985, + 0.0010011249978560954, + 0.0009697079949546605, + 0.0010352499957662076, + 0.001007707993267104, + 0.0010288330086041242, + 0.0009767920128069818, + 0.0010448749962961301, + 0.0009618749900255352, + 0.0008797089976724237, + 0.0008792500011622906, + 0.0008811250008875504 + ] + }, + { + "forward": 0.046334374987054616, + "prefill": 0.05381804099306464, + "decode_steps": [ + 0.0016715419915271923, + 0.0011952089989790693, + 0.0012355409999145195, + 0.0011586249893298373, + 0.001164667002740316, + 0.0011959160037804395, + 0.0012037080014124513, + 0.0012517080031102523, + 0.0011910839966731146, + 0.001659583009313792, + 0.0011816250043921173, + 0.0012619589979294688, + 0.0015669170097680762, + 0.0013685829908354208, + 0.0016970419965218753, + 0.0012621249916264787, + 0.0012280420050956309, + 0.0012037910055369139, + 0.0012020829890388995, + 0.0011200000008102506, + 0.0011428330035414547, + 0.001254500006325543, + 0.0013250839983811602, + 0.0011752090067602694, + 0.0018306249985471368, + 0.0012360830005491152, + 0.0011988330079475418, + 0.001243457998498343, + 0.001221375001478009, + 0.0012615000014193356, + 0.0022069170081522316, + 0.0026988750032614917 + ] + }, + { + "forward": 0.04743725000298582, + "prefill": 0.04115779200219549, + "decode_steps": [ + 0.0015156659937929362, + 0.0012243749952176586, + 0.0011499590036692098, + 0.0011206669878447428, + 0.0011037919903174043, + 0.0010827079968294129, + 0.0010979170037899166, + 0.0011157919943798333, + 0.0011024160048691556, + 0.0011120830022264272, + 0.0010485829989193007, + 0.0009797919919947162, + 0.0009830000053625554, + 0.000976666997303255, + 0.0009813329961616546, + 0.0009710829908726737, + 0.0010266250028507784, + 0.0010135840129805729, + 0.0009715829946799204, + 0.0009737080108607188, + 0.000977582996711135, + 0.000974750000750646, + 0.0009845409949775785, + 0.0009846659959293902, + 0.0009819590050028637, + 0.0009845829918049276, + 0.0009714580082800239, + 0.0009785419970285147, + 0.0010542500094743446, + 0.0009849160123849288, + 0.0010414999997010455, + 0.001014708002912812 + ] + }, + { + "forward": 0.04286058299476281, + "prefill": 0.045661500000278465, + "decode_steps": [ + 0.0014148329937597737, + 0.0012161250051576644, + 0.0012376249942462891, + 0.0011328749969834462, + 0.0011102080025011674, + 0.0011470829922473058, + 0.001109750010073185, + 0.0010677079990273342, + 0.0010598340013530105, + 0.001156291997176595, + 0.001144041001680307, + 0.0010998340003425255, + 0.001153916004113853, + 0.0011431659950176254, + 0.0011712499981513247, + 0.0011003330000676215, + 0.0011331669957144186, + 0.0012180420017102733, + 0.0011972909996984527, + 0.001199457998154685, + 0.0011354159942129627, + 0.0012212080036988482, + 0.0012024580064462498, + 0.001245416991878301, + 0.0012175419979030266, + 0.001253000009455718, + 0.0012011250073555857, + 0.0011649579973891377, + 0.0011367920087650418, + 0.0011912080080946907, + 0.0011877919896505773, + 0.001179124999907799 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 38.4638335017371, + "p95": 46.94095624581678, + "min": 35.295124995172955, + "max": 47.43725000298582, + "iqr": 5.5138639982033055, + "mean": 39.684933397802524, + "cv": 0.10995311439052963 + }, + "prefill_ms": { + "count": 10, + "median": 38.94395849783905, + "p95": 50.147597546310855, + "min": 34.78362500027288, + "max": 53.81804099306464, + "iqr": 4.239240257447818, + "mean": 40.21164989680983, + "cv": 0.14212065705316196 + }, + "decode_ms_per_step": { + "count": 320, + "median": 0.9824795051827095, + "p95": 1.2626395917322955, + "min": 0.7272910006577149, + "max": 2.6988750032614917, + "iqr": 0.2304694899066817, + "mean": 1.0279170220655942, + "cv": 0.19578881971801937 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1008.9984685660693, + "p95": 1151.518906786737, + "min": 730.3427891968629, + "max": 1175.45870153798, + "iqr": 188.99680416634487, + "mean": 991.6543738297403, + "cv": 0.1389985374479586 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 452.74258601534507, + "p95": 508.2890210177915, + "min": 327.7577539436082, + "max": 516.0704336255791, + "iqr": 62.11317438332958, + "mean": 445.13438301003873, + "cv": 0.12812711985482955 + }, + "memory": { + "peak_extra_bytes": 69599232, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "auto", + "batch": 1, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.17134850000729784 + }, + { + "forward": 0.1713740409904858 + }, + { + "forward": 0.17919650000112597 + }, + { + "forward": 0.19249099999433383 + }, + { + "forward": 0.18147179200605024 + }, + { + "forward": 0.22903712501283735 + }, + { + "forward": 0.20669416600139812 + }, + { + "forward": 0.19295299999066629 + }, + { + "forward": 0.22518033398955595 + }, + { + "forward": 0.20014912499755155 + } + ], + "forward_ms": { + "count": 10, + "median": 192.72199999250006, + "p95": 227.30156905236072, + "min": 171.34850000729784, + "max": 229.03712501283735, + "iqr": 25.29258274807944, + "mean": 194.9895582991303, + "cv": 0.1052896870021838 + }, + "memory": { + "peak_extra_bytes": 267763712, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "sdpa", + "batch": 1, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.03986883400648367, + "prefill": 0.038592541008256376, + "decode_steps": [ + 0.0015496669948333874, + 0.0011761670029954985, + 0.0011750840058084577, + 0.0011268329981248826, + 0.001222374994540587, + 0.0011265409993939102, + 0.0011197499989066273, + 0.0011092920030932873, + 0.0010996250057360157, + 0.0010994170006597415, + 0.0011155410029459745, + 0.0011070829932577908, + 0.001110375000280328, + 0.001142540990258567, + 0.0011995410022791475, + 0.0012431659997673705, + 0.001183041007607244, + 0.0011276659934083, + 0.0012503750040195882, + 0.0013156249915482476, + 0.0011825410037999973, + 0.001106291005271487, + 0.0011138749978272244, + 0.0011101669952040538, + 0.0012057499989168718, + 0.0011175419931532815, + 0.0011110409977845848, + 0.0011185000039404258, + 0.0011542080028448254, + 0.0011070839973399416, + 0.001097957996535115, + 0.0010987910063704476 + ] + }, + { + "forward": 0.03915891698852647, + "prefill": 0.04199129200424068, + "decode_steps": [ + 0.001776291013811715, + 0.0013515419996110722, + 0.0012262079981155694, + 0.0011897090007551014, + 0.0011864580010296777, + 0.0012045420007780194, + 0.0013765829935437068, + 0.0012515830021584406, + 0.001175416007754393, + 0.0012421249994076788, + 0.0011792500008596107, + 0.0011680000025080517, + 0.001222708000568673, + 0.0012172909919172525, + 0.0011755830055335537, + 0.0011852500028908253, + 0.0011785419919760898, + 0.001240415993379429, + 0.0012173340073786676, + 0.0014034159976290539, + 0.001259124997886829, + 0.0011702079937094823, + 0.0012896249972982332, + 0.001375874999212101, + 0.0012583750067278743, + 0.0011847080022562295, + 0.001174749995698221, + 0.0011665830097626895, + 0.001160625004558824, + 0.00117816700367257, + 0.0011733330029528588, + 0.0011672079999698326 + ] + }, + { + "forward": 0.05293445799907204, + "prefill": 0.04686741600744426, + "decode_steps": [ + 0.001814709001337178, + 0.0013586249988293275, + 0.0014492079935735092, + 0.0013744580064667389, + 0.0012414170050760731, + 0.0012643329973798245, + 0.0013254160003270954, + 0.0013706250028917566, + 0.0015711250016465783, + 0.0016670000040903687, + 0.001412249999702908, + 0.0014164169988362119, + 0.001348417004919611, + 0.0013291249924805015, + 0.00161179099814035, + 0.00131050001073163, + 0.0012862080038757995, + 0.0014440829982049763, + 0.0013563329994212836, + 0.0014762920036446303, + 0.001978457992663607, + 0.0014736250013811514, + 0.0013737080007558689, + 0.0013069999986328185, + 0.001317291011218913, + 0.0014187089982442558, + 0.0014915000065229833, + 0.0012819580006180331, + 0.0013373330002650619, + 0.0014203750033630058, + 0.0013043749931966886, + 0.001310125007876195 + ] + }, + { + "forward": 0.04471491600270383, + "prefill": 0.042307041003368795, + "decode_steps": [ + 0.0016800830053398386, + 0.001245750012458302, + 0.0013295000098878518, + 0.0012188330001663417, + 0.001204832995426841, + 0.001281207994907163, + 0.0013122500095050782, + 0.0012123339984100312, + 0.0012099169980501756, + 0.0012637090112548321, + 0.0012395410012686625, + 0.0013484999944921583, + 0.0013460839982144535, + 0.0012418749975040555, + 0.0014736249868292361, + 0.0016897919995244592, + 0.0013579170044977218, + 0.0013828340015606955, + 0.0013496659958036616, + 0.0012953749974258244, + 0.0013149579899618402, + 0.0013704169978154823, + 0.0012836249952670187, + 0.0013625830033561215, + 0.0012203340011183172, + 0.0012277920031920075, + 0.0012289169972063974, + 0.001198333004140295, + 0.0013770000077784061, + 0.001224249994265847, + 0.001200667000375688, + 0.0012026670010527596 + ] + }, + { + "forward": 0.04465758299920708, + "prefill": 0.043720375004340895, + "decode_steps": [ + 0.0018870409985538572, + 0.00137870799517259, + 0.0014763329963898286, + 0.0014583750016754493, + 0.001348042002064176, + 0.001540792000014335, + 0.0014173750096233562, + 0.0012794590002158657, + 0.0012637079926207662, + 0.0012024169991491362, + 0.0011849580041598529, + 0.0011933329951716587, + 0.0012313330080360174, + 0.001183833010145463, + 0.0012921659945277497, + 0.0013397919974522665, + 0.00137779199576471, + 0.0012099159939680248, + 0.0016621660033706576, + 0.0014684999914607033, + 0.0013525410031434149, + 0.0013254580117063597, + 0.001278333002119325, + 0.001341582988970913, + 0.0012468749919207767, + 0.0012068749929312617, + 0.0012624579976545647, + 0.0014154170057736337, + 0.0013151659950381145, + 0.0012064589973306283, + 0.0015624579973518848, + 0.0016304579912684858 + ] + }, + { + "forward": 0.0590481249964796, + "prefill": 0.0810206250025658, + "decode_steps": [ + 0.0070367919979617, + 0.00287762499647215, + 0.004239416011841968, + 0.0036536249972414225, + 0.0021129580127308145, + 0.0030585419881390408, + 0.0015720829978818074, + 0.0017604580061743036, + 0.0027581250033108518, + 0.002994541995576583, + 0.0018683340022107586, + 0.0028514579898910597, + 0.0015391670021926984, + 0.0013932919973740354, + 0.0019318749982630834, + 0.0023569999902974814, + 0.0017444999975850806, + 0.0021454590023495257, + 0.002544457995099947, + 0.0044847499957541, + 0.0016812079993542284, + 0.0017949169996427372, + 0.00159983299090527, + 0.001470541988965124, + 0.0017214160034200177, + 0.0016147080023074523, + 0.0016486669919686392, + 0.005322084005456418, + 0.0018936669948743656, + 0.003409541997825727, + 0.00213274999987334, + 0.0022166249982547015 + ] + }, + { + "forward": 0.046131207986036316, + "prefill": 0.07915591601340566, + "decode_steps": [ + 0.006905207992531359, + 0.004335000005085021, + 0.004296625003917143, + 0.007186791001004167, + 0.00918900000397116, + 0.002317250007763505, + 0.001719790991046466, + 0.0016057920001912862, + 0.0014647089992649853, + 0.0014115000085439533, + 0.0014070830075070262, + 0.0015347919979831204, + 0.0013614170020446181, + 0.0015658750053262338, + 0.0016708339971955866, + 0.0014950000040698797, + 0.0013982499949634075, + 0.0015338749944930896, + 0.0014074170030653477, + 0.0029224999889265746, + 0.0016268340114038438, + 0.0015563749911962077, + 0.0015199999907054007, + 0.0015580420003971085, + 0.00767845798691269, + 0.0026836249890038744, + 0.001673291000770405, + 0.0015094999980647117, + 0.0014439579972531646, + 0.0016307920013787225, + 0.002770708000753075, + 0.003365374999702908 + ] + }, + { + "forward": 0.04843670800619293, + "prefill": 0.045777040999382734, + "decode_steps": [ + 0.0018967079959111288, + 0.001469917013309896, + 0.001378917004331015, + 0.0013442500057863072, + 0.0013208330055931583, + 0.001322916999924928, + 0.001328042009845376, + 0.0013009579997742549, + 0.00136749999364838, + 0.00136920899967663, + 0.001343416006420739, + 0.001295791007578373, + 0.0013103749952279031, + 0.0013279580016387627, + 0.001339250011369586, + 0.0013407499936874956, + 0.0013752090017078444, + 0.0013224170106695965, + 0.0013412080006673932, + 0.0013310000067576766, + 0.0013340000004973263, + 0.0014655839913757518, + 0.0013302080042194575, + 0.0013715829991269857, + 0.0013867499947082251, + 0.0013395419955486432, + 0.0013288750051287934, + 0.001411082994309254, + 0.00170208299823571, + 0.0016429579991381615, + 0.0016890830011107028, + 0.0014620000001741573 + ] + }, + { + "forward": 0.04572537499188911, + "prefill": 0.04567670800315682, + "decode_steps": [ + 0.001717917009955272, + 0.0013005410000914708, + 0.0012875410029664636, + 0.001455833000363782, + 0.00137012499908451, + 0.001345374999800697, + 0.001294583998969756, + 0.0012732920004054904, + 0.001294749992666766, + 0.0012646670074900612, + 0.0012188749969936907, + 0.0012013329978799447, + 0.00127704199985601, + 0.0013337920099729672, + 0.0013297499972395599, + 0.0012044169998262078, + 0.0014047499862499535, + 0.0012225410027895123, + 0.0012101660104235634, + 0.0012465000036172569, + 0.0013259170082164928, + 0.0012235839967615902, + 0.0012197920004837215, + 0.001231833011843264, + 0.0012409159971866757, + 0.0013915829913457856, + 0.001807583001209423, + 0.0013091669970890507, + 0.0014576249959645793, + 0.001389333003317006, + 0.0012912919919472188, + 0.0015050420042825863 + ] + }, + { + "forward": 0.09285512499627657, + "prefill": 0.05791570899600629, + "decode_steps": [ + 0.002329459006432444, + 0.0016091249999590218, + 0.0014684159978060052, + 0.001686667004832998, + 0.0016085420065792277, + 0.0015330410096794367, + 0.0017524160066386685, + 0.0015077080024639145, + 0.0014418749924516305, + 0.0014418750070035458, + 0.0014093749923631549, + 0.0013932919973740354, + 0.0014756250020582229, + 0.0015394170040963218, + 0.0014770410052733496, + 0.0013898329925723374, + 0.0013776249979855493, + 0.0013928749976912513, + 0.0013412909902399406, + 0.0014439580118050799, + 0.0013858750025974587, + 0.0013955000031273812, + 0.0014314580039354041, + 0.001353416999336332, + 0.0014503749989671633, + 0.0014857500063953921, + 0.001564083999255672, + 0.0014077079977141693, + 0.0013365829945541918, + 0.001480000006267801, + 0.0013452080020215362, + 0.0015203749935608357 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 45.928291488962714, + "p95": 77.6419749963679, + "min": 39.15891698852647, + "max": 92.85512499627657, + "iqr": 7.1381042507709935, + "mean": 51.35312489728676, + "cv": 0.30590875320518235 + }, + "prefill_ms": { + "count": 10, + "median": 45.726874501269776, + "p95": 80.18150595744373, + "min": 38.592541008256376, + "max": 81.0206250025658, + "iqr": 12.493261245253962, + "mean": 52.30246640421683, + "cv": 0.29628211210921557 + }, + "decode_ms_per_step": { + "count": 320, + "median": 1.3529790012398735, + "p95": 2.8798687460948718, + "min": 1.097957996535115, + "max": 9.18900000397116, + "iqr": 0.28178099819342606, + "mean": 1.5944314563967055, + "cv": 0.5871207000123585 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 723.901019688961, + "p95": 836.5692739875992, + "min": 373.19670052591687, + "max": 861.9887813225026, + "iqr": 82.05083916992476, + "mean": 676.9588526345281, + "cv": 0.24225290161714616 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 357.6200071004535, + "p95": 408.66309108479805, + "min": 194.0551413746663, + "max": 422.63197391809496, + "iqr": 62.967382016237025, + "mean": 331.6680359079126, + "cv": 0.23561570414198413 + }, + "memory": { + "peak_extra_bytes": 38879232, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "reference", + "batch": 1, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.15172779200656805, + "prefill": 0.1357118750020163, + "decode_steps": [ + 0.0019843339978251606, + 0.0015329579910030589, + 0.0014991670032031834, + 0.0014817920018685982, + 0.0014667500072391704, + 0.0014807499974267557, + 0.001545708000776358, + 0.0018509159999666736, + 0.0016389579977840185, + 0.0014969579933676869, + 0.001499292004154995, + 0.0014964580041123554, + 0.0015135830035433173, + 0.0016242919955402613, + 0.001507250010035932, + 0.0014975000085541978, + 0.0014842910022707656, + 0.0019972080044681206, + 0.002245166993816383, + 0.0018090420053340495, + 0.001773541996954009, + 0.0016821250028442591, + 0.0017192920058732852, + 0.001718458006507717, + 0.0016816669958643615, + 0.0016230000037467107, + 0.0020402499940246344, + 0.0018080409936374053, + 0.0018831670022336766, + 0.0016376249986933544, + 0.0016226669977186248, + 0.001636000000871718 + ] + }, + { + "forward": 0.1535294169880217, + "prefill": 0.13406374999613035, + "decode_steps": [ + 0.0019927089888369665, + 0.0015492919919779524, + 0.001491459013777785, + 0.0014964169968152419, + 0.001455499994335696, + 0.0014937910018488765, + 0.0015162920026341453, + 0.0015097089926712215, + 0.0018607079982757568, + 0.0015558750019408762, + 0.001415416001691483, + 0.001515666997875087, + 0.0015223749942379072, + 0.001414749989635311, + 0.0013835420104442164, + 0.0013749579957220703, + 0.0015121250035008416, + 0.0017052500043064356, + 0.0016534159949515015, + 0.0017554579972056672, + 0.0015375829971162602, + 0.0014199580036802217, + 0.0014750830014236271, + 0.0013986249978188425, + 0.0013828340015606955, + 0.0015779169916640967, + 0.0016042909992393106, + 0.0016614159976597875, + 0.001665500007220544, + 0.0016652920021442696, + 0.0020692079997388646, + 0.0018537500000093132 + ] + }, + { + "forward": 0.1841942500032019, + "prefill": 0.16719662499963306, + "decode_steps": [ + 0.0021067920024506748, + 0.0015816660015843809, + 0.0017434999899705872, + 0.001982333997148089, + 0.0016890409897314385, + 0.0017343750077998266, + 0.0017992920038523152, + 0.0020537500095088035, + 0.0016847079969011247, + 0.0015553749981336296, + 0.0015883750020293519, + 0.001510500005679205, + 0.0015010420029284433, + 0.0017252079996978864, + 0.0015739999944344163, + 0.001731208001729101, + 0.0015598750032950193, + 0.001504041996668093, + 0.0015885409957263619, + 0.0014959580003051087, + 0.0014557919930666685, + 0.001374166997266002, + 0.0013947090046713129, + 0.0013901249913033098, + 0.0016186249995371327, + 0.0014490000030491501, + 0.0015901250008028, + 0.001527583008282818, + 0.001489333008066751, + 0.0015215419989544898, + 0.001478500009397976, + 0.001496000011684373 + ] + }, + { + "forward": 0.15440270799444988, + "prefill": 0.14228020800510421, + "decode_steps": [ + 0.0021069590002298355, + 0.0015756669890834019, + 0.001903166004922241, + 0.00189370900625363, + 0.0015375410002889112, + 0.0015626249951310456, + 0.0015098750009201467, + 0.001483667001593858, + 0.001601540992851369, + 0.0014989589981269091, + 0.0014786249957978725, + 0.001453959004720673, + 0.0014502919948427007, + 0.0014685839996673167, + 0.0015084580081747845, + 0.0015252500015776604, + 0.0015075830015121028, + 0.0014912079932400957, + 0.0015549579984508455, + 0.0015911670052446425, + 0.0016107500123325735, + 0.0016107080009533092, + 0.0015824580041226, + 0.0016060420020949095, + 0.0015912079979898408, + 0.001575791000504978, + 0.0016386250063078478, + 0.0016043749928940088, + 0.0016266250022454187, + 0.001702875000773929, + 0.0016847499937284738, + 0.0018905000033555552 + ] + }, + { + "forward": 0.15833091699460056, + "prefill": 0.14168800000334159, + "decode_steps": [ + 0.0020133330108365044, + 0.0016222080012084916, + 0.0015210410056170076, + 0.0014803749945713207, + 0.0014947089948691428, + 0.0014720829931320623, + 0.0014753750001545995, + 0.001435499987564981, + 0.0014462079998338595, + 0.0016314580134348944, + 0.0014558750117430463, + 0.0015198750043055043, + 0.0014495419891318306, + 0.0015002919972175732, + 0.0017545000009704381, + 0.001706875002128072, + 0.001811624999390915, + 0.0016969999996945262, + 0.00165200000628829, + 0.0016816669958643615, + 0.00174170799436979, + 0.0016674579965183511, + 0.0016577920032432303, + 0.0017360839992761612, + 0.001865124999312684, + 0.0018370840116403997, + 0.002071374998195097, + 0.002097667005727999, + 0.0017484580021118745, + 0.00175104099616874, + 0.0016826669889269397, + 0.001657583998166956 + ] + }, + { + "forward": 0.13134112500119954, + "prefill": 0.13505199999781325, + "decode_steps": [ + 0.0019774589891312644, + 0.0015350419998867437, + 0.001524541003163904, + 0.0014918750093784183, + 0.001527832995634526, + 0.0018136250000679865, + 0.0016277079994324595, + 0.0015286669950000942, + 0.0014788339904043823, + 0.0014565419987775385, + 0.0014126669993856922, + 0.0013783750036964193, + 0.0014067500014789402, + 0.0013819589948980138, + 0.0013835419958923012, + 0.0014083749993005767, + 0.0013878750032745302, + 0.0013666669983649626, + 0.0013754169922322035, + 0.001367250006296672, + 0.0013833749981131405, + 0.00133908299903851, + 0.001494458003435284, + 0.001439916988601908, + 0.0014643750037066638, + 0.0014087080053286627, + 0.0013959999923827127, + 0.0015917499986244366, + 0.0019915000011678785, + 0.0015895830001682043, + 0.0014981670101406053, + 0.0015137089940253645 + ] + }, + { + "forward": 0.19043274999421556, + "prefill": 0.17751879199931864, + "decode_steps": [ + 0.0025505000085104257, + 0.0021702090016333386, + 0.0024933330132625997, + 0.0021767920115962625, + 0.0023353339929599315, + 0.0024733330064918846, + 0.002081583996186964, + 0.0020427919953363016, + 0.0020538749959087, + 0.0023939579987199977, + 0.002937416997156106, + 0.002457458002027124, + 0.0019189589947927743, + 0.001819583005271852, + 0.0018977080035256222, + 0.0019144160032738, + 0.001959750006790273, + 0.0024879169941414148, + 0.0019490419945213944, + 0.0026677910063881427, + 0.0021505000040633604, + 0.0018330000020796433, + 0.001933666004333645, + 0.0019261249981354922, + 0.0020325830118963495, + 0.0022792500094510615, + 0.0019637079967651516, + 0.002115209004841745, + 0.0018168749957112595, + 0.001800416997866705, + 0.001849166990723461, + 0.0018374169885646552 + ] + }, + { + "forward": 0.18778987499536015, + "prefill": 0.15283975000784267, + "decode_steps": [ + 0.002428874999168329, + 0.0019820830057142302, + 0.0020249579974915832, + 0.0024399579997407272, + 0.0021288340067258105, + 0.001912167004775256, + 0.0018370000034337863, + 0.0017922919942066073, + 0.0018128329975297675, + 0.0018620420014485717, + 0.0018414999940432608, + 0.002237124994280748, + 0.002013457997236401, + 0.0020059160015080124, + 0.0017877499922178686, + 0.0017617920093471184, + 0.0017477919900557026, + 0.0018111670069629326, + 0.0018810000037774444, + 0.0019403329933993518, + 0.002376167001784779, + 0.0018765830027405173, + 0.0019302909931866452, + 0.001772416988387704, + 0.0017446249985368922, + 0.0017663330072537065, + 0.0018487920024199411, + 0.0018350419995840639, + 0.002046292007435113, + 0.002210415987065062, + 0.0019040829938603565, + 0.0018857499962905422 + ] + }, + { + "forward": 0.14527870899473783, + "prefill": 0.17482545800157823, + "decode_steps": [ + 0.006264958006795496, + 0.0032662500016158447, + 0.0019772499945247546, + 0.001962499998626299, + 0.002316833008080721, + 0.002216749999206513, + 0.0018244999955641106, + 0.001838666998082772, + 0.001774625008692965, + 0.0017579160048626363, + 0.0017422499950043857, + 0.0025129170098807663, + 0.005624500001431443, + 0.002413082998828031, + 0.0025110830029007047, + 0.0026247499918099493, + 0.002319207997061312, + 0.0020017500064568594, + 0.001735291996737942, + 0.0018281250086147338, + 0.0017126250022556633, + 0.0017839999927673489, + 0.0017749999969964847, + 0.0018457499973010272, + 0.0019492919964250177, + 0.0019663749990286306, + 0.002410042012343183, + 0.005244750005658716, + 0.003645749995484948, + 0.0018281660013599321, + 0.0017918749945238233, + 0.0018366669974057004 + ] + }, + { + "forward": 0.15796137499273755, + "prefill": 0.15219845798856113, + "decode_steps": [ + 0.0022350409999489784, + 0.0018880830029956996, + 0.0017425419937353581, + 0.0016545409889658913, + 0.0016965000104391947, + 0.001622292009415105, + 0.0016103329980978742, + 0.0016015419969335198, + 0.0016058749897638336, + 0.001625416989554651, + 0.001604624994797632, + 0.0016498339973622933, + 0.0016219160024775192, + 0.0016048749967012554, + 0.0016171659954125062, + 0.0015953749971231446, + 0.001669707999099046, + 0.0015942079917294905, + 0.001628375001018867, + 0.0016479170008096844, + 0.0015924999897833914, + 0.0016126669943332672, + 0.001633374995435588, + 0.0016207080043386668, + 0.0015958750009303913, + 0.0015869170019868761, + 0.00159275000623893, + 0.0016015830042306334, + 0.0016319580026902258, + 0.0016088749980553985, + 0.0016129590076161548, + 0.0016657500091241673 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 156.18204149359372, + "p95": 189.24345624473062, + "min": 131.34112500119954, + "max": 190.43274999421556, + "iqr": 25.550218499120092, + "mean": 161.49889179650927, + "cv": 0.12119623543421204 + }, + "prefill_ms": { + "count": 10, + "median": 147.23933299683267, + "p95": 176.30679170033545, + "min": 134.06374999613035, + "max": 177.51879199931864, + "iqr": 26.40149999933783, + "mean": 151.33749160013394, + "cv": 0.10956739735731065 + }, + "decode_ms_per_step": { + "count": 320, + "median": 1.6576880007050931, + "p95": 2.429429149196949, + "min": 1.33908299903851, + "max": 6.264958006795496, + "iqr": 0.3648334968602285, + "mean": 1.7815815124777146, + "cv": 0.2777376157279966 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 602.6045139652188, + "p95": 652.3397571915559, + "min": 408.6662844188739, + "max": 667.478376406756, + "iqr": 88.09419056949969, + "mean": 573.8038746354156, + "cv": 0.1439636508802195 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 160.06329301882357, + "p95": 174.20588261446602, + "min": 126.41777683987472, + "max": 174.86947690633858, + "iqr": 21.252165995493158, + "mean": 155.4278588685796, + "cv": 0.1107800592037467 + }, + "memory": { + "peak_extra_bytes": 246218752, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "auto", + "batch": 4, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.0057042919943341985 + }, + { + "forward": 0.005486207999638282 + }, + { + "forward": 0.005260833000647835 + }, + { + "forward": 0.005463667010189965 + }, + { + "forward": 0.005658250011038035 + }, + { + "forward": 0.006129208006314002 + }, + { + "forward": 0.005234583994024433 + }, + { + "forward": 0.005146290990523994 + }, + { + "forward": 0.005037750001065433 + }, + { + "forward": 0.004980124998837709 + } + ], + "forward_ms": { + "count": 10, + "median": 5.3622500054189, + "p95": 5.93799580092309, + "min": 4.980124998837709, + "max": 6.129208006314002, + "iqr": 0.44687526678899303, + "mean": 5.410120800661389, + "cv": 0.06520819352541027 + }, + "memory": { + "peak_extra_bytes": 12091392, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "sdpa", + "batch": 4, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.004271041994797997, + "prefill": 0.004442374993232079, + "decode_steps": [ + 0.0009702499955892563, + 0.0007438340107910335, + 0.0007282919978024438, + 0.0007249580085044727, + 0.0007276249962160364, + 0.000742958000046201, + 0.000726792000932619, + 0.0011577080003917217, + 0.0008789590065134689, + 0.0007982090028235689, + 0.0007744579925201833, + 0.000781084003392607, + 0.0008060840045800433, + 0.0007924160017864779, + 0.0008221250027418137, + 0.0008266670047305524, + 0.0007982499955687672, + 0.0007685830059926957, + 0.0007482080109184608, + 0.0007529589929617941, + 0.0008090420014923438, + 0.0007647080055903643, + 0.0007442500063916668, + 0.0007417079905280843, + 0.0007342500030063093, + 0.0007755840051686391, + 0.0007545000116806477, + 0.0007854580035200343, + 0.0010476249881321564, + 0.0009887500054901466, + 0.0008711249975021929, + 0.0007750420045340434 + ] + }, + { + "forward": 0.004549167002551258, + "prefill": 0.004748666993691586, + "decode_steps": [ + 0.000953249997110106, + 0.0007627500017406419, + 0.0007832500123186037, + 0.0007386659999610856, + 0.0007965840050019324, + 0.0007400829927064478, + 0.0007598750089528039, + 0.000722791999578476, + 0.0007268750050570816, + 0.0007136249914765358, + 0.0007117500063031912, + 0.0007533750031143427, + 0.000757250003516674, + 0.0008030419994611293, + 0.0008293749997392297, + 0.0007988330035004765, + 0.0007401250040857121, + 0.0007378750015050173, + 0.0007403339986922219, + 0.0007306659972527996, + 0.0007254579977598041, + 0.0007132090104278177, + 0.0007275840034708381, + 0.0007487089897040278, + 0.0007337919960264117, + 0.0007524579996243119, + 0.0007418330060318112, + 0.0007418750028591603, + 0.0007833330018911511, + 0.0007722090085735545, + 0.0007628330058651045, + 0.0007560000085504726 + ] + }, + { + "forward": 0.004398500008392148, + "prefill": 0.00451166600396391, + "decode_steps": [ + 0.0009162909991573542, + 0.0007365839992417023, + 0.0007134579936973751, + 0.0006972080009290949, + 0.0006937919970368966, + 0.0006849579949630424, + 0.0007334169931709766, + 0.0007446249946951866, + 0.0007272499933606014, + 0.0007154170016292483, + 0.000727332997485064, + 0.0007610419997945428, + 0.0008264580101240426, + 0.0008227910002460703, + 0.000742958000046201, + 0.0007233340002130717, + 0.0007364999910350889, + 0.0007467090035788715, + 0.0007607090083183721, + 0.000861249995068647, + 0.0007368750084424391, + 0.0007264579908223823, + 0.0008571660000598058, + 0.0008813750027911738, + 0.0009557500015944242, + 0.0008972920040832832, + 0.0009045839979080483, + 0.0008864999981597066, + 0.0008398329955525696, + 0.0007942910015117377, + 0.0007598749944008887, + 0.0007742500019958243 + ] + }, + { + "forward": 0.004210625003906898, + "prefill": 0.0042849160090554506, + "decode_steps": [ + 0.0009655419999035075, + 0.00078600000415463, + 0.0008173749956768006, + 0.0008458750089630485, + 0.0008988749905256554, + 0.0010752920061349869, + 0.0008966669993242249, + 0.0008315419981954619, + 0.0008166250045178458, + 0.0009906250052154064, + 0.000877040991326794, + 0.000850624986924231, + 0.000876124991918914, + 0.0008758330077398568, + 0.0008430830057477579, + 0.000845166010549292, + 0.0009002080041682348, + 0.0009094999986700714, + 0.0008963330037659034, + 0.0008414170006290078, + 0.0007970410078996792, + 0.000803332994109951, + 0.0008976670069387183, + 0.000958333010203205, + 0.0010158329969272017, + 0.0009957919974112883, + 0.0008508339960826561, + 0.0007850420079194009, + 0.000778582994826138, + 0.0007422920025419444, + 0.0007128330034902319, + 0.0007055000023683533 + ] + }, + { + "forward": 0.004513624997343868, + "prefill": 0.004321083004469983, + "decode_steps": [ + 0.0008880419918568805, + 0.000754541004425846, + 0.0007515000033890828, + 0.000734041997930035, + 0.000705291997292079, + 0.0007349590014200658, + 0.0007979169895406812, + 0.000875917001394555, + 0.0007397089939331636, + 0.0007024999940767884, + 0.0007010000117588788, + 0.0007273330120369792, + 0.0007586669962620363, + 0.0009538339945720509, + 0.0008029580058064312, + 0.0007597910007461905, + 0.0007697080000070855, + 0.0007395409920718521, + 0.0007240840059239417, + 0.0007333329995162785, + 0.0007271250069607049, + 0.000734209010261111, + 0.0007206250011222437, + 0.0007865840016165748, + 0.0007544590043835342, + 0.0007490830030292273, + 0.0007699999987380579, + 0.0007819160091457888, + 0.0009033749956870452, + 0.0007661249983357266, + 0.0007677499961573631, + 0.000738000002456829 + ] + }, + { + "forward": 0.0048371250013588, + "prefill": 0.0050660839915508404, + "decode_steps": [ + 0.0010234170040348545, + 0.0007815829885657877, + 0.0007554170006187633, + 0.0007373340049525723, + 0.0007152920006774366, + 0.0007203749992186204, + 0.0007394169952021912, + 0.0007217919919639826, + 0.0007049580017337576, + 0.000693874986609444, + 0.0006939169979887083, + 0.0007309170032385737, + 0.000730999992811121, + 0.0007635829970240593, + 0.0007329579966608435, + 0.0007314999966183677, + 0.0007397089939331636, + 0.0007365830097114667, + 0.0007302500016521662, + 0.0007079159986460581, + 0.0007161660032579675, + 0.00072412499866914, + 0.0007393750129267573, + 0.0007389170059468597, + 0.0007402089977404103, + 0.0007308329950319603, + 0.0007507499976782128, + 0.0007418750028591603, + 0.0007619169919053093, + 0.0007489170093322173, + 0.0007438329921569675, + 0.0007381670002359897 + ] + }, + { + "forward": 0.004148416992393322, + "prefill": 0.004232584004057571, + "decode_steps": [ + 0.0009114999993471429, + 0.0007304580067284405, + 0.0007076250039972365, + 0.0006737079966114834, + 0.0006756249931640923, + 0.0006759170064469799, + 0.0006828750047134236, + 0.0006614160083699971, + 0.000669832996209152, + 0.00068450000253506, + 0.0006914160039741546, + 0.0007218339887913316, + 0.0007170839962782338, + 0.0007022499921731651, + 0.000720083000487648, + 0.0006834590021753684, + 0.0007202080014394596, + 0.0007114159961929545, + 0.0006963750056456774, + 0.0007160410023061559, + 0.0007060839998302981, + 0.0007607089937664568, + 0.0007764580077491701, + 0.0007735410035820678, + 0.0007775410049362108, + 0.0007443329959642142, + 0.0007469580014003441, + 0.0007527499983552843, + 0.0007682499999646097, + 0.0007355839916272089, + 0.0007281250000232831, + 0.0007190420001279563 + ] + }, + { + "forward": 0.004133833004743792, + "prefill": 0.0041505419940222055, + "decode_steps": [ + 0.0008807499980321154, + 0.0007137499924283475, + 0.0007006250089034438, + 0.0006916670099599287, + 0.0006986249936744571, + 0.0007565000123577192, + 0.0007607500010635704, + 0.000714000008883886, + 0.0007072500011418015, + 0.0006860830035293475, + 0.0006190410058479756, + 0.0006330000032903627, + 0.000636124997981824, + 0.0006471250089816749, + 0.0006367500027408823, + 0.0006258330104174092, + 0.0006327500013867393, + 0.0006353750068228692, + 0.0006493330001831055, + 0.000648290995741263, + 0.0006579170003533363, + 0.0006567499949596822, + 0.0006579170003533363, + 0.0006553749990416691, + 0.0006498750008177012, + 0.0006419999990612268, + 0.000649333989713341, + 0.0006512499967357144, + 0.0006614580051973462, + 0.0006522499897982925, + 0.000665250001475215, + 0.0006450420041801408 + ] + }, + { + "forward": 0.004026084003271535, + "prefill": 0.004097832992556505, + "decode_steps": [ + 0.0009074170084204525, + 0.0007942499942146242, + 0.0007557909993920475, + 0.0007517500052927062, + 0.0007314999966183677, + 0.0007067080005072057, + 0.000709084008121863, + 0.0007947499980218709, + 0.0007240840059239417, + 0.0007234999939100817, + 0.0007084579992806539, + 0.0007235000084619969, + 0.0007087910053087398, + 0.0007182500121416524, + 0.0007210829935502261, + 0.000716290989657864, + 0.0007110000005923212, + 0.0007136249914765358, + 0.0007118749927030876, + 0.0006936249992577359, + 0.0006495420093415305, + 0.0006586249946849421, + 0.0006601670029340312, + 0.0006716249918099493, + 0.0006684160034637898, + 0.0006906669877935201, + 0.0007314170070458204, + 0.0006857080006739125, + 0.0006962920015212148, + 0.0006792500062147155, + 0.0006777919916203246, + 0.0006955410062801093 + ] + }, + { + "forward": 0.0038862090004840866, + "prefill": 0.004219249996822327, + "decode_steps": [ + 0.0009634589951019734, + 0.0007464160007657483, + 0.0007212079945020378, + 0.0007077920017763972, + 0.0007151669997256249, + 0.0007224159926408902, + 0.0007222079875646159, + 0.0007070419960655272, + 0.0006421250000130385, + 0.0006312499899649993, + 0.0006264999974519014, + 0.000643749997834675, + 0.0006449579959735274, + 0.0006458339921664447, + 0.0007336660055443645, + 0.0006629999988945201, + 0.0006743330013705418, + 0.0006588749965885654, + 0.0006594580045202747, + 0.0006447920022765175, + 0.0006465000042226166, + 0.0006498750008177012, + 0.0006597499886993319, + 0.0007432080019498244, + 0.0007134999905247241, + 0.0007087499980116263, + 0.0006844159943284467, + 0.0006712080066790804, + 0.0006937919970368966, + 0.0007312079978873953, + 0.0006789580074837431, + 0.0007024169899523258 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 4.240833499352448, + "p95": 4.707543901895406, + "min": 3.8862090004840866, + "max": 4.8371250013588, + "iqr": 0.3473647484497633, + "mean": 4.29746270092437, + "cv": 0.06542790516263838 + }, + "prefill_ms": { + "count": 10, + "median": 4.302999506762717, + "p95": 4.923246342514176, + "min": 4.097832992556505, + "max": 5.06608399155084, + "iqr": 0.2717597526498139, + "mean": 4.407499998342246, + "cv": 0.06817680019392988 + }, + "decode_ms_per_step": { + "count": 320, + "median": 0.7360419913311489, + "p95": 0.9117395493376534, + "min": 0.6190410058479756, + "max": 1.1577080003917217, + "iqr": 0.06977025986998342, + "mean": 0.7528791690674552, + "cv": 0.11197061835220681 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 5330.946779815795, + "p95": 5871.205710023023, + "min": 4623.804379524048, + "max": 5948.496879060049, + "iqr": 446.5711050882128, + "mean": 5339.82895298396, + "cv": 0.0739984794523853 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 4436.131906496437, + "p95": 4926.366331898205, + "min": 4004.03556185815, + "max": 4986.640665171599, + "iqr": 379.4404145741146, + "mean": 4509.503957745013, + "cv": 0.06689578757511394 + }, + "memory": { + "peak_extra_bytes": 11927552, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "reference", + "batch": 4, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.005127041004016064, + "prefill": 0.005338125003618188, + "decode_steps": [ + 0.0011622909951256588, + 0.0010063750087283552, + 0.0011500420077936724, + 0.0010291250073350966, + 0.0010022920032497495, + 0.0009513750119367614, + 0.0009575829899404198, + 0.0009609999979147688, + 0.0009420419955858961, + 0.0009571670088917017, + 0.0009670830040704459, + 0.000991709006484598, + 0.0010089589923154563, + 0.0010496670001884922, + 0.00101570799597539, + 0.000994540998362936, + 0.00098562499624677, + 0.000996250004391186, + 0.0010491659922990948, + 0.0010307089978596196, + 0.0009784579888219014, + 0.000984457990853116, + 0.0009860420104814693, + 0.0009978750022128224, + 0.0009885829931590706, + 0.000972416004515253, + 0.000995875001535751, + 0.0009960830066120252, + 0.000996874994598329, + 0.0010413329873699695, + 0.0010586249991320074, + 0.0010182089899899438 + ] + }, + { + "forward": 0.0052463750034803525, + "prefill": 0.0050705410103546456, + "decode_steps": [ + 0.0011269999959040433, + 0.000996042013866827, + 0.000937834003707394, + 0.0009732920007081702, + 0.0009877499978756532, + 0.0009832089999690652, + 0.0009741249959915876, + 0.0009761249966686592, + 0.0009623340010875836, + 0.001005166006507352, + 0.0009882080048555508, + 0.0010057920007966459, + 0.0011187500058440492, + 0.0010065419919556007, + 0.001038749993313104, + 0.0009744159906404093, + 0.0010100829967996106, + 0.0009602499922038987, + 0.0009805829904507846, + 0.0010219169926131144, + 0.0010308339988114312, + 0.0010005419899243861, + 0.001004625009954907, + 0.0010175420029554516, + 0.000988333995337598, + 0.0010208750027231872, + 0.0010323340102331713, + 0.000993084002402611, + 0.0010009169927798212, + 0.001000667005428113, + 0.0009912499954225495, + 0.0009885000035865232 + ] + }, + { + "forward": 0.00501599999552127, + "prefill": 0.0050313749961787835, + "decode_steps": [ + 0.0011104590084869415, + 0.000993249996099621, + 0.0009607500105630606, + 0.0009631249995436519, + 0.0009604590013623238, + 0.00099516699265223, + 0.0010200419928878546, + 0.0010033339931396767, + 0.0009689589933259413, + 0.0009459159919060767, + 0.000963875005254522, + 0.0010257499961880967, + 0.0009880830039037392, + 0.000990000000456348, + 0.0011103340075351298, + 0.0010606670111883432, + 0.0010090409923577681, + 0.0009986250079236925, + 0.0009978330053854734, + 0.0009895419934764504, + 0.0010291250073350966, + 0.0010056669998448342, + 0.0010326669871574268, + 0.0010249999904772267, + 0.0010133750038221478, + 0.0009944159974111244, + 0.0009932910033967346, + 0.0010720419959397987, + 0.0010281249997206032, + 0.0009953749977285042, + 0.0009806659945752472, + 0.000981207995209843 + ] + }, + { + "forward": 0.005039291994762607, + "prefill": 0.005507374997250736, + "decode_steps": [ + 0.0011721249902620912, + 0.001004208010272123, + 0.0009927499922923744, + 0.0010237910028081387, + 0.0009925000049406663, + 0.000949624998611398, + 0.0009644590027164668, + 0.0010711250070016831, + 0.001357582994387485, + 0.0010707920009735972, + 0.000970374996541068, + 0.0010141669918084517, + 0.0009702920069685206, + 0.0011003750114468858, + 0.0011517920065671206, + 0.0010573340114206076, + 0.001000667005428113, + 0.001007500002742745, + 0.0010296669934177771, + 0.0012354170030448586, + 0.0015446660108864307, + 0.0010164579871343449, + 0.0011012500035576522, + 0.0010136659984709695, + 0.0009862500010058284, + 0.0010412499977974221, + 0.0010137919889530167, + 0.0010070409916806966, + 0.0010422500054119155, + 0.0010681659914553165, + 0.0010414169955765828, + 0.0009968750091502443 + ] + }, + { + "forward": 0.005398041001171805, + "prefill": 0.005062500000349246, + "decode_steps": [ + 0.0011534580116858706, + 0.001007458005915396, + 0.0009722499962663278, + 0.0010577919892966747, + 0.0009591249981895089, + 0.0009436249965801835, + 0.0009872500086203218, + 0.0010028340038843453, + 0.0009970829996746033, + 0.0009677499911049381, + 0.000953249997110106, + 0.0009784580033738166, + 0.0009793750068638474, + 0.0009672909945948049, + 0.0009604160004528239, + 0.0009563329949742183, + 0.0009682089876150712, + 0.0009837079996941611, + 0.0010276669927407056, + 0.0010045410017482936, + 0.001065042000846006, + 0.0009852080111159012, + 0.0009681249939603731, + 0.0009920000011334196, + 0.000983541991445236, + 0.0010755830007838085, + 0.0009837919933488593, + 0.0009937499999068677, + 0.0009918330033542588, + 0.0009830420021899045, + 0.000984041005722247, + 0.0009792500059120357 + ] + }, + { + "forward": 0.005385791999287903, + "prefill": 0.005290790999424644, + "decode_steps": [ + 0.0011625000042840838, + 0.000986750004813075, + 0.0009933340043062344, + 0.0009982920018956065, + 0.0009988750098273158, + 0.0009485420014243573, + 0.0009512079996056855, + 0.0009578750032233074, + 0.0009682920062914491, + 0.0009460409928578883, + 0.0010337919957237318, + 0.0009815000084927306, + 0.0009854160016402602, + 0.0009593750000931323, + 0.0009858749981503934, + 0.0009657500049797818, + 0.0009702079987619072, + 0.0009695000044303015, + 0.0009710830054245889, + 0.0009730000019771978, + 0.0010101250081788749, + 0.0010261249990435317, + 0.0009796669910429046, + 0.000978582989773713, + 0.000956832998781465, + 0.0010274999949615449, + 0.000985207996563986, + 0.00106712500564754, + 0.0009508329967502505, + 0.0009669160062912852, + 0.0010259170085191727, + 0.0010011249978560954 + ] + }, + { + "forward": 0.005027333987527527, + "prefill": 0.004714040987892076, + "decode_steps": [ + 0.0010626669973134995, + 0.0009044159960467368, + 0.0008625419868621975, + 0.0008755420130910352, + 0.0008367499976884574, + 0.0008122079889290035, + 0.0008293330029118806, + 0.0008132910006679595, + 0.0009314579947385937, + 0.0010683750006137416, + 0.0008996670076157898, + 0.0008546669996576384, + 0.0008377500053029507, + 0.0008333749865414575, + 0.00082562500028871, + 0.0008642499888082966, + 0.0008352080039912835, + 0.0008445830026175827, + 0.0008430000016232952, + 0.0008361250074813142, + 0.0008230420062318444, + 0.0008259169990196824, + 0.0008135420066537336, + 0.0008738750038901344, + 0.0008729999972274527, + 0.0008768750121816993, + 0.0008813339954940602, + 0.0008942499989643693, + 0.0009672080050222576, + 0.0009369160106871277, + 0.000906000001123175, + 0.000880833002156578 + ] + }, + { + "forward": 0.004807874996913597, + "prefill": 0.004776999994646758, + "decode_steps": [ + 0.001061666989699006, + 0.0008977079996839166, + 0.0008713340066606179, + 0.0008702909981366247, + 0.0008679159946041182, + 0.0008625000045867637, + 0.0008952500065788627, + 0.0008939590043155476, + 0.0008842499955790117, + 0.0008715000003576279, + 0.0008649590017739683, + 0.0008755410090088844, + 0.0008497920061927289, + 0.0007955000037327409, + 0.0007957909983815625, + 0.0007900839991634712, + 0.0008087080059340224, + 0.0008284159994218498, + 0.0008301670022774488, + 0.0008158749988069758, + 0.0007981670059962198, + 0.0008179580036085099, + 0.0008152079972205684, + 0.0008125829917844385, + 0.0008117500110529363, + 0.0008093749929685146, + 0.0008220419986173511, + 0.0008360840001842007, + 0.0008387499983655289, + 0.0008240000024670735, + 0.0008395829936489463, + 0.0008472919871564955 + ] + }, + { + "forward": 0.004633125005057082, + "prefill": 0.005221916988375597, + "decode_steps": [ + 0.001139500003773719, + 0.0009484589972998947, + 0.0009014579991344362, + 0.0008659590093884617, + 0.0008855420019244775, + 0.0008856250060489401, + 0.0008857499924488366, + 0.0008929579926189035, + 0.0008691249968251213, + 0.0008057499944698066, + 0.0008649160008644685, + 0.0008691670082043856, + 0.0008495420042891055, + 0.0008337499893968925, + 0.0008587079937569797, + 0.0008224580087698996, + 0.0008896250074030831, + 0.0008777080074651167, + 0.0008474999922327697, + 0.0008247499936260283, + 0.0008179160067811608, + 0.0008193749963538721, + 0.0008193749963538721, + 0.0008491669868817553, + 0.000837040992337279, + 0.0008437919896095991, + 0.0008387499983655289, + 0.0008563330047763884, + 0.0008833339961711317, + 0.0009150409896392375, + 0.0009399580012541264, + 0.0009219999919878319 + ] + }, + { + "forward": 0.004948208006680943, + "prefill": 0.0049233750032726675, + "decode_steps": [ + 0.0011525840091053396, + 0.000967707994277589, + 0.0009617500036256388, + 0.0009403750009369105, + 0.000930374997551553, + 0.0008967079920694232, + 0.0009451250080019236, + 0.0009817910031415522, + 0.0009457090054638684, + 0.0008867500000633299, + 0.0008994169911602512, + 0.0008934170036809519, + 0.0008642089960630983, + 0.0008725829975446686, + 0.0008362079970538616, + 0.0008436670032097027, + 0.0008345419919351116, + 0.000884082997799851, + 0.0009705830016173422, + 0.0008862499962560833, + 0.0008392500021727756, + 0.0008432079921476543, + 0.0008323329966515303, + 0.000868874994921498, + 0.0008471660112263635, + 0.0008574170060455799, + 0.0008397089986829087, + 0.0008613750105723739, + 0.0008819579961709678, + 0.0008818750065984204, + 0.0008792080043349415, + 0.0008770419954089448 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 5.033312991145067, + "p95": 5.392528950324049, + "min": 4.633125005057082, + "max": 5.398041001171805, + "iqr": 0.25138549972325563, + "mean": 5.062908299441915, + "cv": 0.04753491067613944 + }, + "prefill_ms": { + "count": 10, + "median": 5.066520505351946, + "p95": 5.431212500116089, + "min": 4.714040987892076, + "max": 5.507374997250736, + "iqr": 0.3231974951631855, + "mean": 5.093703998136334, + "cv": 0.049189130628530774 + }, + "decode_ms_per_step": { + "count": 320, + "median": 0.9703335017547943, + "p95": 1.100418761052424, + "min": 0.7900839991634712, + "max": 1.5446660108864307, + "iqr": 0.12655224782065488, + "mean": 0.9557014311667444, + "cv": 0.09763498653878633 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 4031.751287559223, + "p95": 4657.463975812681, + "min": 3763.6404383921263, + "max": 4722.550175707558, + "iqr": 562.7849406463856, + "mean": 4208.908088092248, + "cv": 0.0796882874722724 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 3466.514003492574, + "p95": 3967.653090978246, + "min": 3239.1122798083015, + "max": 4014.930522552863, + "iqr": 405.7459238771844, + "mean": 3604.985564755989, + "cv": 0.07342686389516359 + }, + "memory": { + "peak_extra_bytes": 11960320, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "auto", + "batch": 4, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.013913832997786812 + }, + { + "forward": 0.016093750004074536 + }, + { + "forward": 0.016357374988729134 + }, + { + "forward": 0.017964250000659376 + }, + { + "forward": 0.02802491698821541 + }, + { + "forward": 0.022372167004505172 + }, + { + "forward": 0.029328582997550257 + }, + { + "forward": 0.026043250007205643 + }, + { + "forward": 0.020613917004084215 + }, + { + "forward": 0.0204438330110861 + } + ], + "forward_ms": { + "count": 10, + "median": 20.528875007585157, + "p95": 28.741933293349575, + "min": 13.913832997786812, + "max": 29.328582997550257, + "iqr": 8.36638551481883, + "mean": 21.115587500389665, + "cv": 0.2503042250228193 + }, + "memory": { + "peak_extra_bytes": 36339712, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "sdpa", + "batch": 4, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.008919290994526818, + "prefill": 0.008847457997035235, + "decode_steps": [ + 0.001145375004853122, + 0.00102599999809172, + 0.0009576250013196841, + 0.0009104589989874512, + 0.0008955420053098351, + 0.0009021250007208437, + 0.0009554169955663383, + 0.0009057080023922026, + 0.0009053750109160319, + 0.0008637080027256161, + 0.0008841660019243136, + 0.0008892089972505346, + 0.0008747500105528161, + 0.0008673750126035884, + 0.0008858329965732992, + 0.0008801670046523213, + 0.0009045839979080483, + 0.0009177079919027165, + 0.0008986659959191456, + 0.000908667003386654, + 0.0008946669986471534, + 0.0009190840064547956, + 0.0009232080046785995, + 0.0009201670036418363, + 0.000927500004763715, + 0.0008926250011427328, + 0.0009149169927695766, + 0.0010215420043095946, + 0.0009889579960145056, + 0.0009218329942086712, + 0.0009255420009139925, + 0.0009122499905060977 + ] + }, + { + "forward": 0.009164333008811809, + "prefill": 0.00908804198843427, + "decode_steps": [ + 0.0011344579979777336, + 0.0009188750118482858, + 0.0008895409991964698, + 0.0008851250022416934, + 0.0008735829906072468, + 0.0008856250060489401, + 0.0008857079956214875, + 0.0008881250105332583, + 0.0009269169968320057, + 0.0009463749884162098, + 0.0009244169923476875, + 0.0009328750020358711, + 0.0009277089993702248, + 0.00091591700038407, + 0.0009084169869311154, + 0.0009112080006161705, + 0.0009224999957950786, + 0.0009647080005379394, + 0.0009575419971952215, + 0.0009443750022910535, + 0.0009201659995596856, + 0.0009040000004461035, + 0.0009510829986538738, + 0.0011266249930486083, + 0.0010108749993378296, + 0.0009309159941039979, + 0.0009668330021668226, + 0.0009150830010185018, + 0.0009279999940190464, + 0.001016708993120119, + 0.0011511670018080622, + 0.0010467920074006543 + ] + }, + { + "forward": 0.010736292009823956, + "prefill": 0.011091374995885417, + "decode_steps": [ + 0.001295041001867503, + 0.0010029169934568927, + 0.0009847500041360036, + 0.0012794580106856301, + 0.0011352919973433018, + 0.001134749996708706, + 0.001138332998380065, + 0.0011170829966431484, + 0.0011150000063935295, + 0.0010078749910462648, + 0.0010523749951971695, + 0.001026375000947155, + 0.0009619170014047995, + 0.0009632500004954636, + 0.0010069580021081492, + 0.0010313329985365272, + 0.0010196249932050705, + 0.0009984579955926165, + 0.0010138329962501302, + 0.00120749999769032, + 0.0011572910007089376, + 0.0011833329917863011, + 0.0011559999984456226, + 0.0011257079895585775, + 0.0011161669972352684, + 0.001016250011161901, + 0.0010667079914128408, + 0.0009937089926097542, + 0.0010858330060727894, + 0.0010873330029426143, + 0.0010684169974410906, + 0.0010466249950695783 + ] + }, + { + "forward": 0.01170358300441876, + "prefill": 0.011322333011776209, + "decode_steps": [ + 0.0013329170033102855, + 0.0010408750094939023, + 0.0010524170065764338, + 0.0009842090075835586, + 0.0009793749923119321, + 0.0009667080012150109, + 0.0010249580082017928, + 0.0010421670012874529, + 0.001000499993097037, + 0.0009609580010874197, + 0.0011085419973824173, + 0.00123741700372193, + 0.0011017499928129837, + 0.0011134160013170913, + 0.0011032500042347237, + 0.0010853749990928918, + 0.0011455420026322827, + 0.0010716659890022129, + 0.001041958006680943, + 0.0009874160023173317, + 0.0009822919964790344, + 0.0009770829929038882, + 0.0010901249916059896, + 0.0010680420091375709, + 0.0010281249997206032, + 0.0010083749948535115, + 0.0011244170018471777, + 0.0010516250040382147, + 0.0010102079977514222, + 0.0009945420024450868, + 0.0009926249913405627, + 0.0009927079954650253 + ] + }, + { + "forward": 0.01885795799898915, + "prefill": 0.018012208995060064, + "decode_steps": [ + 0.0029327080119401217, + 0.0024799579987302423, + 0.001533917005872354, + 0.0014748329995200038, + 0.0014026659919181839, + 0.0015992500120773911, + 0.0022017919982317835, + 0.0016427499940618873, + 0.0015998750022845343, + 0.0012944589980179444, + 0.0017360830097459257, + 0.00220320800144691, + 0.001573291010572575, + 0.002356458004214801, + 0.001674083003308624, + 0.001454541998100467, + 0.0015082919999258593, + 0.0021442500001285225, + 0.001691583005595021, + 0.0015125420031836256, + 0.0019260419940110296, + 0.0013941249926574528, + 0.001412042009178549, + 0.0015220000059343874, + 0.001363541989121586, + 0.0015724580007372424, + 0.001371833001030609, + 0.0027757909992942587, + 0.001817917000153102, + 0.001545957988128066, + 0.001484249994973652, + 0.0019082920043729246 + ] + }, + { + "forward": 0.015474208004889078, + "prefill": 0.015224000002490357, + "decode_steps": [ + 0.001601374999154359, + 0.0013947499974165112, + 0.001344999996945262, + 0.0014424999972106889, + 0.001408084004651755, + 0.0012551669933600351, + 0.0012315839994698763, + 0.0014702089974889532, + 0.0016604999982519075, + 0.0018790000031003729, + 0.001801167003577575, + 0.002027208000072278, + 0.0014176669938024133, + 0.001350332997390069, + 0.0014664589980384335, + 0.0013344170001801103, + 0.0012069160002283752, + 0.0013901249913033098, + 0.0019231249898439273, + 0.002534208004362881, + 0.0015669999993406236, + 0.002058790996670723, + 0.0018808750028256327, + 0.0030615419964306056, + 0.0017391250003129244, + 0.0016043749928940088, + 0.00301091700384859, + 0.0020296250004321337, + 0.001433375000488013, + 0.0015063749888213351, + 0.0016214589995797724, + 0.0014457910001510754 + ] + }, + { + "forward": 0.018906999990576878, + "prefill": 0.037033458007499576, + "decode_steps": [ + 0.005472917007864453, + 0.003218917001504451, + 0.0015483749884879217, + 0.0017014160112012178, + 0.00233387500338722, + 0.0036518749984679744, + 0.0017213749961229041, + 0.002119209006195888, + 0.00235387499560602, + 0.001510332993348129, + 0.001684000002569519, + 0.0016553750028833747, + 0.002045707995421253, + 0.0016420409956481308, + 0.001486665991251357, + 0.0014262920012697577, + 0.0014208750071702525, + 0.00217558401345741, + 0.0014947080053389072, + 0.0013502079964382574, + 0.0012157500023022294, + 0.0012193330039735883, + 0.0013702919968636706, + 0.0014404579997062683, + 0.001546334009617567, + 0.0013786670024273917, + 0.0012603340001078323, + 0.0012307909928495064, + 0.001977332998649217, + 0.00141908299701754, + 0.001295875001233071, + 0.001247292006155476 + ] + }, + { + "forward": 0.016468834000988863, + "prefill": 0.0162268749991199, + "decode_steps": [ + 0.0017695830028969795, + 0.0014458340010605752, + 0.0014007080026203766, + 0.0013931249995948747, + 0.0013868749956600368, + 0.0014839579962426797, + 0.0013902500068070367, + 0.0014760829944862053, + 0.0013688339968211949, + 0.0013960000069346279, + 0.002551875004428439, + 0.0017927090084413067, + 0.001493915988248773, + 0.003107875003479421, + 0.0015826669987291098, + 0.0014336250023916364, + 0.0015647090040147305, + 0.001752207987010479, + 0.001695250000921078, + 0.0018848750041797757, + 0.0017079169920179993, + 0.0016732920048525557, + 0.0020606670004781336, + 0.0015922919992590323, + 0.0015490840014535934, + 0.0018951250094687566, + 0.0015743749972898513, + 0.001476375007769093, + 0.002489250007783994, + 0.001485708009568043, + 0.001455416000680998, + 0.0014350419951369986 + ] + }, + { + "forward": 0.013655208997079171, + "prefill": 0.012977666992810555, + "decode_steps": [ + 0.001732375007122755, + 0.001291416003368795, + 0.0012114589917473495, + 0.0012068329961039126, + 0.0011862920073326677, + 0.001193375006550923, + 0.0012224999954923987, + 0.00121795799350366, + 0.0011785829992732033, + 0.001198333004140295, + 0.0012325420102570206, + 0.0013547499984269962, + 0.0012819999974453822, + 0.0013172920007491484, + 0.0012678329949267209, + 0.0011970839987043291, + 0.0012107920047128573, + 0.0012130830000387505, + 0.0012697079946519807, + 0.001286334008909762, + 0.0012633340083993971, + 0.0012234999885549769, + 0.0012420419952832162, + 0.0013522080116672441, + 0.0013670839980477467, + 0.0012671659933403134, + 0.0014012909960001707, + 0.0013056669995421544, + 0.0012525829952210188, + 0.001237165997736156, + 0.0012413340009516105, + 0.0012497499992605299 + ] + }, + { + "forward": 0.014351208999869414, + "prefill": 0.012883957999292761, + "decode_steps": [ + 0.00152208301005885, + 0.0012989999959245324, + 0.0012366670125629753, + 0.0012035420077154413, + 0.0012052500096615404, + 0.0013426670047920197, + 0.0013110839936416596, + 0.001223458006279543, + 0.0011900000099558383, + 0.0013493340084096417, + 0.0012119999882997945, + 0.0012435829994501546, + 0.0011852500028908253, + 0.0011715830041794106, + 0.0011778329935623333, + 0.0012168749963166192, + 0.0012310409947531298, + 0.0012327080039540306, + 0.0013377079885685816, + 0.0014939169923309237, + 0.0013001659972360358, + 0.0012612499995157123, + 0.0014096669910941273, + 0.0012784589926013723, + 0.0012998750025872141, + 0.00122733399621211, + 0.001229375004186295, + 0.0012459580029826611, + 0.0012752500042552128, + 0.0012679579958785325, + 0.001275499991606921, + 0.0012386250018607825 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 14.003208998474292, + "p95": 18.8849310943624, + "min": 8.919290994526818, + "max": 18.906999990576878, + "iqr": 5.24206274349126, + "mean": 13.82379170099739, + "cv": 0.26524447706548987 + }, + "prefill_ms": { + "count": 10, + "median": 12.930812496051658, + "p95": 28.473895951901774, + "min": 8.847457997035235, + "max": 37.033458007499576, + "iqr": 4.827041750104399, + "mean": 15.270737498940434, + "cv": 0.5371451286079525 + }, + "decode_ms_per_step": { + "count": 320, + "median": 1.2416879981174134, + "p95": 2.20186279839254, + "min": 0.8637080027256161, + "max": 5.472917007864453, + "iqr": 0.4670515118050389, + "mean": 1.350653906183652, + "cv": 0.36488806507745164 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 3146.174406349288, + "p95": 4277.122247635625, + "min": 2183.7351777936797, + "max": 4332.997340861349, + "iqr": 1403.8065330168688, + "mean": 3154.379709203416, + "cv": 0.2617985725405803 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 2387.3861748194386, + "p95": 3292.113502602581, + "min": 1338.2314831536587, + "max": 3334.357082669255, + "iqr": 999.4352627980179, + "mean": 2373.451487317142, + "cv": 0.2858775551663179 + }, + "memory": { + "peak_extra_bytes": 19890176, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "reference", + "batch": 4, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.013136709007085301, + "prefill": 0.0150371250056196, + "decode_steps": [ + 0.0015393750072689727, + 0.0012617910106200725, + 0.0011931660119444132, + 0.0011649999942164868, + 0.0011607499909587204, + 0.0011554170050658286, + 0.0011270420072833076, + 0.0011165829928359017, + 0.0010986249981215224, + 0.0011795830068876967, + 0.0011954170040553436, + 0.0011752499995054677, + 0.0012260829971637577, + 0.0011529590119607747, + 0.0011144999880343676, + 0.0011092500062659383, + 0.0011329160042805597, + 0.0011452500039013103, + 0.0012014999956591055, + 0.0011660419986583292, + 0.0011569159978535026, + 0.001153958000941202, + 0.0011549170012585819, + 0.0011588750057853758, + 0.0011474579951027408, + 0.0011424579861341044, + 0.0012054160033585504, + 0.0011547500034794211, + 0.0011497499945107847, + 0.0011357909970683977, + 0.0011560410057427362, + 0.0012081669992767274 + ] + }, + { + "forward": 0.013028916000621393, + "prefill": 0.012481625002692454, + "decode_steps": [ + 0.0014329590048873797, + 0.0012071670062141493, + 0.0012501250021159649, + 0.0011611250083660707, + 0.0011491250043036416, + 0.0011991670035058632, + 0.0011624589969869703, + 0.001148832991020754, + 0.0011554589873412624, + 0.0011402909876778722, + 0.0011390419967938215, + 0.001158917002612725, + 0.0011383340024622157, + 0.0011494579957798123, + 0.0011418750073062256, + 0.001141375003498979, + 0.001190540991956368, + 0.0011821249936474487, + 0.0011871669994434342, + 0.0011694999993778765, + 0.0012124170025344938, + 0.0011834169999929145, + 0.0012114590062992647, + 0.0011948329920414835, + 0.0011809580028057098, + 0.0011943749996135011, + 0.0012448339984985068, + 0.0012986670044483617, + 0.001263124999240972, + 0.0012148749956395477, + 0.0012133330019423738, + 0.0012144170032115653 + ] + }, + { + "forward": 0.015649791996111162, + "prefill": 0.016410750002251007, + "decode_steps": [ + 0.0015395410009659827, + 0.001214707997860387, + 0.001161042004241608, + 0.0012064579932484776, + 0.001150583993876353, + 0.0011891670001205057, + 0.0011942919954890385, + 0.0011912920017493889, + 0.001181375002488494, + 0.0011599580029724166, + 0.0011820420040749013, + 0.001164667002740316, + 0.0011614169925451279, + 0.0012234590103616938, + 0.0011698750022333115, + 0.0011870409944094718, + 0.0011864589905599132, + 0.0011916660005226731, + 0.0012004169984720647, + 0.001291874999878928, + 0.0013185409916331992, + 0.001307458005612716, + 0.0013528749987017363, + 0.001313750006374903, + 0.0012159170000813901, + 0.0011973339860560372, + 0.0014677500003017485, + 0.0013403339980868623, + 0.0013769579963991418, + 0.0013943750091129914, + 0.0013707080070162192, + 0.0012853329972131178 + ] + }, + { + "forward": 0.028073833003873006, + "prefill": 0.021036416001152247, + "decode_steps": [ + 0.0019969160057371482, + 0.0015399169933516532, + 0.00137854200147558, + 0.0029080419917590916, + 0.004484707998926751, + 0.0021806250006193295, + 0.002209250000305474, + 0.0015534999984083697, + 0.00175012499676086, + 0.002884415996959433, + 0.003746667003724724, + 0.0032233329984592274, + 0.0029382499924395233, + 0.001890625004307367, + 0.001932750004925765, + 0.002054333992418833, + 0.0016275409871013835, + 0.00166937499307096, + 0.00153237501217518, + 0.001865500002168119, + 0.0016837500006658956, + 0.0016177919897018, + 0.0025521670031594113, + 0.0043560000049183145, + 0.0022324169985949993, + 0.0018713750032475218, + 0.0018684160022530705, + 0.0021451659995364025, + 0.001945875003002584, + 0.0018024579912889749, + 0.0017602919979253784, + 0.0018466250039637089 + ] + }, + { + "forward": 0.03099791699787602, + "prefill": 0.0484158750041388, + "decode_steps": [ + 0.0036189579986967146, + 0.003038499999092892, + 0.0026471249875612557, + 0.002093916991725564, + 0.0019272920035291463, + 0.0021508330100914463, + 0.002188875005231239, + 0.0023729999957140535, + 0.0021899580024182796, + 0.0018728750001173466, + 0.0016663749993313104, + 0.0015877499972702935, + 0.0016704159934306517, + 0.0017240410088561475, + 0.0017520420078653842, + 0.002307582995854318, + 0.0022484590008389205, + 0.0017146250029327348, + 0.0017434170003980398, + 0.0016592920001130551, + 0.0015887919871602207, + 0.001618082998902537, + 0.0017471669998485595, + 0.0016277920076390728, + 0.0018860840064007789, + 0.002174624998588115, + 0.00185500000952743, + 0.0018181659979745746, + 0.0016717920079827309, + 0.00159891699149739, + 0.001588750004884787, + 0.0017359579942421988 + ] + }, + { + "forward": 0.028709582998999394, + "prefill": 0.021457625000039116, + "decode_steps": [ + 0.0018993340054294094, + 0.0019532919977791607, + 0.001756750003551133, + 0.0015630830021109432, + 0.0017219580040546134, + 0.0021362919942475855, + 0.002049124988843687, + 0.0017219579895026982, + 0.0016724159941077232, + 0.0016242089914157987, + 0.0020448750001378357, + 0.0018153340060962364, + 0.0017646659980528057, + 0.0019303330045659095, + 0.0023120000114431605, + 0.0019064169900957495, + 0.001867583006969653, + 0.002131417000782676, + 0.0018348750018049031, + 0.0018900830036727712, + 0.0016588329890510067, + 0.0016309999919030815, + 0.0019896250014426187, + 0.002344874999835156, + 0.0018245419923914596, + 0.0016601249953964725, + 0.0016976249899016693, + 0.0018568749947007746, + 0.0018769169982988387, + 0.0016279169940389693, + 0.0016834580019349232, + 0.0031770829955348745 + ] + }, + { + "forward": 0.020235125004546717, + "prefill": 0.019525542011251673, + "decode_steps": [ + 0.002053125004749745, + 0.0016674579965183511, + 0.0016739999991841614, + 0.0018724160036072135, + 0.001608042002771981, + 0.0015669999993406236, + 0.0015893750096438453, + 0.0015951249952195212, + 0.0015847079921513796, + 0.0016366670024581254, + 0.0018260419892612845, + 0.0016497089964104816, + 0.001753167001879774, + 0.001991375000216067, + 0.0016377089923480526, + 0.0018052909872494638, + 0.001968957993085496, + 0.0016844999918248504, + 0.0016895840090001002, + 0.0018346659926464781, + 0.0016765420004958287, + 0.0016205830033868551, + 0.002112708010827191, + 0.0017749590042512864, + 0.0018937499989988282, + 0.0017500840040156618, + 0.002053832999081351, + 0.0017089169996324927, + 0.0018838749965652823, + 0.0017942919948836789, + 0.0017812919977586716, + 0.0025429170054849237 + ] + }, + { + "forward": 0.023912874996312894, + "prefill": 0.023077833000570536, + "decode_steps": [ + 0.0023032919998513535, + 0.0021044169989181682, + 0.0021281660010572523, + 0.002044374996330589, + 0.0021431250061141327, + 0.002169665996916592, + 0.002221082992036827, + 0.0027240839990554377, + 0.001987625000765547, + 0.001826833002269268, + 0.0018040409922832623, + 0.0018387919990345836, + 0.0018498339923098683, + 0.0018315839988645166, + 0.0021061669976916164, + 0.001879041999927722, + 0.001783625004463829, + 0.0019011250114999712, + 0.001737707993015647, + 0.0016222090052906424, + 0.0016900000046007335, + 0.0022557089978363365, + 0.001704791997326538, + 0.0017708750092424452, + 0.0016058750043157488, + 0.0015889170026639476, + 0.0015889580099610612, + 0.0015930829977151006, + 0.0019077499891864136, + 0.0016499999910593033, + 0.0018597090092953295, + 0.0019535840110620484 + ] + }, + { + "forward": 0.01987662499595899, + "prefill": 0.019023500004550442, + "decode_steps": [ + 0.00202266700216569, + 0.0019844589987769723, + 0.0019142910023219883, + 0.0016996249905787408, + 0.0015762090042699128, + 0.0015616659948136657, + 0.0015830839984118938, + 0.0015604169893776998, + 0.0015490829973714426, + 0.0016934580053202808, + 0.001593291002791375, + 0.0017239169974345714, + 0.0020620830036932603, + 0.0016427499940618873, + 0.0015692919987486675, + 0.0015402919962070882, + 0.0017347079992759973, + 0.0017685000057099387, + 0.0017156250105472282, + 0.0017779160116333514, + 0.0016640829999232665, + 0.0018425830057822168, + 0.001758333994075656, + 0.001959582994459197, + 0.001745500005199574, + 0.001643459007027559, + 0.0016555839974898845, + 0.0016234999930020422, + 0.0017589579947525635, + 0.0016864170029293746, + 0.0017985419981414452, + 0.001733707991661504 + ] + }, + { + "forward": 0.017017750011291355, + "prefill": 0.01855400000931695, + "decode_steps": [ + 0.001939624999067746, + 0.0018521669990150258, + 0.0016999579966068268, + 0.0015586249937769026, + 0.0015736659988760948, + 0.0016349580109817907, + 0.0016106249968288466, + 0.0015818329993635416, + 0.0016984590038191527, + 0.0017609169881325215, + 0.0018280829972354695, + 0.0016077500040410087, + 0.001548041997011751, + 0.0015026249893708155, + 0.001524084000266157, + 0.001607832993613556, + 0.001688750009634532, + 0.0015854170051170513, + 0.0015651669964427128, + 0.0015434590022778139, + 0.0016357499989680946, + 0.0018545409984653816, + 0.001681125009781681, + 0.0016587919963058084, + 0.0016420419997302815, + 0.001658541994402185, + 0.0016854580026119947, + 0.0016537500050617382, + 0.0019420000026002526, + 0.0018722499953582883, + 0.0018237920012325048, + 0.0016817079886095598 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 20.055875000252854, + "p95": 29.968166698381534, + "min": 13.028916000621393, + "max": 30.99791699787602, + "iqr": 11.041812002076767, + "mean": 21.063912501267623, + "cv": 0.3123469723030746 + }, + "prefill_ms": { + "count": 10, + "median": 19.274521007901058, + "p95": 37.01375610253305, + "min": 12.481625002692454, + "max": 48.4158750041388, + "iqr": 4.405760246299906, + "mean": 21.502029104158282, + "cv": 0.46374796643370386 + }, + "decode_ms_per_step": { + "count": 320, + "median": 1.6621039976598695, + "p95": 2.3136437608627607, + "min": 1.0986249981215224, + "max": 4.484707998926751, + "iqr": 0.6032292512827553, + "mean": 1.6810186464681465, + "cv": 0.2806947783717981 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 2277.872840021825, + "p95": 3381.8842539533534, + "min": 1801.5704183489545, + "max": 3409.6058589942027, + "iqr": 893.4971316472697, + "mean": 2492.400794463608, + "cv": 0.2380791892348346 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1696.1607407552212, + "p95": 2483.7065462995265, + "min": 1147.9580327770157, + "max": 2523.988788333109, + "iqr": 612.7385100262964, + "mean": 1800.4371860372144, + "cv": 0.25503190705485224 + }, + "memory": { + "peak_extra_bytes": 32800768, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "auto", + "batch": 4, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.05253883299883455 + }, + { + "forward": 0.05357204198662657 + }, + { + "forward": 0.05269837500236463 + }, + { + "forward": 0.05126074999861885 + }, + { + "forward": 0.05081283301115036 + }, + { + "forward": 0.05518950000987388 + }, + { + "forward": 0.05393533399910666 + }, + { + "forward": 0.05218320801213849 + }, + { + "forward": 0.05371933300921228 + }, + { + "forward": 0.05173116699734237 + } + ], + "forward_ms": { + "count": 10, + "median": 52.61860400059959, + "p95": 54.62512530502863, + "min": 50.81283301115036, + "max": 55.18950000987388, + "iqr": 1.8383330025244504, + "mean": 52.764137502526864, + "cv": 0.025541885451018608 + }, + "memory": { + "peak_extra_bytes": 91521024, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "sdpa", + "batch": 4, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.03132466599345207, + "prefill": 0.03408437500183936, + "decode_steps": [ + 0.0017791669961297885, + 0.0015210420096991584, + 0.0014458749938057736, + 0.0014153340016491711, + 0.0014647909993072972, + 0.0014435000048251823, + 0.0013862500054528937, + 0.0014564169978257269, + 0.0013574580079875886, + 0.0013492920115822926, + 0.0013546249974751845, + 0.0014309999969555065, + 0.001414541999110952, + 0.001560708013130352, + 0.0014719999890075997, + 0.0015346250002039596, + 0.0015426250029122457, + 0.001570958993397653, + 0.0015078339929459617, + 0.0015033750096336007, + 0.001490875001763925, + 0.001529333007056266, + 0.001404415990691632, + 0.0014427909918595105, + 0.0015024579915916547, + 0.0017187090124934912, + 0.0014780829951632768, + 0.0014144579909043387, + 0.001415292004821822, + 0.0014408749993890524, + 0.0014188749919412658, + 0.001439874991774559 + ] + }, + { + "forward": 0.029209832995547913, + "prefill": 0.02958904100523796, + "decode_steps": [ + 0.001736165999318473, + 0.0015554169949609786, + 0.0015992089902283624, + 0.0013927499967394397, + 0.0013931249995948747, + 0.0013022920029470697, + 0.001496000011684373, + 0.001436916005332023, + 0.001345333002973348, + 0.001408208001521416, + 0.0016152089956449345, + 0.0015471249935217202, + 0.0013172909966669977, + 0.0016379580047214404, + 0.0015204170049401, + 0.0013879589969292283, + 0.0013785419869236648, + 0.0015285409899661317, + 0.0015400409902213141, + 0.00156883399176877, + 0.0014435420016525313, + 0.0014090839977143332, + 0.0014347499964060262, + 0.001439541985746473, + 0.0013422499905573204, + 0.0013597910001408309, + 0.001408417010679841, + 0.001377833992592059, + 0.0014104580041021109, + 0.0014317909954115748, + 0.0015275829937309027, + 0.001399082990246825 + ] + }, + { + "forward": 0.028700875001959503, + "prefill": 0.03035208399523981, + "decode_steps": [ + 0.0017498329980298877, + 0.001518500008387491, + 0.001592375003383495, + 0.001492042007157579, + 0.0016102079971460626, + 0.0016227500018430874, + 0.00140974999521859, + 0.0014349580014823005, + 0.001382707996526733, + 0.0013978329952806234, + 0.0014597909903386608, + 0.0018142499902751297, + 0.001597750000655651, + 0.0014164590102154762, + 0.0015349589957622811, + 0.0013414580025710166, + 0.0013984999968670309, + 0.0013795840059174225, + 0.0012642090005101636, + 0.0014077500090934336, + 0.0013310000067576766, + 0.0013095000031171367, + 0.0013105419930070639, + 0.001393041995470412, + 0.0013671659980900586, + 0.0013605409912997857, + 0.0013345829938771203, + 0.0014016250061104074, + 0.0013691669882973656, + 0.001370500001939945, + 0.0013504999951692298, + 0.0013431660045171157 + ] + }, + { + "forward": 0.030722290990524925, + "prefill": 0.03227837500162423, + "decode_steps": [ + 0.0018745419947663322, + 0.0014896670036250725, + 0.0015045000036479905, + 0.0015307080029742792, + 0.0014576660032616928, + 0.001393582992022857, + 0.001444249995984137, + 0.0014214170078048483, + 0.0015319579979404807, + 0.0016256249946309254, + 0.0013762499875156209, + 0.0013770000077784061, + 0.0014361669891513884, + 0.0013121669908287004, + 0.0019304170127725229, + 0.0013866249937564135, + 0.0014254590059863403, + 0.001383833005093038, + 0.0014149580092635006, + 0.0014200830046320334, + 0.0014027500001247972, + 0.001444042005459778, + 0.0013888750108890235, + 0.0014357909967657179, + 0.0013732500083278865, + 0.0013291670038597658, + 0.0014514170034090057, + 0.0014671250100946054, + 0.0014705420035170391, + 0.001410541997756809, + 0.0014308749960036948, + 0.0014041249960428104 + ] + }, + { + "forward": 0.03194454198819585, + "prefill": 0.03211391701188404, + "decode_steps": [ + 0.0018698339990805835, + 0.001669749995926395, + 0.0014609170029871166, + 0.001559249998535961, + 0.001455499994335696, + 0.0014224170008674264, + 0.001541749996249564, + 0.0013807500072289258, + 0.0014353330043377355, + 0.0014699580060550943, + 0.00163145900296513, + 0.0016715410019969568, + 0.0014814169990131631, + 0.001560541000799276, + 0.001360624999506399, + 0.0014806669933022931, + 0.001397750005708076, + 0.0014993750082794577, + 0.001625125005375594, + 0.0014440829982049763, + 0.0017462910036556423, + 0.001305708006839268, + 0.0012620000052265823, + 0.0013721669965889305, + 0.0013915839954279363, + 0.001402709007379599, + 0.001280291995499283, + 0.0013623330014524981, + 0.001486124994698912, + 0.0013534999889088795, + 0.0013617079966934398, + 0.0014378749910974875 + ] + }, + { + "forward": 0.031277832997147925, + "prefill": 0.03156408399809152, + "decode_steps": [ + 0.0018051249935524538, + 0.0014854159962851554, + 0.001335875000222586, + 0.001320000010309741, + 0.0013257090031402186, + 0.001396417006617412, + 0.001337791996775195, + 0.0013756249973084778, + 0.0013666669983649626, + 0.001376208005240187, + 0.0013871669943910092, + 0.0013785830087726936, + 0.0014103339926805347, + 0.0015074580005602911, + 0.0013753329985775054, + 0.0013256669917609543, + 0.0013738749985350296, + 0.0013773329992545769, + 0.0013000829931115732, + 0.001334707994828932, + 0.001312958003836684, + 0.001313916000071913, + 0.0013427920057438314, + 0.0013418750022538006, + 0.0013786670024273917, + 0.001353582993033342, + 0.001313165994361043, + 0.0013507499970728531, + 0.001353750005364418, + 0.0013587499997811392, + 0.0013572089956142008, + 0.0013210829929448664 + ] + }, + { + "forward": 0.03324562498892192, + "prefill": 0.03386129200225696, + "decode_steps": [ + 0.0017972089990507811, + 0.0016070830024546012, + 0.0014578750124201179, + 0.0014770420093555003, + 0.001481375002185814, + 0.001481958999647759, + 0.0014499170065391809, + 0.0014582500007236376, + 0.0015122500044526532, + 0.001534916998934932, + 0.0014823750098003075, + 0.0015005829918663949, + 0.00146899999526795, + 0.001458208993426524, + 0.001398125008563511, + 0.0014133340009720996, + 0.0014179160061758012, + 0.001532166003016755, + 0.0013865829969290644, + 0.001414749989635311, + 0.0015181670023594052, + 0.0014998749975347891, + 0.0014653340040240437, + 0.0015379160031443462, + 0.0014967500028433278, + 0.0014030000020284206, + 0.001408249998348765, + 0.0014820840005995706, + 0.001542917001643218, + 0.0015007499896455556, + 0.0014064580027479678, + 0.0015165420045377687 + ] + }, + { + "forward": 0.03117916701012291, + "prefill": 0.03257208300055936, + "decode_steps": [ + 0.0019480830087559298, + 0.0015622079954482615, + 0.0014883330004522577, + 0.0015355420036939904, + 0.0016202499973587692, + 0.00151966699922923, + 0.0015386660088552162, + 0.0016049169935286045, + 0.0014311249979073182, + 0.0015230830031214282, + 0.001514916992164217, + 0.0014875000051688403, + 0.001418542000465095, + 0.001394291000906378, + 0.0013816249993396923, + 0.0015277920028893277, + 0.0013510409917216748, + 0.0013804999907733873, + 0.0013831670075887814, + 0.0013817079889122397, + 0.001393333004671149, + 0.0014027909928699955, + 0.001424208006937988, + 0.001355499989585951, + 0.0014313749998109415, + 0.0016749159985920414, + 0.0016902499919524416, + 0.001316457986831665, + 0.0014620000001741573, + 0.0014163330051815137, + 0.0013261249987408519, + 0.001385708004818298 + ] + }, + { + "forward": 0.029807458005961962, + "prefill": 0.030384540994418785, + "decode_steps": [ + 0.0017764170042937621, + 0.0015507499920204282, + 0.001412208002875559, + 0.0015342499973485246, + 0.001536999989184551, + 0.0014085839939070866, + 0.001466125002480112, + 0.0014162919978844002, + 0.0014454579941229895, + 0.0014684999914607033, + 0.0014452089963015169, + 0.001430916992831044, + 0.0014093750069150701, + 0.001487832996645011, + 0.0014479999954346567, + 0.0014964169968152419, + 0.0014319580077426508, + 0.0015813330101082101, + 0.001348083998891525, + 0.0014174160023685545, + 0.0013375419948715717, + 0.0013651250046677887, + 0.0014317500026663765, + 0.0013898330071242526, + 0.001331124993157573, + 0.0013327500055311248, + 0.001392999998643063, + 0.00130116700893268, + 0.0014519579999614507, + 0.0014201669982867315, + 0.0013281249994179234, + 0.0013365000049816445 + ] + }, + { + "forward": 0.031620041001588106, + "prefill": 0.032407999999122694, + "decode_steps": [ + 0.0017040420061675832, + 0.0014042090042494237, + 0.001384458999382332, + 0.0013633330090669915, + 0.0013422499905573204, + 0.0013684169971384108, + 0.00145087500277441, + 0.00134920900745783, + 0.0013263330038171262, + 0.0013500000059138983, + 0.0013369590014917776, + 0.0013459170004352927, + 0.001337749999947846, + 0.0013635409995913506, + 0.0013743750023422763, + 0.0013953750021755695, + 0.0013577920035459101, + 0.0013994170003570616, + 0.0014204580074874684, + 0.0015080829907674342, + 0.001503040999523364, + 0.0013782080059172586, + 0.0013479170011123642, + 0.0014019999944139272, + 0.0014216249983292073, + 0.0013684169971384108, + 0.0013845840003341436, + 0.0013928330008639023, + 0.0013667919993167743, + 0.0013889170077163726, + 0.001411292003467679, + 0.0014295000000856817 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 31.228500003635418, + "p95": 32.66013763859519, + "min": 28.700875001959503, + "max": 33.24562498892192, + "iqr": 1.5100309974513948, + "mean": 30.90323309734231, + "cv": 0.04372196138214048 + }, + "prefill_ms": { + "count": 10, + "median": 32.196146006754134, + "p95": 33.98398765202728, + "min": 29.58904100523796, + "max": 34.08437500183936, + "iqr": 1.851635504863225, + "mean": 31.92077920102747, + "cv": 0.04630001457446932 + }, + "decode_ms_per_step": { + "count": 320, + "median": 1.418229003320448, + "p95": 1.671709751826711, + "min": 1.2620000052265823, + "max": 1.9480830087559298, + "iqr": 0.12134400094510056, + "mean": 1.4477359467036877, + "cv": 0.07735793598953003 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 2739.8197736384845, + "p95": 2889.450224869769, + "min": 2694.2285664776177, + "max": 2909.4959920818274, + "iqr": 72.89880971377852, + "mean": 2764.582166107086, + "cv": 0.026044829741512614 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1640.1255037070268, + "p95": 1686.8428834374981, + "min": 1573.056432946689, + "max": 1694.06378146588, + "iqr": 70.7717688738121, + "mean": 1636.9545872151482, + "cv": 0.027696738584157166 + }, + "memory": { + "peak_extra_bytes": 36667392, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "reference", + "batch": 4, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.05058412499784026, + "prefill": 0.05056766599591356, + "decode_steps": [ + 0.002199166003265418, + 0.0019044579967157915, + 0.0019870410033036023, + 0.0019142499950248748, + 0.0018082499882439151, + 0.0017826670082286, + 0.0017506250005681068, + 0.0020139580010436475, + 0.0017962080019060522, + 0.0017461660027038306, + 0.001752374999341555, + 0.0018100419983966276, + 0.0017175410030176863, + 0.0017189170030178502, + 0.001687958007096313, + 0.0018557910079834983, + 0.0017430000007152557, + 0.0017972919886233285, + 0.001720000000204891, + 0.0017514579958515242, + 0.0016750829963712022, + 0.0018981250032084063, + 0.0017602919979253784, + 0.0017383329977747053, + 0.0017145000019809231, + 0.0018594169960124418, + 0.0018779579986585304, + 0.0018271250010002404, + 0.0017619999998714775, + 0.001749332994222641, + 0.0017192910017911345, + 0.0018836670060409233 + ] + }, + { + "forward": 0.051295416997163557, + "prefill": 0.05300658299529459, + "decode_steps": [ + 0.0021723339887103066, + 0.001842416008003056, + 0.0018359160021645948, + 0.0017645419866312295, + 0.0018273329915245995, + 0.0018077499989885837, + 0.0018339999951422215, + 0.0018030000064754859, + 0.0018078749999403954, + 0.0018263749952893704, + 0.0018295419868081808, + 0.0017829169955803081, + 0.001789833011571318, + 0.0017853330064099282, + 0.0018282079981872812, + 0.0018100000015692785, + 0.0018318749935133383, + 0.001878708993899636, + 0.0017464590055169538, + 0.0017707910010358319, + 0.0018384159920969978, + 0.0018193750001955777, + 0.0017871250020107254, + 0.001836374998674728, + 0.0018084999901475385, + 0.0018708339921431616, + 0.001756375000695698, + 0.0018358749948674813, + 0.0017816670006141067, + 0.0017439169896533713, + 0.0018325830023968592, + 0.001858415998867713 + ] + }, + { + "forward": 0.0524104580108542, + "prefill": 0.05103866700665094, + "decode_steps": [ + 0.002239000008557923, + 0.0018616249872138724, + 0.0021842919959453866, + 0.0018656249885680154, + 0.0018576250004116446, + 0.001878041002782993, + 0.0020379159977892414, + 0.0017534579965285957, + 0.001826124993385747, + 0.0017910000024130568, + 0.001624582990189083, + 0.0018465409957570955, + 0.0019772499945247546, + 0.001738292005029507, + 0.0017850839940365404, + 0.001747999995131977, + 0.0017742919881129637, + 0.0017753749998519197, + 0.0017694580019451678, + 0.001649167010327801, + 0.0018477079865988344, + 0.0019410829991102219, + 0.0019060000049648806, + 0.0020813330047531053, + 0.0018445830064592883, + 0.0019515829917509109, + 0.002017457998590544, + 0.001809125009458512, + 0.0018529999942984432, + 0.002177292000851594, + 0.001866457998403348, + 0.0018246249965159222 + ] + }, + { + "forward": 0.05222845800744835, + "prefill": 0.05469387500488665, + "decode_steps": [ + 0.002298084000358358, + 0.0019929999980377033, + 0.0018302079988643527, + 0.001989291005884297, + 0.0018292500026291236, + 0.002111791996867396, + 0.0019151669985149056, + 0.002544875009334646, + 0.002053541989880614, + 0.0022318330011330545, + 0.001876791997347027, + 0.0018263749952893704, + 0.0019762500014621764, + 0.0018512090027797967, + 0.0019190000020898879, + 0.0017992089997278526, + 0.0018052500090561807, + 0.002167375001590699, + 0.0019352500094100833, + 0.0019036659941775724, + 0.0018863750010496005, + 0.0018960840097861364, + 0.001988333009649068, + 0.0018423330038785934, + 0.001862457997049205, + 0.0018401250126771629, + 0.00188170799810905, + 0.0019069999980274588, + 0.0020305840007495135, + 0.0018667499971343204, + 0.0018353329942328855, + 0.001803207996999845 + ] + }, + { + "forward": 0.05379020899999887, + "prefill": 0.05366729199886322, + "decode_steps": [ + 0.00212508300319314, + 0.0017989159969147295, + 0.0017862499953480437, + 0.0017667499923845753, + 0.0017995420057559386, + 0.0017768329998943955, + 0.0017739579925546423, + 0.0018272909946972504, + 0.0019276660023024306, + 0.0018561250035418198, + 0.0018222910002805293, + 0.001880915995570831, + 0.0018262090015923604, + 0.001884957993752323, + 0.0018222079961560667, + 0.0018468330090399832, + 0.001862040997366421, + 0.0018757919897325337, + 0.0018389579927315935, + 0.0018617909954627976, + 0.001825166997150518, + 0.0019185419951099902, + 0.001894457993330434, + 0.0019267500028945506, + 0.0019110830035060644, + 0.0018604579963721335, + 0.0018454999953974038, + 0.0018286660051671788, + 0.0018961250025313348, + 0.0018742089887382463, + 0.0018577080045361072, + 0.001830041001085192 + ] + }, + { + "forward": 0.04958483400696423, + "prefill": 0.04932445900340099, + "decode_steps": [ + 0.0021440419950522482, + 0.0017961250123335049, + 0.001724249988910742, + 0.0017122919962275773, + 0.0016727499896660447, + 0.0018444580055074766, + 0.0017590000061318278, + 0.0017163749871542677, + 0.0017442079988541082, + 0.0017094999930122867, + 0.0017558750114403665, + 0.0017545830050949007, + 0.0017202500021085143, + 0.0017044580017682165, + 0.0017054159980034456, + 0.001833917005569674, + 0.0017399999924236909, + 0.0017719579918775707, + 0.0017055409989552572, + 0.0016845000063767657, + 0.001968582990230061, + 0.0017415829934179783, + 0.001749500006553717, + 0.0016636670043226331, + 0.0018620420014485717, + 0.0017632910021347925, + 0.0017608329944778234, + 0.001735832993290387, + 0.0016883339994819835, + 0.0017565830057719722, + 0.001716750004561618, + 0.001800416997866705 + ] + }, + { + "forward": 0.055927332999999635, + "prefill": 0.05570229199656751, + "decode_steps": [ + 0.0023532499908469617, + 0.0020119170076213777, + 0.001981709006940946, + 0.001953500002855435, + 0.0019918750040233135, + 0.002075207987218164, + 0.0019939579942729324, + 0.0019184170087100938, + 0.001898292000987567, + 0.0019492499995976686, + 0.001955958010512404, + 0.0019659579993458465, + 0.001830082997912541, + 0.0021447920007631183, + 0.001908749996800907, + 0.0018532079993747175, + 0.001848167012212798, + 0.0018420419946778566, + 0.0018752089963527396, + 0.0018840830016415566, + 0.0018132080003852025, + 0.0018149590032408014, + 0.0020453750039450824, + 0.001895207999041304, + 0.0018665840034373105, + 0.0019403750047786161, + 0.001949083001818508, + 0.0019101249927189201, + 0.0018445419991621748, + 0.001835458999266848, + 0.001865290993009694, + 0.0020139999978709966 + ] + }, + { + "forward": 0.05175766699539963, + "prefill": 0.05131983300088905, + "decode_steps": [ + 0.002225292002549395, + 0.0018822499987436458, + 0.0018427909963065758, + 0.001847417006501928, + 0.0018219999910797924, + 0.0017932920018211007, + 0.0018221670034108683, + 0.0018608749960549176, + 0.0019071669958066195, + 0.001886459009256214, + 0.0018561249889899045, + 0.0018377499945927411, + 0.0019560410000849515, + 0.00199425000755582, + 0.0018456250109011307, + 0.0018778340017888695, + 0.0018730829906417057, + 0.0017974170041270554, + 0.0017937920056283474, + 0.001779082987923175, + 0.001761958992574364, + 0.0017877079953905195, + 0.0017856670019682497, + 0.0018128749943571165, + 0.0018025000026682392, + 0.0018541669996920973, + 0.0018697500054258853, + 0.0018461250001564622, + 0.0018168750102631748, + 0.001822499994887039, + 0.0019372910028323531, + 0.0018285419937456027 + ] + }, + { + "forward": 0.0519072079914622, + "prefill": 0.055690209002932534, + "decode_steps": [ + 0.0022189589944900945, + 0.002053832999081351, + 0.00201237500004936, + 0.0017486249998910353, + 0.0017532079946249723, + 0.0020400409994181246, + 0.0018307919963262975, + 0.0020195420074742287, + 0.001966458003153093, + 0.0017711249965941533, + 0.0017596250108908862, + 0.0017644589970586821, + 0.0019065420055994764, + 0.0017287090013269335, + 0.0017021670064423233, + 0.0017518750100862235, + 0.0018109590018866584, + 0.0017580420098965988, + 0.0018427909963065758, + 0.001839792006649077, + 0.0016595000051893294, + 0.001805541993235238, + 0.0019623749976744875, + 0.0017688329971861094, + 0.0017393750022165477, + 0.002268290991196409, + 0.0018344999989494681, + 0.0018393750069662929, + 0.0017676659917924553, + 0.0017877500067697838, + 0.0021806250006193295, + 0.0018984589987667277 + ] + }, + { + "forward": 0.04885991600167472, + "prefill": 0.05074912500276696, + "decode_steps": [ + 0.002264666007249616, + 0.0020358329929877073, + 0.0020387919939821586, + 0.0021297500061336905, + 0.0019709170010173693, + 0.0018393749924143776, + 0.0017534159997012466, + 0.0019040420011151582, + 0.0019880840118275955, + 0.001811541005736217, + 0.0019257499952800572, + 0.0017584159941179678, + 0.0016497079923283309, + 0.0017947500018635765, + 0.0017614590033190325, + 0.0017080000106943771, + 0.001694665988907218, + 0.0018017089896602556, + 0.0017705000063870102, + 0.0017050419992301613, + 0.001735208003083244, + 0.00166920899937395, + 0.0017512920021545142, + 0.001751624993630685, + 0.0017747499950928614, + 0.001770916991517879, + 0.002234957995824516, + 0.0020566250022966415, + 0.0017872919997898862, + 0.001807084001484327, + 0.0017505839932709932, + 0.0018703749956330284 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 51.832437493430916, + "p95": 54.96562719999929, + "min": 48.85991600167472, + "max": 55.927332999999635, + "iqr": 1.6030100123316515, + "mean": 51.834562500880565, + "cv": 0.03903046847133827 + }, + "prefill_ms": { + "count": 10, + "median": 52.16320799809182, + "p95": 55.69685464943177, + "min": 49.32445900340099, + "max": 55.70229199656751, + "iqr": 3.615718749642838, + "mean": 52.5760001008166, + "cv": 0.04356594170354459 + }, + "decode_ms_per_step": { + "count": 320, + "median": 1.8349164965911768, + "p95": 2.1459211508044977, + "min": 1.624582990189083, + "max": 2.544875009334646, + "iqr": 0.13239625695860013, + "mean": 1.8600585619424237, + "cv": 0.07046693641801154 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 2153.8707261728823, + "p95": 2242.5098234174975, + "min": 2048.075073133005, + "max": 2269.2253656231546, + "iqr": 52.88526439198495, + "mean": 2152.244874609917, + "cv": 0.030255105601778038 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1150.5994678848451, + "p95": 1196.7630614811846, + "min": 1087.2138578741387, + "max": 1210.6151314242618, + "iqr": 47.31300618584942, + "mean": 1143.052640743092, + "cv": 0.034061076190647556 + }, + "memory": { + "peak_extra_bytes": 86228992, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "auto", + "batch": 4, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.1978910000034375 + }, + { + "forward": 0.20594200000050478 + }, + { + "forward": 0.19778933300403878 + }, + { + "forward": 0.19281145799322985 + }, + { + "forward": 0.17807470899424516 + }, + { + "forward": 0.20036458400136326 + }, + { + "forward": 0.19110366700624581 + }, + { + "forward": 0.1811042500048643 + }, + { + "forward": 0.18017920799320564 + }, + { + "forward": 0.18986633300664835 + } + ], + "forward_ms": { + "count": 10, + "median": 191.95756249973783, + "p95": 203.4321628008911, + "min": 178.07470899424516, + "max": 205.94200000050478, + "iqr": 14.570812498277519, + "mean": 191.51265420077834, + "cv": 0.048876567229137774 + }, + "memory": { + "peak_extra_bytes": 264749056, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "sdpa", + "batch": 4, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.07231979198695626, + "prefill": 0.08216083400475327, + "decode_steps": [ + 0.002678167002159171, + 0.0022069169936003163, + 0.0022286249877652153, + 0.0020150409982306883, + 0.002204749995144084, + 0.0020109170000068843, + 0.0019949169945903122, + 0.0019377080025151372, + 0.0019509999983711168, + 0.0020166670001344755, + 0.0019279159896541387, + 0.0019108749984297901, + 0.0019229999888921157, + 0.0021525829943129793, + 0.0018937910062959418, + 0.0019470410043140873, + 0.0019655409996630624, + 0.0020375409949338064, + 0.0020583750010700896, + 0.001961125002708286, + 0.0019627080037025735, + 0.0021196669986238703, + 0.0020000420045107603, + 0.001991958008147776, + 0.0020372080034576356, + 0.0021206250094110146, + 0.00210470799356699, + 0.0021941250015515834, + 0.0022069169936003163, + 0.0022208340087672696, + 0.00238379200163763, + 0.0024522080057067797 + ] + }, + { + "forward": 0.0878000419907039, + "prefill": 0.07110183400800452, + "decode_steps": [ + 0.0031351670040749013, + 0.017703625009744428, + 0.004687416992965154, + 0.004848249998758547, + 0.002445499994792044, + 0.005132834005053155, + 0.002200208997237496, + 0.0021416249946923926, + 0.0027879999979631975, + 0.0057444590056547895, + 0.0020897499925922602, + 0.0020102090056752786, + 0.0021120419987710193, + 0.00199629100097809, + 0.0020914999913657084, + 0.002138958006980829, + 0.002560458000516519, + 0.0025904999929480255, + 0.0026060829986818135, + 0.00275912499637343, + 0.003253875009249896, + 0.0024042500008363277, + 0.002150166008505039, + 0.0021122500038472936, + 0.00207274999411311, + 0.0020628750062314793, + 0.0020970839977962896, + 0.0020640419970732182, + 0.0020727910014102235, + 0.0020690409874077886, + 0.002099165998515673, + 0.0021044159948360175 + ] + }, + { + "forward": 0.061456750001525506, + "prefill": 0.06021708299522288, + "decode_steps": [ + 0.0025897500017890707, + 0.001928165991557762, + 0.0018702499946812168, + 0.0018376250081928447, + 0.0017980829870793968, + 0.0017879170045489445, + 0.001880708005046472, + 0.0017966250015888363, + 0.0018096669955411926, + 0.0017979580006795004, + 0.0017985419981414452, + 0.0017939160024980083, + 0.0017916670039994642, + 0.0017966250015888363, + 0.001797125005396083, + 0.0018093329999828711, + 0.0018659579945961013, + 0.0017961249977815896, + 0.0018277499912073836, + 0.0019345409964444116, + 0.001870041000074707, + 0.0018675000028451905, + 0.0018552920082584023, + 0.0018498330027796328, + 0.001921916991705075, + 0.0018641670030774549, + 0.00186266700620763, + 0.0018500830046832561, + 0.0018972919933730736, + 0.001891207997687161, + 0.0018731660093180835, + 0.0019449579995125532 + ] + }, + { + "forward": 0.058808791989577, + "prefill": 0.0594655419990886, + "decode_steps": [ + 0.002478624999639578, + 0.0018743330001598224, + 0.0018502500024624169, + 0.0018032090010819957, + 0.0017967910098377615, + 0.0018165829969802871, + 0.0018037919944617897, + 0.0018421660060994327, + 0.0018019999988609925, + 0.001885500008938834, + 0.0017962919955607504, + 0.0018225839885417372, + 0.001797290999093093, + 0.0018021669966401532, + 0.0018538330041337758, + 0.0018031660001724958, + 0.0018349579913774505, + 0.00181049999082461, + 0.0018027079931925982, + 0.0018838749965652823, + 0.0019165839912602678, + 0.0018585829966468737, + 0.0018496249977033585, + 0.0018790409958455712, + 0.0019122909870930016, + 0.001883999997517094, + 0.0018674169987207279, + 0.0018894579989137128, + 0.0018781669932650402, + 0.001917500005220063, + 0.001964540992048569, + 0.0018864169978769496 + ] + }, + { + "forward": 0.06088770799397025, + "prefill": 0.061076459009200335, + "decode_steps": [ + 0.002863333997083828, + 0.0019695829978445545, + 0.001912791994982399, + 0.002039958009845577, + 0.0018687910051085055, + 0.0018702919915085658, + 0.0018482080049579963, + 0.0019319589919177815, + 0.0018569579988252372, + 0.001859249998233281, + 0.0018502499879105017, + 0.0018703750101849437, + 0.0018789159948937595, + 0.0018450829957146198, + 0.001839042000938207, + 0.0018684999959077686, + 0.001982208006666042, + 0.0018764999986160547, + 0.001869332991191186, + 0.0018907909980043769, + 0.0018925420008599758, + 0.0018701669905567542, + 0.0018649580015335232, + 0.0019526249961927533, + 0.001995416998397559, + 0.0020227920031175017, + 0.0019526669930201024, + 0.001994457998080179, + 0.0019391670066397637, + 0.0019092079892288893, + 0.0018724999972619116, + 0.0018914579995907843 + ] + }, + { + "forward": 0.06257174999336712, + "prefill": 0.0628294580092188, + "decode_steps": [ + 0.00284566699701827, + 0.001901166993775405, + 0.0018840830016415566, + 0.0018405830051051453, + 0.0019334169919602573, + 0.001806667001801543, + 0.001790082998923026, + 0.0018421249988023192, + 0.0018288330029463395, + 0.001920915994560346, + 0.0018069170037051663, + 0.0018581670010462403, + 0.0018118339939974248, + 0.0019235420040786266, + 0.0018085420015268028, + 0.0018369580066064373, + 0.0018297920032637194, + 0.0018511670059524477, + 0.0019307499896967784, + 0.0018920839938800782, + 0.0018762910040095448, + 0.0018597079906612635, + 0.0019647090084617957, + 0.0018809999892255291, + 0.0018802499980665743, + 0.001861208991613239, + 0.0018980839959112927, + 0.0019479589973343536, + 0.001965125004062429, + 0.001887792008346878, + 0.0020222910097800195, + 0.00206416699802503 + ] + }, + { + "forward": 0.061582915994222276, + "prefill": 0.06732975000340957, + "decode_steps": [ + 0.0026693749969126657, + 0.0020352089923107997, + 0.001979375010705553, + 0.0019263329886598513, + 0.0020600840070983395, + 0.0019509999983711168, + 0.0018074169929604977, + 0.0018385000003036112, + 0.0018466249894117936, + 0.0018107079959008843, + 0.0019945000094594434, + 0.0019139159994665533, + 0.0018409170006634668, + 0.002019084000494331, + 0.0018234169983770698, + 0.0018077499989885837, + 0.0018631249986356124, + 0.00182304201007355, + 0.001931083999807015, + 0.001976957995793782, + 0.0020425839902600273, + 0.0021450420026667416, + 0.0021282499947119504, + 0.0019258749962318689, + 0.00212583300890401, + 0.002158250004868023, + 0.001961166999535635, + 0.002154457994038239, + 0.0020381670037750155, + 0.0022951249993639067, + 0.001974084007088095, + 0.001932750004925765 + ] + }, + { + "forward": 0.05904670800373424, + "prefill": 0.06082174999755807, + "decode_steps": [ + 0.002500665999832563, + 0.001931541002704762, + 0.0018925420008599758, + 0.0019018330058315769, + 0.001883999997517094, + 0.0018582079937914386, + 0.0018350000027567148, + 0.0019553329912014306, + 0.0018925840122392401, + 0.001855583002907224, + 0.001850499989814125, + 0.0018754579941742122, + 0.0018750409944914281, + 0.0018706249975366518, + 0.001866499995230697, + 0.0018768750014714897, + 0.002006916009122506, + 0.0019850420067086816, + 0.0019124999962514266, + 0.0018471249932190403, + 0.0018877910042647272, + 0.0018688749987632036, + 0.0020429169962881133, + 0.0021920409926678985, + 0.002005457994528115, + 0.002062667001155205, + 0.002077583994832821, + 0.0019785409967880696, + 0.0019772079976974055, + 0.001982040994334966, + 0.0020441249944269657, + 0.0023352919961325824 + ] + }, + { + "forward": 0.057079332997091115, + "prefill": 0.058305665996158496, + "decode_steps": [ + 0.0025042499910341576, + 0.0018859580013668165, + 0.001966874988283962, + 0.002241084002889693, + 0.0018683750095078722, + 0.0018654169980436563, + 0.00192133299424313, + 0.001949917001184076, + 0.0018890829960582778, + 0.0018962500034831464, + 0.0019392909889575094, + 0.0018573750130599365, + 0.0018755420023808256, + 0.0018815420044120401, + 0.0019297919934615493, + 0.001869542000349611, + 0.001919417001772672, + 0.0019820000015897676, + 0.001868917010142468, + 0.0021164999925531447, + 0.002021457999944687, + 0.0018973749974975362, + 0.0018216669996036217, + 0.0018633329891599715, + 0.001833917005569674, + 0.0019779579888563603, + 0.001958082997589372, + 0.0019034160068258643, + 0.0018212919967481866, + 0.0018872090004151687, + 0.0018233750015497208, + 0.0018262499943375587 + ] + }, + { + "forward": 0.056985290997545235, + "prefill": 0.05965366600139532, + "decode_steps": [ + 0.0026076670037582517, + 0.0018539579905336723, + 0.0018242919904878363, + 0.0018129169911844656, + 0.0018097919964930043, + 0.0017957500094780698, + 0.0017747499950928614, + 0.0017928749875864014, + 0.001905917000840418, + 0.0018054579995805398, + 0.0017875420016935095, + 0.0017818329943111166, + 0.0017936670046765357, + 0.0017907919973367825, + 0.0017699579912004992, + 0.001783790998160839, + 0.0017878749931696802, + 0.0018719579966273159, + 0.001794333991711028, + 0.001792416995158419, + 0.0017762499919626862, + 0.001794833005988039, + 0.001802291997591965, + 0.0018133749981643632, + 0.0018644160008989275, + 0.0018472090014256537, + 0.0018860420095734298, + 0.0018676670006243512, + 0.001873917004559189, + 0.0018492919916752726, + 0.0020002500095870346, + 0.0018608750106068328 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 61.17222899774788, + "p95": 80.83392948901745, + "min": 56.985290997545235, + "max": 87.8000419907039, + "iqr": 3.4562705004645977, + "mean": 63.85390819486929, + "cv": 0.14839377165449 + }, + "prefill_ms": { + "count": 10, + "median": 60.9491045033792, + "p95": 77.18428400621632, + "min": 58.305665996158496, + "max": 82.16083400475327, + "iqr": 6.410156755009666, + "mean": 64.29620420240099, + "cv": 0.11553845967591089 + }, + "decode_ms_per_step": { + "count": 320, + "median": 1.892563006549608, + "p95": 2.5897875013470184, + "min": 1.7699579912004992, + "max": 17.703625009744428, + "iqr": 0.161927495355485, + "mean": 2.0557838427976094, + "cv": 0.47161194946554874 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 2067.6206484172817, + "p95": 2152.0147106414483, + "min": 1275.6028949560132, + "max": 2163.113605060071, + "iqr": 112.84480159177588, + "mean": 1989.6308749158688, + "cv": 0.1312832028668834 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1037.3839920109372, + "p95": 1075.1507183127953, + "min": 746.5884030763586, + "max": 1077.1905766533844, + "iqr": 75.80602625606048, + "mean": 996.9280754003505, + "cv": 0.10997033883211972 + }, + "memory": { + "peak_extra_bytes": 39436288, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "reference", + "batch": 4, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.20565854100277647, + "prefill": 0.17739658399659675, + "decode_steps": [ + 0.004038582992507145, + 0.0029977499943925068, + 0.003114125007414259, + 0.002834166996763088, + 0.00337912498798687, + 0.0036482909927144647, + 0.0031828339997446164, + 0.003502083010971546, + 0.0032077079958980903, + 0.0033784589904826134, + 0.0033271659922320396, + 0.0031925829971442, + 0.003520915997796692, + 0.0033881249983096495, + 0.0038518749934155494, + 0.0036894590011797845, + 0.0036023749999003485, + 0.003386875003343448, + 0.003300416996353306, + 0.003555750008672476, + 0.003244042003643699, + 0.003296832990599796, + 0.0032072909962153062, + 0.003139833002933301, + 0.0033434580109314993, + 0.0031799999997019768, + 0.003107832992100157, + 0.0032101660035550594, + 0.0031073330028448254, + 0.0032129169994732365, + 0.003191625000908971, + 0.0034633750037755817 + ] + }, + { + "forward": 0.18226400000276044, + "prefill": 0.1845995000039693, + "decode_steps": [ + 0.004637332996935584, + 0.005240707992925309, + 0.004136124989599921, + 0.0038095829950179905, + 0.0034435409907018766, + 0.0037230829912004992, + 0.0037229160079732537, + 0.003523249994032085, + 0.004128749991650693, + 0.003911792009603232, + 0.0038504169933730736, + 0.0033721670042723417, + 0.003870457992888987, + 0.00332137499935925, + 0.0038438750052591786, + 0.0035050000005867332, + 0.0035033340100198984, + 0.0036648329987656325, + 0.003345707998960279, + 0.003666541990241967, + 0.0032958750089164823, + 0.0033273330045631155, + 0.0036096669937251136, + 0.003515250005875714, + 0.003613458000472747, + 0.0032105840073199943, + 0.0035381659981794655, + 0.003233959010685794, + 0.0033860419935081154, + 0.0035035000037169084, + 0.003314542002044618, + 0.0035566250007832423 + ] + }, + { + "forward": 0.15689312499307562, + "prefill": 0.16649841699108947, + "decode_steps": [ + 0.003937290995963849, + 0.0029861660004826263, + 0.0030552910029655322, + 0.0027759999939007685, + 0.0028851249953731894, + 0.003093207997153513, + 0.002694750000955537, + 0.0030628329986939207, + 0.002962749989819713, + 0.002727541999774985, + 0.0029243330063764006, + 0.003029833998880349, + 0.003086333003011532, + 0.0036261659988667816, + 0.003343874996062368, + 0.0033912090002559125, + 0.003636915993411094, + 0.00326399999903515, + 0.003732207987923175, + 0.0032753749983385205, + 0.003645583987236023, + 0.0033904579904628918, + 0.003442916990024969, + 0.0036255830054869875, + 0.0037219999940134585, + 0.0036342910025268793, + 0.0031072090059751645, + 0.0035195420059608296, + 0.003311249995022081, + 0.003146290997392498, + 0.0034831249940907583, + 0.003083874995354563 + ] + }, + { + "forward": 0.1844525829947088, + "prefill": 0.15776395800639875, + "decode_steps": [ + 0.003642499999841675, + 0.0029445420077536255, + 0.0028098749899072573, + 0.002956250013085082, + 0.002864624999347143, + 0.002744625002378598, + 0.002729166008066386, + 0.002852667006663978, + 0.0030323340033646673, + 0.002852625009836629, + 0.0028267920133657753, + 0.002983916987432167, + 0.002786957993521355, + 0.0027759579970734194, + 0.0029289579979376867, + 0.0028778330015484244, + 0.0032815000013215467, + 0.0037045419885544106, + 0.0036228750104783103, + 0.0029931669996585697, + 0.002657000004546717, + 0.003095709005719982, + 0.0027363749977666885, + 0.002641792001668364, + 0.0025973330048145726, + 0.0029586669988930225, + 0.0028795830003218725, + 0.0029474170005414635, + 0.0026764170033857226, + 0.002930500006186776, + 0.0028451669932110235, + 0.0030576249992009252 + ] + }, + { + "forward": 0.1819812909961911, + "prefill": 0.1444400000036694, + "decode_steps": [ + 0.0033473329967819154, + 0.002562916008173488, + 0.002365624997764826, + 0.0023936249926919118, + 0.0023461249948013574, + 0.002324875007616356, + 0.00231520899978932, + 0.002376332995481789, + 0.002352541996515356, + 0.002325291992747225, + 0.002345416010939516, + 0.002360291007789783, + 0.002319957988220267, + 0.0023148750042309985, + 0.0025095419987337664, + 0.0023702909966232255, + 0.002554042002884671, + 0.0026600000128382817, + 0.0024017919931793585, + 0.002420500008156523, + 0.0023646670015295967, + 0.002370708010857925, + 0.002468332997523248, + 0.0025457919982727617, + 0.00267633399926126, + 0.002550374993006699, + 0.002500665999832563, + 0.002616459009004757, + 0.0025195829948643222, + 0.0025228330050595105, + 0.0025462500052526593, + 0.002527125005144626 + ] + }, + { + "forward": 0.14332658400235232, + "prefill": 0.15016095899045467, + "decode_steps": [ + 0.003499999991618097, + 0.0026379159971838817, + 0.00256995800009463, + 0.002555333005147986, + 0.0024836249940562993, + 0.002570374999777414, + 0.002480292008840479, + 0.002472916996339336, + 0.002502999996067956, + 0.0023859999928390607, + 0.0025098750047618523, + 0.002633666998008266, + 0.0024042500008363277, + 0.002601792002678849, + 0.002745207995758392, + 0.0026594170049065724, + 0.0025318749976577237, + 0.0024699589994270355, + 0.0026015829935204238, + 0.002729375002672896, + 0.0031422080064658076, + 0.002935458003776148, + 0.002746833997662179, + 0.002606333000585437, + 0.0028160000074421987, + 0.0031254579953383654, + 0.0026849580026464537, + 0.0029189580091042444, + 0.002614791999803856, + 0.0025668330054031685, + 0.0026866670086747035, + 0.0026878339995164424 + ] + }, + { + "forward": 0.18718379200436175, + "prefill": 0.15309150000393856, + "decode_steps": [ + 0.003816208991338499, + 0.00302137499966193, + 0.0027641250053420663, + 0.0027624999929685146, + 0.002832083002431318, + 0.003003082994837314, + 0.002719458003412001, + 0.002648916997713968, + 0.0026841660001082346, + 0.002686000007088296, + 0.00281837499642279, + 0.002841249995981343, + 0.00320941700192634, + 0.0028923329955432564, + 0.002813790997606702, + 0.002794375002849847, + 0.003240666992496699, + 0.0037794999952893704, + 0.00291783400462009, + 0.0026397089968668297, + 0.002661207996425219, + 0.0027498749986989424, + 0.0033445000008214265, + 0.003769125003600493, + 0.003746625006897375, + 0.003239833007683046, + 0.003097332999459468, + 0.003568750005797483, + 0.0035302089963806793, + 0.002994708003825508, + 0.003011291002621874, + 0.002954875002615154 + ] + }, + { + "forward": 0.17982837501040194, + "prefill": 0.1503801249928074, + "decode_steps": [ + 0.0031323330040322617, + 0.002574375001131557, + 0.002517499990062788, + 0.0024714589962968603, + 0.0026494580088183284, + 0.0024427089956589043, + 0.0023207920021377504, + 0.002716458009672351, + 0.0025007920048665255, + 0.002424667007289827, + 0.0024239580088760704, + 0.002632250005262904, + 0.002574375001131557, + 0.002509457990527153, + 0.0025804159959079698, + 0.0027542919997358695, + 0.0025653329939814284, + 0.0026569999899948016, + 0.002580874992418103, + 0.002522375012631528, + 0.0025377499987371266, + 0.0026125839940505102, + 0.0025204579869750887, + 0.002529624995077029, + 0.002628125002956949, + 0.00252770799852442, + 0.0025058750034077093, + 0.002653957999427803, + 0.0025772090011741966, + 0.0025091659917961806, + 0.0025379170110682026, + 0.0025076659949263558 + ] + }, + { + "forward": 0.13854683400131762, + "prefill": 0.14665216600405984, + "decode_steps": [ + 0.003555457995389588, + 0.0028239999955985695, + 0.0027058750129071996, + 0.0026765000075101852, + 0.002670624991878867, + 0.0026559580001048744, + 0.0026400420028949156, + 0.0028787909977836534, + 0.0026725829957285896, + 0.0028064590005669743, + 0.0030774580081924796, + 0.0027710839931387454, + 0.0025842499890131876, + 0.002654791998793371, + 0.0026069589948747307, + 0.002577249993919395, + 0.002577916005975567, + 0.002546375006204471, + 0.0025283750001108274, + 0.0026307079970138147, + 0.0025986660039052367, + 0.0025356249971082434, + 0.0025360419967910275, + 0.0025264580035582185, + 0.002537583000957966, + 0.0025479999894741923, + 0.0025690000038594007, + 0.0025173750036628917, + 0.002509250000002794, + 0.0026142499991692603, + 0.002541459005442448, + 0.0025334999954793602 + ] + }, + { + "forward": 0.18128150000120513, + "prefill": 0.15903541700390633, + "decode_steps": [ + 0.0036751250008819625, + 0.0029002090013818815, + 0.0033528750063851476, + 0.002826167008606717, + 0.002961624995805323, + 0.0027769589942181483, + 0.0025784579920582473, + 0.0027445410087238997, + 0.003012667002622038, + 0.0026628749910742044, + 0.002866374998120591, + 0.002632292002090253, + 0.002570999989984557, + 0.002708124986384064, + 0.003054542001336813, + 0.0027160419995198026, + 0.002884292000089772, + 0.002686541003640741, + 0.002575291000539437, + 0.0026846249966183677, + 0.0030487920012092218, + 0.002675083000212908, + 0.002801624999847263, + 0.0029182080033933744, + 0.002814541003317572, + 0.0031426249915966764, + 0.003101957991020754, + 0.0026656670088414103, + 0.002805833995807916, + 0.002972625006805174, + 0.0029861249931855127, + 0.0027601659967331216 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 181.6313954986981, + "p95": 197.34490395348982, + "min": 138.54683400131762, + "max": 205.65854100277647, + "iqr": 21.278499749314506, + "mean": 174.14166250091512, + "cv": 0.12097895289019335 + }, + "prefill_ms": { + "count": 10, + "median": 155.42772900516866, + "p95": 181.35818780065165, + "min": 144.4400000036694, + "max": 184.5995000039693, + "iqr": 14.416916503250832, + "mean": 159.00186259968905, + "cv": 0.08394362144221915 + }, + "decode_ms_per_step": { + "count": 320, + "median": 2.8250835021026433, + "p95": 3.732928838871885, + "min": 2.3148750042309985, + "max": 5.240707992925309, + "iqr": 0.6682712592009921, + "mean": 2.9507724997074547, + "cv": 0.15376549738241718 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1378.1166746962685, + "p95": 1589.9007889622658, + "min": 1090.9792205737022, + "max": 1616.6574926658034, + "iqr": 254.7926212701011, + "mean": 1375.3196245066722, + "cv": 0.12401908069030293 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 510.7290132492932, + "p95": 563.2507102092308, + "min": 423.945935703203, + "max": 572.4105953937445, + "iqr": 67.41074190075312, + "mean": 509.36647214380156, + "cv": 0.0943168115840125 + }, + "memory": { + "peak_extra_bytes": 239550464, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "auto", + "batch": 4, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.6909400419972371 + }, + { + "forward": 0.6721966250042897 + }, + { + "forward": 1.1105696250015171 + }, + { + "forward": 1.0951934160111705 + }, + { + "forward": 0.6761766249983339 + }, + { + "forward": 0.7629807080083992 + }, + { + "forward": 0.8430212499952177 + }, + { + "forward": 0.687946958001703 + }, + { + "forward": 0.6703969579975819 + }, + { + "forward": 0.7732864999998128 + } + ], + "forward_ms": { + "count": 10, + "median": 726.9603750028182, + "p95": 1103.6503309558611, + "min": 670.3969579975819, + "max": 1110.5696250015171, + "iqr": 146.4683542471903, + "mean": 798.2708707015263, + "cv": 0.21308734986426667 + }, + "memory": { + "peak_extra_bytes": 942555136, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "sdpa", + "batch": 4, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.14497108300565742, + "prefill": 0.16008358399267308, + "decode_steps": [ + 0.004405750005389564, + 0.003083124989643693, + 0.003023334007593803, + 0.002915667006163858, + 0.003012708999449387, + 0.0029502909892471507, + 0.0029568750032922253, + 0.0029811249987687916, + 0.003027292012120597, + 0.0030582079925807193, + 0.002970125002320856, + 0.0029721669998252764, + 0.0030107920028967783, + 0.0030075829999987036, + 0.003070291000767611, + 0.0031675420032115653, + 0.0031512079876847565, + 0.0031367500050691888, + 0.003249291010433808, + 0.003116834006505087, + 0.0034694579953793436, + 0.0033264170051552355, + 0.0033646670053713024, + 0.0034532919962657616, + 0.0033932499936781824, + 0.0036246670060791075, + 0.0033698330080369487, + 0.0033971669909078628, + 0.003396750005776994, + 0.0034182079980382696, + 0.0036400829994818196, + 0.0033897910034283996 + ] + }, + { + "forward": 0.15730662499845494, + "prefill": 0.15374333399813622, + "decode_steps": [ + 0.0042091670038644224, + 0.003103166993241757, + 0.0029258750000735745, + 0.003070500009926036, + 0.002942167004221119, + 0.002917584002716467, + 0.0029244170000310987, + 0.003179124993039295, + 0.0031572080042678863, + 0.0030276249890448526, + 0.002974542003357783, + 0.0029265410121297464, + 0.0030971250089351088, + 0.002861582994228229, + 0.0031626249983673915, + 0.003175916994223371, + 0.003076000008150004, + 0.0033297909976681694, + 0.003067917001317255, + 0.0031925000075716525, + 0.003227458000765182, + 0.003097416993114166, + 0.003276916002505459, + 0.0032315420103259385, + 0.0031106249953154474, + 0.0038962499966146424, + 0.0038020830106688663, + 0.0038377500022761524, + 0.0036633749987231568, + 0.003974374994868413, + 0.0036356670025270432, + 0.0036664590006694198 + ] + }, + { + "forward": 0.3856959580007242, + "prefill": 0.3489594999991823, + "decode_steps": [ + 0.006935416007763706, + 0.010861749993637204, + 0.009212457996909507, + 0.007596584007842466, + 0.007837333992938511, + 0.008190125008695759, + 0.005541999998968095, + 0.005401125003118068, + 0.006359457998769358, + 0.010562875002506189, + 0.006181207994814031, + 0.005400666006607935, + 0.006060332991182804, + 0.007942625001305714, + 0.015017249999800697, + 0.014294709006207995, + 0.013060333003522828, + 0.008077833001152612, + 0.00916166600654833, + 0.008017833009944297, + 0.011380083000403829, + 0.0060540830017998815, + 0.00669641699641943, + 0.006521917006466538, + 0.006353167002089322, + 0.016414084006100893, + 0.006688082998152822, + 0.006294459002674557, + 0.005902042001252994, + 0.005671000006259419, + 0.005507125009899028, + 0.006128249995526858 + ] + }, + { + "forward": 0.19834104199253488, + "prefill": 0.18430645899206866, + "decode_steps": [ + 0.005064000011770986, + 0.0033275419991696253, + 0.0033888330071931705, + 0.003364249991136603, + 0.0032153329957509413, + 0.003286542007117532, + 0.003115500003332272, + 0.0034870839881477877, + 0.003373125000507571, + 0.003314499990665354, + 0.003397374995984137, + 0.003226541986805387, + 0.003666292002890259, + 0.0033521250006742775, + 0.0032957079965854064, + 0.003622083007940091, + 0.0032120410032803193, + 0.003522084007272497, + 0.003579666998120956, + 0.003754625009605661, + 0.0035970000026281923, + 0.0038808340032119304, + 0.003987083007814363, + 0.003643375006504357, + 0.00472704200365115, + 0.00488366698846221, + 0.004982875005225651, + 0.0054219580051722005, + 0.005042791992309503, + 0.005204250002861954, + 0.005456000013509765, + 0.0050062499940395355 + ] + }, + { + "forward": 0.17596462501387578, + "prefill": 0.1755002499994589, + "decode_steps": [ + 0.004950250004185364, + 0.0036793339968426153, + 0.0032840829953784123, + 0.0031080830085556954, + 0.003043124990654178, + 0.0030328339926199988, + 0.00301354200928472, + 0.003319833005662076, + 0.0030119159928290173, + 0.0030125420016702265, + 0.003221207996830344, + 0.0030005000007804483, + 0.003096040993114002, + 0.003138583997497335, + 0.0030200830078683794, + 0.003138542000669986, + 0.0030174579878803343, + 0.0031707919988548383, + 0.0031187919958028942, + 0.0030704999953741208, + 0.003167124988976866, + 0.0034132499858969823, + 0.003454874997260049, + 0.0034467919904273003, + 0.0034086670057149604, + 0.00341504201060161, + 0.0034519589971750975, + 0.0035251660010544583, + 0.003497541998513043, + 0.003449167008511722, + 0.003554082999471575, + 0.0035845000093104318 + ] + }, + { + "forward": 0.18134587499662302, + "prefill": 0.1750320409919368, + "decode_steps": [ + 0.005031624998082407, + 0.0033347499993396923, + 0.0036368750006658956, + 0.003094917003181763, + 0.003082875002291985, + 0.00293950000195764, + 0.002922917003161274, + 0.003060417002416216, + 0.0030445409938693047, + 0.0032463750103488564, + 0.0031251249893102795, + 0.003183709006407298, + 0.003276792005635798, + 0.0033252909925067797, + 0.0032155829976545647, + 0.0031892499973764643, + 0.003145333001157269, + 0.0032043329993030056, + 0.0032258749997708946, + 0.0031730829941807315, + 0.0032687089988030493, + 0.0033174580021295696, + 0.003623082986450754, + 0.004035541001940146, + 0.0038422090001404285, + 0.0037636659981217235, + 0.003619833994889632, + 0.003821332997176796, + 0.00417783300508745, + 0.004919875005725771, + 0.004589124990161508, + 0.004739875002996996 + ] + }, + { + "forward": 0.16025733300193679, + "prefill": 0.16343216600944288, + "decode_steps": [ + 0.005126708012539893, + 0.0031479589961236343, + 0.0031829159997869283, + 0.0030924580059945583, + 0.0028872499970020726, + 0.003043292002985254, + 0.0031919170141918585, + 0.0033649170072749257, + 0.0030194169958122075, + 0.0029855419998057187, + 0.0030272500007413328, + 0.0029734580020885915, + 0.00300679198699072, + 0.0033312089944956824, + 0.0037065840006107464, + 0.0032012500014388934, + 0.0038013750017853454, + 0.0038389589899452403, + 0.0034255829959874973, + 0.0032890830043470487, + 0.003679750006995164, + 0.003590874999645166, + 0.0038383749924832955, + 0.0036545420007314533, + 0.0036981250013923272, + 0.0035652919905260205, + 0.0038792920095147565, + 0.0039811659953556955, + 0.0035649580095196143, + 0.003622457996243611, + 0.004468208004254848, + 0.004606208007317036 + ] + }, + { + "forward": 0.1738610000029439, + "prefill": 0.16676112500135787, + "decode_steps": [ + 0.00448399999004323, + 0.0029586669988930225, + 0.003288792009698227, + 0.0029179159901104867, + 0.0029625410097651184, + 0.0029624999879160896, + 0.003158708001137711, + 0.003100832997006364, + 0.0034637079952517524, + 0.0030238339968491346, + 0.0030053329974180087, + 0.0031714170036138967, + 0.00312341601238586, + 0.0034770000056596473, + 0.0035877079935744405, + 0.0033209589892067015, + 0.0034299999970244244, + 0.003212999989045784, + 0.003418416003114544, + 0.00324137500138022, + 0.003609874998801388, + 0.0039095410029403865, + 0.003920750008546747, + 0.0037712920020567253, + 0.0038224169984459877, + 0.0037743749999208376, + 0.00346437499683816, + 0.003868875006446615, + 0.003600209005526267, + 0.0036012500058859587, + 0.004511332997935824, + 0.0037659579975297675 + ] + }, + { + "forward": 0.17309783399105072, + "prefill": 0.18054312499589287, + "decode_steps": [ + 0.005225375003647059, + 0.003478457991150208, + 0.003077333007240668, + 0.0030514170066453516, + 0.00326550001045689, + 0.003596166003262624, + 0.003318332994240336, + 0.0030030000052647665, + 0.0030703339871251956, + 0.003375166008481756, + 0.003244583000196144, + 0.0035721250023925677, + 0.0032718339934945107, + 0.003144834001432173, + 0.0034250839962624013, + 0.0034604999964358285, + 0.0038427079998655245, + 0.003396540996618569, + 0.0034587919944897294, + 0.0036979580036131665, + 0.003656041997601278, + 0.003820583995548077, + 0.0034058749879477546, + 0.0037061250041006133, + 0.003592957989894785, + 0.003915084002073854, + 0.003585583996027708, + 0.0035345840005902573, + 0.004381458013085648, + 0.004576708990498446, + 0.004398082994157448, + 0.004105875006644055 + ] + }, + { + "forward": 0.19173274999775458, + "prefill": 0.2153753339953255, + "decode_steps": [ + 0.005782958003692329, + 0.003831584006547928, + 0.0038131659966893494, + 0.003805291999015026, + 0.004120249999687076, + 0.004167874998529442, + 0.00368345899914857, + 0.0042432500049471855, + 0.004017500003101304, + 0.004603791996487416, + 0.004579250002279878, + 0.005125416995724663, + 0.0043995830055791885, + 0.004343916996731423, + 0.005105832999106497, + 0.0049106669903267175, + 0.014095334001467563, + 0.016933624996454455, + 0.014540957999997772, + 0.014013499996508472, + 0.016877125002793036, + 0.01586683400091715, + 0.00888166599906981, + 0.009260082995751873, + 0.005528790992684662, + 0.007652083993889391, + 0.012010749996989034, + 0.008386417001020163, + 0.004258749992004596, + 0.0044184580037835985, + 0.0041575840004952624, + 0.004184208999504335 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 174.91281250840984, + "p95": 301.3862457970388, + "min": 144.97108300565742, + "max": 385.6959580007242, + "iqr": 25.668572998256423, + "mean": 194.25741250015562, + "cv": 0.35577176955598855 + }, + "prefill_ms": { + "count": 10, + "median": 175.26614549569786, + "p95": 288.8466252974466, + "min": 153.74333399813622, + "max": 348.9594999991823, + "iqr": 19.10121973560308, + "mean": 192.3736917975475, + "cv": 0.2994543921962085 + }, + "decode_ms_per_step": { + "count": 320, + "median": 3.4669164961087517, + "p95": 9.164205606066389, + "min": 2.861582994228229, + "max": 16.933624996454455, + "iqr": 1.0244485092698596, + "mean": 4.3279591377086035, + "cv": 0.5617724096386172 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1132.8926408424784, + "p95": 1230.0503413395852, + "min": 489.81286615626203, + "max": 1236.5890226157503, + "iqr": 153.45752617119706, + "mean": 1025.793110988252, + "cv": 0.266738006076007 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 449.56229383639277, + "p95": 490.87439026062793, + "min": 209.73848867589842, + "max": 495.19385918341436, + "iqr": 44.769785065045994, + "mean": 414.5794154886803, + "cv": 0.22352627771026073 + }, + "memory": { + "peak_extra_bytes": 141557760, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "reference", + "batch": 4, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.6174523749941727, + "prefill": 0.5437997080007335, + "decode_steps": [ + 0.007208540991996415, + 0.005601583994575776, + 0.005914291992667131, + 0.005527459012228064, + 0.005315125003107823, + 0.00522441700741183, + 0.00537208300374914, + 0.005324707992258482, + 0.005183790999581106, + 0.005266332998871803, + 0.005144042006577365, + 0.005350375009584241, + 0.0050858329923357815, + 0.005020291995606385, + 0.005218459002207965, + 0.005085958997369744, + 0.005880209006136283, + 0.005069124992587604, + 0.004997750002075918, + 0.005172375007532537, + 0.005363958000089042, + 0.005532875002245419, + 0.005111459002364427, + 0.005121124995639548, + 0.005330291998689063, + 0.0059771670057671145, + 0.005051958985859528, + 0.00591862500004936, + 0.005488042006618343, + 0.005242290993919596, + 0.005070416998933069, + 0.005279040997265838 + ] + }, + { + "forward": 0.6112949999951525, + "prefill": 0.5386768330063205, + "decode_steps": [ + 0.007291000001714565, + 0.00576033401011955, + 0.005779792001703754, + 0.005556459000217728, + 0.005520792008610442, + 0.006133457995019853, + 0.005981208989396691, + 0.006267709002713673, + 0.006148125001345761, + 0.005926375000854023, + 0.0060655420093098655, + 0.006256958993617445, + 0.006285333001869731, + 0.006070250004995614, + 0.005258582998067141, + 0.005155791994184256, + 0.005693042010534555, + 0.005273125003441237, + 0.0051442909898469225, + 0.005871250003110617, + 0.005371667008148506, + 0.005014333000872284, + 0.005913000000873581, + 0.005517874989891425, + 0.005022457989980467, + 0.0059367499925429, + 0.0054248329979600385, + 0.005039665993535891, + 0.005762666987720877, + 0.005435792001662776, + 0.0055964579951250926, + 0.006078749996959232 + ] + }, + { + "forward": 0.5097834170010174, + "prefill": 0.5746741669863695, + "decode_steps": [ + 0.009263875006581657, + 0.006182500001159497, + 0.0066362910001771525, + 0.006237791996682063, + 0.0071625829878030345, + 0.006890541000757366, + 0.007003791004535742, + 0.008602083995356224, + 0.0066504590067779645, + 0.006857165994006209, + 0.006615209000301547, + 0.006996916999923997, + 0.006057833001250401, + 0.0064641670032870024, + 0.006541916998685338, + 0.006139832999906503, + 0.00618312499136664, + 0.006325207999907434, + 0.006115209005656652, + 0.006644166991463862, + 0.007167999996454455, + 0.006771208005375229, + 0.007137874999898486, + 0.006437834003008902, + 0.0066877500066766515, + 0.006742792000295594, + 0.01110579099622555, + 0.006414874995243736, + 0.006109041001764126, + 0.006385832995874807, + 0.005854040995473042, + 0.006324917005258612 + ] + }, + { + "forward": 0.7357205839944072, + "prefill": 0.6064478750049602, + "decode_steps": [ + 0.007508082999265753, + 0.005984916992019862, + 0.005952791005256586, + 0.005946166987996548, + 0.005592540997895412, + 0.0059522079973248765, + 0.005934917004196905, + 0.005610000007436611, + 0.005396207998273894, + 0.00551020800776314, + 0.005535457996302284, + 0.005766165995737538, + 0.005636000001686625, + 0.005329291001544334, + 0.00537991599412635, + 0.005369916994823143, + 0.005687583005055785, + 0.005694916995707899, + 0.0064035410032374784, + 0.005517624987987801, + 0.005357332993298769, + 0.005442583002150059, + 0.005507249996298924, + 0.005368792000808753, + 0.005767291004303843, + 0.005668458994477987, + 0.005212457996094599, + 0.005367958001443185, + 0.005247583001619205, + 0.005088791993330233, + 0.005467125010909513, + 0.005317000002833083 + ] + }, + { + "forward": 0.6131042500055628, + "prefill": 0.5624999169958755, + "decode_steps": [ + 0.007667749989195727, + 0.00600554198899772, + 0.005733833997510374, + 0.005787208006950095, + 0.005751541000790894, + 0.005372332991100848, + 0.005621708987746388, + 0.005744291003793478, + 0.005276875002891757, + 0.0056812500115484, + 0.00613599999633152, + 0.005395915999542922, + 0.005601249999017455, + 0.005490666997502558, + 0.005214415999944322, + 0.005389875004766509, + 0.00535637499706354, + 0.005087166005978361, + 0.005037167007685639, + 0.0055309159943135455, + 0.005202667001867667, + 0.005240915998001583, + 0.00510966700676363, + 0.005153749996679835, + 0.005132958001922816, + 0.005310749998898245, + 0.005066333003924228, + 0.0054936249944148585, + 0.005446999988635071, + 0.0057478749949950725, + 0.005234249998466112, + 0.005291958004818298 + ] + }, + { + "forward": 0.5508603750058683, + "prefill": 0.5791303750011139, + "decode_steps": [ + 0.008159749995684251, + 0.00571374999708496, + 0.006059875013306737, + 0.005556249991059303, + 0.005398291992605664, + 0.006084666994865984, + 0.00539974999264814, + 0.0055684160033706576, + 0.0060973329964326695, + 0.005611458997009322, + 0.005892332992516458, + 0.005869957996765152, + 0.0057189590006601065, + 0.0061131250113248825, + 0.005921207994106226, + 0.005495541990967467, + 0.005835707997903228, + 0.005660667011397891, + 0.005520374994375743, + 0.005829041998367757, + 0.0059917089965892956, + 0.0054404580005211756, + 0.005734999998821877, + 0.006246541001019068, + 0.006212457999936305, + 0.006204042001627386, + 0.006425416999263689, + 0.006207541999174282, + 0.006319291002000682, + 0.005973834005999379, + 0.005928165992372669, + 0.0064379589894087985 + ] + }, + { + "forward": 0.6392395419970853, + "prefill": 0.5870084589987528, + "decode_steps": [ + 0.00852783300797455, + 0.006449749998864718, + 0.006246207994990982, + 0.005939082999248058, + 0.006303625006694347, + 0.005696541993529536, + 0.006288042000960559, + 0.006586500006960705, + 0.009106665995204821, + 0.007865916995797306, + 0.006824000010965392, + 0.0073627499950816855, + 0.007038291994831525, + 0.007292042006156407, + 0.006775916990591213, + 0.009102916999836452, + 0.006102500003180467, + 0.006709541994496249, + 0.0071904169890331104, + 0.006175374990561977, + 0.0069377909967442974, + 0.00619416699919384, + 0.006940500010387041, + 0.007523958003730513, + 0.006491915992228314, + 0.007159000000683591, + 0.006170332999317907, + 0.005584500002441928, + 0.006222958996659145, + 0.005663749994710088, + 0.006361499996273778, + 0.005646874997182749 + ] + }, + { + "forward": 0.6431018750008661, + "prefill": 0.5554942919989116, + "decode_steps": [ + 0.007138958011637442, + 0.005450000011478551, + 0.005437917003291659, + 0.005234707990894094, + 0.00581975000386592, + 0.00607095799932722, + 0.0052829579944955185, + 0.006007290998240933, + 0.005880624987185001, + 0.005412166006863117, + 0.005618665993097238, + 0.00636875000782311, + 0.005563792001339607, + 0.0052541249897331, + 0.005784166991361417, + 0.0055739999952493235, + 0.00558833300601691, + 0.005878667012439109, + 0.005282208003336564, + 0.005283249993226491, + 0.005696457999874838, + 0.005245166001259349, + 0.005547292006667703, + 0.005312916997354478, + 0.005058458002167754, + 0.005076999994344078, + 0.005309667001711205, + 0.0050360830064164475, + 0.005668584009981714, + 0.005171334007172845, + 0.005037790993810631, + 0.005196666999836452 + ] + }, + { + "forward": 0.5217808330053231, + "prefill": 0.5898621669912245, + "decode_steps": [ + 0.00737325000227429, + 0.005560082994634286, + 0.005637041002046317, + 0.005761708001955412, + 0.005553625000175089, + 0.006123999992269091, + 0.005892415996640921, + 0.00730166700668633, + 0.005820541991852224, + 0.0057806249969871715, + 0.006270916011999361, + 0.005691541009582579, + 0.005621499993139878, + 0.005735416998504661, + 0.005259958008537069, + 0.005249167006695643, + 0.005656541994540021, + 0.005595750000793487, + 0.005806834000395611, + 0.005999916989821941, + 0.005460750006022863, + 0.0054277079907478765, + 0.006203833996551111, + 0.006717833006405272, + 0.006283416994847357, + 0.007006082989391871, + 0.006397707998985425, + 0.00624183400941547, + 0.006774584006052464, + 0.005407916993135586, + 0.005379458001698367, + 0.0055256249906960875 + ] + }, + { + "forward": 0.7285411669872701, + "prefill": 0.6589690410037292, + "decode_steps": [ + 0.007652875006897375, + 0.005787457994301803, + 0.005567708998569287, + 0.00549170799786225, + 0.00539020799624268, + 0.005498208003700711, + 0.005472375007229857, + 0.005746791008277796, + 0.005468041999847628, + 0.005295750001096167, + 0.005614958005025983, + 0.005585332997725345, + 0.006275124993408099, + 0.0059552919992711395, + 0.005543749997741543, + 0.0053211660124361515, + 0.005443791989819147, + 0.0058061669988092035, + 0.005810250004287809, + 0.005997166997985914, + 0.005686500007868744, + 0.0059477920003701, + 0.005861166006070562, + 0.005785959001514129, + 0.0054087089956738055, + 0.0058470829972065985, + 0.005579125005169772, + 0.006570875004399568, + 0.006176459006383084, + 0.006636250007431954, + 0.005966040989733301, + 0.006170125008793548 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 615.2783124998678, + "p95": 732.4898463411955, + "min": 509.78341700101737, + "max": 735.7205839944072, + "iqr": 76.16726049673161, + "mean": 617.0879417986725, + "cv": 0.12400001838601417 + }, + "prefill_ms": { + "count": 10, + "median": 576.9022709937417, + "p95": 635.3345163042832, + "min": 538.6768330063205, + "max": 658.9690410037292, + "iqr": 31.903041744953953, + "mean": 579.6562833987991, + "cv": 0.06040735121250176 + }, + "decode_ms_per_step": { + "count": 320, + "median": 5.7398540011490695, + "p95": 7.291052101936657, + "min": 4.997750002075918, + "max": 11.10579099622555, + "iqr": 0.797198255895637, + "mean": 5.915085658944008, + "cv": 0.12993695617291956 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 692.0928161885954, + "p95": 734.9244615765696, + "min": 585.2482045295109, + "max": 742.2441158382694, + "iqr": 47.19592401306011, + "mean": 680.3286178757055, + "cv": 0.0788992737663982 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 165.15405756395955, + "p95": 178.04267481896522, + "min": 151.42028995758102, + "max": 178.70862359147438, + "iqr": 12.657185708748074, + "mean": 166.8881553134987, + "cv": 0.05291799200013518 + }, + "memory": { + "peak_extra_bytes": 944488448, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "auto", + "batch": 8, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.01188229200488422 + }, + { + "forward": 0.010909707998507656 + }, + { + "forward": 0.0106110839988105 + }, + { + "forward": 0.011764458002289757 + }, + { + "forward": 0.010698583006160334 + }, + { + "forward": 0.010456208008690737 + }, + { + "forward": 0.010696416997234337 + }, + { + "forward": 0.010666249989299104 + }, + { + "forward": 0.010457541997311637 + }, + { + "forward": 0.010634375008521602 + } + ], + "forward_ms": { + "count": 10, + "median": 10.68133349326672, + "p95": 11.829266703716712, + "min": 10.456208008690737, + "max": 11.88229200488422, + "iqr": 0.24001999918255024, + "mean": 10.877691701170988, + "cv": 0.047381096937487786 + }, + "memory": { + "peak_extra_bytes": 21676032, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "sdpa", + "batch": 8, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.010223958001006395, + "prefill": 0.010992042007273994, + "decode_steps": [ + 0.0012674589961534366, + 0.0011580840073293075, + 0.001078791989129968, + 0.0010596249921945855, + 0.001036583009408787, + 0.0010346670023864135, + 0.0010758340067695826, + 0.0010514579917071387, + 0.0010302080045221373, + 0.0010983750107698143, + 0.001048457997967489, + 0.0011265000066487119, + 0.0012155419972259551, + 0.0012935830018250272, + 0.0012732500035781413, + 0.0011665829952107742, + 0.0011273330019321293, + 0.0010523329983698204, + 0.0010610830067889765, + 0.0010993330070050433, + 0.0010432499984744936, + 0.0010645410075085238, + 0.001042584000970237, + 0.0010428340028738603, + 0.001049999991664663, + 0.0011484589922474697, + 0.0011809580028057098, + 0.0012744999985443428, + 0.0012075420090695843, + 0.0012060830049449578, + 0.0011342919897288084, + 0.0010729580098995939 + ] + }, + { + "forward": 0.008376834011869505, + "prefill": 0.008769624997512437, + "decode_steps": [ + 0.0012096249993192032, + 0.0010286249889759347, + 0.001022915996145457, + 0.001014167006360367, + 0.0009933749970514327, + 0.0009981249895645306, + 0.0010536670015426353, + 0.001072415994713083, + 0.001174500008346513, + 0.0010705829918151721, + 0.0010512089938856661, + 0.001050749997375533, + 0.001051875005941838, + 0.0010411670082248747, + 0.001037084002746269, + 0.0011135419917991385, + 0.0010918329935520887, + 0.001036708999890834, + 0.0010894590086536482, + 0.0010588750010356307, + 0.0010443329956615344, + 0.0011170840007252991, + 0.0014063330017961562, + 0.0011802499939221889, + 0.0012285830016480759, + 0.0010800830059451982, + 0.001194125012261793, + 0.0010603749979054555, + 0.001035499997669831, + 0.0010095409961650148, + 0.0010080420033773407, + 0.00098562499624677 + ] + }, + { + "forward": 0.008834167005261406, + "prefill": 0.008985374995972961, + "decode_steps": [ + 0.0014222500030882657, + 0.0011062080011470243, + 0.001048292004270479, + 0.001083499999367632, + 0.001015166999422945, + 0.0010226669983239844, + 0.0010271249921061099, + 0.001003458004561253, + 0.0010143340041395277, + 0.001070708007318899, + 0.0010769999935291708, + 0.0010679999977583066, + 0.001164042012533173, + 0.0011147089971927926, + 0.00104825000744313, + 0.0010425829968880862, + 0.0010614579950924963, + 0.0010664580040611327, + 0.0010719160054577515, + 0.0010468340042280033, + 0.001034375003655441, + 0.0010324169998057187, + 0.0010422080085845664, + 0.001031166990287602, + 0.0010639170068316162, + 0.0010524590034037828, + 0.0010479580087121576, + 0.0010795840062201023, + 0.0011775830062106252, + 0.0010865829972317442, + 0.0011928330059163272, + 0.001091458005248569 + ] + }, + { + "forward": 0.00944300000264775, + "prefill": 0.008694624993950129, + "decode_steps": [ + 0.00142920900543686, + 0.0011298750032437965, + 0.0011505000002216548, + 0.0011957079987041652, + 0.0010304999887011945, + 0.001025624995236285, + 0.0010255420056637377, + 0.0010173750051762909, + 0.0010182910045841709, + 0.0010277500114170834, + 0.0010641670087352395, + 0.001031874999171123, + 0.0010500000062165782, + 0.001051917002769187, + 0.0010772910027299076, + 0.0010832080006366596, + 0.0011576250108191743, + 0.0011039170058211312, + 0.0012401249987306073, + 0.001119874999858439, + 0.0010464999941177666, + 0.0010486249957466498, + 0.001048292004270479, + 0.0010384580091340467, + 0.0010615420032991096, + 0.001064625001163222, + 0.001058416994055733, + 0.001092416001483798, + 0.0011802079970948398, + 0.0010949589923257008, + 0.001063125004293397, + 0.0010468749969732016 + ] + }, + { + "forward": 0.008695666998391971, + "prefill": 0.009626291997847147, + "decode_steps": [ + 0.0013173749903216958, + 0.0010648339957697317, + 0.0010157499928027391, + 0.0010286660108249635, + 0.0010024999937741086, + 0.0010142500104848295, + 0.0010200839897152036, + 0.0010354580008424819, + 0.001090374993509613, + 0.001156874990556389, + 0.0011579169949982315, + 0.0010624170099617913, + 0.0010763749887701124, + 0.0011500000109663233, + 0.0010859169997274876, + 0.0010805829952005297, + 0.001061750008375384, + 0.00102862500352785, + 0.001040499992086552, + 0.0010516670008655638, + 0.0010697910038288683, + 0.001050791994202882, + 0.0010712920047808439, + 0.0010548330028541386, + 0.0012820410047424957, + 0.0011119999981019646, + 0.0010680000123102218, + 0.0010890420089708641, + 0.0011369999992894009, + 0.0011058329982915893, + 0.0010860409965971485, + 0.0010385830100858584 + ] + }, + { + "forward": 0.008388457994442433, + "prefill": 0.00826204199984204, + "decode_steps": [ + 0.0012218329939059913, + 0.001090374993509613, + 0.0011039580131182447, + 0.001014165987726301, + 0.0009969579987227917, + 0.0009972500120056793, + 0.0010361250024288893, + 0.0010126670094905421, + 0.0011886659922311082, + 0.0012753330083796754, + 0.0010706670000217855, + 0.0010425000073155388, + 0.0010505830141482875, + 0.0011434170010033995, + 0.0010495829919818789, + 0.0010347920033382252, + 0.0010512090084375814, + 0.0010267920006299391, + 0.0011530409974511713, + 0.0011062500125262886, + 0.0012013340019620955, + 0.0011349580017849803, + 0.001072333994670771, + 0.0010420839971629903, + 0.0010472080030012876, + 0.0010405420034658164, + 0.0010582499962765723, + 0.001036790999933146, + 0.001062374998582527, + 0.001047458004904911, + 0.0010512080043554306, + 0.0010276669927407056 + ] + }, + { + "forward": 0.008683415988343768, + "prefill": 0.00927925000723917, + "decode_steps": [ + 0.0012257079943083227, + 0.0010387090005679056, + 0.001023166987579316, + 0.0010058329935418442, + 0.0011033750051865354, + 0.0010323329915991053, + 0.0010190000029979274, + 0.0010040839988505468, + 0.0010197090014116839, + 0.0010713749943533912, + 0.001055665998137556, + 0.0010411249968456104, + 0.001051707993610762, + 0.0010386249923612922, + 0.0010155839991057292, + 0.001040250004734844, + 0.0010451669950271025, + 0.0010384159977547824, + 0.001039541995851323, + 0.0010479580087121576, + 0.0010847499943338335, + 0.0011398329952498898, + 0.0010472910071257502, + 0.0010228330065729097, + 0.0010761660087155178, + 0.0010765000042738393, + 0.0010460000048624352, + 0.0010532080050325021, + 0.0010446249943925068, + 0.0010359580046497285, + 0.0010437080054543912, + 0.0010234999936074018 + ] + }, + { + "forward": 0.00859645800665021, + "prefill": 0.008340874992427416, + "decode_steps": [ + 0.0012021660077152774, + 0.0010210000036749989, + 0.0010258330003125593, + 0.0011438750079832971, + 0.001023500008159317, + 0.0010124999971594661, + 0.0010285829921485856, + 0.0010299590067006648, + 0.0010121249943040311, + 0.0009912499954225495, + 0.0009986249933717772, + 0.0010058329935418442, + 0.0009956249996321276, + 0.0009938339935615659, + 0.0009916249982779846, + 0.000996250004391186, + 0.0010638329986250028, + 0.0010056659957626835, + 0.0011194580001756549, + 0.0010311250080121681, + 0.001033583001117222, + 0.001195709002786316, + 0.0012372499913908541, + 0.0011242079926887527, + 0.001065750009729527, + 0.001023290999000892, + 0.001113042002543807, + 0.0010344579932279885, + 0.0010087499977089465, + 0.0010096670011989772, + 0.0010482090001460165, + 0.0010115839977515861 + ] + }, + { + "forward": 0.008607333991676569, + "prefill": 0.008426792002865113, + "decode_steps": [ + 0.0012127079971833155, + 0.0010584170086076483, + 0.0010753750102594495, + 0.0010334999969927594, + 0.0010339169966755435, + 0.0009865000029094517, + 0.0010149169975193217, + 0.0010169580054935068, + 0.0010054580052383244, + 0.0010298749984940514, + 0.0010460409976076335, + 0.0010279590060235932, + 0.0010502910008653998, + 0.0011719999893102795, + 0.001057417000993155, + 0.0010301669972250238, + 0.001038374990457669, + 0.0010382920008851215, + 0.001064666997990571, + 0.0010615419887471944, + 0.0010454159928485751, + 0.001030750005156733, + 0.0010405829962110147, + 0.0010322909947717562, + 0.0010401250037830323, + 0.001036832996760495, + 0.0010487090039532632, + 0.001034000000800006, + 0.0010454160074004903, + 0.0011860000086016953, + 0.0010877919994527474, + 0.0010502079967409372 + ] + }, + { + "forward": 0.009442665992537513, + "prefill": 0.008824415999697521, + "decode_steps": [ + 0.0012237080081831664, + 0.0010306250042049214, + 0.0011249160015722737, + 0.0010827920050360262, + 0.0010800830059451982, + 0.0010680419945856556, + 0.0010847920057130978, + 0.001023875011014752, + 0.001058582987752743, + 0.0010287920013070107, + 0.0010422919876873493, + 0.0010265839955536649, + 0.0010349999938625842, + 0.001060791008058004, + 0.001028667000355199, + 0.0010672920034267008, + 0.0010712079965742305, + 0.0010885410010814667, + 0.0012907500058645383, + 0.0010562920069787651, + 0.0010701669962145388, + 0.0010664580040611327, + 0.0010595839994493872, + 0.001022374999593012, + 0.001044667005771771, + 0.0010331249941373244, + 0.001039499999023974, + 0.0010400410101283342, + 0.0010456250020070001, + 0.0010432090057292953, + 0.001043124997522682, + 0.0010509160056244582 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 8.68954149336787, + "p95": 9.872526901745005, + "min": 8.376834011869505, + "max": 10.223958001006395, + "iqr": 0.6913642428116873, + "mean": 8.929195799282752, + "cv": 0.06614852214546856 + }, + "prefill_ms": { + "count": 10, + "median": 8.79702049860498, + "p95": 10.377454503031911, + "min": 8.26204199984204, + "max": 10.992042007273994, + "iqr": 0.712031003786251, + "mean": 9.020133399462793, + "cv": 0.0899090399445781 + }, + "decode_ms_per_step": { + "count": 320, + "median": 1.0512084991205484, + "p95": 1.22192674461985, + "min": 0.98562499624677, + "max": 1.42920900543686, + "iqr": 0.05526000677491538, + "mean": 1.0753928318536055, + "cv": 0.06644215174313309 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 7420.73404355727, + "p95": 7614.008349857239, + "min": 7146.230060627882, + "max": 7619.463517435138, + "iqr": 185.735487184902, + "mean": 7441.707310632955, + "cv": 0.019470443951358544 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 5930.403935296079, + "p95": 6089.747457871958, + "min": 5468.318076624379, + "max": 6104.097614394853, + "iqr": 103.64603560252908, + "mean": 5899.291380175979, + "cv": 0.03040277626162225 + }, + "memory": { + "peak_extra_bytes": 19120128, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "reference", + "batch": 8, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.01089124999998603, + "prefill": 0.011426832992583513, + "decode_steps": [ + 0.0016606250137556344, + 0.0014266249927459285, + 0.0014186670014169067, + 0.0014997499965829775, + 0.001691415993263945, + 0.0016478750039823353, + 0.0015570000105071813, + 0.0014524160069413483, + 0.0014490000030491501, + 0.0014397909981198609, + 0.0014579580019926652, + 0.0014402920060092583, + 0.00145945799886249, + 0.0014584159944206476, + 0.001541542005725205, + 0.0017086669977288693, + 0.0015414170047733933, + 0.0015056669944897294, + 0.0014776669995626435, + 0.0014720409963047132, + 0.0014483330014627427, + 0.0014563750009983778, + 0.0014720000035595149, + 0.0013727500045206398, + 0.0014116249949438497, + 0.0014633749960921705, + 0.0015561670006718487, + 0.0015559169987682253, + 0.0014051250036573038, + 0.0014186250045895576, + 0.0014150420029181987, + 0.0013788340002065524 + ] + }, + { + "forward": 0.010219000003417023, + "prefill": 0.01024733298982028, + "decode_steps": [ + 0.0016073750011855736, + 0.001510707996203564, + 0.0015886670007603243, + 0.001574709007400088, + 0.0014638329885201529, + 0.0014322500064736232, + 0.0014235830021789297, + 0.0013995839981362224, + 0.001415457998518832, + 0.0014377079933183268, + 0.0014328749966807663, + 0.0014298330061137676, + 0.001521291007520631, + 0.0016448749956907704, + 0.001642333998461254, + 0.001516417003585957, + 0.001446541995392181, + 0.0013786250056000426, + 0.0014057910011615604, + 0.0013911250134697184, + 0.001396084000589326, + 0.0013922090001869947, + 0.0014055000065127388, + 0.0014004169934196398, + 0.0016078330081654713, + 0.0015104170015547425, + 0.001642041010200046, + 0.001473165990319103, + 0.001451667005312629, + 0.0014397920022020116, + 0.0014300419861683622, + 0.001394332997733727 + ] + }, + { + "forward": 0.009503333989414386, + "prefill": 0.010369790994445793, + "decode_steps": [ + 0.001614916996913962, + 0.0014905410062056035, + 0.0014384590031113476, + 0.0014392080047400668, + 0.0014674159901915118, + 0.0015369159955298528, + 0.0015752909966977313, + 0.001484042004449293, + 0.0014267920050770044, + 0.0014973340003052726, + 0.001471874988055788, + 0.0014346670068334788, + 0.001449749994208105, + 0.0014291669940575957, + 0.0015679170028306544, + 0.0014517920062644407, + 0.0015612080023856834, + 0.0014882909890729934, + 0.001619415997993201, + 0.0014312499988591298, + 0.0013961250078864396, + 0.0014265840000007302, + 0.0013945840037195012, + 0.0013708339974982664, + 0.001414500002283603, + 0.0014111670025158674, + 0.0014037079963600263, + 0.00147050000668969, + 0.0016673330101184547, + 0.0015554580022580922, + 0.00158170799841173, + 0.001521375001175329 + ] + }, + { + "forward": 0.010275082997395657, + "prefill": 0.010455124996951781, + "decode_steps": [ + 0.0016158340004039928, + 0.0014352500002132729, + 0.0014140419953037053, + 0.0014208329957909882, + 0.0013986249978188425, + 0.0014382919907802716, + 0.00154020800255239, + 0.0015060000005178154, + 0.0015739579976070672, + 0.001512375005404465, + 0.0014696669968543574, + 0.0014492909976979718, + 0.0014601250004488975, + 0.0014668340008938685, + 0.0014397500053746626, + 0.0014347079995786771, + 0.0014727499947184697, + 0.001596165995579213, + 0.001515874988399446, + 0.0014712090051034465, + 0.001470875009545125, + 0.0015978750016074628, + 0.0016352079983334988, + 0.0015838749968679622, + 0.001502500002970919, + 0.0015335829957621172, + 0.0014937090018065646, + 0.0014969169860705733, + 0.0017436670023016632, + 0.0015202080103335902, + 0.0014957499952288345, + 0.0015253749879775569 + ] + }, + { + "forward": 0.009861291997367516, + "prefill": 0.009886041007121094, + "decode_steps": [ + 0.0016527910047443584, + 0.0014356659958139062, + 0.001515625001047738, + 0.0015078330034157261, + 0.0013892080023651943, + 0.001464417000534013, + 0.0014584999880753458, + 0.001432792007108219, + 0.0014210000081220642, + 0.0014326250093290582, + 0.001448417009669356, + 0.001493749994551763, + 0.001497749995905906, + 0.0014741670020157471, + 0.0014615830004913732, + 0.0015126250073080882, + 0.0016110410069813952, + 0.0016278330003842711, + 0.0015229999989969656, + 0.001415292004821822, + 0.0016374589904444292, + 0.0015244999958667904, + 0.0014560419949702919, + 0.0013755410036537796, + 0.0014896670036250725, + 0.0015803749993210658, + 0.0015106670034583658, + 0.0015580840117763728, + 0.0014562500000465661, + 0.0014422079984797165, + 0.001436999998986721, + 0.0013879999896744266 + ] + }, + { + "forward": 0.010123667001607828, + "prefill": 0.01034133299253881, + "decode_steps": [ + 0.0016012080013751984, + 0.0014361249923240393, + 0.001440000007278286, + 0.001425374997779727, + 0.001387458003591746, + 0.001556957999127917, + 0.00142937499913387, + 0.0014325840020319447, + 0.001443333996576257, + 0.0014536670059897006, + 0.0014567499893018976, + 0.0014175419928506017, + 0.0014619160065194592, + 0.001450499999918975, + 0.0014487080043181777, + 0.001355708998744376, + 0.0014316250017145649, + 0.0015440829884028062, + 0.0014372919977176934, + 0.0014045420102775097, + 0.0014252920082071796, + 0.0014172079972922802, + 0.0014047500008018687, + 0.0013774580002063885, + 0.0014064580027479678, + 0.001388958000461571, + 0.001404084003297612, + 0.0013912080030422658, + 0.0015100840100785717, + 0.0014460840029641986, + 0.0014057089865673333, + 0.0013778749998891726 + ] + }, + { + "forward": 0.010382207998191006, + "prefill": 0.010510292006074451, + "decode_steps": [ + 0.001598333998117596, + 0.0015466250042663887, + 0.0014923340058885515, + 0.001542541998787783, + 0.00142154200875666, + 0.0014167909976094961, + 0.0014022909890627488, + 0.00140974999521859, + 0.0014085419970797375, + 0.0014224170008674264, + 0.001452042008168064, + 0.001436624996131286, + 0.0015997080045053735, + 0.001468957998440601, + 0.0014687500079162419, + 0.0014955420047044754, + 0.0015027920017018914, + 0.001448790993890725, + 0.0014716659934492782, + 0.00148862499918323, + 0.001441750006051734, + 0.001455124991480261, + 0.001455292003811337, + 0.001563249999890104, + 0.0014942080015316606, + 0.0014604999887524173, + 0.0014250410022214055, + 0.001441000000340864, + 0.001401207991875708, + 0.0013975830079289153, + 0.001412999990861863, + 0.0013909590052207932 + ] + }, + { + "forward": 0.010563250005361624, + "prefill": 0.011470957993878983, + "decode_steps": [ + 0.0016311670042341575, + 0.0014834580069873482, + 0.001422125002136454, + 0.0014112499920884147, + 0.0013650410110130906, + 0.0013728750054724514, + 0.0014102079876465723, + 0.0013863339991075918, + 0.0014014160115038976, + 0.0014325410011224449, + 0.001514916992164217, + 0.001354125008219853, + 0.001406124996719882, + 0.0014444160042330623, + 0.0013919589982833713, + 0.001356792010483332, + 0.001376624990371056, + 0.0013571250019595027, + 0.001392999998643063, + 0.001397207990521565, + 0.0013791670062346384, + 0.0013869169924873859, + 0.0013785419869236648, + 0.0013530830037780106, + 0.0013932909932918847, + 0.0013814160047331825, + 0.0013928750122431666, + 0.0013902079954277724, + 0.001375250009004958, + 0.0013787920033792034, + 0.0014062090049264953, + 0.0013760830042883754 + ] + }, + { + "forward": 0.009568334004143253, + "prefill": 0.00945404201047495, + "decode_steps": [ + 0.0015700000076321885, + 0.0014202090096659958, + 0.001393957994878292, + 0.0014064580027479678, + 0.0014854579931125045, + 0.0014159999991534278, + 0.0014154169912217185, + 0.0014296669978648424, + 0.0014104579895501956, + 0.0014578749978682026, + 0.0014173750096233562, + 0.0014127500035101548, + 0.0014660000015283003, + 0.0014960419939598069, + 0.001464166009100154, + 0.0015333750052377582, + 0.0014352080033859238, + 0.0016038330068113282, + 0.0015127910010050982, + 0.0014083749993005767, + 0.0014182079903548583, + 0.0014654999977210537, + 0.0014082920097280294, + 0.0013742500013904646, + 0.0014046669966774061, + 0.0014269590028561652, + 0.001537542004371062, + 0.0014395830075955018, + 0.0014209579967427999, + 0.001416999992216006, + 0.0014709579991176724, + 0.0014023330004420131 + ] + }, + { + "forward": 0.010484083002666011, + "prefill": 0.010213582994765602, + "decode_steps": [ + 0.0016057079919846728, + 0.0014313749998109415, + 0.0014015420019859448, + 0.001400249995640479, + 0.0013630000030389056, + 0.0013753749954048544, + 0.0014033329935045913, + 0.001378374989144504, + 0.0014039170055184513, + 0.0015512499958276749, + 0.0014555829984601587, + 0.0015851669886615127, + 0.0015774159983266145, + 0.0015715839981567115, + 0.0015292500029318035, + 0.0015611249982612208, + 0.0016303329903166741, + 0.0015691669977968559, + 0.001611250001587905, + 0.0015751250030007213, + 0.0015770839963806793, + 0.001575958012836054, + 0.0016165839915629476, + 0.0014706669899169356, + 0.0014810419961577281, + 0.0015562089974991977, + 0.0015537079889327288, + 0.0015237920015351847, + 0.0014361670037033036, + 0.0015750000020489097, + 0.001706041002762504, + 0.0015201250062091276 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 10.24704150040634, + "p95": 10.743650002405047, + "min": 9.503333989414386, + "max": 10.89124999998603, + "iqr": 0.5317285031196661, + "mean": 10.187150099955034, + "cv": 0.043074041468843374 + }, + "prefill_ms": { + "count": 10, + "median": 10.355561993492302, + "p95": 11.451101743296022, + "min": 9.45404201047495, + "max": 11.470957993878983, + "iqr": 0.27447976026451215, + "mean": 10.437533097865526, + "cv": 0.059093352446592966 + }, + "decode_ms_per_step": { + "count": 320, + "median": 1.45012499706354, + "p95": 1.6198368481127545, + "min": 1.3530830037780106, + "max": 1.7436670023016632, + "iqr": 0.10354174082749523, + "mean": 1.471371618890771, + "cv": 0.0528846281265959 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 5406.7546367826035, + "p95": 5641.806322610977, + "min": 5270.471764497361, + "max": 5701.490446018343, + "iqr": 137.54111708681103, + "mean": 5439.83329765247, + "cv": 0.023788509026931276 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 4453.253104277224, + "p95": 4569.263944289355, + "min": 4332.572970569715, + "max": 4588.166610682559, + "iqr": 144.3070532729953, + "mean": 4452.042003061597, + "cv": 0.01953752525929657 + }, + "memory": { + "peak_extra_bytes": 19136512, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "auto", + "batch": 8, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.03377175000787247 + }, + { + "forward": 0.036713791007059626 + }, + { + "forward": 0.03456200000073295 + }, + { + "forward": 0.03423466699314304 + }, + { + "forward": 0.0351170829962939 + }, + { + "forward": 0.03566558301099576 + }, + { + "forward": 0.03630041600263212 + }, + { + "forward": 0.033937166997930035 + }, + { + "forward": 0.03569779099780135 + }, + { + "forward": 0.034667833999264985 + } + ], + "forward_ms": { + "count": 10, + "median": 34.892458497779444, + "p95": 36.52777225506725, + "min": 33.77175000787247, + "max": 36.713791007059626, + "iqr": 1.3732387560594361, + "mean": 35.06680820137262, + "cv": 0.0286012720419252 + }, + "memory": { + "peak_extra_bytes": 41091072, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "sdpa", + "batch": 8, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.025345166999613866, + "prefill": 0.025345124988234602, + "decode_steps": [ + 0.0017437920032534748, + 0.001450916999601759, + 0.0013479580084094778, + 0.0013979580107843503, + 0.0013899580080760643, + 0.001495084012276493, + 0.0014524580037686974, + 0.0014134999946691096, + 0.00143891699553933, + 0.0014316250017145649, + 0.001396958003169857, + 0.0015955840062815696, + 0.00162974999693688, + 0.0014820420037722215, + 0.0014837500057183206, + 0.0013963329984107986, + 0.0014929169992683455, + 0.0015955830021994188, + 0.0014352920115925372, + 0.0014402500091819093, + 0.0014267090009525418, + 0.0013941670040367171, + 0.0016095840110210702, + 0.0015174580039456487, + 0.0014253339904826134, + 0.0014460000093095005, + 0.0015217920008581132, + 0.0015860419953241944, + 0.0016742919979151338, + 0.0015218339976854622, + 0.0014794159942539409, + 0.001465749999624677 + ] + }, + { + "forward": 0.024994707986479625, + "prefill": 0.027253541993559338, + "decode_steps": [ + 0.0018502500024624169, + 0.00160995899932459, + 0.0015682910016039386, + 0.0015750410093460232, + 0.0015690000000176951, + 0.001708915995550342, + 0.0015447910118382424, + 0.0014039999950909987, + 0.0014104169968049973, + 0.0015960830060066655, + 0.0015768750017741695, + 0.0015216659958241507, + 0.001443874993128702, + 0.0014209590008249506, + 0.0015140829927986488, + 0.0014387080009328201, + 0.0014757080061826855, + 0.0014654580008937046, + 0.0014867919962853193, + 0.0015437500114785507, + 0.001683249996858649, + 0.0015235000028042123, + 0.001502125000115484, + 0.0015199580084299669, + 0.001534499999252148, + 0.0015827500028535724, + 0.001528583001345396, + 0.0015899169957265258, + 0.0014909580058883876, + 0.0015060000005178154, + 0.0016191670001717284, + 0.0016241249977611005 + ] + }, + { + "forward": 0.022272665999480523, + "prefill": 0.025603040994610637, + "decode_steps": [ + 0.0016847090009832755, + 0.0014220839948393404, + 0.0013975420006318018, + 0.0014414170000236481, + 0.001377374996081926, + 0.0013437919988064095, + 0.0013289170019561425, + 0.0013273750082589686, + 0.0014382079971255735, + 0.0014085419970797375, + 0.0013847910013282672, + 0.0013521660002879798, + 0.0013712090003537014, + 0.0013622500118799508, + 0.001403457994456403, + 0.001362916998914443, + 0.0014648750075139105, + 0.0013279580016387627, + 0.001350500009721145, + 0.0014359170017996803, + 0.0014590840000892058, + 0.0013702080032089725, + 0.0015346250002039596, + 0.001433416997315362, + 0.0013939589989604428, + 0.0014127500035101548, + 0.0014093749923631549, + 0.001384624993079342, + 0.0014119159895926714, + 0.0014418330101761967, + 0.0014161659928504378, + 0.0013737080007558689 + ] + }, + { + "forward": 0.02534449999802746, + "prefill": 0.025848083998425864, + "decode_steps": [ + 0.0016835839924169704, + 0.0014483330014627427, + 0.0014027500001247972, + 0.0014087089948588982, + 0.0014287499943748116, + 0.00147050000668969, + 0.0014159169950289652, + 0.0014026669960003346, + 0.001440375010133721, + 0.0014496249932562932, + 0.0014610420039389282, + 0.0013771669910056517, + 0.001354541993350722, + 0.0013652499910676852, + 0.00138975000299979, + 0.001392042002407834, + 0.0014962090062908828, + 0.001371082995319739, + 0.0014115829981165007, + 0.001439457992091775, + 0.0013672089989995584, + 0.0015084159967955202, + 0.0015145420038606972, + 0.0014012089959578589, + 0.0014602919982280582, + 0.0014314159925561398, + 0.0014375830069184303, + 0.001552916000946425, + 0.0014881670067552477, + 0.0014047910080989823, + 0.0014306669909274206, + 0.001413209000020288 + ] + }, + { + "forward": 0.025955125005566515, + "prefill": 0.025877250009216368, + "decode_steps": [ + 0.0018709160067373887, + 0.001481333005358465, + 0.001484042004449293, + 0.00154020800255239, + 0.0016265419981209561, + 0.0014369170094141737, + 0.001424624992068857, + 0.001448958006221801, + 0.0014737089950358495, + 0.001546707993838936, + 0.0015094999980647117, + 0.001428999996278435, + 0.0013549160066759214, + 0.0014779170014662668, + 0.0015332499897340313, + 0.0017477079964010045, + 0.0014363340014824644, + 0.0014024580013938248, + 0.0015037499979371205, + 0.001598709000973031, + 0.0016248329920927063, + 0.001507208013208583, + 0.0014808330015512183, + 0.0015622920036548749, + 0.0015080000011948869, + 0.0016276670066872612, + 0.0019242499984102324, + 0.0015679159987485036, + 0.0015277500060619786, + 0.001546833009342663, + 0.00157891699927859, + 0.0016797499993117526 + ] + }, + { + "forward": 0.025530458995490335, + "prefill": 0.024954249995062128, + "decode_steps": [ + 0.0017634160030866042, + 0.0015840409905649722, + 0.0014654170081485063, + 0.0014168750058161095, + 0.0014241669996408746, + 0.0014199580036802217, + 0.0014683750050608069, + 0.001542458005133085, + 0.0014320000045699999, + 0.0013756660046055913, + 0.0014123750006547198, + 0.001442916996893473, + 0.0014072079939069226, + 0.0014739580074092373, + 0.001447250004275702, + 0.001402875001076609, + 0.001411625009495765, + 0.0014305829972727224, + 0.0015802090056240559, + 0.001488958005211316, + 0.0015319579979404807, + 0.0014167500048642978, + 0.001462709013139829, + 0.0015066249907249585, + 0.0015109160012798384, + 0.0015246669936459512, + 0.001461041989387013, + 0.0014600410067941993, + 0.001428207993740216, + 0.0015735410124761984, + 0.0017290000105276704, + 0.0015197079919744283 + ] + }, + { + "forward": 0.026075959001900628, + "prefill": 0.02655175000836607, + "decode_steps": [ + 0.0017507500015199184, + 0.001498833007644862, + 0.0014223339967429638, + 0.0014726670051459223, + 0.0015463339950656518, + 0.0014969170006224886, + 0.0014512089983327314, + 0.0014188339991960675, + 0.0014736669982085004, + 0.001533208996988833, + 0.001590500003658235, + 0.0014570829953299835, + 0.0013655000075232238, + 0.0014465830026892945, + 0.0014574160013580695, + 0.0016100840002764016, + 0.0016130419971887022, + 0.0014125000016065314, + 0.0015019169950392097, + 0.001464458997361362, + 0.0015367080050054938, + 0.0016509160050190985, + 0.001628416997846216, + 0.001501209000707604, + 0.0015045839973026887, + 0.0015339579986175522, + 0.0017099999968195334, + 0.0015351250040112063, + 0.0014637500134995207, + 0.0014515420043608174, + 0.001547082996694371, + 0.0015863750013522804 + ] + }, + { + "forward": 0.025696500000776723, + "prefill": 0.02618108301248867, + "decode_steps": [ + 0.0017802499933168292, + 0.0015883749874774367, + 0.0014896249922458082, + 0.0014513329952023923, + 0.0014171250077197328, + 0.0014900829992257059, + 0.0014904170093359426, + 0.0015812920028110966, + 0.0015390420012408867, + 0.0014674999983981252, + 0.0013651670014951378, + 0.0014467910077655688, + 0.0015156660083448514, + 0.0016527920088265091, + 0.0015065839979797602, + 0.0014902499970048666, + 0.0014747500099474564, + 0.0014351669960888103, + 0.0016736659890739247, + 0.0014896670036250725, + 0.0014662500034319237, + 0.0015665000100852922, + 0.0016278330003842711, + 0.001635666994843632, + 0.001656791995628737, + 0.0016062919894466177, + 0.0016236249939538538, + 0.0015894579992163926, + 0.0015922919992590323, + 0.0014827499981038272, + 0.0015017080004327, + 0.0014851250016363338 + ] + }, + { + "forward": 0.022230083995964378, + "prefill": 0.023459582997020334, + "decode_steps": [ + 0.0017962499987334013, + 0.0014387919945875183, + 0.0014272920088842511, + 0.0016227919986704364, + 0.0016391670069424435, + 0.001419124993844889, + 0.0013541670050472021, + 0.0014384589885594323, + 0.0014129170012893155, + 0.0014412919990718365, + 0.001433750003343448, + 0.0014417079946724698, + 0.001381791997118853, + 0.0014622500020777807, + 0.001409208998666145, + 0.0015074590046424419, + 0.0014713330019731075, + 0.0014647909993072972, + 0.0014316250017145649, + 0.00141033400723245, + 0.0014017920038895682, + 0.0014963330031605437, + 0.0015650000132154673, + 0.0014646250056102872, + 0.0014741670020157471, + 0.0014317080058390275, + 0.0014292909909272566, + 0.0014451669994741678, + 0.0015027089975774288, + 0.0015467500052182004, + 0.0014683330082334578, + 0.0013857909943908453 + ] + }, + { + "forward": 0.025980208010878414, + "prefill": 0.026288458000635728, + "decode_steps": [ + 0.0017520830006105825, + 0.0015046250045998022, + 0.001464792003389448, + 0.0014967090100981295, + 0.0015073749964358285, + 0.0014822499942965806, + 0.001431832992238924, + 0.001461542007746175, + 0.0014705830108141527, + 0.0017530420009279624, + 0.0017823750094976276, + 0.0015008330083219334, + 0.0014783340011490509, + 0.0014135420060483739, + 0.0015014999953564256, + 0.0015868339978624135, + 0.0015172080020420253, + 0.0014781670033698902, + 0.0014137910038698465, + 0.0015452499937964603, + 0.001595084002474323, + 0.0014833330060355365, + 0.0014896670036250725, + 0.0014529589971061796, + 0.0015322090039262548, + 0.0015797090018168092, + 0.0015325419954024255, + 0.0014435829943977296, + 0.0014849169965600595, + 0.0014305840013548732, + 0.0014589589991373941, + 0.0016028750105760992 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 25.4378129975521, + "p95": 26.032871055940632, + "min": 22.230083995964378, + "max": 26.075959001900628, + "iqr": 0.8083127650024835, + "mean": 24.942537599417847, + "cv": 0.058425319517360026 + }, + "prefill_ms": { + "count": 10, + "median": 25.862667003821116, + "p95": 26.937735600222368, + "min": 23.459582997020334, + "max": 27.253541993559338, + "iqr": 0.852010263770353, + "mean": 25.736216599761974, + "cv": 0.03985293030497593 + }, + "decode_ms_per_step": { + "count": 320, + "median": 1.4740625047124922, + "p95": 1.6836402428452857, + "min": 1.3273750082589686, + "max": 1.9242499984102324, + "iqr": 0.1121560089814011, + "mean": 1.4952928852380865, + "cv": 0.06639290962037368 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 5322.431676780781, + "p95": 5630.2391808138045, + "min": 5175.507417064891, + "max": 5685.393255700019, + "iqr": 210.62601990652638, + "mean": 5354.958848388747, + "cv": 0.031997442066361376 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 3463.7939308133364, + "p95": 3628.868836854867, + "min": 3338.419335927897, + "max": 3632.4549789528996, + "iqr": 155.05375841602654, + "mean": 3481.7344201935666, + "cv": 0.029938348036441405 + }, + "memory": { + "peak_extra_bytes": 36503552, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "reference", + "batch": 8, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.03485083399573341, + "prefill": 0.036118832998909056, + "decode_steps": [ + 0.002208167003118433, + 0.0019891660049324855, + 0.0019135829934384674, + 0.0020486670109676197, + 0.0019443750061327592, + 0.0018940409936476499, + 0.0019367089989827946, + 0.0018524999904911965, + 0.0018474160024197772, + 0.001942625007359311, + 0.0019886250083800405, + 0.00209612499747891, + 0.002001665998250246, + 0.001959375003934838, + 0.001970291996258311, + 0.0019208750018151477, + 0.0019662910053739324, + 0.0019959589990321547, + 0.001956374995643273, + 0.0020812079892493784, + 0.002088291003019549, + 0.0020074579952051863, + 0.0019964999955845997, + 0.0020028330036439, + 0.0019672500056913123, + 0.002005292000831105, + 0.0019745419995160773, + 0.00203329099167604, + 0.0020432919991435483, + 0.001923458999954164, + 0.0019655409996630624, + 0.0019524170056683943 + ] + }, + { + "forward": 0.03654366699629463, + "prefill": 0.03686320899578277, + "decode_steps": [ + 0.002391707996139303, + 0.002015459001995623, + 0.0019499589980114251, + 0.0020333749998826534, + 0.0019949169945903122, + 0.0020369999983813614, + 0.0019190409948350862, + 0.0019745000026887283, + 0.001955083993379958, + 0.001984167000046, + 0.001930291997268796, + 0.001918666996061802, + 0.0019005829963134602, + 0.0020044580014655367, + 0.0019275419908808544, + 0.0019420000026002526, + 0.0019410000095376745, + 0.0019104999955743551, + 0.001981583991437219, + 0.002071000009891577, + 0.0019703750003827736, + 0.0020755419973284006, + 0.002017291000811383, + 0.0020743340137414634, + 0.0021514590043807402, + 0.002034624994848855, + 0.0020987500029150397, + 0.002086416003294289, + 0.002051041999948211, + 0.002501250011846423, + 0.0021340419916668907, + 0.002021082997089252 + ] + }, + { + "forward": 0.03502616701007355, + "prefill": 0.0347370840026997, + "decode_steps": [ + 0.0023512080078944564, + 0.002157458002329804, + 0.0021238329936750233, + 0.00192379200598225, + 0.0019409160013310611, + 0.0023123339924495667, + 0.002120374992955476, + 0.0020180419960524887, + 0.0019042500061914325, + 0.0018895840039476752, + 0.0018907080084318295, + 0.0018664170056581497, + 0.0018603750067995861, + 0.0018035840039374307, + 0.0019222499977331609, + 0.0019014580029761419, + 0.0021492499945452437, + 0.0020777080062543973, + 0.0023707919899607077, + 0.0020149170013610274, + 0.001993875004700385, + 0.00208470800134819, + 0.001982333997148089, + 0.0020373330044094473, + 0.002004291003686376, + 0.00255687499884516, + 0.0024910409993026406, + 0.001999958010856062, + 0.00215704100264702, + 0.001931042002979666, + 0.0021511250088224187, + 0.0019673329952638596 + ] + }, + { + "forward": 0.03488637499685865, + "prefill": 0.03667637500620913, + "decode_steps": [ + 0.0021878330007893965, + 0.001992332996451296, + 0.001957665997906588, + 0.001949166995473206, + 0.001981458000955172, + 0.0019772079976974055, + 0.0019480000046314672, + 0.001969707998796366, + 0.0019969169952673838, + 0.0019383750041015446, + 0.0019659999961731955, + 0.0019519589986884966, + 0.0019572500023059547, + 0.0018952500104205683, + 0.001921125003718771, + 0.001956041989615187, + 0.001939624999067746, + 0.0019957910117227584, + 0.0020017500064568594, + 0.001979249995201826, + 0.0019467500096652657, + 0.0019569580035749823, + 0.002021082997089252, + 0.0020368330006022006, + 0.0019880830077454448, + 0.0020635420078178868, + 0.0020044170087203383, + 0.0019717499963007867, + 0.001983624999411404, + 0.0020375829917611554, + 0.0019697499956237152, + 0.002001416010898538 + ] + }, + { + "forward": 0.036465790995862335, + "prefill": 0.038617415993940085, + "decode_steps": [ + 0.0022627500002272427, + 0.002016540995100513, + 0.0019724999874597415, + 0.0020520840043900535, + 0.0020908339938614517, + 0.001974083003005944, + 0.002003166009671986, + 0.0019975410104962066, + 0.0019802080059889704, + 0.0019882920023519546, + 0.0019279170082882047, + 0.0019821669993689284, + 0.0020044999982928857, + 0.001960209003300406, + 0.001904208998894319, + 0.0020413750025909394, + 0.0023896249913377687, + 0.001999041996896267, + 0.001964083989150822, + 0.0021093329996801913, + 0.0020960830006515607, + 0.0020625420002033934, + 0.0019928330002585426, + 0.0021326249989215285, + 0.001982415997190401, + 0.002048708003712818, + 0.0020441249944269657, + 0.0021603750064969063, + 0.0020830829889746383, + 0.00206395800341852, + 0.0020130409975536168, + 0.002704500002437271 + ] + }, + { + "forward": 0.04995962500106543, + "prefill": 0.0333934579975903, + "decode_steps": [ + 0.0025007920048665255, + 0.001980957997147925, + 0.0020862089877482504, + 0.0020391250000102445, + 0.002282041998114437, + 0.0021087500063003972, + 0.002245291994768195, + 0.0019044999935431406, + 0.0019676250085467473, + 0.0019107919943053275, + 0.0018993330013472587, + 0.0019412499968893826, + 0.0019421250035520643, + 0.001936583998030983, + 0.001977332998649217, + 0.001933583989739418, + 0.001967459000297822, + 0.001970625002286397, + 0.0019659579993458465, + 0.0019320420105941594, + 0.0024517909914720803, + 0.0020046249992446974, + 0.0020373330044094473, + 0.0020875420013908297, + 0.0024031249922700226, + 0.0020949579920852557, + 0.0020133749931119382, + 0.002060666010947898, + 0.002074874995741993, + 0.00199012499069795, + 0.002120167002431117, + 0.002226708995294757 + ] + }, + { + "forward": 0.03776008299610112, + "prefill": 0.038413166999816895, + "decode_steps": [ + 0.002352875002543442, + 0.0020837909978581592, + 0.0020452080061659217, + 0.002091959002427757, + 0.002041458006715402, + 0.002043833999778144, + 0.002013250006712042, + 0.0025317089894087985, + 0.00212508300319314, + 0.002058415993815288, + 0.0019856250000884756, + 0.0021057499980088323, + 0.0019549580028979108, + 0.0019159170042257756, + 0.002004750000196509, + 0.0023811250139260665, + 0.002043750006123446, + 0.002011958000366576, + 0.002044708002358675, + 0.0022964160016272217, + 0.0021650000126101077, + 0.002106667001498863, + 0.0021448339975904673, + 0.0023389170091832057, + 0.0020937919907737523, + 0.0021379999961936846, + 0.002160917007131502, + 0.002296500009833835, + 0.0020859159994870424, + 0.00205891698715277, + 0.0023114159994293004, + 0.0020933750056428835 + ] + }, + { + "forward": 0.03510850000020582, + "prefill": 0.0355842919962015, + "decode_steps": [ + 0.0023973750066943467, + 0.0020550409972202033, + 0.0019324580061947927, + 0.001975249993847683, + 0.0019410839886404574, + 0.0020217499986756593, + 0.0019679159886436537, + 0.0019849579985020682, + 0.0019797080021817237, + 0.0018907500052591786, + 0.0018880420102505013, + 0.0019241249974584207, + 0.0019282499997643754, + 0.0018679579952731729, + 0.0019314160017529503, + 0.0019050830014748499, + 0.0019227500015404075, + 0.00191366599756293, + 0.0019674999930430204, + 0.001966791009181179, + 0.0019472500134725124, + 0.002023415989242494, + 0.0020087089942535385, + 0.001987416995689273, + 0.002037083002505824, + 0.0019902089989045635, + 0.0020179999992251396, + 0.001918291993206367, + 0.001931165999849327, + 0.0019763339951168746, + 0.0019676250085467473, + 0.0019604999979492277 + ] + }, + { + "forward": 0.03203079199010972, + "prefill": 0.034439208000549115, + "decode_steps": [ + 0.0022147919953567907, + 0.0019365830085007474, + 0.0019939999911002815, + 0.001992791992961429, + 0.001891417006845586, + 0.001979417007532902, + 0.0019437090086285025, + 0.00197199999820441, + 0.0019184589909855276, + 0.0017718750023050234, + 0.0019226250005885959, + 0.0019164580007782206, + 0.002002583001740277, + 0.0019761250005103648, + 0.001680540997767821, + 0.001877917005913332, + 0.0019792910024989396, + 0.0019136660121148452, + 0.0018386659940006211, + 0.0019743330049095675, + 0.002042833002633415, + 0.001969667006051168, + 0.0018912080122390762, + 0.0020098340028198436, + 0.0021463329903781414, + 0.002096540993079543, + 0.0018422910070512444, + 0.0019757080008275807, + 0.0019665410072775558, + 0.001967916003195569, + 0.001999167012399994, + 0.001993707992369309 + ] + }, + { + "forward": 0.03546345799986739, + "prefill": 0.0354911250033183, + "decode_steps": [ + 0.002292959004989825, + 0.0020073339983355254, + 0.001988290998269804, + 0.0019271250057499856, + 0.0019185830024071038, + 0.001975999999558553, + 0.0020598749979399145, + 0.0019598750077420846, + 0.0019893749995389953, + 0.0019619580125436187, + 0.0020686249918071553, + 0.0020136669918429106, + 0.0018709160067373887, + 0.002024582994636148, + 0.0019904170039808378, + 0.0019539159984560683, + 0.0019817499996861443, + 0.001961166999535635, + 0.0019444589997874573, + 0.002010083000641316, + 0.00198491700575687, + 0.001987374998861924, + 0.002067374996840954, + 0.001995624988921918, + 0.0019852919940603897, + 0.0020185419998597354, + 0.0020776250021299347, + 0.0020271670073270798, + 0.001982583009521477, + 0.0019766670011449605, + 0.0020279169984860346, + 0.002008750001550652 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 35.285979000036605, + "p95": 44.46983109883148, + "min": 32.03079199010972, + "max": 49.95962500106543, + "iqr": 1.602874996024184, + "mean": 36.809529198217206, + "cv": 0.13193725204762058 + }, + "prefill_ms": { + "count": 10, + "median": 35.85156249755528, + "p95": 38.52550394658465, + "min": 33.3934579975903, + "max": 38.617415993940085, + "iqr": 1.8909062455350067, + "mean": 36.033416699501686, + "cv": 0.04637961446400968 + }, + "decode_ms_per_step": { + "count": 320, + "median": 1.9913750002160668, + "p95": 2.313663143286249, + "min": 1.680540997767821, + "max": 2.704500002437271, + "iqr": 0.10561523959040642, + "mean": 2.0244713285592297, + "cv": 0.06474389305895252 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 3967.985732710389, + "p95": 4074.1048196067004, + "min": 3757.699372534046, + "max": 4089.522454656297, + "iqr": 155.3283573833396, + "mean": 3954.1885343729336, + "cv": 0.02655895312582329 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 2563.6850920688075, + "p95": 2618.0043415798295, + "min": 2402.85433441195, + "max": 2638.136211354822, + "iqr": 53.114445452716154, + "mean": 2541.0399459124337, + "cv": 0.027540555155980268 + }, + "memory": { + "peak_extra_bytes": 35946496, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "auto", + "batch": 8, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.20896762500342447 + }, + { + "forward": 0.13424620799196418 + }, + { + "forward": 0.18044254199776333 + }, + { + "forward": 0.13575216599565465 + }, + { + "forward": 0.10365662499680184 + }, + { + "forward": 0.11212854200857691 + }, + { + "forward": 0.09919579100096598 + }, + { + "forward": 0.10987187499995343 + }, + { + "forward": 0.10738649999257177 + }, + { + "forward": 0.10245229200518224 + } + ], + "forward_ms": { + "count": 10, + "median": 111.00020850426517, + "p95": 196.13133765087693, + "min": 99.19579100096598, + "max": 208.96762500342447, + "iqr": 30.786582748987712, + "mean": 129.41001659928588, + "cv": 0.2875860640837468 + }, + "memory": { + "peak_extra_bytes": 138313728, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "sdpa", + "batch": 8, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.10259924999263603, + "prefill": 0.10221266699954867, + "decode_steps": [ + 0.0037423750036396086, + 0.002829291988746263, + 0.0029510419990401715, + 0.003362457995535806, + 0.002765957993688062, + 0.003136583007290028, + 0.0028050410037394613, + 0.0027929170028073713, + 0.002830708006513305, + 0.007068999999319203, + 0.002817957996740006, + 0.002871457996661775, + 0.0029199160053394735, + 0.0031906670046737418, + 0.0032537910010432824, + 0.003371832994162105, + 0.003086083001107909, + 0.0036433339992072433, + 0.0031947920069796965, + 0.0029749999957857653, + 0.003251250003813766, + 0.005175250000320375, + 0.0029459590004989877, + 0.003193209005985409, + 0.0028826250054407865, + 0.0034814999962691218, + 0.0028705410077236593, + 0.0028813749959226698, + 0.002951292000943795, + 0.0029108340095262975, + 0.0030296670011011884, + 0.0029831670108251274 + ] + }, + { + "forward": 0.10299183300230652, + "prefill": 0.08652991699636914, + "decode_steps": [ + 0.0033343749964842573, + 0.0027522089949343354, + 0.0025461249897489324, + 0.002519166999263689, + 0.0025607919960748404, + 0.0025009590026456863, + 0.0024768339935690165, + 0.002509084006305784, + 0.0025778749986784533, + 0.002470624996931292, + 0.0027101250016130507, + 0.0027119580045109615, + 0.0027637500024866313, + 0.0027102909953100607, + 0.003027625003596768, + 0.0027453340007923543, + 0.002731500004301779, + 0.002722417004406452, + 0.0025756249960977584, + 0.0025681659899419174, + 0.002664790998096578, + 0.0026109589962288737, + 0.0026137079985346645, + 0.0025929590046871454, + 0.0025920000043697655, + 0.0026927919971058145, + 0.0031596249900758266, + 0.002989458997035399, + 0.0028009999950882047, + 0.0035332080005900934, + 0.0033617909939493984, + 0.0036665420047938824 + ] + }, + { + "forward": 0.13097583298804238, + "prefill": 0.08889029099373147, + "decode_steps": [ + 0.003409334007301368, + 0.0023818750050850213, + 0.002356833007070236, + 0.002588125003967434, + 0.0023071249888744205, + 0.0022772090014768764, + 0.0023715419956715778, + 0.0027698750054696575, + 0.0024815829965518788, + 0.0037977080064592883, + 0.0029580829868791625, + 0.00252941700455267, + 0.0022959170019021258, + 0.0025170830049319193, + 0.0022442080080509186, + 0.0023959999962244183, + 0.0029717500001424924, + 0.002375916999881156, + 0.002484458003891632, + 0.0025999579956987873, + 0.0027760000084526837, + 0.0025114169984590262, + 0.005979333000141196, + 0.0026590000052237883, + 0.002823083006660454, + 0.004101584010641091, + 0.0029443340026773512, + 0.003789667011005804, + 0.0029567080055130646, + 0.0037015830021118745, + 0.004040959000121802, + 0.004210415994748473 + ] + }, + { + "forward": 0.07336249999934807, + "prefill": 0.06272850000823382, + "decode_steps": [ + 0.0029964169953018427, + 0.0022040409967303276, + 0.002096665994031355, + 0.002093500006594695, + 0.0020257500000298023, + 0.002287915995111689, + 0.0021282500092638656, + 0.0021855000086361542, + 0.00208379200194031, + 0.0021020830026827753, + 0.002061958992271684, + 0.002088749999529682, + 0.0020566249877447262, + 0.0021985839994158596, + 0.00209762500890065, + 0.002220166992628947, + 0.0022019999887561426, + 0.002195000008214265, + 0.0021661250066244975, + 0.002200125003582798, + 0.002286540999193676, + 0.0020485830027610064, + 0.002035249999607913, + 0.002127165993442759, + 0.0021747499995399266, + 0.0020665000047301874, + 0.0020425840048119426, + 0.0021501250012079254, + 0.002459374998579733, + 0.002239749999716878, + 0.0021743329998571426, + 0.0021301249944372103 + ] + }, + { + "forward": 0.06316474999766797, + "prefill": 0.059590042001218535, + "decode_steps": [ + 0.0029047499992884696, + 0.002652082999702543, + 0.0022448329982580617, + 0.002210082995588891, + 0.00218620800296776, + 0.0021720829972764477, + 0.002161292009986937, + 0.002322041997103952, + 0.0021732080058427528, + 0.0021592499979306012, + 0.002156749993446283, + 0.0021934170072199777, + 0.002230458994745277, + 0.002162375007173978, + 0.002170250008930452, + 0.0022895419970154762, + 0.002202250005211681, + 0.0021876670070923865, + 0.002211582992458716, + 0.002188375001423992, + 0.002183792006690055, + 0.0021676250034943223, + 0.002390791996731423, + 0.002210834005381912, + 0.0022861250035930425, + 0.0021917090052738786, + 0.0022475419973488897, + 0.002186457990319468, + 0.002186666999477893, + 0.0022663330018986017, + 0.0021940000005997717, + 0.0022215420030988753 + ] + }, + { + "forward": 0.05964654100534972, + "prefill": 0.05737624999892432, + "decode_steps": [ + 0.002628333997563459, + 0.0019835419952869415, + 0.0018709580035647377, + 0.0018764589913189411, + 0.0018331659957766533, + 0.0018290409934706986, + 0.0019581659871619195, + 0.0018859580013668165, + 0.0018305830017197877, + 0.001815792013076134, + 0.0018717919883783907, + 0.0018313749897060916, + 0.0018572919943835586, + 0.0018229999986942858, + 0.002100166995660402, + 0.0017852919991128147, + 0.0018159169994760305, + 0.0017688329971861094, + 0.0017746660014381632, + 0.001860541000496596, + 0.0018307919963262975, + 0.0018274169997312129, + 0.0018157500016968697, + 0.0019192500039935112, + 0.0018643329967744648, + 0.0018203340005129576, + 0.0018305409903405234, + 0.0018542919860919937, + 0.0019741669966606423, + 0.0018256660114275292, + 0.0018273330060765147, + 0.001823999991756864 + ] + }, + { + "forward": 0.0546015829895623, + "prefill": 0.05999120899650734, + "decode_steps": [ + 0.0026925420097541064, + 0.0020399170025484636, + 0.001909749989863485, + 0.0018652919970918447, + 0.0018521669990150258, + 0.0018283329991390929, + 0.0018498749996069819, + 0.001847583000198938, + 0.001979624998057261, + 0.0018716659978963435, + 0.0018862080032704398, + 0.001843666992499493, + 0.0018243329977849498, + 0.0018168749957112595, + 0.0018303339893464, + 0.0018575830035842955, + 0.001881167001556605, + 0.002070208007353358, + 0.001928750003571622, + 0.0020069169986527413, + 0.001968957993085496, + 0.0020333330030553043, + 0.0019245419971412048, + 0.0019977080082753673, + 0.002079833997413516, + 0.00250791700091213, + 0.002004957990720868, + 0.00211141598992981, + 0.002034291988820769, + 0.0019677919917739928, + 0.0019504999945638701, + 0.0020159170089755207 + ] + }, + { + "forward": 0.0764570419996744, + "prefill": 0.05912995799735654, + "decode_steps": [ + 0.0026374580047558993, + 0.0020131659985054284, + 0.0019668750028358772, + 0.001943083989317529, + 0.0018207500106655061, + 0.0018289589934283867, + 0.0018357919907430187, + 0.001871541011496447, + 0.002013292003539391, + 0.0018579580064397305, + 0.001933666004333645, + 0.002145874997950159, + 0.0019075000018347055, + 0.0019139579962939024, + 0.0019207089935662225, + 0.0018541250028647482, + 0.002004374997341074, + 0.0019892499985871837, + 0.0019281249988125637, + 0.0019066669919993728, + 0.0019137500057695433, + 0.0019231659971410409, + 0.0019038750033359975, + 0.0019457919988781214, + 0.0019189589947927743, + 0.0021458340052049607, + 0.001912791994982399, + 0.0019722919969353825, + 0.0020584169978974387, + 0.002071957991574891, + 0.002018667000811547, + 0.002034666991676204 + ] + }, + { + "forward": 0.05189595799311064, + "prefill": 0.05353149998700246, + "decode_steps": [ + 0.0024599580065114424, + 0.0018917090055765584, + 0.0018334170017624274, + 0.0019474580039968714, + 0.0018361249967711046, + 0.0018225419917143881, + 0.0019025840010726824, + 0.0018283329991390929, + 0.0018354159983573481, + 0.0018275000038556755, + 0.0018169579998357221, + 0.001820416990085505, + 0.0019083329971181229, + 0.0017964580038096756, + 0.0017374169983668253, + 0.0017780420021153986, + 0.0017789999983506277, + 0.001745125002344139, + 0.001735249999910593, + 0.001742124994052574, + 0.0018046659970423207, + 0.0018341250106459484, + 0.0018139579915441573, + 0.0018095840059686452, + 0.001840542012359947, + 0.0018887920014094561, + 0.0018405419978080317, + 0.0018202499923063442, + 0.001821917001507245, + 0.0018386659940006211, + 0.0018344999989494681, + 0.0018304590048501268 + ] + }, + { + "forward": 0.0626210830087075, + "prefill": 0.06379095801094081, + "decode_steps": [ + 0.0026285409985575825, + 0.0020617499976651743, + 0.002011541000683792, + 0.0021780829993076622, + 0.0019859999883919954, + 0.0019590830052038655, + 0.0020467919966904446, + 0.0019425000064074993, + 0.001937124994583428, + 0.0021129999950062484, + 0.0018362499977229163, + 0.0020547080057440326, + 0.0021100830053910613, + 0.0019265419978182763, + 0.0018907909980043769, + 0.0019618340011220425, + 0.0019305409950902686, + 0.0021800410031573847, + 0.0021132090041646734, + 0.0022712920035701245, + 0.0021898339909967035, + 0.0019431669934419915, + 0.002004957990720868, + 0.002019999999902211, + 0.001958625012775883, + 0.0022584169928450137, + 0.0021551249956246465, + 0.002395332994638011, + 0.0020995000086259097, + 0.0020286250073695555, + 0.0020497919904300943, + 0.0021235419990262017 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 68.26362499850802, + "p95": 118.38303299446122, + "min": 51.89595799311064, + "max": 130.97583298804238, + "iqr": 35.67352148820646, + "mean": 77.83163729764055, + "cv": 0.3340024829648121 + }, + "prefill_ms": { + "count": 10, + "median": 61.35985450237058, + "p95": 96.21759779693092, + "min": 53.53149998700246, + "max": 102.21266699954867, + "iqr": 21.600198251690017, + "mean": 69.37712919898331, + "cv": 0.2408353792407154 + }, + "decode_ms_per_step": { + "count": 320, + "median": 2.129187501850538, + "p95": 3.362926745467121, + "min": 1.735249999910593, + "max": 7.068999999319203, + "iqr": 0.6667917550657876, + "mean": 2.315947809120189, + "cv": 0.2663049628961368 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 3767.075135514223, + "p95": 4298.696902517119, + "min": 2457.59508385688, + "max": 4337.353381976492, + "iqr": 1001.233017622541, + "mean": 3583.1160006699547, + "cv": 0.18754106290061687 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1956.7928240270662, + "p95": 2230.625607355973, + "min": 1240.4330267893363, + "max": 2274.4705221194813, + "iqr": 511.1515261002694, + "mean": 1856.5899139661237, + "cv": 0.19322954808094175 + }, + "memory": { + "peak_extra_bytes": 36962304, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "reference", + "batch": 8, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.2881132079928648, + "prefill": 0.2637041670095641, + "decode_steps": [ + 0.012090584001271054, + 0.020052708001458086, + 0.015196832988294773, + 0.013450624988763593, + 0.010970250004902482, + 0.013381958997342736, + 0.010968207992846146, + 0.006315665988950059, + 0.011347124993335456, + 0.01002824999159202, + 0.006481500007794239, + 0.005752833996666595, + 0.009275083997636102, + 0.00540083298983518, + 0.010799958006828092, + 0.0052341249975143, + 0.0054252499976428226, + 0.0047990000020945445, + 0.004247832999681123, + 0.004499249989748932, + 0.007910666987299919, + 0.0054599589930148795, + 0.009185041999444366, + 0.0080063329951372, + 0.008376124998903833, + 0.007369208004092798, + 0.005567582993535325, + 0.0056950830039568245, + 0.005410457990365103, + 0.005183417000807822, + 0.006220833005500026, + 0.005117374996189028 + ] + }, + { + "forward": 0.15233466700010467, + "prefill": 0.12572579200787004, + "decode_steps": [ + 0.004841582995140925, + 0.004132333007873967, + 0.003678625012980774, + 0.0035828339896397665, + 0.0037587919941870496, + 0.0060505000001285225, + 0.0046556249872082844, + 0.004251457998179831, + 0.004377791003207676, + 0.00607862499600742, + 0.004888208000920713, + 0.005343124998034909, + 0.004526916003669612, + 0.0037834999966435134, + 0.003662457995233126, + 0.003902416006894782, + 0.003723957997863181, + 0.00386774999788031, + 0.003691207995871082, + 0.0037077499873703346, + 0.004016291000880301, + 0.003754624995053746, + 0.003669541998533532, + 0.0036421660042833537, + 0.003919958006008528, + 0.003792542003793642, + 0.003670499994768761, + 0.003609667008277029, + 0.0036957080010324717, + 0.0038921660016058013, + 0.0036794579937122762, + 0.003698040993185714 + ] + }, + { + "forward": 0.12374583299970254, + "prefill": 0.11237895899103023, + "decode_steps": [ + 0.004422583006089553, + 0.003519792007864453, + 0.00353729201015085, + 0.0033932500082300976, + 0.003493166994303465, + 0.003548915992723778, + 0.003539832992828451, + 0.0034886670036939904, + 0.0034257920051459223, + 0.0034515000006649643, + 0.0035161669948138297, + 0.0034517919993959367, + 0.0035673749953275546, + 0.003430458004004322, + 0.003825833002338186, + 0.003912333006155677, + 0.00390362499456387, + 0.0038677910051774234, + 0.004972374998033047, + 0.005877291987417266, + 0.007319666998228058, + 0.010388666996732354, + 0.008684540996910073, + 0.006016708008246496, + 0.006295125000178814, + 0.008100582999759354, + 0.006529416001285426, + 0.009303083003032953, + 0.007307457999559119, + 0.006399125006282702, + 0.005626375001156703, + 0.007306166997295804 + ] + }, + { + "forward": 0.10527495900169015, + "prefill": 0.10731545799353626, + "decode_steps": [ + 0.004071249990374781, + 0.0031542920041829348, + 0.0036914999946020544, + 0.0032686670019757003, + 0.003136416999041103, + 0.003252790993428789, + 0.0031496669980697334, + 0.003156958002364263, + 0.0031274169887183234, + 0.003573708003386855, + 0.0037749170005554333, + 0.003531207999913022, + 0.0036227500095264986, + 0.003481584004475735, + 0.003581875003874302, + 0.003653374995337799, + 0.0033784159895731136, + 0.003651541002909653, + 0.003363958006957546, + 0.0034585000103106722, + 0.0033687910035951063, + 0.00340324999706354, + 0.0033791250025387853, + 0.003322542004752904, + 0.0034791669895639643, + 0.003366499993717298, + 0.003432375000556931, + 0.003455458994721994, + 0.003493665994028561, + 0.0035652919905260205, + 0.003487874986603856, + 0.003982959009590559 + ] + }, + { + "forward": 0.09675033300300129, + "prefill": 0.09812183299800381, + "decode_steps": [ + 0.003682250011479482, + 0.0028900420002173632, + 0.0028890829998999834, + 0.0027984579937765375, + 0.0027750829904107377, + 0.0027987909998046234, + 0.00281129201175645, + 0.0027750000008381903, + 0.002786999990348704, + 0.002751542007899843, + 0.0027537910063983873, + 0.0028144999960204586, + 0.002771166997263208, + 0.0027466659958008677, + 0.0027417079982114956, + 0.0027821669937111437, + 0.0028197500068927184, + 0.0029309579986147583, + 0.0029174999945098534, + 0.002796333996229805, + 0.00291837498662062, + 0.0028844170010415837, + 0.0027797910006484017, + 0.0033117499988293275, + 0.0029157500102883205, + 0.002915499993832782, + 0.0028678340022452176, + 0.002808665987686254, + 0.002808499993989244, + 0.002951874994323589, + 0.0028081670025130734, + 0.002847666008165106 + ] + }, + { + "forward": 0.10010100000363309, + "prefill": 0.11930874999961816, + "decode_steps": [ + 0.0037205419939709827, + 0.0029288330115377903, + 0.0027359579980839044, + 0.002719833006267436, + 0.0028904589999001473, + 0.002969209002912976, + 0.0030439999973168597, + 0.0028716249944409356, + 0.002654208001331426, + 0.0026147089956793934, + 0.002854916005162522, + 0.0027177500014659017, + 0.002892583011998795, + 0.00267891600378789, + 0.002622915999381803, + 0.0026061669923365116, + 0.002864458001567982, + 0.002761582989478484, + 0.0028476250008679926, + 0.002922125000623055, + 0.002973042006487958, + 0.0030125419871183112, + 0.0030429580074269325, + 0.0027405840082792565, + 0.0031097079918254167, + 0.0027447920001577586, + 0.0027337089995853603, + 0.0027464580052765086, + 0.003319208000903018, + 0.00330250000115484, + 0.00342287500097882, + 0.0029468749999068677 + ] + }, + { + "forward": 0.09216858299623709, + "prefill": 0.09600216700346209, + "decode_steps": [ + 0.003854416005196981, + 0.003075125001487322, + 0.0030098749994067475, + 0.0030281669896794483, + 0.003005249993293546, + 0.0029474159964593127, + 0.0032022919913288206, + 0.002949916000943631, + 0.0030201669869711623, + 0.002666750006028451, + 0.002649457994266413, + 0.0028226249996805564, + 0.0025962079962482676, + 0.002612124997540377, + 0.0025887079973472282, + 0.0025757090043043718, + 0.00339633300609421, + 0.003900750001776032, + 0.0032392090070061386, + 0.003307334001874551, + 0.003503708998323418, + 0.0032024169922806323, + 0.003386833006516099, + 0.003342916999827139, + 0.0032045420084614307, + 0.003184375003911555, + 0.003157458995701745, + 0.0032763749914010987, + 0.0033929170022020116, + 0.003273041991633363, + 0.0034549999982118607, + 0.0032372499990742654 + ] + }, + { + "forward": 0.09169516699330416, + "prefill": 0.09184779199131299, + "decode_steps": [ + 0.0033225840015802532, + 0.003344249998917803, + 0.0030305410036817193, + 0.002862332999939099, + 0.002900457999203354, + 0.002663499995833263, + 0.0027427079912740737, + 0.0030342920072143897, + 0.002636166987940669, + 0.002925084001617506, + 0.003216458993847482, + 0.0028795830003218725, + 0.0032083749974844977, + 0.003011041000718251, + 0.002916249999543652, + 0.0030009589972905815, + 0.0031854160042712465, + 0.003059665992623195, + 0.0031408750073751435, + 0.002860709006199613, + 0.002961042002425529, + 0.0032217920088442042, + 0.002911332994699478, + 0.0030939999996917322, + 0.0029407499969238415, + 0.0029538329981733114, + 0.002944833002402447, + 0.0032735419954406098, + 0.0029540830000769347, + 0.002981540994369425, + 0.002865000002202578, + 0.0030347920110216364 + ] + }, + { + "forward": 0.09155295799428131, + "prefill": 0.08979937499680091, + "decode_steps": [ + 0.003293749992735684, + 0.002629624999826774, + 0.0026152920036111027, + 0.0025615829945309088, + 0.0025849160010693595, + 0.002612665994092822, + 0.0026106669974979013, + 0.002564499998698011, + 0.0026030000008177012, + 0.0026436249900143594, + 0.0025549999991199, + 0.0026009580033132806, + 0.002627083013067022, + 0.0026820830098586157, + 0.0026526249857852235, + 0.002571875011199154, + 0.0028818330029025674, + 0.00283033300365787, + 0.002723291006986983, + 0.0026167500036535785, + 0.0026269169902661815, + 0.00260920800792519, + 0.0026443329988978803, + 0.00285783399885986, + 0.0030731249862583354, + 0.0028531249990919605, + 0.0028607499989448115, + 0.0031276249937945977, + 0.0030098330025793985, + 0.0033071660000132397, + 0.003187790993251838, + 0.003190374991390854 + ] + }, + { + "forward": 0.09370687499176711, + "prefill": 0.09512745800020639, + "decode_steps": [ + 0.003989125005318783, + 0.0029627089970745146, + 0.0027885410090675578, + 0.0028013330011162907, + 0.002736082999035716, + 0.002670750007382594, + 0.002752250002231449, + 0.002657417004229501, + 0.0025910410040523857, + 0.002611666001030244, + 0.0027900829882128164, + 0.0027780410018749535, + 0.0027434590010670945, + 0.0027872500068042427, + 0.0028914589929627255, + 0.0031279999966500327, + 0.0035412500001257285, + 0.003285375001723878, + 0.003325541998492554, + 0.0031265830039046705, + 0.003165374990203418, + 0.003578625008231029, + 0.003116624997346662, + 0.003219334001187235, + 0.0030763329996261746, + 0.0032250840013148263, + 0.0034910419926745817, + 0.0029220839933259413, + 0.003435249993344769, + 0.0031009160011308268, + 0.003051082996535115, + 0.003160583000862971 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 98.42566650331719, + "p95": 227.0128645461226, + "min": 91.55295799428131, + "max": 288.1132079928648, + "iqr": 26.57495850507985, + "mean": 123.54435829765862, + "cv": 0.4932825261519683 + }, + "prefill_ms": { + "count": 10, + "median": 102.71864549577003, + "p95": 201.61389825880164, + "min": 89.7993749968009, + "max": 263.7041670095641, + "iqr": 22.230166996450862, + "mean": 119.9331750991405, + "cv": 0.43308367865782127 + }, + "decode_ms_per_step": { + "count": 320, + "median": 3.2034795003710315, + "p95": 8.114360099716581, + "min": 2.5549999991199, + "max": 20.052708001458086, + "iqr": 0.8363009947061073, + "mean": 3.876012324280964, + "cv": 0.5466897747954887 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 2591.989366807257, + "p95": 2839.6154019150545, + "min": 965.2365607197752, + "max": 2882.573299259226, + "iqr": 693.2509427975729, + "mean": 2303.942525185472, + "cv": 0.2729750876492158 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1255.695348756643, + "p95": 1401.323135060001, + "min": 484.00136795157056, + "max": 1433.2991404273403, + "iqr": 303.2251188164505, + "mean": 1155.7650540635398, + "cv": 0.24871836496747854 + }, + "memory": { + "peak_extra_bytes": 137707520, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "auto", + "batch": 8, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.44067887499113567 + }, + { + "forward": 0.41294729099899996 + }, + { + "forward": 0.3882039580057608 + }, + { + "forward": 0.451414165989263 + }, + { + "forward": 0.43493379099527374 + }, + { + "forward": 0.44496570799674373 + }, + { + "forward": 0.43004187500628177 + }, + { + "forward": 0.4326918329898035 + }, + { + "forward": 0.4237838749977527 + }, + { + "forward": 0.4097113750030985 + } + ], + "forward_ms": { + "count": 10, + "median": 431.36685399804264, + "p95": 448.5123598926293, + "min": 388.2039580057608, + "max": 451.414165989263, + "iqr": 23.58616699348204, + "mean": 426.93727469741134, + "cv": 0.04432004049422379 + }, + "memory": { + "peak_extra_bytes": 610435072, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "sdpa", + "batch": 8, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.15086804199381731, + "prefill": 0.16864270900259726, + "decode_steps": [ + 0.005124375005834736, + 0.0030691249994561076, + 0.0030290830036392435, + 0.0031361250003101304, + 0.00342304099467583, + 0.004047084003104828, + 0.003092082988587208, + 0.003149374999338761, + 0.0032734580017859116, + 0.00328250000893604, + 0.003711416997248307, + 0.0032924999977694824, + 0.003391124992049299, + 0.0035361670015845448, + 0.003476041994872503, + 0.0037868340004934, + 0.0037280840042512864, + 0.0034966670064022765, + 0.003641791990958154, + 0.004096040996955708, + 0.004152584006078541, + 0.0037725830043200403, + 0.004085291991941631, + 0.004367917004856281, + 0.004849457996897399, + 0.004110082998522557, + 0.004569374999846332, + 0.005335249996278435, + 0.00583862500207033, + 0.0045748330012429506, + 0.005440499997348525, + 0.005320333002600819 + ] + }, + { + "forward": 0.14044541600742377, + "prefill": 0.13694262500212062, + "decode_steps": [ + 0.004546500000287779, + 0.003594166992115788, + 0.0033100000000558794, + 0.0033012919884640723, + 0.003243790997657925, + 0.003193958997144364, + 0.003295667003840208, + 0.003200624996679835, + 0.003079250003793277, + 0.003194749995600432, + 0.0031305419979617, + 0.0032049169967649505, + 0.0031081670022103935, + 0.003538541990565136, + 0.003307667007902637, + 0.0031410840019816533, + 0.003173416989739053, + 0.0030637080053566024, + 0.0031237920047715306, + 0.003996667001047172, + 0.0035537500079954043, + 0.0038688339991495013, + 0.003494499993394129, + 0.003554291994078085, + 0.0036156659916741773, + 0.0035645000025397167, + 0.003566332990885712, + 0.0035574590001488104, + 0.003651208011433482, + 0.0038405410014092922, + 0.0038956670032348484, + 0.003969083001720719 + ] + }, + { + "forward": 0.12929466700006742, + "prefill": 0.15720758298994042, + "decode_steps": [ + 0.00486195799021516, + 0.0037426249909913167, + 0.003283083002315834, + 0.0052820419950876385, + 0.0032456670014653355, + 0.003829916997347027, + 0.0035425420064711943, + 0.0032130409963428974, + 0.0031680410029366612, + 0.0033611249964451417, + 0.00332400000479538, + 0.0033034170046448708, + 0.0038920420047361404, + 0.004419208999024704, + 0.004120125013287179, + 0.003674999999930151, + 0.0037648329889634624, + 0.0033549159998074174, + 0.003637917005107738, + 0.0036337089986773208, + 0.003838666991214268, + 0.003975791987613775, + 0.004215709006530233, + 0.003969042008975521, + 0.0038025420071789995, + 0.003562500001862645, + 0.003845665996777825, + 0.0036915000091539696, + 0.0044340410095173866, + 0.0045243330096127465, + 0.005018665993702598, + 0.005851499998243526 + ] + }, + { + "forward": 0.14088124998670537, + "prefill": 0.14535274999798276, + "decode_steps": [ + 0.004586750001180917, + 0.0030959170107962564, + 0.00300416600657627, + 0.003107124997768551, + 0.003006209008162841, + 0.002961207996122539, + 0.0030203749920474365, + 0.003002124998602085, + 0.0031519170006504282, + 0.003158916995744221, + 0.0031502500060014427, + 0.0033470000053057447, + 0.0034570839925436303, + 0.003625708006438799, + 0.0034208750003017485, + 0.0032866249966900796, + 0.003497915997286327, + 0.0032373329886468127, + 0.0034069160028593615, + 0.00333445800060872, + 0.0032974160130834207, + 0.003590667009120807, + 0.0034155000030295923, + 0.003899041999829933, + 0.0038296249986160547, + 0.003899000003002584, + 0.003628582999226637, + 0.003976000007241964, + 0.003875000009429641, + 0.0050542079989099875, + 0.005260459001874551, + 0.005007166997529566 + ] + }, + { + "forward": 0.1726678749982966, + "prefill": 0.15548624999064486, + "decode_steps": [ + 0.005270458001177758, + 0.003611833992181346, + 0.004640917002689093, + 0.004402749997097999, + 0.004329124989453703, + 0.004830250007216819, + 0.004246333992341533, + 0.0034190000005764887, + 0.0043552499992074445, + 0.0044605829898500815, + 0.003940957991289906, + 0.0036177919973852113, + 0.0035389999975450337, + 0.003436750004766509, + 0.0035500839876476675, + 0.003426749986829236, + 0.004055584009620361, + 0.004249457997502759, + 0.004740417003631592, + 0.0038625000015599653, + 0.0040057500009424984, + 0.004492082996875979, + 0.0038395000010496005, + 0.003858625001157634, + 0.0034509999968577176, + 0.0038008330011507496, + 0.003541333004250191, + 0.0038242500013438985, + 0.004877042010775767, + 0.005192374999751337, + 0.006060083003831096, + 0.004512832994805649 + ] + }, + { + "forward": 0.16524225000466686, + "prefill": 0.1626578330033226, + "decode_steps": [ + 0.004635749995941296, + 0.0030441249982686713, + 0.003115291998255998, + 0.0029689160000998527, + 0.0030377090006368235, + 0.0030371250031748787, + 0.0029117089870851487, + 0.0029910000012023374, + 0.0030424169963225722, + 0.0030732919985894114, + 0.0032416250032838434, + 0.0030180000030668452, + 0.003219457998056896, + 0.003400041998247616, + 0.0034947080130223185, + 0.0035271670058136806, + 0.0032290829985868186, + 0.003464708002866246, + 0.0033940419962164015, + 0.0041154579957947135, + 0.003994458995293826, + 0.003406457995879464, + 0.003726249997271225, + 0.0043072499975096434, + 0.004434624992427416, + 0.004718750002211891, + 0.004738667004858144, + 0.004352500007371418, + 0.004262959002517164, + 0.004656582997995429, + 0.005673749998095445, + 0.004749665997223929 + ] + }, + { + "forward": 0.12684749999607448, + "prefill": 0.12998791699646972, + "decode_steps": [ + 0.004320209001889452, + 0.0032016250042943284, + 0.003006541999639012, + 0.003001417004270479, + 0.003101207999861799, + 0.0030506249895552173, + 0.003252042006351985, + 0.004013917001429945, + 0.004036541999084875, + 0.006196541988174431, + 0.0037417499988805503, + 0.0035351249971427023, + 0.0032852080039447173, + 0.004221666997182183, + 0.004059832994244061, + 0.004492292006034404, + 0.005546665997826494, + 0.004395166994072497, + 0.00462029199115932, + 0.004255584004567936, + 0.0038629579939879477, + 0.00418737500149291, + 0.0037347080069594085, + 0.003765374989598058, + 0.004037582999444567, + 0.003918458998668939, + 0.003795583004830405, + 0.0034593750024214387, + 0.003695832987432368, + 0.005119792011100799, + 0.004882875000475906, + 0.004637000005459413 + ] + }, + { + "forward": 0.16573929198784754, + "prefill": 0.13768608300597407, + "decode_steps": [ + 0.004858500004047528, + 0.003660792004666291, + 0.0035502079990692437, + 0.0035420419881120324, + 0.003898042006767355, + 0.003505917004076764, + 0.0033953340025618672, + 0.0032960410026134923, + 0.0034101670025847852, + 0.0032404160010628402, + 0.0032865840039448813, + 0.003501707993564196, + 0.0031649160082452, + 0.0032700419978937134, + 0.0030931249930290505, + 0.0030617920128861442, + 0.0035080829984508455, + 0.003512957991915755, + 0.003640999988419935, + 0.003569291002349928, + 0.0035319169983267784, + 0.003653500010841526, + 0.0034856250131269917, + 0.0035579579998739064, + 0.0034413339890306816, + 0.0037195420009084046, + 0.0037703329871874303, + 0.003718459003721364, + 0.003548584005329758, + 0.0034945829975185916, + 0.004784916993230581, + 0.004942500003380701 + ] + }, + { + "forward": 0.12485479199676774, + "prefill": 0.17417070800729562, + "decode_steps": [ + 0.0049563329957891256, + 0.003170125011820346, + 0.0031852499960223213, + 0.0032380000047851354, + 0.0037164999957894906, + 0.003267625012085773, + 0.0031633750040782616, + 0.0031184580002445728, + 0.00322099999175407, + 0.0034370000066701323, + 0.0033801250101532787, + 0.0034121250064345077, + 0.0033505840110592544, + 0.0035645830066641793, + 0.0037788750050822273, + 0.0037851670058444142, + 0.0034654160117497668, + 0.0037507919914787635, + 0.00356587499845773, + 0.0036887909955112264, + 0.003797541998210363, + 0.0032925420091487467, + 0.0036261659988667816, + 0.003803624989814125, + 0.004063999993377365, + 0.0038144999998621643, + 0.003908291997504421, + 0.003941999995731749, + 0.0036762499948963523, + 0.003747042006580159, + 0.003885666999849491, + 0.0041584999999031425 + ] + }, + { + "forward": 0.13413237499480601, + "prefill": 0.1302878749993397, + "decode_steps": [ + 0.004167499995674007, + 0.003116791005595587, + 0.0029607079923152924, + 0.0031927079980960116, + 0.0030620420002378523, + 0.002983541999128647, + 0.003000666998559609, + 0.0029578749963548034, + 0.0030552919924957678, + 0.0030582500039599836, + 0.0030247090035118163, + 0.003164207999361679, + 0.003017208000528626, + 0.00302229099906981, + 0.0031174160103546456, + 0.00336625000636559, + 0.003365166994626634, + 0.003255625008023344, + 0.0032507919968338683, + 0.00355520800803788, + 0.0035322500043548644, + 0.0035612500068964437, + 0.004661667000618763, + 0.005013332993257791, + 0.004952582996338606, + 0.004517291992669925, + 0.003989042001194321, + 0.004184083009022288, + 0.005512417003046721, + 0.004691166992415674, + 0.00665941700572148, + 0.006117209006333724 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 140.66333299706457, + "p95": 169.55001264359453, + "min": 124.85479199676774, + "max": 172.6678749982966, + "iqr": 31.144604003202403, + "mean": 145.0973458966473, + "cv": 0.121073401767958 + }, + "prefill_ms": { + "count": 10, + "median": 150.4194999943138, + "p95": 171.68310845518135, + "min": 129.98791699646972, + "max": 174.17070800729562, + "iqr": 24.166780996893067, + "mean": 149.84223329956876, + "cv": 0.10683850226941694 + }, + "decode_ms_per_step": { + "count": 320, + "median": 3.5924170006182976, + "p95": 5.127775005530567, + "min": 2.9117089870851487, + "max": 6.65941700572148, + "iqr": 0.7863857390475459, + "mean": 3.7860853219626733, + "cv": 0.18054219807997254 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 2132.8792963761975, + "p95": 2275.1472176171346, + "min": 1918.4436487431685, + "max": 2308.795292221739, + "iqr": 192.84982430236528, + "mean": 2119.700730346468, + "cv": 0.05898167749511203 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 946.8918765182599, + "p95": 1026.429103629031, + "min": 865.3254167984638, + "max": 1032.995481779215, + "iqr": 114.64702939215738, + "mean": 948.6174864022798, + "cv": 0.06808737200351758 + }, + "memory": { + "peak_extra_bytes": 140705792, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "reference", + "batch": 8, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.41832399999839254, + "prefill": 0.47383841600094456, + "decode_steps": [ + 0.0067202909995103255, + 0.005323082994436845, + 0.005029207997722551, + 0.00618275000306312, + 0.006136291995062493, + 0.0069177910045254976, + 0.007187625000369735, + 0.006249291996937245, + 0.0059013749996665865, + 0.007097124995198101, + 0.007148959004553035, + 0.007955958004458807, + 0.006594124992261641, + 0.006598958003451116, + 0.0065441670012660325, + 0.006020040993462317, + 0.005982999995467253, + 0.0065043749928008765, + 0.0066782499925466254, + 0.006118042001617141, + 0.007998750006663613, + 0.0068169589940225706, + 0.006611499993596226, + 0.0059373749973019585, + 0.006472791996202432, + 0.0057362919906154275, + 0.005604790989309549, + 0.005977833003271371, + 0.006649917006143369, + 0.0056613750057294965, + 0.006526541008497588, + 0.007040790995233692 + ] + }, + { + "forward": 0.3614109170011943, + "prefill": 0.3347488329891348, + "decode_steps": [ + 0.007131874997867271, + 0.006413666007574648, + 0.00907266599824652, + 0.030575124997994862, + 0.013218458014307544, + 0.0065947089897235855, + 0.006376499994075857, + 0.006410083995433524, + 0.006309167001745664, + 0.006111000009695999, + 0.006069042006856762, + 0.00596387499535922, + 0.0062615420029032975, + 0.01098449999699369, + 0.010797332986840047, + 0.007483749999664724, + 0.0072992909990716726, + 0.008132583010592498, + 0.00841249999939464, + 0.008167332998709753, + 0.11777974999858998, + 0.024369040998863056, + 0.006730875000357628, + 0.02210866700625047, + 0.005376582994358614, + 0.005580041994107887, + 0.007216999991214834, + 0.005209082999499515, + 0.005495375007740222, + 0.005637875001411885, + 0.005459665990201756, + 0.005859582990524359 + ] + }, + { + "forward": 0.34793566599546466, + "prefill": 0.33449770900188014, + "decode_steps": [ + 0.007031375003862195, + 0.005593332985881716, + 0.005522249994101003, + 0.005773458993644454, + 0.005535124990274198, + 0.005413458988186903, + 0.005574458002229221, + 0.005583208010648377, + 0.005476874997839332, + 0.00570741698902566, + 0.005535250005777925, + 0.005463000008603558, + 0.005576041992753744, + 0.005428208009107038, + 0.005659083995851688, + 0.0060943340067751706, + 0.0060244170017540455, + 0.005421166992164217, + 0.00548420799896121, + 0.0059377500001573935, + 0.005473416997119784, + 0.0055160829942906275, + 0.005364249998820014, + 0.005289166991133243, + 0.00589600000239443, + 0.005711042002076283, + 0.005397749991971068, + 0.005981125010293908, + 0.006771792002837174, + 0.006986374995904043, + 0.006443958991440013, + 0.006477666989667341 + ] + }, + { + "forward": 0.3816997919930145, + "prefill": 0.32664029199804645, + "decode_steps": [ + 0.007721582995145582, + 0.005862332996912301, + 0.005907833998207934, + 0.005593291003606282, + 0.005730250006308779, + 0.005537584002013318, + 0.005833874995005317, + 0.005586416998994537, + 0.00658816599752754, + 0.006813959000282921, + 0.0060975420055910945, + 0.007413250001263805, + 0.005615374990156852, + 0.0053808749944437295, + 0.005246959000942297, + 0.005800832994282246, + 0.005539792007766664, + 0.006163165991893038, + 0.005473749988595955, + 0.006070000003091991, + 0.00595916599559132, + 0.005475582991493866, + 0.006037374987499788, + 0.006880250002723187, + 0.006728833992383443, + 0.0063239580049412325, + 0.006870499986689538, + 0.0072869170107878745, + 0.006471208995208144, + 0.006108208996010944, + 0.006671958006336354, + 0.006267208009376191 + ] + }, + { + "forward": 0.39456587500171736, + "prefill": 0.3488210000068648, + "decode_steps": [ + 0.007386625002254732, + 0.005763041990576312, + 0.00606141600292176, + 0.0057158330018864945, + 0.006123999992269091, + 0.0065965420071734115, + 0.007018624994088896, + 0.0066634169925237074, + 0.006694666997645982, + 0.006346458001644351, + 0.006337291997624561, + 0.006180916010634974, + 0.006938709004316479, + 0.006604791997233406, + 0.006189874999108724, + 0.006946041990886442, + 0.007340124997426756, + 0.006926916001248173, + 0.007427292002830654, + 0.007491041003959253, + 0.00717416699626483, + 0.007153333004680462, + 0.006058584011043422, + 0.005660209004417993, + 0.0059996250056428835, + 0.006268749988521449, + 0.005549292007344775, + 0.005456250000861473, + 0.005662583003868349, + 0.005462249988340773, + 0.005355291999876499, + 0.005665874996338971 + ] + }, + { + "forward": 0.36502066600951366, + "prefill": 0.3614937079983065, + "decode_steps": [ + 0.0082668749964796, + 0.005955665998044424, + 0.005802415995276533, + 0.005709041986847296, + 0.005747167000663467, + 0.005455666992929764, + 0.005836957992869429, + 0.005642042000545189, + 0.005640541989123449, + 0.005763541994383559, + 0.005609917003312148, + 0.0054997500119498, + 0.005654624997987412, + 0.005639167007757351, + 0.005481457992573269, + 0.005662374998792075, + 0.005560750010772608, + 0.005514250005944632, + 0.006088709007599391, + 0.006091916002333164, + 0.005917458998737857, + 0.005807708002976142, + 0.005857832991750911, + 0.006329124997137114, + 0.00558795798860956, + 0.0060412089951569214, + 0.00571191699418705, + 0.005434666993096471, + 0.005346041987650096, + 0.00570312500349246, + 0.005409959005191922, + 0.005695415995432995 + ] + }, + { + "forward": 0.4624306250043446, + "prefill": 0.3493275830114726, + "decode_steps": [ + 0.007528290996560827, + 0.005898208008147776, + 0.005761584005085751, + 0.005514459000551142, + 0.005861208992428146, + 0.006094249998568557, + 0.006396291006240062, + 0.0058893749956041574, + 0.00650612500612624, + 0.005779792001703754, + 0.005496709010913037, + 0.00582145900989417, + 0.006102415994973853, + 0.0058533749979687855, + 0.006768207997083664, + 0.005760125000961125, + 0.005776625010184944, + 0.005585999999311753, + 0.005654749998939224, + 0.005834750001667999, + 0.005968124998616986, + 0.006165416998555884, + 0.007038750001811422, + 0.006826457989518531, + 0.007310833010706119, + 0.007066332997055724, + 0.0070167920057429, + 0.00721304198668804, + 0.011836458012112416, + 0.009963499993318692, + 0.008585332994698547, + 0.013670917003764771 + ] + }, + { + "forward": 0.3725863750005374, + "prefill": 0.3413963750062976, + "decode_steps": [ + 0.008570374993723817, + 0.006130832989583723, + 0.006529958001920022, + 0.007124290990759619, + 0.006985458006965928, + 0.008839000001898967, + 0.007017708994681016, + 0.006507375001092441, + 0.008652958000311628, + 0.007404584001051262, + 0.007555542004411109, + 0.006596874998649582, + 0.006764874997315928, + 0.008036750004976057, + 0.008615916987764649, + 0.009369667008286342, + 0.008218750008381903, + 0.007231833005789667, + 0.00813920798827894, + 0.007281333004357293, + 0.007403041003271937, + 0.0073324160039192066, + 0.008499083007336594, + 0.007376749999821186, + 0.00741454100352712, + 0.0067420420091366395, + 0.007099249996826984, + 0.008311833007610403, + 0.006736333001754247, + 0.007102332994691096, + 0.007598540993058123, + 0.0066390419960953295 + ] + }, + { + "forward": 0.3142056670039892, + "prefill": 0.34142266600974835, + "decode_steps": [ + 0.007350915999268182, + 0.006019334003212862, + 0.0061556250002468005, + 0.006472042005043477, + 0.010431583999888971, + 0.0068035409931326285, + 0.006744416998117231, + 0.006897875005961396, + 0.0076468750048661605, + 0.006640041989157908, + 0.006468041989137419, + 0.007053917011944577, + 0.00624483400315512, + 0.010264583004754968, + 0.00854895899828989, + 0.009465374998399056, + 0.012131124996813014, + 0.008041332999709994, + 0.006742208992363885, + 0.006935625002370216, + 0.006997042000875808, + 0.008812083004158922, + 0.007298749987967312, + 0.006739750009728596, + 0.007028042004094459, + 0.008056125006987713, + 0.008008457996766083, + 0.006645290995948017, + 0.006794457993237302, + 0.007208334005554207, + 0.007429791003232822, + 0.006654957993305288 + ] + }, + { + "forward": 0.47047229199961293, + "prefill": 0.3835991250089137, + "decode_steps": [ + 0.007538083998952061, + 0.006012124998960644, + 0.005844457991770469, + 0.005839165998622775, + 0.005796750003355555, + 0.005925040997681208, + 0.006089333997806534, + 0.006345208996208385, + 0.0061681660008616745, + 0.0060465419956017286, + 0.0056907079997472465, + 0.005756833008490503, + 0.005746957991505042, + 0.005634000001009554, + 0.005706209005438723, + 0.005780375009635463, + 0.005546417000005022, + 0.005696708001778461, + 0.005459542007884011, + 0.005486958005349152, + 0.005641209005261771, + 0.005466125003295019, + 0.005570875000557862, + 0.005729166005039588, + 0.0059516670007724315, + 0.006352124997647479, + 0.0060437909996835515, + 0.005547583001316525, + 0.005332957996870391, + 0.005871290995855816, + 0.005829582994920202, + 0.005471209005918354 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 377.14308349677594, + "p95": 466.8535418517422, + "min": 314.2056670039892, + "max": 470.47229199961293, + "iqr": 50.07111449594959, + "mean": 388.8651875007781, + "cv": 0.1267368770817817 + }, + "prefill_ms": { + "count": 10, + "median": 345.12183300830657, + "p95": 433.23073505453056, + "min": 326.64029199804645, + "max": 473.83841600094456, + "iqr": 22.041458258172497, + "mean": 359.57857070316095, + "cv": 0.12032411470335605 + }, + "decode_ms_per_step": { + "count": 320, + "median": 6.164291495224461, + "p95": 8.850683301716346, + "min": 5.029207997722551, + "max": 117.77974999858998, + "iqr": 1.3123435091983993, + "mean": 7.057635158753328, + "cv": 0.9352546245245065 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1248.3592012623644, + "p95": 1381.6336631658558, + "min": 648.744197640027, + "max": 1382.71463940979, + "iqr": 258.5537857585148, + "mean": 1187.942419464543, + "cv": 0.18795569144651794 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 449.760481776305, + "p95": 490.9304067737895, + "min": 350.9939157949268, + "max": 492.64804110837383, + "iqr": 26.833720294695922, + "mean": 441.9728321228857, + "cv": 0.10275436647616676 + }, + "memory": { + "peak_extra_bytes": 542228480, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "auto", + "batch": 8, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 1.5470379170001252 + }, + { + "forward": 1.3819939590030117 + }, + { + "forward": 1.4993382910033688 + }, + { + "forward": 1.4812199589941883 + }, + { + "forward": 1.3974327500036452 + }, + { + "forward": 1.3298919999942882 + }, + { + "forward": 1.4968978750112 + }, + { + "forward": 1.4892687910032691 + }, + { + "forward": 1.3364507920050528 + }, + { + "forward": 1.364172375004273 + } + ], + "forward_ms": { + "count": 10, + "median": 1439.3263544989168, + "p95": 1525.5730853015848, + "min": 1329.8919999942882, + "max": 1547.0379170001252, + "iqr": 126.36283300525974, + "mean": 1432.3704709022422, + "cv": 0.05484000705003939 + }, + "memory": { + "peak_extra_bytes": 1923842048, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "sdpa", + "batch": 8, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.31120491700130515, + "prefill": 0.4394486670062179, + "decode_steps": [ + 0.009231917007127777, + 0.0059699159901356325, + 0.005499167004018091, + 0.005768417002400383, + 0.00551616599841509, + 0.005612749999272637, + 0.005617167000309564, + 0.0055768749880371615, + 0.0057862919929903, + 0.006307709001703188, + 0.0065966660040430725, + 0.006586916992091574, + 0.007089542006724514, + 0.00898691700422205, + 0.009597957992809825, + 0.01006283299648203, + 0.010183917009271681, + 0.009389458005898632, + 0.009248000002116896, + 0.008226666002883576, + 0.008085915993433446, + 0.008966459005023353, + 0.009744458991917782, + 0.009815917001105845, + 0.010621958004776388, + 0.011872333008795977, + 0.010865125004784204, + 0.009326833009254187, + 0.014097375009441748, + 0.02474204199097585, + 0.009919042000547051, + 0.009865333006018773 + ] + }, + { + "forward": 0.32665037500555627, + "prefill": 0.3143633330037119, + "decode_steps": [ + 0.007981707996805198, + 0.004911790994810872, + 0.004905374997179024, + 0.004874582999036647, + 0.004965250001987442, + 0.0051445419958326966, + 0.005228624999290332, + 0.005988458986394107, + 0.005950874998234212, + 0.005709625009330921, + 0.005611958011286333, + 0.006107499997597188, + 0.0064334580092690885, + 0.00626291599473916, + 0.006397832999937236, + 0.006572500002221204, + 0.006703499995637685, + 0.007647542006452568, + 0.007188209012383595, + 0.007698375004110858, + 0.007402832998195663, + 0.007412250008201227, + 0.007301709003513679, + 0.008114917000057176, + 0.007630500011146069, + 0.007540500009781681, + 0.007586292005726136, + 0.0073374170024180785, + 0.007591083005536348, + 0.007906082988483831, + 0.007690375001402572, + 0.007709166005952284 + ] + }, + { + "forward": 0.341764458004036, + "prefill": 0.4272034580062609, + "decode_steps": [ + 0.01294345899077598, + 0.009883791004540399, + 0.011663792000035755, + 0.008830042002955452, + 0.00955195800634101, + 0.01737208300619386, + 0.012737874989397824, + 0.01192358300613705, + 0.011131332998047583, + 0.010438833007356152, + 0.008987792010884732, + 0.008150374997057952, + 0.00926879201142583, + 0.012499624994234182, + 0.025178333002259023, + 0.012641125009395182, + 0.00859958300134167, + 0.008826958001009189, + 0.00933679100126028, + 0.009063040997716598, + 0.008664958004374057, + 0.008072208001976833, + 0.007801750005455688, + 0.00861024999176152, + 0.008390707997023128, + 0.00939795799786225, + 0.008941959007643163, + 0.008238041002186947, + 0.008408916997723281, + 0.007861791003961116, + 0.0078869169956306, + 0.00808520900318399 + ] + }, + { + "forward": 0.31541612499859184, + "prefill": 0.3624315420020139, + "decode_steps": [ + 0.008940499988966621, + 0.00543412500701379, + 0.0053008329996373504, + 0.005310916007147171, + 0.005301041004713625, + 0.005681417009327561, + 0.005676125001627952, + 0.00573900000017602, + 0.0055290419986704364, + 0.006257250002818182, + 0.0066314169962424785, + 0.006758708012057468, + 0.006958709011087194, + 0.008219374998589046, + 0.007566083004348911, + 0.008031000004848465, + 0.008468166997772641, + 0.007930874999146909, + 0.007716124993748963, + 0.008270499994978309, + 0.007684541997150518, + 0.007708791992627084, + 0.008000374989933334, + 0.007698040994000621, + 0.007863250008085743, + 0.007699374997173436, + 0.007445374998496845, + 0.007640791009180248, + 0.008026582989259623, + 0.007794542005285621, + 0.007871166002587415, + 0.007393208012217656 + ] + }, + { + "forward": 0.33020570799999405, + "prefill": 0.3589106669969624, + "decode_steps": [ + 0.008719541991013102, + 0.005738959007430822, + 0.005329125007847324, + 0.00523804099066183, + 0.005619999996270053, + 0.005634582994389348, + 0.005587209001532756, + 0.005843000006279908, + 0.005809041002066806, + 0.006581416993867606, + 0.006995791991357692, + 0.0070250420103548095, + 0.007360500007052906, + 0.0077117500040913, + 0.007559999998193234, + 0.008920334003050812, + 0.007673374988371506, + 0.008177957992302254, + 0.007773000004817732, + 0.008439290992100723, + 0.009461792011279613, + 0.008251917010056786, + 0.00818670800072141, + 0.008659708997583948, + 0.00847362499916926, + 0.009024332990520634, + 0.008930416996008717, + 0.008858958011842333, + 0.00935854199633468, + 0.00784258299972862, + 0.007939292001537979, + 0.007490125004551373 + ] + }, + { + "forward": 0.3135928330011666, + "prefill": 0.44269937500939704, + "decode_steps": [ + 0.00991029100259766, + 0.007393209001747891, + 0.006911792006576434, + 0.006646915993769653, + 0.006501874988316558, + 0.00702854098926764, + 0.007925750003778376, + 0.007429792007314973, + 0.008520957999280654, + 0.00858745799632743, + 0.009125917000346817, + 0.009097708010813221, + 0.00943550000374671, + 0.010050915996544063, + 0.008321333007188514, + 0.007831917013390921, + 0.008457292002276517, + 0.008196708993637003, + 0.008456749987090006, + 0.009308124994277023, + 0.009338999996543862, + 0.00819529099680949, + 0.00848108300124295, + 0.008161625009961426, + 0.008251708000898361, + 0.008421792008448392, + 0.008040792003157549, + 0.008121624996419996, + 0.008879457993316464, + 0.008268958001281135, + 0.008209291991079226, + 0.007664958990062587 + ] + }, + { + "forward": 0.3147569580032723, + "prefill": 0.34347729200089816, + "decode_steps": [ + 0.009085832993150689, + 0.005118499990203418, + 0.00519654099480249, + 0.005364208001992665, + 0.005330208005034365, + 0.005651958010275848, + 0.0055886250047478825, + 0.0054797079937998205, + 0.005740167005569674, + 0.006614083002205007, + 0.006294209000770934, + 0.006307375006144866, + 0.006855999992694706, + 0.007358167000347748, + 0.007268792003742419, + 0.007657082998775877, + 0.008472958012134768, + 0.0074886669899569824, + 0.0074478330061538145, + 0.007274749994394369, + 0.007568542001536116, + 0.007637084010639228, + 0.007968958001583815, + 0.008251082996139303, + 0.007533125011832453, + 0.0076202499913051724, + 0.007503749991883524, + 0.007466000010026619, + 0.007553374991402961, + 0.007447832991601899, + 0.007456417006324045, + 0.007466249997378327 + ] + }, + { + "forward": 0.32845504200668074, + "prefill": 0.3144317500118632, + "decode_steps": [ + 0.0079272500006482, + 0.005033583001932129, + 0.004878708001342602, + 0.004946041997754946, + 0.005245375010417774, + 0.0056507499975850806, + 0.005550665999180637, + 0.005750541997258551, + 0.005770417003077455, + 0.005630624989862554, + 0.006265957999858074, + 0.007704208997893147, + 0.008804084005532786, + 0.01019833399914205, + 0.008515583002008498, + 0.0082639169995673, + 0.007946666999487206, + 0.009595250012353063, + 0.008853958002873696, + 0.009578291996149346, + 0.008765167003730312, + 0.00814629100204911, + 0.008060166001087055, + 0.008982083003502339, + 0.008424833999015391, + 0.009645042009651661, + 0.009417124994797632, + 0.007678958005271852, + 0.00872858299408108, + 0.007908625004347414, + 0.009106124998652376, + 0.008503999997628853 + ] + }, + { + "forward": 0.32672108299448155, + "prefill": 0.34265979200426955, + "decode_steps": [ + 0.008469459004118107, + 0.005391458005760796, + 0.005293957990943454, + 0.005213750002440065, + 0.005228375011938624, + 0.005488666996825486, + 0.005499708000570536, + 0.0057958749966928735, + 0.005965290998574346, + 0.006594999998924322, + 0.00634329101012554, + 0.0064754170016385615, + 0.006269707999308594, + 0.0074581250082701445, + 0.007510915995226242, + 0.007265791995450854, + 0.007508541995775886, + 0.0072745840006973594, + 0.0074172079912386835, + 0.007569790992420167, + 0.007544749998487532, + 0.0077705000003334135, + 0.007917791997897439, + 0.007763042012811638, + 0.007779874998959713, + 0.007522790998336859, + 0.007376833993475884, + 0.007663208001758903, + 0.007450958000845276, + 0.007539250000263564, + 0.007451291996403597, + 0.0075654169922927395 + ] + }, + { + "forward": 0.31461650000710506, + "prefill": 0.4013141670002369, + "decode_steps": [ + 0.008354709003469907, + 0.006125499989138916, + 0.006272458995226771, + 0.007015499999397434, + 0.006975791999138892, + 0.006306667011813261, + 0.0065047499956563115, + 0.006552916995133273, + 0.0075781250052386895, + 0.009070457992493175, + 0.007302040990907699, + 0.007638209004653618, + 0.00865608399908524, + 0.00891141701140441, + 0.00875008299772162, + 0.008621209010016173, + 0.009067791994311847, + 0.009506833011982962, + 0.00895529099216219, + 0.009235667006578296, + 0.008925125002861023, + 0.00908016599714756, + 0.009332707995781675, + 0.00942708300135564, + 0.00871624999854248, + 0.008942915999796242, + 0.009559124999213964, + 0.009105292003368959, + 0.009299124998506159, + 0.009499709005467594, + 0.008209833002183586, + 0.0076154580019647256 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 321.03325000207406, + "p95": 336.5630205022171, + "min": 311.20491700130515, + "max": 341.764458004036, + "iqr": 13.369937747484073, + "mean": 322.33839990221895, + "cv": 0.03067567939381647 + }, + "prefill_ms": { + "count": 10, + "median": 360.67110449948814, + "p95": 441.2365564079664, + "min": 314.3633330037119, + "max": 442.69937500939704, + "iqr": 77.86696825132822, + "mean": 374.6940043041832, + "cv": 0.1318169527711922 + }, + "decode_ms_per_step": { + "count": 320, + "median": 7.708978999289684, + "p95": 10.1846378587652, + "min": 4.874582999036647, + "max": 25.178333002259023, + "iqr": 2.1409787477750797, + "mean": 7.8645425599461305, + "cv": 0.27197294176813913 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1054.287189606362, + "p95": 1179.8315861133992, + "min": 777.1946084776102, + "max": 1199.019713033978, + "iqr": 175.2951306076718, + "mean": 1033.88866540676, + "cv": 0.12782559510292768 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 430.2341559656378, + "p95": 472.4795274107061, + "min": 338.35880392046056, + "max": 484.96689570981334, + "iqr": 86.09730775739098, + "mean": 414.60047716330365, + "cv": 0.12320871133988787 + }, + "memory": { + "peak_extra_bytes": 233652224, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "reference", + "batch": 8, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 1.4614228749996983, + "prefill": 1.1245351669931551, + "decode_steps": [ + 0.013599040990811773, + 0.009831999996094964, + 0.00950433399702888, + 0.009569082991220057, + 0.009524167006020434, + 0.00967145900358446, + 0.009681625000666827, + 0.010255417000735179, + 0.009696541994344443, + 0.01041724999959115, + 0.01055537500360515, + 0.010996790995704941, + 0.009862167004030198, + 0.009495457998127677, + 0.009389291997649707, + 0.00959549999970477, + 0.009685124998213723, + 0.009593000009772368, + 0.009503665991360322, + 0.009476292005274445, + 0.009792583005037159, + 0.010792208995553665, + 0.0097609160002321, + 0.009388709004269913, + 0.009139957997831516, + 0.009191083998302929, + 0.008971250004833564, + 0.008975250006187707, + 0.008727166001335718, + 0.009292833012295887, + 0.008900291999452747, + 0.008975207994808443 + ] + }, + { + "forward": 1.2017291249940172, + "prefill": 1.2823072080063866, + "decode_steps": [ + 0.013501832989277318, + 0.009567874993081205, + 0.009759374996065162, + 0.010379415994975716, + 0.009656583002652042, + 0.009679416994913481, + 0.009686457997304387, + 0.010792166998726316, + 0.011254915996687487, + 0.010787707986310124, + 0.010640749998856336, + 0.010919749998720363, + 0.01046154199866578, + 0.010152500006370246, + 0.010784832993522286, + 0.011767250005505048, + 0.01169125000888016, + 0.01013233300182037, + 0.010386875001131557, + 0.009387833008076996, + 0.009343500001705252, + 0.009418208006536588, + 0.010834832995897159, + 0.00964133300294634, + 0.009363208009744994, + 0.009145375006482936, + 0.009156374988378957, + 0.009560083010001108, + 0.009320833007222973, + 0.009019582998007536, + 0.009045249986229464, + 0.00916000000142958 + ] + }, + { + "forward": 1.14174066700798, + "prefill": 1.3128315830108477, + "decode_steps": [ + 0.014558000009856187, + 0.011126124998554587, + 0.011200708002434112, + 0.010783250007079914, + 0.010388208000222221, + 0.0096942499949364, + 0.009450750003452413, + 0.009647208004025742, + 0.0095127499953378, + 0.009471334007685073, + 0.01057083299383521, + 0.009749832999659702, + 0.00977008300833404, + 0.00967404201219324, + 0.009691041996120475, + 0.010766583000076935, + 0.010481999997864477, + 0.0107304999983171, + 0.01135275000706315, + 0.010882916991249658, + 0.011759375003748573, + 0.013185374991735443, + 0.01204887499625329, + 0.012163874998805113, + 0.01045495799917262, + 0.011414833003073, + 0.011259167004027404, + 0.01159516698680818, + 0.011379374991520308, + 0.010572917002718896, + 0.013063500009593554, + 0.011898083001142368 + ] + }, + { + "forward": 1.2450362499948824, + "prefill": 1.3132789579976816, + "decode_steps": [ + 0.013450415994157083, + 0.009917458999552764, + 0.009577667005942203, + 0.00945700000738725, + 0.009500500003923662, + 0.009788166004000232, + 0.01015308299975004, + 0.009760665998328477, + 0.009658459006459452, + 0.009651665997807868, + 0.010249665996525437, + 0.010389542003395036, + 0.01016354099556338, + 0.010128666996024549, + 0.010214499998255633, + 0.010720749996835366, + 0.009667541002272628, + 0.010642583001754247, + 0.010263958989526145, + 0.01013629099179525, + 0.010029082986875437, + 0.010513458008063026, + 0.011337541000102647, + 0.010881291993428022, + 0.010552291001658887, + 0.009451667006942444, + 0.009065250007552095, + 0.009141041999100707, + 0.009475333004957065, + 0.010483541991561651, + 0.00945029201102443, + 0.009179250002489425 + ] + }, + { + "forward": 1.2309430000022985, + "prefill": 1.2441120829898864, + "decode_steps": [ + 0.015767125005368143, + 0.01292216700676363, + 0.012082040993846022, + 0.011472250000224449, + 0.012148958994657733, + 0.013744833006057888, + 0.01257145800627768, + 0.012412374999257736, + 0.011918417003471404, + 0.01067854100256227, + 0.012425249995430931, + 0.013436332999845035, + 0.013238166997325607, + 0.013665708000189625, + 0.012326124997343868, + 0.012358040999970399, + 0.014964832997065969, + 0.011720249996869825, + 0.012048083997797221, + 0.01427020799019374, + 0.011285915999906138, + 0.012237125003593974, + 0.012695249999524094, + 0.01106025000626687, + 0.01212945900624618, + 0.0127490840095561, + 0.011041958001442254, + 0.012161917009507306, + 0.012771166992024519, + 0.010983542000758462, + 0.012167624998255633, + 0.01225762499962002 + ] + }, + { + "forward": 1.2571221250109375, + "prefill": 1.1498467499914113, + "decode_steps": [ + 0.014561000003595836, + 0.01009908301057294, + 0.010668791001080535, + 0.011293167000985704, + 0.010609124990878627, + 0.010850875012692995, + 0.01126304198987782, + 0.01038620799954515, + 0.010270375001709908, + 0.009753291989909485, + 0.009772833000170067, + 0.010226290993159637, + 0.009945124998921528, + 0.009599583005183376, + 0.009419209003681317, + 0.009601333003956825, + 0.009989417012548074, + 0.009981458002584986, + 0.009872000009636395, + 0.009652624998125248, + 0.009942749995389022, + 0.009693791012978181, + 0.009580708996509202, + 0.010342708003008738, + 0.011188625008799136, + 0.011045166989788413, + 0.010341291999793611, + 0.010601332993246615, + 0.009658958006184548, + 0.010856624998268671, + 0.010020957997767255, + 0.010731999995186925 + ] + }, + { + "forward": 1.3354580839950358, + "prefill": 1.1230470830050763, + "decode_steps": [ + 0.013702790995012037, + 0.010608791999402456, + 0.009808582995901816, + 0.009408792000613175, + 0.009867250002571382, + 0.009845041000517085, + 0.010510582986171357, + 0.009533625008771196, + 0.00976845900004264, + 0.009790542011614889, + 0.00994412500585895, + 0.010177125004702248, + 0.009439665998797864, + 0.011177332999068312, + 0.01058937500056345, + 0.009887042004265822, + 0.010379915998782963, + 0.009349125000881031, + 0.009995250002248213, + 0.009850208007264882, + 0.010129750007763505, + 0.009560334001434967, + 0.009127416007686406, + 0.009630207990994677, + 0.009309207991464064, + 0.009193707999656908, + 0.009076542002731003, + 0.00924916700751055, + 0.009356957991258241, + 0.009391124985995702, + 0.00922133299172856, + 0.010356166007113643 + ] + }, + { + "forward": 1.5728519590047654, + "prefill": 1.3675172080111224, + "decode_steps": [ + 0.013455332999001257, + 0.009941624986822717, + 0.009612583002308384, + 0.010118208010680974, + 0.010120708000613376, + 0.01265520800370723, + 0.011432040992076509, + 0.011619500000961125, + 0.01132079200760927, + 0.011381791002349928, + 0.01129387499531731, + 0.01159445800294634, + 0.01080541600822471, + 0.011497416999191046, + 0.011114792010630481, + 0.012036375002935529, + 0.011797041996032931, + 0.012195624993182719, + 0.01166587500483729, + 0.011323124999762513, + 0.011081541000748985, + 0.011371124986908399, + 0.011252707990934141, + 0.010265958000672981, + 0.011858749989187345, + 0.011371291999239475, + 0.011127582998597063, + 0.011472708007204346, + 0.011420708004152402, + 0.010946166003122926, + 0.010856875000172295, + 0.011325875006150454 + ] + }, + { + "forward": 1.138175499989302, + "prefill": 1.3257767080067424, + "decode_steps": [ + 0.017676084011327475, + 0.011545125002157874, + 0.011109250001027249, + 0.0251734579942422, + 0.023369292001007125, + 0.01099516698741354, + 0.011138208006741479, + 0.010944708992610686, + 0.012596458007465117, + 0.01242025000101421, + 0.010785542006487958, + 0.01022058400849346, + 0.010402583007817157, + 0.010257041998556815, + 0.00999254199268762, + 0.010840709001058713, + 0.011922292003873736, + 0.012400124993291683, + 0.014426499998080544, + 0.016941084002610296, + 0.011596458003623411, + 0.011933250003494322, + 0.011849999995320104, + 0.0114412090042606, + 0.01078954100375995, + 0.012641292007174343, + 0.012489583998103626, + 0.010473166999872774, + 0.009542542000417598, + 0.009832750001805834, + 0.010504125006264076, + 0.00958229199750349 + ] + }, + { + "forward": 1.2646499999973457, + "prefill": 1.1781801669858396, + "decode_steps": [ + 0.016825250000692904, + 0.011094916990259662, + 0.011938791009015404, + 0.012477959011448547, + 0.011812416007160209, + 0.012146332999691367, + 0.011827499998616986, + 0.010934249992715195, + 0.011840332997962832, + 0.012323167000431567, + 0.010520125011680648, + 0.011327790998620912, + 0.012941666995175183, + 0.011086832993896678, + 0.01144604200089816, + 0.012384959001792595, + 0.011918250005692244, + 0.01222395799413789, + 0.011684291006531566, + 0.011036042007617652, + 0.011533582990523428, + 0.012497709001763724, + 0.011234167002839968, + 0.011048957996536046, + 0.012190540990559384, + 0.010762042002170347, + 0.010862542010727338, + 0.009939708994352259, + 0.009644083009334281, + 0.009663083998020738, + 0.009318416996393353, + 0.009873625007458031 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 1251.07918750291, + "p95": 1522.708871202485, + "min": 1138.175499989302, + "max": 1572.8519590047654, + "iqr": 108.7234692495258, + "mean": 1284.9129584996263, + "cv": 0.10735608252651829 + }, + "prefill_ms": { + "count": 10, + "median": 1263.2096454981365, + "p95": 1348.7339830091514, + "min": 1123.0470830050763, + "max": 1367.5172080111224, + "iqr": 156.2370100109547, + "mean": 1242.143291499815, + "cv": 0.07349787283211402 + }, + "decode_ms_per_step": { + "count": 320, + "median": 10.571874998277053, + "p95": 13.45765799851506, + "min": 8.727166001335718, + "max": 25.1734579942422, + "iqr": 1.8465724970155861, + "mean": 10.893294097286343, + "cv": 0.1609979893656593 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 751.1540801596017, + "p95": 814.6926500894754, + "min": 640.4609989960469, + "max": 821.0100526105691, + "iqr": 93.26695063283466, + "mean": 739.9290202598211, + "cv": 0.08957251308418927 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 157.88985356743115, + "p95": 178.01080772258905, + "min": 148.07528568001334, + "max": 178.2300105618642, + "iqr": 16.587968090248012, + "mean": 161.63236826870641, + "cv": 0.07012967833588571 + }, + "memory": { + "peak_extra_bytes": 1850310656, + "method": "process_peak_rss_delta" + } + } + ] +} diff --git a/experiments/records/attention/20261022T033255Z_attention_mps_float32_615767d.json b/experiments/records/attention/20261022T033255Z_attention_mps_float32_615767d.json new file mode 100644 index 0000000..b59b030 --- /dev/null +++ b/experiments/records/attention/20261022T033255Z_attention_mps_float32_615767d.json @@ -0,0 +1,14061 @@ +{ + "record_version": 1, + "created_utc": "20261022T033255Z", + "experiment": "attention_backends", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "mps", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "615767d83416226055484fb1f7336bdf4a40eaa0", + "branch": "experiments/attention-kv-audit", + "dirty": false + }, + "command": "python scripts/benchmark_attention.py --device mps --dtype float32" + }, + "settings": { + "dtype": "float32", + "position": "learned", + "backends": [ + "auto", + "sdpa", + "reference" + ], + "seq_lens": [ + 128, + 256, + 512, + 1024, + 2048 + ], + "batch_sizes": [ + 1, + 4, + 8 + ], + "decode_tokens": 32, + "warmup": 3, + "repeats": 10, + "seed": 0, + "weights": "seeded random initialization (timing does not depend on trained values)", + "model_config": { + "vocab_size": 256, + "context_length": 2080, + "embedding_dim": 192, + "num_layers": 4, + "num_heads": 6, + "dropout": 0.0, + "positional_encoding": "learned", + "norm": "rmsnorm", + "activation": "swiglu", + "rope_theta": 10000.0 + }, + "inference_mode": true + }, + "results": [ + { + "backend": "auto", + "batch": 1, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.002109875000314787 + }, + { + "forward": 0.002166999998735264 + }, + { + "forward": 0.0024230830022133887 + }, + { + "forward": 0.001906292003695853 + }, + { + "forward": 0.0019703340076375753 + }, + { + "forward": 0.001990832999581471 + }, + { + "forward": 0.0018760829989332706 + }, + { + "forward": 0.002262207999592647 + }, + { + "forward": 0.00223966600606218 + }, + { + "forward": 0.0019649580062832683 + } + ], + "forward_ms": { + "count": 10, + "median": 2.050353999948129, + "p95": 2.350689251034055, + "min": 1.8760829989332706, + "max": 2.4230830022133887, + "iqr": 0.2551974976086058, + "mean": 2.0910332023049705, + "cv": 0.08563964354183595 + }, + "memory": { + "peak_extra_bytes": 0, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "sdpa", + "batch": 1, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.001695250000921078, + "prefill": 0.0016214170027524233, + "decode_steps": [ + 0.0017119580006692559, + 0.0015833750076126307, + 0.0016993750032270327, + 0.0016707080067135394, + 0.0017150420026155189, + 0.0016875419969437644, + 0.0016832920082379133, + 0.0016774999967310578, + 0.002026874994044192, + 0.0016903330106288195, + 0.0016759590071160346, + 0.0018104589980794117, + 0.0017462499963585287, + 0.001544041995657608, + 0.0017010839947033674, + 0.0015597500023432076, + 0.0016885830118553713, + 0.0016069580015027896, + 0.001574499998241663, + 0.0015621250058757141, + 0.0017677499999990687, + 0.001558666001074016, + 0.0015906670014373958, + 0.0016204590065171942, + 0.0016530419961782172, + 0.001747250003973022, + 0.0015441670111613348, + 0.0015509999939240515, + 0.0015513340040342882, + 0.0015293749893317, + 0.0015650419954909012, + 0.0015961669996613637 + ] + }, + { + "forward": 0.001895042005344294, + "prefill": 0.0018547499930718914, + "decode_steps": [ + 0.0019190840102965012, + 0.0017939999961527064, + 0.001668750002863817, + 0.0017340839985990897, + 0.0015947919891914353, + 0.001564541002153419, + 0.0020004999969387427, + 0.0016175409982679412, + 0.0016907919925870374, + 0.0018878330010920763, + 0.0016997080092551187, + 0.0017213749961229041, + 0.0017820000066421926, + 0.0015666660037823021, + 0.0017118329997174442, + 0.0017622500017751008, + 0.0016743329906603321, + 0.0016184170090127736, + 0.0020215830008964986, + 0.0015943329926813021, + 0.0015422919968841597, + 0.0015577919984934852, + 0.0016087910044007003, + 0.0017048749868990853, + 0.0015448749909410253, + 0.0016467079985886812, + 0.0015714589972048998, + 0.002087417000439018, + 0.0016953329904936254, + 0.001560375007102266, + 0.0015463750023627654, + 0.001937499997438863 + ] + }, + { + "forward": 0.002125459010130726, + "prefill": 0.0021671249996870756, + "decode_steps": [ + 0.0022678749955957755, + 0.0018575840076664463, + 0.0022842920006951317, + 0.0023743329948047176, + 0.0020888330036541447, + 0.00210545799927786, + 0.0020447089918889105, + 0.0015725829871371388, + 0.0015879999991739169, + 0.0015734589978819713, + 0.0016178750083781779, + 0.0017030830058502033, + 0.001862709003034979, + 0.0017155829991679639, + 0.0016893749998416752, + 0.0016847910010255873, + 0.001592417000210844, + 0.0017151670035673305, + 0.0016875829896889627, + 0.0016315839893650264, + 0.0016593750042375177, + 0.0017503749986644834, + 0.0016923329967539757, + 0.0016911249986151233, + 0.0016961249930318445, + 0.0016114169993670657, + 0.001683291993685998, + 0.0016686250019120052, + 0.0015835419908398762, + 0.001592041997355409, + 0.002136958995833993, + 0.0016772499948274344 + ] + }, + { + "forward": 0.0016332499944837764, + "prefill": 0.0016787080094218254, + "decode_steps": [ + 0.002502791990991682, + 0.0017009169969242066, + 0.001673749997280538, + 0.0017457089998060837, + 0.0019617910002125427, + 0.0018147920054616407, + 0.0017004170076688752, + 0.0016033749998314306, + 0.002471208994393237, + 0.0016226250008912757, + 0.00169162500242237, + 0.0016582079988438636, + 0.0020594579982571304, + 0.00168470801145304, + 0.0016172500036191195, + 0.001524582999991253, + 0.002021499996772036, + 0.0016754999960539863, + 0.0015466670010937378, + 0.0018612919957377017, + 0.0018195420125266537, + 0.001868374994955957, + 0.0015510840021306649, + 0.0017221250018337741, + 0.0015410000050906092, + 0.0015829590120119974, + 0.001554125003167428, + 0.0015240419888868928, + 0.001658791006775573, + 0.002129874992533587, + 0.0017277089937124401, + 0.0015340000099968165 + ] + }, + { + "forward": 0.0019088329863734543, + "prefill": 0.002410750006674789, + "decode_steps": [ + 0.0018447079928591847, + 0.0016644170100335032, + 0.002445999998599291, + 0.0037025409983471036, + 0.002449665989843197, + 0.0018699169886531308, + 0.002172709006117657, + 0.0023080420069163665, + 0.002954750001663342, + 0.0037421250017359853, + 0.0027193749992875382, + 0.0026952920015901327, + 0.002875499994843267, + 0.0022441249893745407, + 0.0017536250088596717, + 0.0017332499992335215, + 0.0017107500025304034, + 0.0017259999876841903, + 0.0017714999994495884, + 0.0017224579933099449, + 0.0017108749889302999, + 0.0017035000055329874, + 0.001534499999252148, + 0.0017294999997830018, + 0.001592624990735203, + 0.001654457999393344, + 0.0015725419943919405, + 0.0017157919937744737, + 0.0016536249895580113, + 0.001530541994725354, + 0.0017588750051800162, + 0.0015736670029582456 + ] + }, + { + "forward": 0.0018351670005358756, + "prefill": 0.001792708004359156, + "decode_steps": [ + 0.0018431250064168125, + 0.001585125006386079, + 0.0015815419901628047, + 0.0016849999956320971, + 0.0016810000088298693, + 0.0016656249936204404, + 0.0016573750035604462, + 0.001645333002670668, + 0.0016719580016797408, + 0.001564750011311844, + 0.0015637920005246997, + 0.0015507500065723434, + 0.0017213340033777058, + 0.0016303330048685893, + 0.001692791993264109, + 0.0015796660009073094, + 0.0015797090018168092, + 0.001785542001016438, + 0.0017000420048134401, + 0.0015506249910686165, + 0.0016355410043615848, + 0.0015447500045411289, + 0.001702791007119231, + 0.0016298749978886917, + 0.0016637500084470958, + 0.0016324579919455573, + 0.0016207089938689023, + 0.001584416997502558, + 0.001897124995593913, + 0.0016822500037960708, + 0.0015908339992165565, + 0.0017088750028051436 + ] + }, + { + "forward": 0.0017039999947883189, + "prefill": 0.002021292006247677, + "decode_steps": [ + 0.0017508750024717301, + 0.001665584000875242, + 0.0015608329995302483, + 0.001696041988907382, + 0.0017030000017257407, + 0.001703208006802015, + 0.001559249998535961, + 0.0016623330011498183, + 0.0017056250071618706, + 0.0016897919995244592, + 0.0016882080089999363, + 0.0015915000112727284, + 0.0016997499915305525, + 0.0016260420088656247, + 0.0016509159904671833, + 0.0015433340013260022, + 0.0017424170073354617, + 0.001708750001853332, + 0.0016949160053627566, + 0.001571583008626476, + 0.0015753750049043447, + 0.0016426670044893399, + 0.0016664589929860085, + 0.0015446659963345155, + 0.0016514169983565807, + 0.001686916992184706, + 0.0016215409996220842, + 0.0015623749932274222, + 0.001661500005866401, + 0.0015433749940712005, + 0.0016285409947158769, + 0.0016599999944446608 + ] + }, + { + "forward": 0.0018283329991390929, + "prefill": 0.001795834003132768, + "decode_steps": [ + 0.0017406249971827492, + 0.001897124995593913, + 0.0017136250098701566, + 0.001688417003606446, + 0.0017378330085193738, + 0.0015304170083254576, + 0.001669666002271697, + 0.0016195839998545125, + 0.0015148339880397543, + 0.001524999999674037, + 0.002045333996647969, + 0.0017222089954884723, + 0.0015511670062551275, + 0.0016727499896660447, + 0.0015108749939827248, + 0.0015305829874705523, + 0.0015303749969461933, + 0.00154116700286977, + 0.001658082997892052, + 0.0022799169964855537, + 0.0018340419919695705, + 0.0016465830121887848, + 0.0016060000052675605, + 0.0019138329953420907, + 0.0016434580029454082, + 0.0015636659954907373, + 0.0016594589978922158, + 0.0018324999982723966, + 0.0017208330100402236, + 0.001578707990120165, + 0.0016355830011889338, + 0.0015864579909248278 + ] + }, + { + "forward": 0.001956000007339753, + "prefill": 0.002007375005632639, + "decode_steps": [ + 0.002047582995146513, + 0.0018078330031130463, + 0.001669500008574687, + 0.0016839169984450564, + 0.001711999997496605, + 0.0016372919926652685, + 0.0016550000000279397, + 0.0017587500042282045, + 0.0015416249952977523, + 0.001716041995678097, + 0.0017097500094678253, + 0.001693208992946893, + 0.0015584579959977418, + 0.0016282500000670552, + 0.0016809170047054067, + 0.0016899579932214692, + 0.0016844999918248504, + 0.0015704590041423216, + 0.001681083012954332, + 0.0015999589959392324, + 0.0016753750096540898, + 0.0015641250065527856, + 0.001663415998336859, + 0.0015886659966781735, + 0.0016907500103116035, + 0.0015479580033570528, + 0.0016262910066870973, + 0.0016583749966230243, + 0.0016780829901108518, + 0.0016132919990923256, + 0.001723790992400609, + 0.001720083993859589 + ] + }, + { + "forward": 0.001674833009019494, + "prefill": 0.0016104590031318367, + "decode_steps": [ + 0.001722874992992729, + 0.00162191700655967, + 0.0015675000031478703, + 0.001546166997286491, + 0.0016245839942712337, + 0.0017287079972447827, + 0.0017549999902257696, + 0.0016882500058272853, + 0.0016779160068836063, + 0.0015368750027846545, + 0.0019134169997414574, + 0.0016815840062918141, + 0.0015209169941954315, + 0.0017294580029556528, + 0.0037597079935949296, + 0.004889957999694161, + 0.004377749995910563, + 0.003053000007639639, + 0.0031980420026229694, + 0.0029232919914647937, + 0.0018448749906383455, + 0.001997707993723452, + 0.0018696250044740736, + 0.0019437919982010499, + 0.0017582500004209578, + 0.0021113340044394135, + 0.0017387080006301403, + 0.0017669579974608496, + 0.0017754580039763823, + 0.0020578340045176446, + 0.0019929169939132407, + 0.0020960840047337115 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 1.8317499998374842, + "p95": 2.049202458874788, + "min": 1.6332499944837764, + "max": 2.125459010130726, + "iqr": 0.20794774172827601, + "mean": 1.8256167008075863, + "cv": 0.08372658480822134 + }, + "prefill_ms": { + "count": 10, + "median": 1.8252919981023297, + "p95": 2.3011187535303175, + "min": 1.6104590031318367, + "max": 2.410750006674789, + "iqr": 0.3106047479377594, + "mean": 1.896041803411208, + "cv": 0.13539576868406844 + }, + "decode_ms_per_step": { + "count": 320, + "median": 1.6827709987410344, + "p95": 2.377916344994447, + "min": 1.5108749939827248, + "max": 4.889957999694161, + "iqr": 0.15412526045111008, + "mean": 1.7754458747276658, + "cv": 0.21924834005631075 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 589.7310899682631, + "p95": 606.8058241902716, + "min": 467.35482042966015, + "max": 607.6858086578446, + "iqr": 41.08322501660257, + "mean": 567.7982234947619, + "cv": 0.08936402063315038 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 570.5422598952998, + "p95": 586.4968650325235, + "min": 456.61501024582896, + "max": 587.0262138714146, + "iqr": 40.296779535621226, + "mean": 549.2787088357027, + "cv": 0.08770669259850639 + }, + "memory": { + "peak_extra_bytes": 0, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "reference", + "batch": 1, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.0056137500068871304, + "prefill": 0.0022134589962661266, + "decode_steps": [ + 0.004494915992836468, + 0.004700666002463549, + 0.004385334003018215, + 0.004236958004184999, + 0.004062541993334889, + 0.00436624999565538, + 0.0042322079971199855, + 0.004589792006299831, + 0.00436495900794398, + 0.004178375005722046, + 0.004466749989660457, + 0.004382874991279095, + 0.004168875006143935, + 0.00429858299321495, + 0.004372249997686595, + 0.00429991701093968, + 0.004081999999471009, + 0.004207749996567145, + 0.004274542006896809, + 0.0040025420021265745, + 0.004069041009643115, + 0.004093166993698105, + 0.004315458005294204, + 0.004242042006808333, + 0.003645749995484948, + 0.003992083991761319, + 0.004446165999979712, + 0.004250708007020876, + 0.004228624995448627, + 0.004446792008820921, + 0.0045051670022076, + 0.004271125013474375 + ] + }, + { + "forward": 0.005800291997729801, + "prefill": 0.0027061669970862567, + "decode_steps": [ + 0.0052435420075198635, + 0.004237875007675029, + 0.004386125001474284, + 0.004567541996948421, + 0.004305249996832572, + 0.004250167010468431, + 0.003922957999748178, + 0.004781958996318281, + 0.004607666996889748, + 0.004535667001619004, + 0.0045746250107185915, + 0.004753708999487571, + 0.004034415993373841, + 0.004245040996465832, + 0.004475917012314312, + 0.004224125004839152, + 0.004181624986813404, + 0.0035995840007672086, + 0.004132707996177487, + 0.004470625004614703, + 0.004207624995615333, + 0.004227166995406151, + 0.004282958994735964, + 0.004094999996596016, + 0.0044555410131579265, + 0.004453415996977128, + 0.004539542002021335, + 0.00392008300696034, + 0.003924707998521626, + 0.004556125000817701, + 0.004514041007496417, + 0.00428308300615754 + ] + }, + { + "forward": 0.005314499998348765, + "prefill": 0.002380542005994357, + "decode_steps": [ + 0.003926750010577962, + 0.004163167002843693, + 0.004229458005283959, + 0.0043156669999007136, + 0.004209958991850726, + 0.0038790830003563315, + 0.004255833002389409, + 0.004116292009712197, + 0.004021291999379173, + 0.0041558329976396635, + 0.004110207999474369, + 0.004333041011705063, + 0.004122458005440421, + 0.004237874993123114, + 0.004117499993299134, + 0.004150416993070394, + 0.004143209007452242, + 0.004013833997305483, + 0.004152916997554712, + 0.0041480420040898025, + 0.0040497500012861565, + 0.004092125003808178, + 0.004109917004825547, + 0.00426549999974668, + 0.004436415998497978, + 0.004639999999199063, + 0.0045657499867957085, + 0.004196416004560888, + 0.004377375007607043, + 0.004692499991506338, + 0.004557457999908365, + 0.00445616600336507 + ] + }, + { + "forward": 0.00573600000643637, + "prefill": 0.0023574580118292943, + "decode_steps": [ + 0.004754082998260856, + 0.004795624990947545, + 0.004216625005938113, + 0.004549082994344644, + 0.005508332993485965, + 0.004584458001772873, + 0.004753332992549986, + 0.00445616700744722, + 0.0044859170011477545, + 0.004102291990420781, + 0.004395291005494073, + 0.004494749999139458, + 0.0044999999954598024, + 0.004551416990580037, + 0.004041082996991463, + 0.004441750003024936, + 0.004118124998058192, + 0.004266375006409362, + 0.004421375007950701, + 0.0041303329926449805, + 0.004301499997382052, + 0.004109500005142763, + 0.0038828749966342, + 0.004276459003449418, + 0.003942291994462721, + 0.0038108750013634562, + 0.004843916991376318, + 0.004787583005963825, + 0.004760499999974854, + 0.005163624999113381, + 0.0045952499931445345, + 0.0047778330044820905 + ] + }, + { + "forward": 0.006161707991850562, + "prefill": 0.0022438329906435683, + "decode_steps": [ + 0.005036250004195608, + 0.004453249988728203, + 0.004145166996750049, + 0.00431354099418968, + 0.004375667005660944, + 0.004339874998549931, + 0.004542000009678304, + 0.004357500001788139, + 0.004447667000931688, + 0.003841041005216539, + 0.0041127920121653005, + 0.004307791998144239, + 0.0041846249951049685, + 0.004165291000390425, + 0.004251999998814426, + 0.0037650419981218874, + 0.004164459009189159, + 0.00438458300777711, + 0.003979457993409596, + 0.004122832993743941, + 0.0042891670018434525, + 0.004151542001636699, + 0.0036960830038879067, + 0.004229583995766006, + 0.004090417001862079, + 0.004175166002823971, + 0.003650124999694526, + 0.004303584006265737, + 0.004272459002095275, + 0.004349459006334655, + 0.004716666997410357, + 0.0042969999922206625 + ] + }, + { + "forward": 0.004614083998603746, + "prefill": 0.0022053749999031425, + "decode_steps": [ + 0.004295332997571677, + 0.004135458992095664, + 0.004295916995033622, + 0.003639041999122128, + 0.0038825420051580295, + 0.004199792005238123, + 0.0042283750080969185, + 0.0035355830041226, + 0.004063542000949383, + 0.0040821669972501695, + 0.0041215840028598905, + 0.003747291993931867, + 0.004072750001796521, + 0.004134874994633719, + 0.003849917004117742, + 0.003642667012172751, + 0.004147125000599772, + 0.004161290999036282, + 0.004156708004302345, + 0.0037954580038785934, + 0.004234458989230916, + 0.004330874988227151, + 0.00465299999632407, + 0.004629542003385723, + 0.004350833987700753, + 0.004475666995858774, + 0.004185249999864027, + 0.004174625006271526, + 0.004198750000796281, + 0.00416833400959149, + 0.0041638330003479496, + 0.004118958007893525 + ] + }, + { + "forward": 0.005643666998366825, + "prefill": 0.0021397079981397837, + "decode_steps": [ + 0.004743124998640269, + 0.004027833012514748, + 0.004843041999265552, + 0.00393804200575687, + 0.0038440000062109903, + 0.004405582993058488, + 0.004271707992302254, + 0.00464441700023599, + 0.004426416999194771, + 0.0049130830011563376, + 0.0048980420106090605, + 0.004480457995668985, + 0.0047765420022187755, + 0.004669208006816916, + 0.004488416001549922, + 0.005292000001645647, + 0.004590542012010701, + 0.004376875003799796, + 0.004086582994204946, + 0.004653583004255779, + 0.0038076659984653816, + 0.004271833007805981, + 0.004183750002994202, + 0.004272167003364302, + 0.004423040998517536, + 0.004130375004024245, + 0.0045108749909559265, + 0.004070708004292101, + 0.004193875007331371, + 0.0044889579876326025, + 0.004440417003934272, + 0.004241458998876624 + ] + }, + { + "forward": 0.005519625003216788, + "prefill": 0.002211624989286065, + "decode_steps": [ + 0.004718584008514881, + 0.004445874990778975, + 0.004373916002805345, + 0.003964499992434867, + 0.0048070000048028305, + 0.004313958008424379, + 0.004147665997152217, + 0.004383332998258993, + 0.004567458003293723, + 0.004540333000477403, + 0.0041455409955233335, + 0.0056181250110967085, + 0.004903916007606313, + 0.004573624988552183, + 0.00454570799774956, + 0.004192708001937717, + 0.004581917004543357, + 0.0039439169922843575, + 0.004858291998971254, + 0.00484758400125429, + 0.004311374999815598, + 0.004383915991638787, + 0.004652957999496721, + 0.004275874991435558, + 0.004093583003850654, + 0.0044752079993486404, + 0.004258250002749264, + 0.004326041002059355, + 0.004175957990810275, + 0.0044618750107474625, + 0.004513292005867697, + 0.004354875010903925 + ] + }, + { + "forward": 0.005405874995631166, + "prefill": 0.002517999993870035, + "decode_steps": [ + 0.004475542009458877, + 0.004378375000669621, + 0.004582000008667819, + 0.004656875011278316, + 0.004256499989423901, + 0.0042680000042309985, + 0.004138332995353267, + 0.004187416998320259, + 0.0039579579897690564, + 0.004205958000966348, + 0.004555457999231294, + 0.004213916006847285, + 0.00422008300665766, + 0.004093583003850654, + 0.004330915995524265, + 0.004309415991883725, + 0.00392850000935141, + 0.004017833998659626, + 0.0041521660023136064, + 0.004312792007112876, + 0.004226917008054443, + 0.003981667003245093, + 0.00439225000445731, + 0.004208832993754186, + 0.003943667004932649, + 0.0042678749887272716, + 0.0043080000032205135, + 0.003970958001445979, + 0.004304709000280127, + 0.004146666993619874, + 0.004399209006805904, + 0.003927333003957756 + ] + }, + { + "forward": 0.006819165995693766, + "prefill": 0.002471416999469511, + "decode_steps": [ + 0.004093875002581626, + 0.004276250008842908, + 0.004451249988051131, + 0.0042788749997271225, + 0.004329832998337224, + 0.00407075000111945, + 0.004681499995058402, + 0.004222625007969327, + 0.004433624999364838, + 0.004640333994757384, + 0.004469000006793067, + 0.0042524169984972104, + 0.004082582992850803, + 0.004565457988064736, + 0.00398904099711217, + 0.004449415995622985, + 0.003768583992496133, + 0.004245249991072342, + 0.004386375003377907, + 0.004284666996682063, + 0.0042867090087383986, + 0.004381165999802761, + 0.004211334002320655, + 0.004080749989952892, + 0.004094167001312599, + 0.004091750000952743, + 0.004159125004662201, + 0.004018999999971129, + 0.004177792012342252, + 0.004130041008465923, + 0.004077000005054288, + 0.003860582990455441 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 5.628708502626978, + "p95": 6.523309893964323, + "min": 4.614083998603746, + "max": 6.819165995693766, + "iqr": 0.34990650237887166, + "mean": 5.662866699276492, + "cv": 0.10059811709243595 + }, + "prefill_ms": { + "count": 10, + "median": 2.3006455012364313, + "p95": 2.621491845638957, + "min": 2.1397079981397837, + "max": 2.7061669970862567, + "iqr": 0.23661476006964222, + "mean": 2.344758398248814, + "cv": 0.07585812433540488 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.271416502888314, + "p95": 4.782240196800558, + "min": 3.5355830041226, + "max": 5.6181250110967085, + "iqr": 0.32733401167206466, + "mean": 4.300411584927133, + "cv": 0.06783352213249529 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 235.11206345835166, + "p95": 239.93884639170255, + "min": 224.06041391311172, + "max": 242.66038405426565, + "iqr": 9.109449092697673, + "mean": 232.68652844116733, + "cv": 0.026756079949730015 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 231.13250531559277, + "p95": 235.901794085947, + "min": 220.42198387200366, + "max": 238.66896524453043, + "iqr": 8.606059073318846, + "mean": 228.782713708444, + "cv": 0.02627187816308398 + }, + "memory": { + "peak_extra_bytes": 0, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "auto", + "batch": 1, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.002503542011254467 + }, + { + "forward": 0.0026552499912213534 + }, + { + "forward": 0.0029338340100366622 + }, + { + "forward": 0.002490915998350829 + }, + { + "forward": 0.0027558329893508926 + }, + { + "forward": 0.002498333007679321 + }, + { + "forward": 0.002496125001925975 + }, + { + "forward": 0.003038750000996515 + }, + { + "forward": 0.0025463749916525558 + }, + { + "forward": 0.002429250002023764 + } + ], + "forward_ms": { + "count": 10, + "median": 2.5249585014535114, + "p95": 2.9915378050645813, + "min": 2.429250002023764, + "max": 3.038750000996515, + "iqr": 0.23401023645419627, + "mean": 2.6348208004492335, + "cv": 0.07936163532525956 + }, + "memory": { + "peak_extra_bytes": 0, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "sdpa", + "batch": 1, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.0024437500105705112, + "prefill": 0.002272083002026193, + "decode_steps": [ + 0.0016730000061215833, + 0.0015432090003741905, + 0.001778709003701806, + 0.0016436670121038333, + 0.0017111670022131875, + 0.0017642909951973706, + 0.0017516670050099492, + 0.001684375005424954, + 0.0017008330032695085, + 0.0017267499933950603, + 0.001714708996587433, + 0.0016753329982748255, + 0.0017467500001657754, + 0.0017331249982817098, + 0.0017019589868141338, + 0.0016989159921649843, + 0.0016667920135660097, + 0.0016526250110473484, + 0.0018048750062007457, + 0.0017122920107794926, + 0.0016582910029683262, + 0.001713332996587269, + 0.0015588750102324411, + 0.0016602500109001994, + 0.0016357919957954437, + 0.0016272499924525619, + 0.0016167080029845238, + 0.0016951669967966154, + 0.0016569169965805486, + 0.0016126249975059181, + 0.0016162499960046262, + 0.001683624999714084 + ] + }, + { + "forward": 0.0023279169981833547, + "prefill": 0.002265625007566996, + "decode_steps": [ + 0.0018772500043269247, + 0.0017112920031649992, + 0.0016492920112796128, + 0.0016089170094346628, + 0.0016583750111749396, + 0.0018615829903865233, + 0.0017503330018371344, + 0.0016554580070078373, + 0.0016870830004336312, + 0.001691208002739586, + 0.0017076670046662912, + 0.0017288340022787452, + 0.0016596249915892258, + 0.0016901659982977435, + 0.001669332996243611, + 0.0016758750134613365, + 0.0017035420023603365, + 0.001697334009804763, + 0.0017853750032372773, + 0.001678874992649071, + 0.0017335000011371449, + 0.0018024589953711256, + 0.0016543339879717678, + 0.0017160410061478615, + 0.0017019169899867848, + 0.0017244999908143654, + 0.001585374993737787, + 0.0016714589874027297, + 0.0017430419975426048, + 0.001642499992158264, + 0.0018772089970298111, + 0.0015771249891258776 + ] + }, + { + "forward": 0.0025472919951425865, + "prefill": 0.0023457500064978376, + "decode_steps": [ + 0.0018413749930914491, + 0.0016251659981207922, + 0.001668083990807645, + 0.0016676660015946254, + 0.0016961669898591936, + 0.0016891669947654009, + 0.0017103330028476194, + 0.001758874990628101, + 0.0017329590045846999, + 0.0016886249941308051, + 0.0017050830065272748, + 0.0016865829966263846, + 0.0016665839939378202, + 0.0016480829945066944, + 0.0016917500033741817, + 0.0016358749999199063, + 0.0016443339991383255, + 0.0016882080089999363, + 0.0016650000034132972, + 0.0016058329929364845, + 0.0015859999984968454, + 0.0017957499949261546, + 0.0015859999984968454, + 0.0017104169965023175, + 0.002045875007752329, + 0.001757707999786362, + 0.0015814159996807575, + 0.0017168329941341653, + 0.0017207079945364967, + 0.0015901250008028, + 0.0016739589918870479, + 0.0016593329928582534 + ] + }, + { + "forward": 0.0021716250048484653, + "prefill": 0.0023491250030929223, + "decode_steps": [ + 0.0018359160021645948, + 0.0016422920016339049, + 0.00169012499100063, + 0.0017119580006692559, + 0.001706875002128072, + 0.0016050839913077652, + 0.0016954589955275878, + 0.0016715829988243058, + 0.0016943329974310473, + 0.001719958003377542, + 0.0016897919995244592, + 0.0017176249966723844, + 0.0016834580019349232, + 0.0017087910091504455, + 0.0017129170009866357, + 0.0016841249889694154, + 0.0017042920080712065, + 0.001698499996564351, + 0.0015970000094966963, + 0.0016116659971885383, + 0.0017105409933719784, + 0.001655958010815084, + 0.0017172079969896004, + 0.0017309579998254776, + 0.0016209159948630258, + 0.0017024170083459467, + 0.0017264170019188896, + 0.0017165000026579946, + 0.001606499994522892, + 0.0016933330043684691, + 0.0016779169964138418, + 0.001636417000554502 + ] + }, + { + "forward": 0.0024692910083103925, + "prefill": 0.002260499997646548, + "decode_steps": [ + 0.0019450419931672513, + 0.0018623750074766576, + 0.0017395409959135577, + 0.0018365830037510023, + 0.001767834008205682, + 0.0017145000019809231, + 0.0017034579941537231, + 0.0016587909922236577, + 0.001661166999838315, + 0.001800915997591801, + 0.0017359579942421988, + 0.0015923749888315797, + 0.0018014579982263967, + 0.0017265829956158996, + 0.0017174170061480254, + 0.0015730830054963008, + 0.0021867080067750067, + 0.0016898339963518083, + 0.0017016659985529259, + 0.001658499997574836, + 0.00200229199253954, + 0.0017143330042017624, + 0.0017349580011796206, + 0.0016854169953148812, + 0.0016930839919950813, + 0.0016279589908663183, + 0.0016095840110210702, + 0.0016747920017223805, + 0.0016672089986968786, + 0.001731666998239234, + 0.0016909590049181134, + 0.0018557089933892712 + ] + }, + { + "forward": 0.0022398750006686896, + "prefill": 0.002200083006755449, + "decode_steps": [ + 0.0018138750019716099, + 0.0018212919967481866, + 0.0017287499940721318, + 0.0017082499980460852, + 0.001713915989967063, + 0.0018663750088308007, + 0.001756792000378482, + 0.0017652079986874014, + 0.0016966669936664402, + 0.0016743330052122474, + 0.001780791993951425, + 0.0017104589933296666, + 0.0017144579906016588, + 0.0017926250002346933, + 0.001698124993708916, + 0.0017182080046040937, + 0.0017084159917430952, + 0.0016745829925639555, + 0.001640000002225861, + 0.0017383749946020544, + 0.001661500005866401, + 0.0016016250010579824, + 0.001661166999838315, + 0.001586707992828451, + 0.0016062499926192686, + 0.0017924999992828816, + 0.0016466660017613322, + 0.0016200420068344101, + 0.001622709009097889, + 0.001720416999887675, + 0.0016066669923020527, + 0.0015899580030236393 + ] + }, + { + "forward": 0.0022516250028274953, + "prefill": 0.00227499999164138, + "decode_steps": [ + 0.0023602500004926696, + 0.0017437499918742105, + 0.0017352909926557913, + 0.0017262500041397288, + 0.0017174999957205728, + 0.0017356669995933771, + 0.0016346249904017895, + 0.0018274580070283264, + 0.0027853749925270677, + 0.001969083008589223, + 0.0018665419920580462, + 0.001827958010835573, + 0.0017982499994104728, + 0.0019497909961501136, + 0.0018757499929051846, + 0.0016046670061768964, + 0.001839582997490652, + 0.001657208995311521, + 0.0015772920014569536, + 0.001769541995599866, + 0.0016914580046432093, + 0.0015913749957690015, + 0.0016475000011269003, + 0.0016434580029454082, + 0.0016207080043386668, + 0.0030166659998940304, + 0.001647625002078712, + 0.0016871669940883294, + 0.001698124993708916, + 0.0019017080048797652, + 0.001788749999832362, + 0.0016992499877233058 + ] + }, + { + "forward": 0.0030163750052452087, + "prefill": 0.0029117500089341775, + "decode_steps": [ + 0.0022773750097258016, + 0.0019646250002551824, + 0.0018837079987861216, + 0.0023952499905135483, + 0.001847042003646493, + 0.0017827499978011474, + 0.002118124990374781, + 0.0019379160075914115, + 0.002013457997236401, + 0.001722874992992729, + 0.001816541000152938, + 0.0016943749942583963, + 0.0016240409895544872, + 0.0016955829923972487, + 0.0017391250003129244, + 0.0017752089916029945, + 0.0016149999864865094, + 0.0018198329926235601, + 0.0017147499893326312, + 0.002248666001833044, + 0.0018361249967711046, + 0.0016523329977644607, + 0.0018222079961560667, + 0.0019554580067051575, + 0.0017515420040581375, + 0.0016115839971462265, + 0.0016873330023372546, + 0.002826750001986511, + 0.0017456670029787347, + 0.0017134579975390807, + 0.002473999993526377, + 0.0023079999955371022 + ] + }, + { + "forward": 0.0021581670007435605, + "prefill": 0.002271959005156532, + "decode_steps": [ + 0.001791208007489331, + 0.0016591250023338944, + 0.0017269579984713346, + 0.0017943749990081415, + 0.00177237500611227, + 0.0017187920020660385, + 0.0016985420079436153, + 0.0017633749957894906, + 0.001735208003083244, + 0.0017183329910039902, + 0.0016332920058630407, + 0.0017360420024488121, + 0.0016957090119831264, + 0.001720375003060326, + 0.0016673750069458038, + 0.0017303749918937683, + 0.0017832910089055076, + 0.0017275839927606285, + 0.0017188329948112369, + 0.0016604999982519075, + 0.0017702909972285852, + 0.0017034580087056383, + 0.0016340829897671938, + 0.0016872909909579903, + 0.0017229170043719932, + 0.0017693750123726204, + 0.0016204580024350435, + 0.0016995000041788444, + 0.0015938750002533197, + 0.0024986659991554916, + 0.0016741660074330866, + 0.0017287499940721318 + ] + }, + { + "forward": 0.0023172079963842407, + "prefill": 0.0022406250063795596, + "decode_steps": [ + 0.001687958007096313, + 0.0018027499900199473, + 0.0023925410059746355, + 0.0016702910070307553, + 0.0016353329992853105, + 0.001674125000135973, + 0.0019215409993194044, + 0.001686832998530008, + 0.0016506249958183616, + 0.0015750839957036078, + 0.002460874995449558, + 0.0017128749896073714, + 0.001701500004855916, + 0.0016123329987749457, + 0.00172004199703224, + 0.001705917005892843, + 0.0016205840074690059, + 0.001668209006311372, + 0.0016504169907420874, + 0.001791040995158255, + 0.0018022910080617294, + 0.001755208009853959, + 0.0016208330052904785, + 0.001935958003741689, + 0.0018081669986713678, + 0.0016987919952953234, + 0.0016240420081885532, + 0.0025670420000096783, + 0.0016875419969437644, + 0.0015892919909674674, + 0.0016703750006854534, + 0.0018208330002380535 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 2.3225624972837977, + "p95": 2.8052876506990283, + "min": 2.1581670007435605, + "max": 3.0163750052452087, + "iqr": 0.22009325766703114, + "mean": 2.3943125022924505, + "cv": 0.1058748628391761 + }, + "prefill_ms": { + "count": 10, + "median": 2.2720210035913624, + "p95": 2.658568756305612, + "min": 2.200083006755449, + "max": 2.9117500089341775, + "iqr": 0.0662812526570633, + "mean": 2.3392500035697594, + "cv": 0.088045823496749 + }, + "decode_ms_per_step": { + "count": 320, + "median": 1.7035000055329874, + "p95": 2.0150788477621977, + "min": 1.5432090003741905, + "max": 3.0166659998940304, + "iqr": 0.09790599142434075, + "mean": 1.7465476368670352, + "cv": 0.10667269945871301 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 581.1699298001929, + "p95": 593.7963378944276, + "min": 523.9828169975558, + "max": 594.5731041601397, + "iqr": 25.150475256740833, + "mean": 573.442576676703, + "cv": 0.04049675797562508 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 558.1694752234206, + "p95": 569.4238416404345, + "min": 500.1371084549721, + "max": 570.4891994177358, + "iqr": 22.59051779347749, + "mean": 550.4357962178726, + "cv": 0.041081385584451874 + }, + "memory": { + "peak_extra_bytes": 0, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "reference", + "batch": 1, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.0062394590058829635, + "prefill": 0.0031644580012653023, + "decode_steps": [ + 0.004208541999105364, + 0.004490499995881692, + 0.004660082995542325, + 0.004697084004874341, + 0.00458516700018663, + 0.004409750006743707, + 0.004576124993036501, + 0.004480167001020163, + 0.004235125001287088, + 0.004257708991644904, + 0.00420833399402909, + 0.004241709000780247, + 0.0037631250015692785, + 0.00422887499735225, + 0.0041598329989938065, + 0.004020124993985519, + 0.004203458011033945, + 0.004162957993685268, + 0.004142082994803786, + 0.004099000012502074, + 0.004125457999180071, + 0.004012000004877336, + 0.004225291995680891, + 0.0038720839947927743, + 0.0046058750012889504, + 0.00437525000597816, + 0.004299749998608604, + 0.004163541001616977, + 0.003809208996244706, + 0.0036526250041788444, + 0.00419224999495782, + 0.004205249992082827 + ] + }, + { + "forward": 0.0065548340062377974, + "prefill": 0.0031451249960809946, + "decode_steps": [ + 0.004541874994174577, + 0.003881291995639913, + 0.004224916992825456, + 0.004378499987069517, + 0.00420624999969732, + 0.0036910829949192703, + 0.003980875000706874, + 0.0039450000040233135, + 0.004272125006536953, + 0.003714625010616146, + 0.004153457994107157, + 0.004593874997226521, + 0.004531082988251001, + 0.004316042002756149, + 0.004109957997570746, + 0.00480345799587667, + 0.004345000008470379, + 0.004254624989698641, + 0.004231041995808482, + 0.004781374998856336, + 0.003913541993824765, + 0.004234332998748869, + 0.004473542008781806, + 0.0051785410032607615, + 0.004266499992809258, + 0.004279957996914163, + 0.004609041992807761, + 0.005028957995818928, + 0.004304040994611569, + 0.004843250004341826, + 0.0046135419979691505, + 0.0045523750013671815 + ] + }, + { + "forward": 0.006229207996511832, + "prefill": 0.003579291995265521, + "decode_steps": [ + 0.004478707996895537, + 0.004372334005893208, + 0.004196416994091123, + 0.0044350419921102, + 0.004168125000433065, + 0.004226250006468035, + 0.004190874999039806, + 0.004227791010634974, + 0.004135582988965325, + 0.004083874999196269, + 0.004225332988426089, + 0.004209000006085262, + 0.004332457989221439, + 0.004170790998614393, + 0.004125291990931146, + 0.004167542007053271, + 0.004152416993747465, + 0.0041219590057153255, + 0.003665707990876399, + 0.004091458002221771, + 0.004140082994126715, + 0.0041882499936036766, + 0.004100959005882032, + 0.004139916010899469, + 0.004098499994142912, + 0.004251500009559095, + 0.0042055840021930635, + 0.0040531670092605054, + 0.004640415994799696, + 0.00448833400150761, + 0.0042164170008618385, + 0.0041205410088878125 + ] + }, + { + "forward": 0.0065369170042686164, + "prefill": 0.0030346249986905605, + "decode_steps": [ + 0.004540790992905386, + 0.004180375006399117, + 0.0058174169971607625, + 0.004141291996347718, + 0.004937584002618678, + 0.005510791001142934, + 0.004634750002878718, + 0.015747707991977222, + 0.009483250003540888, + 0.005557416996452957, + 0.00567041699832771, + 0.005084959004307166, + 0.004706249994342215, + 0.0042501249990891665, + 0.004568458010908216, + 0.0038356660079443827, + 0.004345374996773899, + 0.004454207999515347, + 0.005157041989150457, + 0.004696249990956858, + 0.004273624988854863, + 0.005087792000267655, + 0.004309875002945773, + 0.004174832996795885, + 0.004799500005901791, + 0.005243999999947846, + 0.004605291993357241, + 0.004240750000462867, + 0.004415290997712873, + 0.00411375000840053, + 0.004302375004044734, + 0.004470166997634806 + ] + }, + { + "forward": 0.00790762499673292, + "prefill": 0.0031220420059980825, + "decode_steps": [ + 0.005140624998603016, + 0.005036457994719967, + 0.0044028330012224615, + 0.004786999998032115, + 0.004355292010586709, + 0.0041148750024149194, + 0.004134457994950935, + 0.004623707995051518, + 0.0044242079893592745, + 0.0043929160019615665, + 0.004147416009800509, + 0.00458533299388364, + 0.004215624998323619, + 0.0043141249916516244, + 0.004026208000141196, + 0.004477874987060204, + 0.003790583010413684, + 0.004474040993954986, + 0.004426583007443696, + 0.004844374998356216, + 0.004850291996262968, + 0.004485124998609535, + 0.0050365000060992315, + 0.00394754100125283, + 0.004534209001576528, + 0.004678833007346839, + 0.004400791003718041, + 0.004312542005209252, + 0.004127707987208851, + 0.004773624998051673, + 0.004303374997107312, + 0.004412124995724298 + ] + }, + { + "forward": 0.006127665998064913, + "prefill": 0.0032374160073231906, + "decode_steps": [ + 0.004422625002916902, + 0.004095291005796753, + 0.004393499999423511, + 0.0038577500090468675, + 0.004355416996986605, + 0.004749915999127552, + 0.004391707989270799, + 0.004391750000650063, + 0.004234749998431653, + 0.00434625000343658, + 0.004638582991901785, + 0.004190042003756389, + 0.004123083999729715, + 0.004215207998640835, + 0.003722667010151781, + 0.004233582993037999, + 0.003629541999544017, + 0.0041702909948071465, + 0.0036802499962504953, + 0.004095040989341214, + 0.004152916997554712, + 0.004210040991893038, + 0.003698291999171488, + 0.004241707996698096, + 0.004171458000200801, + 0.004159208998316899, + 0.003615834008087404, + 0.004123042002902366, + 0.004232874998706393, + 0.004132499991101213, + 0.004230708989780396, + 0.003926416989997961 + ] + }, + { + "forward": 0.006460582997533493, + "prefill": 0.0030005409935256466, + "decode_steps": [ + 0.004427583000506274, + 0.004378707992145792, + 0.0045516249956563115, + 0.004208541999105364, + 0.005221542000072077, + 0.003965167008573189, + 0.0045148749923100695, + 0.004060708000906743, + 0.004375457996502519, + 0.004321291999076493, + 0.004037040998809971, + 0.004532083999947645, + 0.004270666991942562, + 0.0044167919986648485, + 0.004424541999469511, + 0.00456608300737571, + 0.0042284170049242675, + 0.004223417010507546, + 0.004266332995030098, + 0.003913583001121879, + 0.004283999995095655, + 0.00460420799208805, + 0.00423337500251364, + 0.0047686660109320655, + 0.004136042000027373, + 0.004023125002277084, + 0.004955083000822924, + 0.003986875002738088, + 0.004486207995796576, + 0.0045232080010464415, + 0.004710541004897095, + 0.004168833009316586 + ] + }, + { + "forward": 0.008034166996367276, + "prefill": 0.004033958000945859, + "decode_steps": [ + 0.006452917004935443, + 0.008108499998343177, + 0.005623999997624196, + 0.00498887499270495, + 0.0047116249916143715, + 0.004950167000060901, + 0.0051742079958785325, + 0.005119667010148987, + 0.0051125000027241185, + 0.005137167012435384, + 0.005342499993275851, + 0.005055832996731624, + 0.005855707990122028, + 0.005894417001400143, + 0.004666334003559314, + 0.0056433340068906546, + 0.00477600000158418, + 0.005047583996201865, + 0.005066541998530738, + 0.00621116699767299, + 0.005159333988558501, + 0.005180625012144446, + 0.004209125007037073, + 0.004557250009384006, + 0.004910792005830444, + 0.004554625003947876, + 0.00453262499650009, + 0.0042357909987913445, + 0.004750416992465034, + 0.003956792003009468, + 0.004510291997576132, + 0.004494459004490636 + ] + }, + { + "forward": 0.006415957992430776, + "prefill": 0.0040322499989997596, + "decode_steps": [ + 0.004925540997646749, + 0.004076124998391606, + 0.004547125005046837, + 0.004249875011737458, + 0.004826916992897168, + 0.004102416001842357, + 0.00417300000844989, + 0.00464941599057056, + 0.0056394999992335215, + 0.004418208001879975, + 0.004956667005899362, + 0.008151375004672445, + 0.004652292001992464, + 0.005085458004032262, + 0.004859708991716616, + 0.004499957998632453, + 0.0053280410065781325, + 0.004796666995389387, + 0.0049014579999493435, + 0.004450249994988553, + 0.00494308301131241, + 0.004997958996682428, + 0.004735124995931983, + 0.004364583001006395, + 0.004839874993194826, + 0.004492167005082592, + 0.00469600000360515, + 0.005002208999940194, + 0.004733333989861421, + 0.0048287079989677295, + 0.0043508749949978665, + 0.00405295800010208 + ] + }, + { + "forward": 0.00653845799388364, + "prefill": 0.0039440420077880844, + "decode_steps": [ + 0.004679791993112303, + 0.004881959001068026, + 0.005644290999043733, + 0.004145458995481022, + 0.005182374996365979, + 0.0043710409954655915, + 0.004058999998960644, + 0.004280541994376108, + 0.004542333001154475, + 0.004495875007705763, + 0.004409958011819981, + 0.004386375003377907, + 0.004795166998519562, + 0.005073167005321011, + 0.005185792004340328, + 0.004816999993636273, + 0.004630416005966254, + 0.004256790998624638, + 0.004590084005030803, + 0.004851457997574471, + 0.0044556669890880585, + 0.004488916005357169, + 0.004426957995747216, + 0.0045275410084286705, + 0.004494041990255937, + 0.004508874990278855, + 0.005296333009027876, + 0.004225625001708977, + 0.00448420800967142, + 0.004324084002291784, + 0.004443375000846572, + 0.004492541993386112 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 6.498750000901055, + "p95": 7.977223096531816, + "min": 6.127665998064913, + "max": 8.034166996367276, + "iqr": 0.26715625062934123, + "mean": 6.704487498791423, + "cv": 0.10202763010371298 + }, + "prefill_ms": { + "count": 10, + "median": 3.2009370042942464, + "p95": 4.033189400070114, + "min": 3.0005409935256466, + "max": 4.033958000945859, + "iqr": 0.7250417511386331, + "mean": 3.4293749005883, + "cv": 0.12441313073256137 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.393208000692539, + "p95": 5.350914543669207, + "min": 3.615834008087404, + "max": 15.747707991977222, + "iqr": 0.5068644932180177, + "mean": 4.544873690110762, + "cv": 0.1903340079588639 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 226.68170691851384, + "p95": 239.6243614364338, + "min": 193.52024670233803, + "max": 240.9053525243307, + "iqr": 23.957888669291293, + "mean": 221.2899531431136, + "cv": 0.07778371627175995 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 221.87171664601624, + "p95": 233.6933963697116, + "min": 190.03278067976422, + "max": 235.1736500244079, + "iqr": 24.063623539337783, + "mean": 216.17239327740722, + "cv": 0.07722120589073866 + }, + "memory": { + "peak_extra_bytes": 0, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "auto", + "batch": 1, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.0038004999951226637 + }, + { + "forward": 0.0040468750084983185 + }, + { + "forward": 0.00535279099131003 + }, + { + "forward": 0.004052167001646012 + }, + { + "forward": 0.004200082999886945 + }, + { + "forward": 0.003999916996690445 + }, + { + "forward": 0.003945208998629823 + }, + { + "forward": 0.006145124993054196 + }, + { + "forward": 0.004942416999256238 + }, + { + "forward": 0.00426383399462793 + } + ], + "forward_ms": { + "count": 10, + "median": 4.1261250007664785, + "p95": 5.788574692269321, + "min": 3.8004999951226637, + "max": 6.145124993054196, + "iqr": 0.7611147484567482, + "mean": 4.47489179787226, + "cv": 0.17021987548497478 + }, + "memory": { + "peak_extra_bytes": 262144, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "sdpa", + "batch": 1, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.003556582989403978, + "prefill": 0.0035482920065987855, + "decode_steps": [ + 0.0020271250104997307, + 0.0018294170004082844, + 0.0016279999981634319, + 0.0018202079954789951, + 0.0016455830045742914, + 0.0016713330114725977, + 0.0016599999944446608, + 0.002131792003638111, + 0.001839207994635217, + 0.0019156250054948032, + 0.0017633329989621416, + 0.0017622090090299025, + 0.0017063749983208254, + 0.0017244999908143654, + 0.0017323749925708398, + 0.002643583997269161, + 0.0017718750023050234, + 0.0017654169932939112, + 0.0017516660009277984, + 0.001703250003629364, + 0.0017524590075481683, + 0.001737709011649713, + 0.0017341659986414015, + 0.0017338330071652308, + 0.0017416660120943561, + 0.0017819999920902774, + 0.0017071670008590445, + 0.0017141669959528372, + 0.00174937499104999, + 0.0016717079997761175, + 0.0016961670044111088, + 0.001692000005277805 + ] + }, + { + "forward": 0.0034610829898156226, + "prefill": 0.004330291994847357, + "decode_steps": [ + 0.0019191250030416995, + 0.0017074580100597814, + 0.002157082999474369, + 0.0018322910036658868, + 0.0017791249993024394, + 0.0016845000063767657, + 0.0027737079944927245, + 0.0017051659960998222, + 0.0019299590057926252, + 0.0017346249951515347, + 0.0020045830024173483, + 0.001844416998210363, + 0.0018845830054488033, + 0.0017403749952791259, + 0.001778292004019022, + 0.001692165998974815, + 0.001746499998262152, + 0.0018303749966435134, + 0.001874082998256199, + 0.0017467919969931245, + 0.0016902080096770078, + 0.0017377909971401095, + 0.002609250004752539, + 0.0018005419988185167, + 0.0017500420071883127, + 0.0017187919875141233, + 0.0018625410011736676, + 0.0017791669961297885, + 0.0017821669898694381, + 0.0016694999940227717, + 0.0026639999996405095, + 0.0017613749951124191 + ] + }, + { + "forward": 0.003404249990126118, + "prefill": 0.0033930420031538233, + "decode_steps": [ + 0.001995999991777353, + 0.0017282500048168004, + 0.002659167003002949, + 0.0017554579972056672, + 0.0016170000017154962, + 0.0016599160007899627, + 0.0016160840023076162, + 0.001953749990207143, + 0.0017815840110415593, + 0.0016923329967539757, + 0.0016275419911835343, + 0.0016996659978758544, + 0.001804209008696489, + 0.00164958399545867, + 0.0017273749981541187, + 0.002056708006421104, + 0.00176229199860245, + 0.0017793750012060627, + 0.0018609999970067292, + 0.0027678329934133217, + 0.001803624996682629, + 0.0017555000085849315, + 0.0017641669983277097, + 0.0018952090031234547, + 0.001764417000231333, + 0.0017827080009737983, + 0.0017353750008624047, + 0.00217575000715442, + 0.0018962920003104955, + 0.0017189589998451993, + 0.0018159169994760305, + 0.002047916001174599 + ] + }, + { + "forward": 0.004189416998997331, + "prefill": 0.003499417012790218, + "decode_steps": [ + 0.001882041004137136, + 0.0020027499995194376, + 0.0017191670049214736, + 0.001771291994373314, + 0.0016726670000934973, + 0.002590999996755272, + 0.0017791249993024394, + 0.001635459004319273, + 0.0017267919902224094, + 0.0019256250088801607, + 0.0019011670083273202, + 0.001829332992201671, + 0.0017678330041235313, + 0.002556833002017811, + 0.0017512909980723634, + 0.0017076670046662912, + 0.0018471250077709556, + 0.0017981660057557747, + 0.0017298750026384369, + 0.00173200000426732, + 0.0016571660089539364, + 0.0027148750086780638, + 0.0017992500070249662, + 0.0016923329967539757, + 0.0017429159925086424, + 0.0017534590006107464, + 0.0016979159991024062, + 0.0017605419998290017, + 0.0017122089921031147, + 0.0017663750040810555, + 0.0017380839999532327, + 0.0017661250021774322 + ] + }, + { + "forward": 0.0034283340064575896, + "prefill": 0.003545958999893628, + "decode_steps": [ + 0.0019825000053970143, + 0.0017371670110151172, + 0.0019495830056257546, + 0.001827999993111007, + 0.001723666995530948, + 0.0017423329991288483, + 0.0020368749974295497, + 0.002004917012527585, + 0.0018316659989068285, + 0.0017408330022590235, + 0.0016942499933065847, + 0.001639417008846067, + 0.001738499995553866, + 0.0016270839987555519, + 0.001738082995871082, + 0.0017078750097425655, + 0.0018019159906543791, + 0.001639624999370426, + 0.0017217499989783391, + 0.0017590419884072617, + 0.0017118339892476797, + 0.0017021670064423233, + 0.0018149590032408014, + 0.001843083999119699, + 0.0018347500008530915, + 0.0019375409901840612, + 0.0017104589933296666, + 0.0017017920035868883, + 0.0024398329987889156, + 0.0017294999997830018, + 0.0017781659989850596, + 0.001934874992002733 + ] + }, + { + "forward": 0.0036363340041134506, + "prefill": 0.0034921670012408867, + "decode_steps": [ + 0.0023849580029491335, + 0.0018540000019129366, + 0.0016957089974312112, + 0.0017552909994265065, + 0.0017067500011762604, + 0.0023606670001754537, + 0.002140125012374483, + 0.0019061670027440414, + 0.0017985830054385588, + 0.0018210840062238276, + 0.0017847910057753325, + 0.0016328749916283414, + 0.001817917000153102, + 0.0017507500015199184, + 0.0018008330080192536, + 0.001687249998212792, + 0.0018690840079216287, + 0.0019573330064304173, + 0.001838458003476262, + 0.0017905000131577253, + 0.0017180420109070837, + 0.001891624997369945, + 0.0017779159970814362, + 0.00172079200274311, + 0.0017358339973725379, + 0.0019183330005034804, + 0.0017990000051213428, + 0.0016875420114956796, + 0.001745290996041149, + 0.0026487909926800057, + 0.0017841670050984249, + 0.0017313330026809126 + ] + }, + { + "forward": 0.003433374993619509, + "prefill": 0.0034016249992419034, + "decode_steps": [ + 0.0018595000001369044, + 0.001729333002003841, + 0.0017810419958550483, + 0.0017293339915340766, + 0.0020084159914404154, + 0.0017910830065375194, + 0.0016467079985886812, + 0.002076707998639904, + 0.002812709004501812, + 0.0018262919911649078, + 0.0017564170120749623, + 0.0017090420005843043, + 0.0019657080119941384, + 0.0018132919940399006, + 0.001831792003940791, + 0.001720333006232977, + 0.0018700420041568577, + 0.0017591659998288378, + 0.0017646250053076074, + 0.0016672079946147278, + 0.0019571660086512566, + 0.0017627079942030832, + 0.0017537500098114833, + 0.0016629580059088767, + 0.0026782920031109825, + 0.0017545830050949007, + 0.0017446660058340058, + 0.0016088749980553985, + 0.0019547919946489856, + 0.0018189169932156801, + 0.0017738749884301797, + 0.0017421670054318383 + ] + }, + { + "forward": 0.00751829199725762, + "prefill": 0.005416041996795684, + "decode_steps": [ + 0.0024874579976312816, + 0.0017570000054547563, + 0.0028320419951342046, + 0.0017875000048661605, + 0.0016965830000117421, + 0.0018233749869978055, + 0.002022874992690049, + 0.001820333010982722, + 0.0018445419991621748, + 0.0017372080037603155, + 0.0017629590001888573, + 0.0018857079994631931, + 0.0017928750021383166, + 0.001732375007122755, + 0.0019307920010760427, + 0.0018199170008301735, + 0.001764000000548549, + 0.0017188329948112369, + 0.0026407080003991723, + 0.0017405409889761358, + 0.0017299160099355504, + 0.00178899998718407, + 0.0018150830001104623, + 0.0017690829990897328, + 0.00177425000583753, + 0.0016892499988898635, + 0.0025798339920584112, + 0.0018184590007876977, + 0.0017831250006565824, + 0.0017213749961229041, + 0.0019854170095641166, + 0.0018160420004278421 + ] + }, + { + "forward": 0.0034034169948427007, + "prefill": 0.0034747919999063015, + "decode_steps": [ + 0.0021338340011425316, + 0.0018050000071525574, + 0.0018199170008301735, + 0.001776959004928358, + 0.001659208006458357, + 0.003595083995605819, + 0.002598915991256945, + 0.001935583000886254, + 0.0019443339988356456, + 0.002842332993168384, + 0.0017545409937156364, + 0.0017309999966528267, + 0.0018806669977493584, + 0.001879792005638592, + 0.0016699169937055558, + 0.0017588329938007519, + 0.0016527080006198958, + 0.0019595000048866495, + 0.001751208008499816, + 0.0016987920098472387, + 0.0017693750123726204, + 0.0020364590018289164, + 0.0017714169953251258, + 0.0017147500038845465, + 0.001736165999318473, + 0.002445124991936609, + 0.0018408750038361177, + 0.0017409159918315709, + 0.0017422910023014992, + 0.002016541999182664, + 0.0018312090105609968, + 0.0017814169987104833 + ] + }, + { + "forward": 0.0035292079992359504, + "prefill": 0.0033372500038240105, + "decode_steps": [ + 0.0020743329951073974, + 0.0018886250036302954, + 0.0017408749990863726, + 0.00175350000790786, + 0.0018793749914038926, + 0.0017435829940950498, + 0.0016374160040868446, + 0.0017002499953377992, + 0.0017602080042706802, + 0.0017027080029947683, + 0.001626875004149042, + 0.00169237500813324, + 0.001727666996885091, + 0.001773166994098574, + 0.001867458995548077, + 0.001873917004559189, + 0.0019034160068258643, + 0.0017678750009508803, + 0.00167891698947642, + 0.0016715829988243058, + 0.0027913330122828484, + 0.0017694580019451678, + 0.00161087499873247, + 0.0018544580088928342, + 0.0020217080018483102, + 0.0018195829907199368, + 0.0018353329942328855, + 0.0017495409992989153, + 0.0019547499978216365, + 0.001723042005323805, + 0.0018953329999931157, + 0.0018010839994531125 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 3.4951454945257865, + "p95": 6.020298248040486, + "min": 3.4034169948427007, + "max": 7.51829199725762, + "iqr": 0.18680199718801305, + "mean": 3.956029296386987, + "cv": 0.32189545781475026 + }, + "prefill_ms": { + "count": 10, + "median": 3.4957920070155524, + "p95": 4.927454495918936, + "min": 3.3372500038240105, + "max": 5.416041996795684, + "iqr": 0.1277920055144932, + "mean": 3.74388780182926, + "cv": 0.17398201456373075 + }, + "decode_ms_per_step": { + "count": 320, + "median": 1.770374998159241, + "p95": 2.591395796480356, + "min": 1.6088749980553985, + "max": 3.595083995605819, + "iqr": 0.14290650142356753, + "mean": 1.8560511661689816, + "cv": 0.14135901736800358 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 539.2540877973437, + "p95": 556.8675897762494, + "min": 513.8525988052854, + "max": 559.3888058791503, + "iqr": 13.868053325066171, + "mean": 539.0780691445375, + "cv": 0.024757670116821305 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 510.0531227861848, + "p95": 524.490204915706, + "min": 482.76746456516327, + "max": 526.717896666517, + "iqr": 18.224031955730197, + "mean": 507.2123382824863, + "cv": 0.028742087515330157 + }, + "memory": { + "peak_extra_bytes": 262144, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "reference", + "batch": 1, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.00968933300464414, + "prefill": 0.006000459005008452, + "decode_steps": [ + 0.004535291009233333, + 0.0047210410120896995, + 0.0045228329981910065, + 0.004132125002797693, + 0.004204208991723135, + 0.00454366700432729, + 0.004090917005669326, + 0.004608624993124977, + 0.004334915996878408, + 0.004381874998216517, + 0.0045439999958034605, + 0.004134208007599227, + 0.004044833010993898, + 0.004665417000069283, + 0.004517749999649823, + 0.004175875001237728, + 0.004231958999298513, + 0.004424707993166521, + 0.004263582988642156, + 0.004235500004142523, + 0.004143417012528516, + 0.004039084000396542, + 0.004235208994941786, + 0.004235000000335276, + 0.004173042005277239, + 0.00430979099473916, + 0.004092540999408811, + 0.00420549999398645, + 0.0041548750014044344, + 0.0036234580038581043, + 0.004156875002081506, + 0.004206957994028926 + ] + }, + { + "forward": 0.00980637499014847, + "prefill": 0.007575957992230542, + "decode_steps": [ + 0.005349582992494106, + 0.0047152079932857305, + 0.003992416997789405, + 0.0041857080068439245, + 0.0048614999977871776, + 0.004909208000754006, + 0.00457616698986385, + 0.004775959008838981, + 0.004590500000631437, + 0.004121124991797842, + 0.0042132090020459145, + 0.004484707998926751, + 0.004560457993648015, + 0.004318250008509494, + 0.0043731670011766255, + 0.004664457999751903, + 0.004168208004557528, + 0.004300749991671182, + 0.004558000000542961, + 0.004291125005693175, + 0.004410334004205652, + 0.004132709000259638, + 0.004706917010480538, + 0.004388791989185847, + 0.004226000004564412, + 0.004482040996663272, + 0.004340875006164424, + 0.0041702920134412125, + 0.004304874993977137, + 0.0043049579981015995, + 0.004180666990578175, + 0.004258999993908219 + ] + }, + { + "forward": 0.008636208003736101, + "prefill": 0.007229249997180887, + "decode_steps": [ + 0.004659959013224579, + 0.0043905419879592955, + 0.004415292001795024, + 0.0037942079943604767, + 0.004556084008072503, + 0.00426008299109526, + 0.004286666997359134, + 0.00430179201066494, + 0.004132250003749505, + 0.003887208004016429, + 0.0041369999962626025, + 0.004200707990094088, + 0.00421262500458397, + 0.004207459001918323, + 0.004102499995497055, + 0.004161000004387461, + 0.004161750010098331, + 0.004103375002159737, + 0.004050583986099809, + 0.004018457999336533, + 0.004686125001171604, + 0.00401445799798239, + 0.00454441701003816, + 0.004348542002844624, + 0.004356374993221834, + 0.004176415997790173, + 0.0043456660059746355, + 0.004146666004089639, + 0.004385041000205092, + 0.004261833004420623, + 0.004262583010131493, + 0.004337709004175849 + ] + }, + { + "forward": 0.009908374995575286, + "prefill": 0.0070871670031920075, + "decode_steps": [ + 0.00597449998895172, + 0.004819166992092505, + 0.004038958999444731, + 0.004209666003589518, + 0.0049993749998975545, + 0.004120708006666973, + 0.00436608400195837, + 0.004495874993153848, + 0.004434375005075708, + 0.004226666002068669, + 0.004257208987837657, + 0.004344917004345916, + 0.004212375002680346, + 0.004058542006532662, + 0.004489458995522, + 0.004371167000499554, + 0.004300790998968296, + 0.004279708999092691, + 0.004460874988581054, + 0.0047047080006450415, + 0.004896665996056981, + 0.0044596669904422015, + 0.004553709004539996, + 0.00438437498814892, + 0.004330291994847357, + 0.004292874989914708, + 0.004258624991052784, + 0.004922540989355184, + 0.004015667000203393, + 0.0046473750117002055, + 0.004123666993109509, + 0.004360332997748628 + ] + }, + { + "forward": 0.010162958002183586, + "prefill": 0.006124417006503791, + "decode_steps": [ + 0.005917458998737857, + 0.004657165991375223, + 0.0044024579983670264, + 0.00503166600537952, + 0.00442420800391119, + 0.004247208999004215, + 0.0043411659862613305, + 0.004839415996684693, + 0.004034875004435889, + 0.0044251250074012205, + 0.00488066699472256, + 0.004515833003097214, + 0.00388812500750646, + 0.0042461249977350235, + 0.004731249995529652, + 0.005207750000408851, + 0.004363333006040193, + 0.004144499995163642, + 0.004486834004637785, + 0.004247958000632934, + 0.0047774580016266555, + 0.0049412500084144995, + 0.006089457994676195, + 0.004205000004731119, + 0.005187375005334616, + 0.0053555000049527735, + 0.004446791994269006, + 0.00470925000263378, + 0.004151833010837436, + 0.00376604100165423, + 0.00411241699475795, + 0.004983999999240041 + ] + }, + { + "forward": 0.009088040998904034, + "prefill": 0.005960957991192117, + "decode_steps": [ + 0.004503333999309689, + 0.0041229170019505545, + 0.0044987910077907145, + 0.004595833001076244, + 0.004847874995903112, + 0.004803707997780293, + 0.004153000001679175, + 0.00455858300847467, + 0.0047075829934328794, + 0.00457970799470786, + 0.004036750004161149, + 0.004313792000175454, + 0.004417457996169105, + 0.004449583007954061, + 0.004233625004417263, + 0.0043516250007087365, + 0.004311083001084626, + 0.004011874989373609, + 0.003996583996922709, + 0.004135292008868419, + 0.004333416989538819, + 0.0043904160120291635, + 0.004335833000368439, + 0.004158374998951331, + 0.004444249992957339, + 0.003959209003369324, + 0.004327041999204084, + 0.004447541999979876, + 0.004234708001604304, + 0.004227792000165209, + 0.00430291700467933, + 0.004621583997504786 + ] + }, + { + "forward": 0.010025915995356627, + "prefill": 0.006050832991604693, + "decode_steps": [ + 0.005368416997953318, + 0.0044611670018639416, + 0.004949209003825672, + 0.004717124989838339, + 0.0045729590055998415, + 0.004623000000719912, + 0.004799125003046356, + 0.00446587499754969, + 0.0043125829979544505, + 0.004360250008176081, + 0.00422495900420472, + 0.004469999999855645, + 0.0047593329945812, + 0.004150000007939525, + 0.004171624997979961, + 0.00411895799334161, + 0.0045110840001143515, + 0.004864292001002468, + 0.004520125003182329, + 0.0039238330064108595, + 0.00435970799298957, + 0.004304874993977137, + 0.004014749996713363, + 0.004465541001991369, + 0.004334500001277775, + 0.003994792001321912, + 0.004218041009153239, + 0.004304083005990833, + 0.004564874994684942, + 0.004421375007950701, + 0.004996499992557801, + 0.004832792008528486 + ] + }, + { + "forward": 0.010954792000120506, + "prefill": 0.006203375000040978, + "decode_steps": [ + 0.0061959580052644014, + 0.004786999998032115, + 0.005580083001405001, + 0.005783540997072123, + 0.00450775001081638, + 0.004708166990894824, + 0.004351332987425849, + 0.004644083994207904, + 0.004086957997060381, + 0.0045894580107415095, + 0.004088499990757555, + 0.004350292001618072, + 0.004358082995167933, + 0.004207499994663522, + 0.00425970800279174, + 0.004269708995707333, + 0.00515058399469126, + 0.004709125001681969, + 0.00443391699809581, + 0.004622000007657334, + 0.004126459010876715, + 0.004118999990168959, + 0.004093958006706089, + 0.004394540999783203, + 0.00412812500144355, + 0.004304082991438918, + 0.004054709002957679, + 0.004276499996194616, + 0.00438679099897854, + 0.00437166600022465, + 0.0047512079909211025, + 0.00525862499489449 + ] + }, + { + "forward": 0.009592791990144178, + "prefill": 0.00754837499698624, + "decode_steps": [ + 0.0051220419991295785, + 0.004727124993223697, + 0.0038918330101296306, + 0.004370166992885061, + 0.004995958006475121, + 0.004515125008765608, + 0.004453625006135553, + 0.004429874999914318, + 0.004306750008254312, + 0.0045672080013900995, + 0.004386291999253444, + 0.004860000000917353, + 0.004164291996858083, + 0.004612667005858384, + 0.015947500010952353, + 0.009199666004860774, + 0.00515112500579562, + 0.004369125002995133, + 0.005528458001208492, + 0.005089167010737583, + 0.0048677919985493645, + 0.005741916989791207, + 0.005168582996702753, + 0.0046679579972987995, + 0.005130500008817762, + 0.005666915996698663, + 0.0043672080064425245, + 0.004908416987746023, + 0.004419499993673526, + 0.004623166998499073, + 0.004265374998794869, + 0.004302582994569093 + ] + }, + { + "forward": 0.010303292001481168, + "prefill": 0.006660541010205634, + "decode_steps": [ + 0.005783208005595952, + 0.0043847500055562705, + 0.004602957997121848, + 0.004953583993483335, + 0.004531625003437512, + 0.004571375000523403, + 0.004695333002018742, + 0.004299166001146659, + 0.004393541006720625, + 0.004491792002227157, + 0.004823166993446648, + 0.004533832994638942, + 0.004067917005158961, + 0.004124957995372824, + 0.004625750007107854, + 0.004011750002973713, + 0.004339374994742684, + 0.004183165990980342, + 0.004370832990389317, + 0.003802292005275376, + 0.004138165997574106, + 0.007537292010965757, + 0.01190395800222177, + 0.017541458990308456, + 0.01570516699575819, + 0.014877790992613882, + 0.007390833998215385, + 0.011728083991329186, + 0.01039270800538361, + 0.005229125003097579, + 0.004746042002807371, + 0.004407667001942173 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 9.857374992861878, + "p95": 10.661617000732804, + "min": 8.636208003736101, + "max": 10.954792000120506, + "iqr": 0.5117702567076776, + "mean": 9.81680819822941, + "cv": 0.06527087615776636 + }, + "prefill_ms": { + "count": 10, + "median": 6.431958005123306, + "p95": 7.563545644370606, + "min": 5.960957991192117, + "max": 7.575957992230542, + "iqr": 1.1245002533541992, + "mean": 6.644133299414534, + "cv": 0.09935961077794253 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.383124993182719, + "p95": 5.743981540581445, + "min": 3.6234580038581043, + "max": 17.541458990308456, + "iqr": 0.4461664939299226, + "mean": 4.704393353131309, + "cv": 0.32609751840675044 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 224.1173152598834, + "p95": 234.69652523742562, + "min": 155.9540330631049, + "max": 235.4579426443619, + "iqr": 11.282795316143108, + "mean": 215.60796024369924, + "cv": 0.11254239452907246 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 214.05311416726096, + "p95": 223.77665429925705, + "min": 151.05083683390163, + "max": 223.9492111975552, + "iqr": 10.16899667617426, + "mean": 206.3048697679722, + "cv": 0.10931924824078731 + }, + "memory": { + "peak_extra_bytes": 33816576, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "auto", + "batch": 1, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.00805062499421183 + }, + { + "forward": 0.009125334006967023 + }, + { + "forward": 0.009036916992045008 + }, + { + "forward": 0.008089499999186955 + }, + { + "forward": 0.008124708998366259 + }, + { + "forward": 0.009374332992592826 + }, + { + "forward": 0.008125833002850413 + }, + { + "forward": 0.008584542010794394 + }, + { + "forward": 0.008197834002203308 + }, + { + "forward": 0.008007290991372429 + } + ], + "forward_ms": { + "count": 10, + "median": 8.16183350252686, + "p95": 9.262283449061215, + "min": 8.007290991372429, + "max": 9.374332992592826, + "iqr": 0.8255209977505729, + "mean": 8.471691799059045, + "cv": 0.06131368019076238 + }, + "memory": { + "peak_extra_bytes": 33554432, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "sdpa", + "batch": 1, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.006316250000963919, + "prefill": 0.005976457992801443, + "decode_steps": [ + 0.0020530409965431318, + 0.0018313340115128085, + 0.001958042004844174, + 0.002997208997840062, + 0.0018802080012392253, + 0.0017733749991748482, + 0.0018915420077973977, + 0.001893125008791685, + 0.0018191249982919544, + 0.0018522500031394884, + 0.002191624997067265, + 0.0019157079950673506, + 0.0017949159955605865, + 0.0018854999943869188, + 0.0017432919994462281, + 0.001884375000372529, + 0.001746499998262152, + 0.001796165990526788, + 0.0017795419989852235, + 0.0017698750016279519, + 0.0018046249897452071, + 0.0017816670006141067, + 0.0017882079991977662, + 0.0017779999907361344, + 0.0018254590104334056, + 0.001747791000525467, + 0.0022839169978396967, + 0.0017465419950895011, + 0.0018002500000875443, + 0.001851375011028722, + 0.0017739579925546423, + 0.0018420829874230549 + ] + }, + { + "forward": 0.006554416992003098, + "prefill": 0.006114916002843529, + "decode_steps": [ + 0.002036290999967605, + 0.00193162499635946, + 0.0017766249948181212, + 0.0021636250021401793, + 0.0019385409977985546, + 0.0018916669941972941, + 0.0018184589862357825, + 0.001880333002191037, + 0.0019079159974353388, + 0.0018532079993747175, + 0.0018399589898763224, + 0.002206750010373071, + 0.0017495409992989153, + 0.0018797499942593277, + 0.0018532909889472649, + 0.0018283329991390929, + 0.0018433340010233223, + 0.0018059580033877864, + 0.001782457999070175, + 0.0025015829887706786, + 0.0017806249961722642, + 0.0018040840077446774, + 0.0017322089988738298, + 0.0018351659964537248, + 0.0018649580015335232, + 0.0018495420081308112, + 0.0017789169942261651, + 0.002284707996295765, + 0.0018899580027209595, + 0.001786124994396232, + 0.0019350419897818938, + 0.0018095840059686452 + ] + }, + { + "forward": 0.00624012500338722, + "prefill": 0.007027500003459863, + "decode_steps": [ + 0.0024564170016674325, + 0.0020606249890988693, + 0.0020374169980641454, + 0.0018839169933926314, + 0.001822416001232341, + 0.0016890830011107028, + 0.002507707991753705, + 0.0018332500039832667, + 0.0019018330058315769, + 0.0020859590003965423, + 0.001972792000742629, + 0.0018355420033913106, + 0.0018414999940432608, + 0.0019938329933211207, + 0.0019337499979883432, + 0.0018974580016219988, + 0.0018837079987861216, + 0.0018756670033326373, + 0.0019101250072708353, + 0.0017640840087551624, + 0.0019896250014426187, + 0.0019074580050073564, + 0.0018559589952928945, + 0.0018269169959239662, + 0.001964374998351559, + 0.0018638750043464825, + 0.001869833009550348, + 0.0018034999957308173, + 0.0018162499909522012, + 0.002017874998273328, + 0.0020999170083086938, + 0.001961500005563721 + ] + }, + { + "forward": 0.006700042009470053, + "prefill": 0.006145874998765066, + "decode_steps": [ + 0.0022402500035241246, + 0.0019054160075029358, + 0.0020951249898644164, + 0.0019539159984560683, + 0.0018053749954560772, + 0.0017852080054581165, + 0.00200899998890236, + 0.0018713750032475218, + 0.0018314580083824694, + 0.001783541010809131, + 0.0022314999951049685, + 0.00209612499747891, + 0.0018985409988090396, + 0.0017939590034075081, + 0.0019064590014750138, + 0.0019298330007586628, + 0.0018294579931534827, + 0.0018401660054223612, + 0.001755459001287818, + 0.0017692499968688935, + 0.0018139170133508742, + 0.0018121670000255108, + 0.001850625005317852, + 0.00184208300197497, + 0.001834292008425109, + 0.0017502080008853227, + 0.0026479579973965883, + 0.0020229579968145117, + 0.0018045420001726598, + 0.001871916014351882, + 0.001939583002240397, + 0.0018285410042153671 + ] + }, + { + "forward": 0.006249458994716406, + "prefill": 0.006178540992550552, + "decode_steps": [ + 0.002448083003400825, + 0.0020787079993169755, + 0.0019404999911785126, + 0.0018640829948708415, + 0.0030387499864446, + 0.0019486670062178746, + 0.00181462500768248, + 0.001870250009233132, + 0.0021206669916864485, + 0.0018720829975791276, + 0.00186191700049676, + 0.0018362910050200298, + 0.0020287920051487163, + 0.0019305830064695328, + 0.0018838329997379333, + 0.002236916989204474, + 0.00198229200032074, + 0.0018438330007484183, + 0.0017358339973725379, + 0.0017921250109793618, + 0.0018062909948639572, + 0.0017578749975655228, + 0.00176229199860245, + 0.002100500001688488, + 0.00187070899119135, + 0.001746334004565142, + 0.0018781249964376912, + 0.0017726660007610917, + 0.0019816669955616817, + 0.0018945420015370473, + 0.0017336250020889565, + 0.001819040990085341 + ] + }, + { + "forward": 0.0064755829953355715, + "prefill": 0.0062497090111719444, + "decode_steps": [ + 0.0020625000033760443, + 0.0017875420016935095, + 0.0018212080030934885, + 0.0017781250062398612, + 0.0017966250015888363, + 0.001987957992241718, + 0.001868915991508402, + 0.0018440409912727773, + 0.001746499998262152, + 0.0018260419892612845, + 0.0017582919972483069, + 0.0018392920028418303, + 0.0017909159942064434, + 0.001991750003071502, + 0.0018755420023808256, + 0.0019004999921889976, + 0.0017606670007808134, + 0.0019619579979917035, + 0.0018331659957766533, + 0.0017948750028153881, + 0.0018560839962447062, + 0.00198041700059548, + 0.0018222910002805293, + 0.0018653750012163073, + 0.0018616249872138724, + 0.0019102090009255335, + 0.0017490830068709329, + 0.001903000011225231, + 0.0018227499967906624, + 0.0017551249911775813, + 0.0017485830030636862, + 0.0017861250089481473 + ] + }, + { + "forward": 0.006320542001049034, + "prefill": 0.006603832996916026, + "decode_steps": [ + 0.002208707999670878, + 0.001919792004628107, + 0.0021740419906564057, + 0.0022789170034229755, + 0.0021904580062255263, + 0.002356458004214801, + 0.0022975419997237623, + 0.001966417010407895, + 0.0020021250093122944, + 0.0021503750031115487, + 0.0021217500034254044, + 0.0021729160071117803, + 0.0020841660007135943, + 0.002187417005188763, + 0.0022327920014504343, + 0.0022642080002697185, + 0.002203750002081506, + 0.002030666990322061, + 0.0021238750050542876, + 0.0021429170010378584, + 0.002311208998435177, + 0.002069541995297186, + 0.0022369579965015873, + 0.0022234999923966825, + 0.002442625002004206, + 0.0022884589998284355, + 0.0022237079974729568, + 0.002298084000358358, + 0.0022874159913044423, + 0.002245459007099271, + 0.002128125008312054, + 0.0020967089949408546 + ] + }, + { + "forward": 0.0063000839872984216, + "prefill": 0.0063635830010753125, + "decode_steps": [ + 0.0020772499992744997, + 0.0023266659991350025, + 0.0018639589980011806, + 0.00188079199870117, + 0.0017875840130727738, + 0.0017844590038293973, + 0.0017810409917728975, + 0.0019047500099986792, + 0.001815625000745058, + 0.0018507500062696636, + 0.0018365419964538887, + 0.0018565419886726886, + 0.0018814170034602284, + 0.001771916009602137, + 0.001767583002219908, + 0.0017890419985633343, + 0.001816000003600493, + 0.0016967920091701671, + 0.001822499994887039, + 0.001805500010959804, + 0.001743625005474314, + 0.0018328330043004826, + 0.0017863749962998554, + 0.0018317499925615266, + 0.0017961249977815896, + 0.0017258750012842938, + 0.0017087090091081336, + 0.0018328750011278316, + 0.0017790830024750903, + 0.001745500005199574, + 0.0018593339918879792, + 0.002056958997854963 + ] + }, + { + "forward": 0.006374750009854324, + "prefill": 0.006141083998954855, + "decode_steps": [ + 0.0021275000035529956, + 0.0019267919997218996, + 0.001792915994883515, + 0.0017678750009508803, + 0.0017720830073812976, + 0.001993124998989515, + 0.002021457999944687, + 0.0019742909935303032, + 0.0017872500029625371, + 0.001826082996558398, + 0.001777707992005162, + 0.002666165994014591, + 0.0018650420097401366, + 0.0017791249993024394, + 0.001970458004507236, + 0.0018702079978538677, + 0.0017906249995576218, + 0.0018214580049971119, + 0.001915624990942888, + 0.0018738329963525757, + 0.0018920830043498427, + 0.001849916996434331, + 0.001772708012140356, + 0.001775042008375749, + 0.0018877499969676137, + 0.001830458000767976, + 0.001835834002122283, + 0.002441499993437901, + 0.0017731249972712249, + 0.0018367079901508987, + 0.0019376249983906746, + 0.0017519590037409216 + ] + }, + { + "forward": 0.006106874992838129, + "prefill": 0.006155208000564016, + "decode_steps": [ + 0.00197966699488461, + 0.0020431250013643876, + 0.001853709007264115, + 0.0018041250004898757, + 0.0018978340085595846, + 0.0019099169876426458, + 0.0018905409961007535, + 0.0017772919964045286, + 0.0018153749988414347, + 0.0017878750077215955, + 0.0017856250051409006, + 0.0017858749924926087, + 0.0017939580138772726, + 0.0017665829946054146, + 0.0019721249991562217, + 0.0019265000009909272, + 0.001790833004633896, + 0.0024036669929046184, + 0.0019249169999966398, + 0.0019113750022370368, + 0.0019094169983873144, + 0.001856959002907388, + 0.001937499997438863, + 0.0017894579941639677, + 0.0018610419938340783, + 0.002388416003668681, + 0.0019840830063913018, + 0.0018814159993780777, + 0.0018559169984655455, + 0.0019252079946454614, + 0.0019186249992344528, + 0.001811666996218264 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 6.318396001006477, + "p95": 6.634510751609923, + "min": 6.106874992838129, + "max": 6.700042009470053, + "iqr": 0.18825950610334985, + "mean": 6.363812698691618, + "cv": 0.026942657018448044 + }, + "prefill_ms": { + "count": 10, + "median": 6.166874496557284, + "p95": 6.8368498505151365, + "min": 5.976457992801443, + "max": 7.027500003459863, + "iqr": 0.19283275469206274, + "mean": 6.295670699910261, + "cv": 0.04886060168351411 + }, + "decode_ms_per_step": { + "count": 320, + "median": 1.8645204982021824, + "p95": 2.297569099755492, + "min": 1.6890830011107028, + "max": 3.0387499864446, + "iqr": 0.18051074584946036, + "mean": 1.9274891842997022, + "cv": 0.1013007383924697 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 525.2577356474465, + "p95": 542.947650789852, + "min": 457.3971305329947, + "max": 544.0719253598475, + "iqr": 11.839965941254945, + "mean": 519.9018693644705, + "cv": 0.04624748004246355 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 477.1090921322941, + "p95": 490.4207287527767, + "min": 417.9459784698875, + "max": 490.95314468982946, + "iqr": 11.374976517444907, + "mean": 471.654251624241, + "cv": 0.04406600680655123 + }, + "memory": { + "peak_extra_bytes": 0, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "reference", + "batch": 1, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.020958541994332336, + "prefill": 0.017802417001803406, + "decode_steps": [ + 0.0057217919966205955, + 0.005132833000971004, + 0.004474833011045121, + 0.004002083995146677, + 0.004506541998125613, + 0.0047718339919811115, + 0.0048361670051235706, + 0.004355208002380095, + 0.00388212499092333, + 0.004320749998441897, + 0.004510833998210728, + 0.004151750006712973, + 0.003863417005049996, + 0.004373750009108335, + 0.003733499994268641, + 0.0041417500033276156, + 0.004259416004060768, + 0.004774959001224488, + 0.0041046249971259385, + 0.003991040997789241, + 0.004772915999637917, + 0.004523750001681037, + 0.003984082999522798, + 0.004001166991656646, + 0.004524083997239359, + 0.005099125002743676, + 0.004523666997556575, + 0.004620707986759953, + 0.004381499995361082, + 0.004324124995036982, + 0.004233999992720783, + 0.004104541993001476 + ] + }, + { + "forward": 0.02059012500103563, + "prefill": 0.016630500002065673, + "decode_steps": [ + 0.00417595800536219, + 0.0043508749949978665, + 0.004467082995688543, + 0.0042741249926621094, + 0.0037577920011244714, + 0.004227333003655076, + 0.0044395830045687035, + 0.004278958003851585, + 0.0038599169929511845, + 0.004252499988069758, + 0.00427841700729914, + 0.0047847499954514205, + 0.004268082993803546, + 0.004949292007950135, + 0.004486500009079464, + 0.0044212909997440875, + 0.003780166996875778, + 0.004176540998741984, + 0.0045516249956563115, + 0.004522458999417722, + 0.004419250006321818, + 0.004327625007135794, + 0.003776375000597909, + 0.00427483300154563, + 0.004335500008892268, + 0.004341792009654455, + 0.00390720900031738, + 0.004182999997283332, + 0.0051077919924864545, + 0.00450849998742342, + 0.0044144160056021065, + 0.004535500003839843 + ] + }, + { + "forward": 0.01983591700263787, + "prefill": 0.017902124993270263, + "decode_steps": [ + 0.006244791991775855, + 0.004718500000308268, + 0.0044474159949459136, + 0.004540916997939348, + 0.004636083001969382, + 0.00417149999702815, + 0.004238291003275663, + 0.005591749999439344, + 0.00458737499138806, + 0.004758082999614999, + 0.00493337499210611, + 0.004922167005133815, + 0.00447745798737742, + 0.00434270799451042, + 0.004443375000846572, + 0.004036458005430177, + 0.004315458005294204, + 0.0048163749888772145, + 0.004575875005684793, + 0.004610167001374066, + 0.005070665996754542, + 0.0051888750022044405, + 0.005493832999491133, + 0.00489825000113342, + 0.004733791996841319, + 0.0041569160093786195, + 0.004554625003947876, + 0.004979999997885898, + 0.005145458999322727, + 0.004612874996382743, + 0.004614375007804483, + 0.004809749996638857 + ] + }, + { + "forward": 0.022081332994275726, + "prefill": 0.016779541998403147, + "decode_steps": [ + 0.006047332994057797, + 0.005442583002150059, + 0.00596554100047797, + 0.004405833999044262, + 0.004866415998549201, + 0.005226249995757826, + 0.004725042002974078, + 0.004608832998201251, + 0.004912749995128252, + 0.004673917006584816, + 0.0055099580058595166, + 0.005112792001455091, + 0.004282666006474756, + 0.004314833000535145, + 0.005069709004601464, + 0.0042752919980557635, + 0.004784667005878873, + 0.004708292006398551, + 0.004686917003709823, + 0.004082667001057416, + 0.004354290998890065, + 0.004693916998803616, + 0.004343916996731423, + 0.004421999998157844, + 0.004792458013980649, + 0.004419582997797988, + 0.004785209006513469, + 0.0044112080067861825, + 0.004699207987869158, + 0.004556999992928468, + 0.004492583000683226, + 0.004629000002751127 + ] + }, + { + "forward": 0.02019108401145786, + "prefill": 0.01651716700871475, + "decode_steps": [ + 0.005530791007913649, + 0.00501224999607075, + 0.00471616699360311, + 0.0046426250046351925, + 0.004481500000110827, + 0.004112833004910499, + 0.004736957998829894, + 0.003966832999140024, + 0.004988916000002064, + 0.004839874993194826, + 0.0046565840020775795, + 0.004811167003936134, + 0.004953249997925013, + 0.004800291993888095, + 0.004584875001455657, + 0.004870541990385391, + 0.0045074170047882944, + 0.0043672500032698736, + 0.0043744160066125914, + 0.0043886659987038, + 0.004200082999886945, + 0.004524999996647239, + 0.004412875001435168, + 0.004635874996893108, + 0.004839666988118552, + 0.004613833996700123, + 0.004571416007820517, + 0.00446008300059475, + 0.004342457992606796, + 0.004420749988639727, + 0.0043624999962048605, + 0.004231416998663917 + ] + }, + { + "forward": 0.019063458006712608, + "prefill": 0.016095833998406306, + "decode_steps": [ + 0.004426500003319234, + 0.0037787079927511513, + 0.004265750001650304, + 0.00418904100661166, + 0.004327167000155896, + 0.004254832994774915, + 0.003960957998060621, + 0.0044538750080391765, + 0.004803208008524962, + 0.003974917010054924, + 0.004387209002743475, + 0.003986541996710002, + 0.005507708003278822, + 0.004242959010298364, + 0.004563916998449713, + 0.0038177499955054373, + 0.004505874996539205, + 0.004138792006415315, + 0.004425791994435713, + 0.004300500004319474, + 0.0038201249990379438, + 0.004440625009010546, + 0.004614667006535456, + 0.003884125006152317, + 0.0045190840028226376, + 0.004324207999161445, + 0.004786416000570171, + 0.003926208999473602, + 0.004407957996590994, + 0.005191958000068553, + 0.00676891699549742, + 0.004506833996856585 + ] + }, + { + "forward": 0.0218847500073025, + "prefill": 0.01656270799867343, + "decode_steps": [ + 0.0053389160020742565, + 0.004617416998371482, + 0.004601250009727664, + 0.0044125829881522804, + 0.004377082994324155, + 0.004830999998375773, + 0.004590875003486872, + 0.0039037079986883327, + 0.00426570899435319, + 0.004283124988432974, + 0.004446542006917298, + 0.004338166996603832, + 0.004694792005466297, + 0.00429054100823123, + 0.004966208012774587, + 0.004327834001742303, + 0.004590707991155796, + 0.004493291999096982, + 0.004453708010260016, + 0.004199666000204161, + 0.004910417002975009, + 0.005228416994214058, + 0.0047134580090641975, + 0.004612582997651771, + 0.004401209007482976, + 0.0045167080097598955, + 0.004377124991151504, + 0.004424500002642162, + 0.003909791994374245, + 0.004477499998756684, + 0.004906749993097037, + 0.004701915997429751 + ] + }, + { + "forward": 0.02162658299494069, + "prefill": 0.01644000000669621, + "decode_steps": [ + 0.005147207994014025, + 0.004360167004051618, + 0.004904959001578391, + 0.003918292000889778, + 0.00434083299478516, + 0.004266500007361174, + 0.0046786250022705644, + 0.003894791007041931, + 0.004089833004400134, + 0.00451187499857042, + 0.0043013329996028915, + 0.004960833000950515, + 0.005603249999694526, + 0.004875333004747517, + 0.005065540986834094, + 0.004917250000289641, + 0.004783541997312568, + 0.00545262498781085, + 0.004529042009380646, + 0.005221249986789189, + 0.004602500004693866, + 0.004398582997964695, + 0.004839999994146638, + 0.004615750003722496, + 0.004594582991558127, + 0.004626875001122244, + 0.004410333000123501, + 0.003946124998037703, + 0.004309875002945773, + 0.0041849999979604036, + 0.004833541999687441, + 0.0047746250056661665 + ] + }, + { + "forward": 0.021099582998431288, + "prefill": 0.016695957994670607, + "decode_steps": [ + 0.0053224169969325885, + 0.005281749996356666, + 0.004670709007768892, + 0.004812332990695722, + 0.004818291999981739, + 0.004913208991638385, + 0.004641291001462378, + 0.0044573329942068085, + 0.004673957999330014, + 0.004436040995642543, + 0.0044327500072540715, + 0.004401665995828807, + 0.004539790999842808, + 0.0041229170019505545, + 0.004489750004722737, + 0.004614166988176294, + 0.004554583996650763, + 0.00450420800189022, + 0.005207375012105331, + 0.004439791999175213, + 0.00486441700195428, + 0.004089333990123123, + 0.004813958003069274, + 0.004815457999939099, + 0.00454945900128223, + 0.004688165994593874, + 0.004382125000120141, + 0.004576457999064587, + 0.004496167006436735, + 0.004702874997747131, + 0.0046187910047592595, + 0.004345042005297728 + ] + }, + { + "forward": 0.01966483300202526, + "prefill": 0.016653541009873152, + "decode_steps": [ + 0.005287292005959898, + 0.004925625005853362, + 0.0046370830095838755, + 0.004141124998568557, + 0.004384333005873486, + 0.004373333998955786, + 0.003868542000418529, + 0.004369874994154088, + 0.004224459000397474, + 0.00444291700841859, + 0.004400334000820294, + 0.004250416997820139, + 0.004366792010841891, + 0.004403499988256954, + 0.004171124994172715, + 0.004343375010648742, + 0.004557874999591149, + 0.004343833003076725, + 0.004233292012941092, + 0.004371916991658509, + 0.004320958993048407, + 0.004339207996963523, + 0.004380709011456929, + 0.004317957995226607, + 0.004412624999531545, + 0.004331749994889833, + 0.004286625000531785, + 0.004429332999279723, + 0.0038668750057695433, + 0.00434474999201484, + 0.0041693749953992665, + 0.004704124992713332 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 20.774333497683983, + "p95": 21.992870650137775, + "min": 19.063458006712608, + "max": 22.081332994275726, + "iqr": 1.5701242409704719, + "mean": 20.699620801315177, + "cv": 0.04874910312110405 + }, + "prefill_ms": { + "count": 10, + "median": 16.642020505969413, + "p95": 17.857256397110177, + "min": 16.095833998406306, + "max": 17.902124993270263, + "iqr": 0.23009374126559123, + "mean": 16.807979201257695, + "cv": 0.03458971634427599 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.479499999433756, + "p95": 5.282027096836828, + "min": 3.733499994268641, + "max": 6.76891699549742, + "iqr": 0.41725049959495664, + "mean": 4.54160220929225, + "cv": 0.09105383145902382 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 219.06396351152313, + "p95": 229.99160165361795, + "min": 210.10794383559517, + "max": 231.15534742492585, + "iqr": 10.817119733713014, + "mean": 220.41236947934811, + "cv": 0.03374425678005709 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 196.78166831771256, + "p95": 205.42220442234952, + "min": 188.71300002419053, + "max": 206.3643527217448, + "iqr": 8.521124361404276, + "mean": 197.53379137650631, + "cv": 0.031121821959672745 + }, + "memory": { + "peak_extra_bytes": 1074003968, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "auto", + "batch": 1, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.020988582997233607 + }, + { + "forward": 0.02614683299907483 + }, + { + "forward": 0.02529237500857562 + }, + { + "forward": 0.025108791000093333 + }, + { + "forward": 0.02766829199390486 + }, + { + "forward": 0.025197791008395143 + }, + { + "forward": 0.02437058399664238 + }, + { + "forward": 0.021829499994055368 + }, + { + "forward": 0.02370533300563693 + }, + { + "forward": 0.023976541007868946 + } + ], + "forward_ms": { + "count": 10, + "median": 24.739687498367857, + "p95": 26.983635446231347, + "min": 20.988582997233607, + "max": 27.66829199390486, + "iqr": 1.4955940023355652, + "mean": 24.4284623011481, + "cv": 0.0802271132562034 + }, + "memory": { + "peak_extra_bytes": 1073741824, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "sdpa", + "batch": 1, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.013244625006336719, + "prefill": 0.013642249992699362, + "decode_steps": [ + 0.002287750001414679, + 0.0021890420030103996, + 0.0020895829948130995, + 0.002050291994237341, + 0.0020644580072257668, + 0.002050874987617135, + 0.0021120839955983683, + 0.0020392920123413205, + 0.0020149159972788766, + 0.00202379199618008, + 0.002119040989782661, + 0.002078040997730568, + 0.0020721249893540516, + 0.002023000008193776, + 0.002075582990073599, + 0.0022472499986179173, + 0.002140750002581626, + 0.0020390829886309803, + 0.0020547499880194664, + 0.0020369999983813614, + 0.002033541997661814, + 0.0019949580018874258, + 0.0020729999960167333, + 0.002035415993304923, + 0.0019762920128414407, + 0.0020191249932395294, + 0.002098082986776717, + 0.0020631250081351027, + 0.002075791999232024, + 0.0020684170012827963, + 0.002099791992804967, + 0.0025688329915283248 + ] + }, + { + "forward": 0.013914750001276843, + "prefill": 0.014094415993895382, + "decode_steps": [ + 0.0032783750066300854, + 0.0027651249984046444, + 0.00268079100351315, + 0.002580832995590754, + 0.002289333991939202, + 0.0022882500052219257, + 0.0033672910067252815, + 0.002142624987754971, + 0.00284770799044054, + 0.002264500013552606, + 0.0021733750036219135, + 0.002306790993316099, + 0.0033286659891018644, + 0.002110208006342873, + 0.00211333399056457, + 0.002657541001099162, + 0.002221250004367903, + 0.002550416989834048, + 0.0024007920001167804, + 0.0021324159897631034, + 0.002106041007209569, + 0.0025101249921135604, + 0.0026867090055020526, + 0.002175957997678779, + 0.0022224999993341044, + 0.0037271250039339066, + 0.0022676660009892657, + 0.002995125003508292, + 0.002580083004431799, + 0.002180334005970508, + 0.0022323750017676502, + 0.0021132499969098717 + ] + }, + { + "forward": 0.016737917001591995, + "prefill": 0.016287000005831942, + "decode_steps": [ + 0.0038246250041993335, + 0.003398458007723093, + 0.003403458002139814, + 0.0029597499960800633, + 0.003385083007742651, + 0.003425542003242299, + 0.0033099169959314167, + 0.003293582994956523, + 0.003549125001882203, + 0.003428167008678429, + 0.00373333299648948, + 0.003567084000678733, + 0.003423082991503179, + 0.003499291997286491, + 0.003562084006262012, + 0.0034374170063529164, + 0.003407208001590334, + 0.0034244999988004565, + 0.0032934580085566267, + 0.0025743339938344434, + 0.003323458004160784, + 0.002914166994742118, + 0.003456582999206148, + 0.003493166994303465, + 0.002882165994378738, + 0.0034431670064805076, + 0.0034773340012179688, + 0.003469542003585957, + 0.003440041997237131, + 0.002813040991895832, + 0.0035886659898096696, + 0.003566250001313165 + ] + }, + { + "forward": 0.017193166000652127, + "prefill": 0.017264624999370426, + "decode_steps": [ + 0.0038056250050431117, + 0.0037057919980725273, + 0.0035678749991348013, + 0.0034836670092772692, + 0.0036477499961620197, + 0.003678249995573424, + 0.0036652499984484166, + 0.003049375009140931, + 0.003828665998298675, + 0.0032073750044219196, + 0.003549958986695856, + 0.0036534169921651483, + 0.0032985840080073103, + 0.0035514160117600113, + 0.0034420409938320518, + 0.0036429169995244592, + 0.0041560420067980886, + 0.00352158400346525, + 0.0040250000020023435, + 0.0034595839970279485, + 0.0034987499966518953, + 0.0034960829943884164, + 0.003616165995481424, + 0.0036442499986151233, + 0.0037063329946249723, + 0.0030056670075282454, + 0.003537124997819774, + 0.0036393339978531003, + 0.003701458001160063, + 0.004836833002627827, + 0.003599083996959962, + 0.003457666010945104 + ] + }, + { + "forward": 0.016981166001642123, + "prefill": 0.01673833299719263, + "decode_steps": [ + 0.003772291005589068, + 0.0033786669955588877, + 0.003408417003811337, + 0.0035768749949056655, + 0.003393749997485429, + 0.0037292079941835254, + 0.0033031659986590967, + 0.0036914999946020544, + 0.0034447499929228798, + 0.003534083007252775, + 0.003437791994656436, + 0.005203334003454074, + 0.0034718749957391992, + 0.0033919169945875183, + 0.003492374991765246, + 0.0033570420055184513, + 0.003508791996864602, + 0.003499583006487228, + 0.0037447499926202, + 0.0034321669954806566, + 0.003968458986491896, + 0.003592916007619351, + 0.003639417001977563, + 0.0037489579990506172, + 0.003609749997849576, + 0.003750415999093093, + 0.0035498749930411577, + 0.002813915998558514, + 0.0035600830015027896, + 0.0036699159973068163, + 0.003463291999651119, + 0.0032917919888859615 + ] + }, + { + "forward": 0.01633854200190399, + "prefill": 0.016247833002125844, + "decode_steps": [ + 0.004286874987883493, + 0.003959208988817409, + 0.003274665999924764, + 0.003862416007905267, + 0.005016082999645732, + 0.003792165996856056, + 0.0035507920110831037, + 0.0038624580047326162, + 0.003329416998894885, + 0.00378758400620427, + 0.003923625001334585, + 0.003784917003940791, + 0.0037947920063743368, + 0.0037294590001692995, + 0.0037442920001922175, + 0.0035749999951804057, + 0.0038708750071236864, + 0.003711209006723948, + 0.004603708992362954, + 0.0038010419957572594, + 0.003979583008913323, + 0.003668167002615519, + 0.00405262500862591, + 0.0037920419999863952, + 0.0031881249888101593, + 0.0036030830087838694, + 0.003938833993743174, + 0.0038435000024037436, + 0.003996750005171634, + 0.003570875007426366, + 0.003894000008585863, + 0.0036407920124474913 + ] + }, + { + "forward": 0.017248958989512175, + "prefill": 0.016418875005911104, + "decode_steps": [ + 0.0038304170011542737, + 0.003971000012825243, + 0.004114042007131502, + 0.0036632910050684586, + 0.00369654200039804, + 0.0037816249969182536, + 0.003653708001365885, + 0.003513333009323105, + 0.0036518749984679744, + 0.0037500420003198087, + 0.003812041992205195, + 0.0029510000022128224, + 0.003705665993038565, + 0.003846708990749903, + 0.0038883340021129698, + 0.0035282500030007213, + 0.0037751249910797924, + 0.0037437080027302727, + 0.003814958006842062, + 0.003871249995427206, + 0.00368174999312032, + 0.0038713339890819043, + 0.0038897080085007474, + 0.0037144170055398718, + 0.003833709008176811, + 0.003707000010763295, + 0.004068541995366104, + 0.00395333299820777, + 0.003487583002424799, + 0.0035899999929824844, + 0.003950625003199093, + 0.003831207999610342 + ] + }, + { + "forward": 0.014695165999000892, + "prefill": 0.014716542005771771, + "decode_steps": [ + 0.004839875007746741, + 0.003481167004792951, + 0.0038208330079214647, + 0.0032890839938772842, + 0.0036765830009244382, + 0.003600582989747636, + 0.0041009999986272305, + 0.0036740000068675727, + 0.0030266249959822744, + 0.0034633330069482327, + 0.003925166995031759, + 0.0035363329952815548, + 0.003416583000216633, + 0.003611958003602922, + 0.0034728330065263435, + 0.003606332989875227, + 0.003742250002687797, + 0.003017750001163222, + 0.003492792005999945, + 0.0037564999947790056, + 0.003549917004420422, + 0.003457875005551614, + 0.0037678329972550273, + 0.003670166988740675, + 0.0036029159964527935, + 0.0036507499899016693, + 0.0034475419961381704, + 0.0031852909887675196, + 0.003198499995050952, + 0.00320866699621547, + 0.0031555419991491362, + 0.003098500004853122 + ] + }, + { + "forward": 0.014570208004442975, + "prefill": 0.014455000011366792, + "decode_steps": [ + 0.0033702080108923838, + 0.0030001249979250133, + 0.00293687499652151, + 0.0030807500006631017, + 0.0035794160066870973, + 0.0030874170042807236, + 0.0030232079880079255, + 0.003172833996359259, + 0.003077000001212582, + 0.004374875003122725, + 0.003309708001324907, + 0.0036235000006854534, + 0.0032308340014424175, + 0.003089417004957795, + 0.0030840419931337237, + 0.003017708004335873, + 0.0030262499931268394, + 0.003494874996249564, + 0.0037966670060995966, + 0.0030119580042082816, + 0.0032629159977659583, + 0.0033855829969979823, + 0.0037489579990506172, + 0.002638791993376799, + 0.00318137499561999, + 0.003053084001294337, + 0.0032654579990776256, + 0.0030946250044507906, + 0.0025549999991199, + 0.003023541998118162, + 0.00330137500714045, + 0.0032510839955648407 + ] + }, + { + "forward": 0.014179292003973387, + "prefill": 0.013763708004262298, + "decode_steps": [ + 0.004356124991318211, + 0.0032443329982925206, + 0.00373333299648948, + 0.0031220829987432808, + 0.003204750013537705, + 0.002555125000071712, + 0.0031504170037806034, + 0.003303249992313795, + 0.002937958008260466, + 0.0030947089981054887, + 0.002754000001004897, + 0.0031334999948740005, + 0.003284834005171433, + 0.003271041001426056, + 0.0030477919935947284, + 0.002596083009848371, + 0.003158291001454927, + 0.003069041995331645, + 0.0031537499889964238, + 0.0033189579989993945, + 0.003310583997517824, + 0.003156416001729667, + 0.003487042005872354, + 0.0032979580137180164, + 0.0031319999980041757, + 0.0025295000086771324, + 0.003029792002053, + 0.003544584003975615, + 0.003233582989196293, + 0.0030771669989917427, + 0.003215124990674667, + 0.0031612500024493784 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 15.51685400045244, + "p95": 17.223852144525154, + "min": 13.244625006336719, + "max": 17.248958989512175, + "iqr": 2.643332747538807, + "mean": 15.510379101033323, + "cv": 0.09897598625975247 + }, + "prefill_ms": { + "count": 10, + "median": 15.482187503948808, + "p95": 17.027793598390417, + "min": 13.642249992699362, + "max": 17.264624999370426, + "iqr": 2.2013442576280795, + "mean": 15.362858201842755, + "cv": 0.088480747086277 + }, + "decode_ms_per_step": { + "count": 320, + "median": 3.4347920009167865, + "p95": 3.971429162629647, + "min": 1.9762920128414407, + "max": 5.203334003454074, + "iqr": 0.6528022640850395, + "mean": 3.268809765313563, + "cv": 0.18900924947300995 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 289.38153192447385, + "p95": 442.35434362266636, + "min": 261.3760621827211, + "max": 478.2180501589823, + "iqr": 36.24638772107079, + "mean": 316.4491908769154, + "cv": 0.2184515931080732 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 253.7210326957687, + "p95": 371.03432670055173, + "min": 230.75237631488838, + "max": 397.23261982347026, + "iqr": 34.42873562788549, + "mean": 274.27591012796427, + "cv": 0.19498250451714028 + }, + "memory": { + "peak_extra_bytes": 33554432, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "reference", + "batch": 1, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.05704425000294577, + "prefill": 0.057017208004253916, + "decode_steps": [ + 0.006465832993853837, + 0.005996667008730583, + 0.00537583400728181, + 0.005019832999096252, + 0.005292582995025441, + 0.005088874997454695, + 0.004906916001345962, + 0.005392041988670826, + 0.004850458004511893, + 0.005008792009903118, + 0.005594166999799199, + 0.004926082998281345, + 0.004750583000713959, + 0.005447042000014335, + 0.005239082995103672, + 0.0055240419897018, + 0.004925041997921653, + 0.004640374987502582, + 0.004512957995757461, + 0.005042832999606617, + 0.005238084006123245, + 0.004704292005044408, + 0.004892875003861263, + 0.004848041004152037, + 0.004888540992396884, + 0.005007874991861172, + 0.005177707993425429, + 0.004988249987945892, + 0.004910542003926821, + 0.005123041002661921, + 0.004730542001198046, + 0.005113959006848745 + ] + }, + { + "forward": 0.06136704199889209, + "prefill": 0.05618929100455716, + "decode_steps": [ + 0.006979041005251929, + 0.006896792008774355, + 0.005828791996464133, + 0.006216875000973232, + 0.005546458007302135, + 0.006017458013957366, + 0.00626758299767971, + 0.005178833001991734, + 0.006010249999235384, + 0.005941082999925129, + 0.0057737500028451905, + 0.006226874989806674, + 0.006241874987608753, + 0.005270666995784268, + 0.005780542007414624, + 0.006133415998192504, + 0.006748957996023819, + 0.006409207999240607, + 0.005733666999731213, + 0.005843917009769939, + 0.006323291992885061, + 0.007112374994903803, + 0.006933249998837709, + 0.005758708997745998, + 0.007115833999705501, + 0.006583750000572763, + 0.006412000002455898, + 0.006346249996568076, + 0.006955208009458147, + 0.008233665997977369, + 0.007240167004056275, + 0.007866750005632639 + ] + }, + { + "forward": 0.06375545900664292, + "prefill": 0.05665458300791215, + "decode_steps": [ + 0.008668082999065518, + 0.007282833001227118, + 0.006598166000912897, + 0.006356749989208765, + 0.007775582998874597, + 0.006095375007134862, + 0.006854292005300522, + 0.0069231670058798045, + 0.006469375002779998, + 0.006784540993976407, + 0.006446999992476776, + 0.00656012499530334, + 0.0065956249891314656, + 0.0061268749996088445, + 0.007342709010117687, + 0.007100125003489666, + 0.007057750000967644, + 0.0061400419945130125, + 0.007203583998489194, + 0.006918041995959356, + 0.006530917002237402, + 0.00698016700334847, + 0.006546499993419275, + 0.006206792008015327, + 0.0060019590018782765, + 0.006496999994851649, + 0.008096958001260646, + 0.007727582997176796, + 0.008167749998392537, + 0.006851124999229796, + 0.006777750008041039, + 0.006372165997163393 + ] + }, + { + "forward": 0.061298459011595696, + "prefill": 0.05829762500070501, + "decode_steps": [ + 0.008923667002818547, + 0.00809737500094343, + 0.007230791001347825, + 0.00702462499612011, + 0.007035916991299018, + 0.006999249992077239, + 0.007020207995083183, + 0.006877583000459708, + 0.007438166998326778, + 0.00682016700739041, + 0.006663708001724444, + 0.006327458002488129, + 0.00656479099416174, + 0.00743454199982807, + 0.006695834003039636, + 0.006955083008506335, + 0.006865207993541844, + 0.006980207996093668, + 0.006807541009038687, + 0.006651124989730306, + 0.006943832995602861, + 0.007258541998453438, + 0.006869917007861659, + 0.006640541003434919, + 0.006859208006062545, + 0.006316458006040193, + 0.007518124999478459, + 0.006866833005915396, + 0.006807291996665299, + 0.00683920799929183, + 0.006779417002690025, + 0.006783499993616715 + ] + }, + { + "forward": 0.06298212498950306, + "prefill": 0.05600854101066943, + "decode_steps": [ + 0.010507583006983623, + 0.008499000003212132, + 0.008325874994625337, + 0.007456624996848404, + 0.007601999997859821, + 0.006852624996099621, + 0.007323958998313174, + 0.007972374995006248, + 0.007990583995706402, + 0.007061791999149136, + 0.007864541999879293, + 0.007443249996867962, + 0.00905833300203085, + 0.009047750005265698, + 0.00858470899402164, + 0.0073389159952057526, + 0.007394040992949158, + 0.007094875007169321, + 0.00876004200836178, + 0.009144417010247707, + 0.007484916000976227, + 0.007092040992574766, + 0.0072023749962681904, + 0.007490666990634054, + 0.007673166997847147, + 0.007705957992584445, + 0.006852124992292374, + 0.006971374998101965, + 0.00747416599188, + 0.008232042004237883, + 0.007926958001917228, + 0.008277665998321027 + ] + }, + { + "forward": 0.06810545800544787, + "prefill": 0.06325533299241215, + "decode_steps": [ + 0.013969749998068437, + 0.007892082998296246, + 0.0076429170003393665, + 0.007872375004808418, + 0.007731000005151145, + 0.0076507079938892275, + 0.009292457994888537, + 0.007529249996878207, + 0.00901200000953395, + 0.007165457995142788, + 0.007592000009026378, + 0.007205415997304954, + 0.009159875000477768, + 0.00805308400595095, + 0.007474124999134801, + 0.007515541001339443, + 0.007716541993431747, + 0.008048250005231239, + 0.009766250004759058, + 0.007360709001659416, + 0.0092597909970209, + 0.0072375830059172586, + 0.0071995829930529, + 0.007730083001661114, + 0.007907750012236647, + 0.0077998329943511635, + 0.007623832992976531, + 0.0075857080082641914, + 0.007440041998052038, + 0.00758704099280294, + 0.007204708992503583, + 0.007596875002491288 + ] + }, + { + "forward": 0.06109958300658036, + "prefill": 0.056285667000338435, + "decode_steps": [ + 0.011317959011648782, + 0.009333916998002678, + 0.009452832993702032, + 0.008200500000384636, + 0.009053792004124261, + 0.008956167002907023, + 0.008883957998477854, + 0.007767999995849095, + 0.008262374991318211, + 0.007989042002009228, + 0.00889104200177826, + 0.007445625000400469, + 0.008035624996409751, + 0.007942083000671118, + 0.008721499994862825, + 0.00782370900560636, + 0.00769587499962654, + 0.007070499996189028, + 0.008174959002644755, + 0.00774033299239818, + 0.007659959010197781, + 0.007617333001689985, + 0.008781249998719431, + 0.007424542010994628, + 0.00794224999845028, + 0.0077809999929741025, + 0.008046040995395742, + 0.007744707996607758, + 0.007819542006473057, + 0.007204916997579858, + 0.007121208996977657, + 0.006722457997966558 + ] + }, + { + "forward": 0.061317333005717956, + "prefill": 0.055807165990700014, + "decode_steps": [ + 0.009652957989601418, + 0.006995458999881521, + 0.00831979198846966, + 0.007745666007394902, + 0.007951332998345606, + 0.007351124993874691, + 0.007464958005584776, + 0.006818833004217595, + 0.007086832993081771, + 0.007034374997601844, + 0.006739208009094, + 0.00716574999387376, + 0.00684387500223238, + 0.007307915991987102, + 0.006967666995478794, + 0.008497290997183882, + 0.008124542000587098, + 0.007022290999884717, + 0.008236207999289036, + 0.007184959002188407, + 0.0069126669986872, + 0.006545791999087669, + 0.007003000006079674, + 0.007153250000556, + 0.00700762499764096, + 0.006310875003691763, + 0.0071737909893272445, + 0.006948792011826299, + 0.007245542001328431, + 0.00722354200843256, + 0.006864374998258427, + 0.006659332997514866 + ] + }, + { + "forward": 0.0592097499902593, + "prefill": 0.05546433299605269, + "decode_steps": [ + 0.008328916999744251, + 0.007107833007466979, + 0.006858124994323589, + 0.007510291994549334, + 0.007038041003397666, + 0.007297833988559432, + 0.007047207996947691, + 0.006921499996678904, + 0.007350124986260198, + 0.007000999990850687, + 0.006654083001194522, + 0.005758333005360328, + 0.005826916996738873, + 0.006114832998719066, + 0.00628929199592676, + 0.006114000003435649, + 0.006371749987010844, + 0.005873875008546747, + 0.005899499999941327, + 0.007009042004938237, + 0.007765083995764144, + 0.006552958002430387, + 0.006895582992001437, + 0.007035083996015601, + 0.006134249997558072, + 0.006001708010444418, + 0.0053452499996637926, + 0.00519995900685899, + 0.005800832994282246, + 0.005693041006452404, + 0.005637292008032091, + 0.0057815829932224005 + ] + }, + { + "forward": 0.061988792003830895, + "prefill": 0.057887167000444606, + "decode_steps": [ + 0.009670250001363456, + 0.007197041995823383, + 0.00847866699041333, + 0.007228125003166497, + 0.008291416001156904, + 0.006765167010598816, + 0.00698066598852165, + 0.006205790996318683, + 0.007717083004536107, + 0.006015540988300927, + 0.00616220899973996, + 0.006696540993289091, + 0.006207541999174282, + 0.006414999996195547, + 0.006707082997309044, + 0.006205875004525296, + 0.00658554099209141, + 0.005881916004000232, + 0.006398541998350993, + 0.005527624991373159, + 0.005723958995076828, + 0.006722457997966558, + 0.005560082994634286, + 0.006127875007223338, + 0.006682874998659827, + 0.005777083992143162, + 0.006010375000187196, + 0.005567583997617476, + 0.0068174999905750155, + 0.0071121249930001795, + 0.006191583001054823, + 0.0065162910032086074 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 61.34218750230502, + "p95": 66.14795845598564, + "min": 57.04425000294577, + "max": 68.10545800544787, + "iqr": 1.584489735250827, + "mean": 61.81682510214159, + "cv": 0.04685448187327463 + }, + "prefill_ms": { + "count": 10, + "median": 56.47012500412529, + "p95": 61.02436439614393, + "min": 55.46433299605269, + "max": 63.25533299241215, + "iqr": 1.6159487422555685, + "mean": 57.28669140080456, + "cv": 0.03981845147387819 + }, + "decode_ms_per_step": { + "count": 320, + "median": 6.980187499721069, + "p95": 8.95895865323837, + "min": 4.512957995757461, + "max": 13.969749998068437, + "iqr": 1.367447999655269, + "mean": 6.986188902510548, + "cv": 0.16733920296014201 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 144.1702419443296, + "p95": 178.17722308390498, + "min": 122.78177317030023, + "max": 195.5705818227944, + "iqr": 23.37348112877902, + "mean": 145.63956175794482, + "cv": 0.14694932833735153 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 114.52063289948214, + "p95": 135.12737910551496, + "min": 99.66427000599965, + "max": 145.03197569266894, + "iqr": 15.004692570174342, + "mean": 115.2175436801756, + "cv": 0.11550725893796289 + }, + "memory": { + "peak_extra_bytes": 1074003968, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "auto", + "batch": 4, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.0038232080114539713 + }, + { + "forward": 0.004477458001929335 + }, + { + "forward": 0.0037661660026060417 + }, + { + "forward": 0.0038030840078135952 + }, + { + "forward": 0.0038093750044936314 + }, + { + "forward": 0.003878417002852075 + }, + { + "forward": 0.003954208004870452 + }, + { + "forward": 0.003727792005520314 + }, + { + "forward": 0.0037718749954365194 + }, + { + "forward": 0.003724791997228749 + } + ], + "forward_ms": { + "count": 10, + "median": 3.8062295061536133, + "p95": 4.241995503252837, + "min": 3.724791997228749, + "max": 4.477458001929335, + "iqr": 0.09702150418888777, + "mean": 3.8736375034204684, + "cv": 0.05760301796148238 + }, + "memory": { + "peak_extra_bytes": 1441792, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "sdpa", + "batch": 4, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.0032590420014457777, + "prefill": 0.0032100410026032478, + "decode_steps": [ + 0.0018754159973468632, + 0.0018461250001564622, + 0.001860165997641161, + 0.0018285830010427162, + 0.001825707993702963, + 0.0018497499986551702, + 0.0018322919931961223, + 0.001915541011840105, + 0.0017645839980104938, + 0.0020043340045958757, + 0.001828749998821877, + 0.002090457986923866, + 0.006299832995864563, + 0.004540541005553678, + 0.005906915990635753, + 0.002531666003051214, + 0.002949333007563837, + 0.00188079199870117, + 0.0018440830026520416, + 0.0023927500005811453, + 0.0026507920119911432, + 0.0024724590039113536, + 0.00228529199375771, + 0.002398374999756925, + 0.002516124994144775, + 0.0022705409937771037, + 0.0018522919999668375, + 0.0019619999948190525, + 0.001941875001648441, + 0.0017936250078491867, + 0.0020292919944040477, + 0.001868374994955957 + ] + }, + { + "forward": 0.003279083000961691, + "prefill": 0.003205417000572197, + "decode_steps": [ + 0.002846749994205311, + 0.0019112919981125742, + 0.0020351249986561015, + 0.0019569169962778687, + 0.0019035000004805624, + 0.0019638749945443124, + 0.0018600409966893494, + 0.0018343749979976565, + 0.0028762909932993352, + 0.0018187500099884346, + 0.0018098339933203533, + 0.0017762920033419505, + 0.002030541989370249, + 0.001966458003153093, + 0.0019249169999966398, + 0.002192707994254306, + 0.0019108339911326766, + 0.0019621670071501285, + 0.0018792500050039962, + 0.0018707080016611144, + 0.0019218749948777258, + 0.001857499999459833, + 0.0018411669880151749, + 0.001895249995868653, + 0.001919624992297031, + 0.0018366250005783513, + 0.001924374999362044, + 0.001780957987648435, + 0.0018931660015368834, + 0.0020167089969618246, + 0.0018599999893922359, + 0.00219574999937322 + ] + }, + { + "forward": 0.003389542005606927, + "prefill": 0.0033620419999351725, + "decode_steps": [ + 0.002239207999082282, + 0.002031583004281856, + 0.0024445829913020134, + 0.0020462079992285, + 0.0018434999947203323, + 0.0019476669986033812, + 0.0018919169961009175, + 0.0018525420018704608, + 0.0018725410045590252, + 0.0017521249974379316, + 0.0020444159890757874, + 0.0019730839994736016, + 0.0019055409939028323, + 0.002282208006363362, + 0.0019141249940730631, + 0.0018077919958159328, + 0.002287333001731895, + 0.0019045000080950558, + 0.0019236660009482875, + 0.0018490410002414137, + 0.0021089999936521053, + 0.0019302080036140978, + 0.0019234579958720133, + 0.002285624999785796, + 0.0019568330026231706, + 0.0018111249955836684, + 0.0018198750040028244, + 0.0028337080002529547, + 0.0019976670009782538, + 0.0019446670048637316, + 0.0019477910100249574, + 0.0020883340039290488 + ] + }, + { + "forward": 0.0033246670063817874, + "prefill": 0.003319249997730367, + "decode_steps": [ + 0.0019375840056454763, + 0.002406375002465211, + 0.0018610420083859935, + 0.002009917006944306, + 0.0020745419897139072, + 0.0019522079965099692, + 0.0019442079938016832, + 0.001867583006969653, + 0.0018220000056317076, + 0.001936791988555342, + 0.0019306659960420802, + 0.0018922920135082677, + 0.0021157089940970764, + 0.0019645419961307198, + 0.0019383749895496294, + 0.0017774589941836894, + 0.00184229199658148, + 0.0018162500055041164, + 0.0020842500089202076, + 0.0020582079887390137, + 0.001980665998416953, + 0.002231375008705072, + 0.0019317080004839227, + 0.0018219580088043585, + 0.001988916003028862, + 0.002074457996059209, + 0.0019372079987078905, + 0.0019230840116506442, + 0.002000500011490658, + 0.0018562920013209805, + 0.001909500002511777, + 0.002080707999994047 + ] + }, + { + "forward": 0.003424374997848645, + "prefill": 0.0035400410124566406, + "decode_steps": [ + 0.00208583299536258, + 0.0019848340016324073, + 0.0019702089921338484, + 0.0019492919964250177, + 0.001894457993330434, + 0.0019462079944787547, + 0.0018240839999634773, + 0.0018054169922834262, + 0.0018710410076892003, + 0.0019009170064236969, + 0.001875957997981459, + 0.0018746249988907948, + 0.0018485000036889687, + 0.0018250419961987063, + 0.0020792499999515712, + 0.0018705840047914535, + 0.0018397910025669262, + 0.002149958992959, + 0.00202433300728444, + 0.0018580410105641931, + 0.0019481249910313636, + 0.0017827500123530626, + 0.0018980839959112927, + 0.0020166250033071265, + 0.0019049580005230382, + 0.001815958006773144, + 0.001828999986173585, + 0.001883792006992735, + 0.0018294170004082844, + 0.0021479169954545796, + 0.0018019159906543791, + 0.001913833009894006 + ] + }, + { + "forward": 0.0032549159950576723, + "prefill": 0.0035073750041192397, + "decode_steps": [ + 0.0019490839913487434, + 0.0019275420054327697, + 0.00194191699847579, + 0.0019298749975860119, + 0.0018129169911844656, + 0.0019191250030416995, + 0.0019367910135770217, + 0.001817917000153102, + 0.0018770839960779995, + 0.0018586249934742227, + 0.001930250000441447, + 0.0017631660011829808, + 0.0018009170016739517, + 0.0018189999973401427, + 0.0018257920019095764, + 0.001713707999442704, + 0.0017892920004669577, + 0.0019462499913061038, + 0.001847249994170852, + 0.0018454579985700548, + 0.0018682079971767962, + 0.0018715409969445318, + 0.0018122080073226243, + 0.002491916995495558, + 0.0018687910051085055, + 0.0018567079969216138, + 0.0018795420037349686, + 0.0018254999886266887, + 0.0018191249982919544, + 0.0018752079922705889, + 0.00208416699024383, + 0.0018161250045523047 + ] + }, + { + "forward": 0.0032951670000329614, + "prefill": 0.0033256669994443655, + "decode_steps": [ + 0.0018141670007025823, + 0.0018484589963918552, + 0.0021049579954706132, + 0.0018191670096712187, + 0.0019057919998886064, + 0.001993124998989515, + 0.0018176250014221296, + 0.001922666997415945, + 0.0017856250051409006, + 0.0017590409988770261, + 0.0019334579992573708, + 0.0019301249994896352, + 0.0018065420008497313, + 0.0018463340093148872, + 0.0017673749971436337, + 0.002187249992857687, + 0.0018761660030577332, + 0.001854874994023703, + 0.0021068330097477883, + 0.0019692079949891195, + 0.001962415990419686, + 0.001948707998963073, + 0.0020539999968605116, + 0.00208470800134819, + 0.001839958000346087, + 0.002850000004400499, + 0.0020537499949568883, + 0.0019252499914728105, + 0.001853541994933039, + 0.0019659579993458465, + 0.0018562909972388297, + 0.0018446249887347221 + ] + }, + { + "forward": 0.003419625005335547, + "prefill": 0.003199708997271955, + "decode_steps": [ + 0.002564875001553446, + 0.0019830829987768084, + 0.0019389579974813387, + 0.001997125000343658, + 0.0018735829944489524, + 0.00199295899074059, + 0.001833917005569674, + 0.001872584005468525, + 0.00194454200391192, + 0.0017908750014612451, + 0.001765332999639213, + 0.00205650000134483, + 0.002098625001963228, + 0.0019807909993687645, + 0.0018957919965032488, + 0.0030006659944774583, + 0.001960416993824765, + 0.001832542009651661, + 0.002032500007771887, + 0.0020784169901162386, + 0.0019663749990286306, + 0.001853875000961125, + 0.001965709001524374, + 0.0019325839966768399, + 0.00187329099571798, + 0.0018050829967251047, + 0.002033665994531475, + 0.0019747500045923516, + 0.0018583750061225146, + 0.0020683339971583337, + 0.0020233329996699467, + 0.0019067079992964864 + ] + }, + { + "forward": 0.0035006669932045043, + "prefill": 0.0033431659976486117, + "decode_steps": [ + 0.0019408329972065985, + 0.0019456250010989606, + 0.0020469589944696054, + 0.00203424999199342, + 0.0021070829970994964, + 0.0018526249914430082, + 0.0022042079945094883, + 0.0019784169999184087, + 0.0019916669989470392, + 0.0019775420078076422, + 0.0018260000069858506, + 0.002038249993347563, + 0.0019485419907141477, + 0.0018988340016221628, + 0.0018238330085296184, + 0.0019631250033853576, + 0.0018522919999668375, + 0.0019215409993194044, + 0.0019081250065937638, + 0.002095042000291869, + 0.001998457999434322, + 0.0019428750092629343, + 0.0017974170041270554, + 0.0017594999953871593, + 0.0021877090039197356, + 0.0019042500061914325, + 0.0018289589934283867, + 0.002194957996835001, + 0.0020023329998366535, + 0.0020808330009458587, + 0.001930291997268796, + 0.002653957999427803 + ] + }, + { + "forward": 0.003174541998305358, + "prefill": 0.003450625008554198, + "decode_steps": [ + 0.0020360420021461323, + 0.0018477089906809852, + 0.0018995409918716177, + 0.0019292499928269535, + 0.001947958007804118, + 0.002130958004272543, + 0.0019545830000424758, + 0.0017682500038063154, + 0.001969832999748178, + 0.0021187499951338395, + 0.0018750420131254941, + 0.0019189580052625388, + 0.0017662089958321303, + 0.0018683340022107586, + 0.0021785419958177954, + 0.001955250001628883, + 0.0018694169993977994, + 0.00298920800560154, + 0.0019952910079155117, + 0.00180283299414441, + 0.00213649999932386, + 0.001982415997190401, + 0.001994082995224744, + 0.001933709005243145, + 0.0019737499969778582, + 0.0019231670012231916, + 0.0019747499900404364, + 0.0018959590088343248, + 0.0020134170044912025, + 0.001890874991659075, + 0.0018510830122977495, + 0.001912332998472266 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 3.3099170032073744, + "p95": 3.4663355952943675, + "min": 3.174541998305358, + "max": 3.5006669932045043, + "iqr": 0.14805200407863595, + "mean": 3.332162600418087, + "cv": 0.02971551057822734 + }, + "prefill_ms": { + "count": 10, + "median": 3.3344164985464886, + "p95": 3.52534130870481, + "min": 3.199708997271955, + "max": 3.5400410124566406, + "iqr": 0.19113600501441397, + "mean": 3.3463333020335995, + "cv": 0.03669206722833829 + }, + "decode_ms_per_step": { + "count": 320, + "median": 1.92639599845279, + "p95": 2.473431903490564, + "min": 1.713707999442704, + "max": 6.299832995864563, + "iqr": 0.16161476014531218, + "mean": 2.00677837128751, + "cv": 0.20484710256892263 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 2022.4744529052123, + "p95": 2107.4860388572943, + "min": 1664.3118392670867, + "max": 2121.980738027773, + "iqr": 40.823088820219255, + "mean": 2001.348767417078, + "cv": 0.06256467152837561 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1921.4444759158182, + "p95": 1991.9729537687901, + "min": 1597.6294178963117, + "max": 2005.3777036450647, + "iqr": 34.65141011333344, + "mean": 1901.3248079972277, + "cv": 0.05897061298583757 + }, + "memory": { + "peak_extra_bytes": 262144, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "reference", + "batch": 4, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.008280540991108865, + "prefill": 0.004385084001114592, + "decode_steps": [ + 0.004859707987634465, + 0.004588041003444232, + 0.003957499997341074, + 0.004526167002040893, + 0.004608208007994108, + 0.0042720000055851415, + 0.004502540992689319, + 0.004602165994583629, + 0.00437858299119398, + 0.0038364579959306866, + 0.004414417009684257, + 0.004030791998957284, + 0.004639875012799166, + 0.006651499992585741, + 0.005170332995476201, + 0.005929249993641861, + 0.004596457991283387, + 0.004214250002405606, + 0.004569416996673681, + 0.00406158299301751, + 0.00457562500378117, + 0.003970624995417893, + 0.004411582995089702, + 0.0044572909973794594, + 0.004482999996980652, + 0.004344499990111217, + 0.004515332999289967, + 0.004346458008512855, + 0.004279583998140879, + 0.0041006249957717955, + 0.0044887919939355925, + 0.004446541002835147 + ] + }, + { + "forward": 0.007827875000657514, + "prefill": 0.00439520800136961, + "decode_steps": [ + 0.004904458008240908, + 0.004401874990435317, + 0.004507874997216277, + 0.004290374999982305, + 0.004561624999041669, + 0.004429250009707175, + 0.004452499997569248, + 0.004314707999583334, + 0.004453959001693875, + 0.004350458999397233, + 0.004409083005157299, + 0.004022749999421649, + 0.004122040991205722, + 0.004060916995513253, + 0.004868208008701913, + 0.005656499997712672, + 0.00563729100394994, + 0.006243791998713277, + 0.005172707999008708, + 0.004652540999813937, + 0.004356666991952807, + 0.004489375001867302, + 0.004930833005346358, + 0.005320625001331791, + 0.005300125005305745, + 0.005139708009664901, + 0.00520929200865794, + 0.004836583000724204, + 0.005009791988413781, + 0.005489374991157092, + 0.0048262080090353265, + 0.005209708004258573 + ] + }, + { + "forward": 0.008744709004531614, + "prefill": 0.005146834009792656, + "decode_steps": [ + 0.005065833989647217, + 0.004241916001774371, + 0.004691249996540137, + 0.0047990000020945445, + 0.004847666990826838, + 0.00479579099919647, + 0.004497834001085721, + 0.004300375003367662, + 0.0045971669896971434, + 0.0044124169944552705, + 0.004529208003077656, + 0.004975000003469177, + 0.004839458997594193, + 0.004699541997979395, + 0.005304541991790757, + 0.005575832998147234, + 0.00487170901033096, + 0.006118250006693415, + 0.005854207993252203, + 0.005101792005007155, + 0.0042687920067692176, + 0.004849125005421229, + 0.004272250007488765, + 0.00455337499442976, + 0.00444570800755173, + 0.0040792500076349825, + 0.004699500001152046, + 0.00540012501005549, + 0.004328041992266662, + 0.004130749992327765, + 0.004454958005226217, + 0.00447300000814721 + ] + }, + { + "forward": 0.008296624990180135, + "prefill": 0.00474887499876786, + "decode_steps": [ + 0.004694707997259684, + 0.004871165991062298, + 0.005171332988538779, + 0.004701416997704655, + 0.00462308299029246, + 0.0040713339985813946, + 0.004394040995975956, + 0.003920500006643124, + 0.004251708000083454, + 0.004422250000061467, + 0.0044552080071298406, + 0.004107417000341229, + 0.004262583999661729, + 0.004682957995100878, + 0.004596041995682754, + 0.004021707994979806, + 0.004220333998091519, + 0.0043440419976832345, + 0.005026041006203741, + 0.004258750006556511, + 0.0045699170004809275, + 0.004770915998960845, + 0.004436082992469892, + 0.004145250000874512, + 0.004458416995476, + 0.004851542005781084, + 0.004320249994634651, + 0.004514375003054738, + 0.004256000000168569, + 0.003981792004196905, + 0.004338333004852757, + 0.004420124998432584 + ] + }, + { + "forward": 0.007803125001373701, + "prefill": 0.004337582999141887, + "decode_steps": [ + 0.004635000004782341, + 0.004595749996951781, + 0.0038830830017104745, + 0.0044394159922376275, + 0.004422582991537638, + 0.004429000007803552, + 0.004373666000901721, + 0.004353417010861449, + 0.004380500002298504, + 0.00429187499685213, + 0.005334916990250349, + 0.004095166994375177, + 0.004769249993842095, + 0.0044451669964473695, + 0.003917124995496124, + 0.0038350830000126734, + 0.004327792004914954, + 0.004411749992868863, + 0.004417958989506587, + 0.004344083994510584, + 0.0043300830002408475, + 0.004440749995410442, + 0.004056000005220994, + 0.004378958998131566, + 0.004289832999347709, + 0.0043389579950599, + 0.004362250008853152, + 0.004409750006743707, + 0.004059457991388626, + 0.004305166992708109, + 0.004114084003958851, + 0.004406041000038385 + ] + }, + { + "forward": 0.0068109999992884696, + "prefill": 0.004487041995162144, + "decode_steps": [ + 0.004348665999714285, + 0.004144749997067265, + 0.004342208005255088, + 0.004276040999684483, + 0.004129707987885922, + 0.0043692089966498315, + 0.004376833996502683, + 0.004577708008582704, + 0.00492308298998978, + 0.003994708007667214, + 0.00444587500533089, + 0.00465466600144282, + 0.004301707987906411, + 0.004329999996116385, + 0.004345958004705608, + 0.00433037499897182, + 0.0042097499972442165, + 0.0043259170051896945, + 0.004324166991864331, + 0.004393709008581936, + 0.004290832992410287, + 0.004300790998968296, + 0.0043179589993087575, + 0.004250582991517149, + 0.004309583993745036, + 0.004046750007546507, + 0.0040078330057440326, + 0.004226582997944206, + 0.004447458006325178, + 0.0045027089945506305, + 0.004165958001976833, + 0.004204458004096523 + ] + }, + { + "forward": 0.007948707992909476, + "prefill": 0.0047734580002725124, + "decode_steps": [ + 0.005227833011304028, + 0.004742667006212287, + 0.004384041007142514, + 0.005560999998124316, + 0.004476916990824975, + 0.004444499994860962, + 0.005413666993263178, + 0.005880958997295238, + 0.0048102080036187544, + 0.005197666003368795, + 0.004509375008638017, + 0.006487332997494377, + 0.006090333001338877, + 0.005842625003424473, + 0.004724375001387671, + 0.005628666986012831, + 0.004242249997332692, + 0.004563916998449713, + 0.004983874998288229, + 0.0049155409942613915, + 0.004780540999490768, + 0.004479791998164728, + 0.004882082997937687, + 0.004203874996164814, + 0.004413333008415066, + 0.005290291999699548, + 0.005199999999604188, + 0.004900667001493275, + 0.004687874999945052, + 0.0047848750109551474, + 0.004599791995133273, + 0.004347958005382679 + ] + }, + { + "forward": 0.008110958006000146, + "prefill": 0.004453750007087365, + "decode_steps": [ + 0.00472254199848976, + 0.005506916990270838, + 0.004697833996033296, + 0.005206874993746169, + 0.005070125000202097, + 0.004732207991764881, + 0.004108082997845486, + 0.004287709001800977, + 0.004624457986210473, + 0.004656791003071703, + 0.00427283400495071, + 0.004578624997520819, + 0.004410125009599142, + 0.0045287919929251075, + 0.0043442499882075936, + 0.0044068749994039536, + 0.004350667004473507, + 0.0043512499978533015, + 0.004319416999351233, + 0.004640707993530668, + 0.004382457991596311, + 0.00442720900173299, + 0.004507958001340739, + 0.00453633299912326, + 0.006775374989956617, + 0.004808334007975645, + 0.00451466599770356, + 0.004691124995588325, + 0.004392667004140094, + 0.0045923330035293475, + 0.004521333001321182, + 0.00440550000348594 + ] + }, + { + "forward": 0.006876750005176291, + "prefill": 0.004410834008012898, + "decode_steps": [ + 0.004477499998756684, + 0.004391374997794628, + 0.004627916001481935, + 0.004371332994196564, + 0.0043912499968428165, + 0.0043407500052126125, + 0.004346041998360306, + 0.004232749997754581, + 0.004349957991507836, + 0.0047485420072916895, + 0.004186374993878417, + 0.004325916001107544, + 0.004503916003159247, + 0.004476583999348804, + 0.004379582998808473, + 0.004360666003776714, + 0.004380708996905014, + 0.0043466249917401, + 0.004287582996767014, + 0.004344874992966652, + 0.003878832998452708, + 0.003967040989664383, + 0.004344999993918464, + 0.003942499999538995, + 0.005956708002486266, + 0.004483500000787899, + 0.004977208998752758, + 0.004805042000953108, + 0.0046094999997876585, + 0.004527458004304208, + 0.00470420801138971, + 0.004089708003448322 + ] + }, + { + "forward": 0.008368249997147359, + "prefill": 0.00475374999223277, + "decode_steps": [ + 0.005288582993671298, + 0.004762915996252559, + 0.005304708000039682, + 0.004709457993158139, + 0.0048827909922692925, + 0.004602583998348564, + 0.004616500009433366, + 0.0042969169880962, + 0.0044455419993028045, + 0.004738999996334314, + 0.004765416000736877, + 0.004575833008857444, + 0.0043693749903468415, + 0.004736083006719127, + 0.003822042010142468, + 0.004629083996405825, + 0.005512834002729505, + 0.004981749996659346, + 0.004777291003847495, + 0.004819624999072403, + 0.004603209003107622, + 0.004397625001729466, + 0.004268749995389953, + 0.00460962500073947, + 0.00398716599738691, + 0.0044430420093704015, + 0.004579125001328066, + 0.004682042010244913, + 0.004677584001910873, + 0.004527125012828037, + 0.004509958002017811, + 0.004163208999671042 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 8.029832999454811, + "p95": 8.5753024512087, + "min": 6.8109999992884696, + "max": 8.744709004531614, + "iqr": 0.4832914892176632, + "mean": 7.906854098837357, + "cv": 0.07919408711732173 + }, + "prefill_ms": { + "count": 10, + "median": 4.4703960011247545, + "p95": 4.978814805508591, + "min": 4.337582999141887, + "max": 5.146834009792656, + "iqr": 0.35341674083610997, + "mean": 4.5892418012954295, + "cv": 0.056293845620451616 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.45624950225465, + "p95": 5.515242302499247, + "min": 3.822042010142468, + "max": 6.775374989956617, + "iqr": 0.40883275869418867, + "mean": 4.58568473068226, + "cv": 0.10132005064901939 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 874.0788144535171, + "p95": 923.1568884684085, + "min": 806.559181203951, + "max": 926.0830935585128, + "iqr": 53.462421062797034, + "mean": 873.8745498478544, + "cv": 0.044745410174621005 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 847.689913817904, + "p95": 894.6359339695829, + "min": 783.0073218883057, + "max": 896.9641765460814, + "iqr": 52.83376823215485, + "mean": 847.3356780961971, + "cv": 0.04420261149557027 + }, + "memory": { + "peak_extra_bytes": 262144, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "auto", + "batch": 4, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.006441958001232706 + }, + { + "forward": 0.006396041993866675 + }, + { + "forward": 0.0065243330027442425 + }, + { + "forward": 0.006357875012326986 + }, + { + "forward": 0.006620957996346988 + }, + { + "forward": 0.006583958005649038 + }, + { + "forward": 0.00654124999709893 + }, + { + "forward": 0.006554958003107458 + }, + { + "forward": 0.007979874993907288 + }, + { + "forward": 0.006357334001222625 + } + ], + "forward_ms": { + "count": 10, + "median": 6.532791499921586, + "p95": 7.368362345005151, + "min": 6.357334001222625, + "max": 7.979874993907288, + "iqr": 0.16918700930546038, + "mean": 6.635854100750294, + "cv": 0.07257096151782182 + }, + "memory": { + "peak_extra_bytes": 36175872, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "sdpa", + "batch": 4, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.005532417009817436, + "prefill": 0.005524125008378178, + "decode_steps": [ + 0.0021419170079752803, + 0.001998124993406236, + 0.00195220900059212, + 0.0019557919877115637, + 0.0020244170009391382, + 0.001986624993151054, + 0.002035042009083554, + 0.0018969580123666674, + 0.001962416004971601, + 0.001973082995391451, + 0.001945541996974498, + 0.0030935000104364008, + 0.0020631670049624518, + 0.002052708005066961, + 0.001963083996088244, + 0.0019360419973963872, + 0.002047750007477589, + 0.001999790998524986, + 0.002843333000782877, + 0.001987750001717359, + 0.0020130000048084185, + 0.0019270829943707213, + 0.002027541006100364, + 0.0020293749985285103, + 0.001966583004104905, + 0.0022042920027161017, + 0.002008791998378001, + 0.0019406249921303242, + 0.002022499989834614, + 0.0019674579962156713, + 0.0019340839935466647, + 0.0019351249939063564 + ] + }, + { + "forward": 0.005558874996495433, + "prefill": 0.005616458001895808, + "decode_steps": [ + 0.002194291999330744, + 0.0020646670018322766, + 0.002041917003225535, + 0.00196620800124947, + 0.001996209000935778, + 0.0020129160111537203, + 0.001995041995542124, + 0.001976209008716978, + 0.002050833005341701, + 0.0022050839907024056, + 0.0020090420002816245, + 0.00217145899659954, + 0.002054083000984974, + 0.0020609999919543043, + 0.0019369999936316162, + 0.0023525420110672712, + 0.0021062920131953433, + 0.002064542000880465, + 0.0019691659981617704, + 0.0020547500025713816, + 0.0021057919948361814, + 0.0021253330050967634, + 0.0019810420053545386, + 0.0020462910033529624, + 0.0023134170041885227, + 0.0020465419947868213, + 0.0021750829910160974, + 0.0020183330052532256, + 0.002090417008730583, + 0.0020617089903680608, + 0.0020347499958006665, + 0.0020017909992020577 + ] + }, + { + "forward": 0.005560207995586097, + "prefill": 0.005512374991667457, + "decode_steps": [ + 0.0023147089959820732, + 0.0020915830100420862, + 0.0019444170029601082, + 0.0024049999919952825, + 0.0022974159946898, + 0.0021645829983754084, + 0.001989708995097317, + 0.0019329579954501241, + 0.001965500006917864, + 0.002051957999356091, + 0.002311542004463263, + 0.002089874993544072, + 0.0020305829966673627, + 0.0022857080039102584, + 0.001972291007405147, + 0.002092957991408184, + 0.0019520000059856102, + 0.0021262090012896806, + 0.002027209004154429, + 0.0021025000023655593, + 0.0020893330074613914, + 0.0020935829961672425, + 0.0019947499968111515, + 0.0020255420095054433, + 0.002031208001426421, + 0.002099000004818663, + 0.0020890000014333054, + 0.002047750007477589, + 0.002370375004829839, + 0.00198341699433513, + 0.0020292080007493496, + 0.0020023329998366535 + ] + }, + { + "forward": 0.005564749997574836, + "prefill": 0.00555545900715515, + "decode_steps": [ + 0.002131000001099892, + 0.0020792499999515712, + 0.0019383750041015446, + 0.0021805830037919804, + 0.002038708989857696, + 0.002025124995270744, + 0.0020629159989766777, + 0.002049707996775396, + 0.00205441600701306, + 0.002027750000706874, + 0.00203216599766165, + 0.0020204589964123443, + 0.0018992919940501451, + 0.001954667008249089, + 0.0020488750014919788, + 0.0020090420002816245, + 0.0019441659969743341, + 0.0022427499934565276, + 0.0032039579964475706, + 0.0022322079894365743, + 0.0019264580041635782, + 0.0019569169962778687, + 0.001920041991979815, + 0.0021532499958993867, + 0.00212079199263826, + 0.0019983329984825104, + 0.0019889999966835603, + 0.001977083011297509, + 0.0023954170028446242, + 0.0020057920046383515, + 0.0021105839987285435, + 0.002092791997711174 + ] + }, + { + "forward": 0.005551207999815233, + "prefill": 0.005553542010602541, + "decode_steps": [ + 0.0021423750004032627, + 0.001938625006005168, + 0.0020661250018747523, + 0.002007624992984347, + 0.0019374160037841648, + 0.0020149589981883764, + 0.001975333987502381, + 0.002967083011753857, + 0.0020055829954799265, + 0.0020059160015080124, + 0.0020576249953592196, + 0.002066415996523574, + 0.0020756660087499768, + 0.002031959011219442, + 0.0020088339952053502, + 0.0020240419980837032, + 0.0019875419966410846, + 0.0019721249991562217, + 0.002053832999081351, + 0.0019981660007033497, + 0.0019795420084847137, + 0.0020689579978352413, + 0.0019764589960686862, + 0.0021794579952256754, + 0.0020375830063130707, + 0.002160125004593283, + 0.002145124992239289, + 0.002091207992634736, + 0.0019731249922188, + 0.0020086659933440387, + 0.002069041001959704, + 0.0020889580046059564 + ] + }, + { + "forward": 0.0067062080051982775, + "prefill": 0.00549374999536667, + "decode_steps": [ + 0.0021612499986076728, + 0.0021775839995825663, + 0.002016792001086287, + 0.0020935409993398935, + 0.001987750001717359, + 0.0019851660035783425, + 0.0019987499981652945, + 0.002054457989288494, + 0.0020454589975997806, + 0.0019592919998103753, + 0.0020232910028425977, + 0.0020444999972824007, + 0.00207366599352099, + 0.001955833999090828, + 0.002227042001322843, + 0.0020345419907243922, + 0.002096957992762327, + 0.002001875007408671, + 0.0020402919908519834, + 0.002114041999448091, + 0.0020251660025678575, + 0.002028416987741366, + 0.0020179580023977906, + 0.002013083008932881, + 0.001998666994040832, + 0.002023790992097929, + 0.0020682500035036355, + 0.0021080840087961406, + 0.0020737919985549524, + 0.002010790994972922, + 0.0020503330015344545, + 0.0020191249932395294 + ] + }, + { + "forward": 0.005523749990970828, + "prefill": 0.005583999998634681, + "decode_steps": [ + 0.0020231250091455877, + 0.001963834001799114, + 0.0019519589986884966, + 0.001925916993059218, + 0.0020038749935338274, + 0.0020092500053578988, + 0.0019009169918717816, + 0.0021822910057380795, + 0.0028205830021761358, + 0.0020010829903185368, + 0.0019128749991068617, + 0.0019608750008046627, + 0.00207425000553485, + 0.0020200839935569093, + 0.0019705420127138495, + 0.0019913749856641516, + 0.002066667002509348, + 0.0019822499889414757, + 0.001997542000026442, + 0.0019958329939981923, + 0.002083208004478365, + 0.002028125003562309, + 0.002040416991803795, + 0.002070832997560501, + 0.0020982079877285287, + 0.0020067919977009296, + 0.002020540996454656, + 0.0019968330016126856, + 0.002022208005655557, + 0.0020149589981883764, + 0.002034417004324496, + 0.0020303750061430037 + ] + }, + { + "forward": 0.006111083988798782, + "prefill": 0.0066810410062316805, + "decode_steps": [ + 0.002253083002869971, + 0.0020555000082822517, + 0.0019933750008931383, + 0.002064707994577475, + 0.002051458999630995, + 0.00199162500211969, + 0.001988749994779937, + 0.0020912920008413494, + 0.001980957997147925, + 0.0019543749949662015, + 0.002146458995412104, + 0.0020143749861745164, + 0.0021322919928934425, + 0.0020317079906817526, + 0.0020982089918106794, + 0.002107249994878657, + 0.002089917004923336, + 0.0020107499876758084, + 0.0019870410033036023, + 0.0020401249930728227, + 0.0019788750068983063, + 0.002072083007078618, + 0.002068125002551824, + 0.0020933750056428835, + 0.0021166669903323054, + 0.001959417000762187, + 0.002304500012542121, + 0.0021791670005768538, + 0.0020430830045370385, + 0.0021314579935278744, + 0.0020184589957352728, + 0.001993917001527734 + ] + }, + { + "forward": 0.00551058299606666, + "prefill": 0.00640479099820368, + "decode_steps": [ + 0.002140374999726191, + 0.0020524589926935732, + 0.001978416010388173, + 0.001977332998649217, + 0.0020798750047106296, + 0.0019690409972099587, + 0.0034235420025652274, + 0.002046084002358839, + 0.0019821250025415793, + 0.0020894170011160895, + 0.002014457990298979, + 0.00203558400971815, + 0.0019441250042291358, + 0.0027265419921604916, + 0.002084500010823831, + 0.002128374995663762, + 0.0020157499966444448, + 0.002122624995536171, + 0.002043000000412576, + 0.0019414160051383078, + 0.0036512920050881803, + 0.002018250001128763, + 0.0020069579913979396, + 0.001958833003300242, + 0.002007417002459988, + 0.0019976250041509047, + 0.0019007920054718852, + 0.0034762499999487773, + 0.0019749589991988614, + 0.0022853330010548234, + 0.001960958994459361, + 0.0019995000038761646 + ] + }, + { + "forward": 0.006070333998650312, + "prefill": 0.005616916998405941, + "decode_steps": [ + 0.002126457999111153, + 0.0032393330038757995, + 0.0020324999932199717, + 0.001987790994462557, + 0.0019171249878127128, + 0.0020311249973019585, + 0.001990208009374328, + 0.001983583002584055, + 0.003268583008321002, + 0.0020795419986825436, + 0.001896333007607609, + 0.0019785829936154187, + 0.0020016249909531325, + 0.002003417001105845, + 0.001957917003892362, + 0.0031522920035058632, + 0.002001500004553236, + 0.0019253749924246222, + 0.0019510830024955794, + 0.002021125008468516, + 0.0020217499986756593, + 0.002060165992588736, + 0.0027999170124530792, + 0.0020398329943418503, + 0.0019635839998954907, + 0.0019713329966180027, + 0.002148375002434477, + 0.0021074579999549314, + 0.002039624989265576, + 0.0034397499985061586, + 0.0020078330126125365, + 0.002002041001105681 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 5.559541496040765, + "p95": 6.438402197818504, + "min": 5.51058299606666, + "max": 6.7062080051982775, + "iqr": 0.4068232410645578, + "mean": 5.7689416978973895, + "cv": 0.0694806022907391 + }, + "prefill_ms": { + "count": 10, + "median": 5.569729502894916, + "p95": 6.55672850261908, + "min": 5.49374999536667, + "max": 6.6810410062316805, + "iqr": 0.08532299034413882, + "mean": 5.754245801654179, + "cv": 0.0734593140212452 + }, + "decode_ms_per_step": { + "count": 320, + "median": 2.0282709956518374, + "p95": 2.3716271047305786, + "min": 1.896333007607609, + "max": 3.6512920050881803, + "iqr": 0.10153176117455587, + "mean": 2.090673046586744, + "cv": 0.11931498202475564 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1933.548818570403, + "p95": 1958.7136504894843, + "min": 1824.7352355030832, + "max": 1963.1286715872955, + "iqr": 31.081651126326733, + "mean": 1914.4234702678373, + "cv": 0.025626755825652273 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1774.3772926018785, + "p95": 1805.547904053077, + "min": 1674.5597515379245, + "max": 1808.2660738240916, + "iqr": 30.44310708051239, + "mean": 1762.8181413396785, + "cv": 0.02581012958018446 + }, + "memory": { + "peak_extra_bytes": 0, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "reference", + "batch": 4, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.01188883300346788, + "prefill": 0.00846595800248906, + "decode_steps": [ + 0.004621667001629248, + 0.004627000002074055, + 0.004512125000474043, + 0.004520791000686586, + 0.0045375420013442636, + 0.004543292001471855, + 0.00438416599354241, + 0.0044284169998718426, + 0.00444904099276755, + 0.0044900829961989075, + 0.004400958001497202, + 0.004461874996195547, + 0.004869333002716303, + 0.004370416005258448, + 0.004344166998635046, + 0.004832999999052845, + 0.004501499992329627, + 0.004531708007561974, + 0.004095540993148461, + 0.0045348330022534356, + 0.004432166999322362, + 0.004564457995002158, + 0.004491790998145007, + 0.004539290996035561, + 0.004521333001321182, + 0.0044694590033032, + 0.004312667006161064, + 0.004595457998220809, + 0.004521249997196719, + 0.004342375003034249, + 0.004411166999489069, + 0.003962249989854172 + ] + }, + { + "forward": 0.011765957999159582, + "prefill": 0.008778625007835217, + "decode_steps": [ + 0.005018833006033674, + 0.004558333006571047, + 0.00458345799415838, + 0.004551625010208227, + 0.004532541002845392, + 0.004453208006452769, + 0.004447500003152527, + 0.0046963749919086695, + 0.005059083006926812, + 0.004688000000896864, + 0.004638707992853597, + 0.00453466598992236, + 0.004591292003169656, + 0.0040716670046094805, + 0.004537625005468726, + 0.004547000004095025, + 0.004464708996238187, + 0.004514457992627285, + 0.004433250011061318, + 0.00401054100075271, + 0.004472417000215501, + 0.004448332998435944, + 0.004512583996984176, + 0.004484332996071316, + 0.004493833999731578, + 0.004441875003976747, + 0.003980250010499731, + 0.004443541998625733, + 0.004492250009207055, + 0.004539250003290363, + 0.0085618750017602, + 0.00957570799801033 + ] + }, + { + "forward": 0.01272008300293237, + "prefill": 0.009525375004159287, + "decode_steps": [ + 0.005123458991874941, + 0.0049729169986676425, + 0.004579040993121453, + 0.004560542001854628, + 0.004705957995611243, + 0.004595332997268997, + 0.004413041999214329, + 0.00455708299705293, + 0.004560333996778354, + 0.0044927919952897355, + 0.0044786670041503385, + 0.004546083000604995, + 0.004520167000009678, + 0.004487624988541938, + 0.004609290990629233, + 0.004527249999227934, + 0.004380958998808637, + 0.004896833997918293, + 0.004667542001698166, + 0.004582665991620161, + 0.004534415988018736, + 0.004393249997519888, + 0.004495750006753951, + 0.004517124994890764, + 0.004539250003290363, + 0.0044710420042974874, + 0.004471916006878018, + 0.00421466700208839, + 0.004452249995665625, + 0.004443500001798384, + 0.004471542008104734, + 0.004500416005612351 + ] + }, + { + "forward": 0.012589499994646758, + "prefill": 0.013896709002438001, + "decode_steps": [ + 0.0063247080106521025, + 0.005988290999084711, + 0.005873250003787689, + 0.006026999995810911, + 0.006005832998198457, + 0.005582624988164753, + 0.004866167000727728, + 0.005164750007679686, + 0.005627916005323641, + 0.00488420799956657, + 0.004701666999608278, + 0.004569208002067171, + 0.00448329201026354, + 0.004569207987515256, + 0.004539041998214088, + 0.0046651250013383105, + 0.004518250003457069, + 0.004436415998497978, + 0.004483916985918768, + 0.004506790995947085, + 0.004589375006617047, + 0.004290291995857842, + 0.004486832986003719, + 0.0044938750070286915, + 0.004441124998265877, + 0.004409957997268066, + 0.0044632919889409095, + 0.004481249998207204, + 0.004434332993696444, + 0.004535500003839843, + 0.004461832999368198, + 0.004589624993968755 + ] + }, + { + "forward": 0.011639333999482915, + "prefill": 0.008438958000624552, + "decode_steps": [ + 0.004658999998355284, + 0.004149874992435798, + 0.004256125001120381, + 0.004586958006257191, + 0.004664874999434687, + 0.004263334005372599, + 0.004079000005731359, + 0.004561249996186234, + 0.004616458012606017, + 0.004509874997893348, + 0.004608125003869645, + 0.004609958006767556, + 0.004503000003751367, + 0.004553624996333383, + 0.004431583001860417, + 0.004759333998663351, + 0.004495000001043081, + 0.004483707991312258, + 0.004555209001409821, + 0.004486582998652011, + 0.004516417000559159, + 0.004241874994477257, + 0.0048314169980585575, + 0.00992770899028983, + 0.011846958004753105, + 0.005785291999927722, + 0.005940582996117882, + 0.005442333989776671, + 0.0055855829996289685, + 0.004676042008213699, + 0.004568458010908216, + 0.004458084003999829 + ] + }, + { + "forward": 0.010951375006698072, + "prefill": 0.008552166997105815, + "decode_steps": [ + 0.004634083001292311, + 0.004420042008860037, + 0.003875374997733161, + 0.004526290998910554, + 0.004467332997592166, + 0.004619125000317581, + 0.00452699999732431, + 0.004473624998354353, + 0.004471416003070772, + 0.004702374993939884, + 0.004968374996678904, + 0.004719291013316251, + 0.004557457999908365, + 0.004164458005107008, + 0.004335292003815994, + 0.00418158300453797, + 0.004532666993327439, + 0.004476290996535681, + 0.004526958000496961, + 0.004452667009900324, + 0.004274209000868723, + 0.004586458002449945, + 0.004464167010155506, + 0.004709332992206328, + 0.004508500001975335, + 0.004433042005985044, + 0.004537040993454866, + 0.004513209001743235, + 0.004498207999859005, + 0.004695458002970554, + 0.004361833009170368, + 0.004756750000524335 + ] + }, + { + "forward": 0.011896750002051704, + "prefill": 0.00846799999999348, + "decode_steps": [ + 0.004812458995729685, + 0.004566875009913929, + 0.004555124993203208, + 0.0048102500004461035, + 0.00428549999196548, + 0.004488708000280894, + 0.0044864999945275486, + 0.00449320899497252, + 0.004563124995911494, + 0.004459165997104719, + 0.004055999990669079, + 0.004576624996843748, + 0.003894458001013845, + 0.00443450000602752, + 0.004728583007818088, + 0.004498540991335176, + 0.004544499999610707, + 0.0045413329935399815, + 0.00457933300640434, + 0.004054124990943819, + 0.004500333001487888, + 0.004520417001913302, + 0.00445729200146161, + 0.004431041001225822, + 0.0042872500052908435, + 0.0053756250126753, + 0.005045125013566576, + 0.0048862080002436414, + 0.004671499991673045, + 0.004935875011142343, + 0.004938917001709342, + 0.004550665995338932 + ] + }, + { + "forward": 0.011334291004459374, + "prefill": 0.008556208005757071, + "decode_steps": [ + 0.004763332995935343, + 0.004658416990423575, + 0.004667041008360684, + 0.004538958994089626, + 0.004201165997073986, + 0.004810290993191302, + 0.004546042007859796, + 0.004583583999192342, + 0.004563374997815117, + 0.004193417000351474, + 0.004575250000925735, + 0.004433624999364838, + 0.004518750007264316, + 0.0045402090036077425, + 0.004519916998106055, + 0.004557042004307732, + 0.00442345799820032, + 0.004255082996678539, + 0.004324249995988794, + 0.00518104100774508, + 0.004989916997146793, + 0.004324707988416776, + 0.004762541997479275, + 0.004598292012815364, + 0.004585667003993876, + 0.004370583003037609, + 0.004642583007807843, + 0.004754374996991828, + 0.004551874997559935, + 0.00448662499547936, + 0.004569625001749955, + 0.004539250003290363 + ] + }, + { + "forward": 0.01227741599723231, + "prefill": 0.008494916997733526, + "decode_steps": [ + 0.004342915999586694, + 0.003965708005125634, + 0.004265541996574029, + 0.00445916700118687, + 0.005636834001052193, + 0.004678999990574084, + 0.004692124988650903, + 0.004361832994618453, + 0.004239582995069213, + 0.004332374999648891, + 0.004849542005104013, + 0.005839416000526398, + 0.004611625001416542, + 0.004495333007071167, + 0.005675332999089733, + 0.00449970799672883, + 0.004009583994047716, + 0.0039268750115297735, + 0.004487625003093854, + 0.004458583003724925, + 0.0046937079896451905, + 0.005895207999856211, + 0.004535625004791655, + 0.003935125001589768, + 0.00441691699961666, + 0.004076750003150664, + 0.004343125008745119, + 0.00463241599209141, + 0.004238374996930361, + 0.00471395900240168, + 0.004832667007576674, + 0.004231416998663917 + ] + }, + { + "forward": 0.012319917004788294, + "prefill": 0.008467416992061771, + "decode_steps": [ + 0.004699416997027583, + 0.0047114170010900125, + 0.005173792000277899, + 0.004639040998881683, + 0.005607874991255812, + 0.004071167000802234, + 0.004405583007610403, + 0.004659375001210719, + 0.004205666991765611, + 0.00452308300009463, + 0.004571209006826393, + 0.005001000012271106, + 0.004491833009524271, + 0.003999499997007661, + 0.004392625007312745, + 0.004526459000771865, + 0.004383583000162616, + 0.004505791992414743, + 0.0039307080005528405, + 0.004632208001567051, + 0.00423879201116506, + 0.004631124989828095, + 0.004213957989122719, + 0.0049557500024093315, + 0.0046352079953067005, + 0.005775333003839478, + 0.004055666999192908, + 0.004127167005208321, + 0.004512125000474043, + 0.003966083007981069, + 0.004480417002923787, + 0.004549833000055514 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 11.892791502759792, + "p95": 12.661320649203844, + "min": 10.951375006698072, + "max": 12.72008300293237, + "iqr": 0.6383017534972169, + "mean": 11.938345701491926, + "cv": 0.04636926607010656 + }, + "prefill_ms": { + "count": 10, + "median": 8.52354199741967, + "p95": 11.929608703212576, + "min": 8.438958000624552, + "max": 13.896709002438001, + "iqr": 0.2554580132709816, + "mean": 9.164433401019778, + "cv": 0.1849342242473735 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.5263749998412095, + "p95": 5.628361905110069, + "min": 3.875374997733161, + "max": 11.846958004753105, + "iqr": 0.19298924235044979, + "mean": 4.647461546937848, + "cv": 0.15411701089485183 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 876.5338520844186, + "p95": 891.6312043225779, + "min": 791.8164433608141, + "max": 893.7229308631956, + "iqr": 35.84251912238847, + "mean": 861.9105738518738, + "cv": 0.038924269619108894 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 827.2604064331999, + "p95": 841.7635998372615, + "min": 752.5313345532675, + "max": 843.8424740074576, + "iqr": 33.46882402829567, + "mean": 812.0459374642686, + "cv": 0.041645744744505976 + }, + "memory": { + "peak_extra_bytes": 33816576, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "auto", + "batch": 4, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.01390791599988006 + }, + { + "forward": 0.019002750006620772 + }, + { + "forward": 0.013477915999828838 + }, + { + "forward": 0.013184625000576489 + }, + { + "forward": 0.01290658299694769 + }, + { + "forward": 0.013046334002865478 + }, + { + "forward": 0.014821832999587059 + }, + { + "forward": 0.015855874997214414 + }, + { + "forward": 0.014623958995798603 + }, + { + "forward": 0.016066999989561737 + } + ], + "forward_ms": { + "count": 10, + "median": 14.265937497839332, + "p95": 17.681662498944203, + "min": 12.90658299694769, + "max": 19.002750006620772, + "iqr": 2.3394167474179994, + "mean": 14.689479098888114, + "cv": 0.1285778718337424 + }, + "memory": { + "peak_extra_bytes": 38535168, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "sdpa", + "batch": 4, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.0109646659984719, + "prefill": 0.010458625009050593, + "decode_steps": [ + 0.002391165995504707, + 0.0038482079980894923, + 0.002256332998513244, + 0.0025787079939618707, + 0.0023795410088496283, + 0.0023280000023078173, + 0.002316083002369851, + 0.0033839579991763458, + 0.00242866700864397, + 0.0023249160003615543, + 0.002387875007116236, + 0.002333249998628162, + 0.002271417004521936, + 0.0038001249922672287, + 0.0022729160118615255, + 0.002333666998310946, + 0.002337083002203144, + 0.0023475000052712858, + 0.0023427910055033863, + 0.0024143749906215817, + 0.0023274580016732216, + 0.0022624580014962703, + 0.002334792006877251, + 0.0022974169987719506, + 0.0023640409926883876, + 0.002347209010622464, + 0.003325917001347989, + 0.0023602500004926696, + 0.0023086669971235096, + 0.002332125004613772, + 0.002393874994595535, + 0.002398291995632462 + ] + }, + { + "forward": 0.011949917010497302, + "prefill": 0.012223249999806285, + "decode_steps": [ + 0.0024554999981774017, + 0.0023908750008558854, + 0.002302624998264946, + 0.0024967909994302318, + 0.0034287089947611094, + 0.002454000001307577, + 0.0023716249997960404, + 0.002306249996763654, + 0.002371915994444862, + 0.0038185419980436563, + 0.002267750009195879, + 0.002376500007812865, + 0.0023450419976143166, + 0.002386042004218325, + 0.002379792000283487, + 0.0038807500095572323, + 0.002364290994592011, + 0.002472499996656552, + 0.002299041996593587, + 0.0022747499897377566, + 0.0023083339910954237, + 0.0037972910067765042, + 0.0023101670085452497, + 0.002343209009268321, + 0.0023110839974833652, + 0.0023901670065242797, + 0.002396500000031665, + 0.003849500004434958, + 0.002343375002965331, + 0.0023431250010617077, + 0.002317874997970648, + 0.0024053750094026327 + ] + }, + { + "forward": 0.012641041001188569, + "prefill": 0.011004999993019737, + "decode_steps": [ + 0.0032978329982142895, + 0.002376709002419375, + 0.002579416992375627, + 0.0024819160025799647, + 0.0023824999952921644, + 0.0025015419960254803, + 0.0025666669971542433, + 0.0023141249985201284, + 0.0023891659948276356, + 0.0023005830007605255, + 0.0024594999995315447, + 0.002459916999214329, + 0.0024624579964438453, + 0.002448334009386599, + 0.0023141669953474775, + 0.00237333400582429, + 0.0023783329961588606, + 0.002335458993911743, + 0.003399583001737483, + 0.002417249997961335, + 0.0024154999991878867, + 0.0023679580044699833, + 0.0024013750080484897, + 0.002538207991165109, + 0.0024152499972842634, + 0.0025019169988809153, + 0.0024028750049183145, + 0.0024445830058539286, + 0.0029772919951938093, + 0.002359791993512772, + 0.0038592920027440414, + 0.0023476670030504465 + ] + }, + { + "forward": 0.01107370899990201, + "prefill": 0.012660665990551934, + "decode_steps": [ + 0.003098500004853122, + 0.0026026660052593797, + 0.002410999994026497, + 0.0025325829919893295, + 0.0023257500142790377, + 0.002559000000474043, + 0.002384124993113801, + 0.0023356250021606684, + 0.0037652910104952753, + 0.0024089999933494255, + 0.0024349169980268925, + 0.0022728749900124967, + 0.0024649160041008145, + 0.0024631249980302528, + 0.0023424999963026494, + 0.002449292005621828, + 0.0022695000079693273, + 0.0023709170054644346, + 0.0025158749922411516, + 0.002296874998137355, + 0.002328665999812074, + 0.002825000003213063, + 0.0023964170104591176, + 0.0026233750104438514, + 0.002477541987900622, + 0.002352875002543442, + 0.002325875000678934, + 0.0034156249894294888, + 0.0022556670010089874, + 0.0025123750092461705, + 0.0023630409996258095, + 0.0023205830075312406 + ] + }, + { + "forward": 0.011268791000475176, + "prefill": 0.010709374997531995, + "decode_steps": [ + 0.002511166996555403, + 0.0029765840008622035, + 0.002338999998755753, + 0.0024994999985210598, + 0.0023257080028997734, + 0.0023722500045550987, + 0.0025710839981911704, + 0.002428541993140243, + 0.0023409999994328246, + 0.002353292002226226, + 0.0023520420072600245, + 0.002342624997254461, + 0.0023695419949945062, + 0.0037791669892612845, + 0.002291750002768822, + 0.002510833990527317, + 0.0023689999943599105, + 0.0023522499977843836, + 0.0023904999980004504, + 0.002492041006917134, + 0.0023491659958381206, + 0.002383541999734007, + 0.0022797920100856572, + 0.0023244170006364584, + 0.00234433299920056, + 0.0039310410065809265, + 0.002369624999118969, + 0.0024127920041792095, + 0.002326999994693324, + 0.0023059999948600307, + 0.0024883330042939633, + 0.002434416994219646 + ] + }, + { + "forward": 0.010830541999894194, + "prefill": 0.012699083003099076, + "decode_steps": [ + 0.0030210830009309575, + 0.002504625008441508, + 0.002419624986941926, + 0.0024467079929308966, + 0.0029155410011298954, + 0.002510165999410674, + 0.002406375002465211, + 0.0023975829972187057, + 0.002362250001169741, + 0.0027902919973712415, + 0.0024449579941574484, + 0.0023493750049965456, + 0.0024185000074794516, + 0.00237724999897182, + 0.002437499992083758, + 0.0024950420047389343, + 0.0023929579911055043, + 0.0023934170021675527, + 0.0023285829956876114, + 0.002429291998851113, + 0.002366000000620261, + 0.0037974169972585514, + 0.0024093749962048605, + 0.0023998749966267496, + 0.0024517089914297685, + 0.002384125007665716, + 0.0023585839953739196, + 0.002569499993114732, + 0.00247837501228787, + 0.002433166009723209, + 0.0023999580007512122, + 0.0023933330085128546 + ] + }, + { + "forward": 0.011251958989305422, + "prefill": 0.011379917006706819, + "decode_steps": [ + 0.0031115409947233275, + 0.0024433340004179627, + 0.0024639580078655854, + 0.002466541001922451, + 0.002446333004627377, + 0.0025380830047652125, + 0.0024767919967416674, + 0.0023991669877432287, + 0.0024124589981511235, + 0.0023231249942909926, + 0.0024144580092979595, + 0.00251649999700021, + 0.004032625001855195, + 0.002337291996809654, + 0.002739917006692849, + 0.0024789159942884, + 0.002753666994976811, + 0.0036616249999497086, + 0.00254658299672883, + 0.0024895409878809005, + 0.002454582994687371, + 0.002555125000071712, + 0.0025398750003660098, + 0.003942082999856211, + 0.0025042089982889593, + 0.002551707992097363, + 0.0024326669954461977, + 0.0025768339983187616, + 0.0025885409995680675, + 0.0026639999996405095, + 0.002532665996113792, + 0.00245520900352858 + ] + }, + { + "forward": 0.011842292005894706, + "prefill": 0.011883540995768271, + "decode_steps": [ + 0.00284658299642615, + 0.002608124996186234, + 0.0030727499979548156, + 0.002962707992992364, + 0.0023857080086600035, + 0.002546416988479905, + 0.0024402919952990487, + 0.0035659580025821924, + 0.0025102500076172873, + 0.0029438339988701046, + 0.0024366249999729916, + 0.002456458008964546, + 0.0026543329877313226, + 0.004061792002175935, + 0.002569250005763024, + 0.0025469170068390667, + 0.002614041994092986, + 0.002559375003329478, + 0.0026640839932952076, + 0.0027101669984403998, + 0.0025445420033065602, + 0.0024998340086312965, + 0.0024857500102370977, + 0.003112791004241444, + 0.003107541997451335, + 0.0024690830032341182, + 0.00251520900928881, + 0.0027382919943192974, + 0.002672333997907117, + 0.0026022920064860955, + 0.0031340419955085963, + 0.0025312920042779297 + ] + }, + { + "forward": 0.011962999997194856, + "prefill": 0.013454333995468915, + "decode_steps": [ + 0.0037479159946087748, + 0.003512583003612235, + 0.003367083001649007, + 0.0033260409982176498, + 0.0032925419945968315, + 0.003271833003964275, + 0.004743541998323053, + 0.003253417002269998, + 0.0034527079988038167, + 0.0033681249915389344, + 0.0033432909986004233, + 0.003368833989952691, + 0.0031426669884240255, + 0.003230208996683359, + 0.0032966660073725507, + 0.004773249995196238, + 0.0033023330033756793, + 0.00339445800636895, + 0.0033189160021720454, + 0.004465207995963283, + 0.0031347089970950037, + 0.002850875011063181, + 0.0028210830059833825, + 0.0033574580011190847, + 0.004674959011026658, + 0.003176792000886053, + 0.0034007079957518727, + 0.0033504589955555275, + 0.0032405000092694536, + 0.0034330829948885366, + 0.0033580000017536804, + 0.0033085000031860545 + ] + }, + { + "forward": 0.012016959008178674, + "prefill": 0.012363749992800876, + "decode_steps": [ + 0.0030660000047646463, + 0.003415708997636102, + 0.003545707993907854, + 0.0034075419971486554, + 0.0033259999909205362, + 0.003118708002148196, + 0.002874584009987302, + 0.0033416249934816733, + 0.0033965000038733706, + 0.0036121250013820827, + 0.00334324999130331, + 0.0034210419980809093, + 0.0035418329935055226, + 0.0034267080045538023, + 0.0033535830007167533, + 0.002893791999667883, + 0.0033538749994477257, + 0.0035271249944344163, + 0.00476195800001733, + 0.0035968750016763806, + 0.003399624998564832, + 0.0035909170110244304, + 0.0033843330020317808, + 0.003388708006241359, + 0.002947874992969446, + 0.0034472920087864622, + 0.0034159170027123764, + 0.002969042005133815, + 0.0034637500066310167, + 0.0034185410040663555, + 0.0033709999988786876, + 0.004324541994719766 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 11.55554150318494, + "p95": 12.360204104334116, + "min": 10.830541999894194, + "max": 12.641041001188569, + "iqr": 0.8414577532676049, + "mean": 11.58028760110028, + "cv": 0.05043608582649604 + }, + "prefill_ms": { + "count": 10, + "median": 12.053395497787278, + "p95": 13.114471048902487, + "min": 10.458625009050593, + "max": 13.454333995468915, + "iqr": 1.4877077446726616, + "mean": 11.88375409838045, + "cv": 0.08185912222784912 + }, + "decode_ms_per_step": { + "count": 320, + "median": 2.477166992321145, + "p95": 3.7975523970089853, + "min": 2.2556670010089874, + "max": 4.773249995196238, + "iqr": 0.7421037516905926, + "mean": 2.742514328156176, + "cv": 0.19438444551001333 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1562.694025071401, + "p95": 1599.739373317831, + "min": 1152.3356385043028, + "max": 1601.6349494026274, + "iqr": 119.86676759861552, + "mean": 1480.0966133313898, + "cv": 0.11759731175277356 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1362.2049070274054, + "p95": 1412.7146005541647, + "min": 1027.8393335307094, + "max": 1412.9952839051866, + "iqr": 86.69203470118873, + "mean": 1300.9399339796832, + "cv": 0.10993995164692195 + }, + "memory": { + "peak_extra_bytes": 33554432, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "reference", + "batch": 4, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.025707666005473584, + "prefill": 0.022673458006465808, + "decode_steps": [ + 0.005340165997040458, + 0.005919791990891099, + 0.005053582994150929, + 0.005453167002997361, + 0.004955417010933161, + 0.004782375006470829, + 0.004855582999880426, + 0.004727749997982755, + 0.005357750007533468, + 0.005222791995038278, + 0.0050250409985892475, + 0.00452329199470114, + 0.006455249997088686, + 0.0047443749936064705, + 0.0046432079980149865, + 0.005329042003722861, + 0.005034290996263735, + 0.005390709004132077, + 0.004935874996590428, + 0.0049997500027529895, + 0.004873333004070446, + 0.004974957992089912, + 0.004791833998751827, + 0.004404874998726882, + 0.004872958001215011, + 0.006106166998506524, + 0.004440249991603196, + 0.004798542009666562, + 0.004867166993790306, + 0.00487224999233149, + 0.004853042002650909, + 0.004745416998048313 + ] + }, + { + "forward": 0.025210874999174848, + "prefill": 0.02241916699858848, + "decode_steps": [ + 0.005866583000170067, + 0.005448957992484793, + 0.004703375001554377, + 0.005347832993720658, + 0.00535712500277441, + 0.005749500007368624, + 0.006021957990014926, + 0.005346292004105635, + 0.006282707996433601, + 0.005688542005373165, + 0.00559787500242237, + 0.006630459000007249, + 0.006445624996558763, + 0.006995625008130446, + 0.0057773750013438985, + 0.005626333004329354, + 0.005989415993099101, + 0.006429417000617832, + 0.005599833995802328, + 0.008326917013619095, + 0.005638667003950104, + 0.006036334001692012, + 0.005423750000772998, + 0.005661709001287818, + 0.007317708994378336, + 0.00680875001125969, + 0.005590207991190255, + 0.00547937500232365, + 0.005429791999631561, + 0.006247375000384636, + 0.005691583006409928, + 0.005270832989481278 + ] + }, + { + "forward": 0.027899875000002794, + "prefill": 0.02421991700248327, + "decode_steps": [ + 0.006640458988840692, + 0.006498374990769662, + 0.006454874994233251, + 0.005244207990472205, + 0.005560792007599957, + 0.0051469160098349676, + 0.005172542005311698, + 0.005372958999942057, + 0.005537167002330534, + 0.0047662500001024455, + 0.005398707988206297, + 0.005925625009695068, + 0.005485375004354864, + 0.0055810420017223805, + 0.00502591700933408, + 0.004716249997727573, + 0.005499541002791375, + 0.005357292000553571, + 0.004839250002987683, + 0.0052315840002847835, + 0.006034040998201817, + 0.00485970800218638, + 0.005306916995323263, + 0.005817500001285225, + 0.00469804200110957, + 0.005521458006114699, + 0.005513875003089197, + 0.004634541008272208, + 0.005394875013735145, + 0.0064571249968139455, + 0.004881708009634167, + 0.005189208997762762 + ] + }, + { + "forward": 0.024465875001624227, + "prefill": 0.022630666993791237, + "decode_steps": [ + 0.00546645799477119, + 0.005192416996578686, + 0.0048841250099940225, + 0.004936249999445863, + 0.005305041995598003, + 0.005130332996486686, + 0.004738374991575256, + 0.0052503750048344955, + 0.004862458998104557, + 0.0049307920126011595, + 0.00616237499343697, + 0.0048022919945651665, + 0.004617584010702558, + 0.004921375002595596, + 0.004778332993737422, + 0.004794833002961241, + 0.004720124998129904, + 0.004310000003897585, + 0.004750916006742045, + 0.004765459001646377, + 0.004664416002924554, + 0.004713125003036112, + 0.004734540998470038, + 0.004505417004111223, + 0.0042989580106223, + 0.004745875005028211, + 0.004654624994145706, + 0.005394292005803436, + 0.0051805419934680685, + 0.0051817909989040345, + 0.006114375006291084, + 0.0050972500030184165 + ] + }, + { + "forward": 0.02588991599623114, + "prefill": 0.021480583993252367, + "decode_steps": [ + 0.0058217500045429915, + 0.00538345800305251, + 0.004853749996982515, + 0.005315958012943156, + 0.006528958008857444, + 0.006571792007889599, + 0.005214749995502643, + 0.005331749998731539, + 0.005318166993674822, + 0.005280500001390465, + 0.005040165997343138, + 0.005955041997367516, + 0.00525358400773257, + 0.006183749996125698, + 0.004722041994682513, + 0.005033458000980318, + 0.005914791996474378, + 0.005043625002144836, + 0.005435625003883615, + 0.005406874988693744, + 0.005250917005469091, + 0.005341292009688914, + 0.005260999998426996, + 0.005110416008392349, + 0.004919125000014901, + 0.004984458006219938, + 0.005118208006024361, + 0.005358124995836988, + 0.005440000008093193, + 0.006371666997438297, + 0.004999625001801178, + 0.005210583010921255 + ] + }, + { + "forward": 0.024856125004589558, + "prefill": 0.022517332996358164, + "decode_steps": [ + 0.005339290990377776, + 0.004966958993463777, + 0.004603041990776546, + 0.00466720899567008, + 0.004772542000864632, + 0.005039625000790693, + 0.004835500003537163, + 0.005160874992725439, + 0.004752166991238482, + 0.0047167080047074705, + 0.004953374998876825, + 0.004866167000727728, + 0.004659124999307096, + 0.004720291995909065, + 0.004909167008008808, + 0.004610499992850237, + 0.004345916997408494, + 0.0048689170071156695, + 0.0055459160066675395, + 0.00491820799652487, + 0.004937167002935894, + 0.004732333007268608, + 0.004757750008138828, + 0.004750332998810336, + 0.004841250003664754, + 0.0049082909972639754, + 0.0049047079955926165, + 0.004914374992949888, + 0.0054280829936033115, + 0.005002208010409959, + 0.005115249994560145, + 0.0044215420057298616 + ] + }, + { + "forward": 0.026887791987974197, + "prefill": 0.023429916996974498, + "decode_steps": [ + 0.005630291998386383, + 0.015288957991288044, + 0.009484624999458902, + 0.005808499990962446, + 0.00547037499200087, + 0.005527082990738563, + 0.0055311250034719706, + 0.005790042006992735, + 0.0055270830052904785, + 0.00523395900381729, + 0.005399250003392808, + 0.005220125007326715, + 0.005297499999869615, + 0.0053960410004947335, + 0.005314667010679841, + 0.005136125007993542, + 0.005336416012141854, + 0.005140708002727479, + 0.00506479199975729, + 0.005105832999106497, + 0.005272416994557716, + 0.005186499998671934, + 0.006511875006253831, + 0.005334250003215857, + 0.005353957996703684, + 0.004932583004119806, + 0.004815500011318363, + 0.005295875002047978, + 0.005167875002371147, + 0.005277124990243465, + 0.00522629200713709, + 0.005303915997501463 + ] + }, + { + "forward": 0.0262968750030268, + "prefill": 0.023787459009326994, + "decode_steps": [ + 0.007623207988217473, + 0.005590499989921227, + 0.005545957988942973, + 0.006174750000354834, + 0.005224209002335556, + 0.005100666006910615, + 0.005525166998268105, + 0.005765583002357744, + 0.005725459006498568, + 0.006167707993881777, + 0.005730082993977703, + 0.005928124999627471, + 0.005728709002141841, + 0.005769916999270208, + 0.005928958009462804, + 0.006971875001909211, + 0.006799208000302315, + 0.0059689999907277524, + 0.005902582997805439, + 0.006917207996593788, + 0.006937750003999099, + 0.005998915992677212, + 0.005784499997389503, + 0.005859249999048188, + 0.006619083011173643, + 0.006065791007131338, + 0.005504209009814076, + 0.00575899999239482, + 0.005704375013010576, + 0.005396707987529226, + 0.005692332997568883, + 0.0054328330006683245 + ] + }, + { + "forward": 0.028307957996730693, + "prefill": 0.023967083005118184, + "decode_steps": [ + 0.0073294169997097924, + 0.0062171669997042045, + 0.005570875000557862, + 0.006571417005034164, + 0.00547562500287313, + 0.006127458997070789, + 0.006003375005093403, + 0.005792916999780573, + 0.0063022500107763335, + 0.005836917000124231, + 0.006726082996465266, + 0.006308040989097208, + 0.005989874989609234, + 0.006045666988939047, + 0.0074630830058595166, + 0.005881500008399598, + 0.005831958013004623, + 0.005528458001208492, + 0.005598582996753976, + 0.006006374998833053, + 0.005320249998476356, + 0.006151415989734232, + 0.005967292003333569, + 0.005716083003790118, + 0.005125958006829023, + 0.005699040993931703, + 0.006306541996309534, + 0.005966083001112565, + 0.005345833007595502, + 0.005259582991129719, + 0.00567958400642965, + 0.005484459004946984 + ] + }, + { + "forward": 0.028334542002994567, + "prefill": 0.024273541013826616, + "decode_steps": [ + 0.006643457993050106, + 0.00643283300451003, + 0.0061295839986996725, + 0.006330000003799796, + 0.005775042009190656, + 0.006660250001004897, + 0.006316208993666805, + 0.006077291996916756, + 0.006400166996172629, + 0.0065967080008704215, + 0.006505750003270805, + 0.006421375001082197, + 0.005804417000035755, + 0.005839167002704926, + 0.007575875002657995, + 0.006497750000562519, + 0.007812457988620736, + 0.006509790997370146, + 0.0066895000054501, + 0.005867333005880937, + 0.006316666011116467, + 0.008423999999649823, + 0.007750374992610887, + 0.007788916991557926, + 0.008270958001958206, + 0.006620250002015382, + 0.0072315419965889305, + 0.007164249997003935, + 0.00637191699934192, + 0.006591417011804879, + 0.006484917001216672, + 0.0067850829946110025 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 26.09339549962897, + "p95": 28.322579200175824, + "min": 24.465875001624227, + "max": 28.334542002994567, + "iqr": 2.3117814962461125, + "mean": 26.38574989978224, + "cv": 0.05383063639859467 + }, + "prefill_ms": { + "count": 10, + "median": 23.051687501720153, + "p95": 24.24941020872211, + "min": 21.480583993252367, + "max": 24.273541013826616, + "iqr": 1.3765105104539543, + "mean": 23.139912601618562, + "cv": 0.04017071190939563 + }, + "decode_ms_per_step": { + "count": 320, + "median": 5.398978995799553, + "p95": 6.973062502220273, + "min": 4.2989580106223, + "max": 15.288957991288044, + "iqr": 0.974385238805553, + "mean": 5.606333722289492, + "cv": 0.1689232493188327 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 713.5490894156546, + "p95": 814.5499955989665, + "min": 596.2216752236162, + "max": 820.698045861942, + "iqr": 107.80810919634644, + "mean": 720.0812665330702, + "cv": 0.10017862521461039 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 629.7547835994387, + "p95": 712.2552749530827, + "min": 535.6572106788967, + "max": 717.1586511425023, + "iqr": 88.11499394925056, + "mean": 636.8310533004865, + "cv": 0.09139031221369771 + }, + "memory": { + "peak_extra_bytes": 1074003968, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "auto", + "batch": 4, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.03239183299592696 + }, + { + "forward": 0.03523166599916294 + }, + { + "forward": 0.0393442919885274 + }, + { + "forward": 0.04625112500798423 + }, + { + "forward": 0.04480591700121295 + }, + { + "forward": 0.050367249990813434 + }, + { + "forward": 0.04921625000133645 + }, + { + "forward": 0.038019624989829026 + }, + { + "forward": 0.03156433299591299 + }, + { + "forward": 0.031178917008219287 + } + ], + "forward_ms": { + "count": 10, + "median": 38.681958489178214, + "p95": 49.84929999554879, + "min": 31.178917008219287, + "max": 50.367249990813434, + "iqr": 12.788031759555452, + "mean": 39.83712079789257, + "cv": 0.18489358205919165 + }, + "memory": { + "peak_extra_bytes": 1116995584, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "sdpa", + "batch": 4, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.026456666993908584, + "prefill": 0.026181417008046992, + "decode_steps": [ + 0.003711540994117968, + 0.003754083998501301, + 0.0039766249974491075, + 0.00384266598848626, + 0.003648957994300872, + 0.003267082996899262, + 0.003214290991309099, + 0.0034071249974658713, + 0.0032589999900665134, + 0.002750792002188973, + 0.002743916993495077, + 0.0027231659914832562, + 0.002659791993210092, + 0.002633249998325482, + 0.004101291007827967, + 0.0026725000061560422, + 0.0026409160054754466, + 0.002659916994161904, + 0.0026481670065550134, + 0.004222333009238355, + 0.0031313749932451174, + 0.002981208002893254, + 0.002699542004847899, + 0.0029079579981043935, + 0.00446291700063739, + 0.0028021250036545098, + 0.0029080419917590916, + 0.002779791990178637, + 0.003177582999342121, + 0.004612874996382743, + 0.003337000001920387, + 0.0031294579966925085 + ] + }, + { + "forward": 0.026667291997000575, + "prefill": 0.025306666997494176, + "decode_steps": [ + 0.004093166993698105, + 0.003126374998828396, + 0.0031839589937590063, + 0.0031241669930750504, + 0.003131833000225015, + 0.003162124994560145, + 0.004312666002078913, + 0.003110874997219071, + 0.0030518340063281357, + 0.0027763329999288544, + 0.0028801669977838174, + 0.0028257090016268194, + 0.0025915410078596324, + 0.002666749991476536, + 0.0026783339999383315, + 0.002729250001721084, + 0.0044247920013731346, + 0.002710957996896468, + 0.002722582998103462, + 0.002756208006758243, + 0.002876790997106582, + 0.002870833995984867, + 0.0028807500057155266, + 0.00262641700101085, + 0.0027859159890795127, + 0.0027783340046880767, + 0.002706165993004106, + 0.004009124997537583, + 0.0027409589965827763, + 0.0030848750029690564, + 0.0029370830015977845, + 0.002863708999939263 + ] + }, + { + "forward": 0.030486208997899666, + "prefill": 0.03158558400173206, + "decode_steps": [ + 0.0042059579864144325, + 0.004458417010027915, + 0.0040014169935602695, + 0.003665124997496605, + 0.003677541986689903, + 0.00378158398962114, + 0.005504208005731925, + 0.003707666997797787, + 0.0036193330015521497, + 0.0031968750117812306, + 0.0033145830093417317, + 0.0030550840019714087, + 0.0031894580024527386, + 0.0033689159899950027, + 0.005142541995155625, + 0.0032098749943543226, + 0.003264207989559509, + 0.003352541010826826, + 0.0033926670002983883, + 0.003325957994093187, + 0.003416959007154219, + 0.0032294580014422536, + 0.0033439159888075665, + 0.003406666000955738, + 0.003530415997374803, + 0.0031822080054553226, + 0.0031890420068521053, + 0.003381500006071292, + 0.0034823750029318035, + 0.0032570840121479705, + 0.0032430419960292056, + 0.003657666005892679 + ] + }, + { + "forward": 0.032204040995566174, + "prefill": 0.03644166700541973, + "decode_steps": [ + 0.007268791989190504, + 0.004584541005897336, + 0.004600207990733907, + 0.0050942079978995025, + 0.004006165996543132, + 0.0039265420055016875, + 0.004581165994750336, + 0.0044068749994039536, + 0.003821332997176796, + 0.0034507500095060095, + 0.003820875004748814, + 0.004307042006985284, + 0.0034088750107912347, + 0.0034925420040963218, + 0.003760542007512413, + 0.004337541991844773, + 0.0035033329913858324, + 0.003456999998888932, + 0.004075000004377216, + 0.004459749994566664, + 0.0035140419931849465, + 0.0035146250011166558, + 0.004053124997881241, + 0.004469999999855645, + 0.0037355000094976276, + 0.0037817920092493296, + 0.0042497079994063824, + 0.005626832993584685, + 0.004195875008008443, + 0.004002666988526471, + 0.004624124994734302, + 0.004134582995902747 + ] + }, + { + "forward": 0.035703000001376495, + "prefill": 0.037848958003451116, + "decode_steps": [ + 0.005253332987194881, + 0.005291541994665749, + 0.005135208004503511, + 0.004793708998477086, + 0.005410833007772453, + 0.005033249995904043, + 0.004080124999745749, + 0.004714166003395803, + 0.004679290999774821, + 0.003468208000413142, + 0.0036210419930284843, + 0.004362791994935833, + 0.0040354579978156835, + 0.0035364579962333664, + 0.003547957996488549, + 0.0040103329956764355, + 0.004066166991833597, + 0.003899584000464529, + 0.0037083339993841946, + 0.004037165999761783, + 0.005159207998076454, + 0.003621166993980296, + 0.003834583010757342, + 0.004199750008410774, + 0.0040169999992940575, + 0.0037205830012680963, + 0.004064125008881092, + 0.004475540990824811, + 0.0041548750014044344, + 0.0039074999949662015, + 0.0047262500011129305, + 0.004656750010326505 + ] + }, + { + "forward": 0.03666349999548402, + "prefill": 0.03510795900365338, + "decode_steps": [ + 0.006128958004410379, + 0.004441833996679634, + 0.00463037499866914, + 0.007334041001740843, + 0.004582415989716537, + 0.004794458000105806, + 0.005025499995099381, + 0.004513292005867697, + 0.0047015000018291175, + 0.0037646249984391034, + 0.003627042009611614, + 0.0036616249999497086, + 0.006137833988759667, + 0.0038755419955123216, + 0.0038700830045854673, + 0.007061500000418164, + 0.004789583996171132, + 0.003814583003986627, + 0.007301165998796932, + 0.0038369169924408197, + 0.003950416008592583, + 0.0037442080065375194, + 0.0038254999963101, + 0.003787084002397023, + 0.0038699590077158064, + 0.005509167007403448, + 0.003963957991800271, + 0.0038660830032313243, + 0.003921457988326438, + 0.00550570800260175, + 0.0040303749992745, + 0.004129000008106232 + ] + }, + { + "forward": 0.03743325000687037, + "prefill": 0.037192542004049756, + "decode_steps": [ + 0.008606166011304595, + 0.0073114999977406114, + 0.006862124995677732, + 0.008207291000871919, + 0.006736582989105955, + 0.0094693749997532, + 0.006241292008780874, + 0.008151166999596171, + 0.006221292002010159, + 0.0061272500024642795, + 0.006064000001060776, + 0.006862499998533167, + 0.0058097919973079115, + 0.006115874988608994, + 0.004762083000969142, + 0.005091750004794449, + 0.004329750008764677, + 0.004567333002341911, + 0.004229458005283959, + 0.004062250009155832, + 0.004263249997165985, + 0.00407804200949613, + 0.00461804200313054, + 0.004061832994921133, + 0.004006874994956888, + 0.004203542004688643, + 0.0050654160004341975, + 0.003982541995355859, + 0.00420362499426119, + 0.004955249998602085, + 0.004213958993204869, + 0.0042795830086106434 + ] + }, + { + "forward": 0.026132291997782886, + "prefill": 0.026370416991994716, + "decode_steps": [ + 0.004486666002776474, + 0.004707000000053085, + 0.00417579100758303, + 0.0043348330073058605, + 0.00429579199408181, + 0.004379249992780387, + 0.004105291998712346, + 0.004064542008563876, + 0.004210540995700285, + 0.004161832999670878, + 0.003415790997678414, + 0.003239625002606772, + 0.003351499995915219, + 0.0035378749889787287, + 0.003412624995689839, + 0.003210290989954956, + 0.0032520829990971833, + 0.0034327499888604507, + 0.0033862499985843897, + 0.0034742499992717057, + 0.003285041995695792, + 0.0032815000013215467, + 0.003675250001833774, + 0.003583208002964966, + 0.0033193749986821786, + 0.0032254160032607615, + 0.003332958003738895, + 0.004565124996588565, + 0.0034277080121682957, + 0.003221374994609505, + 0.0033349160075886175, + 0.0035341659968253225 + ] + }, + { + "forward": 0.024211833995650522, + "prefill": 0.02407654101261869, + "decode_steps": [ + 0.004888042007223703, + 0.004466957994736731, + 0.004497000001720153, + 0.004319707994000055, + 0.004362040999694727, + 0.0043158329935977235, + 0.004373832998680882, + 0.0049926659994525835, + 0.0043668750004144385, + 0.00346604100195691, + 0.003907333011738956, + 0.003548042004695162, + 0.0033818340016296133, + 0.0034793329978128895, + 0.003390666999621317, + 0.0033974580001085997, + 0.0034610419970704243, + 0.004203666001558304, + 0.0034957919997395948, + 0.003364416988915764, + 0.0034656250063562766, + 0.003447416005656123, + 0.0034712909982772544, + 0.0034867920039687306, + 0.0034326669992879033, + 0.0038659580022795126, + 0.00413916700927075, + 0.0034879159939009696, + 0.003829124994808808, + 0.003526125001371838, + 0.0035838330077240244, + 0.0035327079967828467 + ] + }, + { + "forward": 0.02473916699818801, + "prefill": 0.024726457995711826, + "decode_steps": [ + 0.004757167000207119, + 0.004747667000629008, + 0.004672624985687435, + 0.004495874993153848, + 0.004459042000235058, + 0.004628583003068343, + 0.0048641670000506565, + 0.0045563330058939755, + 0.004859291992033832, + 0.003670874997624196, + 0.0037420000007841736, + 0.003506667009787634, + 0.0037602920056087896, + 0.003679708010167815, + 0.0035907499986933544, + 0.0037059580063214526, + 0.003631250001490116, + 0.0038708750071236864, + 0.0037318329996196553, + 0.0036118330026511103, + 0.004377999997814186, + 0.0037037500005681068, + 0.0035942079994129017, + 0.0034727499878499657, + 0.004482541000470519, + 0.004131082998355851, + 0.0036754999891854823, + 0.0035734589910134673, + 0.00352549999661278, + 0.0035022090014535934, + 0.003547790998709388, + 0.004368542009615339 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 28.57675049745012, + "p95": 37.08686250174651, + "min": 24.211833995650522, + "max": 37.43325000687037, + "iqr": 8.614874503109604, + "mean": 30.06972519797273, + "cv": 0.170666024378975 + }, + "prefill_ms": { + "count": 10, + "median": 28.978000496863388, + "p95": 37.553570803720504, + "min": 24.07654101261869, + "max": 37.848958003451116, + "iqr": 10.582885504845763, + "mean": 30.483821002417244, + "cv": 0.18742157049212596 + }, + "decode_ms_per_step": { + "count": 320, + "median": 3.800833503191825, + "p95": 6.1273354025615845, + "min": 2.5915410078596324, + "max": 9.4693749997532, + "iqr": 0.9965934950741939, + "mean": 4.007977937135365, + "cv": 0.25792158617278704 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1018.6103834681171, + "p95": 1280.8241630732268, + "min": 720.0688029473916, + "max": 1316.5936004348564, + "iqr": 168.86550392538902, + "mean": 1025.8322969886112, + "cv": 0.1702941936611744 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 852.9913803272234, + "p95": 1018.843533902974, + "min": 595.4780891269152, + "max": 1044.6655576309877, + "iqr": 145.8903167036459, + "mean": 827.2126677007038, + "cv": 0.16402262686837535 + }, + "memory": { + "peak_extra_bytes": 1107296256, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "reference", + "batch": 4, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.07153433401254006, + "prefill": 0.07273787500162143, + "decode_steps": [ + 0.020504165993770584, + 0.012905916009913199, + 0.00858262499968987, + 0.0076798329973826185, + 0.00826000000233762, + 0.007283874991117045, + 0.007212708005681634, + 0.006588709002244286, + 0.006251625003642403, + 0.006713916998705827, + 0.006248625009902753, + 0.006814915992435999, + 0.005926082987571135, + 0.00603058299748227, + 0.006721792000462301, + 0.006086042005335912, + 0.006419167009880766, + 0.005801416002213955, + 0.006408041997929104, + 0.0065555829933146015, + 0.006071541996789165, + 0.006745749997207895, + 0.006884750007884577, + 0.005749374991864897, + 0.005962208000710234, + 0.0057989170018117875, + 0.005880374999833293, + 0.007416916996589862, + 0.005930750005063601, + 0.0062167919968487695, + 0.005824917010613717, + 0.005877582996618003 + ] + }, + { + "forward": 0.07012954200035892, + "prefill": 0.06817337500979193, + "decode_steps": [ + 0.016536625000298955, + 0.0116907089977758, + 0.006889417010825127, + 0.006886208997457288, + 0.006673333991784602, + 0.006134625000413507, + 0.00733024999499321, + 0.006033625002601184, + 0.006177750008646399, + 0.0054103750007925555, + 0.005984291012282483, + 0.006175917005748488, + 0.0058661250077420846, + 0.005601374999969266, + 0.007338874987908639, + 0.00595900000189431, + 0.006068333997973241, + 0.0058476250123931095, + 0.00580404199718032, + 0.005825374988489784, + 0.005800291008199565, + 0.005990124991512857, + 0.006011999998008832, + 0.005901124997762963, + 0.005964166994090192, + 0.00550199999997858, + 0.0063771669956622645, + 0.007368833001237363, + 0.00607433301047422, + 0.005879958000150509, + 0.005890083004487678, + 0.005655916000250727 + ] + }, + { + "forward": 0.07236512498639058, + "prefill": 0.07156891698832624, + "decode_steps": [ + 0.007618709001690149, + 0.0084811249980703, + 0.0066799579944927245, + 0.0064866250031627715, + 0.006392583003616892, + 0.00630408400320448, + 0.007321542012505233, + 0.0062534169992432, + 0.006299415996181779, + 0.0062537499907193705, + 0.006208124992554076, + 0.006819625006755814, + 0.006430125009501353, + 0.006270832993322983, + 0.0062386670033447444, + 0.006106375003582798, + 0.00736799999140203, + 0.006957499994314276, + 0.007864832994528115, + 0.005918832990573719, + 0.006118042001617141, + 0.006038457999238744, + 0.006154125003376976, + 0.006366707995766774, + 0.006298708001850173, + 0.006234083004528657, + 0.006270708006923087, + 0.00596050001331605, + 0.0063367499969899654, + 0.006036292004864663, + 0.00629674999800045, + 0.00689808301103767 + ] + }, + { + "forward": 0.08808791599585675, + "prefill": 0.09177533299953211, + "decode_steps": [ + 0.021216416993411258, + 0.013351750007132068, + 0.012741667000227608, + 0.010151875001611188, + 0.009310958994319662, + 0.009630874992581084, + 0.008753457994316705, + 0.009067749997484498, + 0.008513542008586228, + 0.008347916998900473, + 0.007990833997610025, + 0.009831292001763359, + 0.008386041008634493, + 0.0077305000013438985, + 0.008232791005866602, + 0.007906166996690445, + 0.0077893330017104745, + 0.007982541996170767, + 0.00808783300453797, + 0.007316624993109144, + 0.007770416996208951, + 0.007570041008875705, + 0.007999292007298209, + 0.009251708994270302, + 0.0071924579970072955, + 0.007635999994818121, + 0.007924332996481098, + 0.0078085410059429705, + 0.007562457991298288, + 0.00800070799596142, + 0.006885042006615549, + 0.007947250007418916 + ] + }, + { + "forward": 0.08809575000486802, + "prefill": 0.08714116700866725, + "decode_steps": [ + 0.02428725000936538, + 0.01890145799552556, + 0.019241582995164208, + 0.011690708008245565, + 0.008596874991781078, + 0.00948262499878183, + 0.009515666999504901, + 0.00980504100152757, + 0.00943566599744372, + 0.00857141699816566, + 0.009191874996758997, + 0.008052000004681759, + 0.008116917000734247, + 0.00820450000173878, + 0.008684874992468394, + 0.008574375009629875, + 0.008388083995669149, + 0.008627624993096106, + 0.007446750008966774, + 0.008775124995736405, + 0.009001583006465808, + 0.01065591700898949, + 0.007822666986612603, + 0.008162750003975816, + 0.00830708300054539, + 0.007965875003719702, + 0.008136624994222075, + 0.007873124995967373, + 0.007687666002311744, + 0.0088598330039531, + 0.008570499994675629, + 0.008387959009269252 + ] + }, + { + "forward": 0.08045150000543799, + "prefill": 0.0767621249979129, + "decode_steps": [ + 0.009027624997543171, + 0.007617249997565523, + 0.007358834001934156, + 0.007668791993637569, + 0.007615750000695698, + 0.0068685419973917305, + 0.007593709000502713, + 0.007360416988376528, + 0.007238958001835272, + 0.007024208010989241, + 0.007351082997047342, + 0.007313458001590334, + 0.007076832989696413, + 0.00718929099093657, + 0.007703250012127683, + 0.008488916995702311, + 0.009311332993092947, + 0.009196207989589311, + 0.00875495899526868, + 0.012578124995343387, + 0.008586083989939652, + 0.008456374998786487, + 0.007940750001580454, + 0.007705541996983811, + 0.008154374998412095, + 0.008140124991768971, + 0.008149375003995374, + 0.007969208003487438, + 0.008711416012374684, + 0.008578250010032207, + 0.009121542010689154, + 0.008936416998039931 + ] + }, + { + "forward": 0.07783774999552406, + "prefill": 0.07137133300420828, + "decode_steps": [ + 0.01116599999659229, + 0.007661458002985455, + 0.007608499989146367, + 0.008004667004570365, + 0.010892082995269448, + 0.008172417001333088, + 0.00750762500683777, + 0.008152000009431504, + 0.00793120899470523, + 0.008059749990934506, + 0.007490416988730431, + 0.007737958992947824, + 0.0071618340007262304, + 0.0073545839986763895, + 0.008226542006013915, + 0.00790108299406711, + 0.008371040996280499, + 0.007811500006937422, + 0.008756957991863601, + 0.007436374988174066, + 0.00779362500179559, + 0.007381167000858113, + 0.008046083006775007, + 0.007289832996320911, + 0.007450958990375511, + 0.0072581670101499185, + 0.0073665420059114695, + 0.0083493330021156, + 0.008047874987823889, + 0.007717417000094429, + 0.0075482499960344285, + 0.007171124991145916 + ] + }, + { + "forward": 0.06995045900112018, + "prefill": 0.06420241699379403, + "decode_steps": [ + 0.013675499998498708, + 0.023207292004371993, + 0.03089666699816007, + 0.013229290998424403, + 0.010354457990615629, + 0.010716832999605685, + 0.011975999994319864, + 0.010223500008578412, + 0.008536042005289346, + 0.00885566599026788, + 0.009011125002871267, + 0.009458540997002274, + 0.009269665999454446, + 0.010666792004485615, + 0.010262957992381416, + 0.009416332992259413, + 0.009485207992838696, + 0.00901516700105276, + 0.010670292002032511, + 0.009553249998134561, + 0.00848208399838768, + 0.008383459004107863, + 0.007248084002640098, + 0.007756749997497536, + 0.007704667004873045, + 0.007820249986252747, + 0.007465832997695543, + 0.007881541998358443, + 0.008625749993370846, + 0.007961708004586399, + 0.009320541008492, + 0.009496041995589621 + ] + }, + { + "forward": 0.07833637499425095, + "prefill": 0.0681315830006497, + "decode_steps": [ + 0.011346332990797237, + 0.008648041999549605, + 0.010595375002594665, + 0.008724291998078115, + 0.008413667004788294, + 0.008485500002279878, + 0.008771291992161423, + 0.008925208006985486, + 0.009556582997902296, + 0.00834279099944979, + 0.008001584006706253, + 0.008201583012123592, + 0.00792274999548681, + 0.00792070799798239, + 0.008279083005618304, + 0.008671290997881442, + 0.007654874993022531, + 0.007917125010862947, + 0.007950915998662822, + 0.007639207993634045, + 0.0078295419953065, + 0.007684416006668471, + 0.007775000005494803, + 0.007808209003997035, + 0.007659459006390534, + 0.007726333002210595, + 0.007888166001066566, + 0.008540416994947009, + 0.007916375005152076, + 0.007920207994175144, + 0.008453332993667573, + 0.00787062500603497 + ] + }, + { + "forward": 0.06916954201005865, + "prefill": 0.06374629199854098, + "decode_steps": [ + 0.012475291994633153, + 0.010220666998066008, + 0.011407208992750384, + 0.00953812499938067, + 0.008576332998927683, + 0.008751667002798058, + 0.008807582998997532, + 0.008234124994487502, + 0.008129540990921669, + 0.00869566599430982, + 0.007836749995476566, + 0.007897542003775015, + 0.008071250005741604, + 0.007884499995270744, + 0.008668791997479275, + 0.009142499999143183, + 0.008680416009156033, + 0.009149499994236976, + 0.007838959005312063, + 0.009613834001356736, + 0.008686832996318117, + 0.007018666990916245, + 0.009346500009996817, + 0.009222874999977648, + 0.008462500001769513, + 0.008704375009983778, + 0.00803529200493358, + 0.008369417002541013, + 0.008035624996409751, + 0.007989624995389022, + 0.010350124997785315, + 0.007575916999485344 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 75.10143749095732, + "p95": 88.09222470081295, + "min": 69.16954201005865, + "max": 88.09575000486802, + "iqr": 9.441978749237023, + "mean": 76.59582930064062, + "cv": 0.09418461276177431 + }, + "prefill_ms": { + "count": 10, + "median": 71.47012499626726, + "p95": 89.68995830364292, + "min": 63.74629199854098, + "max": 91.77533299953211, + "iqr": 7.614031495904783, + "mean": 73.56104170030449, + "cv": 0.12641564933415975 + }, + "decode_ms_per_step": { + "count": 320, + "median": 7.911271000921261, + "p95": 11.704973547603007, + "min": 5.4103750007925555, + "max": 30.89666699816007, + "iqr": 1.6819892516650725, + "mean": 8.290300262251549, + "cv": 0.32640306484847403 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 485.075215356699, + "p95": 607.2538623661845, + "min": 380.242492027363, + "max": 611.6110574328885, + "iqr": 94.6356776929124, + "mean": 493.32120815944563, + "cv": 0.15727926375743612 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 380.548842161505, + "p95": 455.7815363726283, + "min": 318.2787754247716, + "max": 455.80271409536135, + "iqr": 68.97931017044044, + "mean": 383.61486749400603, + "cv": 0.1308691076163438 + }, + "memory": { + "peak_extra_bytes": 1107558400, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "auto", + "batch": 4, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.10848112500389107 + }, + { + "forward": 0.0894650419941172 + }, + { + "forward": 0.10817408299772069 + }, + { + "forward": 0.13125833400408737 + }, + { + "forward": 0.10744379200332332 + }, + { + "forward": 0.12048112500633579 + }, + { + "forward": 0.1509334169968497 + }, + { + "forward": 0.09365154198894743 + }, + { + "forward": 0.09557070799928624 + }, + { + "forward": 0.10315829201135784 + } + ], + "forward_ms": { + "count": 10, + "median": 107.808937500522, + "p95": 142.07962965010663, + "min": 89.4650419941172, + "max": 150.9334169968497, + "iqr": 20.013521003420465, + "mean": 110.86174600059167, + "cv": 0.1697157965874008 + }, + "memory": { + "peak_extra_bytes": 1126432768, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "sdpa", + "batch": 4, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.07622533399262466, + "prefill": 0.08329549999325536, + "decode_steps": [ + 0.011004042011336423, + 0.010425457992823794, + 0.0097069590119645, + 0.011367083003278822, + 0.012567249999847263, + 0.006371750001562759, + 0.005247957989922725, + 0.0055667080014245585, + 0.005031458000303246, + 0.007921166994492523, + 0.005749500007368624, + 0.0060175840044394135, + 0.0053259999986039475, + 0.006362334010191262, + 0.004997583004296757, + 0.0058349169994471595, + 0.0063603750022593886, + 0.006061249994672835, + 0.007594292008434422, + 0.005226415989454836, + 0.00586033400031738, + 0.006350375013425946, + 0.0068721669958904386, + 0.006625542009714991, + 0.00531245899037458, + 0.006769125000573695, + 0.006224124997970648, + 0.005917458009207621, + 0.0063735000003362074, + 0.005921792006120086, + 0.00593620800646022, + 0.0059305000031599775 + ] + }, + { + "forward": 0.06343287498748396, + "prefill": 0.06370762499864213, + "decode_steps": [ + 0.005039041992858984, + 0.004607666007359512, + 0.004330291005317122, + 0.00461412500590086, + 0.004665208005462773, + 0.004462541997781955, + 0.004237458997522481, + 0.004581917004543357, + 0.004633375006960705, + 0.004218875008518808, + 0.00455691700335592, + 0.0049646660045254976, + 0.004529958998318762, + 0.004197707996354438, + 0.004656167002394795, + 0.004731915993033908, + 0.004139125012443401, + 0.004661166996811517, + 0.0049157500034198165, + 0.004577291998430155, + 0.004504833996179514, + 0.005114833009429276, + 0.005098290988826193, + 0.004588749987306073, + 0.005278208001982421, + 0.0056174170022131875, + 0.005261708007310517, + 0.007615374997840263, + 0.006470249994890764, + 0.00941850000526756, + 0.0071711250056978315, + 0.006383792002452537 + ] + }, + { + "forward": 0.08363712500431575, + "prefill": 0.08936966699548066, + "decode_steps": [ + 0.007077417001710273, + 0.005809875001432374, + 0.006460457996581681, + 0.0070899999991524965, + 0.005871999994269572, + 0.007157083004130982, + 0.00540758301212918, + 0.005885499995201826, + 0.007598750002216548, + 0.005877417002920993, + 0.006459291995270178, + 0.005859208002220839, + 0.005728957999963313, + 0.008182332996511832, + 0.005784958993899636, + 0.006407666995073669, + 0.00819812499685213, + 0.009241916995961219, + 0.009959291011909954, + 0.0102574159973301, + 0.012013583997031674, + 0.0068786670017289, + 0.0069637499982491136, + 0.006460040996898897, + 0.005816916993353516, + 0.006317833001958206, + 0.005556707998039201, + 0.005975041989586316, + 0.005760915999417193, + 0.005785334011306986, + 0.005849541994393803, + 0.00583095800539013 + ] + }, + { + "forward": 0.10016041700146161, + "prefill": 0.10502437500690576, + "decode_steps": [ + 0.01164058300491888, + 0.010972125004627742, + 0.009823500004131347, + 0.00952095800312236, + 0.010953958990285173, + 0.00958804199763108, + 0.008541417002561502, + 0.010163250000914559, + 0.008877667001797818, + 0.008393582989810966, + 0.010181542005739175, + 0.007439333989168517, + 0.00881287500669714, + 0.006154290997073986, + 0.007350582993240096, + 0.007991332997335121, + 0.009623666992411017, + 0.0057301250053569674, + 0.006471458007581532, + 0.006492790998890996, + 0.006076874997233972, + 0.007803957996657118, + 0.0055629170092288405, + 0.008097124999039806, + 0.008144958002958447, + 0.00970450000022538, + 0.005737917002988979, + 0.008166958999936469, + 0.006432583002606407, + 0.006545291995280422, + 0.00825529100256972, + 0.006514583001262508 + ] + }, + { + "forward": 0.06916995899518952, + "prefill": 0.07186433298920747, + "decode_steps": [ + 0.005515042008482851, + 0.005171708995476365, + 0.004760958006954752, + 0.004896582991932519, + 0.005565040992223658, + 0.005421374997240491, + 0.005307750005158596, + 0.005263625003863126, + 0.005077292007626966, + 0.005335042005754076, + 0.0063302500057034194, + 0.005357207992346957, + 0.0056190830073319376, + 0.007613708003191277, + 0.007489125011488795, + 0.008885167000698857, + 0.008106749999569729, + 0.007707208991632797, + 0.006411791997379623, + 0.005773042008513585, + 0.005702417009160854, + 0.005609665997326374, + 0.005458750005345792, + 0.005437832995085046, + 0.005569958011619747, + 0.006023833993822336, + 0.005366124998545274, + 0.00548795799841173, + 0.0052189160051057115, + 0.005578790995059535, + 0.006148166008642875, + 0.00589787500211969 + ] + }, + { + "forward": 0.09364841699425597, + "prefill": 0.10163558399653994, + "decode_steps": [ + 0.008797666989266872, + 0.008038041007239372, + 0.009030208006151952, + 0.007482500004698522, + 0.007194667006842792, + 0.008144749997882172, + 0.008698582998476923, + 0.007131959006073885, + 0.0074876670114463195, + 0.007641667005373165, + 0.008130208996590227, + 0.0076386669970816, + 0.00715233400114812, + 0.007507250003982335, + 0.006443624995881692, + 0.010390833005658351, + 0.010367124996264465, + 0.011232999997446314, + 0.011212875004275702, + 0.011026292006135918, + 0.01147687500633765, + 0.00897104199975729, + 0.007783207998727448, + 0.0076817080116597936, + 0.015444542004843242, + 0.0077504580112872645, + 0.00742799999716226, + 0.006727874992066063, + 0.0083676249923883, + 0.008581792004406452, + 0.007896459006587975, + 0.00927212499664165 + ] + }, + { + "forward": 0.11971841599734034, + "prefill": 0.12739904200134333, + "decode_steps": [ + 0.008910707998438738, + 0.010795874986797571, + 0.010097958002006635, + 0.009878041993943043, + 0.009574540992616676, + 0.009295000010752119, + 0.008679167003720067, + 0.008371874995646067, + 0.009107374993618578, + 0.009105832999921404, + 0.00848900001437869, + 0.009810167000978254, + 0.008727000007638708, + 0.01016074999643024, + 0.010372124990681186, + 0.010239499999443069, + 0.008676208002725616, + 0.008300082990899682, + 0.007625250000273809, + 0.007851583999581635, + 0.007197165992693044, + 0.008117790988762863, + 0.006889041993417777, + 0.00851475000672508, + 0.007157750005717389, + 0.007962540999869816, + 0.007375832996331155, + 0.008020749999559484, + 0.006852500009699725, + 0.0077868330117780715, + 0.0075373750005383044, + 0.007515958001022227 + ] + }, + { + "forward": 0.06178975000511855, + "prefill": 0.06408924999414012, + "decode_steps": [ + 0.008828707999782637, + 0.014261750009609386, + 0.008698667006683536, + 0.007839665995561518, + 0.005289082997478545, + 0.005317333998391405, + 0.004893666991847567, + 0.005825040992931463, + 0.0054070830083219334, + 0.004849125005421229, + 0.00540591700701043, + 0.005192166994675063, + 0.00482666700554546, + 0.0050526249979157, + 0.005411292004282586, + 0.0048892920021899045, + 0.006061124993721023, + 0.004957083001499996, + 0.004787459009094164, + 0.005322917000739835, + 0.005092459003208205, + 0.004909374998533167, + 0.005012458001147024, + 0.005116707994602621, + 0.005078333007986657, + 0.004844708004384302, + 0.004863707988988608, + 0.0054355419997591525, + 0.004875708007602952, + 0.005117249995237216, + 0.005411541002104059, + 0.00488158299413044 + ] + }, + { + "forward": 0.06872308399761096, + "prefill": 0.07070424999983516, + "decode_steps": [ + 0.005462957997224294, + 0.005045750003773719, + 0.005332583008566871, + 0.005193832999793813, + 0.004871291006566025, + 0.005130666002514772, + 0.0051587920024758205, + 0.004744833000586368, + 0.005304374994011596, + 0.005089541999041103, + 0.004759834002470598, + 0.005602625009487383, + 0.005289209002512507, + 0.004896291997283697, + 0.005129917000886053, + 0.00523974999669008, + 0.004869208001764491, + 0.005174374993657693, + 0.005170791002456099, + 0.004896291997283697, + 0.005355791989131831, + 0.00542158298776485, + 0.005095291999168694, + 0.005277666001347825, + 0.005863833008334041, + 0.005648292004480027, + 0.008120416998281144, + 0.006892541001434438, + 0.008501208008965477, + 0.008010416990146041, + 0.007861916004912928, + 0.0076025839953217655 + ] + }, + { + "forward": 0.07496774999890476, + "prefill": 0.0783574170054635, + "decode_steps": [ + 0.010670000003301539, + 0.009064249999937601, + 0.008048709001741372, + 0.008238624999648891, + 0.00833641599456314, + 0.007914832996902987, + 0.008247457997640595, + 0.008041374996537343, + 0.006558041000971571, + 0.006874457991216332, + 0.0061152499984018505, + 0.006749042004230432, + 0.005940457995166071, + 0.005674209009157494, + 0.005848542001331225, + 0.00581824999244418, + 0.005722375004552305, + 0.006385082990163937, + 0.005709749995730817, + 0.005858249991433695, + 0.005804375003208406, + 0.005571957997744903, + 0.005825749991345219, + 0.006068583010346629, + 0.005827166998642497, + 0.005993625003611669, + 0.005703041999367997, + 0.005793416989035904, + 0.005984666990116239, + 0.00548125000204891, + 0.005825125001138076, + 0.005746416994952597 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 75.59654199576471, + "p95": 110.91731644919489, + "min": 61.78975000511855, + "max": 119.71841599734034, + "iqr": 22.310791249765316, + "mean": 81.1473126974306, + "cv": 0.2272466266579632 + }, + "prefill_ms": { + "count": 10, + "median": 80.82645849935943, + "p95": 117.3304418538464, + "min": 63.70762499864213, + "max": 127.39904200134333, + "iqr": 27.57483399909688, + "mean": 85.54470429808134, + "cv": 0.2403949075437281 + }, + "decode_ms_per_step": { + "count": 320, + "median": 6.131708003522363, + "p95": 10.392564255016623, + "min": 4.139125012443401, + "max": 15.444542004843242, + "iqr": 2.7044472481065895, + "mean": 6.823174209876015, + "cv": 0.28042195823162147 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 595.1448342498425, + "p95": 746.1347330070728, + "min": 460.21375164311576, + "max": 781.2297347523361, + "iqr": 179.13796102174717, + "mean": 604.5936835889568, + "cv": 0.182052393856158 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 432.6732844336091, + "p95": 541.7830476626685, + "min": 318.09510971173154, + "max": 562.5090981028549, + "iqr": 137.16214639922543, + "mean": 435.9770410400034, + "cv": 0.19088452430638297 + }, + "memory": { + "peak_extra_bytes": 1140850688, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "reference", + "batch": 4, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.24719283300510142, + "prefill": 0.24439320899546146, + "decode_steps": [ + 0.02201874999445863, + 0.018254124996019527, + 0.01305112500267569, + 0.011006375003489666, + 0.010866833996260539, + 0.009957500005839393, + 0.009894292001263238, + 0.00934654199227225, + 0.008333999998285435, + 0.009366583006340079, + 0.008638500003144145, + 0.009266875000321306, + 0.008575832995120436, + 0.008368875001906417, + 0.008619542000815272, + 0.008496125010424294, + 0.009929082996677607, + 0.008758249998209067, + 0.008884416005457751, + 0.009076582995476201, + 0.008444834005786106, + 0.008245083998190239, + 0.008513416003552265, + 0.008222416989156045, + 0.008453332993667573, + 0.008174041009624489, + 0.008457832998828962, + 0.009318249998614192, + 0.008179540993296541, + 0.00869625000632368, + 0.008378583006560802, + 0.009217832994181663 + ] + }, + { + "forward": 0.21302266701241024, + "prefill": 0.22605145900161006, + "decode_steps": [ + 0.02218329098832328, + 0.015856124999118038, + 0.011343874997692183, + 0.010887749987887219, + 0.010316374988178723, + 0.009648749997722916, + 0.009834584008785896, + 0.009747207994223572, + 0.009456708998186514, + 0.00928445799218025, + 0.008562374991015531, + 0.008726667001610622, + 0.00845754100009799, + 0.008592041995143518, + 0.008518999995430931, + 0.010270041006151587, + 0.008975624994491227, + 0.008352709002792835, + 0.008039833002840169, + 0.008053625002503395, + 0.00887779200274963, + 0.007976583001436666, + 0.008671250005136244, + 0.008408709007198922, + 0.009149709003395401, + 0.008113167001283728, + 0.008734333008760586, + 0.007981333998031914, + 0.008976666009402834, + 0.007520500003010966, + 0.008976124998298474, + 0.007893957998021506 + ] + }, + { + "forward": 0.228974500001641, + "prefill": 0.24600704199110623, + "decode_steps": [ + 0.020377167005790398, + 0.01459133300522808, + 0.011862208004458807, + 0.010277583001879975, + 0.01025120799022261, + 0.01065349999407772, + 0.009401208008057438, + 0.010577458000625484, + 0.012377999999444, + 0.020856040995568037, + 0.013547084003221244, + 0.011058500007493421, + 0.010298125009285286, + 0.010011374994064681, + 0.011308917004498653, + 0.009613917005481198, + 0.010327374999178573, + 0.011905499995918944, + 0.009815042008995079, + 0.010088625000207685, + 0.009636041999328882, + 0.009359666000818834, + 0.011274709002464078, + 0.009169833007035777, + 0.009316499999840744, + 0.009977875000913627, + 0.009233957985998131, + 0.010688042006222531, + 0.009180540990200825, + 0.009852582996245474, + 0.010332958001527004, + 0.009663999997428618 + ] + }, + { + "forward": 0.26421800001116935, + "prefill": 0.27805816699401475, + "decode_steps": [ + 0.022940374998142943, + 0.017079708995879628, + 0.013731624989304692, + 0.013952209003036842, + 0.012879375004558824, + 0.011381874996004626, + 0.010154832998523489, + 0.009865999993053265, + 0.011101833006250672, + 0.00939908299187664, + 0.009591249996447004, + 0.009498833009274676, + 0.009660833995440044, + 0.0112937090016203, + 0.010367833005147986, + 0.008467041014228016, + 0.010194165995926596, + 0.010155334006412886, + 0.010583209004835226, + 0.01005366699246224, + 0.008924708003178239, + 0.00895587500417605, + 0.008464250000542961, + 0.008691374998306856, + 0.00830149999819696, + 0.008466832994599827, + 0.008077374994172715, + 0.008514790999470279, + 0.007983708012034185, + 0.008479291995172389, + 0.008139832993037999, + 0.00807979199453257 + ] + }, + { + "forward": 0.23028058299678378, + "prefill": 0.24749937499291264, + "decode_steps": [ + 0.01830762501049321, + 0.015820333996089175, + 0.013798082989524119, + 0.011568959002033807, + 0.010621583001920953, + 0.012504792000981979, + 0.01149204200191889, + 0.012042832997394726, + 0.011022958991816267, + 0.013073875001282431, + 0.011483374997624196, + 0.01069929200457409, + 0.01178266700298991, + 0.014058708999073133, + 0.012530040999990888, + 0.012221041004522704, + 0.010950458992738277, + 0.011139875001390465, + 0.0111946670076577, + 0.010899417000473477, + 0.009985708995372988, + 0.010611707999487408, + 0.011476250001578592, + 0.010437250006361865, + 0.01140116598980967, + 0.011595167001360096, + 0.014326042000902817, + 0.013468582998029888, + 0.012788374995579943, + 0.011204790993360803, + 0.01143970899283886, + 0.012005707991193049 + ] + }, + { + "forward": 0.24503204200300388, + "prefill": 0.31757641599688213, + "decode_steps": [ + 0.028714540996588767, + 0.024224209002568386, + 0.026730165991466492, + 0.02131033300247509, + 0.02092854199872818, + 0.025401417005923577, + 0.02317416699952446, + 0.019355791999259964, + 0.018591749991173856, + 0.017658207987551577, + 0.01821404199290555, + 0.01759587500419002, + 0.01586833300825674, + 0.01628274998802226, + 0.016898709000088274, + 0.02171358300256543, + 0.020250166999176145, + 0.021408583997981623, + 0.01985366700682789, + 0.02326312501099892, + 0.0209029170073336, + 0.02435525000328198, + 0.02106116700451821, + 0.01975120800489094, + 0.01677699999709148, + 0.016261875003692694, + 0.017376124989823438, + 0.01548495799943339, + 0.01614412499475293, + 0.016451416013296694, + 0.01446574999135919, + 0.014343792005092837 + ] + }, + { + "forward": 0.27942316699773073, + "prefill": 0.2779292089981027, + "decode_steps": [ + 0.021195416993577965, + 0.020383875002153218, + 0.01467041700379923, + 0.013131708998116665, + 0.013915916992118582, + 0.012062708003213629, + 0.012135166994994506, + 0.012621665999176912, + 0.011880083999130875, + 0.011654125002678484, + 0.012369332995149307, + 0.009968500002287328, + 0.011576667006011121, + 0.011337833988363855, + 0.0117561250081053, + 0.011527833004947752, + 0.010877291002543643, + 0.010935250000329688, + 0.010535374996834435, + 0.010208374995272607, + 0.0106544589943951, + 0.010358041006838903, + 0.01068808299896773, + 0.011062500008847564, + 0.00980695900216233, + 0.010297500004526228, + 0.010574833999271505, + 0.010131750008440576, + 0.010261874995194376, + 0.010285041993483901, + 0.009771124998223968, + 0.009938000002875924 + ] + }, + { + "forward": 0.22245008399477229, + "prefill": 0.22548029100289568, + "decode_steps": [ + 0.021539583991398104, + 0.014460374994087033, + 0.012397250000503846, + 0.012404500012053177, + 0.011564374988665804, + 0.010714042000472546, + 0.011573083000257611, + 0.010553542000707239, + 0.010671916999854147, + 0.01209033299528528, + 0.01234608399681747, + 0.010509499988984317, + 0.010252084000967443, + 0.01121574999706354, + 0.01132362500356976, + 0.010738625001977198, + 0.010886124990065582, + 0.011999958995147608, + 0.010409333001007326, + 0.010824249999132007, + 0.011312875009025447, + 0.012189665998448618, + 0.010876542000914924, + 0.01122245799342636, + 0.010922000001301058, + 0.011138292000396177, + 0.00990766599716153, + 0.010066374990856275, + 0.011030166002456099, + 0.012467207998270169, + 0.011036625000997446, + 0.010598082997603342 + ] + }, + { + "forward": 0.23147154199250508, + "prefill": 0.23910529099521227, + "decode_steps": [ + 0.022455457990872674, + 0.018864459008909762, + 0.012868083009379916, + 0.013167500001145527, + 0.011308583008940332, + 0.010485584003617987, + 0.010505250000278465, + 0.011573541996767744, + 0.009833792006247677, + 0.010593583006993867, + 0.010626584000419825, + 0.01034583400178235, + 0.010141291000763886, + 0.009864457999356091, + 0.009824666994973086, + 0.009925082995323464, + 0.01016041700495407, + 0.010066834001918323, + 0.009945875004632398, + 0.009785000002011657, + 0.009533375006867573, + 0.009637832990847528, + 0.009739333007019013, + 0.010131291986908764, + 0.009545666995109059, + 0.009924916012096219, + 0.010596459003863856, + 0.00996320799458772, + 0.00938079200568609, + 0.00920337499701418, + 0.009283208011765964, + 0.009578917000908405 + ] + }, + { + "forward": 0.23909008299233392, + "prefill": 0.2232333749998361, + "decode_steps": [ + 0.02234899999166373, + 0.017477917004725896, + 0.012444874999346212, + 0.011714624997694045, + 0.00976620800793171, + 0.010417916011647321, + 0.009178624997730367, + 0.009001708996947855, + 0.009189417003653944, + 0.00884579100238625, + 0.00906562501040753, + 0.008615582992206328, + 0.008759208008996211, + 0.009066833998076618, + 0.008364082998014055, + 0.008454208000330254, + 0.008395000011660159, + 0.008157332995324396, + 0.008846625001751818, + 0.009123208001255989, + 0.008207082995795645, + 0.008581332993344404, + 0.009034834001795389, + 0.00897212499694433, + 0.009047084007761441, + 0.008669500006362796, + 0.009534291995805688, + 0.008308249991387129, + 0.00838320900220424, + 0.008512124986737035, + 0.008332375000463799, + 0.007750832999590784 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 235.2808124924195, + "p95": 272.5808418537781, + "min": 213.02266701241024, + "max": 279.42316699773073, + "iqr": 17.351614504150348, + "mean": 240.11555010074517, + "cv": 0.08263107526272677 + }, + "prefill_ms": { + "count": 10, + "median": 245.20012549328385, + "p95": 299.79320394559176, + "min": 223.2333749998361, + "max": 317.57641599688213, + "iqr": 41.00683349679457, + "mean": 252.5333833968034, + "cv": 0.11903954380872066 + }, + "decode_ms_per_step": { + "count": 320, + "median": 10.413624506327324, + "p95": 21.20116279402282, + "min": 7.520500003010966, + "max": 28.714540996588767, + "iqr": 2.901655996538466, + "mean": 11.680403662421668, + "cv": 0.3358417420809196 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 362.78308313520387, + "p95": 415.24080614003344, + "min": 202.91257443695915, + "max": 417.76996425770534, + "iqr": 62.889063715333975, + "mean": 356.4051206703345, + "cv": 0.17555300658387427 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 213.40718110363076, + "p95": 240.12696981539125, + "min": 134.9655790696824, + "max": 240.4025986336849, + "iqr": 23.24704441115952, + "mean": 209.50364538783742, + "cv": 0.144161544141251 + }, + "memory": { + "peak_extra_bytes": 1141112832, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "auto", + "batch": 8, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.006083667001803406 + }, + { + "forward": 0.0060360000061336905 + }, + { + "forward": 0.005987374999676831 + }, + { + "forward": 0.0060052089975215495 + }, + { + "forward": 0.006143209000583738 + }, + { + "forward": 0.006337208003969863 + }, + { + "forward": 0.006041291999281384 + }, + { + "forward": 0.006563833012478426 + }, + { + "forward": 0.006041000000550412 + }, + { + "forward": 0.005998375010676682 + } + ], + "forward_ms": { + "count": 10, + "median": 6.041145999915898, + "p95": 6.461851758649573, + "min": 5.987374999676831, + "max": 6.563833012478426, + "iqr": 0.11541675121407025, + "mean": 6.123716803267598, + "cv": 0.03035295465503303 + }, + "memory": { + "peak_extra_bytes": 36175872, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "sdpa", + "batch": 8, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.005258374992990866, + "prefill": 0.005323749996023253, + "decode_steps": [ + 0.00228245799371507, + 0.002167083992389962, + 0.002097124990541488, + 0.0021642080100718886, + 0.0021833329956280068, + 0.0021071249939268455, + 0.002088374996674247, + 0.0021742090029874817, + 0.0021242499933578074, + 0.002267957999720238, + 0.002195500012021512, + 0.002069124995614402, + 0.0021276250045048073, + 0.00209237499802839, + 0.0022189579904079437, + 0.0022439580061472952, + 0.002109082997776568, + 0.002103000006172806, + 0.002089250003336929, + 0.00214583300112281, + 0.0021227499964879826, + 0.0020682920003309846, + 0.0021197499881964177, + 0.0021221669885562733, + 0.0021775829955004156, + 0.002127375002601184, + 0.0021760000090580434, + 0.0020957090018782765, + 0.0021901250001974404, + 0.002280583998071961, + 0.002242041999124922, + 0.002129416010575369 + ] + }, + { + "forward": 0.008889541000826284, + "prefill": 0.008109791990136728, + "decode_steps": [ + 0.003246167005272582, + 0.0024952079984359443, + 0.0023381250066449866, + 0.0025051249976968393, + 0.002401249992544763, + 0.0023909170122351497, + 0.00242491700919345, + 0.00227483399794437, + 0.002331416995730251, + 0.0026344999932916835, + 0.002477625006577, + 0.002129832995706238, + 0.0023187089973362163, + 0.002261458008433692, + 0.0022407910000765696, + 0.0021566669893218204, + 0.002294207995873876, + 0.002122667006915435, + 0.002052666008239612, + 0.00215274999209214, + 0.00241499999538064, + 0.00225479100481607, + 0.002356875003897585, + 0.0022893749992363155, + 0.0020813750015804544, + 0.0021379579993663356, + 0.002120332996128127, + 0.00225687499914784, + 0.002540707995649427, + 0.002247791999252513, + 0.0022684160067001358, + 0.0021267079864628613 + ] + }, + { + "forward": 0.005962416005786508, + "prefill": 0.005227832996752113, + "decode_steps": [ + 0.002151417007553391, + 0.0022719169937772676, + 0.0023014170001260936, + 0.002184375000069849, + 0.0019979580101789907, + 0.0022031249973224476, + 0.0020739579922519624, + 0.002100707992212847, + 0.002068875008262694, + 0.002183542004786432, + 0.0020537499949568883, + 0.0020272500114515424, + 0.0020444999972824007, + 0.0021452079963637516, + 0.0021899580024182796, + 0.0020396250038174912, + 0.0019966249965364113, + 0.0023569999902974814, + 0.0020817909971810877, + 0.0020438750070752576, + 0.0021779160015285015, + 0.0022166669950820506, + 0.0020740410109283403, + 0.0020674999977927655, + 0.002419750002445653, + 0.002148333005607128, + 0.0020110409968765453, + 0.00212079199263826, + 0.0021653330040862784, + 0.0023944999993545935, + 0.0022902909986441955, + 0.00222491699969396 + ] + }, + { + "forward": 0.005366541998228058, + "prefill": 0.005862457997864112, + "decode_steps": [ + 0.002103999999235384, + 0.0020637080015148968, + 0.0020513339986791834, + 0.0021685410029022023, + 0.0020967089949408546, + 0.002593084005638957, + 0.00209612499747891, + 0.002017916995100677, + 0.0020249579974915832, + 0.0019937920005759224, + 0.002046458001132123, + 0.002103749997331761, + 0.0021004580048611388, + 0.0020890000014333054, + 0.0020248749933671206, + 0.002102999991620891, + 0.002137667004717514, + 0.0020524999999906868, + 0.002159333002055064, + 0.0021724160033045337, + 0.0021779579983558506, + 0.0020555000082822517, + 0.0023493330081691965, + 0.002223165996838361, + 0.002053707998129539, + 0.0022838750010123476, + 0.002118082993547432, + 0.0023417500051436946, + 0.002114667004207149, + 0.002161458003683947, + 0.0021485830075107515, + 0.002293458004714921 + ] + }, + { + "forward": 0.0053052080038469285, + "prefill": 0.005453916994156316, + "decode_steps": [ + 0.0025129159912467003, + 0.0020179999992251396, + 0.002149082996766083, + 0.0021453329973155633, + 0.0021104160114191473, + 0.0020278750016586855, + 0.0020345420052763075, + 0.0021889169875066727, + 0.0020372919971123338, + 0.0023105419968487695, + 0.002043708009296097, + 0.0021415410010376945, + 0.0021942089952062815, + 0.0021438749972730875, + 0.00221633400360588, + 0.0020772090065293014, + 0.002217082990682684, + 0.002214583000750281, + 0.002168125007301569, + 0.002075999989756383, + 0.002164915989851579, + 0.0021442919969558716, + 0.0021261250076349825, + 0.002178415990783833, + 0.002121166995493695, + 0.002245875002699904, + 0.0020984580041840672, + 0.0021232500002952293, + 0.0022110829886514693, + 0.002176916997996159, + 0.0021252089936751872, + 0.002138042007572949 + ] + }, + { + "forward": 0.0053223750001052395, + "prefill": 0.0053591250034514815, + "decode_steps": [ + 0.0022144169925013557, + 0.0020983340073144063, + 0.0020513339986791834, + 0.0023514159984188154, + 0.002166375008528121, + 0.002209500002209097, + 0.0021400000114226714, + 0.0019739580020541325, + 0.0024605840008007362, + 0.0021227079996606335, + 0.0021429170010378584, + 0.0021332499891286716, + 0.0021541669993894175, + 0.002069292007945478, + 0.0022077080066083, + 0.0020725829963339493, + 0.002230666999821551, + 0.0021952499955659732, + 0.0021275420003803447, + 0.0022551669972017407, + 0.002168125007301569, + 0.0021275000035529956, + 0.002219583999249153, + 0.002128125008312054, + 0.0021275420003803447, + 0.0020793329895241186, + 0.0021559169981628656, + 0.0021499580034287646, + 0.0022464170033345, + 0.002143916004570201, + 0.0022029169922461733, + 0.00218341700383462 + ] + }, + { + "forward": 0.005278291006106883, + "prefill": 0.005546290994971059, + "decode_steps": [ + 0.0021267919946694747, + 0.002042834006715566, + 0.0021101250022184104, + 0.0021428750042105094, + 0.00347445800434798, + 0.002258249995065853, + 0.0028759999986505136, + 0.0028736250096699223, + 0.002595666010165587, + 0.002708415995584801, + 0.0025970419956138358, + 0.0022719999979017302, + 0.0021244999952614307, + 0.002221250004367903, + 0.0020126250019529834, + 0.0023009580036159605, + 0.002151583001250401, + 0.0021362500119721517, + 0.002190708997659385, + 0.00298791698878631, + 0.00212154199834913, + 0.0023214589891722426, + 0.0022620419913437217, + 0.0023437909985659644, + 0.002214249994722195, + 0.0029068329895380884, + 0.0022167910065036267, + 0.002195417007897049, + 0.002157833005185239, + 0.0021682920050807297, + 0.0021168340026633814, + 0.0022611660097027197 + ] + }, + { + "forward": 0.00583337499119807, + "prefill": 0.005472083008498885, + "decode_steps": [ + 0.0022684160067001358, + 0.002126000006683171, + 0.0021704169921576977, + 0.0021246659889584407, + 0.002053457996225916, + 0.002134666996425949, + 0.002207792000262998, + 0.002193416003137827, + 0.002070667003863491, + 0.002333417010959238, + 0.0021254589955788106, + 0.0021310829906724393, + 0.0021553339902311563, + 0.0022013749985489994, + 0.0020863749959971756, + 0.002071541006444022, + 0.0024228749971371144, + 0.0021032920049037784, + 0.002141125005437061, + 0.002304291003383696, + 0.0021419160038931295, + 0.0022781669977121055, + 0.0020957910019205883, + 0.002593583005364053, + 0.002165500001865439, + 0.002077666998957284, + 0.0022798749996582046, + 0.0020769580005435273, + 0.0020142090070294216, + 0.002195708002545871, + 0.0026857079938054085, + 0.002412667003227398 + ] + }, + { + "forward": 0.005512957999599166, + "prefill": 0.0055994580034166574, + "decode_steps": [ + 0.0023019590007606894, + 0.002210624996223487, + 0.002138417010428384, + 0.002114499991876073, + 0.0026378750044386834, + 0.002072125003905967, + 0.0021409580076579005, + 0.0021550829987972975, + 0.0022037079907022417, + 0.002092041992000304, + 0.0020975420047761872, + 0.0026442499947734177, + 0.002125290993717499, + 0.002061666004010476, + 0.00206229199829977, + 0.0021129160013515502, + 0.002195374996517785, + 0.0021257920016068965, + 0.002658792000147514, + 0.0021428750042105094, + 0.002157291994080879, + 0.00208191700221505, + 0.002213415995356627, + 0.002237916996818967, + 0.002251291007269174, + 0.0027312500023981556, + 0.0021890000061830506, + 0.002086792010231875, + 0.0023000420042080805, + 0.0021891659998800606, + 0.0021436669921968132, + 0.002238790999399498 + ] + }, + { + "forward": 0.005346042002202012, + "prefill": 0.005748791998485103, + "decode_steps": [ + 0.0022477089951280504, + 0.0020585419988492504, + 0.0020025419944431633, + 0.0021270409924909472, + 0.0021705000108340755, + 0.002100875004543923, + 0.0021031670039519668, + 0.002168374994653277, + 0.0021867500036023557, + 0.00229350000154227, + 0.002075249998597428, + 0.002122249992680736, + 0.002117290991009213, + 0.00216658299905248, + 0.0021948750072624534, + 0.002186792000429705, + 0.0021235419990262017, + 0.002135582995833829, + 0.0020985000010114163, + 0.0022880420001456514, + 0.00231762501061894, + 0.0023872080055298284, + 0.0022349170030793175, + 0.002119208002113737, + 0.0021206250094110146, + 0.0021632089919876307, + 0.0021595839934889227, + 0.002184458004194312, + 0.0024803330015856773, + 0.002268875003210269, + 0.0021680420031771064, + 0.002119000011589378 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 5.356292000215035, + "p95": 7.5723347530583816, + "min": 5.258374992990866, + "max": 8.889541000826284, + "iqr": 0.4437709903868381, + "mean": 5.807512300089002, + "cv": 0.19111564898790773 + }, + "prefill_ms": { + "count": 10, + "median": 5.509187001734972, + "p95": 7.098491693614048, + "min": 5.227832996752113, + "max": 8.109791990136728, + "iqr": 0.3286354985903017, + "mean": 5.770349898375571, + "cv": 0.14629114104228705 + }, + "decode_ms_per_step": { + "count": 320, + "median": 2.1569794917013496, + "p95": 2.593108955625212, + "min": 1.9739580020541325, + "max": 3.47445800434798, + "iqr": 0.13741700604441576, + "mean": 2.2047749814191775, + "cv": 0.08176046446269078 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 3683.4619973794397, + "p95": 3728.5781079567864, + "min": 3391.172023328696, + "max": 3736.071966234977, + "iqr": 107.8523019045515, + "mean": 3632.2814058922436, + "cv": 0.03342945720670359 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 3410.953063230659, + "p95": 3453.278324180621, + "min": 3104.692461770015, + "max": 3456.8533654857015, + "iqr": 97.45494433309341, + "mean": 3358.6146627401818, + "cv": 0.03752973469465487 + }, + "memory": { + "peak_extra_bytes": 0, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "reference", + "batch": 8, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.010970999996061437, + "prefill": 0.007406583012198098, + "decode_steps": [ + 0.005263290993752889, + 0.005350624996935949, + 0.004298875006497838, + 0.00483525000163354, + 0.004487875004997477, + 0.004671707996749319, + 0.00460775000101421, + 0.00417258299421519, + 0.004244709009071812, + 0.004432832996826619, + 0.004726915998617187, + 0.004243874995154329, + 0.004609542011166923, + 0.004704250008217059, + 0.004415791991050355, + 0.00459374999627471, + 0.004545333998976275, + 0.004590209005982615, + 0.004131167006562464, + 0.0045146670017857105, + 0.0041022910008905455, + 0.004359750004368834, + 0.004596625003614463, + 0.004827249998925254, + 0.005585083999903873, + 0.00554554199334234, + 0.00578854201012291, + 0.005426083007478155, + 0.006075749988667667, + 0.009666042009484954, + 0.005890250002266839, + 0.005328500003088266 + ] + }, + { + "forward": 0.01060416700784117, + "prefill": 0.0069794169976376, + "decode_steps": [ + 0.004979584002285264, + 0.004807833000086248, + 0.004741625001770444, + 0.0047778330044820905, + 0.005011582994484343, + 0.0046028749929973856, + 0.004498875001445413, + 0.00445729200146161, + 0.004782582996995188, + 0.004629040995496325, + 0.004598540996084921, + 0.004949750000378117, + 0.004606541988323443, + 0.004768457991303876, + 0.004769416991621256, + 0.004725125007098541, + 0.004225500000757165, + 0.00460962500073947, + 0.004585792004945688, + 0.004541124988463707, + 0.004694292001659051, + 0.004558582993922755, + 0.004118249999010004, + 0.0045742920046905056, + 0.004673249990446493, + 0.004621291998773813, + 0.004266042000381276, + 0.004252875005477108, + 0.004587542003719136, + 0.004324834008002654, + 0.004667790999519639, + 0.005055625006207265 + ] + }, + { + "forward": 0.009221416999935172, + "prefill": 0.007224749992019497, + "decode_steps": [ + 0.0046940829925006256, + 0.004187083992292173, + 0.004440332995727658, + 0.004125500010559335, + 0.0045698750036535785, + 0.0046830000064801425, + 0.004543750008451752, + 0.004588958996464498, + 0.004084083993802778, + 0.004420458993990906, + 0.004525500000454485, + 0.0046392079966608435, + 0.004628249997040257, + 0.00446008300059475, + 0.0046611249999841675, + 0.004193083004793152, + 0.004611458993167616, + 0.004498665992286988, + 0.004540249996352941, + 0.0045917909883428365, + 0.004509209000389092, + 0.004129583001486026, + 0.004207416990539059, + 0.005045124999014661, + 0.004765417004819028, + 0.004691333000664599, + 0.0045946669997647405, + 0.004125042003579438, + 0.004688792003435083, + 0.004131209003389813, + 0.004627583999536, + 0.004573832993628457 + ] + }, + { + "forward": 0.011359374999301508, + "prefill": 0.007504332999815233, + "decode_steps": [ + 0.00543824999476783, + 0.004647624999051914, + 0.0051153329986846074, + 0.004927707996102981, + 0.004860957997152582, + 0.005160791988600977, + 0.005135083003551699, + 0.004920084000332281, + 0.0049700840027071536, + 0.005371874998672865, + 0.004981334001058713, + 0.004923500004224479, + 0.004739959011203609, + 0.004666709006414749, + 0.004345833003753796, + 0.004792334002559073, + 0.004646333007258363, + 0.005015500006265938, + 0.0047626250016037375, + 0.004639666003640741, + 0.004620791005436331, + 0.004728667001472786, + 0.004725999999209307, + 0.0046353750076377764, + 0.004647333989851177, + 0.00454662500123959, + 0.004154833004577085, + 0.0047010830021463335, + 0.0045624579943250865, + 0.004591959004756063, + 0.004675083007896319, + 0.004598917003022507 + ] + }, + { + "forward": 0.010516207999899052, + "prefill": 0.007485334004741162, + "decode_steps": [ + 0.00491283398878295, + 0.004832457998418249, + 0.004783374999533407, + 0.004804833006346598, + 0.004768375001731329, + 0.004584334004903212, + 0.004609708004863933, + 0.0045148749923100695, + 0.0044020829955115914, + 0.004794458000105806, + 0.004440499993506819, + 0.0047298749996116385, + 0.004757625007187016, + 0.004706334002548829, + 0.004725250008050352, + 0.004718166994280182, + 0.004747124999994412, + 0.0071667919983156025, + 0.005950333012151532, + 0.00693883300118614, + 0.0052071670070290565, + 0.00563079200219363, + 0.005202333995839581, + 0.004369082991615869, + 0.004553083999780938, + 0.005044291989179328, + 0.005179665997275151, + 0.0050073330057784915, + 0.004793333995621651, + 0.005090041013318114, + 0.004232291001244448, + 0.004803666000952944 + ] + }, + { + "forward": 0.009391958999913186, + "prefill": 0.0070899999991524965, + "decode_steps": [ + 0.005104708994622342, + 0.00510216700786259, + 0.004929500006255694, + 0.004505624994635582, + 0.004557499996735714, + 0.004815209002117626, + 0.004707166997832246, + 0.004373250005301088, + 0.004604999994626269, + 0.0045348330022534356, + 0.004400709003675729, + 0.004980792000424117, + 0.004996542003937066, + 0.005002874997444451, + 0.004873625002801418, + 0.004693417009548284, + 0.00450720899971202, + 0.004601459004334174, + 0.004126959000132047, + 0.004569333992549218, + 0.004595916994730942, + 0.004761875010444783, + 0.004345208988524973, + 0.004193999993731268, + 0.004604707995895296, + 0.00460345798637718, + 0.004585874994518235, + 0.004681332997279242, + 0.004371917006210424, + 0.005037166993133724, + 0.005005792001611553, + 0.004509207996306941 + ] + }, + { + "forward": 0.01102845800051, + "prefill": 0.007641167001565918, + "decode_steps": [ + 0.005631250009173527, + 0.005547917011426762, + 0.005061541989562102, + 0.004859874999965541, + 0.005055500005255453, + 0.0048934579972410575, + 0.00491504100500606, + 0.004841208996367641, + 0.004653333002352156, + 0.004835167012060992, + 0.005675082997186109, + 0.005416500003775582, + 0.005083541997009888, + 0.005127916010678746, + 0.004647083987947553, + 0.0043451250094221905, + 0.0048365000111516565, + 0.005189209012314677, + 0.004793999993125908, + 0.0046422500017797574, + 0.005279457996948622, + 0.004795500004547648, + 0.004655208002077416, + 0.004868124990025535, + 0.004533042010734789, + 0.004231041995808482, + 0.004655915996409021, + 0.00497429100505542, + 0.005116250002174638, + 0.004520082991803065, + 0.005131749989232048, + 0.004807666991837323 + ] + }, + { + "forward": 0.011078708004788496, + "prefill": 0.00769229099387303, + "decode_steps": [ + 0.005244165993644856, + 0.004843583999900147, + 0.004824083007406443, + 0.004915790996165015, + 0.004786083998624235, + 0.004252625003573485, + 0.004576332998112775, + 0.004460333002498373, + 0.004614834004314616, + 0.004517791996477172, + 0.004644333996111527, + 0.004690083995228633, + 0.004093083000043407, + 0.004563957991194911, + 0.0045110420032870024, + 0.005257333992631175, + 0.005446208000648767, + 0.004935167002258822, + 0.004707041007350199, + 0.004582999987178482, + 0.004894875004538335, + 0.004882749999524094, + 0.00476758299919311, + 0.005326667000190355, + 0.0049997500027529895, + 0.004582209003274329, + 0.004409041997860186, + 0.004683459002990276, + 0.0044975000055273995, + 0.0046895419945940375, + 0.004797834000783041, + 0.004674040988902561 + ] + }, + { + "forward": 0.00956058299925644, + "prefill": 0.007610417000250891, + "decode_steps": [ + 0.00463037499866914, + 0.0045813329925294966, + 0.004168333005509339, + 0.004571500001475215, + 0.003980916997534223, + 0.004691791997174732, + 0.004587791001540609, + 0.0047590000031050295, + 0.004482000003918074, + 0.004269500001100823, + 0.004405750005389564, + 0.004345041990745813, + 0.004601124994223937, + 0.004199584000161849, + 0.0045847500005038455, + 0.004667625005822629, + 0.004520415997831151, + 0.004491375002544373, + 0.004605415990226902, + 0.0041078749927692115, + 0.00419429199246224, + 0.0045188749936642125, + 0.0046422919986071065, + 0.004552166006760672, + 0.004532709004706703, + 0.004483083001105115, + 0.00457133399322629, + 0.004289832999347709, + 0.004488666992983781, + 0.004602458007866517, + 0.004522709001321346, + 0.004575500002829358 + ] + }, + { + "forward": 0.011402374992030673, + "prefill": 0.007147875003283843, + "decode_steps": [ + 0.0054990830103633925, + 0.005005790997529402, + 0.004689666995545849, + 0.004687708991696127, + 0.004298833999200724, + 0.0041145419963868335, + 0.004444874997716397, + 0.0051160830043954775, + 0.004484375007450581, + 0.004535166997811757, + 0.004526916993199848, + 0.004248333993018605, + 0.004514749991358258, + 0.004560875007882714, + 0.0047074169997358695, + 0.004459625008166768, + 0.00400624999019783, + 0.004576667008223012, + 0.0046417090052273124, + 0.004579832995659672, + 0.004276292005670257, + 0.004598125000484288, + 0.0045804169931216165, + 0.004063709013280459, + 0.004773415988893248, + 0.004655500000808388, + 0.0044243750016903505, + 0.004553125007078052, + 0.005265749990940094, + 0.004892708006082103, + 0.004717207993962802, + 0.004745999991428107 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 10.787583501951303, + "p95": 11.383024995302549, + "min": 9.221416999935172, + "max": 11.402374992030673, + "iqr": 1.266656254301779, + "mean": 10.513424999953713, + "cv": 0.07858334854376564 + }, + "prefill_ms": { + "count": 10, + "median": 7.44595850846963, + "p95": 7.66928519733483, + "min": 6.9794169976376, + "max": 7.69229099387303, + "iqr": 0.41680224967421964, + "mean": 7.378216700453777, + "cv": 0.034066934246634845 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.642271000193432, + "p95": 5.426691356842639, + "min": 3.980916997534223, + "max": 9.666042009484954, + "iqr": 0.31907301308820024, + "mean": 4.724284174926652, + "cv": 0.10131594025913673 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1701.349816650741, + "p95": 1784.3134962818924, + "min": 1610.0502353612796, + "max": 1787.3922478433817, + "iqr": 100.4574003655498, + "mean": 1695.6693251405443, + "cv": 0.03871317647744947 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1621.7391981316632, + "p95": 1696.3765533496407, + "min": 1537.6614369035951, + "max": 1697.2094312648208, + "iqr": 96.44829112162279, + "mean": 1616.6321598738893, + "cv": 0.03751887113634344 + }, + "memory": { + "peak_extra_bytes": 262144, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "auto", + "batch": 8, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.011942750003072433 + }, + { + "forward": 0.012293583000428043 + }, + { + "forward": 0.013951374989119358 + }, + { + "forward": 0.01220566700794734 + }, + { + "forward": 0.01240512500226032 + }, + { + "forward": 0.011946375001571141 + }, + { + "forward": 0.012835707995691337 + }, + { + "forward": 0.015025792003143579 + }, + { + "forward": 0.015463874995475635 + }, + { + "forward": 0.013816791994031519 + } + ], + "forward_ms": { + "count": 10, + "median": 12.620416498975828, + "p95": 15.26673764892621, + "min": 11.942750003072433, + "max": 15.463874995475635, + "iqr": 1.6900832342798822, + "mean": 13.18870419927407, + "cv": 0.09830783207250145 + }, + "memory": { + "peak_extra_bytes": 38535168, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "sdpa", + "batch": 8, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.01045045799401123, + "prefill": 0.01043495800695382, + "decode_steps": [ + 0.0025182920071529225, + 0.002412125002592802, + 0.0025384580076206475, + 0.002433834000839852, + 0.0028157500055385754, + 0.0024183339992305264, + 0.002522791997762397, + 0.002517040993552655, + 0.002420125005301088, + 0.0025540409988025203, + 0.002522708004107699, + 0.0023919589875731617, + 0.002480290990206413, + 0.0024451670033158734, + 0.0026887080020969734, + 0.0025275829975726083, + 0.002542041998822242, + 0.0023952920018928126, + 0.002520291003747843, + 0.0025140409998130053, + 0.0025161670055240393, + 0.0025613750040065497, + 0.0025108749978244305, + 0.00239799999690149, + 0.002506000004359521, + 0.0025054580037249252, + 0.0025087499961955473, + 0.0025689579924801365, + 0.0024535839911550283, + 0.0024178750027203932, + 0.002424916994641535, + 0.0025098330079345033 + ] + }, + { + "forward": 0.010204167003394105, + "prefill": 0.010818540991749614, + "decode_steps": [ + 0.002512125007342547, + 0.002480458002537489, + 0.002715082999202423, + 0.0025104999949689955, + 0.002602332999231294, + 0.0028135839966125786, + 0.002542834001360461, + 0.002545208000810817, + 0.0025268330064136535, + 0.0025210830062860623, + 0.002555875005782582, + 0.002408332991763018, + 0.002495708002243191, + 0.002478000009432435, + 0.0024966669880086556, + 0.002499540991266258, + 0.002555291008320637, + 0.002526333002606407, + 0.0024910840002121404, + 0.0024883750011213124, + 0.0023615420068381354, + 0.0025341669970657676, + 0.0025184999976772815, + 0.00268154100922402, + 0.002395125004113652, + 0.002503374998923391, + 0.0024933329987106845, + 0.002467915997840464, + 0.0027899590058950707, + 0.0023917910002637655, + 0.002523125003790483, + 0.0025110420101555064 + ] + }, + { + "forward": 0.011837583006126806, + "prefill": 0.01034991699270904, + "decode_steps": [ + 0.0037052080006105825, + 0.0025169999862555414, + 0.002675333002116531, + 0.0031037920125527307, + 0.002533457998652011, + 0.0025203749974025413, + 0.003057083988096565, + 0.0028699170070467517, + 0.003422291003516875, + 0.00262716700672172, + 0.002449666993925348, + 0.003003709003678523, + 0.0026244170003337786, + 0.0025837089924607426, + 0.0026851250004256144, + 0.0025990419962909073, + 0.002533457998652011, + 0.0030318339995574206, + 0.002432875000522472, + 0.002586417002021335, + 0.0024397920060437173, + 0.0033270839921897277, + 0.002649415997439064, + 0.0027988340007141232, + 0.002642541003297083, + 0.0028676250076387078, + 0.0025649580056779087, + 0.002498834001016803, + 0.0027144159976160154, + 0.0027632080018520355, + 0.002640708989929408, + 0.0025857500004349276 + ] + }, + { + "forward": 0.010252041989588179, + "prefill": 0.011087624996434897, + "decode_steps": [ + 0.002815874991938472, + 0.002567584000644274, + 0.0030767080024816096, + 0.002587958995718509, + 0.002472125008353032, + 0.002586750008049421, + 0.0027033330115955323, + 0.002499707989045419, + 0.0026604169979691505, + 0.002525709001929499, + 0.002478791997418739, + 0.002531749996705912, + 0.002426834005746059, + 0.002526708005461842, + 0.0027159999881405383, + 0.0025463749916525558, + 0.002555207989644259, + 0.0026198330015176907, + 0.002406540996162221, + 0.0025825420016190037, + 0.002675166993867606, + 0.0025744580052560195, + 0.002444874990032986, + 0.0026040840020868927, + 0.0025740840064827353, + 0.0025771670043468475, + 0.0026769999967655167, + 0.0027412089984863997, + 0.0027167920052306727, + 0.0026782909990288317, + 0.0025796670088311657, + 0.0025774999958230183 + ] + }, + { + "forward": 0.010301415997673757, + "prefill": 0.010380291001638398, + "decode_steps": [ + 0.0028241669933777303, + 0.0024695000029169023, + 0.0025386670022271574, + 0.002720082993619144, + 0.0028157920023659244, + 0.0031601250084349886, + 0.0030594590061809868, + 0.0027022499998565763, + 0.0026024999970104545, + 0.002629666996654123, + 0.0026091250038007274, + 0.0026616670074872673, + 0.002625250010169111, + 0.002534916013246402, + 0.0026363749930169433, + 0.002621459003421478, + 0.002634750009747222, + 0.0027138750010635704, + 0.002652458002557978, + 0.0025869580131256953, + 0.0025355830002808943, + 0.002516041000490077, + 0.00268641700677108, + 0.0026265000051353127, + 0.0026332909910706803, + 0.002488541998900473, + 0.002550750010414049, + 0.002643334009917453, + 0.002637583005707711, + 0.002679458004422486, + 0.0026514589990256354, + 0.002547875003074296 + ] + }, + { + "forward": 0.010575124993920326, + "prefill": 0.010529708000831306, + "decode_steps": [ + 0.0034338749974267557, + 0.0027606249932432547, + 0.0031376249971799552, + 0.002590834003058262, + 0.0025121670041698962, + 0.002551375000621192, + 0.002605250003398396, + 0.0025211669999407604, + 0.002562417008448392, + 0.0032079579978017136, + 0.002880291998735629, + 0.0030939999996917322, + 0.0029873750027036294, + 0.0029479579970939085, + 0.002499957990949042, + 0.002890540999942459, + 0.0024795419885776937, + 0.003069750004215166, + 0.002606167006888427, + 0.0025575829931767657, + 0.0032891250011743978, + 0.00249837500450667, + 0.002503875002730638, + 0.0025586669944459572, + 0.0034900000027846545, + 0.0025934580044122413, + 0.003070874998229556, + 0.002884041008655913, + 0.0029780829936498776, + 0.003063916999963112, + 0.002748499988229014, + 0.0037914580025244504 + ] + }, + { + "forward": 0.011153083003591746, + "prefill": 0.010919874999672174, + "decode_steps": [ + 0.0034235419880133122, + 0.003324291988974437, + 0.002756625006441027, + 0.003335457993671298, + 0.0032222500012721866, + 0.0026652090018615127, + 0.002718707997701131, + 0.002697375006391667, + 0.0026224579924019054, + 0.003229541005566716, + 0.0027331669989507645, + 0.002675708004971966, + 0.0028182079986436293, + 0.0027144159976160154, + 0.0032435000030091032, + 0.002678917007870041, + 0.0028942500066477805, + 0.0027541250019567087, + 0.002834332990460098, + 0.0027407500019762665, + 0.002680709003470838, + 0.002552874997491017, + 0.002700166995055042, + 0.002585000009275973, + 0.0032065830018837005, + 0.003240792008000426, + 0.002789999998640269, + 0.0032922090031206608, + 0.002907166999648325, + 0.0027739590004784986, + 0.0029104579880367965, + 0.0027750000008381903 + ] + }, + { + "forward": 0.01322700000309851, + "prefill": 0.012840374998631887, + "decode_steps": [ + 0.0029797500028507784, + 0.004731167005957104, + 0.0035016250039916486, + 0.002813375002006069, + 0.00290024999412708, + 0.0033062080037780106, + 0.004964749998180196, + 0.002766042001894675, + 0.00284866699075792, + 0.002975916999275796, + 0.0028729999903589487, + 0.0028198340005474165, + 0.0025977500044973567, + 0.0028007499931845814, + 0.003171499993186444, + 0.0028355829999782145, + 0.005433250000351109, + 0.003265167004428804, + 0.003198665988747962, + 0.0028017910080961883, + 0.00326045899419114, + 0.0038111249887151644, + 0.002941416998510249, + 0.0034952090063598007, + 0.005503374995896593, + 0.0035037080087931827, + 0.003495999990263954, + 0.0033618330053286627, + 0.003347833000589162, + 0.0029694580007344484, + 0.003361416995176114, + 0.0032331669935956597 + ] + }, + { + "forward": 0.012087750001228414, + "prefill": 0.011367999992216937, + "decode_steps": [ + 0.004465249992790632, + 0.004197958012809977, + 0.0035456250043353066, + 0.0033589999948162585, + 0.003364792006323114, + 0.0032045000116340816, + 0.0034775830135913566, + 0.0034002920001512393, + 0.005118875007610768, + 0.0036001249973196536, + 0.0034425000048941, + 0.0030830420000711456, + 0.0035968750016763806, + 0.003418124993913807, + 0.0033845839934656397, + 0.0030856669909553602, + 0.003637208996224217, + 0.0033700420026434585, + 0.0035540409880923107, + 0.003424208000069484, + 0.00359745800960809, + 0.004902209009742364, + 0.003525374995660968, + 0.0031353329977719113, + 0.0035726659989450127, + 0.0035538749943953007, + 0.003698291999171488, + 0.0034501670015743002, + 0.003682249996927567, + 0.0036781660019187257, + 0.004487083002459258, + 0.0036858329985989258 + ] + }, + { + "forward": 0.011657958006253466, + "prefill": 0.011359625001205131, + "decode_steps": [ + 0.003744166999240406, + 0.005227332992944866, + 0.0035149590112268925, + 0.0035856670001521707, + 0.0036464169970713556, + 0.00374287499289494, + 0.00357258299482055, + 0.0031386250047944486, + 0.003693874998134561, + 0.0037402500020107254, + 0.0036583329929271713, + 0.0035159169929102063, + 0.0036972919915569946, + 0.0036987499915994704, + 0.0036303329980000854, + 0.003672500009997748, + 0.0031516660092165694, + 0.0036552090023178607, + 0.0038286249910015613, + 0.00305333299911581, + 0.0031311250058934093, + 0.003644166004960425, + 0.004666207998525351, + 0.0036109999928157777, + 0.003622709002229385, + 0.0030701249925186858, + 0.0036675419978564605, + 0.0038562080007977784, + 0.0036194999993313104, + 0.0036351669987197965, + 0.003640750001068227, + 0.00491225000587292 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 10.864103998756036, + "p95": 12.714337502256965, + "min": 10.204167003394105, + "max": 13.22700000309851, + "iqr": 1.4540002594003454, + "mean": 11.174658199888654, + "cv": 0.09027466590754522 + }, + "prefill_ms": { + "count": 10, + "median": 10.869207995710894, + "p95": 12.177806245745158, + "min": 10.34991699270904, + "max": 12.840374998631887, + "iqr": 0.8329794945893809, + "mean": 11.00889149820432, + "cv": 0.06800932548124355 + }, + "decode_ms_per_step": { + "count": 320, + "median": 2.714749498409219, + "p95": 3.7465315494046085, + "min": 2.3615420068381354, + "max": 5.503374995896593, + "iqr": 0.717021495802328, + "mean": 2.9501074439849617, + "cv": 0.19069556343419664 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 2854.5821973468687, + "p95": 3182.001523302602, + "min": 2164.98800228333, + "max": 3197.5751985872544, + "iqr": 585.8764777692736, + "mean": 2765.411355710689, + "cv": 0.1407146764698185 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 2556.923520993403, + "p95": 2811.3909418693593, + "min": 1975.2311572275298, + "max": 2828.8660672272276, + "iqr": 495.6067965867346, + "mean": 2469.923472153202, + "cv": 0.13108003685527114 + }, + "memory": { + "peak_extra_bytes": 33554432, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "reference", + "batch": 8, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.019373583010747097, + "prefill": 0.01605712498712819, + "decode_steps": [ + 0.005242333994829096, + 0.00509487499948591, + 0.004896458995062858, + 0.005062500000349246, + 0.005764125002315268, + 0.005279582997900434, + 0.005029375010053627, + 0.005071417006547563, + 0.00480329200217966, + 0.00499620899790898, + 0.005131541998707689, + 0.004902290995232761, + 0.005122749993461184, + 0.004903708992060274, + 0.005095874992548488, + 0.005011582994484343, + 0.004934458003845066, + 0.004874791993643157, + 0.00494141700619366, + 0.004917625003145076, + 0.0048392079916084185, + 0.004601041990099475, + 0.0048826669953996316, + 0.005275499992421828, + 0.005382541014114395, + 0.005114375002449378, + 0.004792749998159707, + 0.005040707997977734, + 0.005118165994645096, + 0.00487766599690076, + 0.005060915995272808, + 0.005129500001203269 + ] + }, + { + "forward": 0.019835832994431257, + "prefill": 0.017202415998326614, + "decode_steps": [ + 0.005291875000693835, + 0.005063625008915551, + 0.005113624996738508, + 0.004881999993813224, + 0.004924875000142492, + 0.004819042005692609, + 0.004935499993734993, + 0.00514929200289771, + 0.005150290991878137, + 0.005030290994909592, + 0.004999791010050103, + 0.0050889589911093935, + 0.0049515840073581785, + 0.004895125006441958, + 0.00482683299924247, + 0.0050605419964995235, + 0.004901334003079683, + 0.004969250003341585, + 0.005016292008804157, + 0.005120250003528781, + 0.0046750839974265546, + 0.0049590419948799536, + 0.0050069580029230565, + 0.004888417010079138, + 0.004968749999534339, + 0.004916249992675148, + 0.004994250004529022, + 0.004729583000880666, + 0.004881000000750646, + 0.005152709010872059, + 0.005203458000323735, + 0.005299416996422224 + ] + }, + { + "forward": 0.019153208995703608, + "prefill": 0.01633899999433197, + "decode_steps": [ + 0.005227916000876576, + 0.005209833005210385, + 0.004981125006452203, + 0.004989124994608574, + 0.005093583007692359, + 0.004935624994686805, + 0.004961291997460648, + 0.004980541008990258, + 0.004683416991611011, + 0.005012750014429912, + 0.004935374992783181, + 0.004853582999203354, + 0.004918708000332117, + 0.004950417001964524, + 0.004789832993992604, + 0.004832042002817616, + 0.004790708000655286, + 0.0048393750039394945, + 0.006647832997259684, + 0.007606833009049296, + 0.01177875000576023, + 0.007342666998738423, + 0.011199875007150695, + 0.01577608300431166, + 0.010268000012729317, + 0.006898082996485755, + 0.006710165995173156, + 0.005974416999379173, + 0.00593004199618008, + 0.0059554999897954985, + 0.00722108299669344, + 0.006634083008975722 + ] + }, + { + "forward": 0.02007662500545848, + "prefill": 0.01683975000923965, + "decode_steps": [ + 0.006396041993866675, + 0.005490875002578832, + 0.005750791009631939, + 0.0051313330041011795, + 0.005208292001043446, + 0.005072750005638227, + 0.005206917005125433, + 0.005180249994737096, + 0.005220582999754697, + 0.0051844999979948625, + 0.005206625006394461, + 0.005215999990468845, + 0.005442249996121973, + 0.004928249996737577, + 0.005089541999041103, + 0.005139499990036711, + 0.005241708000539802, + 0.005105124990222976, + 0.005011790999560617, + 0.005226499997661449, + 0.00536012499651406, + 0.005085875003715046, + 0.005388291989220306, + 0.005233666001004167, + 0.005226291003054939, + 0.00506158301141113, + 0.005391041006078012, + 0.00530741699913051, + 0.005104374999064021, + 0.0053269579948391765, + 0.005273750008200295, + 0.005336707996320911 + ] + }, + { + "forward": 0.019804790994385257, + "prefill": 0.016925583011470735, + "decode_steps": [ + 0.00669454199669417, + 0.005742958004702814, + 0.006123750004917383, + 0.004933375006658025, + 0.005452750003314577, + 0.005647375000989996, + 0.004919749990222044, + 0.00516924999828916, + 0.00605658401036635, + 0.00525299999571871, + 0.005404458002885804, + 0.0058141670015174896, + 0.005023082994739525, + 0.0051671670080395415, + 0.005760832995292731, + 0.004895666992524639, + 0.004960416001267731, + 0.0058580830082064494, + 0.005149541000719182, + 0.005205333000048995, + 0.0070172499981708825, + 0.006099916994571686, + 0.006548624995048158, + 0.007120291003957391, + 0.005680832997313701, + 0.005604167003184557, + 0.0050612920022103935, + 0.005168625008082017, + 0.005269957997370511, + 0.004875208003795706, + 0.005286666011670604, + 0.005324875004589558 + ] + }, + { + "forward": 0.01891887500823941, + "prefill": 0.016688290997990407, + "decode_steps": [ + 0.006222625001100823, + 0.005299707991071045, + 0.00601637500221841, + 0.005791916002635844, + 0.005137583008036017, + 0.005782874999567866, + 0.005187332993955351, + 0.00530345800507348, + 0.00552945799427107, + 0.004998959004296921, + 0.005162834000657313, + 0.005418290995294228, + 0.005348000006051734, + 0.0051223329937784, + 0.00531766600033734, + 0.004777166992425919, + 0.005273875009152107, + 0.005331667009158991, + 0.005551583002670668, + 0.005089124999358319, + 0.005599249998340383, + 0.005072333005955443, + 0.005244540996500291, + 0.0051919170073233545, + 0.006188125000335276, + 0.0057462910044705495, + 0.005718375003198162, + 0.0056977920030476525, + 0.005900290998397395, + 0.005306958992150612, + 0.005572000009124167, + 0.005532750001293607 + ] + }, + { + "forward": 0.019886833004420623, + "prefill": 0.01666583400219679, + "decode_steps": [ + 0.006593500002054498, + 0.005747707997215912, + 0.005605625003227033, + 0.005545791995245963, + 0.005934166998486035, + 0.005985124997096136, + 0.005522292005480267, + 0.005542915998375975, + 0.005627417005598545, + 0.005810583999846131, + 0.006038749997969717, + 0.005486916998052038, + 0.005808791000163183, + 0.006317290986771695, + 0.005846000000019558, + 0.005634417000692338, + 0.0054709590040147305, + 0.005747334012994543, + 0.006206250007380731, + 0.005887958002858795, + 0.0056097920023603365, + 0.006186541999340989, + 0.0051705829973798245, + 0.005212750009377487, + 0.0056537500058766454, + 0.006702832994051278, + 0.008808624988887459, + 0.008861665992299095, + 0.008456915995338932, + 0.006423875005566515, + 0.006926000001840293, + 0.0069555840018438175 + ] + }, + { + "forward": 0.02471979201072827, + "prefill": 0.0196629580023, + "decode_steps": [ + 0.007321500001125969, + 0.007032541994703934, + 0.0071892080013640225, + 0.009199499996611848, + 0.006751875000190921, + 0.007870917004765943, + 0.005932291998760775, + 0.005996708991006017, + 0.005795541001134552, + 0.006057334001525305, + 0.006122417005826719, + 0.005712290992960334, + 0.005862915990292095, + 0.007016458999714814, + 0.006725500003085472, + 0.006908249997650273, + 0.006339916988508776, + 0.006297457992332056, + 0.006123458995716646, + 0.0063148749904939905, + 0.007655583991436288, + 0.005645625002216548, + 0.0061225419922266155, + 0.005813249998027459, + 0.006321458000456914, + 0.007331207991228439, + 0.006227874997421168, + 0.006035750004230067, + 0.006067916998290457, + 0.005961707996902987, + 0.005925292003666982, + 0.006152834001113661 + ] + }, + { + "forward": 0.023776749992975965, + "prefill": 0.027623749992926605, + "decode_steps": [ + 0.008269917001598515, + 0.007123250004951842, + 0.006690458001685329, + 0.008237166999606416, + 0.006643166998401284, + 0.0070724999968660995, + 0.006795041001169011, + 0.007112374994903803, + 0.007350166997639462, + 0.007259082995005883, + 0.008197292001568712, + 0.00660612499632407, + 0.0071115419996203855, + 0.007020417004241608, + 0.010534791988902725, + 0.006507917001727037, + 0.007897208008216694, + 0.006846040996606462, + 0.01029874999949243, + 0.006926749992999248, + 0.007815958000719547, + 0.006893291007145308, + 0.006905083006131463, + 0.00646483400487341, + 0.006090958006097935, + 0.006250624996027909, + 0.006176124996272847, + 0.0075242500024614856, + 0.005649333004839718, + 0.006183959005284123, + 0.00617454199527856, + 0.0061360419931588694 + ] + }, + { + "forward": 0.024262874998385087, + "prefill": 0.01929912499326747, + "decode_steps": [ + 0.006828084005974233, + 0.006722667007124983, + 0.008543333999114111, + 0.006411415990442038, + 0.00821400000131689, + 0.006629124996834435, + 0.006789084000047296, + 0.00695304200053215, + 0.006589583004824817, + 0.006319708001683466, + 0.006303000001935288, + 0.00660424999659881, + 0.006604332986171357, + 0.006501583993667737, + 0.0061879999993834645, + 0.0062807080103084445, + 0.00676595899858512, + 0.006778708004276268, + 0.0065657090017339215, + 0.006035332989995368, + 0.007447708005202003, + 0.007564749990706332, + 0.006758792005712166, + 0.00816875000600703, + 0.021420375007437542, + 0.011112083011539653, + 0.009630333006498404, + 0.008300458008307032, + 0.007756666003842838, + 0.007471749995602295, + 0.007702957998844795, + 0.00809737500094343 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 19.86133299942594, + "p95": 24.514179355173837, + "min": 18.91887500823941, + "max": 24.71979201072827, + "iqr": 3.370333739439957, + "mean": 20.980916601547506, + "cv": 0.10944186894580267 + }, + "prefill_ms": { + "count": 10, + "median": 16.882666510355193, + "p95": 24.041393597144626, + "min": 16.05712498712819, + "max": 27.623749992926605, + "iqr": 2.103499493387062, + "mean": 18.330383198917843, + "cv": 0.18998414946256978 + }, + "decode_ms_per_step": { + "count": 320, + "median": 5.604896003205795, + "p95": 8.23880449970602, + "min": 4.601041990099475, + "max": 21.420375007437542, + "iqr": 1.4479585042863619, + "mean": 6.0226232876175345, + "cv": 0.25837179621214423 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1366.5961589703343, + "p95": 1595.420498909384, + "min": 1040.398236677578, + "max": 1601.3481947600765, + "iqr": 269.93210392398805, + "mean": 1354.9630308305625, + "cv": 0.14459865270646144 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1253.832163296183, + "p95": 1445.1114742059206, + "min": 964.7317074579282, + "max": 1445.774613331825, + "iqr": 237.32785962045864, + "mean": 1236.359191332677, + "cv": 0.14134645478461721 + }, + "memory": { + "peak_extra_bytes": 1074003968, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "auto", + "batch": 8, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.025245708995498717 + }, + { + "forward": 0.029381499989540316 + }, + { + "forward": 0.025163874990539625 + }, + { + "forward": 0.02605266700265929 + }, + { + "forward": 0.02831541698833462 + }, + { + "forward": 0.030559250008082017 + }, + { + "forward": 0.03552183300780598 + }, + { + "forward": 0.03419408299669158 + }, + { + "forward": 0.0316725410084473 + }, + { + "forward": 0.03153762500733137 + } + ], + "forward_ms": { + "count": 10, + "median": 29.970374998811167, + "p95": 34.9243455028045, + "min": 25.163874990539625, + "max": 35.52183300780598, + "iqr": 5.020457509090193, + "mean": 29.76444999949308, + "cv": 0.12154638256295808 + }, + "memory": { + "peak_extra_bytes": 1116995584, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "sdpa", + "batch": 8, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.021161832992220297, + "prefill": 0.021441125005367212, + "decode_steps": [ + 0.004639499995391816, + 0.0044365409994497895, + 0.004506958997808397, + 0.004429791995789856, + 0.00438120799663011, + 0.004903375011053868, + 0.00398999999742955, + 0.003945624994230457, + 0.0037121249915799126, + 0.003645959004643373, + 0.0036970840010326356, + 0.004529292011284269, + 0.0043512499978533015, + 0.0036869169998681173, + 0.003648999991128221, + 0.00434083299478516, + 0.0035970000026281923, + 0.003633791988249868, + 0.004424625003593974, + 0.004550625002593733, + 0.0036187080113450065, + 0.0036223750066710636, + 0.003649124992080033, + 0.004381833001389168, + 0.0038889999996172264, + 0.0034618329955264926, + 0.0043632080050883815, + 0.0036497500113910064, + 0.003686458003357984, + 0.0036692919966299087, + 0.003710832999786362, + 0.0038251660007517785 + ] + }, + { + "forward": 0.020123209003941156, + "prefill": 0.020431833996553905, + "decode_steps": [ + 0.0034137920010834932, + 0.0027597500011324883, + 0.002757416994427331, + 0.0026217499980702996, + 0.0027755409973906353, + 0.002665833992068656, + 0.0027137500001117587, + 0.0026574999938020483, + 0.0027245420060353354, + 0.002897375001339242, + 0.0027774579939432442, + 0.0029085419955663383, + 0.002823667004122399, + 0.0029706250061281025, + 0.002862749999621883, + 0.002842749992851168, + 0.003264792001573369, + 0.00451391599199269, + 0.0028382500022416934, + 0.003245957996114157, + 0.0028665830031968653, + 0.003674334002425894, + 0.003056124987779185, + 0.0030553749966202304, + 0.0029202909936429933, + 0.003048250000574626, + 0.004804665994015522, + 0.002922125000623055, + 0.0029519579984480515, + 0.002886125002987683, + 0.004367457993794233, + 0.0029990420007379726 + ] + }, + { + "forward": 0.022163667003042065, + "prefill": 0.022957334003876895, + "decode_steps": [ + 0.0029207500047050416, + 0.0027645000081975013, + 0.0028468329983297735, + 0.0027164170023752376, + 0.00359241699334234, + 0.0027698750054696575, + 0.0027285420073894784, + 0.0027375839999876916, + 0.0027821249968837947, + 0.0028336670075077564, + 0.002783125004498288, + 0.0028487919917097315, + 0.00283816599403508, + 0.002820709007210098, + 0.0027610830002231523, + 0.0028159159992355853, + 0.002877000006265007, + 0.0028357499977573752, + 0.0029586249875137582, + 0.0028830000082962215, + 0.0029572080093203112, + 0.003564582992112264, + 0.003478500002529472, + 0.0032123330020112917, + 0.0031884160125628114, + 0.004001833003712818, + 0.00337129199760966, + 0.003388708006241359, + 0.0032867920090211555, + 0.0034262499975739047, + 0.003605916994274594, + 0.0035477089986670762 + ] + }, + { + "forward": 0.022594541995204054, + "prefill": 0.022719625005265698, + "decode_steps": [ + 0.006956582990824245, + 0.004925583998556249, + 0.004569750002701767, + 0.004594833997543901, + 0.004465667007025331, + 0.0037853749963687733, + 0.003772583993850276, + 0.004384458996355534, + 0.003722458000993356, + 0.003741874999832362, + 0.003629583996371366, + 0.0036669580003945157, + 0.0037820830038981512, + 0.00378233399533201, + 0.003773665986955166, + 0.0037332500069169328, + 0.0038072500028647482, + 0.0037924999924143776, + 0.003804791995207779, + 0.0035708340001292527, + 0.003476125013548881, + 0.0035157079983036965, + 0.003302666009403765, + 0.003311042004497722, + 0.003193958997144364, + 0.0031564580131089315, + 0.0032330420071957633, + 0.0029897079948568717, + 0.002966750005725771, + 0.004579084008582868, + 0.0029515409987652674, + 0.003068082995014265 + ] + }, + { + "forward": 0.026281958998879418, + "prefill": 0.02856995799811557, + "decode_steps": [ + 0.0035453339951345697, + 0.0029360420012380928, + 0.0073050000064540654, + 0.003661125010694377, + 0.0031959579937392846, + 0.0029062499961582944, + 0.006275749998167157, + 0.0036945829924661666, + 0.00326116599899251, + 0.003126875002635643, + 0.005697166998288594, + 0.0034022079926216975, + 0.0033692500001052395, + 0.003382916998816654, + 0.003928042002371512, + 0.00383408299239818, + 0.0035615830129245296, + 0.0033361670066369697, + 0.006812749998061918, + 0.003515249991323799, + 0.0031831250089453533, + 0.006764833000488579, + 0.003545791987562552, + 0.003288249994511716, + 0.003231666996725835, + 0.007020082994131371, + 0.003599040996050462, + 0.003396833999431692, + 0.003381749993423, + 0.005623999997624196, + 0.003446707996772602, + 0.0034433330001775175 + ] + }, + { + "forward": 0.026020458011771552, + "prefill": 0.02541929201106541, + "decode_steps": [ + 0.004067249989020638, + 0.0034474580024834722, + 0.0037271670007612556, + 0.0034425830090185627, + 0.004414334005559795, + 0.0036036660021636635, + 0.003593125002225861, + 0.003363333991728723, + 0.003973832994233817, + 0.003688791999593377, + 0.0035715840058401227, + 0.0033008330065058544, + 0.0037964579969411716, + 0.005057917005615309, + 0.0035901660012314096, + 0.003433416990446858, + 0.0035264589969301596, + 0.0038036250043660402, + 0.0035009169951081276, + 0.0036660000041592866, + 0.0035417910112300888, + 0.004292124998755753, + 0.003619166003772989, + 0.003687042000819929, + 0.003674250008771196, + 0.0041049580031540245, + 0.003749499999685213, + 0.0036551249941112474, + 0.003823582999757491, + 0.0038789590034866706, + 0.0037053749983897433, + 0.0036975830007577315 + ] + }, + { + "forward": 0.029564083000877872, + "prefill": 0.029733584000496194, + "decode_steps": [ + 0.0058036670088768005, + 0.005760915999417193, + 0.0052572499989764765, + 0.005382832998293452, + 0.004725666003650986, + 0.004782082993187942, + 0.004506708006374538, + 0.00458308300585486, + 0.005341334006516263, + 0.004806917000678368, + 0.005285708990413696, + 0.004990917004761286, + 0.004652792005799711, + 0.0047567080036969855, + 0.005178166989935562, + 0.004530249992967583, + 0.0046352499921340495, + 0.004739583004266024, + 0.004957707991707139, + 0.004530707999947481, + 0.005310250009642914, + 0.004841000001761131, + 0.00475337500392925, + 0.004368082998553291, + 0.005119583001942374, + 0.004551750011160038, + 0.00432587499381043, + 0.0045975419925525784, + 0.0045512499928008765, + 0.004343082997365855, + 0.004636208002921194, + 0.006178499999805354 + ] + }, + { + "forward": 0.028939583004103042, + "prefill": 0.02871874999254942, + "decode_steps": [ + 0.004927040994516574, + 0.0038087090069893748, + 0.003688374999910593, + 0.0040032920078374445, + 0.003920207993360236, + 0.0037882920005358756, + 0.003660000002128072, + 0.0042070409981533885, + 0.0038995830109342933, + 0.004040957996039651, + 0.003720000007888302, + 0.004046417001518421, + 0.003910416999133304, + 0.0038605830050073564, + 0.003763166008866392, + 0.005112541999551468, + 0.0039553340029669926, + 0.0038627500034635887, + 0.0038625000015599653, + 0.004000708009698428, + 0.00410195899894461, + 0.004049208000651561, + 0.004182999997283332, + 0.005084292002720758, + 0.00448925000091549, + 0.0043402499868534505, + 0.004435374998138286, + 0.0041409580007893965, + 0.004341249994467944, + 0.0047559579979861155, + 0.0044272499944781885, + 0.004349791997810826 + ] + }, + { + "forward": 0.029363249996094964, + "prefill": 0.02576187500380911, + "decode_steps": [ + 0.0050612920022103935, + 0.003881874989019707, + 0.003661333001218736, + 0.003918000002158806, + 0.003932291001547128, + 0.003810624999459833, + 0.0036157919967081398, + 0.003918500005966052, + 0.003980250010499731, + 0.0038653339870506898, + 0.0037644580006599426, + 0.0053495830070460215, + 0.003911583000444807, + 0.0038869169948156923, + 0.003911459003575146, + 0.005279625009279698, + 0.004034333003801294, + 0.0037305829901015386, + 0.003910291998181492, + 0.003971915997681208, + 0.003919750000932254, + 0.003921875002561137, + 0.004080333004822023, + 0.004163292003795505, + 0.003894834007951431, + 0.004562375004752539, + 0.006121041995356791, + 0.004389749999972992, + 0.004455792004591785, + 0.00627604199689813, + 0.005178125007660128, + 0.0050355409912299365 + ] + }, + { + "forward": 0.028560042002936825, + "prefill": 0.028623833000892773, + "decode_steps": [ + 0.006371458002831787, + 0.005684833013219759, + 0.005557666008826345, + 0.0043744580034399405, + 0.00427224999293685, + 0.005055125002400018, + 0.004109999994398095, + 0.004178750008577481, + 0.00456083299650345, + 0.008265374999609776, + 0.006625207999604754, + 0.006353457996738143, + 0.005697542001144029, + 0.005610583000816405, + 0.005998749998980202, + 0.004406417006975971, + 0.005184833993553184, + 0.0044720000005327165, + 0.005197375008719973, + 0.005144082999322563, + 0.0042114160023629665, + 0.004903832988929935, + 0.005285209001158364, + 0.005439208005554974, + 0.004704333987319842, + 0.005043791999923997, + 0.004180625008302741, + 0.00418179199914448, + 0.004241082991939038, + 0.004363333995570429, + 0.0048866249999264255, + 0.00409279199084267 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 26.151208505325485, + "p95": 29.473708148725564, + "min": 20.123209003941156, + "max": 29.564083000877872, + "iqr": 6.573312002728926, + "mean": 25.477262600907125, + "cv": 0.14380947385815562 + }, + "prefill_ms": { + "count": 10, + "median": 25.59058350743726, + "p95": 29.276908696920145, + "min": 20.431833996553905, + "max": 29.733584000496194, + "iqr": 5.831311995279975, + "mean": 25.43772100179922, + "cv": 0.13354363274110764 + }, + "decode_ms_per_step": { + "count": 320, + "median": 3.8296244965749793, + "p95": 5.700710701057688, + "min": 2.6217499980702996, + "max": 8.265374999609776, + "iqr": 1.066479249857366, + "mean": 4.02362720947167, + "cv": 0.23048487330900402 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1975.041278065014, + "p95": 2603.116335297052, + "min": 1573.8829760574781, + "max": 2608.4010451509293, + "iqr": 244.9922187128618, + "mean": 2038.5322809469387, + "cv": 0.1699858595441661 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1657.0220168688047, + "p95": 2134.25939193497, + "min": 1338.360032371696, + "max": 2150.896750670253, + "iqr": 193.5766653297369, + "mean": 1697.3671585801671, + "cv": 0.15951947287125046 + }, + "memory": { + "peak_extra_bytes": 1107296256, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "reference", + "batch": 8, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.04625266599759925, + "prefill": 0.04153437500644941, + "decode_steps": [ + 0.011256333003984764, + 0.009413166000740603, + 0.008833250001771376, + 0.008010500008822419, + 0.007709291996434331, + 0.007446540999808349, + 0.007034958005533554, + 0.0067755409982055426, + 0.006622708999202587, + 0.006029583004419692, + 0.005840750003699213, + 0.005656499997712672, + 0.005436874998849817, + 0.005714250000892207, + 0.005787250003777444, + 0.0057562080037314445, + 0.005298957999912091, + 0.005444540991447866, + 0.0054235419956967235, + 0.0054461670079035684, + 0.005416457992396317, + 0.0055006670008879155, + 0.005662124996888451, + 0.005522834006114863, + 0.005409874996985309, + 0.005256958000245504, + 0.0053526250121649355, + 0.005486249996465631, + 0.00541187499766238, + 0.00518329199985601, + 0.0053405419894261286, + 0.006249500002013519 + ] + }, + { + "forward": 0.05083949999243487, + "prefill": 0.04519620900100563, + "decode_steps": [ + 0.009790791998966597, + 0.008585042000049725, + 0.007666040997719392, + 0.006931583993718959, + 0.007161790999816731, + 0.006678000005194917, + 0.006574500002898276, + 0.006581249996088445, + 0.006520667011500336, + 0.006185207996168174, + 0.006173667003167793, + 0.005912958993576467, + 0.006150250002974644, + 0.007557208999060094, + 0.006566916999872774, + 0.006244375006644987, + 0.005780708001111634, + 0.0057958749966928735, + 0.00740724999923259, + 0.005615292000584304, + 0.005604124991805293, + 0.006694666997645982, + 0.005524624997633509, + 0.005409958001109771, + 0.0050827920058509335, + 0.005680749993189238, + 0.0056101670052157715, + 0.005233584000961855, + 0.005484124994836748, + 0.005601958007900976, + 0.005295291994116269, + 0.005707458010874689 + ] + }, + { + "forward": 0.04825616600282956, + "prefill": 0.04445241700159386, + "decode_steps": [ + 0.0063739170000189915, + 0.006033249999745749, + 0.00662666599964723, + 0.0057332079886691645, + 0.006642833002842963, + 0.005443540998385288, + 0.00560645799851045, + 0.007108250007149763, + 0.005466499991598539, + 0.005552000002353452, + 0.005101458998979069, + 0.005588874992099591, + 0.0054031250037951395, + 0.00521245900017675, + 0.0057310410047648475, + 0.006065291992854327, + 0.006051290998584591, + 0.005762750006397255, + 0.005608125007711351, + 0.005484084002091549, + 0.005485374989802949, + 0.005722999994759448, + 0.005685458003426902, + 0.005587084000580944, + 0.0054897920053917915, + 0.0055819580011302605, + 0.005351499989046715, + 0.005510167000466026, + 0.00571937499626074, + 0.00808916699315887, + 0.0059516670007724315, + 0.005735750004532747 + ] + }, + { + "forward": 0.04642858299484942, + "prefill": 0.04125120799290016, + "decode_steps": [ + 0.008414083000388928, + 0.007987541001057252, + 0.0070465829921886325, + 0.007165667004301213, + 0.006674000003840774, + 0.006497125010355376, + 0.006579124994459562, + 0.006215582994627766, + 0.006685125001240522, + 0.00615529200877063, + 0.006064583008992486, + 0.006025166992913, + 0.0062731250072829425, + 0.006174415990244597, + 0.00592366598721128, + 0.005934457993134856, + 0.006734291993780062, + 0.006949874994461425, + 0.005829541987623088, + 0.005913500004680827, + 0.005689583995263092, + 0.005817292010760866, + 0.00883662499836646, + 0.01216195899178274, + 0.008092749994830228, + 0.008948750008130446, + 0.008314499995321967, + 0.006862624999484979, + 0.007427207994624041, + 0.008878624998033047, + 0.009819916987908073, + 0.008079040999291465 + ] + }, + { + "forward": 0.060622582997893915, + "prefill": 0.055122708989074454, + "decode_steps": [ + 0.012058874999638647, + 0.00951179199910257, + 0.007649999999557622, + 0.011519042003783397, + 0.0073191249975934625, + 0.006851999991340563, + 0.006748708998202346, + 0.00675379199674353, + 0.006186457991134375, + 0.006007000003592111, + 0.006122582999523729, + 0.006077582991565578, + 0.006095834003644995, + 0.006298124993918464, + 0.006453416994190775, + 0.006193458000780083, + 0.005999750006594695, + 0.005837292002979666, + 0.006128667009761557, + 0.006638165999902412, + 0.006473874993389472, + 0.006293833997915499, + 0.00618387499707751, + 0.005901833006646484, + 0.005943667012616061, + 0.006482249998953193, + 0.006034916004864499, + 0.006390959009877406, + 0.006049167001037858, + 0.005964875002973713, + 0.006856125008198433, + 0.0069696669961558655 + ] + }, + { + "forward": 0.04670350000378676, + "prefill": 0.04473316599614918, + "decode_steps": [ + 0.008659709012135863, + 0.006719208002323285, + 0.007224250002764165, + 0.006686541993985884, + 0.00659845799964387, + 0.00636145900352858, + 0.006743166988599114, + 0.006753584006219171, + 0.006464958001743071, + 0.006570957993972115, + 0.006444875005399808, + 0.006385582993971184, + 0.006692332986858673, + 0.006362041996908374, + 0.006430666995584033, + 0.006543916999362409, + 0.00627845800772775, + 0.0068325000029290095, + 0.007540000005974434, + 0.006713667011354119, + 0.006683874991722405, + 0.006867750009405427, + 0.0068826249917037785, + 0.006679958998574875, + 0.006883834008476697, + 0.006557457993039861, + 0.006568583004991524, + 0.006649333008681424, + 0.006644749999395572, + 0.006731250003213063, + 0.006618875006097369, + 0.006734124996000901 + ] + }, + { + "forward": 0.05885191701236181, + "prefill": 0.057075541000813246, + "decode_steps": [ + 0.019318332997499965, + 0.013695540998014621, + 0.011477417006972246, + 0.010003082992625423, + 0.009699957998236641, + 0.008770750006078742, + 0.009409334001247771, + 0.007863792008720338, + 0.007902208002633415, + 0.008933666002121754, + 0.007585125000332482, + 0.008023959002457559, + 0.007749709009658545, + 0.007590332999825478, + 0.007296332987607457, + 0.007226332993013784, + 0.00833045800391119, + 0.0072244999901158735, + 0.00807925000844989, + 0.007193750003352761, + 0.007767124989186414, + 0.008586415991885588, + 0.008145792002324015, + 0.007509333998314105, + 0.007793915996444412, + 0.007463832997018471, + 0.008177041003364138, + 0.007795249999617226, + 0.00784274999750778, + 0.007480583997676149, + 0.00775699999940116, + 0.007532458010246046 + ] + }, + { + "forward": 0.058924249999108724, + "prefill": 0.05473299999721348, + "decode_steps": [ + 0.01579887499974575, + 0.012012874998617917, + 0.009614040987798944, + 0.00884487500297837, + 0.008381999999983236, + 0.00893141599954106, + 0.009422709001228213, + 0.008003249997273088, + 0.008088290996965952, + 0.008149625005898997, + 0.007827374996850267, + 0.00804483299725689, + 0.007716334002907388, + 0.008802917000139132, + 0.008185541999409907, + 0.00805495800159406, + 0.007914583999081515, + 0.0084624589944724, + 0.008914167003240436, + 0.00783029200101737, + 0.00813533399195876, + 0.007652959000552073, + 0.00775679100479465, + 0.007945249992189929, + 0.00785324999014847, + 0.008067832997767255, + 0.007744083006400615, + 0.007968874997459352, + 0.007710000005317852, + 0.008346208007424138, + 0.008464667000225745, + 0.008627791990875266 + ] + }, + { + "forward": 0.05294333299389109, + "prefill": 0.0517729590064846, + "decode_steps": [ + 0.009700249996967614, + 0.00822387498919852, + 0.00809754199872259, + 0.008163625010638498, + 0.008319833999848925, + 0.007838916993932799, + 0.008941791005781852, + 0.008193333997041918, + 0.008150958004989661, + 0.007616332994075492, + 0.00816412499989383, + 0.008074249999481253, + 0.0078205410100054, + 0.0075880840013269335, + 0.008976541997981258, + 0.007928499995614402, + 0.0074345839966554195, + 0.007953291991725564, + 0.008099124999716878, + 0.008133124996675178, + 0.008631125005194917, + 0.007776708007440902, + 0.009479583008214831, + 0.007845625004847534, + 0.007825791006325744, + 0.007109165992005728, + 0.008796290989266708, + 0.007564207990071736, + 0.007793957993271761, + 0.007659999988391064, + 0.009344958001747727, + 0.0075062910036649555 + ] + }, + { + "forward": 0.053050500006065704, + "prefill": 0.04765270900679752, + "decode_steps": [ + 0.010857667002710514, + 0.008081291991402395, + 0.008631709002656862, + 0.008405665997997858, + 0.007932125008665025, + 0.00775324999995064, + 0.007137957989471033, + 0.007547207991592586, + 0.007553667004685849, + 0.007385624994640239, + 0.007598959011374973, + 0.007407750003039837, + 0.00818670800072141, + 0.007265542008099146, + 0.00764016700850334, + 0.007512541997130029, + 0.007531541996286251, + 0.007296333002159372, + 0.007234667005832307, + 0.006562667011166923, + 0.006990916997892782, + 0.00835979200201109, + 0.007910375003120862, + 0.00755454100726638, + 0.00689754199993331, + 0.007566291998955421, + 0.007324041987885721, + 0.00733454100554809, + 0.00730854099674616, + 0.0081257919955533, + 0.006683250001515262, + 0.0070904169988352805 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 51.89141649316298, + "p95": 59.85833314844058, + "min": 46.25266599759925, + "max": 60.622582997893915, + "iqr": 10.309896257240325, + "mean": 52.28729980008211, + "cv": 0.10627384195296037 + }, + "prefill_ms": { + "count": 10, + "median": 46.424459003901575, + "p95": 56.19676659553079, + "min": 41.25120799290016, + "max": 57.075541000813246, + "iqr": 9.470385499298573, + "mean": 48.352429299848154, + "cv": 0.12138126080550303 + }, + "decode_ms_per_step": { + "count": 320, + "median": 7.012937501713168, + "p95": 9.51690444853739, + "min": 5.0827920058509335, + "max": 19.318332997499965, + "iqr": 1.899031256471062, + "mean": 7.2238828279296285, + "cv": 0.2207558721667563 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1143.2715991354041, + "p95": 1328.8433468522926, + "min": 923.4365383581116, + "max": 1372.6142873016202, + "iqr": 245.767115532223, + "mean": 1126.6165160340431, + "cv": 0.13707858284535793 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 940.2483797363249, + "p95": 1085.1488361333468, + "min": 765.7772427304541, + "max": 1108.4274371275997, + "iqr": 186.81560514210332, + "mean": 929.7956611036683, + "cv": 0.12798629932511907 + }, + "memory": { + "peak_extra_bytes": 1107558400, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "auto", + "batch": 8, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.07938237499911338 + }, + { + "forward": 0.07065062499896158 + }, + { + "forward": 0.0668822919978993 + }, + { + "forward": 0.07547945799888112 + }, + { + "forward": 0.07225391699466854 + }, + { + "forward": 0.06972220799070783 + }, + { + "forward": 0.08207025000592694 + }, + { + "forward": 0.06744995800545439 + }, + { + "forward": 0.06684045899601188 + }, + { + "forward": 0.07572829199489206 + } + ], + "forward_ms": { + "count": 10, + "median": 71.45227099681506, + "p95": 80.86070625286084, + "min": 66.84045899601188, + "max": 82.07025000592694, + "iqr": 7.648062994121574, + "mean": 72.6459833982517, + "cv": 0.07374895271920916 + }, + "memory": { + "peak_extra_bytes": 1126432768, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "sdpa", + "batch": 8, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.06910716599668376, + "prefill": 0.0748224169947207, + "decode_steps": [ + 0.010342749999836087, + 0.009700708003947511, + 0.0091700830089394, + 0.009589124994818121, + 0.009380833987961523, + 0.008251999999629334, + 0.009957083006156608, + 0.008897499996237457, + 0.008592041995143518, + 0.006129667002824135, + 0.007369958999333903, + 0.007601374993100762, + 0.007481416992959566, + 0.006690167007036507, + 0.008152042006258853, + 0.0060021249955752864, + 0.009531374991638586, + 0.005849916997249238, + 0.00766687499708496, + 0.0060157089901622385, + 0.005385625001508743, + 0.007020292003289796, + 0.0058722499961731955, + 0.00629658300022129, + 0.006847707991255447, + 0.006010750003042631, + 0.007949665989144705, + 0.005957542001851834, + 0.006740875003742985, + 0.006898958992678672, + 0.006057083999621682, + 0.00667041601263918 + ] + }, + { + "forward": 0.06292324999230914, + "prefill": 0.06440991700219456, + "decode_steps": [ + 0.005985875002807006, + 0.005539707999560051, + 0.005198708007810637, + 0.0051340420031920075, + 0.005165499998838641, + 0.00509279100515414, + 0.005479291008668952, + 0.005402875001891516, + 0.005379124995670281, + 0.005390208010794595, + 0.005409083008999005, + 0.0051890829927288, + 0.005470082993269898, + 0.005575582996243611, + 0.00545262498781085, + 0.005716332991141826, + 0.005494208002346568, + 0.006034083999111317, + 0.006392458992195316, + 0.008840750000672415, + 0.008601249995990656, + 0.0070909579953877255, + 0.006423457991331816, + 0.006443292004405521, + 0.006772958004148677, + 0.005826374996104278, + 0.005831957998452708, + 0.0057809999998426065, + 0.005579707998549566, + 0.006246958000701852, + 0.005702459005988203, + 0.005952624997007661 + ] + }, + { + "forward": 0.0530879590078257, + "prefill": 0.05705054099962581, + "decode_steps": [ + 0.0054216249991441146, + 0.005902916003833525, + 0.004850124998483807, + 0.004723624995676801, + 0.006016665996867232, + 0.004913458004011773, + 0.00490179201005958, + 0.005597958996077068, + 0.004931083007249981, + 0.005137542000738904, + 0.0055850420030765235, + 0.0050849589897552505, + 0.004781916999490932, + 0.004778583999723196, + 0.004943125008139759, + 0.005035375012084842, + 0.00462254200829193, + 0.005886290993657894, + 0.00517112499801442, + 0.004988957996829413, + 0.005237791992840357, + 0.0053477079927688465, + 0.004938166995998472, + 0.005662791998474859, + 0.005812792005599476, + 0.007284541992703453, + 0.007781000007526018, + 0.008090792005532421, + 0.008849082994856872, + 0.007795957993948832, + 0.007988249999471009, + 0.007511582996812649 + ] + }, + { + "forward": 0.06256054199184291, + "prefill": 0.06325616700632963, + "decode_steps": [ + 0.010019290988566354, + 0.008545082993805408, + 0.008254207990830764, + 0.006921625012182631, + 0.0059305000031599775, + 0.007485082998755388, + 0.006132124995929189, + 0.005643166994559579, + 0.006625041001825593, + 0.005977042004815303, + 0.005815541007905267, + 0.005512125004315749, + 0.005620417010504752, + 0.0060971670027356595, + 0.005379292008001357, + 0.0058937499998137355, + 0.005542749990127049, + 0.005388040997786447, + 0.005809916008729488, + 0.006059207997168414, + 0.005711582998628728, + 0.005738791995099746, + 0.005747625007643364, + 0.005764834000729024, + 0.006338500010315329, + 0.0056794580013956875, + 0.005680917005520314, + 0.0055584159999853, + 0.0053949579887557775, + 0.005886416998691857, + 0.005326167010935023, + 0.005603917001280934 + ] + }, + { + "forward": 0.05336037500819657, + "prefill": 0.0561692920018686, + "decode_steps": [ + 0.0056397090083919466, + 0.006274250001297332, + 0.004665832995669916, + 0.004448625011718832, + 0.005127249998622574, + 0.0050458340119803324, + 0.0044176660012453794, + 0.00487016599799972, + 0.004802915995242074, + 0.004537999993772246, + 0.004344667002442293, + 0.005120583999087103, + 0.004654832999221981, + 0.00440383299428504, + 0.005247000008239411, + 0.00486029101011809, + 0.004378915997222066, + 0.004839707995415665, + 0.004987332999007776, + 0.005493832999491133, + 0.0051964580052299425, + 0.006210500010638498, + 0.006612083001527935, + 0.005890667001949623, + 0.005644749995553866, + 0.00786804199742619, + 0.007940124996821396, + 0.008302083006128669, + 0.007560583995655179, + 0.0075406249961815774, + 0.007512624986702576, + 0.0064435829990543425 + ] + }, + { + "forward": 0.05784000000858214, + "prefill": 0.05920683300064411, + "decode_steps": [ + 0.005299374999594875, + 0.00506720801058691, + 0.0053585840068990365, + 0.0049251249874942005, + 0.005577708012424409, + 0.0065454580035293475, + 0.005039709008997306, + 0.005037374998209998, + 0.00543637499504257, + 0.004840207999222912, + 0.0051012920011999086, + 0.0051515420054784045, + 0.004951874987455085, + 0.0048274589935317636, + 0.005535667005460709, + 0.004891667005722411, + 0.004764333003549837, + 0.006507208003313281, + 0.005785875007859431, + 0.005494416996953078, + 0.00661995800328441, + 0.005417709005996585, + 0.00588087500364054, + 0.006985749991144985, + 0.005505958994035609, + 0.009863125000265427, + 0.008245624994742684, + 0.007795958998030983, + 0.009464332993957214, + 0.008793125001830049, + 0.008470083004795015, + 0.007985000003827736 + ] + }, + { + "forward": 0.06607112499477807, + "prefill": 0.07199495800887235, + "decode_steps": [ + 0.0112473330082139, + 0.009689833997981623, + 0.007954750006319955, + 0.0064138750021811575, + 0.006455249997088686, + 0.00622483401093632, + 0.007379292001132853, + 0.005943291005678475, + 0.006152791000204161, + 0.005738541003665887, + 0.006190125001012348, + 0.006385540997143835, + 0.006131833986728452, + 0.006234540996956639, + 0.006048291004844941, + 0.006059417006326839, + 0.0063403749954886734, + 0.006115458003478125, + 0.006089499991503544, + 0.006007083007716574, + 0.006039499989128672, + 0.006375708006089553, + 0.006262167007662356, + 0.0059114999894518405, + 0.007620874996064231, + 0.006319499996607192, + 0.006684209001832642, + 0.0061228749982547015, + 0.006136249998235144, + 0.006146749990875833, + 0.006074999997508712, + 0.006435499992221594 + ] + }, + { + "forward": 0.05728533300862182, + "prefill": 0.05665933300042525, + "decode_steps": [ + 0.005276333002257161, + 0.004808833997230977, + 0.004921457992168143, + 0.0045982919982634485, + 0.004511041988735087, + 0.005164832997252233, + 0.005280999990645796, + 0.005733707992476411, + 0.004661458995542489, + 0.00472812500083819, + 0.00475337500392925, + 0.004576292005367577, + 0.005153667007107288, + 0.0062358340073842555, + 0.004940792001434602, + 0.005099917005281895, + 0.0051204580086050555, + 0.004922209001961164, + 0.005300417004036717, + 0.005458416999317706, + 0.005253374998574145, + 0.005745749993366189, + 0.005814208998344839, + 0.008191917004296556, + 0.008567583994590677, + 0.00861920800525695, + 0.008619750005891547, + 0.007945582998218015, + 0.008331542005180381, + 0.0064833750075194985, + 0.006645208006375469, + 0.006282834001467563 + ] + }, + { + "forward": 0.055910499999299645, + "prefill": 0.05584041700058151, + "decode_steps": [ + 0.005401875008828938, + 0.004918874998111278, + 0.004726374987512827, + 0.005718582993722521, + 0.005013540998334065, + 0.004647791007300839, + 0.004775542009156197, + 0.005092250008601695, + 0.004548791999695823, + 0.0047926660045050085, + 0.004833583006984554, + 0.0048102500004461035, + 0.005085083001176827, + 0.005011208006180823, + 0.00503095900057815, + 0.005219124999712221, + 0.00530308298766613, + 0.005394958003307693, + 0.005476999998791143, + 0.005840915997396223, + 0.00570012500975281, + 0.007652083004359156, + 0.007926082995254546, + 0.008538875001249835, + 0.00819137500366196, + 0.008407290995819494, + 0.00725941700511612, + 0.00615937499969732, + 0.006338124992907979, + 0.006743291989550926, + 0.006104583997512236, + 0.005745499991462566 + ] + }, + { + "forward": 0.06567133299540728, + "prefill": 0.07110083299630787, + "decode_steps": [ + 0.008053416007896885, + 0.00542983399645891, + 0.005781040992587805, + 0.0059561660018516704, + 0.005494958997587673, + 0.005204874993069097, + 0.006915417005075142, + 0.005557625001529232, + 0.005380207992857322, + 0.005509082999196835, + 0.006003416987368837, + 0.005509417009307072, + 0.006119208002928644, + 0.005424000002676621, + 0.00601787501364015, + 0.005723249996663071, + 0.005609041007119231, + 0.005768040995462798, + 0.005721459005144425, + 0.00541712500853464, + 0.005681291993823834, + 0.005414208004367538, + 0.00528233298973646, + 0.005101999995531514, + 0.005109500008984469, + 0.004754624998895451, + 0.005258707999018952, + 0.005765709007391706, + 0.005269375003990717, + 0.006677917001070455, + 0.005082125004264526, + 0.005056957990746014 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 60.200271000212524, + "p95": 67.7409475458262, + "min": 53.0879590078257, + "max": 69.10716599668376, + "iqr": 8.730103993002558, + "mean": 60.381758300354704, + "cv": 0.09339606452660039 + }, + "prefill_ms": { + "count": 10, + "median": 61.23150000348687, + "p95": 73.55006045108894, + "min": 55.84041700058151, + "max": 74.8224169947207, + "iqr": 12.670968997554155, + "mean": 63.05107080115704, + "cv": 0.11537121283187743 + }, + "decode_ms_per_step": { + "count": 320, + "median": 5.756229504186194, + "p95": 8.628418755688473, + "min": 4.344667002442293, + "max": 11.2473330082139, + "iqr": 1.2860000097134616, + "mean": 6.114127334512887, + "cv": 0.21016430670974193 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1356.8911058457313, + "p95": 1414.68970923449, + "min": 1066.3089957018212, + "max": 1415.2765793260378, + "iqr": 79.41774818034628, + "mean": 1317.316891361773, + "cv": 0.08144844959944939 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1011.5586842464077, + "p95": 1069.504606924895, + "min": 812.9489559383729, + "max": 1079.9288042743744, + "iqr": 66.0655049405201, + "mean": 996.2025635250633, + "cv": 0.08153344188483598 + }, + "memory": { + "peak_extra_bytes": 1140850688, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "reference", + "batch": 8, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.17549529200186953, + "prefill": 0.17491779099509586, + "decode_steps": [ + 0.022657416004221886, + 0.019442707998678088, + 0.013222250010585412, + 0.016227750005782582, + 0.01183258299715817, + 0.011861708990181796, + 0.010925541995675303, + 0.010669458002666943, + 0.011333458998706192, + 0.010888832999626175, + 0.010291124999639578, + 0.012110249997931533, + 0.009982291012420319, + 0.011091166990809143, + 0.011585667001781985, + 0.009935499998391606, + 0.010145791995455511, + 0.01168695799424313, + 0.009846041997661814, + 0.009863334009423852, + 0.011135209002532065, + 0.01002824999159202, + 0.010674167002434842, + 0.012012833001790568, + 0.0105256249953527, + 0.01037049999285955, + 0.010704458007239737, + 0.009884541010251269, + 0.009974916989449412, + 0.009677666996140033, + 0.009367958002258092, + 0.011255666002398357 + ] + }, + { + "forward": 0.1459856250003213, + "prefill": 0.14590175000193994, + "decode_steps": [ + 0.02030225000635255, + 0.01773191698885057, + 0.012717750010779127, + 0.014639875007560477, + 0.010627541996655054, + 0.010054541999124922, + 0.010056915998575278, + 0.009098750000703149, + 0.009004125007777475, + 0.009414375002961606, + 0.008652165997773409, + 0.009370291998493485, + 0.028764125003363006, + 0.01103145799424965, + 0.009358040988445282, + 0.010279958005412482, + 0.009428833000129089, + 0.009539749997202307, + 0.009441791989956982, + 0.009362750002765097, + 0.008810915998765267, + 0.009254917007638142, + 0.009209875002852641, + 0.008695541007909924, + 0.00920041700010188, + 0.008826833000057377, + 0.010687417001463473, + 0.008996999997179955, + 0.008751041998039, + 0.009093915999983437, + 0.008721666992641985, + 0.00915287499083206 + ] + }, + { + "forward": 0.13239408300432842, + "prefill": 0.13605529100459535, + "decode_steps": [ + 0.019504792013322003, + 0.01726595799846109, + 0.01213070799713023, + 0.01154508299077861, + 0.01093333400785923, + 0.010197333001997322, + 0.01005120800982695, + 0.009359332994790748, + 0.00906391700846143, + 0.008734999995795079, + 0.009435667001525871, + 0.008809249993646517, + 0.009580416008247994, + 0.009295291994931176, + 0.008747540996409953, + 0.008816166999167763, + 0.00857995799742639, + 0.008493125002132729, + 0.008433750001131557, + 0.008409083005972207, + 0.008514415996614844, + 0.00829866599815432, + 0.008765584003413096, + 0.008869208002579398, + 0.008965500004705973, + 0.00783229200169444, + 0.008411832997808233, + 0.008351958997081965, + 0.008826334000332281, + 0.008236833004048094, + 0.00838729199313093, + 0.008070915995631367 + ] + }, + { + "forward": 0.14514899998903275, + "prefill": 0.14129791699815542, + "decode_steps": [ + 0.019877165992511436, + 0.01386291699600406, + 0.011444874995504506, + 0.011096375004854053, + 0.010343582995119505, + 0.010834500004420988, + 0.009812582997255959, + 0.009326207989943214, + 0.010734499999671243, + 0.010148625005967915, + 0.008064792011282407, + 0.0077771670039510354, + 0.008901749999495223, + 0.011344040991389193, + 0.009009625006001443, + 0.008702542007085867, + 0.008380875005968846, + 0.008408792011323385, + 0.009248875008779578, + 0.00871829099196475, + 0.009174124992568977, + 0.007623124998644926, + 0.008908624993637204, + 0.007837874989490956, + 0.008779666997725144, + 0.007958624992170371, + 0.008643624998512678, + 0.008044917005463503, + 0.00800820799486246, + 0.007619500000146218, + 0.009533916992950253, + 0.0077309170010266826 + ] + }, + { + "forward": 0.13869383299606852, + "prefill": 0.14626658300403506, + "decode_steps": [ + 0.022349958002450876, + 0.01886341700446792, + 0.011769791002734564, + 0.012703999993391335, + 0.011897374992258847, + 0.010004791998653673, + 0.010704665997764096, + 0.00968262500828132, + 0.01116754200484138, + 0.008809457998722792, + 0.008559833004255779, + 0.008729541004868224, + 0.008519458002410829, + 0.00843195800553076, + 0.007919125011540018, + 0.008431458001723513, + 0.00839845799782779, + 0.00893516700307373, + 0.00780054199276492, + 0.009722958988277242, + 0.008262250004918315, + 0.009987124998588115, + 0.008602707996033132, + 0.007972042003530078, + 0.008607958996435627, + 0.00878483300039079, + 0.008463250007480383, + 0.008993166004074737, + 0.010607000003801659, + 0.009148749988526106, + 0.010131917006219737, + 0.008602874993812293 + ] + }, + { + "forward": 0.135674334000214, + "prefill": 0.13890508298936766, + "decode_steps": [ + 0.01189541700296104, + 0.011989542006631382, + 0.011428625002736226, + 0.010071291995700449, + 0.011205375005374663, + 0.009266375011065975, + 0.009433958999579772, + 0.009796083002584055, + 0.008272333012428135, + 0.009176207997370511, + 0.008619208994787186, + 0.00965474999975413, + 0.008951249998062849, + 0.008862083006533794, + 0.008962208012235351, + 0.008632625002064742, + 0.009284332991228439, + 0.009516500009340234, + 0.01082325000606943, + 0.009163208000245504, + 0.009034707996761426, + 0.008754125010455027, + 0.009356041991850361, + 0.009157832988421433, + 0.008828459001961164, + 0.009664375000284053, + 0.008700874997884966, + 0.009817666999879293, + 0.01007237499288749, + 0.009105917008128017, + 0.01249600001028739, + 0.009834209005930461 + ] + }, + { + "forward": 0.16545424998912495, + "prefill": 0.14871100000164006, + "decode_steps": [ + 0.02381204199627973, + 0.0173192500078585, + 0.013207249998231418, + 0.012768957996740937, + 0.010538415997871198, + 0.010527042002649978, + 0.009973749998607673, + 0.009730125006171875, + 0.009975666995160282, + 0.008960084000136703, + 0.009200833010254428, + 0.009148874989477918, + 0.00887475001218263, + 0.0089402089943178, + 0.008888208991265856, + 0.009548667003400624, + 0.009547000008751638, + 0.010371707990998402, + 0.00895812499220483, + 0.008422375001828186, + 0.008802166994428262, + 0.00855254199996125, + 0.00850445800460875, + 0.008340459011378698, + 0.00850408399128355, + 0.00812633300665766, + 0.010762707999674603, + 0.008514291999745183, + 0.008233000000473112, + 0.008277625005575828, + 0.008172625006409362, + 0.008147040993208066 + ] + }, + { + "forward": 0.13536766701145098, + "prefill": 0.14284337499702815, + "decode_steps": [ + 0.016705792004358955, + 0.016438874998129904, + 0.013203084003180265, + 0.010939915999188088, + 0.01041929199709557, + 0.01088658299704548, + 0.009917832998326048, + 0.009624416998121887, + 0.009908125008223578, + 0.009279917008825578, + 0.009921374992700294, + 0.008894459009752609, + 0.008576292006182484, + 0.009728166987770237, + 0.009040874996571802, + 0.00866712500283029, + 0.008464041005936451, + 0.009816791993216611, + 0.008041207998758182, + 0.010129165995749645, + 0.008091499999864027, + 0.00783025000419002, + 0.008085166991804726, + 0.008113625008263625, + 0.00861849999637343, + 0.008219124996685423, + 0.00814704100775998, + 0.007575292009278201, + 0.00764145799621474, + 0.008320375010953285, + 0.00797454199346248, + 0.007959875001688488 + ] + }, + { + "forward": 0.13499966700328514, + "prefill": 0.13486408299650066, + "decode_steps": [ + 0.013274208002258092, + 0.009634333997382782, + 0.008637500010081567, + 0.008390874994802289, + 0.008465915991109796, + 0.008946332993218675, + 0.009027791995322332, + 0.01007999999274034, + 0.008412332987063564, + 0.00867554200522136, + 0.007816709010512568, + 0.009499791995040141, + 0.008930041003623046, + 0.008491000000503846, + 0.009009708010125905, + 0.00834558300266508, + 0.008747207990381867, + 0.00896783301141113, + 0.010428291992866434, + 0.00849441699392628, + 0.009075457986909896, + 0.009030957997310907, + 0.008881292000296526, + 0.008929749994422309, + 0.008102916995994747, + 0.009571374990628101, + 0.008170749992132187, + 0.00899291699170135, + 0.008193749992642552, + 0.00953066699730698, + 0.008665667002787814, + 0.008946667003328912 + ] + }, + { + "forward": 0.14524270800757222, + "prefill": 0.16113045800011605, + "decode_steps": [ + 0.014077707994147204, + 0.010856708002393134, + 0.011190457997145131, + 0.012499416989157908, + 0.009756290994118899, + 0.009210082993377, + 0.009633791007217951, + 0.008936625003116205, + 0.009762708999915048, + 0.009072333996300586, + 0.008966124994913116, + 0.009089125000173226, + 0.010074208010337315, + 0.009756207989994437, + 0.008959082988440059, + 0.00969141699897591, + 0.009154125000350177, + 0.008840124995913357, + 0.008722958009457216, + 0.008763417004956864, + 0.008682917003170587, + 0.008826207995298319, + 0.00905941599921789, + 0.01126487499277573, + 0.009631333989091218, + 0.009579709003446624, + 0.009045790997333825, + 0.009173750004265457, + 0.00888879100966733, + 0.009368917002575472, + 0.008517791997292079, + 0.00882145800278522 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 141.92141649255063, + "p95": 170.97682309613447, + "min": 132.39408300432842, + "max": 175.49529200186953, + "iqr": 10.355561993492302, + "mean": 145.44564590032678, + "cv": 0.09792854018389592 + }, + "prefill_ms": { + "count": 10, + "median": 144.37256249948405, + "p95": 168.71349114735492, + "min": 134.86408299650066, + "max": 174.91779099509586, + "iqr": 8.596604260674212, + "mean": 147.08933309884742, + "cv": 0.08375890407067643 + }, + "decode_ms_per_step": { + "count": 320, + "median": 9.173937498417217, + "p95": 14.105816344817875, + "min": 7.575292009278201, + "max": 28.764125003363006, + "iqr": 1.4940415021555964, + "mean": 9.964923818688476, + "cv": 0.26186391018604793 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 827.4832780038836, + "p95": 865.7477369904144, + "min": 689.6335749270081, + "max": 887.7558167770447, + "iqr": 39.58405604071902, + "mean": 806.6625891604551, + "cv": 0.07044825959109469 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 558.850174210499, + "p95": 591.57041003656, + "min": 468.75336230649657, + "max": 604.8696730602787, + "iqr": 27.532784700808065, + "mean": 551.8617357065455, + "cv": 0.06801621132954826 + }, + "memory": { + "peak_extra_bytes": 1107558400, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "auto", + "batch": 8, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.23265937498945277 + }, + { + "forward": 0.19915737499832176 + }, + { + "forward": 0.20839475000684615 + }, + { + "forward": 0.226246792008169 + }, + { + "forward": 0.19226162499398924 + }, + { + "forward": 0.1899290829896927 + }, + { + "forward": 0.30714295800135005 + }, + { + "forward": 0.18596454200451262 + }, + { + "forward": 0.18870537501061335 + }, + { + "forward": 0.2901228330010781 + } + ], + "forward_ms": { + "count": 10, + "median": 203.77606250258395, + "p95": 299.4839017512277, + "min": 185.96454200451262, + "max": 307.14295800135005, + "iqr": 40.544010753365, + "mean": 222.05847080040257, + "cv": 0.19599960882210482 + }, + "memory": { + "peak_extra_bytes": 1145307136, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "sdpa", + "batch": 8, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.1957603329938138, + "prefill": 0.21780970800318755, + "decode_steps": [ + 0.01541491701209452, + 0.013557500002207235, + 0.012363832996925339, + 0.012750915993819945, + 0.011155125001096167, + 0.011022749997209758, + 0.009052583001903258, + 0.008450791996438056, + 0.00951458400231786, + 0.008621957997092977, + 0.010584916002699174, + 0.009008042005007155, + 0.008410916998400353, + 0.008658042002934963, + 0.00816633399517741, + 0.00933141699351836, + 0.008022875001188368, + 0.009006291991681792, + 0.008715207994100638, + 0.009882375001325272, + 0.013614084004075266, + 0.012019417001283728, + 0.011596999989706092, + 0.010602416004985571, + 0.011809290997916833, + 0.011343125006533228, + 0.011070749998907559, + 0.010704542000894435, + 0.008829625003272668, + 0.008783583005424589, + 0.00871683300647419, + 0.009667291000369005 + ] + }, + { + "forward": 0.11924487500800751, + "prefill": 0.12671675000456162, + "decode_steps": [ + 0.007223125008749776, + 0.007312540998100303, + 0.0068599590013036504, + 0.007309749998967163, + 0.006930500007001683, + 0.006978791992878541, + 0.008338375002495013, + 0.006899625004734844, + 0.0072244999901158735, + 0.0068652920017484576, + 0.0077825000043958426, + 0.006842125003458932, + 0.0076942909945501015, + 0.007113832994946279, + 0.00749808301043231, + 0.007302875004825182, + 0.00823099999979604, + 0.007988333003595471, + 0.009937834009178914, + 0.009078333008801565, + 0.01157275000878144, + 0.010576999993645586, + 0.010854916996322572, + 0.010313916005543433, + 0.011079375006374903, + 0.010728916997322813, + 0.013325667008757591, + 0.01080845900287386, + 0.010991166011081077, + 0.010219874995527789, + 0.0105145829875255, + 0.010548749996814877 + ] + }, + { + "forward": 0.1600853330019163, + "prefill": 0.1843109579931479, + "decode_steps": [ + 0.0107134580030106, + 0.009475749990087934, + 0.012313416998949833, + 0.011668916005874053, + 0.01012191700283438, + 0.010445040999911726, + 0.01161404101003427, + 0.01006262500595767, + 0.010538582995650358, + 0.01307766599347815, + 0.012743583996780217, + 0.009831333009060472, + 0.010624083006405272, + 0.009311292000347748, + 0.011126458004582673, + 0.012150791997555643, + 0.014628374992753379, + 0.0158932499907678, + 0.014993874996434897, + 0.013230124997789972, + 0.012276082998141646, + 0.011701208000886254, + 0.01306920799834188, + 0.010093582997797057, + 0.009476457998971455, + 0.010039000000688247, + 0.008237624992034398, + 0.007796374993631616, + 0.007927708007628098, + 0.008716583004570566, + 0.008613166995928623, + 0.009508874994935468 + ] + }, + { + "forward": 0.19271320800180547, + "prefill": 0.19285775000753347, + "decode_steps": [ + 0.009507458002190106, + 0.009591541995177977, + 0.007913540990557522, + 0.0086536670132773, + 0.008210042011342011, + 0.008580290988902561, + 0.010408292000647634, + 0.007929083993076347, + 0.009497959006694146, + 0.007843708008294925, + 0.009316666997619905, + 0.007918165996670723, + 0.008333334000781178, + 0.007878999997046776, + 0.009537167003145441, + 0.008766624989220873, + 0.01038620799954515, + 0.010020583009463735, + 0.009331458000815473, + 0.00994454199098982, + 0.009974625005270354, + 0.008356749996892177, + 0.009638082992751151, + 0.009666124999057502, + 0.009953207991202362, + 0.008368584007257596, + 0.009203084002365358, + 0.008635542006231844, + 0.009024458995554596, + 0.009157083011814393, + 0.008469499996863306, + 0.009189499993226491 + ] + }, + { + "forward": 0.11918287500157021, + "prefill": 0.12816837499849498, + "decode_steps": [ + 0.0070940000005066395, + 0.007004125000094064, + 0.006784165991120972, + 0.006846082993433811, + 0.007139374996768311, + 0.006706000000122003, + 0.006968250003410503, + 0.00671100000909064, + 0.007220084007713012, + 0.007102416988345794, + 0.006822041992563754, + 0.007389208010863513, + 0.007165874994825572, + 0.007722666006884538, + 0.007052083004964516, + 0.007825708002201281, + 0.007171749995904975, + 0.008631208998849615, + 0.008012833990505897, + 0.009631166001781821, + 0.011125208009616472, + 0.011470666999230161, + 0.010932542005321011, + 0.01014808300533332, + 0.0118662500026403, + 0.009993167011998594, + 0.009249042006558739, + 0.007318874995689839, + 0.007272707996889949, + 0.0072884160035755485, + 0.007270375004736707, + 0.007557874996564351 + ] + }, + { + "forward": 0.14031545900797937, + "prefill": 0.15540291600336786, + "decode_steps": [ + 0.00949366700660903, + 0.011007833003532141, + 0.010245208002743311, + 0.008813541004201397, + 0.008895624996512197, + 0.008091415991657414, + 0.00946825000573881, + 0.007894791997387074, + 0.008807958001852967, + 0.00783041599788703, + 0.00907325001026038, + 0.007915542009868659, + 0.010322625006665476, + 0.008708000008482486, + 0.00843999999051448, + 0.00843062499188818, + 0.008345166992512532, + 0.009319958990090527, + 0.0081895000039367, + 0.00950708299933467, + 0.011147666999022476, + 0.01162358300643973, + 0.013016417011385784, + 0.011794207995990291, + 0.01269279100233689, + 0.01196245799656026, + 0.01605195899901446, + 0.011904290993697941, + 0.01042787499318365, + 0.008029000004171394, + 0.012999625003430992, + 0.009071332999155857 + ] + }, + { + "forward": 0.20099458300683182, + "prefill": 0.19603775000723545, + "decode_steps": [ + 0.013138624999555759, + 0.012884540992672555, + 0.011837916987133212, + 0.012319292000029236, + 0.012927791991387494, + 0.013003707994357683, + 0.011431167004047893, + 0.013109125007758848, + 0.013104917001328431, + 0.012474499992094934, + 0.011674000008497387, + 0.010100917003001086, + 0.011759459011955187, + 0.011607167005422525, + 0.010083000001031905, + 0.01022808300331235, + 0.008370042007300071, + 0.010139499994693324, + 0.00837475000298582, + 0.008505792007781565, + 0.00838837499031797, + 0.00861175000318326, + 0.0089882499887608, + 0.00866820800001733, + 0.00911008300317917, + 0.007973333005793393, + 0.008994333998998627, + 0.009507458002190106, + 0.011585375003051013, + 0.009348874998977408, + 0.007926250007585622, + 0.008098916994640604 + ] + }, + { + "forward": 0.12537720799446106, + "prefill": 0.14131954200274777, + "decode_steps": [ + 0.008023959002457559, + 0.01202320800803136, + 0.011780415996327065, + 0.011859999998705462, + 0.011459582994575612, + 0.011826624991954304, + 0.012560916991787963, + 0.009644958001445048, + 0.00979758400353603, + 0.007937249989481643, + 0.008926666996558197, + 0.009100374998524785, + 0.010059749998617917, + 0.009285541993449442, + 0.009890166998957284, + 0.008615540995378979, + 0.008747667001443915, + 0.008905584007152356, + 0.009082791992113926, + 0.010285333002684638, + 0.00960954200127162, + 0.008912040997529402, + 0.009778833002201281, + 0.008968374997493811, + 0.010636167004122399, + 0.01166699999885168, + 0.008927916001994163, + 0.010717790995840915, + 0.011893584000063129, + 0.009491749995504506, + 0.011098625007434748, + 0.010081416010507382 + ] + }, + { + "forward": 0.1393258330062963, + "prefill": 0.1584103750064969, + "decode_steps": [ + 0.01022862498939503, + 0.010087791990372352, + 0.011401249998016283, + 0.00943350000306964, + 0.01005983300274238, + 0.008536291003110819, + 0.009014583003590815, + 0.009444957991945557, + 0.008716750002349727, + 0.010115917000803165, + 0.008264541989774443, + 0.010039750006399117, + 0.009251291005057283, + 0.01082283399591688, + 0.013639583994518034, + 0.011392707994673401, + 0.013907458007452078, + 0.011158499997691251, + 0.01181016699410975, + 0.011330417008139193, + 0.012803958001313731, + 0.010326999996323138, + 0.010410583010525443, + 0.009402125011547469, + 0.009190833996399306, + 0.009683541997219436, + 0.009036290997755714, + 0.010082499997224659, + 0.008549083999241702, + 0.012719582999125123, + 0.009830625000176951, + 0.01056791699375026 + ] + }, + { + "forward": 0.23436833299638238, + "prefill": 0.26742424999247305, + "decode_steps": [ + 0.01934708299813792, + 0.016206250002142042, + 0.015425667006638832, + 0.014926083997124806, + 0.014371375000337139, + 0.014532832996337675, + 0.016891499995836057, + 0.015115041998797096, + 0.014434084005188197, + 0.015467083008843474, + 0.01595904199348297, + 0.014892958992277272, + 0.01350125001044944, + 0.011258291007834487, + 0.01349729199137073, + 0.011883917002705857, + 0.01333333300135564, + 0.01047837499936577, + 0.010618458007229492, + 0.009621374993002973, + 0.008901417008019052, + 0.009935917012626305, + 0.00954670799546875, + 0.009638375006034039, + 0.009359625008073635, + 0.009230916999513283, + 0.008572542006731965, + 0.008942707994719967, + 0.0085758750064997, + 0.008342540997546166, + 0.009238207989255898, + 0.00989050000498537 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 150.20039600494783, + "p95": 219.3501455010846, + "min": 119.18287500157021, + "max": 234.36833299638238, + "iqr": 66.13418749839184, + "mean": 162.73680400190642, + "cv": 0.24940425256865456 + }, + "prefill_ms": { + "count": 10, + "median": 171.3606664998224, + "p95": 245.09770609729452, + "min": 126.71675000456162, + "max": 267.42424999247305, + "iqr": 50.40236450440716, + "mean": 176.84583740192465, + "cv": 0.24939708976561092 + }, + "decode_ms_per_step": { + "count": 320, + "median": 9.626270497392397, + "p95": 14.374510450579692, + "min": 6.706000000122003, + "max": 19.34708299813792, + "iqr": 2.709645003051264, + "mean": 10.036585925263353, + "cv": 0.2137658502140028 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 785.3650794562575, + "p95": 951.8384044988379, + "min": 653.1668208401225, + "max": 990.3546841649427, + "iqr": 96.60855721535916, + "mean": 807.1265652823201, + "cv": 0.11925101983460745 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 526.9250270899113, + "p95": 645.3488849113767, + "min": 388.25476200713354, + "max": 662.0775998768058, + "iqr": 70.84886691769714, + "mean": 524.8116479211847, + "cv": 0.15020278219731084 + }, + "memory": { + "peak_extra_bytes": 1107296256, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "reference", + "batch": 8, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.45448970900906716, + "prefill": 0.4561821249953937, + "decode_steps": [ + 0.015243332993122749, + 0.01298216701252386, + 0.013478750013746321, + 0.015391791996080428, + 0.016554666988668032, + 0.014883374999044463, + 0.01466758300375659, + 0.014833749999525025, + 0.012886291995528154, + 0.01166491600451991, + 0.011659750001854263, + 0.013217625004472211, + 0.011454541003331542, + 0.01051066699437797, + 0.011270458999206312, + 0.013258666993351653, + 0.010710417001973838, + 0.010625499999150634, + 0.011192459001904353, + 0.010913207996054552, + 0.009883749997243285, + 0.010707583001931198, + 0.010003874995163642, + 0.010115249999216758, + 0.012193917005788535, + 0.010029500001110137, + 0.009723916999064386, + 0.010092209005961195, + 0.009609458997147158, + 0.009737458996823989, + 0.009402750001754612, + 0.009085291007068008 + ] + }, + { + "forward": 0.47551345800457057, + "prefill": 0.5197565830021631, + "decode_steps": [ + 0.021287167008267716, + 0.018162916006986052, + 0.016820958990138024, + 0.01782074999937322, + 0.015883833999396302, + 0.014888167002936825, + 0.014061750000109896, + 0.013840624989825301, + 0.01518404099624604, + 0.01528166700154543, + 0.014731000002939254, + 0.01467504100583028, + 0.016903166004340164, + 0.015103165991604328, + 0.013525874994229525, + 0.013973499997518957, + 0.012735333002638072, + 0.01278566698601935, + 0.012167416993179359, + 0.014798166987020522, + 0.012967332993866876, + 0.012178083008620888, + 0.012329167002462782, + 0.012211625013151206, + 0.012871291997726075, + 0.01260024998919107, + 0.012862667004810646, + 0.012744666004437022, + 0.012917750005726703, + 0.01285412498691585, + 0.012647166004171595, + 0.01167162500496488 + ] + }, + { + "forward": 0.5073569170053815, + "prefill": 0.5727002079947852, + "decode_steps": [ + 0.02007670799503103, + 0.015565124995191582, + 0.016659374989103526, + 0.01810808300797362, + 0.01407787500647828, + 0.013402874988969415, + 0.014769500005058944, + 0.013121875002980232, + 0.013812000004691072, + 0.013338125005248003, + 0.012824707999243401, + 0.012830874999053776, + 0.012960459003807046, + 0.012609874989720993, + 0.011811917007435113, + 0.012841999996453524, + 0.014734958007466048, + 0.014450542003032751, + 0.013150416998541914, + 0.01654654199955985, + 0.01404049999837298, + 0.01276941598916892, + 0.012902625006972812, + 0.012894958010292612, + 0.012448500012396835, + 0.012271875006263144, + 0.01175633400271181, + 0.01178362499922514, + 0.012276958994334564, + 0.01172233400575351, + 0.01095316700229887, + 0.011765167000703514 + ] + }, + { + "forward": 0.462160084003699, + "prefill": 0.4730570839892607, + "decode_steps": [ + 0.015124333003768697, + 0.01240529200003948, + 0.011590832989895716, + 0.012385374997393228, + 0.011709750004229136, + 0.012233707995619625, + 0.011448290999396704, + 0.012494875001721084, + 0.011004166008206084, + 0.010465999992447905, + 0.011082167009590194, + 0.01272400000016205, + 0.010504332996788435, + 0.010229291990981437, + 0.011362374993041158, + 0.011915666997083463, + 0.013286166999023408, + 0.011459041998023167, + 0.012261625000974163, + 0.011806999988039024, + 0.011532791002537124, + 0.012393291995977052, + 0.013497916996129788, + 0.010751583002274856, + 0.009912207999150269, + 0.011107166996225715, + 0.01028733300336171, + 0.010188583997660317, + 0.011102208009106107, + 0.009701708011562005, + 0.010955417004879564, + 0.012403125001583248 + ] + }, + { + "forward": 0.4595974580006441, + "prefill": 0.548478791009984, + "decode_steps": [ + 0.027422541999840178, + 0.02396820898866281, + 0.01711829200212378, + 0.015750250007840805, + 0.015591125003993511, + 0.014759167010197416, + 0.014432958996621892, + 0.01739599999564234, + 0.016224582999711856, + 0.013921584002673626, + 0.014815791990258731, + 0.014058166998438537, + 0.01297016600437928, + 0.012588416997459717, + 0.01252083299914375, + 0.01277424998988863, + 0.014787041989620775, + 0.013053041999228299, + 0.011884124993230216, + 0.011427874997025356, + 0.017533749996800907, + 0.016777832992374897, + 0.011314707997371443, + 0.013481540998327546, + 0.012398083999869414, + 0.011252458003582433, + 0.01145329100836534, + 0.011496208011521958, + 0.010921875000349246, + 0.011692916988977231, + 0.012255708003067411, + 0.014902249997248873 + ] + }, + { + "forward": 0.48464958398835734, + "prefill": 0.593432917012251, + "decode_steps": [ + 0.029954707992146723, + 0.026413166997372173, + 0.021864083988475613, + 0.02050345900352113, + 0.018648375000339, + 0.01568537499406375, + 0.016395999991800636, + 0.015584042004775256, + 0.016739665996283293, + 0.01631979200465139, + 0.01555266699870117, + 0.01597945899993647, + 0.015460790993529372, + 0.014545416997862048, + 0.014639917004387826, + 0.014341959002194926, + 0.013148541998816654, + 0.011854500000481494, + 0.012990250004804693, + 0.013080040997010656, + 0.012144874999648891, + 0.011944083991693333, + 0.01199033300508745, + 0.013276124998810701, + 0.012230041000293568, + 0.010890667006606236, + 0.012126874993555248, + 0.012249874998815358, + 0.010835416993359104, + 0.010057417006464675, + 0.012228332998347469, + 0.010189125008764677 + ] + }, + { + "forward": 0.455163167003775, + "prefill": 0.5025105840031756, + "decode_steps": [ + 0.014997541991760954, + 0.016196083000977524, + 0.015622999999322928, + 0.01543975000095088, + 0.015017917001387104, + 0.015246832990669645, + 0.015192708000540733, + 0.013111624997691251, + 0.012137541998527013, + 0.013163916999474168, + 0.014613624996854924, + 0.014208416003384627, + 0.014873250009259209, + 0.015495416999328882, + 0.016068042008555494, + 0.016873250002390705, + 0.015050624991999939, + 0.01504199999908451, + 0.012296792003326118, + 0.011976915993727744, + 0.022659624999505468, + 0.013012666007853113, + 0.012592041995958425, + 0.015147041995078325, + 0.013374332993407734, + 0.01285391699639149, + 0.010938833002001047, + 0.01071087499440182, + 0.01056008400337305, + 0.011000541999237612, + 0.010208291001617908, + 0.010884000002988614 + ] + }, + { + "forward": 0.7141004169970984, + "prefill": 0.9321897089976119, + "decode_steps": [ + 0.035107416988466866, + 0.025613042002078146, + 0.017479957998148166, + 0.0187087919912301, + 0.016591625011642464, + 0.016760333004640415, + 0.016303708995110355, + 0.014487332999124192, + 0.013884667001548223, + 0.01351783299469389, + 0.011980832990957424, + 0.012974375000339933, + 0.012439250000170432, + 0.012740165999275632, + 0.01258679099555593, + 0.011752000005799346, + 0.01151837500219699, + 0.011605541003518738, + 0.01155966600344982, + 0.011128999991342425, + 0.011931165994610637, + 0.014202415986801498, + 0.014774333001696505, + 0.011234582998440601, + 0.011223541994695552, + 0.012974375000339933, + 0.0120830409869086, + 0.012755542004015297, + 0.01143275000504218, + 0.01135654200334102, + 0.011759375003748573, + 0.011253417003899813 + ] + }, + { + "forward": 0.5544361659995047, + "prefill": 0.5218709580076393, + "decode_steps": [ + 0.030630666995421052, + 0.015860084007726982, + 0.016776915988884866, + 0.014199833996826783, + 0.013415291003184393, + 0.01262495800619945, + 0.01262662498629652, + 0.012815666006645188, + 0.0139898329944117, + 0.014891375001752749, + 0.015213500009849668, + 0.014022167000803165, + 0.01280458300607279, + 0.012885624993941747, + 0.012175875002867542, + 0.013595500000519678, + 0.014081375004025176, + 0.013979791008750908, + 0.012483500002417713, + 0.0122402920096647, + 0.012734249990899116, + 0.01237070899514947, + 0.010916708008153364, + 0.010534000000916421, + 0.012173750001238659, + 0.011028792010620236, + 0.010798290997627191, + 0.011660542004392482, + 0.011192291000043042, + 0.010598249995382503, + 0.01071408401185181, + 0.010563833988271654 + ] + }, + { + "forward": 0.4633527080004569, + "prefill": 0.49838495899166446, + "decode_steps": [ + 0.027768583997385576, + 0.01992374999099411, + 0.015959958007442765, + 0.014372042001923546, + 0.015201750007690862, + 0.01412908299244009, + 0.013592292001703754, + 0.012952083998243324, + 0.012567124998895451, + 0.012784790989826433, + 0.012706583002000116, + 0.01195200000074692, + 0.012019083005725406, + 0.012447582994354889, + 0.012834459004807286, + 0.011902208992978558, + 0.011715500004356727, + 0.011439249996328726, + 0.011262250001891516, + 0.013912875001551583, + 0.012356374994851649, + 0.012373459001537412, + 0.011837917001685128, + 0.009919124990119599, + 0.010854708991246298, + 0.010946958005661145, + 0.010531749998335727, + 0.010105624998686835, + 0.00984116698964499, + 0.009553249998134561, + 0.009848125002463348, + 0.009150790996500291 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 469.43308300251374, + "p95": 642.251504048181, + "min": 454.48970900906716, + "max": 714.1004169970984, + "iqr": 41.44196924971766, + "mean": 503.0819668012555, + "cv": 0.1596884799140421 + }, + "prefill_ms": { + "count": 10, + "median": 520.8137705049012, + "p95": 779.7491526041991, + "min": 456.1821249953937, + "max": 932.1897089976119, + "iqr": 67.22848850404262, + "mean": 561.8563918003929, + "cv": 0.24346873843833175 + }, + "decode_ms_per_step": { + "count": 320, + "median": 12.771832989528775, + "p95": 18.187188956653706, + "min": 9.085291007068008, + "max": 35.107416988466866, + "iqr": 3.1598862587998156, + "mean": 13.501684130869762, + "cv": 0.2453675071604059 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 580.6948686916122, + "p95": 683.9540745378055, + "min": 533.4829553271437, + "max": 689.4181403779718, + "iqr": 57.915261676323894, + "mean": 596.4707102692496, + "cv": 0.08823514309514863 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 266.1171246749261, + "p95": 305.22184454920335, + "min": 184.4498014179189, + "max": 306.89298315268144, + "iqr": 25.804817541421954, + "mean": 262.44671223807666, + "cv": 0.13289927839300383 + }, + "memory": { + "peak_extra_bytes": 1912864768, + "method": "mps_driver_allocated_delta" + } + } + ] +} diff --git a/experiments/records/attention/20261022T033548Z_attention_mps_float16_615767d.json b/experiments/records/attention/20261022T033548Z_attention_mps_float16_615767d.json new file mode 100644 index 0000000..ae63270 --- /dev/null +++ b/experiments/records/attention/20261022T033548Z_attention_mps_float16_615767d.json @@ -0,0 +1,14061 @@ +{ + "record_version": 1, + "created_utc": "20261022T033548Z", + "experiment": "attention_backends", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "mps", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "615767d83416226055484fb1f7336bdf4a40eaa0", + "branch": "experiments/attention-kv-audit", + "dirty": false + }, + "command": "python scripts/benchmark_attention.py --device mps --dtype float16" + }, + "settings": { + "dtype": "float16", + "position": "learned", + "backends": [ + "auto", + "sdpa", + "reference" + ], + "seq_lens": [ + 128, + 256, + 512, + 1024, + 2048 + ], + "batch_sizes": [ + 1, + 4, + 8 + ], + "decode_tokens": 32, + "warmup": 3, + "repeats": 10, + "seed": 0, + "weights": "seeded random initialization (timing does not depend on trained values)", + "model_config": { + "vocab_size": 256, + "context_length": 2080, + "embedding_dim": 192, + "num_layers": 4, + "num_heads": 6, + "dropout": 0.0, + "positional_encoding": "learned", + "norm": "rmsnorm", + "activation": "swiglu", + "rope_theta": 10000.0 + }, + "inference_mode": true + }, + "results": [ + { + "backend": "auto", + "batch": 1, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.0018057910056086257 + }, + { + "forward": 0.002490792001481168 + }, + { + "forward": 0.0021970419911667705 + }, + { + "forward": 0.0019100000063190237 + }, + { + "forward": 0.0022806250053690746 + }, + { + "forward": 0.002118250005878508 + }, + { + "forward": 0.001931042002979666 + }, + { + "forward": 0.002110250003170222 + }, + { + "forward": 0.003362292001838796 + }, + { + "forward": 0.0028866250067949295 + } + ], + "forward_ms": { + "count": 10, + "median": 2.1576459985226393, + "p95": 3.1482418540690555, + "min": 1.8057910056086257, + "max": 3.362292001838796, + "iqr": 0.46240624942583963, + "mean": 2.3092709030606784, + "cv": 0.21020133034584723 + }, + "memory": { + "peak_extra_bytes": 0, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "sdpa", + "batch": 1, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.0024748750001890585, + "prefill": 0.0016719579871278256, + "decode_steps": [ + 0.0016905840020626783, + 0.002105833002133295, + 0.0016193749906960875, + 0.0015193750004982576, + 0.0015151250117924064, + 0.001807958004064858, + 0.00155050000466872, + 0.0020134590013185516, + 0.001543833001051098, + 0.0015586249937769026, + 0.001759624996338971, + 0.0015816669911146164, + 0.0015648750122636557, + 0.002546999996411614, + 0.0024095419939840212, + 0.002376167001784779, + 0.0016910830017877743, + 0.001568583000334911, + 0.0026580410049064085, + 0.0016926249954849482, + 0.0015698329953011125, + 0.0016392910038121045, + 0.0020882920071016997, + 0.002969207998830825, + 0.0033734160097083077, + 0.002873249992262572, + 0.0016329170030076057, + 0.001643666997551918, + 0.0015205420058919117, + 0.0015077920106705278, + 0.0020661250018747523, + 0.001727459006360732 + ] + }, + { + "forward": 0.0020873329922324046, + "prefill": 0.0020887910068267956, + "decode_steps": [ + 0.002925207998487167, + 0.0019219160021748394, + 0.00184395800170023, + 0.0019164589903084561, + 0.001910667007905431, + 0.0017726250080158934, + 0.0019933750008931383, + 0.0014950830081943423, + 0.0023781659983797, + 0.0017166250036098063, + 0.0015397499955724925, + 0.0016312079969793558, + 0.0017853750032372773, + 0.0015074999973876402, + 0.0015464169991901144, + 0.0015403750003315508, + 0.0015192080027190968, + 0.001813166993088089, + 0.001675999999861233, + 0.0015289169969037175, + 0.001906457997392863, + 0.0015458749985555187, + 0.0015834999940125272, + 0.0015479580033570528, + 0.0015170410042628646, + 0.0014890000020386651, + 0.0022680419933749363, + 0.0016526249964954332, + 0.0015683339879615232, + 0.0018346249999012798, + 0.0016011669940780848, + 0.0015255000034812838 + ] + }, + { + "forward": 0.0017661669990047812, + "prefill": 0.0019512919971020892, + "decode_steps": [ + 0.001967375006643124, + 0.0017301670013694093, + 0.001716624989057891, + 0.0016039170004660264, + 0.0014826249971520156, + 0.0018217909964732826, + 0.0015734170010546222, + 0.0015945409977575764, + 0.001538458003778942, + 0.001527457992779091, + 0.0015651660069124773, + 0.0015640829951735213, + 0.0015183329960564151, + 0.0015540409949608147, + 0.0015480000001844019, + 0.0015856250101933256, + 0.0017910419992404059, + 0.0016079579945653677, + 0.0015673750021960586, + 0.0015283329994417727, + 0.001552582994918339, + 0.001599249997525476, + 0.0015485000039916486, + 0.001537167001515627, + 0.0015135830035433173, + 0.001524999999674037, + 0.0016117089980980381, + 0.0016443749918835238, + 0.0015032499941298738, + 0.0016164590051630512, + 0.0015398750110762194, + 0.001653958999668248 + ] + }, + { + "forward": 0.0016551250009797513, + "prefill": 0.0016879589966265485, + "decode_steps": [ + 0.001588041996001266, + 0.0015856660029385239, + 0.0015194589941529557, + 0.0016571250016568229, + 0.0016462909989058971, + 0.0016611250030109659, + 0.0015414590016007423, + 0.001545209001051262, + 0.0016255829978035763, + 0.0015184999938355759, + 0.0015840830019442365, + 0.0017408749990863726, + 0.0015089170046849176, + 0.001494334006565623, + 0.0015231249999487773, + 0.0015438339905813336, + 0.0016726250032661483, + 0.001580416996148415, + 0.001513041992438957, + 0.0015391250053653494, + 0.0015750419988762587, + 0.0015152080013649538, + 0.0015367090090876445, + 0.001513041992438957, + 0.0015065830084495246, + 0.001598666000063531, + 0.0015314590127673, + 0.0016690000047674403, + 0.001524624996818602, + 0.0015307910070987418, + 0.0017265829956158996, + 0.0015001669962657616 + ] + }, + { + "forward": 0.0021151669934624806, + "prefill": 0.0020067080040462315, + "decode_steps": [ + 0.0019802919996436685, + 0.0015347500011557713, + 0.0014812920126132667, + 0.0015504589973716065, + 0.0016676249942975119, + 0.0015842080028960481, + 0.0016199579986277968, + 0.0016429170063929632, + 0.001555666996864602, + 0.001631999999517575, + 0.0015824159927433357, + 0.0015039160061860457, + 0.0021715420007240027, + 0.0015959589945850894, + 0.0015286249981727451, + 0.0016142079985002056, + 0.001630458005820401, + 0.0017087089945562184, + 0.0015465420001419261, + 0.0015237920015351847, + 0.0015661670040572062, + 0.0015247499977704138, + 0.0016614999913144857, + 0.0017483750125393271, + 0.0015644999948563054, + 0.0015696670016041026, + 0.0015572499978588894, + 0.0015427089965669438, + 0.0015202920039882883, + 0.0015772920014569536, + 0.0015172499988693744, + 0.0018444579909555614 + ] + }, + { + "forward": 0.001717374994768761, + "prefill": 0.001695250000921078, + "decode_steps": [ + 0.0017322920029982924, + 0.0016530830034753308, + 0.0016077920008683577, + 0.0015502079913858324, + 0.001539625009172596, + 0.0016216249932767823, + 0.0015040410071378574, + 0.0015125420031836256, + 0.0015466670010937378, + 0.0014961249980842695, + 0.0016540409997105598, + 0.0015399590047309175, + 0.0015201670030364767, + 0.001725708003505133, + 0.0015254999889293686, + 0.0015017920086393133, + 0.0016185840067919344, + 0.001585332996910438, + 0.0016237920062849298, + 0.0015545420028502122, + 0.0015491660014959052, + 0.0015394999936688691, + 0.0014983750006649643, + 0.001635624998016283, + 0.0015185829979600385, + 0.001524999999674037, + 0.0015502079913858324, + 0.0015364579885499552, + 0.0016747920017223805, + 0.001524582999991253, + 0.001508249988546595, + 0.0015191249985946342 + ] + }, + { + "forward": 0.0018717500061029568, + "prefill": 0.0016553339955862612, + "decode_steps": [ + 0.0016347079945262522, + 0.001695790997473523, + 0.0023689160007052124, + 0.0017222919996129349, + 0.0015952909889165312, + 0.001540792000014335, + 0.0015614159929100424, + 0.0014762909995624796, + 0.0015285409899661317, + 0.0020776250021299347, + 0.001560459000756964, + 0.002453082997817546, + 0.0015538750012638047, + 0.001582875003805384, + 0.0015475830005016178, + 0.0015167499950621277, + 0.001662124996073544, + 0.0015804580034455284, + 0.001499416001024656, + 0.0015324999985750765, + 0.0015776249929331243, + 0.00161087499873247, + 0.0015683749952586368, + 0.0015907500055618584, + 0.0016844159981701523, + 0.0015449589991476387, + 0.0015621250058757141, + 0.001548209009342827, + 0.0015730409941170365, + 0.0015324169944506139, + 0.0015151670086197555, + 0.0015347500011557713 + ] + }, + { + "forward": 0.0017183750023832545, + "prefill": 0.001767624999047257, + "decode_steps": [ + 0.0018739160004770383, + 0.0016212920018006116, + 0.00205650000134483, + 0.0017878329963423312, + 0.0015902910090517253, + 0.0017154999950435013, + 0.0017090000037569553, + 0.0017103749996749684, + 0.0015250410069711506, + 0.0015002090076450258, + 0.0017111249908339232, + 0.00160395800776314, + 0.0015307919966289774, + 0.0016590000013820827, + 0.001554083006340079, + 0.0015018749982118607, + 0.0015621670027030632, + 0.0016787920030765235, + 0.0015201669884845614, + 0.001568583000334911, + 0.0016473750001750886, + 0.0015397499955724925, + 0.0016329580103047192, + 0.0015205419913399965, + 0.0015366669977083802, + 0.001600999996298924, + 0.00154116700286977, + 0.0015092919929884374, + 0.0015370829933090135, + 0.0015322919934988022, + 0.0015793330094311386, + 0.0015298329963115975 + ] + }, + { + "forward": 0.0026694580010371283, + "prefill": 0.0025969999987864867, + "decode_steps": [ + 0.002714624992222525, + 0.0026270840025972575, + 0.0019196250068489462, + 0.00165908299095463, + 0.002286375005496666, + 0.0016822079924168065, + 0.0017786660027923062, + 0.0016607090074103326, + 0.001516916003311053, + 0.002041832995018922, + 0.0015399999974761158, + 0.0015398749965243042, + 0.0016073750011855736, + 0.0021022090077167377, + 0.0015426250029122457, + 0.0015296250057872385, + 0.001607291997061111, + 0.0016077499894890934, + 0.0023347079986706376, + 0.004021542001282796, + 0.001810832996852696, + 0.0032377909956267104, + 0.003270874993177131, + 0.0025895409926306456, + 0.0016887499950826168, + 0.0016272079956252128, + 0.0016947910044109449, + 0.0015881250001257285, + 0.0015841249987715855, + 0.00157200000830926, + 0.0015280410007108003, + 0.0015694159956183285 + ] + }, + { + "forward": 0.001777500001480803, + "prefill": 0.0017221250018337741, + "decode_steps": [ + 0.0024692910083103925, + 0.001598290997208096, + 0.0015730829909443855, + 0.0017698750016279519, + 0.001507082997704856, + 0.001560624994453974, + 0.0015094999980647117, + 0.001692582998657599, + 0.0015483329916605726, + 0.0017537910025566816, + 0.0015767080039950088, + 0.0015337079967139289, + 0.0017327499954262748, + 0.0015416250098496675, + 0.0014931659970898181, + 0.0020429589931154624, + 0.0015593749994877726, + 0.0015772500046296045, + 0.001752416996168904, + 0.0016961250075837597, + 0.001594500005012378, + 0.0015626669919583946, + 0.0015619579935446382, + 0.0015672079898649827, + 0.001863249999587424, + 0.0015499580040341243, + 0.0016115000034915283, + 0.0016920829948503524, + 0.0016808749933261424, + 0.0015780419926159084, + 0.0015763749979669228, + 0.0015356660005636513 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 1.8246250037918799, + "p95": 2.581895650655497, + "min": 1.6551250009797513, + "max": 2.6694580010371283, + "iqr": 0.37788549161632545, + "mean": 1.985312499164138, + "cv": 0.17538678920360137 + }, + "prefill_ms": { + "count": 10, + "median": 1.7448750004405156, + "p95": 2.3683059524046253, + "min": 1.6553339955862612, + "max": 2.5969999987864867, + "iqr": 0.3030722546100151, + "mean": 1.8844041987904347, + "cv": 0.15632037357483122 + }, + "decode_ms_per_step": { + "count": 320, + "median": 1.5798750027897768, + "p95": 2.376266951614525, + "min": 1.4762909995624796, + "max": 4.021542001282796, + "iqr": 0.15356275252997875, + "mean": 1.6910771809307334, + "cv": 0.19002037129730037 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 613.1527423691311, + "p95": 636.8331517711563, + "min": 507.2792915884316, + "max": 637.538044931124, + "iqr": 43.341433031001884, + "mean": 594.8849188005048, + "cv": 0.07810158170228632 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 592.3509394852307, + "p95": 616.0881385841652, + "min": 487.2209483996628, + "max": 616.7089623819286, + "iqr": 41.22003959321853, + "mean": 574.8860501441121, + "cv": 0.07826533411965429 + }, + "memory": { + "peak_extra_bytes": 0, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "reference", + "batch": 1, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.009875749994534999, + "prefill": 0.002761958006885834, + "decode_steps": [ + 0.014783916994929314, + 0.005006125007639639, + 0.005107375007355586, + 0.008006375006516464, + 0.009252417003153823, + 0.006219165996299125, + 0.005364834010833874, + 0.005934624990914017, + 0.005311083004926331, + 0.004837000000406988, + 0.005311832996085286, + 0.004277333006029949, + 0.0038579170068260282, + 0.0054467500012833625, + 0.0041887920087901875, + 0.004596749990014359, + 0.004283375004888512, + 0.004583750007441267, + 0.004411874993820675, + 0.003906540994648822, + 0.004649666007026099, + 0.004605082998750731, + 0.004406749998452142, + 0.004160625001532026, + 0.004468125000130385, + 0.004488582999329083, + 0.004694041999755427, + 0.004266999996616505, + 0.004413417002069764, + 0.004224834003252909, + 0.0045420410024235025, + 0.004678375000366941 + ] + }, + { + "forward": 0.006296166990068741, + "prefill": 0.002257874992210418, + "decode_steps": [ + 0.00508516700938344, + 0.00486941599228885, + 0.004445625003427267, + 0.0042127920023631305, + 0.004554625003947876, + 0.004244209005264565, + 0.0037757499958388507, + 0.004158833005931228, + 0.004549582998151891, + 0.004512582992902026, + 0.004334583994932473, + 0.00439300001016818, + 0.004225874989060685, + 0.004384292013128288, + 0.004097958008060232, + 0.0038269170036073774, + 0.004156124996370636, + 0.004369125002995133, + 0.004228791003697552, + 0.0042792919994099066, + 0.004350667004473507, + 0.004361667000921443, + 0.004343791995779611, + 0.0043156250030733645, + 0.004352084011770785, + 0.004139833006775007, + 0.004096416989341378, + 0.004125082996324636, + 0.004546249998384155, + 0.004536499996902421, + 0.004698333999840543, + 0.004271999991033226 + ] + }, + { + "forward": 0.005270207999274135, + "prefill": 0.0029024590039625764, + "decode_steps": [ + 0.0045688339887419716, + 0.004264708011760376, + 0.004061458996147849, + 0.004125459003262222, + 0.004409291999763809, + 0.0037604170065606013, + 0.004311542012146674, + 0.004138666990911588, + 0.004187665996141732, + 0.005188959010411054, + 0.014176999990013428, + 0.005229417001828551, + 0.006369374998030253, + 0.0069780000048922375, + 0.00522345800709445, + 0.004584290989441797, + 0.004115916002774611, + 0.004245332995196804, + 0.005035958994994871, + 0.005439291999209672, + 0.004882041001110338, + 0.004381834005471319, + 0.004574584003421478, + 0.004356917008408345, + 0.004022582987090573, + 0.004351916999439709, + 0.0042821669921977445, + 0.004171166001469828, + 0.003927541998564266, + 0.00422866600274574, + 0.004288832991733216, + 0.004067665999173187 + ] + }, + { + "forward": 0.006037832994479686, + "prefill": 0.0023271249956451356, + "decode_steps": [ + 0.0050489589921198785, + 0.004428542000823654, + 0.004481458992813714, + 0.0043988749966956675, + 0.004329832998337224, + 0.004299791995435953, + 0.004228082994814031, + 0.004176916001597419, + 0.004870916993240826, + 0.005091416998766363, + 0.004760916999657638, + 0.004606167000019923, + 0.0046098750026430935, + 0.004412124995724298, + 0.0044064579997211695, + 0.004173041001195088, + 0.003986582989455201, + 0.0037186660047154874, + 0.0040945000073406845, + 0.00397879199590534, + 0.0042692499991972, + 0.0038282919995253906, + 0.005086499993922189, + 0.004587250004988164, + 0.0045087500038789585, + 0.005050124993431382, + 0.004644625005312264, + 0.004413624992594123, + 0.0047382500051753595, + 0.004444084013812244, + 0.004297834006138146, + 0.004188541992334649 + ] + }, + { + "forward": 0.005561458005104214, + "prefill": 0.0025265419972129166, + "decode_steps": [ + 0.005158291998668574, + 0.00460216699866578, + 0.004797708010300994, + 0.0047235410020221025, + 0.004015166996396147, + 0.004135334005695768, + 0.004327374990680255, + 0.004291334000299685, + 0.004368582987808622, + 0.0038004999951226637, + 0.004246749987942167, + 0.004108000008272938, + 0.004260916990460828, + 0.004306833012378775, + 0.0043963750067632645, + 0.0042289160046493635, + 0.0039215839933604, + 0.0040573330043116584, + 0.004113458009669557, + 0.004175540991127491, + 0.0040646249981364235, + 0.004083624997292645, + 0.004082291008671746, + 0.0036844580026809126, + 0.004056541001773439, + 0.0044779169984394684, + 0.004578583990223706, + 0.00430979099473916, + 0.004028540992294438, + 0.004269833007128909, + 0.00426683398836758, + 0.00422308300039731 + ] + }, + { + "forward": 0.005052207998232916, + "prefill": 0.0026359579933341593, + "decode_steps": [ + 0.003726875002030283, + 0.004058541002450511, + 0.004268167002010159, + 0.0035697910061571747, + 0.003557041010935791, + 0.0040101670019794255, + 0.004015707992948592, + 0.004095291995326988, + 0.0037799170095240697, + 0.004128000000491738, + 0.00414291700872127, + 0.0039661250048084185, + 0.0036557080020429567, + 0.004163916994002648, + 0.00410287499835249, + 0.004038250001030974, + 0.003683958013425581, + 0.004044833010993898, + 0.004011917000752874, + 0.003934000007575378, + 0.0042322079971199855, + 0.0039773750031599775, + 0.004333083998062648, + 0.004473332999623381, + 0.0040303749992745, + 0.004208375001326203, + 0.00416008300089743, + 0.004238542009261437, + 0.004242666997015476, + 0.003923875003238209, + 0.004035249992739409, + 0.004079499994986691 + ] + }, + { + "forward": 0.0058939159935107455, + "prefill": 0.002220915994257666, + "decode_steps": [ + 0.004536082997219637, + 0.004483083001105115, + 0.004633709002519026, + 0.004179208001005463, + 0.004578916996251792, + 0.004221707989927381, + 0.003997124993475154, + 0.004119042001548223, + 0.0042392500035930425, + 0.004279249988030642, + 0.004529207988525741, + 0.00498591699579265, + 0.004459916002815589, + 0.004198249996989034, + 0.004196915993816219, + 0.004375082993647084, + 0.00410195799486246, + 0.004266208008630201, + 0.0040969999972730875, + 0.004228999998304062, + 0.004223332987749018, + 0.003878832998452708, + 0.0036177500005578622, + 0.004169332998571917, + 0.004218166999635287, + 0.003587957995478064, + 0.00413391699839849, + 0.004076625002198853, + 0.004083332998561673, + 0.003547792002791539, + 0.0039867499872343615, + 0.004084041996975429 + ] + }, + { + "forward": 0.005476791993714869, + "prefill": 0.0022319169947877526, + "decode_steps": [ + 0.004244708994519897, + 0.004283000002033077, + 0.004638541999156587, + 0.004454249996342696, + 0.00467670799116604, + 0.004591040997183882, + 0.004401874990435317, + 0.004001541004981846, + 0.004228541991324164, + 0.003999833992565982, + 0.004214166008750908, + 0.00467058400681708, + 0.004661291997763328, + 0.004466500002308749, + 0.004210790997603908, + 0.00418625000747852, + 0.0038244580064201728, + 0.004195749992504716, + 0.004236791995936073, + 0.004180250005447306, + 0.0043040000018663704, + 0.004278833002899773, + 0.004103957995539531, + 0.004117874996154569, + 0.004209667007671669, + 0.004075290999026038, + 0.004651082999771461, + 0.004103041996131651, + 0.0037792910006828606, + 0.004287957999622449, + 0.004461874996195547, + 0.004227375000482425 + ] + }, + { + "forward": 0.005196208003326319, + "prefill": 0.0023235830012708902, + "decode_steps": [ + 0.0038450829888461158, + 0.00400941600673832, + 0.004677416000049561, + 0.004230040998663753, + 0.0044102500105509534, + 0.004685291991336271, + 0.004318582999985665, + 0.004329916002461687, + 0.0041284589970018715, + 0.004368082998553291, + 0.004082291998201981, + 0.0037427079951157793, + 0.0039116670086514205, + 0.004474792003748007, + 0.004309583004214801, + 0.003930582999601029, + 0.0042964589956682175, + 0.00454270800400991, + 0.0043872499954886734, + 0.004257584005245008, + 0.004555208011879586, + 0.0041374580032425, + 0.004472500004339963, + 0.004069375005201437, + 0.005365667006117292, + 0.005001208002795465, + 0.005097041997942142, + 0.009148375000222586, + 0.005019374992116354, + 0.004761749994941056, + 0.004830999998375773, + 0.006484333993284963 + ] + }, + { + "forward": 0.0058320840034866706, + "prefill": 0.0024668750120326877, + "decode_steps": [ + 0.0050185830041300505, + 0.00432699998782482, + 0.004757458009407856, + 0.004674416995840147, + 0.004766041005495936, + 0.004797084009624086, + 0.004223875002935529, + 0.004434333008248359, + 0.00460329199268017, + 0.0041254169918829575, + 0.005028708997997455, + 0.004616833990439773, + 0.004696667005191557, + 0.0043043750047218055, + 0.0044039999920642, + 0.004348832997493446, + 0.003974167004344054, + 0.004138249991228804, + 0.0046432079980149865, + 0.0044579170062206686, + 0.004436500006704591, + 0.004410958994412795, + 0.004035750011098571, + 0.004687957989517599, + 0.0042539160058368, + 0.004299916996387765, + 0.004296084007364698, + 0.003619249997427687, + 0.004257958993548527, + 0.004319791987654753, + 0.004633416989236139, + 0.004380583006422967 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 5.696771004295442, + "p95": 8.26493764252518, + "min": 5.052207998232916, + "max": 9.875749994535, + "iqr": 0.6799997463531327, + "mean": 6.0492623975733295, + "cv": 0.23162214910134013 + }, + "prefill_ms": { + "count": 10, + "median": 2.3970000038389117, + "p95": 2.8392335552780423, + "min": 2.220915994257666, + "max": 2.9024590039625764, + "iqr": 0.33430199982831255, + "mean": 2.4655207991600037, + "cv": 0.09639746208922235 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.2883954956778325, + "p95": 5.233500301983441, + "min": 3.547792002791539, + "max": 14.783916994929314, + "iqr": 0.4350417584646493, + "mean": 4.4732045560976985, + "cv": 0.22436120524687922 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 227.93609154226596, + "p95": 243.7628899845872, + "min": 190.15205587540203, + "max": 248.2762912957815, + "iqr": 16.10187552444455, + "mean": 224.7876456599243, + "cv": 0.07565054698301057 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 224.16730181360379, + "p95": 239.2822611894992, + "min": 187.08162550663815, + "max": 243.3004483363687, + "iqr": 15.371284616601429, + "mean": 220.96677527153724, + "cv": 0.07512471430236507 + }, + "memory": { + "peak_extra_bytes": 0, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "auto", + "batch": 1, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.002498582995031029 + }, + { + "forward": 0.0031310420017689466 + }, + { + "forward": 0.002629541006172076 + }, + { + "forward": 0.0023861670051701367 + }, + { + "forward": 0.0029177910037105903 + }, + { + "forward": 0.002855541999451816 + }, + { + "forward": 0.002471915999194607 + }, + { + "forward": 0.0025929999974323437 + }, + { + "forward": 0.002666333006345667 + }, + { + "forward": 0.002420666001853533 + } + ], + "forward_ms": { + "count": 10, + "median": 2.6112705018022098, + "p95": 3.0350790526426863, + "min": 2.3861670051701367, + "max": 3.1310420017689466, + "iqr": 0.3296570030215662, + "mean": 2.6570581016130745, + "cv": 0.09095402533304904 + }, + "memory": { + "peak_extra_bytes": 8650752, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "sdpa", + "batch": 1, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.002173334010876715, + "prefill": 0.0022003339981893077, + "decode_steps": [ + 0.0015961250028340146, + 0.0016951669967966154, + 0.001684000002569519, + 0.0015697079943493009, + 0.0015185420052148402, + 0.0016031669947551563, + 0.0015255420003086329, + 0.0015717079950263724, + 0.0017619160062167794, + 0.0015669580025132746, + 0.0016852909902809188, + 0.0016386250063078478, + 0.001575292000779882, + 0.001528166001662612, + 0.0016033749998314306, + 0.0015736250061308965, + 0.0017251669924007729, + 0.0015917080017970875, + 0.0016097920015454292, + 0.0015597499877912924, + 0.0017534159997012466, + 0.001668750002863817, + 0.0015721669915365055, + 0.00169779198768083, + 0.0017085830040741712, + 0.001561333003337495, + 0.0015549579984508455, + 0.001695042010396719, + 0.0015783749986439943, + 0.0015512079990003258, + 0.001559041003929451, + 0.0016307909972965717 + ] + }, + { + "forward": 0.002230457990663126, + "prefill": 0.0022432500118156895, + "decode_steps": [ + 0.0018507920030970126, + 0.0015433749940712005, + 0.0016692500066710636, + 0.0015743749972898513, + 0.0016551250009797513, + 0.001579250005306676, + 0.0016153330070665106, + 0.0017054170020855963, + 0.0016089999990072101, + 0.0015815000078873709, + 0.0015544170018984005, + 0.0016414999990956858, + 0.00153104099445045, + 0.0016006250079954043, + 0.00161070900503546, + 0.0015960840100888163, + 0.0016623749979771674, + 0.0015130410029087216, + 0.001795999996829778, + 0.001604624994797632, + 0.0016286669997498393, + 0.0016835419955896214, + 0.0015594170108670369, + 0.0015390000044135377, + 0.0016327080084010959, + 0.0015848750044824556, + 0.0017019169899867848, + 0.001563333993544802, + 0.0015624590014340356, + 0.001557333001983352, + 0.0018018330010818318, + 0.0018138339946744964 + ] + }, + { + "forward": 0.0021881660068174824, + "prefill": 0.002186666999477893, + "decode_steps": [ + 0.0018493749957997352, + 0.0017602909938432276, + 0.001686832998530008, + 0.0016026249941205606, + 0.0015799579996382818, + 0.0016125419933814555, + 0.0015724999975645915, + 0.001669666002271697, + 0.0015667499974370003, + 0.0015702500095358118, + 0.0016127080016303807, + 0.0015462079900316894, + 0.0015583330095978454, + 0.0017011660092975944, + 0.001550332992337644, + 0.0017229579971171916, + 0.0015583329950459301, + 0.0015696250047767535, + 0.001584624988026917, + 0.0015568329981761053, + 0.0015784589922986925, + 0.0015525839990004897, + 0.0015920420119073242, + 0.0015620409976691008, + 0.0015692909946665168, + 0.0015604159998474643, + 0.0015495830011786893, + 0.0015935419942252338, + 0.0015393749927170575, + 0.0015481669979635626, + 0.00156170800619293, + 0.001586166996276006 + ] + }, + { + "forward": 0.002029707990004681, + "prefill": 0.0022674999927403405, + "decode_steps": [ + 0.0017656249983701855, + 0.0015700410003773868, + 0.0016823750047478825, + 0.0015820420085219666, + 0.0015592089912388474, + 0.0016261660057352856, + 0.0016154589975485578, + 0.0015615000011166558, + 0.001600834002601914, + 0.0016811660025268793, + 0.0015532079996773973, + 0.0015508749929722399, + 0.0015898749988991767, + 0.001758333994075656, + 0.0015780419926159084, + 0.0015770409954711795, + 0.0015427500038640574, + 0.0016205839929170907, + 0.0016681659908499569, + 0.001579583011334762, + 0.0015615829906892031, + 0.0015902079903753474, + 0.0016798330034362152, + 0.0015690000000176951, + 0.0016846660000737756, + 0.0015269580035237595, + 0.0015524169866694137, + 0.001546500003314577, + 0.0015525409980909899, + 0.001524958002846688, + 0.0016682909918017685, + 0.0015426249883603305 + ] + }, + { + "forward": 0.002200332994107157, + "prefill": 0.0021855000086361542, + "decode_steps": [ + 0.001828749998821877, + 0.0015764580020913854, + 0.0016372920072171837, + 0.0016923329967539757, + 0.001658541994402185, + 0.00172433300758712, + 0.0015937920106807724, + 0.0015540000022156164, + 0.0016652079939376563, + 0.0016363340109819546, + 0.0017453339969506487, + 0.001558290998218581, + 0.0015483749884879217, + 0.0015543329936917871, + 0.0015800420078448951, + 0.0016873330023372546, + 0.0015720840019639581, + 0.0016027919918997213, + 0.001578499999595806, + 0.001598125003511086, + 0.0015229590062517673, + 0.0015669999993406236, + 0.0015902919985819608, + 0.0015842909924685955, + 0.0015455829998245463, + 0.0015419169940287247, + 0.0015575830038869753, + 0.0017496669897809625, + 0.001557333001983352, + 0.001528958004200831, + 0.001585707999765873, + 0.0016067909891717136 + ] + }, + { + "forward": 0.0024594159913249314, + "prefill": 0.0025283750001108274, + "decode_steps": [ + 0.0019876660080626607, + 0.001782874998752959, + 0.0022328339982777834, + 0.0016441249899799004, + 0.001865124999312684, + 0.0017949999892152846, + 0.001800375001039356, + 0.0015698329953011125, + 0.0015423750010086223, + 0.001708791998680681, + 0.0017445830017095432, + 0.0015850829950068146, + 0.0015911249938653782, + 0.00152658399019856, + 0.0015660000062780455, + 0.0016670830082148314, + 0.0015851669886615127, + 0.0016402909968746826, + 0.001965332994586788, + 0.0016867499944055453, + 0.001530166991869919, + 0.0015172499988693744, + 0.0015249170101014897, + 0.0015588750102324411, + 0.001629291000426747, + 0.0015441249997820705, + 0.0015700000076321885, + 0.001598499991814606, + 0.0015657079929951578, + 0.001708791998680681, + 0.0016058750043157488, + 0.0015461670118384063 + ] + }, + { + "forward": 0.002239540990558453, + "prefill": 0.0021752920001745224, + "decode_steps": [ + 0.0017038339865393937, + 0.001632041996344924, + 0.0015672499866923317, + 0.0015714170003775507, + 0.001560708013130352, + 0.0015526670031249523, + 0.0015423750010086223, + 0.0016489169938722625, + 0.0015079580043675378, + 0.001537167001515627, + 0.0015383339923573658, + 0.0016987919952953234, + 0.001574541995069012, + 0.0015803749993210658, + 0.001637374996789731, + 0.0015424169978359714, + 0.0015629999979864806, + 0.0016890410042833537, + 0.0015277080092346296, + 0.0019814590050373226, + 0.0015580829931423068, + 0.001521540994872339, + 0.00156170800619293, + 0.001491874994826503, + 0.0015407499886350706, + 0.0015648749977117404, + 0.0015358749951701611, + 0.00165108400688041, + 0.0015920409932732582, + 0.0017504590068710968, + 0.0015512910031247884, + 0.001577791990712285 + ] + }, + { + "forward": 0.0025808339996729046, + "prefill": 0.0022514580050483346, + "decode_steps": [ + 0.001854667003499344, + 0.0018887920014094561, + 0.0023899590014480054, + 0.0019564579997677356, + 0.001985499999136664, + 0.001934667001478374, + 0.0019148749997839332, + 0.0019480409973766655, + 0.001899415990919806, + 0.001561291006510146, + 0.0017335840093437582, + 0.0015392499917652458, + 0.0016909160040086135, + 0.0016195839998545125, + 0.001577332994202152, + 0.0015596670127706602, + 0.0015845000016270205, + 0.0016052499995566905, + 0.0015629999979864806, + 0.0015656670002499595, + 0.0015764999989187345, + 0.0016902079951250926, + 0.0015734999906271696, + 0.0016366670024581254, + 0.001597209004103206, + 0.0015840410051168874, + 0.0015474999963771552, + 0.0015560830070171505, + 0.0015497910062549636, + 0.0015732499887235463, + 0.0015949579974403605, + 0.0015502500027650967 + ] + }, + { + "forward": 0.0024245410022558644, + "prefill": 0.0024373749911319464, + "decode_steps": [ + 0.0018227079999633133, + 0.0016707920003682375, + 0.0015825829905224964, + 0.0017277919978369027, + 0.0016182499966816977, + 0.0015806670126039535, + 0.0015854170051170513, + 0.001658707988099195, + 0.0016703330038581043, + 0.0015171249979175627, + 0.0015534580015810207, + 0.0018415840022498742, + 0.001586124999448657, + 0.001789332993212156, + 0.0015626250096829608, + 0.0021912500087637454, + 0.0015667920088162646, + 0.0015567080117762089, + 0.001527415995951742, + 0.0016202080005314201, + 0.0016115829930640757, + 0.001590500003658235, + 0.0016824589984025806, + 0.001542500001960434, + 0.0017130829946836457, + 0.0015856670070206746, + 0.0015987910010153428, + 0.0016457500023534521, + 0.0015878749982221052, + 0.0016158330108737573, + 0.0016811249952297658, + 0.0015502090100198984 + ] + }, + { + "forward": 0.002197458001319319, + "prefill": 0.0022233329946175218, + "decode_steps": [ + 0.0017068339948309585, + 0.0015733339969301596, + 0.0015525409980909899, + 0.0015742090035928413, + 0.0016055830055847764, + 0.0015422500000568107, + 0.0015433749940712005, + 0.001581374992383644, + 0.0015812499914318323, + 0.0015406250022351742, + 0.001557708004838787, + 0.0015545839996775612, + 0.001520958001492545, + 0.00155237500439398, + 0.0016223749989876524, + 0.0015622499922756106, + 0.0015250830037984997, + 0.0015330000023823231, + 0.0015590420080116019, + 0.0016808329964987934, + 0.0015436249959748238, + 0.0015177910099737346, + 0.0016265829908661544, + 0.0015291250019799918, + 0.0015317080105887726, + 0.002105791005305946, + 0.0015791250043548644, + 0.0015322090039262548, + 0.0015652909933123738, + 0.0018235000025015324, + 0.0015975419955793768, + 0.0017664170009084046 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 2.2153954923851416, + "p95": 2.5261958959163167, + "min": 2.029707990004681, + "max": 2.5808339996729046, + "iqr": 0.18780199388857, + "mean": 2.2723788977600634, + "cv": 0.07226745601955976 + }, + "prefill_ms": { + "count": 10, + "median": 2.2332915032166056, + "p95": 2.487424996070331, + "min": 2.1752920001745224, + "max": 2.5283750001108274, + "iqr": 0.07340574666159227, + "mean": 2.2699084001942538, + "cv": 0.052110217430724165 + }, + "decode_ms_per_step": { + "count": 320, + "median": 1.5841659987927414, + "p95": 1.8509857531171292, + "min": 1.491874994826503, + "max": 2.3899590014480054, + "iqr": 0.11052048648707569, + "mean": 1.626635124648601, + "cv": 0.07231526303333381 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 619.8842455175121, + "p95": 626.0422845633752, + "min": 588.2096247402043, + "max": 626.7780916914086, + "iqr": 15.126201173112463, + "mean": 615.0122736522331, + "cv": 0.02087096914924733 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 594.6249054616076, + "p95": 600.2445776423053, + "min": 564.8338137629061, + "max": 601.1642603375158, + "iqr": 15.847657919104222, + "mean": 589.3232254163797, + "cv": 0.02132267953218782 + }, + "memory": { + "peak_extra_bytes": 0, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "reference", + "batch": 1, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.007040540993330069, + "prefill": 0.003311041000415571, + "decode_steps": [ + 0.0046683749969815835, + 0.0043872919923160225, + 0.004429124994203448, + 0.004388459012261592, + 0.004216916000586934, + 0.004300166998291388, + 0.004091124996193685, + 0.004222709001624025, + 0.004220040995278396, + 0.004207082994980738, + 0.0041400839982088655, + 0.003547250002156943, + 0.004522749994066544, + 0.004340125000453554, + 0.004135082999709994, + 0.0034968330000992864, + 0.004117666001548059, + 0.004172500004642643, + 0.0044039999920642, + 0.0037962920032441616, + 0.0042465000005904585, + 0.0046167499967850745, + 0.004129916007514112, + 0.004396958000143059, + 0.004187750004348345, + 0.0043480419990373775, + 0.0042253330029780045, + 0.004335374993388541, + 0.004196833004243672, + 0.004076332988915965, + 0.00412887500715442, + 0.0041514999902574345 + ] + }, + { + "forward": 0.006044291993021034, + "prefill": 0.0033412920020055026, + "decode_steps": [ + 0.0045997500128578395, + 0.004271749989129603, + 0.004069875009008683, + 0.003976708991103806, + 0.004299165986594744, + 0.004124541999772191, + 0.004174417001195252, + 0.00426287499431055, + 0.004046374990139157, + 0.003987125004641712, + 0.004257375010638498, + 0.0040613750024931505, + 0.003602875003707595, + 0.00407054100651294, + 0.0037904170021647587, + 0.004423917009262368, + 0.00416475000383798, + 0.003951749997213483, + 0.004261249996488914, + 0.004290540993679315, + 0.0037745839945273474, + 0.0040282499976456165, + 0.005093833999126218, + 0.003968958000768907, + 0.0045297079923329875, + 0.004604333997122012, + 0.004795917004230432, + 0.003958959001465701, + 0.0041760419990168884, + 0.004604417001246475, + 0.004447167011676356, + 0.003984458991908468 + ] + }, + { + "forward": 0.007392792002065107, + "prefill": 0.005039208001107909, + "decode_steps": [ + 0.004229124999255873, + 0.004586459006532095, + 0.0043493330013006926, + 0.004227583995088935, + 0.004375999997137114, + 0.004410583002027124, + 0.0039745000103721395, + 0.004402375008794479, + 0.0045888750028098, + 0.004653540992876515, + 0.004454750000149943, + 0.004317166996770538, + 0.011888833992998116, + 0.00610124999366235, + 0.005079957991256379, + 0.0054322920041158795, + 0.005524166001123376, + 0.004983709004591219, + 0.004918250007904135, + 0.00443579199782107, + 0.004412750000483356, + 0.0042598749860189855, + 0.004316499995184131, + 0.00444233299640473, + 0.0040790829953039065, + 0.004333290999056771, + 0.004411041998537257, + 0.004221499999403022, + 0.004774124987307005, + 0.004469833002076484, + 0.00411429199448321, + 0.004566542003885843 + ] + }, + { + "forward": 0.006528500001877546, + "prefill": 0.0032124580029631034, + "decode_steps": [ + 0.0048030000034486875, + 0.004827374999877065, + 0.005026416998589411, + 0.004799666989129037, + 0.004816333996132016, + 0.004305915994336829, + 0.004204665994620882, + 0.004125542007386684, + 0.004164083002251573, + 0.004347582987975329, + 0.004223374999128282, + 0.00429279099626001, + 0.004391041002236307, + 0.004240750000462867, + 0.003970458012190647, + 0.00422754100873135, + 0.004490874998737127, + 0.004241082991939038, + 0.004004457994597033, + 0.004238915993482806, + 0.00427316699642688, + 0.004181707990937866, + 0.0040901250031311065, + 0.004203959004371427, + 0.004062749998411164, + 0.0041781250038184226, + 0.004145790997426957, + 0.004117708012927324, + 0.004164458005107008, + 0.004154584006755613, + 0.004034458004753105, + 0.004155917005846277 + ] + }, + { + "forward": 0.006672624993370846, + "prefill": 0.003312417000415735, + "decode_steps": [ + 0.004539165995083749, + 0.004206333003821783, + 0.004375832999357954, + 0.004186374993878417, + 0.004245667005307041, + 0.004117250005947426, + 0.00407745799748227, + 0.004144334001466632, + 0.004133166003157385, + 0.004161874996498227, + 0.0043914590060012415, + 0.00411483300558757, + 0.004067958987434395, + 0.004127749998588115, + 0.004122958009247668, + 0.004063416999997571, + 0.004039959007059224, + 0.004143332989769988, + 0.003900916999555193, + 0.004348292000941001, + 0.0043281660036882386, + 0.0043333329958841205, + 0.003952375001972541, + 0.0038779159949626774, + 0.004112499998882413, + 0.004188874998362735, + 0.004063290994963609, + 0.004138250005780719, + 0.004336666999734007, + 0.0045391669991659, + 0.004244874988216907, + 0.0042681659979280084 + ] + }, + { + "forward": 0.005726957999286242, + "prefill": 0.0032579999970039353, + "decode_steps": [ + 0.004234833002556115, + 0.003921625000657514, + 0.003610084007959813, + 0.004045833004056476, + 0.00413504199241288, + 0.004145000013522804, + 0.0040634999895701185, + 0.004002834000857547, + 0.0040901250031311065, + 0.004203417003736831, + 0.003784707994782366, + 0.004149583997786976, + 0.003991791003500111, + 0.004032458004076034, + 0.0035625409946078435, + 0.003988374999607913, + 0.004120541998418048, + 0.004227916986565106, + 0.0037059590104036033, + 0.004224999996949919, + 0.004005125010735355, + 0.004195708999759518, + 0.0037600409996230155, + 0.004085792010300793, + 0.00418287499633152, + 0.0039042499993229285, + 0.0037021250027464703, + 0.0040054169949144125, + 0.0038328749942593277, + 0.004183541997917928, + 0.004088916000910103, + 0.00414999999338761 + ] + }, + { + "forward": 0.0066500420070951805, + "prefill": 0.0031138330086832866, + "decode_steps": [ + 0.004514209009357728, + 0.00387466698884964, + 0.004265083000063896, + 0.004326750000473112, + 0.0043284589919494465, + 0.0041832919960143045, + 0.004164792000665329, + 0.003662292001536116, + 0.004301292006857693, + 0.004055291006807238, + 0.004118458993616514, + 0.0041045830002985895, + 0.004097625002032146, + 0.004185332989436574, + 0.004094999996596016, + 0.00365520799823571, + 0.004031542004668154, + 0.004279207991203293, + 0.0039050420018611476, + 0.004576166000333615, + 0.004321333006373607, + 0.004287499992642552, + 0.004598875006195158, + 0.004274459002772346, + 0.004283666989067569, + 0.004158542011282407, + 0.0041219590057153255, + 0.0039472920034313574, + 0.003954667001380585, + 0.00433579200762324, + 0.003615666995756328, + 0.00422870900365524 + ] + }, + { + "forward": 0.006285334005951881, + "prefill": 0.0033238339965464547, + "decode_steps": [ + 0.005132457998115569, + 0.004780041999765672, + 0.0044419170008040965, + 0.003929250000510365, + 0.004235125001287088, + 0.00476495799375698, + 0.0050365000060992315, + 0.004974833005690016, + 0.005047625003498979, + 0.004398917008074932, + 0.003795374999754131, + 0.004265417010174133, + 0.004149375003180467, + 0.004134000002522953, + 0.00412812500144355, + 0.00404120801249519, + 0.004129667009692639, + 0.004175666996161453, + 0.00422962500306312, + 0.0044344999914756045, + 0.004174542002147064, + 0.004579291999107227, + 0.004260916000930592, + 0.004315375001169741, + 0.004527207987848669, + 0.004302250003092922, + 0.0041250419890275225, + 0.0044064999965485185, + 0.004243749994202517, + 0.004156292008701712, + 0.004171541993855499, + 0.004136290997848846 + ] + }, + { + "forward": 0.005468374991323799, + "prefill": 0.0033796659990912303, + "decode_steps": [ + 0.004222749994369224, + 0.004077000005054288, + 0.004038084007333964, + 0.00416925000899937, + 0.004154750000452623, + 0.004092625007615425, + 0.0038583330024266616, + 0.004185750003671274, + 0.003957416003686376, + 0.004829832992982119, + 0.004276874999050051, + 0.004111166999791749, + 0.004238749999785796, + 0.003759375002118759, + 0.003928292004275136, + 0.004411250003613532, + 0.004274332997738384, + 0.004336290992796421, + 0.004072915995493531, + 0.004184500008705072, + 0.0042274579900549725, + 0.004257832988514565, + 0.004150958004174754, + 0.0041381250048289075, + 0.004054999997606501, + 0.004048333008540794, + 0.004063750006025657, + 0.004152083012741059, + 0.0040760420088190585, + 0.0041228750051232055, + 0.00412454099569004, + 0.004090333997737616 + ] + }, + { + "forward": 0.006712958987918682, + "prefill": 0.0031518339965259656, + "decode_steps": [ + 0.004730917004053481, + 0.004461791992071085, + 0.004265875002602115, + 0.0042148330103373155, + 0.0036580000014510006, + 0.004441832992597483, + 0.004109541012439877, + 0.0038914999895496294, + 0.004160959011642262, + 0.004264208997483365, + 0.003882832999806851, + 0.004135458002565429, + 0.004087499997694977, + 0.004149208005401306, + 0.0035699999862117693, + 0.004179541996563785, + 0.004921707994071767, + 0.005780958992545493, + 0.0066647079947870225, + 0.004372750001493841, + 0.004598750005243346, + 0.004318082996178418, + 0.004668082998250611, + 0.004031333010061644, + 0.004602208995493129, + 0.004393666997202672, + 0.004376249999040738, + 0.0040194160101236776, + 0.004467416001716629, + 0.004310707998229191, + 0.004423125006724149, + 0.00444158299069386 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 6.589271004486363, + "p95": 7.23427904813434, + "min": 5.468374991323799, + "max": 7.392792002065107, + "iqr": 0.5983229930279776, + "mean": 6.452241797524039, + "cv": 0.09067351807435456 + }, + "prefill_ms": { + "count": 10, + "median": 3.311729000415653, + "p95": 4.292414100200402, + "min": 3.1138330086832866, + "max": 5.039208001107909, + "iqr": 0.11308399916742928, + "mean": 3.4443583004758693, + "cv": 0.16455969079075128 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.18831250135554, + "p95": 4.834253843728221, + "min": 3.4968330000992864, + "max": 11.888833992998116, + "iqr": 0.2666252512426581, + "mean": 4.2760190093758865, + "cv": 0.12878525858554554 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 237.4786506059637, + "p95": 245.66423144493842, + "min": 208.6924188504588, + "max": 249.3419029443773, + "iqr": 10.330220062692291, + "mean": 234.3539453215862, + "cv": 0.046918405600905606 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 231.7578636995285, + "p95": 239.65120676395995, + "min": 202.05219748845724, + "max": 243.16879025406232, + "iqr": 9.945922269289014, + "mean": 228.63756770827825, + "cv": 0.04850788760607576 + }, + "memory": { + "peak_extra_bytes": 8650752, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "auto", + "batch": 1, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.003710291988682002 + }, + { + "forward": 0.003965084004448727 + }, + { + "forward": 0.003849083004752174 + }, + { + "forward": 0.003841708996333182 + }, + { + "forward": 0.003923374999430962 + }, + { + "forward": 0.005739125001127832 + }, + { + "forward": 0.0037833329988643527 + }, + { + "forward": 0.003953249994083308 + }, + { + "forward": 0.004143209007452242 + }, + { + "forward": 0.0037646249984391034 + } + ], + "forward_ms": { + "count": 10, + "median": 3.886229002091568, + "p95": 5.020962803973815, + "min": 3.710291988682002, + "max": 5.739125001127832, + "iqr": 0.164198503625812, + "mean": 4.067308499361388, + "cv": 0.14758928054818363 + }, + "memory": { + "peak_extra_bytes": 8650752, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "sdpa", + "batch": 1, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.003242666003643535, + "prefill": 0.003205167013220489, + "decode_steps": [ + 0.0017886660061776638, + 0.001725916998111643, + 0.0016872090054675937, + 0.0016675419901730493, + 0.0018546659994171932, + 0.0016681249981047586, + 0.0016583749966230243, + 0.0016982499946607277, + 0.0015664170059608296, + 0.0016843339981278405, + 0.0015811249904800206, + 0.0016160419909283519, + 0.001724750007269904, + 0.0016517499898327515, + 0.0017814590100897476, + 0.0016706669994164258, + 0.0016713330114725977, + 0.0017001250089379027, + 0.001709707998088561, + 0.00171329099975992, + 0.0017067500011762604, + 0.0017114579968620092, + 0.0015971669927239418, + 0.0017149999912362546, + 0.0017337919998681173, + 0.0015983750054147094, + 0.0017262499895878136, + 0.0017093339993152767, + 0.0017197500128531829, + 0.0015919579891487956, + 0.0016058750043157488, + 0.001705500006210059 + ] + }, + { + "forward": 0.0032142499985639006, + "prefill": 0.0032951250032056123, + "decode_steps": [ + 0.001778292004019022, + 0.001600207993760705, + 0.0016825839993543923, + 0.0017041670071193948, + 0.0019922500068787485, + 0.0016406250069849193, + 0.0016227089945459738, + 0.001854333997471258, + 0.0017368749977322295, + 0.0017149589984910563, + 0.001694874998065643, + 0.0016015419969335198, + 0.001636999993934296, + 0.0017372909933328629, + 0.0016090420103864744, + 0.0018297920032637194, + 0.0016922089998843148, + 0.001614499997231178, + 0.0016035829903557897, + 0.0017403749952791259, + 0.001708374998997897, + 0.0016977500054053962, + 0.001592582993907854, + 0.0016307500045513734, + 0.001730832998873666, + 0.0017211249942192808, + 0.0017181250004796311, + 0.0015975419955793768, + 0.0017117090028477833, + 0.001699625005130656, + 0.0017031250026775524, + 0.0017151249921880662 + ] + }, + { + "forward": 0.0040239169902633876, + "prefill": 0.004194749999442138, + "decode_steps": [ + 0.0019835409912047908, + 0.0017178330017486587, + 0.001696207997156307, + 0.0017090829933295026, + 0.0017275839927606285, + 0.0017022499960148707, + 0.0017242080066353083, + 0.001704750000499189, + 0.0016847079969011247, + 0.00159275000623893, + 0.0016682080022292212, + 0.0015717080095782876, + 0.0017855409969342873, + 0.0017225000046892092, + 0.001835999995819293, + 0.0017145000019809231, + 0.0017046249995473772, + 0.0016866669902810827, + 0.001687583004240878, + 0.0016979579959297553, + 0.001734457997372374, + 0.0017034999909810722, + 0.001601374999154359, + 0.0018315420020371675, + 0.0017574170051375404, + 0.0017400829965481535, + 0.0015954580012476072, + 0.0016071249992819503, + 0.0016082090005511418, + 0.0015888329944573343, + 0.0017041670071193948, + 0.0015607090026605874 + ] + }, + { + "forward": 0.0032007079862523824, + "prefill": 0.003192792006302625, + "decode_steps": [ + 0.0017743750067893416, + 0.0016957079933490604, + 0.001553374997456558, + 0.001701959001366049, + 0.0017102080018958077, + 0.0017058329976862296, + 0.0016878750029718503, + 0.0015792079939274117, + 0.0017067500011762604, + 0.0016208750021178275, + 0.0015739579976070672, + 0.0015755000058561563, + 0.001575250003952533, + 0.0016185410058824345, + 0.0015609169931849465, + 0.0015879159909673035, + 0.0016141670057550073, + 0.0017013750039041042, + 0.0016844589990796521, + 0.0015737079957034439, + 0.0015923339960863814, + 0.001604375007445924, + 0.001718833009363152, + 0.0016981660010060295, + 0.0016934169980231673, + 0.001570874999742955, + 0.0017034579941537231, + 0.0016343340103048831, + 0.0016952090081758797, + 0.0016830829990794882, + 0.0016487079992657527, + 0.0016722910077078268 + ] + }, + { + "forward": 0.0037181660009082407, + "prefill": 0.00352250000287313, + "decode_steps": [ + 0.0019478329923003912, + 0.001631791004911065, + 0.0017355000018142164, + 0.0017059579986380413, + 0.0016856250003911555, + 0.0017195420077769086, + 0.0015909999929135665, + 0.0016405420028604567, + 0.0015938750002533197, + 0.0017287499940721318, + 0.0017159589915536344, + 0.0016594170010648668, + 0.0016880419861990958, + 0.001594083005329594, + 0.0016380409942939878, + 0.001685708004515618, + 0.0015901250008028, + 0.0015935419942252338, + 0.0015635000017937273, + 0.001565708007547073, + 0.0015710839943494648, + 0.0015667499974370003, + 0.0015553749981336296, + 0.0015476249973289669, + 0.0015707499987911433, + 0.0015895830001682043, + 0.0016200420068344101, + 0.0015644590021111071, + 0.0016281250136671588, + 0.0015929580113152042, + 0.0016624170093564317, + 0.0015531249955529347 + ] + }, + { + "forward": 0.004079874997842126, + "prefill": 0.006733125002938323, + "decode_steps": [ + 0.003851750007015653, + 0.004508208003244363, + 0.0025845409982139245, + 0.004514208005275577, + 0.002203957992605865, + 0.003395790990907699, + 0.002198292000684887, + 0.002299959000083618, + 0.002042708991211839, + 0.0022133750026114285, + 0.001608417005627416, + 0.0016122919914778322, + 0.0021891670039622113, + 0.00166937499307096, + 0.0016019169997889549, + 0.0023044589906930923, + 0.0018871249922085553, + 0.0016660419933032244, + 0.001588833998539485, + 0.0015594160067848861, + 0.0019265830051153898, + 0.0017475420027039945, + 0.0017109159962274134, + 0.0017077079974114895, + 0.001654457999393344, + 0.0016626249998807907, + 0.001763417007168755, + 0.0016233329952228814, + 0.0015869170019868761, + 0.0018083339964505285, + 0.0018343330011703074, + 0.0018071660015266389 + ] + }, + { + "forward": 0.0031628339929739013, + "prefill": 0.0032103749981615692, + "decode_steps": [ + 0.0021789590100524947, + 0.001818082993850112, + 0.0017669589869910851, + 0.001799582998501137, + 0.0017326250090263784, + 0.0017780000052880496, + 0.0016950830031419173, + 0.0016339579888153821, + 0.001704957991023548, + 0.0016994170000543818, + 0.0016534589958610013, + 0.001588999992236495, + 0.00156879099085927, + 0.0016200420068344101, + 0.001640750007936731, + 0.0016359580040443689, + 0.0016510839923284948, + 0.0016755839897086844, + 0.0016467089881189167, + 0.0017610829963814467, + 0.001583040997502394, + 0.001636624991078861, + 0.0016299580020131543, + 0.0018165829969802871, + 0.0018707079871091992, + 0.002117666997946799, + 0.002267541000037454, + 0.002275374994496815, + 0.0023787090030964464, + 0.002193666994571686, + 0.002042207997874357, + 0.0020202909945510328 + ] + }, + { + "forward": 0.0032639590062899515, + "prefill": 0.0032123749988386407, + "decode_steps": [ + 0.0019545830000424758, + 0.001749917006236501, + 0.0015976249997038394, + 0.001676458996371366, + 0.0015856250101933256, + 0.0015530829987255856, + 0.0016635000065434724, + 0.0017444580007577315, + 0.0016672079946147278, + 0.0016589580045547336, + 0.0016425829962827265, + 0.0015899160061962903, + 0.0015760829992359504, + 0.0015587079979013652, + 0.001709167001536116, + 0.0017097920062951744, + 0.0017367499967804179, + 0.0015862080035731196, + 0.0018021660071099177, + 0.0017536249943077564, + 0.0017306660010945052, + 0.0015984170022420585, + 0.001575791000504978, + 0.0017233750113518909, + 0.0017230829980690032, + 0.0017696249997243285, + 0.0017067920125555247, + 0.0017094170034397393, + 0.0016093339945655316, + 0.0018296249909326434, + 0.001619832997675985, + 0.0016058750043157488 + ] + }, + { + "forward": 0.004066042005433701, + "prefill": 0.003188916001818143, + "decode_steps": [ + 0.0020402909867698327, + 0.0015927089989418164, + 0.0016877919988473877, + 0.0015776670043123886, + 0.0016938749904511496, + 0.0023284159979084507, + 0.0017702500044833869, + 0.0015994160057744011, + 0.0018372089980402961, + 0.0016597089997958392, + 0.0016015000001061708, + 0.001799999998183921, + 0.0015799999964656308, + 0.0015688749990658835, + 0.0017106250015785918, + 0.0017106669984059408, + 0.001582875003805384, + 0.0017284589994233102, + 0.0015885419998085126, + 0.0017137920076493174, + 0.001709167001536116, + 0.0015586249937769026, + 0.0017006669950205833, + 0.0017181250004796311, + 0.0016089170094346628, + 0.0016048749967012554, + 0.0016660840046824887, + 0.0016513750015292317, + 0.0016895830049179494, + 0.001639249996514991, + 0.0015593749994877726, + 0.002510459002223797 + ] + }, + { + "forward": 0.003193374999682419, + "prefill": 0.0032561669941060245, + "decode_steps": [ + 0.001817875003325753, + 0.0015594580036122352, + 0.0022082920040702447, + 0.0016927920078160241, + 0.001713374993414618, + 0.0015795000072102994, + 0.0017022090032696724, + 0.0015899169957265258, + 0.001687084004515782, + 0.0016207080043386668, + 0.0015193329891189933, + 0.0020244580082362518, + 0.0016300419956678525, + 0.0015815830120118335, + 0.0016726250032661483, + 0.0015762089897179976, + 0.0016062079957919195, + 0.0015538329898845404, + 0.0015831249911570922, + 0.0017481670074630529, + 0.001744749999488704, + 0.001749707997078076, + 0.001778833000571467, + 0.0016674580110702664, + 0.0020162500004516914, + 0.001776916004018858, + 0.0016008329985197634, + 0.0016156249912455678, + 0.0024141250032698736, + 0.0016360830049961805, + 0.0015913340030238032, + 0.0017805829993449152 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 3.2533125049667433, + "p95": 4.0736501512583345, + "min": 3.1628339929739013, + "max": 4.079874997842126, + "iqr": 0.7433857535943389, + "mean": 3.5165791981853545, + "cv": 0.11519350863773764 + }, + "prefill_ms": { + "count": 10, + "median": 3.2342709964723326, + "p95": 5.590856251365037, + "min": 3.188916001818143, + "max": 6.733125002938323, + "iqr": 0.2591872435004916, + "mean": 3.7011292020906694, + "cv": 0.29975923712341745 + }, + "decode_ms_per_step": { + "count": 320, + "median": 1.687958494585473, + "p95": 2.198575300280936, + "min": 1.5193329891189933, + "max": 4.514208005275577, + "iqr": 0.12729175432468764, + "mean": 1.7430691497793305, + "cv": 0.18083826844078454 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 588.9663744846371, + "p95": 609.6969083367998, + "min": 468.29382301735916, + "max": 611.8917255090383, + "iqr": 13.550544291071901, + "mean": 576.8351072597636, + "cv": 0.07207777219839587 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 552.7519993828959, + "p95": 572.8604992434637, + "min": 426.2898720355988, + "max": 573.2780687349876, + "iqr": 14.390519599485742, + "mean": 541.7069749848376, + "cv": 0.07957971129947872 + }, + "memory": { + "peak_extra_bytes": 8650752, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "reference", + "batch": 1, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.009670125000411645, + "prefill": 0.006421874990337528, + "decode_steps": [ + 0.0046971249976195395, + 0.004330958006903529, + 0.004294999991543591, + 0.004453832996659912, + 0.0040718329983064905, + 0.004466541999136098, + 0.004445667000254616, + 0.004581166998832487, + 0.004212082989397459, + 0.00414066699158866, + 0.003928416001144797, + 0.004149041997152381, + 0.004747290993691422, + 0.004438332995050587, + 0.004930541996145621, + 0.004605958005413413, + 0.004587375005939975, + 0.004472249987884425, + 0.004402209000545554, + 0.004304666988900863, + 0.0043916669965256006, + 0.004226917008054443, + 0.004226334000122733, + 0.004176708986051381, + 0.0039074999949662015, + 0.004339874998549931, + 0.0042305419920012355, + 0.004207417005090974, + 0.004341749998275191, + 0.004222749994369224, + 0.004117207994568162, + 0.004156334005529061 + ] + }, + { + "forward": 0.009531665986287408, + "prefill": 0.006380250008078292, + "decode_steps": [ + 0.0044963749969610944, + 0.004245624993927777, + 0.003667417011456564, + 0.004337334001320414, + 0.004324167006416246, + 0.004317874991102144, + 0.00400029199954588, + 0.004453458997886628, + 0.004265459007001482, + 0.004141709010582417, + 0.004211167004541494, + 0.003696750005474314, + 0.004315958998631686, + 0.004272875012247823, + 0.0041912500018952414, + 0.004185332989436574, + 0.00424774999555666, + 0.004222500007017516, + 0.004131790992687456, + 0.004178167000645772, + 0.004223665993777104, + 0.004354125005193055, + 0.0042345419933553785, + 0.004186167003354058, + 0.004103166997083463, + 0.004305625014239922, + 0.0041293749964097515, + 0.004105290994630195, + 0.004383291001431644, + 0.003677208995213732, + 0.004116541997063905, + 0.004338667000411078 + ] + }, + { + "forward": 0.009516999998595566, + "prefill": 0.0068330829963088036, + "decode_steps": [ + 0.005207166992477141, + 0.0040937500016298145, + 0.004412291993503459, + 0.004464791010832414, + 0.004092332994332537, + 0.004631666990462691, + 0.004449416999705136, + 0.004305374997784384, + 0.004247125005349517, + 0.004243417002726346, + 0.004202708005323075, + 0.004207917008898221, + 0.0042910409974865615, + 0.004266332995030098, + 0.0041733329999260604, + 0.004018959007225931, + 0.004154082998866215, + 0.0040857919957488775, + 0.004137875002925284, + 0.004114542010938749, + 0.003908291007974185, + 0.00430699999560602, + 0.003977374988608062, + 0.004437332987436093, + 0.004243957999278791, + 0.004241834001732059, + 0.004391875001601875, + 0.004067083995323628, + 0.004372749986941926, + 0.004323667002609, + 0.0041842499922495335, + 0.004200750001473352 + ] + }, + { + "forward": 0.009663249991717748, + "prefill": 0.006386000008205883, + "decode_steps": [ + 0.00450700000510551, + 0.0039035419904394075, + 0.0043047500075772405, + 0.004256208005244844, + 0.004091166993021034, + 0.0044024999951943755, + 0.0038815840089228004, + 0.004395750002004206, + 0.004323291999753565, + 0.004279290995327756, + 0.004017208993900567, + 0.0040830829966580495, + 0.004096874996321276, + 0.004395875002956018, + 0.004204666998703033, + 0.004120124998735264, + 0.0037402920133899897, + 0.004144708000239916, + 0.004086750006536022, + 0.004094750009244308, + 0.003777667006943375, + 0.004229750004014932, + 0.004139833996305242, + 0.004105250001884997, + 0.0036507500044535846, + 0.00412812500144355, + 0.004178707997198217, + 0.0037918749876553193, + 0.0042802920070244, + 0.004316417005611584, + 0.00414587500563357, + 0.004205040997476317 + ] + }, + { + "forward": 0.009086334001040086, + "prefill": 0.007199999992735684, + "decode_steps": [ + 0.0039059580012690276, + 0.004174332992988639, + 0.004037792008602992, + 0.004225374999805354, + 0.003618709000875242, + 0.004045458001201041, + 0.004201500007184222, + 0.004179832991212606, + 0.00366700001177378, + 0.004542208000202663, + 0.004214625005261041, + 0.004112541995709762, + 0.00445429100363981, + 0.004289750009775162, + 0.004151957997237332, + 0.003557290998287499, + 0.004211167004541494, + 0.004138207994401455, + 0.00422141699527856, + 0.004228749996400438, + 0.004115374991670251, + 0.004116374999284744, + 0.004081541002960876, + 0.004052417003549635, + 0.004253916995367035, + 0.004074333002790809, + 0.004113041999517009, + 0.003716125007485971, + 0.004234834006638266, + 0.004199041999527253, + 0.004112957991310395, + 0.004009500000393018 + ] + }, + { + "forward": 0.009498082989011891, + "prefill": 0.006681625003693625, + "decode_steps": [ + 0.0045719160116277635, + 0.004139625001698732, + 0.00479750000522472, + 0.004469457999221049, + 0.004269124998245388, + 0.004179083000053652, + 0.004041207997943275, + 0.004114459006814286, + 0.004538167006103322, + 0.0037985829985700548, + 0.004319667001254857, + 0.0041833339928416535, + 0.004131333000259474, + 0.003670416001114063, + 0.0039526250038761646, + 0.00429579199408181, + 0.004841417001443915, + 0.004458041992620565, + 0.004514207990723662, + 0.004349625000031665, + 0.004410041990922764, + 0.0047666250029578805, + 0.004498624999541789, + 0.004504874988924712, + 0.0049048750079236925, + 0.0053832089906791225, + 0.004345792011008598, + 0.004540124995401129, + 0.005411499994806945, + 0.005252624992863275, + 0.004464790996280499, + 0.005271333997370675 + ] + }, + { + "forward": 0.010944000008748844, + "prefill": 0.007218707993160933, + "decode_steps": [ + 0.004799959002411924, + 0.004551374993752688, + 0.004216583998640999, + 0.004217624999000691, + 0.004430750006577, + 0.00447916699340567, + 0.004051416995935142, + 0.004320499996538274, + 0.004073208998306654, + 0.0042966249893652275, + 0.004259332999936305, + 0.004116707990760915, + 0.004128792003029957, + 0.004220875009195879, + 0.004316499995184131, + 0.004145999992033467, + 0.004197500005830079, + 0.00411616699420847, + 0.004116167008760385, + 0.00411954200535547, + 0.004188165999948978, + 0.004120792000321671, + 0.004346125002484769, + 0.003859042000840418, + 0.0044632919889409095, + 0.00441300000238698, + 0.004281166999135166, + 0.0040140830096788704, + 0.0044360000028973445, + 0.004167499995674007, + 0.0041846249951049685, + 0.004233165993355215 + ] + }, + { + "forward": 0.009585291991243139, + "prefill": 0.006319541003904305, + "decode_steps": [ + 0.004448000006959774, + 0.004286542010959238, + 0.00433504200191237, + 0.004382041995995678, + 0.0042540830036159605, + 0.0041140829998767, + 0.004236458000377752, + 0.00475150000420399, + 0.005018833006033674, + 0.004991124995285645, + 0.004728041007183492, + 0.004053333002957515, + 0.004407957996590994, + 0.004321790998801589, + 0.00458345799415838, + 0.004378375000669621, + 0.004043249995447695, + 0.004537583998171613, + 0.004550708006718196, + 0.0038291670061880723, + 0.004087334003997967, + 0.0037937500019324943, + 0.004408749999129213, + 0.004341249994467944, + 0.004302249988541007, + 0.004381541992188431, + 0.004304708010749891, + 0.004258500004652888, + 0.004296499988413416, + 0.00413204099459108, + 0.00425916699168738, + 0.0042828330042539164 + ] + }, + { + "forward": 0.009465665993047878, + "prefill": 0.006503417011117563, + "decode_steps": [ + 0.0046144160005496815, + 0.004241167000145651, + 0.00417241599643603, + 0.004224415999487974, + 0.0042211250110995024, + 0.00411429199448321, + 0.0036682499921880662, + 0.004091707989573479, + 0.004091916009201668, + 0.004098416000488214, + 0.004154499998548999, + 0.004001167006208561, + 0.0042718329932540655, + 0.003962290997151285, + 0.004195500005153008, + 0.004200709008728154, + 0.0041447500116191804, + 0.003807750006671995, + 0.004118249999010004, + 0.004100916004972532, + 0.004264000002876855, + 0.0036550830118358135, + 0.004136458010179922, + 0.003974542007199489, + 0.004134583999984898, + 0.004301249995478429, + 0.00417991699941922, + 0.00404216701281257, + 0.0040606249967822805, + 0.003618082992034033, + 0.004132333007873967, + 0.004063792002853006 + ] + }, + { + "forward": 0.01035145900095813, + "prefill": 0.006512416002806276, + "decode_steps": [ + 0.0057806249969871715, + 0.004430917004356161, + 0.0044216660025995225, + 0.004645708002499305, + 0.004418750002514571, + 0.0037564999947790056, + 0.004131749999942258, + 0.004597040999215096, + 0.004449540996574797, + 0.004204041993943974, + 0.0044994589989073575, + 0.004265540992491879, + 0.0038880000065546483, + 0.004327041999204084, + 0.004225374999805354, + 0.004365790999145247, + 0.004351333001977764, + 0.004751749991555698, + 0.0041989169985754415, + 0.00535116701212246, + 0.004519875001278706, + 0.004365084008895792, + 0.004177249997155741, + 0.004305499998736195, + 0.0041577919910196215, + 0.004201875010039657, + 0.004189459010376595, + 0.0042531670042080805, + 0.004282791007426567, + 0.0042207920050714165, + 0.004240500013111159, + 0.004186584003036842 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 9.558478988765273, + "p95": 10.677356555243023, + "min": 9.086334001040086, + "max": 10.944000008748844, + "iqr": 0.16559400683036074, + "mean": 9.731287496106233, + "cv": 0.05428592257085273 + }, + "prefill_ms": { + "count": 10, + "median": 6.507916506961919, + "p95": 7.210289392969571, + "min": 6.319541003904305, + "max": 7.218707993160933, + "iqr": 0.4002497444162145, + "mean": 6.645691501034889, + "cv": 0.05026763136288295 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.224040996632539, + "p95": 4.7515125035715755, + "min": 3.557290998287499, + "max": 5.7806249969871715, + "iqr": 0.251030498475302, + "mean": 4.257661093788556, + "cv": 0.06769668637922485 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 235.0032788513254, + "p95": 243.99896326916308, + "min": 223.1675833948315, + "max": 244.16554589059947, + "iqr": 10.936897035158324, + "mean": 235.06413206522728, + "cv": 0.030164125360885575 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 223.47187727846423, + "p95": 231.94521357860742, + "min": 213.23151389784334, + "max": 232.62231510290266, + "iqr": 9.916775434154658, + "mean": 224.1107944689833, + "cv": 0.028419101826137362 + }, + "memory": { + "peak_extra_bytes": 8650752, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "auto", + "batch": 1, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.007624208999914117 + }, + { + "forward": 0.008041125009185635 + }, + { + "forward": 0.007210333002149127 + }, + { + "forward": 0.007996166998054832 + }, + { + "forward": 0.008429250010522082 + }, + { + "forward": 0.009905291008180939 + }, + { + "forward": 0.007028958993032575 + }, + { + "forward": 0.007328541993047111 + }, + { + "forward": 0.007807125002727844 + }, + { + "forward": 0.007324540987610817 + } + ], + "forward_ms": { + "count": 10, + "median": 7.7156670013209805, + "p95": 9.241072559234452, + "min": 7.028958993032575, + "max": 9.905291008180939, + "iqr": 0.7043442674330436, + "mean": 7.869554200442508, + "cv": 0.1062987587623226 + }, + "memory": { + "peak_extra_bytes": 8650752, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "sdpa", + "batch": 1, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.005461959008243866, + "prefill": 0.005580415992881171, + "decode_steps": [ + 0.002202417002990842, + 0.0018088750075548887, + 0.001961667003342882, + 0.001695291997748427, + 0.001872915992862545, + 0.00171237500035204, + 0.0018303749966435134, + 0.0017030419985530898, + 0.0016603330004727468, + 0.0023093329946277663, + 0.0018149999959859997, + 0.0016612920007901266, + 0.0017979999975068495, + 0.0017263749905396253, + 0.0017767499957699329, + 0.00170808398979716, + 0.0017252499965252355, + 0.0017340000049443915, + 0.0018014999950537458, + 0.0017894999909913167, + 0.0016749579954193905, + 0.001701500004855916, + 0.0017246659990632907, + 0.001766874993336387, + 0.0017009580042213202, + 0.0016101660003187135, + 0.0016842920013004914, + 0.001669749995926395, + 0.0018227079999633133, + 0.0017744169890647754, + 0.0017508750024717301, + 0.001730874995701015 + ] + }, + { + "forward": 0.005691042009857483, + "prefill": 0.0054819999932078645, + "decode_steps": [ + 0.0020835830073338, + 0.0017040409875335172, + 0.0017735000001266599, + 0.0017860000079963356, + 0.00178116699680686, + 0.0017787079996196553, + 0.0021747499995399266, + 0.001709167001536116, + 0.0016990830044960603, + 0.0017559590050950646, + 0.0017794999876059592, + 0.0017289999959757552, + 0.0017488750017946586, + 0.0017646670021349564, + 0.0016387500072596595, + 0.0017787500109989196, + 0.0016671670018695295, + 0.0017725840007187799, + 0.0017948339955182746, + 0.0017512079939479008, + 0.001728165996610187, + 0.0017405419930582866, + 0.0018360000103712082, + 0.0018818329990608618, + 0.001746458001434803, + 0.001793208997696638, + 0.0018291249871253967, + 0.0017617920093471184, + 0.001759624996338971, + 0.0017672920075710863, + 0.0017457500071031973, + 0.0017782079958124086 + ] + }, + { + "forward": 0.005601500000921078, + "prefill": 0.005574458991759457, + "decode_steps": [ + 0.0020876250055152923, + 0.0017995839880313724, + 0.0018322500109206885, + 0.0016808750078780577, + 0.001711249991785735, + 0.0017752500134520233, + 0.0016638750093989074, + 0.0016968749987427145, + 0.0017707499937387183, + 0.001764167012879625, + 0.0017761250055627897, + 0.0017240829911315814, + 0.0017202920134877786, + 0.001724000001559034, + 0.0017637910059420392, + 0.0017115410009864718, + 0.0016535000031581149, + 0.0017492920014774427, + 0.0017679589946055785, + 0.0016239999968092889, + 0.0019693330104928464, + 0.001862292003352195, + 0.0017155410023406148, + 0.0017877499922178686, + 0.0017652090027695522, + 0.0023415829928126186, + 0.0018360000103712082, + 0.0017112090135924518, + 0.0017478330119047314, + 0.0017206250049639493, + 0.001880374999018386, + 0.0017053339979611337 + ] + }, + { + "forward": 0.0056423329951940104, + "prefill": 0.005613208006252535, + "decode_steps": [ + 0.001984041999094188, + 0.0017959160031750798, + 0.0016920839989325032, + 0.001814292001654394, + 0.0018631669954629615, + 0.0018119169981218874, + 0.0016897919995244592, + 0.0017806249961722642, + 0.0017781659989850596, + 0.0019020000036107376, + 0.0017582919972483069, + 0.001723792011034675, + 0.0016435420111520216, + 0.0016662499983794987, + 0.0017726660007610917, + 0.0017408329877071083, + 0.0017230830126209185, + 0.0016398750012740493, + 0.0016887910023797303, + 0.0016462919884361327, + 0.0017682080069789663, + 0.001694124992354773, + 0.001797125005396083, + 0.0016917920002015308, + 0.001621749994228594, + 0.0018393749924143776, + 0.0017210420046467334, + 0.0017172920051962137, + 0.0024834170035319403, + 0.0018137080041924492, + 0.0018645419913809747, + 0.0017909159942064434 + ] + }, + { + "forward": 0.005994332997943275, + "prefill": 0.0058613330038497224, + "decode_steps": [ + 0.0024507079942850396, + 0.001964957991731353, + 0.0017962500132853165, + 0.0018668749980861321, + 0.0018045830074697733, + 0.0018707920098677278, + 0.0017518750100862235, + 0.0017707919905660674, + 0.00189275000593625, + 0.0017954159993678331, + 0.002150291998987086, + 0.0017843340028775856, + 0.0016560420044697821, + 0.0016632920014671981, + 0.0017106250015785918, + 0.0017278339946642518, + 0.0018408750038361177, + 0.0018107920041074976, + 0.0017583330045454204, + 0.001804957995773293, + 0.0017837079940363765, + 0.0016897080058697611, + 0.0017113750072894618, + 0.0018612499989103526, + 0.002111042005708441, + 0.001786124994396232, + 0.0017534159997012466, + 0.0018242500082124025, + 0.0017698750016279519, + 0.001757374993758276, + 0.0016964579990599304, + 0.0017290839896304533 + ] + }, + { + "forward": 0.006054583005607128, + "prefill": 0.00604741700226441, + "decode_steps": [ + 0.002655332995345816, + 0.0018107919895555824, + 0.0018008339975494891, + 0.001652541002840735, + 0.001736207996145822, + 0.0017453750042477623, + 0.0017580419953446835, + 0.0017288329981965944, + 0.0020358750043669716, + 0.0019471669947961345, + 0.0018804580031428486, + 0.0018140830070478842, + 0.0018275409966008738, + 0.001765332999639213, + 0.0017482909897807986, + 0.001807249995181337, + 0.0017422079981770366, + 0.0017011250020004809, + 0.001834874987252988, + 0.0018077080021612346, + 0.0017258750012842938, + 0.0017258750012842938, + 0.0017937919910764322, + 0.0017725419893395156, + 0.0017298329912591726, + 0.0017190420039696619, + 0.0016856669972185045, + 0.0019334590033395216, + 0.0017547500028740615, + 0.0021937919955234975, + 0.0020806249958695844, + 0.0017079579993151128 + ] + }, + { + "forward": 0.005501499996171333, + "prefill": 0.00599695899290964, + "decode_steps": [ + 0.0020025419944431633, + 0.0018051670049317181, + 0.001712999990559183, + 0.0017562919965712354, + 0.0017170000064652413, + 0.0024489999923389405, + 0.0018226249958388507, + 0.0017267499933950603, + 0.0017320409970125183, + 0.0018594579887576401, + 0.0018612079875310883, + 0.001862874996731989, + 0.001743250002618879, + 0.0017366660031257197, + 0.001795459000277333, + 0.0016776669945102185, + 0.0017161249998025596, + 0.0017833750025602058, + 0.001816959003917873, + 0.001697332991170697, + 0.0017701250035315752, + 0.0017562499997438863, + 0.0018936669948743656, + 0.0016747079935157672, + 0.001966291994904168, + 0.0017229999939445406, + 0.0017512909980723634, + 0.0017371659923810512, + 0.0016259580006590113, + 0.0016974169993773103, + 0.0017329170077573508, + 0.001778124991687946 + ] + }, + { + "forward": 0.013686209000297822, + "prefill": 0.007441249996190891, + "decode_steps": [ + 0.0025879580061882734, + 0.001791834001778625, + 0.0020253749971743673, + 0.001842916986788623, + 0.0018303329998161644, + 0.001807375010685064, + 0.0017573750083101913, + 0.0017633750103414059, + 0.001779625003109686, + 0.0018102910107700154, + 0.0016302500007441267, + 0.001724249988910742, + 0.0018349580059293658, + 0.001669584002229385, + 0.001709500007564202, + 0.0017399579955963418, + 0.0017423749959561974, + 0.0016542079974897206, + 0.0017491249891463667, + 0.0017539580003358424, + 0.0017594579985598102, + 0.0017871250020107254, + 0.0017964580038096756, + 0.0016990419971989468, + 0.0017088750028051436, + 0.0020844579994445667, + 0.0016894589934963733, + 0.0016341669979738072, + 0.0017356669995933771, + 0.0017354169976897538, + 0.0017767499957699329, + 0.0017202500021085143 + ] + }, + { + "forward": 0.005515291995834559, + "prefill": 0.005599832991720177, + "decode_steps": [ + 0.002098916011163965, + 0.0017829999997047707, + 0.0017738750029820949, + 0.0017198330024257302, + 0.0017857920029200613, + 0.0017838329949881881, + 0.0017649590008659288, + 0.0016714169905753806, + 0.001708166004391387, + 0.0016910419944906607, + 0.0018728750001173466, + 0.001677332998951897, + 0.001802874990971759, + 0.0017901660030474886, + 0.001786082997568883, + 0.0016639579989714548, + 0.0017725830111885443, + 0.0017829999997047707, + 0.0017554579972056672, + 0.00178208299621474, + 0.0017022910033119842, + 0.001723834007862024, + 0.0017677499999990687, + 0.0017875829944387078, + 0.001726292000967078, + 0.001781541999662295, + 0.001869458006694913, + 0.0017632910021347925, + 0.001680375004070811, + 0.0018999159947270527, + 0.0018318330112379044, + 0.0017949590110220015 + ] + }, + { + "forward": 0.0056827499938663095, + "prefill": 0.0055233340099221095, + "decode_steps": [ + 0.0019916669989470392, + 0.0017961660050787032, + 0.0016635420033708215, + 0.0017050829919753596, + 0.001757583988364786, + 0.0017309590039076284, + 0.0016492090071551502, + 0.0016472910065203905, + 0.0017461670067859814, + 0.002081541999359615, + 0.0017432919994462281, + 0.001726292000967078, + 0.0017978329997276887, + 0.0017585419991519302, + 0.0017543330031912774, + 0.0017572499928064644, + 0.001668375000008382, + 0.0017497500084573403, + 0.0018232089933007956, + 0.0016600829985691234, + 0.0018793339986586943, + 0.001852750006946735, + 0.0018540830060373992, + 0.0016640829999232665, + 0.0019656249933177605, + 0.002439541000057943, + 0.0018425000016577542, + 0.00178958399919793, + 0.0017246250063180923, + 0.00183012499473989, + 0.001808375003747642, + 0.001765708002494648 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 5.66254149453016, + "p95": 10.251977302687003, + "min": 5.461959008243866, + "max": 13.686209000297822, + "iqr": 0.3816662538156379, + "mean": 6.483150100393686, + "cv": 0.39157509502029614 + }, + "prefill_ms": { + "count": 10, + "median": 5.606520498986356, + "p95": 6.814025148923973, + "min": 5.4819999932078645, + "max": 7.441249996190891, + "iqr": 0.3871042536047753, + "mean": 5.872020898095798, + "cv": 0.09990419335413457 + }, + "decode_ms_per_step": { + "count": 320, + "median": 1.7644170075072907, + "p95": 2.0836267569393385, + "min": 1.6101660003187135, + "max": 2.655332995345816, + "iqr": 0.09027050327858888, + "mean": 1.7937688780421013, + "cv": 0.08150317981724062 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 559.5815567076955, + "p95": 562.9348850062065, + "min": 545.860076706083, + "max": 563.4162784892969, + "iqr": 6.001418697519171, + "mean": 557.5448336640994, + "cv": 0.010825277161608929 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 508.9050333533774, + "p95": 512.5243702514794, + "min": 494.03300753824595, + "max": 512.8517110834538, + "iqr": 11.789613164938544, + "mean": 505.8486273681045, + "cv": 0.014804822110046719 + }, + "memory": { + "peak_extra_bytes": 8650752, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "reference", + "batch": 1, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.02209320801193826, + "prefill": 0.01759791700169444, + "decode_steps": [ + 0.005267250002361834, + 0.004847624993999489, + 0.0046323330025188625, + 0.004504041004111059, + 0.0043728339951485395, + 0.004413125003338791, + 0.004327709000790492, + 0.004467167003895156, + 0.004411917005199939, + 0.004450042004464194, + 0.004423874997883104, + 0.004319875006331131, + 0.004311624987167306, + 0.004475665991776623, + 0.004251042002579197, + 0.004081333012436517, + 0.004200124996714294, + 0.004868709002039395, + 0.004160124997724779, + 0.004306624992750585, + 0.004294750004191883, + 0.004214707994833589, + 0.004389374997117557, + 0.004186374993878417, + 0.0046171669964678586, + 0.004439959011506289, + 0.004397500000777654, + 0.004328042006818578, + 0.004340917002991773, + 0.00432774999353569, + 0.0043223340035183355, + 0.004307707989937626 + ] + }, + { + "forward": 0.020594333007466048, + "prefill": 0.017417791998013854, + "decode_steps": [ + 0.004723583988379687, + 0.004300000000512227, + 0.004325875008362345, + 0.00434008298907429, + 0.004203791992040351, + 0.004197165995719843, + 0.004235999993397854, + 0.004202540992991999, + 0.004092000002856366, + 0.00408866599900648, + 0.004039749997900799, + 0.004417417003423907, + 0.004241457994794473, + 0.004721166013041511, + 0.004634750002878718, + 0.0041396669985260814, + 0.004252541999449022, + 0.004267332988092676, + 0.004375459000584669, + 0.0041777500009629875, + 0.004197957998258062, + 0.003880165997543372, + 0.00453187499078922, + 0.003885458005242981, + 0.004212167012155987, + 0.00505374999193009, + 0.004446250008186325, + 0.0037343330041039735, + 0.0042164170008618385, + 0.004662666004151106, + 0.004545374991721474, + 0.004530167003395036 + ] + }, + { + "forward": 0.021451416992931627, + "prefill": 0.019101917001535185, + "decode_steps": [ + 0.006277041989960708, + 0.004769749997649342, + 0.003903000004356727, + 0.004825500000151806, + 0.004773958004079759, + 0.004384083003969863, + 0.005046584003139287, + 0.005846625004778616, + 0.004781166993780062, + 0.004629832998034544, + 0.00577750000229571, + 0.005030542000895366, + 0.00488795799901709, + 0.0047878339973976836, + 0.0049662499950500205, + 0.005167791998246685, + 0.005060666997451335, + 0.005108791010570712, + 0.004019625004730187, + 0.004320709005696699, + 0.005141667003044859, + 0.004917624988593161, + 0.004497709000133909, + 0.004658167003071867, + 0.004819542009499855, + 0.0046084169880487025, + 0.0044015419989591464, + 0.004379374993732199, + 0.00437262500054203, + 0.004211374995065853, + 0.004326540991314687, + 0.00433074998727534 + ] + }, + { + "forward": 0.021028207993367687, + "prefill": 0.018967832991620526, + "decode_steps": [ + 0.005938667003647424, + 0.005350499995984137, + 0.004059583006892353, + 0.004223334006383084, + 0.005258417004370131, + 0.004549499994027428, + 0.00436624999565538, + 0.004522584000369534, + 0.0048165420012082905, + 0.0039246669912245125, + 0.004202416006592102, + 0.004942375002428889, + 0.004516999993938953, + 0.00443112499488052, + 0.004275917002814822, + 0.0045514170051319525, + 0.004026916009024717, + 0.004083124993485399, + 0.004341207997640595, + 0.004643749998649582, + 0.004228584002703428, + 0.004168041006778367, + 0.004888499999651685, + 0.004165292004472576, + 0.004438916002982296, + 0.0044312499958323315, + 0.004141917001106776, + 0.0048210829991148785, + 0.004445209007826634, + 0.004260208006598987, + 0.004266874995664693, + 0.004653583004255779 + ] + }, + { + "forward": 0.021699334000004455, + "prefill": 0.018848166990210302, + "decode_steps": [ + 0.005349375001969747, + 0.004615624988218769, + 0.004825333002372645, + 0.004742249991977587, + 0.004415290997712873, + 0.004508625002927147, + 0.0041666659963084385, + 0.004335416000685655, + 0.004310959004214965, + 0.005643249998684041, + 0.004889125004410744, + 0.004323000001022592, + 0.004584624999552034, + 0.004577750005410053, + 0.004315665995818563, + 0.004387583001516759, + 0.005080666989670135, + 0.004621666987077333, + 0.0046209590072976425, + 0.004324166991864331, + 0.0041804170032264665, + 0.004779499999131076, + 0.004689667010097764, + 0.004610916992533021, + 0.004822959002922289, + 0.004167708000750281, + 0.004724790996988304, + 0.004087833003723063, + 0.004099792000488378, + 0.005016333001549356, + 0.003962832997785881, + 0.004444749996764585 + ] + }, + { + "forward": 0.020793042000150308, + "prefill": 0.017458208007155918, + "decode_steps": [ + 0.004549833000055514, + 0.004296040991903283, + 0.0042887919989880174, + 0.004332666998379864, + 0.004304125002818182, + 0.004376875003799796, + 0.004211790990666486, + 0.004267291005817242, + 0.004411625006468967, + 0.004025250003905967, + 0.004408207998494618, + 0.004319375002523884, + 0.004263915994670242, + 0.00433654100925196, + 0.003747792012291029, + 0.004256625004927628, + 0.004222749994369224, + 0.004334792000008747, + 0.004373625008156523, + 0.004321875007008202, + 0.004190875013591722, + 0.004077625009813346, + 0.004203916992992163, + 0.004267333002644591, + 0.004130957997404039, + 0.004084917003638111, + 0.004823792012757622, + 0.004196333000436425, + 0.004277042011381127, + 0.004353042008006014, + 0.004457417002413422, + 0.00431445799767971 + ] + }, + { + "forward": 0.022165625006891787, + "prefill": 0.018782875005854294, + "decode_steps": [ + 0.005852916001458652, + 0.004394082992803305, + 0.004955291995429434, + 0.004962874998454936, + 0.004515916996751912, + 0.0047387910017278045, + 0.004632666998077184, + 0.004506124998442829, + 0.004167874998529442, + 0.004394624993437901, + 0.004246666998369619, + 0.004298166997614317, + 0.004368665991933085, + 0.00433520799560938, + 0.004310167001676746, + 0.004295374994399026, + 0.004327832997660153, + 0.004240124995703809, + 0.00415445800172165, + 0.004201792005915195, + 0.004127791995415464, + 0.004254166997270659, + 0.004250666999723762, + 0.004218333997414447, + 0.004225750002660789, + 0.0042276250023860484, + 0.004097458004252985, + 0.00427208399923984, + 0.004979542005457915, + 0.0049915840063476935, + 0.0044893749873153865, + 0.005029708001529798 + ] + }, + { + "forward": 0.02115274999232497, + "prefill": 0.01755270799912978, + "decode_steps": [ + 0.004750167005113326, + 0.004472833010368049, + 0.004309208001359366, + 0.004308542003855109, + 0.004375124990474433, + 0.004291625009500422, + 0.004337999998824671, + 0.004113333998247981, + 0.004203916992992163, + 0.004610000003594905, + 0.00435708298755344, + 0.0041510829905746505, + 0.004792791994987056, + 0.004498499998589978, + 0.004320791995269246, + 0.004136500007007271, + 0.004814083993551321, + 0.0041871660068864, + 0.004095750002306886, + 0.0042100830032723024, + 0.004930291004711762, + 0.004468250001082197, + 0.004213457999867387, + 0.004902542001218535, + 0.004771459003677592, + 0.004294749989639968, + 0.005044125005952083, + 0.004910874995402992, + 0.004505250006332062, + 0.004316749997087754, + 0.004692792004789226, + 0.0039946670003701 + ] + }, + { + "forward": 0.021443624995299615, + "prefill": 0.018435750011121854, + "decode_steps": [ + 0.005587415987974964, + 0.005415334002464078, + 0.004943166990415193, + 0.00406045799900312, + 0.004370125010609627, + 0.0042868330056080595, + 0.004349124996224418, + 0.004230874998029321, + 0.004448625011718832, + 0.004352290998212993, + 0.004343042004620656, + 0.004152334004174918, + 0.0038426660030381754, + 0.004141917001106776, + 0.004245792006258853, + 0.004009874988696538, + 0.004240167007083073, + 0.0041382909985259175, + 0.004152458001044579, + 0.004385749998618849, + 0.004303207999328151, + 0.004316916994866915, + 0.004353083000751212, + 0.0048091249918797985, + 0.0046984159998828545, + 0.004370624999864958, + 0.004078792000655085, + 0.004291292003472336, + 0.004621416999725625, + 0.00425804199767299, + 0.004220000002533197, + 0.004401458005304448 + ] + }, + { + "forward": 0.021322207991033792, + "prefill": 0.017733915999997407, + "decode_steps": [ + 0.0052822079887846485, + 0.004450249994988553, + 0.004829291996429674, + 0.004281999994418584, + 0.004483707991312258, + 0.004235542000969872, + 0.003760958992643282, + 0.004254333005519584, + 0.00419595799758099, + 0.0043615839967969805, + 0.004177541995886713, + 0.00425520799763035, + 0.004153625006438233, + 0.004298375002690591, + 0.004243957999278791, + 0.004253790990333073, + 0.004421540987095796, + 0.004261875001247972, + 0.004567708005197346, + 0.004488709004363045, + 0.004484959004912525, + 0.004460624986677431, + 0.004110833004233427, + 0.004442374993232079, + 0.004236916996887885, + 0.0042899170075543225, + 0.004352417003246956, + 0.004378999990876764, + 0.0036803340044571087, + 0.004347458001575433, + 0.0041897919873008505, + 0.004136125004151836 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 21.382916493166704, + "p95": 22.1330373591627, + "min": 20.594333007466048, + "max": 22.165625006891787, + "iqr": 0.5780112551292405, + "mean": 21.374374999140855, + "cv": 0.024044690525169124 + }, + "prefill_ms": { + "count": 10, + "median": 18.08483300555963, + "p95": 19.041579197073588, + "min": 17.417791998013854, + "max": 19.101917001535185, + "iqr": 1.2678337443503551, + "mean": 18.189708300633356, + "cv": 0.0383444496115181 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.341062500316184, + "p95": 5.082073190715164, + "min": 3.6803340044571087, + "max": 6.277041989960708, + "iqr": 0.38307299837470055, + "mean": 4.448702621675693, + "cv": 0.08227155500941061 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 225.64053419002175, + "p95": 232.88487773538455, + "min": 209.10858029135053, + "max": 233.52976918268868, + "iqr": 7.7603760918383955, + "mean": 225.00110421705662, + "cv": 0.032204391503980835 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 200.76012312996718, + "p95": 206.65530937947358, + "min": 185.9033423061855, + "max": 207.13890500938663, + "iqr": 7.348464726809112, + "mean": 199.50691749981863, + "cv": 0.032086173162932774 + }, + "memory": { + "peak_extra_bytes": 1082392576, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "auto", + "batch": 1, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.01729699999850709 + }, + { + "forward": 0.01833737498964183 + }, + { + "forward": 0.017477458008215763 + }, + { + "forward": 0.018046416007564403 + }, + { + "forward": 0.017974791990127414 + }, + { + "forward": 0.017973541995161213 + }, + { + "forward": 0.018362040995270945 + }, + { + "forward": 0.017880291008623317 + }, + { + "forward": 0.018430540992994793 + }, + { + "forward": 0.01875754199863877 + } + ], + "forward_ms": { + "count": 10, + "median": 18.01060399884591, + "p95": 18.61039154609898, + "min": 17.29699999850709, + "max": 18.75754199863877, + "iqr": 0.4522707386058755, + "mean": 18.053699798474554, + "cv": 0.0244971156434368 + }, + "memory": { + "peak_extra_bytes": 42205184, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "sdpa", + "batch": 1, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.011119290997157805, + "prefill": 0.011170666010002606, + "decode_steps": [ + 0.002315749996341765, + 0.0017909999878611416, + 0.002165625002817251, + 0.0018180839979322627, + 0.0017865419940790161, + 0.002015582998865284, + 0.0017714999994495884, + 0.0017944999999599531, + 0.001827083993703127, + 0.0017549170006532222, + 0.0018020839925156906, + 0.0017387920088367537, + 0.0024017499963520095, + 0.001781124999979511, + 0.0019112920126644894, + 0.0018373749917373061, + 0.001904208998894319, + 0.0017857909988379106, + 0.0017620840080780908, + 0.002382792008575052, + 0.0017526250012451783, + 0.0017317080055363476, + 0.001780375008820556, + 0.0017935419891728088, + 0.0020585829915944487, + 0.0017992909997701645, + 0.0017719169991323724, + 0.0017645830084802583, + 0.0027175839932169765, + 0.0017892920004669577, + 0.0017612500087125227, + 0.0018116660066880286 + ] + }, + { + "forward": 0.011682500000461005, + "prefill": 0.011407583006075583, + "decode_steps": [ + 0.0024829589965520427, + 0.001916374996653758, + 0.0023360420018434525, + 0.00187070899119135, + 0.0021592089906334877, + 0.002000958993448876, + 0.0018663340015336871, + 0.0024604170030215755, + 0.0019087500113528222, + 0.002308791008545086, + 0.0019000839965883642, + 0.0019712089997483417, + 0.0018027909973170608, + 0.0018851670029107481, + 0.0019381250021979213, + 0.0018126669892808422, + 0.0019427079969318584, + 0.0022008749947417527, + 0.0018681249930523336, + 0.0019365829939488322, + 0.001797458011424169, + 0.001919792004628107, + 0.002460292002069764, + 0.0018677909974940121, + 0.0024242079962277785, + 0.002407541003776714, + 0.0021036670077592134, + 0.0018877499969676137, + 0.0020522919949144125, + 0.002100333003909327, + 0.001875624991953373, + 0.001840957993408665 + ] + }, + { + "forward": 0.011183000009623356, + "prefill": 0.011305209001875482, + "decode_steps": [ + 0.002952833005110733, + 0.0025415840063942596, + 0.0020212499948684126, + 0.0018159579922212288, + 0.0018502910097595304, + 0.001890291998279281, + 0.0019848340016324073, + 0.0019000829925062135, + 0.0017518749955343083, + 0.002309415998752229, + 0.0019099589990219101, + 0.0023689999943599105, + 0.0018692500016186386, + 0.0018881669966503978, + 0.001850082990131341, + 0.002060375001747161, + 0.0018817500094883144, + 0.0018746669957181439, + 0.0019392500107642263, + 0.0021012499928474426, + 0.0018586249934742227, + 0.0018721669912338257, + 0.0017862079985206947, + 0.001774625008692965, + 0.0019226250005885959, + 0.0018957080028485507, + 0.0018530000088503584, + 0.0019252919883001596, + 0.0018405410082777962, + 0.002064707994577475, + 0.001856375005445443, + 0.001806124986615032 + ] + }, + { + "forward": 0.011558958009118214, + "prefill": 0.012298292000195943, + "decode_steps": [ + 0.00274024999816902, + 0.002144541998859495, + 0.0019848329975502566, + 0.002512416001991369, + 0.001942791001056321, + 0.0018395410006633028, + 0.0020323339995229617, + 0.0019429170060902834, + 0.002037958998698741, + 0.0018542089965194464, + 0.0018714170000748709, + 0.0024282079975819215, + 0.0018922919989563525, + 0.001983624999411404, + 0.0019601669919211417, + 0.001974208003957756, + 0.0019949579873355106, + 0.0018919589929282665, + 0.0019452499982435256, + 0.0020606670004781336, + 0.002037041005678475, + 0.002060541999526322, + 0.0018814999930327758, + 0.0019550830038497224, + 0.001785208994988352, + 0.0017840420041466132, + 0.0018382919952273369, + 0.0019420419994276017, + 0.0018969170050695539, + 0.002040167004452087, + 0.0017981660057557747, + 0.0021406250016298145 + ] + }, + { + "forward": 0.011742957998649217, + "prefill": 0.011087625010986812, + "decode_steps": [ + 0.0029525840072892606, + 0.001980375003768131, + 0.0019459579925751314, + 0.002033749988186173, + 0.001848375002737157, + 0.002404041006229818, + 0.0019080000056419522, + 0.0019244160066591576, + 0.002009083007578738, + 0.0019209589954698458, + 0.0019063329964410514, + 0.001876333000836894, + 0.0018344170093769208, + 0.0021157089940970764, + 0.0019724580051843077, + 0.001875666988780722, + 0.0017835000035120174, + 0.0018783330015139654, + 0.0018757910002022982, + 0.0019282089924672619, + 0.001848333005909808, + 0.0020088339952053502, + 0.0019296249956823885, + 0.0018526670028222725, + 0.0018364580027991906, + 0.0019191250030416995, + 0.0018817089876392856, + 0.0019464590004645288, + 0.0018608749960549176, + 0.0019105829996988177, + 0.0019868330127792433, + 0.0018304999975953251 + ] + }, + { + "forward": 0.01193558301019948, + "prefill": 0.01163516599626746, + "decode_steps": [ + 0.00251816600211896, + 0.0021274580067256466, + 0.002188958998885937, + 0.0020303750061430037, + 0.0018533750117057934, + 0.0018932499951915815, + 0.001841707999119535, + 0.0018681669898796827, + 0.002023415989242494, + 0.0017879579972941428, + 0.0022111669968580827, + 0.0019105829996988177, + 0.001967792006325908, + 0.0018773750052787364, + 0.001843249992816709, + 0.0019761250005103648, + 0.001981292007258162, + 0.0018143750057788566, + 0.0018757499929051846, + 0.0018490829970687628, + 0.0019105829996988177, + 0.0018737499922281131, + 0.0018297500064363703, + 0.0019100839999737218, + 0.001959834000444971, + 0.001784957989002578, + 0.001938790999702178, + 0.0019540420034900308, + 0.001765916997101158, + 0.0018642079958226532, + 0.0018503750034142286, + 0.0018347919976804405 + ] + }, + { + "forward": 0.011281667000730522, + "prefill": 0.011844458000268787, + "decode_steps": [ + 0.0022194579942151904, + 0.0019734160014195368, + 0.0018753750046016648, + 0.0019231659971410409, + 0.001879834002465941, + 0.0019699580006999895, + 0.0020154999947408214, + 0.001859625001088716, + 0.0019549580028979108, + 0.0018496669945307076, + 0.001947083990671672, + 0.0018452920048730448, + 0.0019122499943478033, + 0.0018297920032637194, + 0.0018526250059949234, + 0.0018227499967906624, + 0.0018469999922672287, + 0.0018198339967057109, + 0.0018705420079641044, + 0.0019470419938443229, + 0.0018769580055959523, + 0.0018486670014681295, + 0.00194620800903067, + 0.0018237089971080422, + 0.00201237500004936, + 0.0018370000034337863, + 0.001956708001671359, + 0.001870457999757491, + 0.001859249998233281, + 0.0018083339964505285, + 0.00185087499266956, + 0.001897499998449348 + ] + }, + { + "forward": 0.011528957998962142, + "prefill": 0.01209741600905545, + "decode_steps": [ + 0.0030047500040382147, + 0.0021107500069774687, + 0.0019674169889185578, + 0.0020890000014333054, + 0.0021126670035300776, + 0.002006667011301033, + 0.0019027499947696924, + 0.0019656249933177605, + 0.001947583004948683, + 0.0019070830021519214, + 0.0019464580109342933, + 0.0020016249909531325, + 0.0020047089928993955, + 0.0018545000057201833, + 0.0019004579953616485, + 0.001940791989909485, + 0.0018455829995218664, + 0.0018620000046212226, + 0.00186191700049676, + 0.0018806250009220093, + 0.0018947499920614064, + 0.0018097910069627687, + 0.0017244159971596673, + 0.002044749999186024, + 0.0019128330022795126, + 0.0018048329948214814, + 0.0021893329976592213, + 0.002069124995614402, + 0.0018390829936834052, + 0.001955958010512404, + 0.00190395800746046, + 0.0018917080014944077 + ] + }, + { + "forward": 0.01202341700263787, + "prefill": 0.012032208003802225, + "decode_steps": [ + 0.0023431249865097925, + 0.0020360000053187832, + 0.002229667006758973, + 0.002044416993157938, + 0.0020374169980641454, + 0.0019447079976089299, + 0.0020697500003734604, + 0.0020405830000527203, + 0.001979624998057261, + 0.002166207996197045, + 0.002031374999205582, + 0.0019969160057371482, + 0.0020644580072257668, + 0.002055624994682148, + 0.0020884170080535114, + 0.0020293749985285103, + 0.0019612500036600977, + 0.002019375009695068, + 0.0019066249951720238, + 0.0019272500067017972, + 0.0019277919927844778, + 0.002044666005531326, + 0.0019991250010207295, + 0.0019945420062867925, + 0.0020276670111343265, + 0.001870124993729405, + 0.0018879579874919727, + 0.0020074580097571015, + 0.001953958999365568, + 0.0019333329983055592, + 0.0020432500023161992, + 0.0019515000021783635 + ] + }, + { + "forward": 0.012187708998681046, + "prefill": 0.012224582998896949, + "decode_steps": [ + 0.002492749990778975, + 0.002030874995398335, + 0.001958041000762023, + 0.0019887920061592013, + 0.0020093750063097104, + 0.0019277919927844778, + 0.0019114169990643859, + 0.0019065829983446747, + 0.001908209000248462, + 0.0018858330004150048, + 0.0019471249979687855, + 0.0019758330017793924, + 0.001967333009815775, + 0.0020010419975733384, + 0.001919417001772672, + 0.0019560420041671023, + 0.0018719590007094666, + 0.0020580830023391172, + 0.00205966699286364, + 0.0020396250038174912, + 0.0020265829953132197, + 0.002016166996327229, + 0.001973334001377225, + 0.0019963750091847032, + 0.0020275419956305996, + 0.0018648330005817115, + 0.002001125001697801, + 0.001994709004065953, + 0.001989500000490807, + 0.0020510829926934093, + 0.0021141670003999025, + 0.0019871669937856495 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 11.62072900478961, + "p95": 12.113777600461617, + "min": 11.119290997157805, + "max": 12.187708998681046, + "iqr": 0.5439370070234872, + "mean": 11.624404102622066, + "cv": 0.031038615170642186 + }, + "prefill_ms": { + "count": 10, + "median": 11.739811998268124, + "p95": 12.265122949611396, + "min": 11.087625010986812, + "max": 12.298292000195943, + "iqr": 0.7503115048166364, + "mean": 11.71032060374273, + "cv": 0.038422516523132805 + }, + "decode_ms_per_step": { + "count": 320, + "median": 1.9262709975009784, + "p95": 2.4018645468459, + "min": 1.7244159971596673, + "max": 3.0047500040382147, + "iqr": 0.16075000894488767, + "mean": 1.9710338810000394, + "cv": 0.09670436288371072 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 506.44278355244296, + "p95": 525.9881363123125, + "min": 489.9965221983352, + "max": 526.2880036792116, + "iqr": 15.890903350075348, + "mean": 507.61292020646516, + "cv": 0.024160462669429245 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 427.3197424899289, + "p95": 442.4885124290015, + "min": 417.13285079887635, + "max": 444.1300927981595, + "iqr": 15.688150033018417, + "mean": 428.1062748642286, + "cv": 0.02307749207381846 + }, + "memory": { + "peak_extra_bytes": 8650752, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "reference", + "batch": 1, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.06426029199792538, + "prefill": 0.058276749987271614, + "decode_steps": [ + 0.006115041993325576, + 0.0051582079904619604, + 0.004953084004228003, + 0.0039469589974032715, + 0.004364666005130857, + 0.004369291011244059, + 0.004517624998698011, + 0.004390665999380872, + 0.004902041997411288, + 0.004879541011177935, + 0.004505792006966658, + 0.004324624998844229, + 0.004640958009986207, + 0.005047917002229951, + 0.004792665989953093, + 0.006910249998327345, + 0.005386374992667697, + 0.006430499997804873, + 0.004887749993940815, + 0.00489804100652691, + 0.004997249998268671, + 0.005304291000356898, + 0.004802042007213458, + 0.004707458007032983, + 0.00509541601059027, + 0.004655875003663823, + 0.004311957993195392, + 0.004194624998490326, + 0.004592166995280422, + 0.0042444589926162735, + 0.004935959004797041, + 0.005370291008148342 + ] + }, + { + "forward": 0.06583350000437349, + "prefill": 0.06275912499404512, + "decode_steps": [ + 0.00641304200689774, + 0.005429541997727938, + 0.005121209003846161, + 0.004434583999682218, + 0.0046965420042397454, + 0.004571165991364978, + 0.004523916999460198, + 0.004521041992120445, + 0.004857000007177703, + 0.004632625001249835, + 0.005286791987600736, + 0.005852042013430037, + 0.00499787500302773, + 0.005778457998530939, + 0.005670500002452172, + 0.005437084008008242, + 0.006967833993257955, + 0.005321499993442558, + 0.005168082992895506, + 0.00568366699735634, + 0.005047083002864383, + 0.004764541998156346, + 0.006253624989767559, + 0.005131958998390473, + 0.004689708002842963, + 0.005327624996425584, + 0.004941249993862584, + 0.004503165997448377, + 0.005865707993507385, + 0.005200375002459623, + 0.0045367499988060445, + 0.004851958001381718 + ] + }, + { + "forward": 0.06802795799740124, + "prefill": 0.06347475000075065, + "decode_steps": [ + 0.006392874987795949, + 0.005094666994409636, + 0.005442333000246435, + 0.004788791004102677, + 0.00505691600847058, + 0.00467449999996461, + 0.004774749992066063, + 0.004228667006827891, + 0.004377749995910563, + 0.005363124990253709, + 0.004606041999068111, + 0.004705250001279637, + 0.005143917005625553, + 0.004785915996762924, + 0.004667832996346988, + 0.004550875004497357, + 0.005287624997436069, + 0.004566709001665004, + 0.005182834007428028, + 0.005735500002629124, + 0.004202083000564016, + 0.004699500001152046, + 0.0052991669945186, + 0.0046279580128612, + 0.0045963330048834905, + 0.005203124994295649, + 0.005204458997468464, + 0.00467916700290516, + 0.004166750004515052, + 0.004742874996736646, + 0.004532541002845392, + 0.004499832997680642 + ] + }, + { + "forward": 0.06397991599806119, + "prefill": 0.061260583010152914, + "decode_steps": [ + 0.0067037919943686575, + 0.004881707995082252, + 0.005290665998472832, + 0.005097749992273748, + 0.004351541996584274, + 0.004432542002177797, + 0.004949541995301843, + 0.004294957994716242, + 0.005236832992522977, + 0.004593582998495549, + 0.005044957986683585, + 0.004762500000651926, + 0.004958542005624622, + 0.004663874991820194, + 0.004642792002414353, + 0.004694957999163307, + 0.004713415997684933, + 0.0043029169901274145, + 0.0045196249993750826, + 0.004620874999091029, + 0.004605791997164488, + 0.0045179169974289834, + 0.004460916999960318, + 0.004658542005927302, + 0.0041409999976167455, + 0.004929625007207505, + 0.004822292001335882, + 0.004432458008523099, + 0.005044624995207414, + 0.005197375008719973, + 0.0045615419949172065, + 0.004459000003407709 + ] + }, + { + "forward": 0.06423324999923352, + "prefill": 0.062037750001763925, + "decode_steps": [ + 0.006350124996970408, + 0.004907333001028746, + 0.00481095899885986, + 0.0043190840078750625, + 0.004340333005529828, + 0.005006916006095707, + 0.004563249996863306, + 0.004452084001968615, + 0.004727457999251783, + 0.004690083995228633, + 0.004525625001406297, + 0.004951875002007, + 0.004671791000873782, + 0.004226124990964308, + 0.004564084010780789, + 0.004852666999795474, + 0.004760084004374221, + 0.004579166998155415, + 0.0045487500028684735, + 0.004350415998487733, + 0.004329500006861053, + 0.004169541993178427, + 0.0044790420070057735, + 0.00441766700532753, + 0.004533291998086497, + 0.004345540990470909, + 0.004824125004233792, + 0.004516624991083518, + 0.004511207996984012, + 0.004624875000445172, + 0.0045514170051319525, + 0.00438962499902118 + ] + }, + { + "forward": 0.059984458013786934, + "prefill": 0.05788587499409914, + "decode_steps": [ + 0.004935707998811267, + 0.004646917004720308, + 0.004578916996251792, + 0.004614958001184277, + 0.004369417001726106, + 0.0049670000007608905, + 0.0045460410037776455, + 0.004928166003082879, + 0.004620916006388143, + 0.0045809579896740615, + 0.004590458993334323, + 0.0044759580050595105, + 0.00463691700133495, + 0.004421207995619625, + 0.004403458995511755, + 0.004494791995966807, + 0.004389875000924803, + 0.004501709001488052, + 0.004368042005808093, + 0.004394959003548138, + 0.00423262499680277, + 0.004320958003518172, + 0.004396083008032292, + 0.004351208001025952, + 0.004602209010045044, + 0.0044723750033881515, + 0.004539499990642071, + 0.0043996250024065375, + 0.0044708749919664115, + 0.004447874991456047, + 0.0045425420103129, + 0.004444457998033613 + ] + }, + { + "forward": 0.06226029200479388, + "prefill": 0.05892820900771767, + "decode_steps": [ + 0.005951957995421253, + 0.004983292004908435, + 0.004602249988238327, + 0.0046319999964907765, + 0.003909209000994451, + 0.004523334006080404, + 0.004505291988607496, + 0.004378541998448782, + 0.0047123330004978925, + 0.004436040995642543, + 0.00427187500463333, + 0.004361083003459498, + 0.004418415992404334, + 0.004322416993090883, + 0.004321958011132665, + 0.0044462920050136745, + 0.004422750003868714, + 0.004324416004237719, + 0.004359292011940852, + 0.004019792002509348, + 0.004423416001372971, + 0.004837166998186149, + 0.004997750002075918, + 0.004670583002734929, + 0.004653625001083128, + 0.004437124996911734, + 0.004588208001223393, + 0.004337042002589442, + 0.004203874996164814, + 0.004299417007132433, + 0.004738291012472473, + 0.004502000010688789 + ] + }, + { + "forward": 0.061727292006253265, + "prefill": 0.058396666005137376, + "decode_steps": [ + 0.0060908750019734725, + 0.004952250004862435, + 0.004471207997994497, + 0.005044584002462216, + 0.004574207996483892, + 0.0044616660015890375, + 0.004550083001959138, + 0.00449691699759569, + 0.00455691700335592, + 0.004457750008441508, + 0.004549749995931052, + 0.00420737499371171, + 0.004365083994343877, + 0.00436700000136625, + 0.0043851670052390546, + 0.004499000002397224, + 0.004334166995249689, + 0.00450887500483077, + 0.004228333011269569, + 0.0045053329959046096, + 0.004534541003522463, + 0.004318333012633957, + 0.004408332999446429, + 0.004400125006213784, + 0.0047242910077329725, + 0.004470750005566515, + 0.004376500000944361, + 0.004372499999590218, + 0.0044278329878579825, + 0.004275708008208312, + 0.004382167011499405, + 0.004397333002998494 + ] + }, + { + "forward": 0.0612541250011418, + "prefill": 0.05906487499305513, + "decode_steps": [ + 0.005410042009316385, + 0.00517375000345055, + 0.004683166989707388, + 0.0046958339953562245, + 0.004265041992766783, + 0.004889708987320773, + 0.004808500001672655, + 0.015302792002330534, + 0.008790583000518382, + 0.005243416992016137, + 0.004200374998617917, + 0.005128499993588775, + 0.004676249998738058, + 0.004505874996539205, + 0.004712958994787186, + 0.004185999991022982, + 0.004502499999944121, + 0.004840708003030159, + 0.004652000003261492, + 0.004380667000077665, + 0.004285249990061857, + 0.004341499996371567, + 0.004849958000704646, + 0.004993500013370067, + 0.004304917005356401, + 0.004993375012418255, + 0.004990791989257559, + 0.00443745800293982, + 0.0046866250049788505, + 0.0045232080010464415, + 0.004624707988114096, + 0.004622958993422799 + ] + }, + { + "forward": 0.06269045900262427, + "prefill": 0.05995499998971354, + "decode_steps": [ + 0.006030709002516232, + 0.004949790993123315, + 0.005145832998096012, + 0.004633667005691677, + 0.004931542003760114, + 0.004914874996757135, + 0.004612666001776233, + 0.004802583003765903, + 0.005227375004324131, + 0.005055332992924377, + 0.004800082999281585, + 0.004702249992988072, + 0.004896041005849838, + 0.004734499991172925, + 0.004757167000207119, + 0.004702125006588176, + 0.0045658750022994354, + 0.004722583005786873, + 0.004790708000655286, + 0.00474120800208766, + 0.004639999999199063, + 0.004684458006522618, + 0.004980666999472305, + 0.004799665999598801, + 0.004776333007612266, + 0.004939165999530815, + 0.004686959000537172, + 0.004662624996853992, + 0.004562458998407237, + 0.004549707999103703, + 0.004632958007277921, + 0.00472345799789764 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 63.33518750034273, + "p95": 67.04045190053876, + "min": 59.984458013786934, + "max": 68.02795799740124, + "iqr": 2.392989492363995, + "mean": 63.4251542025595, + "cv": 0.037153103781733705 + }, + "prefill_ms": { + "count": 10, + "median": 59.509937491384335, + "p95": 63.15271874773316, + "min": 57.88587499409914, + "max": 63.474750000750646, + "iqr": 3.3139064980787225, + "mean": 60.20395829837071, + "cv": 0.03370010575367892 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.632791504263878, + "p95": 5.782137199275895, + "min": 3.9092090009944513, + "max": 15.302792002330534, + "iqr": 0.48674999925424345, + "mean": 4.7890713532979134, + "cv": 0.16391285920547172 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 208.51411425559135, + "p95": 221.24799972627738, + "min": 192.21729823070405, + "max": 221.31385022043273, + "iqr": 15.415242544567803, + "mean": 209.3130502829469, + "cv": 0.051311829515556 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 149.4772095659568, + "p95": 157.78639692731525, + "min": 139.59329095564607, + "max": 157.96806935602507, + "iqr": 9.396922347952568, + "mean": 150.15328484143527, + "cv": 0.04180985390848837 + }, + "memory": { + "peak_extra_bytes": 1082392576, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "auto", + "batch": 4, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.006336167003610171 + }, + { + "forward": 0.00391466700239107 + }, + { + "forward": 0.004705416009528562 + }, + { + "forward": 0.003569916996639222 + }, + { + "forward": 0.0034085419902112335 + }, + { + "forward": 0.005837499993504025 + }, + { + "forward": 0.003698832995723933 + }, + { + "forward": 0.0035958749940618873 + }, + { + "forward": 0.003460791995166801 + }, + { + "forward": 0.003587583007174544 + } + ], + "forward_ms": { + "count": 10, + "median": 3.64735399489291, + "p95": 6.111766849062405, + "min": 3.4085419902112335, + "max": 6.336167003610171, + "iqr": 0.9333952584711369, + "mean": 4.211529198801145, + "cv": 0.2520478977279319 + }, + "memory": { + "peak_extra_bytes": 9240576, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "sdpa", + "batch": 4, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.0037388339987955987, + "prefill": 0.0033984170004259795, + "decode_steps": [ + 0.0035557079972932115, + 0.007715375002590008, + 0.003448999996180646, + 0.0030150840029818937, + 0.002081124999676831, + 0.004808874989976175, + 0.0021823330025654286, + 0.0020367500110296533, + 0.0018802919948939234, + 0.0019783330062637106, + 0.0022564589889952913, + 0.004355875003966503, + 0.0021639170008711517, + 0.0023497500078519806, + 0.001986874995054677, + 0.001928375000716187, + 0.002588791001471691, + 0.006517999994684942, + 0.002661417005583644, + 0.0024012909998418763, + 0.0030755420011701062, + 0.00645716700819321, + 0.0028782909939764068, + 0.002729999992880039, + 0.002738750001299195, + 0.0025836669956333935, + 0.005295541006489657, + 0.002620041006593965, + 0.002586708011222072, + 0.002502540999557823, + 0.0030096250120550394, + 0.002908042006311007 + ] + }, + { + "forward": 0.0030810830066911876, + "prefill": 0.003027290993486531, + "decode_steps": [ + 0.0020167089969618246, + 0.00394925000728108, + 0.0018680000066524372, + 0.001890832994831726, + 0.0017928750021383166, + 0.0018520830053603277, + 0.0018031250074272975, + 0.0017672920075710863, + 0.0017743330099619925, + 0.001825665996875614, + 0.0017059579986380413, + 0.0017756669985828921, + 0.0018018330010818318, + 0.0018329160084249452, + 0.001742041000397876, + 0.0019208330049877986, + 0.0030207499949028715, + 0.0018858340044971555, + 0.0018025000026682392, + 0.001914833002956584, + 0.001887958002043888, + 0.0018781250109896064, + 0.0017922080005519092, + 0.0018437919934513047, + 0.0018210419948445633, + 0.0018923339957837015, + 0.0017914170020958409, + 0.0018377090018475428, + 0.0017716250004014, + 0.0017802499933168292, + 0.0018399170076008886, + 0.0017628330097068101 + ] + }, + { + "forward": 0.003067708996240981, + "prefill": 0.0030786249990342185, + "decode_steps": [ + 0.0018925829936051741, + 0.0018866250029532239, + 0.0017831250006565824, + 0.001917416011565365, + 0.0017342499922960997, + 0.001793915987946093, + 0.0017489160090917721, + 0.0018338340014452115, + 0.002001458007725887, + 0.001876791997347027, + 0.0017395000031683594, + 0.0017781670030672103, + 0.00380474999838043, + 0.0018394999933661893, + 0.0019420419994276017, + 0.0017757919995347038, + 0.0017925830034073442, + 0.0020119590044487268, + 0.0017979580006795004, + 0.0017942919948836789, + 0.0018057909910567105, + 0.001836584007833153, + 0.0018170419934904203, + 0.001808417000574991, + 0.0018578749877633527, + 0.0019456250010989606, + 0.0018214589945273474, + 0.002945041997008957, + 0.0019105829996988177, + 0.0019231249898439273, + 0.00180170800013002, + 0.0019660419930005446 + ] + }, + { + "forward": 0.003323291995911859, + "prefill": 0.003054917004192248, + "decode_steps": [ + 0.001891000007162802, + 0.001787999994121492, + 0.0018817920063156635, + 0.0019073750008828938, + 0.0018005000019911677, + 0.0017682080069789663, + 0.002790167010971345, + 0.0018703340028878301, + 0.001773541996954009, + 0.0018764590058708563, + 0.0018866250029532239, + 0.0018801669939421117, + 0.0018098750006174669, + 0.0017791249993024394, + 0.0019465420045889914, + 0.0017901670071296394, + 0.0017282089975196868, + 0.001955250001628883, + 0.001846874991315417, + 0.0018208750116173178, + 0.0018129579984815791, + 0.0017872909957077354, + 0.0017972500063478947, + 0.001917500005220063, + 0.0017429169965907931, + 0.0018988750089192763, + 0.0019046249944949523, + 0.0019026249938178807, + 0.0017714999994495884, + 0.002379375000600703, + 0.0018274169997312129, + 0.0018319590017199516 + ] + }, + { + "forward": 0.0032755839929450303, + "prefill": 0.003278125004726462, + "decode_steps": [ + 0.0019061659986618906, + 0.0018516659911256284, + 0.0030275839962996542, + 0.001866499995230697, + 0.0017758339963620529, + 0.0019117499905405566, + 0.0018075840052915737, + 0.0018136250000679865, + 0.0017633329989621416, + 0.002289166994160041, + 0.001941875001648441, + 0.001879792005638592, + 0.0017979999975068495, + 0.0019132090092170984, + 0.0019473330030450597, + 0.0019214579951949418, + 0.0017511670012027025, + 0.0019144579855492339, + 0.0033538330026203766, + 0.0018829999899026006, + 0.0019334169919602573, + 0.0019164580007782206, + 0.0019165410049026832, + 0.001865541998995468, + 0.0018901250005001202, + 0.0018139579915441573, + 0.0018598750029923394, + 0.0018854999943869188, + 0.0018464589957147837, + 0.0018655830062925816, + 0.0018207920074928552, + 0.0018669999990379438 + ] + }, + { + "forward": 0.00394645900814794, + "prefill": 0.003861500008497387, + "decode_steps": [ + 0.002107374995830469, + 0.002156332993763499, + 0.002347916000871919, + 0.001979084001504816, + 0.0022944579977774993, + 0.0020431249868124723, + 0.0018390409968560562, + 0.0019048329995712265, + 0.0018730419978965074, + 0.0018457909900462255, + 0.001818499993532896, + 0.001896916000987403, + 0.0018252500012749806, + 0.0018221659993287176, + 0.0017515419895062223, + 0.0020907500002067536, + 0.0019555839971872047, + 0.0017934159986907616, + 0.0018728750001173466, + 0.0019228750024922192, + 0.0018015410023508593, + 0.0018750000017462298, + 0.0018549999949755147, + 0.00197725000907667, + 0.0018057909910567105, + 0.0019357499986654148, + 0.0019301249994896352, + 0.0018728750001173466, + 0.0019498339970596135, + 0.0019620419916464016, + 0.001945875003002584, + 0.0019652499904623255 + ] + }, + { + "forward": 0.0029820839990861714, + "prefill": 0.003035791000002064, + "decode_steps": [ + 0.0019374160037841648, + 0.0018060830043395981, + 0.0018482089944882318, + 0.0018362080008955672, + 0.001829875007388182, + 0.0016918749897740781, + 0.0018265830003656447, + 0.001851958004408516, + 0.0018314170010853559, + 0.0017222500027855858, + 0.0018312909960513934, + 0.0018065839976770803, + 0.0018894590029958636, + 0.0017414579924661666, + 0.0018650419951882213, + 0.00184116700256709, + 0.0018201250059064478, + 0.0021806250006193295, + 0.0018221670034108683, + 0.0017362090002279729, + 0.001808750006603077, + 0.001810667003155686, + 0.0018239589990116656, + 0.001846832994488068, + 0.0018402089917799458, + 0.0018113339901901782, + 0.0018182499916292727, + 0.0018865000020014122, + 0.00182245799805969, + 0.0018933749961433932, + 0.0018685830000322312, + 0.0019856250000884756 + ] + }, + { + "forward": 0.0033500840072520077, + "prefill": 0.003250167006626725, + "decode_steps": [ + 0.0019067499961238354, + 0.0019204999989597127, + 0.0019273330108262599, + 0.001778499994543381, + 0.0017280830070376396, + 0.0019401250028749928, + 0.001913709013024345, + 0.0017856669874163345, + 0.0017655830015428364, + 0.0017700839962344617, + 0.0018075410043820739, + 0.001810041008866392, + 0.0017180419963551685, + 0.001757374993758276, + 0.0017321670020464808, + 0.0017810419958550483, + 0.001956374995643273, + 0.0018365839932812378, + 0.001795459000277333, + 0.0018564579950179905, + 0.001753749995259568, + 0.0018548750085756183, + 0.0019143340032314882, + 0.0019170419982401654, + 0.0017786669923225418, + 0.0017550829943502322, + 0.0017728340026224032, + 0.0019629159942269325, + 0.0018930829974124208, + 0.0018369589961366728, + 0.0021063330059405416, + 0.001929166988702491 + ] + }, + { + "forward": 0.0030812500044703484, + "prefill": 0.0031445420027012005, + "decode_steps": [ + 0.001924750002217479, + 0.001814041010220535, + 0.0017613749951124191, + 0.0018162089982070029, + 0.0018783749983413145, + 0.0018869170016841963, + 0.001801167003577575, + 0.0019036249868804589, + 0.0018996659928234294, + 0.0018129999953089282, + 0.0018265419930685312, + 0.0019650419999379665, + 0.0019930419948650524, + 0.002136374998372048, + 0.0018401249981252477, + 0.0018222079961560667, + 0.0019174579938407987, + 0.0018507500062696636, + 0.0019087079999735579, + 0.0018036660039797425, + 0.0018890420033130795, + 0.001906082994537428, + 0.0018021249998128042, + 0.0017970829940168187, + 0.00178358300763648, + 0.0018328750011278316, + 0.0018829579930752516, + 0.0018507920030970126, + 0.0018933329993160442, + 0.0019057500030612573, + 0.0017966250015888363, + 0.0017522499983897433 + ] + }, + { + "forward": 0.0029679590079467744, + "prefill": 0.003042917000129819, + "decode_steps": [ + 0.001976249986910261, + 0.0018938330031232908, + 0.0018568340019555762, + 0.0018424579902784899, + 0.0018454170058248565, + 0.0017694160051178187, + 0.0017882909887703136, + 0.0018086250056512654, + 0.0018181670020567253, + 0.0017550830089021474, + 0.0017398330091964453, + 0.0017936249932972714, + 0.0018187920068157837, + 0.0017925830034073442, + 0.001795249991118908, + 0.0017562080029165372, + 0.001797958990209736, + 0.0017324999935226515, + 0.0017526250012451783, + 0.0017856670019682497, + 0.0018144159985240549, + 0.0017941249971045181, + 0.001758041005814448, + 0.0017345840024063364, + 0.001744584005791694, + 0.0017933339986484498, + 0.0018128749943571165, + 0.0017462090036133304, + 0.0018578749877633527, + 0.0017768329998943955, + 0.0018125829956261441, + 0.0020441249944269657 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 3.1784169987076893, + "p95": 3.8530277539393865, + "min": 2.9679590079467744, + "max": 3.94645900814794, + "iqr": 0.27233350556343794, + "mean": 3.28143380174879, + "cv": 0.10013606329179435 + }, + "prefill_ms": { + "count": 10, + "median": 3.1115835008677095, + "p95": 3.6531126548652533, + "min": 3.027290993486531, + "max": 3.861500008497387, + "iqr": 0.22521850405610166, + "mean": 3.2172292019822635, + "cv": 0.08045063241900917 + }, + "decode_ms_per_step": { + "count": 320, + "median": 1.8520205048844218, + "p95": 2.948271147761262, + "min": 1.6918749897740781, + "max": 7.715375002590008, + "iqr": 0.12439550118870102, + "mean": 2.018336077753702, + "cv": 0.3175443570634946 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 2095.0966800345095, + "p95": 2195.192113474799, + "min": 1263.5792813288842, + "max": 2214.1881031079174, + "iqr": 104.7947611833356, + "mean": 2031.7048167665441, + "cv": 0.13593305998983263 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1995.7317851712776, + "p95": 2086.4138383826694, + "min": 1222.564447491359, + "max": 2103.4670123812857, + "iqr": 105.10406446308184, + "mean": 1932.0131083269753, + "cv": 0.13226629917881072 + }, + "memory": { + "peak_extra_bytes": 8650752, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "reference", + "batch": 4, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.010372832999564707, + "prefill": 0.008201292002922855, + "decode_steps": [ + 0.005429791999631561, + 0.005564999999478459, + 0.004588541996781714, + 0.004569209006149322, + 0.004692917005741037, + 0.004350707997218706, + 0.00449541601119563, + 0.0044577079970622435, + 0.004695458002970554, + 0.004172291999566369, + 0.0043076250003650784, + 0.004163374993368052, + 0.004267541997251101, + 0.004256208005244844, + 0.004074249998666346, + 0.005153875012183562, + 0.004361957995570265, + 0.0046826250036247075, + 0.004579208994982764, + 0.003832750007859431, + 0.004396834003273398, + 0.004307791998144239, + 0.0043158329935977235, + 0.0040857499989215285, + 0.0043460830056574196, + 0.0043009999935748056, + 0.005528041991055943, + 0.004326041002059355, + 0.004374542011646554, + 0.004383040999528021, + 0.0043380830029491335, + 0.004371333008748479 + ] + }, + { + "forward": 0.007394958010991104, + "prefill": 0.00428720899799373, + "decode_steps": [ + 0.00446833400928881, + 0.004434083995874971, + 0.004473042004974559, + 0.003806249995250255, + 0.004477334005059674, + 0.00406624999595806, + 0.00431633299740497, + 0.004321999993408099, + 0.004821249996894039, + 0.004281249988707714, + 0.0044059170031687245, + 0.0043107090023113415, + 0.004241582995746285, + 0.00415633300144691, + 0.00425037500099279, + 0.004316249993280508, + 0.004297624996979721, + 0.004307957991841249, + 0.004152332985540852, + 0.004128500004298985, + 0.004437792013050057, + 0.004450541993719526, + 0.004556792002404109, + 0.004498875001445413, + 0.0051032089977525175, + 0.004628124996088445, + 0.011895292002009228, + 0.005372583007556386, + 0.004503500007558614, + 0.004636707992176525, + 0.004669667003327049, + 0.004324415989685804 + ] + }, + { + "forward": 0.006821458009653725, + "prefill": 0.00434158300049603, + "decode_steps": [ + 0.004448374995263293, + 0.0043120000045746565, + 0.00424179200490471, + 0.0042369170114398, + 0.004471375010325573, + 0.004382665996672586, + 0.004385249994811602, + 0.003971665995777585, + 0.0042577499989420176, + 0.004310083997552283, + 0.004235333006363362, + 0.003953207997255959, + 0.00417225000273902, + 0.004302124987589195, + 0.004245332995196804, + 0.004386082990095019, + 0.004162084005656652, + 0.00452945900906343, + 0.0068292500072857365, + 0.004368833004264161, + 0.004322082997532561, + 0.004019291009171866, + 0.004228624995448627, + 0.004465249992790632, + 0.004427957988809794, + 0.0054139580024639145, + 0.004434917005710304, + 0.004513416002737358, + 0.0040379579877480865, + 0.0043485409987624735, + 0.0042636669968487695, + 0.004316999998991378 + ] + }, + { + "forward": 0.007896791998064145, + "prefill": 0.004182042001048103, + "decode_steps": [ + 0.004708959007984959, + 0.004762208991451189, + 0.004350082992459647, + 0.004552291007712483, + 0.00435208399721887, + 0.004323457993450575, + 0.005397040993557312, + 0.004250541009241715, + 0.004388917004689574, + 0.004130582994548604, + 0.004934083990519866, + 0.004281957997591235, + 0.005257917000562884, + 0.006083166997996159, + 0.00476045798859559, + 0.00495204099570401, + 0.004184457997325808, + 0.004510166996624321, + 0.004747041006339714, + 0.004728083993541077, + 0.004213499996694736, + 0.0045226670044939965, + 0.004647750000003725, + 0.004247834003763273, + 0.004476959002204239, + 0.004544457988231443, + 0.004285667004296556, + 0.003931625004042871, + 0.004119666991755366, + 0.004411041009007022, + 0.00489620900771115, + 0.0043820830032927915 + ] + }, + { + "forward": 0.007172000012360513, + "prefill": 0.004270665987860411, + "decode_steps": [ + 0.004465584002900869, + 0.0038526669959537685, + 0.004165999998804182, + 0.004821834008907899, + 0.004946000000927597, + 0.007229708004160784, + 0.0066317079908913, + 0.013987208003527485, + 0.017631333990721032, + 0.011951749998843297, + 0.008859499997925013, + 0.00809229101287201, + 0.0071600829978706315, + 0.005908500010264106, + 0.004720958997495472, + 0.005371250008465722, + 0.004969540997990407, + 0.004747874991153367, + 0.005204790999414399, + 0.0042533329979050905, + 0.004586124996421859, + 0.005079583002952859, + 0.0048165839980356395, + 0.0047047499974723905, + 0.004229250000207685, + 0.004404374994919635, + 0.005464750007377006, + 0.004875667000305839, + 0.0058492090029176325, + 0.0047843749925959855, + 0.004445208003744483, + 0.00533991600968875 + ] + }, + { + "forward": 0.006854999999632128, + "prefill": 0.006859582994366065, + "decode_steps": [ + 0.004834457999095321, + 0.004646125002182089, + 0.005221166997216642, + 0.004709375003585592, + 0.004543707997072488, + 0.0041693749953992665, + 0.005288000000291504, + 0.004751915999804623, + 0.005663124989951029, + 0.004791625004145317, + 0.004925250002997927, + 0.00483525000163354, + 0.0053663750004488975, + 0.005255374999251217, + 0.0048873750056372955, + 0.005329124993295409, + 0.00513150000188034, + 0.0044647500035353005, + 0.005622124997898936, + 0.0045548329944722354, + 0.004731125009129755, + 0.004764124998473562, + 0.00478570799168665, + 0.005221375002292916, + 0.006509166996693239, + 0.020370707992697135, + 0.00789624999742955, + 0.011679709001327865, + 0.006090249997214414, + 0.004857583000557497, + 0.00503166600537952, + 0.005211041003349237 + ] + }, + { + "forward": 0.007989874997292645, + "prefill": 0.004221084003802389, + "decode_steps": [ + 0.004881916000158526, + 0.00478533300338313, + 0.004509374994086102, + 0.004537249988061376, + 0.004612999997334555, + 0.0053544169932138175, + 0.004539708999800496, + 0.004491749990847893, + 0.004705208993982524, + 0.004485999990720302, + 0.003940833994420245, + 0.0043956250010523945, + 0.00433729198994115, + 0.004551707999780774, + 0.004432542002177797, + 0.0045449579920386896, + 0.004453292000107467, + 0.004282042005797848, + 0.004525083000771701, + 0.004393625000375323, + 0.004264709001290612, + 0.004379292004159652, + 0.004371000002720393, + 0.0038537910004379228, + 0.004385833002743311, + 0.004240500013111159, + 0.004357041005278006, + 0.005048584003816359, + 0.004713832997367717, + 0.004658292004023679, + 0.004407624990562908, + 0.004599915992002934 + ] + }, + { + "forward": 0.007320332995732315, + "prefill": 0.004802750001545064, + "decode_steps": [ + 0.004872916004387662, + 0.004822333008632995, + 0.004293667007004842, + 0.004782999996677972, + 0.004720625001937151, + 0.004795458007720299, + 0.004588334006257355, + 0.0044944580004084855, + 0.004339792008977383, + 0.0059141669917153195, + 0.004725417005829513, + 0.004162083991104737, + 0.004355667013442144, + 0.0043847500055562705, + 0.00439466699026525, + 0.004374541007564403, + 0.004403541999636218, + 0.004556333995424211, + 0.004452207998838276, + 0.004479374998481944, + 0.004742291988804936, + 0.00453616700542625, + 0.00451450000400655, + 0.004440959004568867, + 0.004409333007060923, + 0.004593125006067567, + 0.004622541993740015, + 0.004736708011478186, + 0.004558208005619235, + 0.003989875011029653, + 0.004230500009725802, + 0.004381083999760449 + ] + }, + { + "forward": 0.006936999998288229, + "prefill": 0.004157000003033318, + "decode_steps": [ + 0.005285499995807186, + 0.004132292000576854, + 0.004435750000993721, + 0.004549625009531155, + 0.004615209007170051, + 0.004490291001275182, + 0.004143625003052875, + 0.004167332997894846, + 0.004276624997146428, + 0.004078250000020489, + 0.004500833005295135, + 0.004150250009843148, + 0.0045338750060182065, + 0.004462541997781955, + 0.0051732499996433035, + 0.005121999987750314, + 0.004548749988316558, + 0.004452542008948512, + 0.0042343340028310195, + 0.004231541999615729, + 0.004469583000172861, + 0.00418179199914448, + 0.0045427089935401455, + 0.004320666004787199, + 0.004319708998082206, + 0.0042430840112501755, + 0.0042638329905457795, + 0.004314208010328002, + 0.0038109589950181544, + 0.004201667004963383, + 0.004139500000746921, + 0.0041589169995859265 + ] + }, + { + "forward": 0.007721207992290147, + "prefill": 0.0044204169971635565, + "decode_steps": [ + 0.004940083003020845, + 0.004571582991047762, + 0.004730291999294423, + 0.0046366660099010915, + 0.004487625003093854, + 0.004332374999648891, + 0.004383207997307181, + 0.004233500003465451, + 0.004332541997428052, + 0.004239582995069213, + 0.004370249997009523, + 0.004323166998801753, + 0.003946875003748573, + 0.004193874992779456, + 0.004213665990391746, + 0.0042137080017710105, + 0.004513792009674944, + 0.0047130000020843, + 0.004497582995099947, + 0.004345833003753796, + 0.005875332994037308, + 0.0057762499927775934, + 0.005626334008411504, + 0.0047992079926189035, + 0.004600416999892332, + 0.00422141699527856, + 0.0044289579964242876, + 0.004373292002128437, + 0.00427783299528528, + 0.0039638330054003745, + 0.004325667003286071, + 0.004305624999688007 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 7.357645503361709, + "p95": 9.300501898542278, + "min": 6.821458009653725, + "max": 10.372832999564707, + "iqr": 0.8571459948143456, + "mean": 7.648145701386966, + "cv": 0.1366075826560465 + }, + "prefill_ms": { + "count": 10, + "median": 4.31439599924488, + "p95": 7.597522949072298, + "min": 4.157000003033318, + "max": 8.201292002922855, + "iqr": 0.4736872506327927, + "mean": 4.974362599023152, + "cv": 0.28060019179975454 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.466959006094839, + "p95": 5.922616992029363, + "min": 3.806249995250255, + "max": 20.370707992697135, + "iqr": 0.44630124830291606, + "mean": 4.809241284419841, + "cv": 0.3229845361144401 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 881.362531369453, + "p95": 909.4122214391504, + "min": 647.9316290758973, + "max": 910.7011618077526, + "iqr": 27.396611291039108, + "mean": 842.891515709857, + "cv": 0.11149790879676273 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 850.5310768733486, + "p95": 882.8195792941115, + "min": 634.2210601439664, + "max": 884.5396257881496, + "iqr": 25.571535990677376, + "mean": 816.1607136779043, + "cv": 0.11010335203999363 + }, + "memory": { + "peak_extra_bytes": 8650752, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "auto", + "batch": 4, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.00697883398970589 + }, + { + "forward": 0.006218166003236547 + }, + { + "forward": 0.009406249999301508 + }, + { + "forward": 0.00555224998970516 + }, + { + "forward": 0.005730832999688573 + }, + { + "forward": 0.007807665999280289 + }, + { + "forward": 0.006127249987912364 + }, + { + "forward": 0.005690000005415641 + }, + { + "forward": 0.005744165988289751 + }, + { + "forward": 0.00603545899502933 + } + ], + "forward_ms": { + "count": 10, + "median": 6.081354491470847, + "p95": 8.686887199291958, + "min": 5.55224998970516, + "max": 9.406249999301508, + "iqr": 1.054500746249687, + "mean": 6.529087395756505, + "cv": 0.18778079183636764 + }, + "memory": { + "peak_extra_bytes": 9830400, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "sdpa", + "batch": 4, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.005197958002099767, + "prefill": 0.00501695899583865, + "decode_steps": [ + 0.00451654200151097, + 0.001907000012579374, + 0.002190832994529046, + 0.0020657499990193173, + 0.0019596670026658103, + 0.0019271670025773346, + 0.002014541008975357, + 0.0027880419947905466, + 0.002044291002675891, + 0.002023749999352731, + 0.0020115420047659427, + 0.002003540997975506, + 0.0018645829986780882, + 0.0021092499955557287, + 0.0032601249986328185, + 0.002107207998051308, + 0.0019967090047430247, + 0.0019819160079350695, + 0.0021245829993858933, + 0.001981416004127823, + 0.003989916993305087, + 0.002136666007572785, + 0.002204125004936941, + 0.0019942910003010184, + 0.0020742909982800484, + 0.0019667090091388673, + 0.0038624169974355027, + 0.0022417909931391478, + 0.002014625002630055, + 0.002007083996431902, + 0.0019728750048670918, + 0.0019711669883690774 + ] + }, + { + "forward": 0.005436709005152807, + "prefill": 0.005477167011122219, + "decode_steps": [ + 0.0038208330079214647, + 0.0025510829873383045, + 0.002527125005144626, + 0.002309667004738003, + 0.002158624993171543, + 0.0025773329980438575, + 0.00209237499802839, + 0.0019168339931638911, + 0.002421958008198999, + 0.0024695829924894497, + 0.002091957998345606, + 0.002639749989612028, + 0.0027310000004945323, + 0.0025860419991659, + 0.0026002090016845614, + 0.0027764170081354678, + 0.0025419579906156287, + 0.00262079099775292, + 0.002633249998325482, + 0.001992542005609721, + 0.0021333330078050494, + 0.001953916987986304, + 0.002152166998712346, + 0.0030798340012552217, + 0.0024704999959794804, + 0.002468833001330495, + 0.001979583001229912, + 0.0019739999988814816, + 0.0018957919965032488, + 0.002045166998868808, + 0.0020716659928439185, + 0.0021854590013390407 + ] + }, + { + "forward": 0.007090541999787092, + "prefill": 0.006218959009856917, + "decode_steps": [ + 0.002258583001093939, + 0.0023141669953474775, + 0.0027229169936617836, + 0.0024506250047124922, + 0.00252341698796954, + 0.00259800000640098, + 0.0024442080029984936, + 0.0022813749965280294, + 0.002346583001781255, + 0.0022720420092809945, + 0.0022844159975647926, + 0.0023194590030470863, + 0.0022826670028734952, + 0.0022293750080280006, + 0.0021414590009953827, + 0.0022335420071613044, + 0.001931916005560197, + 0.0021429170010378584, + 0.0020090000034542754, + 0.002059249993180856, + 0.0020345420052763075, + 0.0020538340031635016, + 0.0020883750112261623, + 0.0019280420092400163, + 0.0019968749984400347, + 0.0020083339913981035, + 0.0020040840026922524, + 0.0019605830020736903, + 0.0018967910000355914, + 0.001972541998839006, + 0.001980707995244302, + 0.0019276659877505153 + ] + }, + { + "forward": 0.005087917001219466, + "prefill": 0.005099332993268035, + "decode_steps": [ + 0.0020274999988032505, + 0.0024514159886166453, + 0.0019042499916395172, + 0.0019827920041279867, + 0.0019228750024922192, + 0.0019504999945638701, + 0.001927207995322533, + 0.0019513339939294383, + 0.0020002499950351194, + 0.0018674580060178414, + 0.0020001249940833077, + 0.0018360410031164065, + 0.0020246250060154125, + 0.001937124994583428, + 0.0019663330022012815, + 0.0019021250045625493, + 0.0019357080018380657, + 0.002018166007474065, + 0.0018715419864747673, + 0.0019200840033590794, + 0.0019949160050600767, + 0.0020090000034542754, + 0.001873374989372678, + 0.00194191699847579, + 0.0023457079951185733, + 0.0020147500035818666, + 0.0018752920004772022, + 0.0019449579995125532, + 0.001901499999803491, + 0.0019366250053280964, + 0.0019139579962939024, + 0.001938874993356876 + ] + }, + { + "forward": 0.006465708996984176, + "prefill": 0.0054642920003971085, + "decode_steps": [ + 0.002079417012282647, + 0.002095042000291869, + 0.001891750012873672, + 0.001891583000542596, + 0.0018970420060213655, + 0.002057292003883049, + 0.0018943750037578866, + 0.0018869170016841963, + 0.0022908749961061403, + 0.0019315420067869127, + 0.001919207992614247, + 0.0018817910022335127, + 0.0019324169988976792, + 0.0019861250038957223, + 0.001953541999682784, + 0.001970458004507236, + 0.0019544169917935506, + 0.0019246250012656674, + 0.0019274579972261563, + 0.0019345420005265623, + 0.0019109999993816018, + 0.002024334011366591, + 0.0019644999993033707, + 0.00250733298889827, + 0.002005875008762814, + 0.0019468749960651621, + 0.002001083004870452, + 0.0019652499904623255, + 0.0020409999997355044, + 0.00185425000381656, + 0.002654416995937936, + 0.00205554100102745 + ] + }, + { + "forward": 0.0382995830004802, + "prefill": 0.013618583005154505, + "decode_steps": [ + 0.0036244999937480316, + 0.005523624990019016, + 0.00319408300856594, + 0.0030113329994492233, + 0.003270166998845525, + 0.0026529160095378757, + 0.002963874998386018, + 0.0024238340120064095, + 0.0024039580021053553, + 0.0021954999974695966, + 0.00240529200527817, + 0.002234625004348345, + 0.002836832994944416, + 0.0026986659941030666, + 0.002482209005393088, + 0.002208083009463735, + 0.0022035000001778826, + 0.002098249999107793, + 0.0020088750025024638, + 0.0019849999953294173, + 0.002158083996619098, + 0.0020827080006711185, + 0.002055249991826713, + 0.002165625002817251, + 0.0019568749994505197, + 0.002006082999287173, + 0.002515125001082197, + 0.0019825829949695617, + 0.0019480839982861653, + 0.0019964999955845997, + 0.0020771250128746033, + 0.0019849999953294173 + ] + }, + { + "forward": 0.00493412499781698, + "prefill": 0.0051132080116076395, + "decode_steps": [ + 0.002043333006440662, + 0.001959458997589536, + 0.0018853340006899089, + 0.002085291998810135, + 0.0019852499972330406, + 0.0019085000094491988, + 0.0019324169988976792, + 0.002023374996497296, + 0.0019015420111827552, + 0.002433333997032605, + 0.002098416007356718, + 0.0019333749951329082, + 0.0019818339933408424, + 0.001999708008952439, + 0.002009167001233436, + 0.0019427500083111227, + 0.00202379199618008, + 0.0019248750031692907, + 0.00203124999825377, + 0.0019979159987997264, + 0.0019897500023944303, + 0.0019250839977758005, + 0.0021399169927462935, + 0.0019648750021588057, + 0.0019353329989826307, + 0.0020189999922877178, + 0.0018958339933305979, + 0.0020002079982077703, + 0.0018949999939650297, + 0.002029041002970189, + 0.0019585410045692697, + 0.00205350000760518 + ] + }, + { + "forward": 0.006099958001868799, + "prefill": 0.004990667002857663, + "decode_steps": [ + 0.002147958002751693, + 0.00192320799396839, + 0.0018660420028027147, + 0.00203408399829641, + 0.001994666992686689, + 0.002043999993475154, + 0.0019385829946259037, + 0.0018807089945767075, + 0.0019802500028163195, + 0.0018689169955905527, + 0.0019629589951364323, + 0.0020558329997584224, + 0.0019248330063419417, + 0.0019941250066040084, + 0.0019294170051580295, + 0.0019653750059660524, + 0.0019233330094721168, + 0.001865290993009694, + 0.0019665829895529896, + 0.0019150000007357448, + 0.001963375005288981, + 0.0019369170040590689, + 0.0018845000013243407, + 0.0019612080068327487, + 0.0018441249994793907, + 0.001967000003787689, + 0.002059916005237028, + 0.0022227080044103786, + 0.0018998340092366561, + 0.0022211670002434403, + 0.002052041993010789, + 0.0020215840049786493 + ] + }, + { + "forward": 0.005031499997130595, + "prefill": 0.0052341249975143, + "decode_steps": [ + 0.0021559169981628656, + 0.0018796670046867803, + 0.001987582989386283, + 0.0019909170077880844, + 0.0018951249949168414, + 0.0019000839965883642, + 0.0030706669931532815, + 0.0019485410011839122, + 0.001972582991584204, + 0.0021389589965110645, + 0.0019833330006804317, + 0.0020302909979363903, + 0.0019399999873712659, + 0.0021897499973420054, + 0.0022329159983200952, + 0.0019075000018347055, + 0.0020738329913001508, + 0.0020037079957546666, + 0.0018861660064430907, + 0.0019370420050108805, + 0.002007458999287337, + 0.0019825839990517125, + 0.002070792004815303, + 0.0020391669968375936, + 0.0020867080020252615, + 0.0020114159997319803, + 0.001931958002387546, + 0.0019654169882414863, + 0.004075291988556273, + 0.0019015830039279535, + 0.0020397089974721894, + 0.002039040991803631 + ] + }, + { + "forward": 0.005327459002728574, + "prefill": 0.004917792000924237, + "decode_steps": [ + 0.002010832991800271, + 0.0019798749999608845, + 0.0019551250006770715, + 0.003597290997277014, + 0.0020995419909013435, + 0.002037374986684881, + 0.0019905829976778477, + 0.0020353340078145266, + 0.0019889169925590977, + 0.0022444589994847775, + 0.0020387500117067248, + 0.002013500008615665, + 0.001943042007042095, + 0.0019555420003598556, + 0.001920374998007901, + 0.001954666993697174, + 0.0027102909953100607, + 0.0021105000050738454, + 0.0018824170110747218, + 0.002048166992608458, + 0.0020680419984273612, + 0.0018947919888887554, + 0.002008374998695217, + 0.003227292007068172, + 0.002342792009585537, + 0.0020269160013413057, + 0.0020441669912543148, + 0.0020756669982802123, + 0.001956332998815924, + 0.0019518330082064494, + 0.0038782079936936498, + 0.001868791994638741 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 5.38208400394069, + "p95": 24.255514550168268, + "min": 4.93412499781698, + "max": 38.2995830004802, + "iqr": 1.2588439967657905, + "mean": 8.897146000526845, + "cv": 1.1638740343258163 + }, + "prefill_ms": { + "count": 10, + "median": 5.17366650456097, + "p95": 10.288752207270583, + "min": 4.917792000924237, + "max": 13.618583005154505, + "iqr": 0.43639576324494556, + "mean": 6.115108502854127, + "cv": 0.43559792827245075 + }, + "decode_ms_per_step": { + "count": 320, + "median": 2.0089375029783696, + "p95": 2.9662478984391787, + "min": 1.8360410031164065, + "max": 5.523624990019016, + "iqr": 0.21182349883019924, + "mean": 2.157545974887398, + "cv": 0.20029101747215147 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1869.5526234010295, + "p95": 2027.064061604066, + "min": 1613.0218038527125, + "max": 2028.9013521827587, + "iqr": 232.8887649453677, + "mean": 1865.3551748300583, + "cv": 0.08095992021790695 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1738.5103488337472, + "p95": 1876.94269693633, + "min": 1376.7475024290677, + "max": 1877.1723882496144, + "iqr": 202.95852335516724, + "mean": 1718.079432039079, + "cv": 0.09391850222954365 + }, + "memory": { + "peak_extra_bytes": 8650752, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "reference", + "batch": 4, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.012213375011924654, + "prefill": 0.010886666001169942, + "decode_steps": [ + 0.005685916999937035, + 0.005048791994340718, + 0.004927500005578622, + 0.004928790993290022, + 0.004657375000533648, + 0.006084332999307662, + 0.004928041002131067, + 0.004781042007380165, + 0.004685666004661471, + 0.004672541996114887, + 0.00455766700906679, + 0.004514750005910173, + 0.004483374999836087, + 0.0048513330111745745, + 0.004527416007476859, + 0.005697083004633896, + 0.00491841699113138, + 0.004954541000188328, + 0.005184749999898486, + 0.004762250013300218, + 0.004902458997094072, + 0.00525470900174696, + 0.004858874992351048, + 0.004824874995392747, + 0.004589000003761612, + 0.005028874991694465, + 0.004893125005764887, + 0.004149209009483457, + 0.0044824579963460565, + 0.004985709005268291, + 0.005007415995351039, + 0.006797374997404404 + ] + }, + { + "forward": 0.012166333006462082, + "prefill": 0.008513374996255152, + "decode_steps": [ + 0.005082332994788885, + 0.004757000002427958, + 0.004556416999548674, + 0.005124167000758462, + 0.004695291005191393, + 0.005156749990419485, + 0.004657249999581836, + 0.004626082998584025, + 0.0049620420031715184, + 0.0047374580026371405, + 0.0045876249932916835, + 0.004589333999319933, + 0.004550833007670008, + 0.00402254200889729, + 0.004427207997650839, + 0.005167124996660277, + 0.0046088749950286, + 0.00442066699906718, + 0.004402207996463403, + 0.004525167008978315, + 0.004079000005731359, + 0.0044493749883258715, + 0.004370084003312513, + 0.004304625006625429, + 0.00485316599952057, + 0.004786916993907653, + 0.004926209003315307, + 0.0044063340028515086, + 0.004722375000710599, + 0.004650999995646998, + 0.004903207998722792, + 0.004549334000330418 + ] + }, + { + "forward": 0.011307167005725205, + "prefill": 0.008364875000552274, + "decode_steps": [ + 0.004126209008973092, + 0.004420833007316105, + 0.004462541997781955, + 0.004613791999872774, + 0.004368374997284263, + 0.004406000007293187, + 0.004362042003776878, + 0.0042977499979315326, + 0.004405083993333392, + 0.005211374998907559, + 0.0050673749938141555, + 0.004853749996982515, + 0.004953290990670212, + 0.0049864589964272454, + 0.004381833990919404, + 0.004697750002378598, + 0.005362666997825727, + 0.0060126249882159755, + 0.005407874996308237, + 0.004532458013272844, + 0.00402066700917203, + 0.004529792000539601, + 0.004422124999109656, + 0.004401708007208072, + 0.004357957994216122, + 0.004476082991459407, + 0.00438774999929592, + 0.004334541998105124, + 0.004464000012376346, + 0.004395667012431659, + 0.00807491700106766, + 0.01196358299057465 + ] + }, + { + "forward": 0.011903291000635363, + "prefill": 0.008860707996063866, + "decode_steps": [ + 0.00457299999834504, + 0.004494707987760194, + 0.004374166004708968, + 0.004337334001320414, + 0.004204208991723135, + 0.004003874986665323, + 0.003935000000637956, + 0.004469040999538265, + 0.004423332997248508, + 0.0043952919950243086, + 0.0043917909933952615, + 0.00435299999662675, + 0.004656250006519258, + 0.005395582993514836, + 0.004853625010582618, + 0.004796583001734689, + 0.004505082994000986, + 0.004556791987852193, + 0.004168915998889133, + 0.004367499990621582, + 0.004197083995677531, + 0.004477958005736582, + 0.0045674170105485246, + 0.004439666998223402, + 0.004115624993573874, + 0.004097917000763118, + 0.004482041011215188, + 0.004336875004810281, + 0.004864124988671392, + 0.004641666993848048, + 0.004934458003845066, + 0.004776250003487803 + ] + }, + { + "forward": 0.011218166997423396, + "prefill": 0.00905120899551548, + "decode_steps": [ + 0.004749750005430542, + 0.0046432079980149865, + 0.004480167001020163, + 0.0044426670065149665, + 0.004470207990380004, + 0.004261624999344349, + 0.004321375003200956, + 0.004043167005875148, + 0.004504415992414579, + 0.0045595830015372485, + 0.004563917013001628, + 0.0045260829938342795, + 0.003967500000726432, + 0.004394332994706929, + 0.0044497079943539575, + 0.004391374997794628, + 0.0039078749978216365, + 0.00428141700103879, + 0.0044045410031685606, + 0.004429874999914318, + 0.0046479160082526505, + 0.004136750008910894, + 0.0044784579949919134, + 0.005075166001915932, + 0.004715374991064891, + 0.004542458002106287, + 0.0044436250027501956, + 0.004597916005877778, + 0.004648250003810972, + 0.004496041001402773, + 0.004307208000682294, + 0.004284124996047467 + ] + }, + { + "forward": 0.011473915990791284, + "prefill": 0.008415292002609931, + "decode_steps": [ + 0.00408233399502933, + 0.004342083993833512, + 0.004437540992512368, + 0.004426959014381282, + 0.004376083001261577, + 0.004117874996154569, + 0.004314165998948738, + 0.004611666998243891, + 0.004267541997251101, + 0.00444533399422653, + 0.004592749988660216, + 0.004507958001340739, + 0.004058000005898066, + 0.0044872080034110695, + 0.004404082996188663, + 0.004383750012493692, + 0.004055082987179048, + 0.004351750001660548, + 0.004447457991773263, + 0.004503167001530528, + 0.004117749995202757, + 0.004385042004287243, + 0.003922791991499253, + 0.004466208003577776, + 0.004356583012850024, + 0.004434041999047622, + 0.0043105419899802655, + 0.0039200420142151415, + 0.004146792009123601, + 0.004744083009427413, + 0.008055583995883353, + 0.005998333988827653 + ] + }, + { + "forward": 0.01196062499366235, + "prefill": 0.009003250001114793, + "decode_steps": [ + 0.004884458001470193, + 0.0048978330014506355, + 0.004638500002329238, + 0.0045639580057468265, + 0.004524541989667341, + 0.004676458993344568, + 0.004725209000753239, + 0.0040414169925497845, + 0.004444999998668209, + 0.004422708007041365, + 0.00451187499857042, + 0.004258750006556511, + 0.004576000006636605, + 0.004476583999348804, + 0.004428333006217144, + 0.004509417005465366, + 0.0039082080038497224, + 0.004618124992703088, + 0.004519041001913138, + 0.004875208003795706, + 0.004582207999192178, + 0.004527917000814341, + 0.004688165994593874, + 0.004556583997327834, + 0.004373667004983872, + 0.004445250000571832, + 0.004300875007174909, + 0.004745833997731097, + 0.004817333989194594, + 0.004649042006349191, + 0.0048502499994356185, + 0.004538167006103322 + ] + }, + { + "forward": 0.011904874991159886, + "prefill": 0.00891645799856633, + "decode_steps": [ + 0.004886499998974614, + 0.004812666011275724, + 0.004674625000916421, + 0.004619790997821838, + 0.004313249999540858, + 0.004363166997791268, + 0.004652000003261492, + 0.004441833007149398, + 0.004386083004646935, + 0.004467416991246864, + 0.004409957997268066, + 0.004412375012179837, + 0.004571708006551489, + 0.004367582994746044, + 0.004684084007749334, + 0.004528499994194135, + 0.004567083000438288, + 0.004548875003820285, + 0.004395166994072497, + 0.00440566599718295, + 0.0043342919962015, + 0.0045399999944493175, + 0.004542666996712796, + 0.004392959002871066, + 0.004414415991050191, + 0.004486207995796576, + 0.004498666996369138, + 0.004262749993358739, + 0.0042667090019676834, + 0.004439041993464343, + 0.0044304999901214615, + 0.004353042008006014 + ] + }, + { + "forward": 0.010893459009821527, + "prefill": 0.008848541998304427, + "decode_steps": [ + 0.004932499999995343, + 0.004525834010564722, + 0.004545207993942313, + 0.004423207996296696, + 0.0043960000039078295, + 0.004318708990467712, + 0.004347542009782046, + 0.004518792004091665, + 0.004441791010322049, + 0.004441249999217689, + 0.004317250000895001, + 0.004450332999113016, + 0.004393708004499786, + 0.004354917007731274, + 0.004333208009484224, + 0.004306582995923236, + 0.004366207998828031, + 0.004414957991684787, + 0.004216458008158952, + 0.004688624991104007, + 0.004674583004089072, + 0.004433583002537489, + 0.0042752500012284145, + 0.004326666006818414, + 0.004454790992895141, + 0.004385832988191396, + 0.004197665999527089, + 0.004512583007453941, + 0.004152707988396287, + 0.0049443749885540456, + 0.004766833008034155, + 0.005612624998320825 + ] + }, + { + "forward": 0.013671292006620206, + "prefill": 0.008506250000209548, + "decode_steps": [ + 0.005042040997068398, + 0.004847500007599592, + 0.004460208001546562, + 0.00507787500100676, + 0.004342583997640759, + 0.004407208994962275, + 0.004415125004015863, + 0.004476457994314842, + 0.00429633300518617, + 0.004310041986173019, + 0.0063959580002119765, + 0.004680832993471995, + 0.004429916996741667, + 0.004225583004881628, + 0.004922957989037968, + 0.005086000004666857, + 0.004786916993907653, + 0.0042762079974636436, + 0.005138250009622425, + 0.005597416995442472, + 0.009014750001369976, + 0.0052321249968372285, + 0.006636374993831851, + 0.005034166999394074, + 0.00492870899324771, + 0.0067825830046785995, + 0.004666915992856957, + 0.004645584005629644, + 0.005220417006057687, + 0.004453499990631826, + 0.004424290993483737, + 0.004518957997788675 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 11.904082995897625, + "p95": 13.015229359007206, + "min": 10.893459009821527, + "max": 13.671292006620206, + "iqr": 0.7660517512704246, + "mean": 11.871250001422595, + "cv": 0.06472250636191108 + }, + "prefill_ms": { + "count": 10, + "median": 8.854624997184146, + "p95": 10.060710348625433, + "min": 8.364875000552274, + "max": 10.886666001169942, + "iqr": 0.47352075125672854, + "mean": 8.936662499036174, + "cv": 0.08169539718719376 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.503791496972553, + "p95": 5.396197593654506, + "min": 3.9078749978216365, + "max": 11.96358299057465, + "iqr": 0.35888551428797655, + "mean": 4.652363268587578, + "cv": 0.1471603014419174 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 883.8568243379159, + "p95": 897.4689872407777, + "min": 796.1496662051967, + "max": 900.3554278530235, + "iqr": 71.71603885274317, + "mean": 861.6726076085845, + "cv": 0.048604175602627614 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 833.7356453577071, + "p95": 844.3857843936138, + "min": 755.1073037610115, + "max": 846.4640685468546, + "iqr": 62.22468697670308, + "mean": 812.7641902437731, + "cv": 0.04704026765907522 + }, + "memory": { + "peak_extra_bytes": 42205184, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "auto", + "batch": 4, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.011858334008138627 + }, + { + "forward": 0.01073312500375323 + }, + { + "forward": 0.011112124993815087 + }, + { + "forward": 0.011023332990589552 + }, + { + "forward": 0.015237291998346336 + }, + { + "forward": 0.01747583299584221 + }, + { + "forward": 0.012776457995641977 + }, + { + "forward": 0.012406791996909305 + }, + { + "forward": 0.022471583011792973 + }, + { + "forward": 0.011422583003877662 + } + ], + "forward_ms": { + "count": 10, + "median": 12.132563002523966, + "p95": 20.223495504615123, + "min": 10.73312500375323, + "max": 22.471583011792973, + "iqr": 3.4323440013395157, + "mean": 13.651745799870696, + "cv": 0.27549497572354253 + }, + "memory": { + "peak_extra_bytes": 44564480, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "sdpa", + "batch": 4, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.009330207991297357, + "prefill": 0.010832625004695728, + "decode_steps": [ + 0.003095709005719982, + 0.004129417007789016, + 0.0024600000033387914, + 0.002744082987192087, + 0.002493292005965486, + 0.0021885830065002665, + 0.0024855410010786727, + 0.0027733330061892048, + 0.003807458997471258, + 0.0023026670096442103, + 0.0025286250020144507, + 0.0023364170046988875, + 0.0023261669994099066, + 0.0036697500036098063, + 0.0025822089955909178, + 0.0023931250034365803, + 0.0024375420034630224, + 0.0024578339944127947, + 0.002312707991222851, + 0.004343250009696931, + 0.002380332996835932, + 0.0024466659961035475, + 0.0023061249958118424, + 0.0022922920034034178, + 0.00238566700136289, + 0.0022817499993834645, + 0.0023385420063277707, + 0.002252874997793697, + 0.002325624998775311, + 0.0023868749995017424, + 0.002371791997575201, + 0.002332499992917292 + ] + }, + { + "forward": 0.009037624986376613, + "prefill": 0.009360458003357053, + "decode_steps": [ + 0.002550207995227538, + 0.0023453750036424026, + 0.00234416700550355, + 0.0023818339977879077, + 0.002406041996437125, + 0.002371583992498927, + 0.002390583002124913, + 0.0023410419962601736, + 0.002275458988151513, + 0.0023676670098211616, + 0.002373875002376735, + 0.002358957994147204, + 0.0029719160083914176, + 0.002366540997172706, + 0.0022971669968683273, + 0.0023750000109430403, + 0.0024316249910043553, + 0.0023779159964760765, + 0.0029750419926131144, + 0.002405999999609776, + 0.002277124993270263, + 0.0029494159971363842, + 0.0023908750008558854, + 0.0023968750028871, + 0.0025314590020570904, + 0.002419917000224814, + 0.002230250000138767, + 0.0023594170052092522, + 0.002419374999590218, + 0.0023491669999202713, + 0.0025463329948252067, + 0.0025022089976118878 + ] + }, + { + "forward": 0.009892874993965961, + "prefill": 0.009048542007803917, + "decode_steps": [ + 0.002535500010708347, + 0.003010915999766439, + 0.0024645829980727285, + 0.0022805409971624613, + 0.00234695800463669, + 0.002332499992917292, + 0.002358084006118588, + 0.0023854169994592667, + 0.002422249992378056, + 0.0023420409997925162, + 0.0025229169987142086, + 0.0023345420049736276, + 0.0023570839985040948, + 0.002478958005667664, + 0.002430291991913691, + 0.002358832993195392, + 0.00236508299713023, + 0.0026687499921536073, + 0.002367125009186566, + 0.002411125009530224, + 0.0027487920015119016, + 0.0025642499967943877, + 0.002327584006707184, + 0.0023232080129673705, + 0.002466375008225441, + 0.0023665000044275075, + 0.0024566249921917915, + 0.0024468329938827083, + 0.00231500000518281, + 0.002360709011554718, + 0.002645042011863552, + 0.0024725840048631653 + ] + }, + { + "forward": 0.009527957998216152, + "prefill": 0.009380124989547767, + "decode_steps": [ + 0.0024341249954886734, + 0.002401249992544763, + 0.0024489580100635067, + 0.00245258399809245, + 0.002385457992204465, + 0.0024810000031720847, + 0.00243649999902118, + 0.002359584002988413, + 0.0024278339988086373, + 0.0024927920021582395, + 0.0024290000001201406, + 0.0024404579889960587, + 0.002460125004290603, + 0.0023484170087613165, + 0.0027674159937305376, + 0.0024682499933987856, + 0.0023765410005580634, + 0.0024518749996786937, + 0.002362250001169741, + 0.0023807910038158298, + 0.0024231660063378513, + 0.0024609579995740205, + 0.0024149590026354417, + 0.002375791998929344, + 0.002455083988024853, + 0.002376457996433601, + 0.0024082919990178198, + 0.0024733749887673184, + 0.002385459010838531, + 0.002336292003747076, + 0.0023600840067956597, + 0.0023428330023307353 + ] + }, + { + "forward": 0.012670916999923065, + "prefill": 0.010841625000466593, + "decode_steps": [ + 0.0038682500016875565, + 0.003399000008357689, + 0.002853790996596217, + 0.0031734579970361665, + 0.003040583003894426, + 0.0039013749919831753, + 0.0031796249968465418, + 0.0033802920079324394, + 0.003073540996410884, + 0.0035945420095231384, + 0.004973709001205862, + 0.003183124994393438, + 0.002926749992184341, + 0.0031769170018378645, + 0.004044832996441983, + 0.00331016699783504, + 0.003362707997439429, + 0.00286137500370387, + 0.004041291002067737, + 0.003857250005239621, + 0.00324062499566935, + 0.00477675000729505, + 0.004850292010814883, + 0.003720000007888302, + 0.003519625010085292, + 0.0038979590026428923, + 0.004793583997525275, + 0.003986791998613626, + 0.0030668749968754128, + 0.0028164169925730675, + 0.0030395409994525835, + 0.003828707995126024 + ] + }, + { + "forward": 0.010731166999903508, + "prefill": 0.010839249996934086, + "decode_steps": [ + 0.004237374989315867, + 0.004650083006708883, + 0.003848542008199729, + 0.004372750001493841, + 0.003677166998386383, + 0.0038791249971836805, + 0.004294250000384636, + 0.0042013749916804954, + 0.004036833008285612, + 0.004798207999556325, + 0.004475667010410689, + 0.003844542006845586, + 0.004597624996677041, + 0.0038284590118564665, + 0.0038302909961203113, + 0.0038227500044740736, + 0.004214999993564561, + 0.0038522079994436353, + 0.004694750008638948, + 0.00446083300630562, + 0.003941500006476417, + 0.004566000003251247, + 0.0037671669997507706, + 0.004671875009080395, + 0.003985417002695613, + 0.0038503329997183755, + 0.0038463329983642325, + 0.005005125000025146, + 0.004051000010804273, + 0.003764791996218264, + 0.0034265420108567923, + 0.004182749995379709 + ] + }, + { + "forward": 0.011115957997390069, + "prefill": 0.010552167004789226, + "decode_steps": [ + 0.006020082990289666, + 0.0046033750113565475, + 0.00668349998886697, + 0.006139124991022982, + 0.011462791997473687, + 0.008676333993207663, + 0.0073635000007925555, + 0.008869291996234097, + 0.005666416996973567, + 0.029863292002119124, + 0.010907542004133575, + 0.00807491599698551, + 0.00900166601059027, + 0.011033208997105248, + 0.01173354200727772, + 0.008662499996717088, + 0.014408208007807843, + 0.010431792004965246, + 0.0092381669965107, + 0.00966395800060127, + 0.00934600000618957, + 0.010682082996936515, + 0.011902458005351946, + 0.008665499990456738, + 0.007819666992872953, + 0.0072718750016065314, + 0.008389249996980652, + 0.007134332990972325, + 0.007844124993425794, + 0.007929624989628792, + 0.006869915989227593, + 0.0076573330006795 + ] + }, + { + "forward": 0.02306454199424479, + "prefill": 0.01262962500914, + "decode_steps": [ + 0.006077666999772191, + 0.006333292010822333, + 0.007102290997863747, + 0.007827916007954627, + 0.006331124997814186, + 0.006804916993132792, + 0.00556645801407285, + 0.004853999998886138, + 0.006946749999769963, + 0.004752791995997541, + 0.005211791998590343, + 0.005884917001822032, + 0.005482458000187762, + 0.006069499999284744, + 0.006890875010867603, + 0.007033000001683831, + 0.005004334001569077, + 0.004742416989756748, + 0.00467204098822549, + 0.005230583003140055, + 0.005078750007669441, + 0.0043836249969899654, + 0.004360749997431412, + 0.0042879590037046, + 0.004260332992998883, + 0.00438887499331031, + 0.004108041001018137, + 0.004562957998132333, + 0.004155874994467013, + 0.004117582997423597, + 0.005563792001339607, + 0.007092416999512352 + ] + }, + { + "forward": 0.010331166995456442, + "prefill": 0.009907625004416332, + "decode_steps": [ + 0.003480750005110167, + 0.0033795000053942204, + 0.004161250006291084, + 0.00332679100392852, + 0.0037318750109989196, + 0.003955916006816551, + 0.0032780839974293485, + 0.0033002920099534094, + 0.0031243330013239756, + 0.0033483329898444936, + 0.003240083999116905, + 0.003350166996824555, + 0.00323950000165496, + 0.0034930840047309175, + 0.003220583006623201, + 0.00314079099916853, + 0.003140291999443434, + 0.0033355000050505623, + 0.003489499998977408, + 0.0033018749963957816, + 0.0031075410079210997, + 0.0031822919991100207, + 0.003207958012353629, + 0.002850375007255934, + 0.0031431660027010366, + 0.003137332998448983, + 0.0032318339945049956, + 0.004260790999978781, + 0.00326383400533814, + 0.0031890829995973036, + 0.003165417001582682, + 0.0034568749979371205 + ] + }, + { + "forward": 0.010413333991891704, + "prefill": 0.010070375006762333, + "decode_steps": [ + 0.003161083994200453, + 0.003411958008655347, + 0.003575583003112115, + 0.0031587920093443245, + 0.0031387089984491467, + 0.003500459002680145, + 0.00321408300078474, + 0.0031016249995445833, + 0.00315108300128486, + 0.0034469590027583763, + 0.0033397499937564135, + 0.0032746250071795657, + 0.0031815839902264997, + 0.003217749996110797, + 0.004323500004829839, + 0.0031692920019850135, + 0.0033726670080795884, + 0.003253584000049159, + 0.0034326670138398185, + 0.0035657089902088046, + 0.002903665998019278, + 0.003185915993526578, + 0.00321241699566599, + 0.0031135830067796633, + 0.0031477919983444735, + 0.0032103339908644557, + 0.0032726670033298433, + 0.003123958013020456, + 0.0031637079955544323, + 0.003916417001164518, + 0.0032117499940795824, + 0.0031233329937094823 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 10.372250493674073, + "p95": 18.387410746800004, + "min": 9.037624986376613, + "max": 23.06454199424479, + "iqr": 1.4005730008648243, + "mean": 11.611575094866566, + "cv": 0.3579910690980748 + }, + "prefill_ms": { + "count": 10, + "median": 10.31127100577578, + "p95": 11.825025005236965, + "min": 9.048542007803917, + "max": 12.62962500914, + "iqr": 1.3255937556095887, + "mean": 10.346241702791303, + "cv": 0.10105789053244144 + }, + "decode_ms_per_step": { + "count": 320, + "median": 3.198520505975466, + "p95": 8.66264999640407, + "min": 2.1885830065002665, + "max": 29.863292002119124, + "iqr": 1.7846874943643343, + "mean": 3.9167437848391273, + "cv": 0.6330669377559833 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1203.0126868637008, + "p95": 1644.7075339301375, + "min": 426.64480119831114, + "max": 1649.102862435156, + "iqr": 605.2091495678885, + "mean": 1209.1908628102933, + "cv": 0.3453748026211852 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1099.7489530796533, + "p95": 1468.8895788418117, + "min": 412.1486720609903, + "max": 1471.2967535598198, + "iqr": 511.7023178983861, + "mean": 1095.5615876013476, + "cv": 0.32836454812155025 + }, + "memory": { + "peak_extra_bytes": 8650752, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "reference", + "batch": 4, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.025636708989623003, + "prefill": 0.021468874998390675, + "decode_steps": [ + 0.005714040991733782, + 0.005203416003496386, + 0.005615457994281314, + 0.005197457998292521, + 0.005025083999498747, + 0.005044999998062849, + 0.004919250000966713, + 0.004580167005769908, + 0.004966791995684616, + 0.005011083994759247, + 0.004936666999128647, + 0.005013582995161414, + 0.0049574999866308644, + 0.00489262500195764, + 0.004849500008276664, + 0.005009208005503751, + 0.0048609170044073835, + 0.005205624998779967, + 0.0049614999879850075, + 0.004877040992141701, + 0.004658207995817065, + 0.005092874998808838, + 0.005118499990203418, + 0.004928583002765663, + 0.005025291990023106, + 0.004949541995301843, + 0.0048603749892208725, + 0.004959708996466361, + 0.0050292499945499, + 0.004877250001300126, + 0.004990458008251153, + 0.005177416998776607 + ] + }, + { + "forward": 0.02533883300202433, + "prefill": 0.02221583400387317, + "decode_steps": [ + 0.005381041002692655, + 0.005034667003201321, + 0.005066333003924228, + 0.004908832997898571, + 0.00553395900351461, + 0.005229333997704089, + 0.004980249999789521, + 0.005161124994629063, + 0.00490591699781362, + 0.004538458990282379, + 0.005047874990850687, + 0.0050905410025734454, + 0.005118875007610768, + 0.004804584008525126, + 0.004960667007253505, + 0.004974458002834581, + 0.004862250003498048, + 0.004997583004296757, + 0.00497787501080893, + 0.0048683329951018095, + 0.0049590419948799536, + 0.004854708997299895, + 0.004892249999102205, + 0.005656665991409682, + 0.005218582999077626, + 0.005030207990785129, + 0.004627708010957576, + 0.005154457991011441, + 0.004979624995030463, + 0.004784624994499609, + 0.0049226249975617975, + 0.004990041998098604 + ] + }, + { + "forward": 0.02455845900112763, + "prefill": 0.022722167006577365, + "decode_steps": [ + 0.007198042003437877, + 0.005912041990086436, + 0.005777124999440275, + 0.005415167004684918, + 0.005216542005655356, + 0.005914750014198944, + 0.004860457993345335, + 0.0051077919924864545, + 0.005344500008504838, + 0.00521245900017675, + 0.005043041004682891, + 0.005379417008953169, + 0.005196750003960915, + 0.0048466250009369105, + 0.0048802910023368895, + 0.004979124991223216, + 0.004911666997941211, + 0.004878374995314516, + 0.004999625001801178, + 0.004510667000431567, + 0.004840374997002073, + 0.004988042011973448, + 0.0049041250022128224, + 0.005324540994479321, + 0.005171209006221034, + 0.005048624996561557, + 0.004839749992243014, + 0.00508608300879132, + 0.005107707998831756, + 0.004674249998060986, + 0.004774832996190526, + 0.004956707998644561 + ] + }, + { + "forward": 0.024933207998401485, + "prefill": 0.02149987498705741, + "decode_steps": [ + 0.005427249998319894, + 0.00535166599729564, + 0.005159207998076454, + 0.005069125007139519, + 0.004964291991200298, + 0.0046274170017568395, + 0.005453167002997361, + 0.0053843339992454275, + 0.0053149999876040965, + 0.005371207997086458, + 0.005726250004954636, + 0.005384625008446164, + 0.005438875014078803, + 0.005084625008748844, + 0.005085957993287593, + 0.005652000007103197, + 0.005203207998420112, + 0.005346750011085533, + 0.005259750003460795, + 0.006401207996532321, + 0.010053833000711165, + 0.018309624996618368, + 0.00763079100579489, + 0.008948500006226823, + 0.007185874987044372, + 0.008852416998706758, + 0.010155666997889057, + 0.016578082999330945, + 0.010215207992587239, + 0.009852083006990142, + 0.007156040999689139, + 0.008362290987861343 + ] + }, + { + "forward": 0.02745566600060556, + "prefill": 0.02386837499216199, + "decode_steps": [ + 0.007876542003941722, + 0.008596250001573935, + 0.007919208001112565, + 0.016552834000322036, + 0.01596179199987091, + 0.008782333999988623, + 0.007521249994169921, + 0.00924320898775477, + 0.008138958000927232, + 0.009920333992340602, + 0.008763250007177703, + 0.009477083003730513, + 0.012114208991988562, + 0.020077833003597334, + 0.017294125005719252, + 0.021998249998432584, + 0.02023570799792651, + 0.017277500010095537, + 0.01802741600840818, + 0.016224209000938572, + 0.011689959006616846, + 0.011452291990281083, + 0.013175915999454446, + 0.011947417006012984, + 0.009474749997025356, + 0.009852458999375813, + 0.01335916698735673, + 0.022639999995590188, + 0.014428959009819664, + 0.01655995799228549, + 0.017847958995844238, + 0.012857959009124897 + ] + }, + { + "forward": 0.03277874999912456, + "prefill": 0.025307417003205046, + "decode_steps": [ + 0.015220417000818998, + 0.010109625000040978, + 0.015245874994434416, + 0.013926415995229036, + 0.010935291997157037, + 0.01658112500444986, + 0.010942125009023584, + 0.011070000007748604, + 0.011958124989178032, + 0.011031625006580725, + 0.011165584000991657, + 0.010851500002900138, + 0.00951287499628961, + 0.015685500009567477, + 0.01959300000453368, + 0.01650349999545142, + 0.015192708000540733, + 0.012653833007789217, + 0.012246540994965471, + 0.009893207999994047, + 0.008649249997688457, + 0.009319124990724958, + 0.009393958011060022, + 0.014199832992744632, + 0.010275874999933876, + 0.011301874998025596, + 0.00938487499661278, + 0.012432375006028451, + 0.010747584005002864, + 0.009816417004913092, + 0.009997707995353267, + 0.009594583010766655 + ] + }, + { + "forward": 0.035440916995867155, + "prefill": 0.026900291995843872, + "decode_steps": [ + 0.01567212499503512, + 0.01580758299678564, + 0.014459958008956164, + 0.02399079198949039, + 0.019936416996642947, + 0.021220125010586344, + 0.017192083003465086, + 0.023969250003574416, + 0.01838599999609869, + 0.015672292007366195, + 0.015201583009911701, + 0.015218250002362765, + 0.020784082997124642, + 0.02166558400494978, + 0.018151582989958115, + 0.019249749995651655, + 0.01985833300568629, + 0.021782833006000146, + 0.016256791990599595, + 0.017600084000150673, + 0.02060058299684897, + 0.019201749993953854, + 0.016652374994009733, + 0.01964616699842736, + 0.019369792003999464, + 0.015723208998679183, + 0.016938291999395005, + 0.014749583002412692, + 0.025339125000755303, + 0.017516583000542596, + 0.015219583990983665, + 0.01266641600523144 + ] + }, + { + "forward": 0.055788457990274765, + "prefill": 0.024282291997224092, + "decode_steps": [ + 0.014148125002975576, + 0.009519208004348911, + 0.010476167008164339, + 0.010684084001695737, + 0.01005291698675137, + 0.010025750001659617, + 0.009640832999139093, + 0.011340000011841767, + 0.011202875000890344, + 0.013938916992628947, + 0.01278795899997931, + 0.012853124993853271, + 0.010794292000355199, + 0.009566416003508493, + 0.010341667002649046, + 0.010910040989983827, + 0.011255834004259668, + 0.011718124995240942, + 0.014264625002397224, + 0.01376962500216905, + 0.012822417003917508, + 0.00942824999219738, + 0.00963587500154972, + 0.010453708004206419, + 0.029545582990976982, + 0.013651708999532275, + 0.011963874989305623, + 0.010867125005461276, + 0.008772165994741954, + 0.008223457989515737, + 0.008301833004225045, + 0.00788408299558796 + ] + }, + { + "forward": 0.028250540999579243, + "prefill": 0.023537667002528906, + "decode_steps": [ + 0.00938770899665542, + 0.008767375009483658, + 0.007694375002756715, + 0.008048832998611033, + 0.009425291005754843, + 0.00880279199918732, + 0.00827700000081677, + 0.008023458998650312, + 0.007117750006727874, + 0.007893792004324496, + 0.007000709010753781, + 0.0070328330039046705, + 0.007026666993624531, + 0.007222875006846152, + 0.00666687500779517, + 0.00703154101211112, + 0.0069128750037634745, + 0.006641165993642062, + 0.0074067499954253435, + 0.00770258299598936, + 0.007104208998498507, + 0.007473542005755007, + 0.0074463749915594235, + 0.007687875011470169, + 0.00829770800191909, + 0.009062124998308718, + 0.009081500000320375, + 0.009825875007663853, + 0.018477875011740252, + 0.01297320899902843, + 0.009544916989398189, + 0.008792291991994716 + ] + }, + { + "forward": 0.029010874990490265, + "prefill": 0.022644083001068793, + "decode_steps": [ + 0.008259999987785704, + 0.00838970800396055, + 0.007468333991710097, + 0.006988959008594975, + 0.007573541006422602, + 0.007675000000745058, + 0.007310249988222495, + 0.006905083995661698, + 0.007514625001931563, + 0.00855737499659881, + 0.009032583009684458, + 0.007234417003928684, + 0.008930540992878377, + 0.006974584001000039, + 0.007749958007480018, + 0.00865704199532047, + 0.007462915993528441, + 0.00705791701329872, + 0.00841700000455603, + 0.007944541997858323, + 0.007274709001649171, + 0.007711875005043112, + 0.009244125001714565, + 0.007686666998779401, + 0.00753587500366848, + 0.009249291004380211, + 0.009116792003624141, + 0.00890991699998267, + 0.010000708003644831, + 0.007910915999673307, + 0.007158334003179334, + 0.009169416007352993 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 27.853103500092402, + "p95": 46.63206454279132, + "min": 24.55845900112763, + "max": 55.788457990274765, + "iqr": 6.423479248041986, + "mean": 30.9192415967118, + "cv": 0.30501772056537524 + }, + "prefill_ms": { + "count": 10, + "median": 23.129917004553135, + "p95": 26.1834982491564, + "min": 21.468874998390675, + "max": 26.900291995843872, + "iqr": 1.8559164927864913, + "mean": 23.444687698793132, + "cv": 0.0734114300365696 + }, + "decode_ms_per_step": { + "count": 320, + "median": 8.036145998630673, + "p95": 19.595658354228362, + "min": 4.510667000431567, + "max": 29.545582990976982, + "iqr": 6.192208005813882, + "mean": 9.414108481314543, + "cv": 0.5168602379319327 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 486.1246575840075, + "p95": 797.4519855277172, + "min": 218.54230408655005, + "max": 797.4612078615583, + "iqr": 384.524979305388, + "mean": 509.33656300675295, + "cv": 0.42753159382249184 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 446.94346699508964, + "p95": 702.0797435478803, + "min": 208.94573376123884, + "max": 703.3807614623508, + "iqr": 323.0975240530544, + "mean": 461.57023670954504, + "cv": 0.3977678895649848 + }, + "memory": { + "peak_extra_bytes": 1082392576, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "auto", + "batch": 4, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.031771000009030104 + }, + { + "forward": 0.034537333995103836 + }, + { + "forward": 0.028769083990482613 + }, + { + "forward": 0.02982641699782107 + }, + { + "forward": 0.034039250007481314 + }, + { + "forward": 0.028815625002607703 + }, + { + "forward": 0.029245125013403594 + }, + { + "forward": 0.03654074999212753 + }, + { + "forward": 0.02934862500114832 + }, + { + "forward": 0.05464908300200477 + } + ], + "forward_ms": { + "count": 10, + "median": 30.798708503425587, + "p95": 46.50033314755999, + "min": 28.769083990482613, + "max": 54.64908300200477, + "iqr": 5.1418129878584296, + "mean": 33.754229301121086, + "cv": 0.23231073985515235 + }, + "memory": { + "peak_extra_bytes": 38535168, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "sdpa", + "batch": 4, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.025969292008085176, + "prefill": 0.025406582994037308, + "decode_steps": [ + 0.005054791996371932, + 0.004633832999388687, + 0.0049331659974996, + 0.00520029199833516, + 0.0047708340134704486, + 0.014303583011496812, + 0.005129791999934241, + 0.005044499994255602, + 0.004715082992333919, + 0.004769749997649342, + 0.005537041986826807, + 0.006844624993391335, + 0.005073207998066209, + 0.0048073750076582655, + 0.004753084009280428, + 0.004905790992779657, + 0.004710499997599982, + 0.004804500000318512, + 0.005084375006845221, + 0.005193916993448511, + 0.005691040991223417, + 0.005265334009891376, + 0.004829208002774976, + 0.005142374997376464, + 0.004573291997076012, + 0.00960845900408458, + 0.006785874997149222, + 0.005160582993994467, + 0.0062862920021871105, + 0.005210457995417528, + 0.004944083004374988, + 0.0055225420073838905 + ] + }, + { + "forward": 0.03054433398938272, + "prefill": 0.023899667008663528, + "decode_steps": [ + 0.0063552499923389405, + 0.0049452500097686425, + 0.005203624998102896, + 0.005682500006514601, + 0.00489562499569729, + 0.005098084002383985, + 0.012037500011501834, + 0.005286374987917952, + 0.005524332998902537, + 0.004865334005444311, + 0.010586042000795715, + 0.007536749995779246, + 0.00598087499383837, + 0.0059669170004781336, + 0.005542334009078331, + 0.005540333004319109, + 0.005426750009064563, + 0.005940207993262447, + 0.006089041999075562, + 0.005624958997941576, + 0.005433083002571948, + 0.006352416996378452, + 0.00558029099192936, + 0.0057542089925846085, + 0.006026457995176315, + 0.0061813749925931916, + 0.006111332986620255, + 0.006071333002182655, + 0.005746750000980683, + 0.006678250007098541, + 0.005812125004013069, + 0.006375917000696063 + ] + }, + { + "forward": 0.027903499998501502, + "prefill": 0.020796540993615054, + "decode_steps": [ + 0.004717749994597398, + 0.004506500001298264, + 0.004292666999390349, + 0.004163292003795505, + 0.004262249989551492, + 0.004336583995609544, + 0.0068484580115182325, + 0.004102833001525141, + 0.00406887500139419, + 0.003999042004579678, + 0.0040166249964386225, + 0.004035624995594844, + 0.004204292010399513, + 0.004093040988664143, + 0.004135332987061702, + 0.0040637089987285435, + 0.004053541997564025, + 0.004046874993946403, + 0.004013833007775247, + 0.004013415993540548, + 0.004233832994941622, + 0.004129583991016261, + 0.004077499994309619, + 0.004059083003085107, + 0.014986875001341105, + 0.005320917000062764, + 0.0038421669887611642, + 0.004434166999999434, + 0.004210917002637871, + 0.003824041996267624, + 0.004120124998735264, + 0.0042041660053655505 + ] + }, + { + "forward": 0.022524583007907495, + "prefill": 0.02261562499916181, + "decode_steps": [ + 0.004674916999647394, + 0.0039263749931706116, + 0.003446875009103678, + 0.003551208006683737, + 0.003414958991925232, + 0.0034767089964589104, + 0.003501375002088025, + 0.003916417001164518, + 0.003434875005041249, + 0.0034630419977474958, + 0.003462416003458202, + 0.0034618339996086434, + 0.003421332992729731, + 0.0034872909891419113, + 0.008937083010096103, + 0.0042382079991512, + 0.0036657919990830123, + 0.003507084009470418, + 0.0034938749886350706, + 0.0038602499989792705, + 0.003618667004047893, + 0.0036683340003946796, + 0.0035195839882362634, + 0.003681665999465622, + 0.0038417500036302954, + 0.0036442080017877743, + 0.003745333000551909, + 0.003523417006363161, + 0.00355279199720826, + 0.0036678330070571974, + 0.003691207995871082, + 0.0036149579973425716 + ] + }, + { + "forward": 0.022653916996205226, + "prefill": 0.0227519170002779, + "decode_steps": [ + 0.0049405419995309785, + 0.00444570800755173, + 0.004308834002586082, + 0.004183916011243127, + 0.0040805830067256466, + 0.004269791999831796, + 0.0041602080018492416, + 0.0044472080044215545, + 0.004158333002123982, + 0.004028541996376589, + 0.003999749998911284, + 0.004077999998116866, + 0.0040381669969065115, + 0.004771124993567355, + 0.004207874997518957, + 0.004303792011342011, + 0.004268917007721029, + 0.004150958004174754, + 0.003916749992640689, + 0.004184290999546647, + 0.00420270899485331, + 0.003653542007668875, + 0.0038842920039314777, + 0.003524624989950098, + 0.0039125830080593005, + 0.0035709999938262627, + 0.003706916992086917, + 0.004006792005384341, + 0.0037589589919662103, + 0.0036541250010486692, + 0.003619084003730677, + 0.004400749996420927 + ] + }, + { + "forward": 0.021540875008213334, + "prefill": 0.0219774159922963, + "decode_steps": [ + 0.00477970800420735, + 0.009876416996121407, + 0.004330791009124368, + 0.004268292002961971, + 0.004011542012449354, + 0.004271749989129603, + 0.004063875006977469, + 0.0040686249994905666, + 0.004055582990986295, + 0.0040541250054957345, + 0.004209166989312507, + 0.004133665992412716, + 0.003784665997955017, + 0.004075584001839161, + 0.004110333989956416, + 0.004381749997264706, + 0.0034721249976428226, + 0.0043371250067139044, + 0.010281833994667977, + 0.0041558329976396635, + 0.0035084999981336296, + 0.0035648329940158874, + 0.003744208996067755, + 0.0040788750047795475, + 0.003517125005600974, + 0.0037889589875703678, + 0.004100000005564652, + 0.0038846670067869127, + 0.0035532920010155067, + 0.0036709160049213096, + 0.003608833998441696, + 0.003554665992851369 + ] + }, + { + "forward": 0.023036208003759384, + "prefill": 0.02181412500794977, + "decode_steps": [ + 0.004256790998624638, + 0.003655790991615504, + 0.0035100000095553696, + 0.004073709002113901, + 0.004145583996432833, + 0.004153332993155345, + 0.00409225000475999, + 0.004125457999180071, + 0.009413624997250736, + 0.004058124992297962, + 0.004024000008939765, + 0.004163500008871779, + 0.0039846250001573935, + 0.007172334007918835, + 0.0045047500025248155, + 0.004068333000759594, + 0.003948332989239134, + 0.004174916990450583, + 0.004394417002913542, + 0.004481374999159016, + 0.004055707991938107, + 0.004250208992743865, + 0.004209916005493142, + 0.004366666995338164, + 0.004289375006919727, + 0.004147875006310642, + 0.0040585000097053126, + 0.0043295420036884025, + 0.004194250010186806, + 0.004158291994826868, + 0.015047166991280392, + 0.010653916993760504 + ] + }, + { + "forward": 0.02721475000726059, + "prefill": 0.025235915993107483, + "decode_steps": [ + 0.004638958009309135, + 0.004544249997707084, + 0.004565833005472086, + 0.004184499994153157, + 0.00672837499587331, + 0.004272874997695908, + 0.0041764170018723235, + 0.004578083011438139, + 0.004480874995351769, + 0.004585749993566424, + 0.004499167000176385, + 0.004536624997854233, + 0.00418641600117553, + 0.004149332991801202, + 0.004337207996286452, + 0.004250957994372584, + 0.0045999160065548494, + 0.004432582994922996, + 0.004691416004789062, + 0.004637582998839207, + 0.005062834010459483, + 0.006144583996501751, + 0.004891125005087815, + 0.00470329201198183, + 0.0050167090084869415, + 0.005113542007165961, + 0.016804959013825282, + 0.004911749987513758, + 0.005462250002892688, + 0.004796916007762775, + 0.004797209010575898, + 0.004843875009100884 + ] + }, + { + "forward": 0.021465290992637165, + "prefill": 0.02294566700584255, + "decode_steps": [ + 0.013864041000488214, + 0.004475291003473103, + 0.004230624996125698, + 0.004414832990732975, + 0.0042141250014537945, + 0.004567333002341911, + 0.004754042005515657, + 0.005023416000767611, + 0.0049879159923875704, + 0.004416916010086425, + 0.004391833004774526, + 0.005511125011253171, + 0.008908500007237308, + 0.004525874988758005, + 0.00469691600301303, + 0.004517375011346303, + 0.004500166993238963, + 0.005028833998949267, + 0.005031249995226972, + 0.005050209001637995, + 0.005554041999857873, + 0.005434917009552009, + 0.005888125000637956, + 0.0066066250001313165, + 0.005229207992670126, + 0.0052867920021526515, + 0.00689491699449718, + 0.005440708002424799, + 0.004870375007158145, + 0.005421750000095926, + 0.005224125008680858, + 0.005134040999109857 + ] + }, + { + "forward": 0.03347016700718086, + "prefill": 0.030776500003412366, + "decode_steps": [ + 0.004998542004614137, + 0.004464208002900705, + 0.004260375004378147, + 0.004327916001784615, + 0.004216166998958215, + 0.004182125005172566, + 0.004505375007283874, + 0.004746041988255456, + 0.0037399169959826395, + 0.004546208001556806, + 0.003941291986848228, + 0.003781667008297518, + 0.008844915995723568, + 0.0037740419938927516, + 0.004409207991557196, + 0.0039021249976940453, + 0.003593916990212165, + 0.0037885830097366124, + 0.0037961249909130856, + 0.003898625000147149, + 0.004011750002973713, + 0.003736749989911914, + 0.00380458400468342, + 0.004538832989055663, + 0.004025917005492374, + 0.0063192499947035685, + 0.003990375000284985, + 0.004295958002330735, + 0.004306250004447065, + 0.004605042006005533, + 0.004391999988001771, + 0.0036733750021085143 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 24.50275000592228, + "p95": 32.1535421491717, + "min": 21.465290992637165, + "max": 33.47016700718086, + "iqr": 5.174395995709347, + "mean": 25.632291701913346, + "cv": 0.16070254329820874 + }, + "prefill_ms": { + "count": 10, + "median": 22.848792003060225, + "p95": 28.360037349193583, + "min": 20.796540993615054, + "max": 30.776500003412366, + "iqr": 2.764885502983816, + "mean": 23.821995699836407, + "cv": 0.11941987636700005 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.328729002736509, + "p95": 7.19055480731186, + "min": 3.414958991925232, + "max": 16.804959013825282, + "iqr": 1.0180414974456653, + "mean": 4.845342731186975, + "cv": 0.3677425816528002 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 838.7865883862389, + "p95": 1016.7583068064141, + "min": 652.2238921859386, + "max": 1048.2208793092855, + "iqr": 171.71124669886945, + "mean": 842.4748742126646, + "cv": 0.14907211430942693 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 731.6429516697578, + "p95": 861.4550774390385, + "min": 581.4183404421235, + "max": 884.421986700613, + "iqr": 128.84779246808353, + "mean": 726.9787284289284, + "cv": 0.13171882944052463 + }, + "memory": { + "peak_extra_bytes": 33816576, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "reference", + "batch": 4, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.08574987499741837, + "prefill": 0.07307250000303611, + "decode_steps": [ + 0.019463458011159673, + 0.02051233299425803, + 0.01585250000061933, + 0.02028299999074079, + 0.04096249998838175, + 0.01607954199425876, + 0.018819249991793185, + 0.01884287499706261, + 0.03525616599654313, + 0.02194824999605771, + 0.024041166994720697, + 0.0203082079970045, + 0.03738116599561181, + 0.023578874999657273, + 0.019457749993307516, + 0.02479545900132507, + 0.029291667000507005, + 0.03328583401162177, + 0.02299062500242144, + 0.01856729200517293, + 0.04686700001184363, + 0.01939691699226387, + 0.01864091599418316, + 0.01959608400647994, + 0.019765999997616746, + 0.015136500005610287, + 0.018420916007016785, + 0.015574958000797778, + 0.015406416001496837, + 0.01583366699924227, + 0.013689333994989283, + 0.025361333988257684 + ] + }, + { + "forward": 0.10824429200147279, + "prefill": 0.07742045800841879, + "decode_steps": [ + 0.027475249997223727, + 0.013328291999641806, + 0.012192833004519343, + 0.012627125004655682, + 0.012099250001483597, + 0.011807041999418288, + 0.012994292002986185, + 0.012591082995641045, + 0.01162662499700673, + 0.010995999997248873, + 0.020933958003297448, + 0.012497167001129128, + 0.011764416994992644, + 0.010626375005813316, + 0.019538499996997416, + 0.012530207997770049, + 0.01050395899801515, + 0.011442874994827434, + 0.012481249999837019, + 0.011859249992994592, + 0.01794191700173542, + 0.010312291997252032, + 0.009816582998610102, + 0.011178874992765486, + 0.01079129200661555, + 0.009714832995086908, + 0.01022374999593012, + 0.009555209006066434, + 0.011275458004092798, + 0.012385583002469502, + 0.02753445800044574, + 0.01207633299054578 + ] + }, + { + "forward": 0.09034179200534709, + "prefill": 0.07123299999511801, + "decode_steps": [ + 0.014063750000786968, + 0.012014000007184222, + 0.01002208401041571, + 0.018215207994217053, + 0.01209054198989179, + 0.01103391700598877, + 0.014918041997589171, + 0.011004624990164302, + 0.022368833000655286, + 0.012894458006485365, + 0.011368916995706968, + 0.011669415995129384, + 0.011447707991464995, + 0.014851791987894103, + 0.02013795799575746, + 0.012742666993290186, + 0.013874208001652732, + 0.013268999988213181, + 0.012603750001289882, + 0.02057787499506958, + 0.012463709004805423, + 0.010916958999587223, + 0.01026954200642649, + 0.009860959005891345, + 0.010458333999849856, + 0.009463083013542928, + 0.009940040996298194, + 0.018088458993588574, + 0.011563708001631312, + 0.0119112920074258, + 0.013070500004687347, + 0.010683083994081244 + ] + }, + { + "forward": 0.07658670899400022, + "prefill": 0.07272070899489336, + "decode_steps": [ + 0.013625583000248298, + 0.009941917000105605, + 0.00854300000355579, + 0.0170723749906756, + 0.011232792006921954, + 0.009817957994528115, + 0.009406208002474159, + 0.009887291002087295, + 0.009220874999300577, + 0.009343124998849817, + 0.01288399999612011, + 0.008529667000402696, + 0.00922075001290068, + 0.00890054200135637, + 0.011277166006038897, + 0.012038750006468035, + 0.011620459001278505, + 0.010526624988415278, + 0.012979749997612089, + 0.010132584007806145, + 0.009943958997610025, + 0.010917334002442658, + 0.010038292006356642, + 0.008385624998481944, + 0.011230000003706664, + 0.016990792006254196, + 0.011335083996527828, + 0.010083875007694587, + 0.009718457993585616, + 0.009599375000107102, + 0.009239457998774014, + 0.009106083001825027 + ] + }, + { + "forward": 0.0747650419943966, + "prefill": 0.06904687499627471, + "decode_steps": [ + 0.012698791004368104, + 0.009638958013965748, + 0.008888124997611158, + 0.009851499999058433, + 0.009294832998421043, + 0.012015458007226698, + 0.010937708997516893, + 0.009012415990582667, + 0.010821957999723963, + 0.00952691699785646, + 0.00924562499858439, + 0.01104299999133218, + 0.009442666996619664, + 0.009696875000372529, + 0.009823833010159433, + 0.012459958001272753, + 0.0140225420036586, + 0.010957500009681098, + 0.0110828750039218, + 0.009551625000312924, + 0.010150041998713277, + 0.007870208995882422, + 0.01534033298958093, + 0.01258345801034011, + 0.009733416998642497, + 0.008373042001039721, + 0.009743167000124231, + 0.008755125003517605, + 0.008383125008549541, + 0.02247733299736865, + 0.010045167000498623, + 0.012305708005442284 + ] + }, + { + "forward": 0.07541566599684302, + "prefill": 0.0708757499960484, + "decode_steps": [ + 0.0113991249963874, + 0.010935041995253414, + 0.010655582998879254, + 0.01734612499421928, + 0.011588374996790662, + 0.01928675000090152, + 0.011940124997636303, + 0.009864874999038875, + 0.009473375001107343, + 0.01011008299246896, + 0.009922333003487438, + 0.008955375000368804, + 0.012580874987179413, + 0.009116959001403302, + 0.009301208003307693, + 0.01028395799221471, + 0.009473250000155531, + 0.009457957989070565, + 0.00919962499756366, + 0.009688874997664243, + 0.009724624993395992, + 0.009436874999664724, + 0.00933791599527467, + 0.01948491600342095, + 0.011165541989612393, + 0.00993850000668317, + 0.009648457998991944, + 0.00946858299721498, + 0.009837541001616046, + 0.008694292002473958, + 0.013938165997387841, + 0.008296290994621813 + ] + }, + { + "forward": 0.08063791699532885, + "prefill": 0.07066012499853969, + "decode_steps": [ + 0.01298158300051, + 0.012295916996663436, + 0.011385207995772362, + 0.010967207999783568, + 0.009428333010873757, + 0.010355291000450961, + 0.010687791000236757, + 0.010011082995333709, + 0.009474417005549185, + 0.017285708003328182, + 0.009913041998515837, + 0.009492875004070811, + 0.010282916991855018, + 0.011072833003709093, + 0.009307707994594239, + 0.009169083990855142, + 0.010602541995467618, + 0.014749041001778096, + 0.009262916995794512, + 0.016760999991674908, + 0.011600375000853091, + 0.010818124996148981, + 0.009820791005040519, + 0.01119612499314826, + 0.009716542001115158, + 0.009506417001830414, + 0.019081207996350713, + 0.028405249991919845, + 0.01956487500865478, + 0.011393834007321857, + 0.009805166992009617, + 0.018038832989986986 + ] + }, + { + "forward": 0.08984087499266025, + "prefill": 0.07255704099952709, + "decode_steps": [ + 0.012022208000416867, + 0.010590249992674217, + 0.009290207992307842, + 0.010229999999864958, + 0.01486999999906402, + 0.01012883399380371, + 0.017170791994431056, + 0.011640500000794418, + 0.01351224999234546, + 0.011214416997972876, + 0.009899041993776336, + 0.011198583000805229, + 0.011575249998713844, + 0.01182883299770765, + 0.010802707998664118, + 0.012387666007271037, + 0.012666708003962412, + 0.012566791992867365, + 0.01240495800448116, + 0.012187542000901885, + 0.011074875001213513, + 0.030804165988229215, + 0.012236000009579584, + 0.011176749991136603, + 0.010889583005337045, + 0.014067874988541007, + 0.012251291002030484, + 0.022329833009280264, + 0.011280667007667944, + 0.015685708000091836, + 0.0094336670008488, + 0.009224209003150463 + ] + }, + { + "forward": 0.07951112500450108, + "prefill": 0.07267483299074229, + "decode_steps": [ + 0.013546125002903864, + 0.01035329099977389, + 0.010140124999452382, + 0.020331916995928623, + 0.011301334001473151, + 0.012873584011686035, + 0.016938166998443194, + 0.015184250005404465, + 0.014501541998470202, + 0.01380758399318438, + 0.0172725000011269, + 0.02784175000851974, + 0.016216249990975484, + 0.014973333993111737, + 0.014499416007311083, + 0.02273599999898579, + 0.026441541005624458, + 0.022448042000178248, + 0.02010166698892135, + 0.01580579199071508, + 0.023677792007219978, + 0.015472875005798414, + 0.013539249994209968, + 0.012680208994424902, + 0.011120124996523373, + 0.011412666994147003, + 0.01140099999611266, + 0.02023366600042209, + 0.010603165996144526, + 0.011546875000931323, + 0.012366333001409657, + 0.011333083006320521 + ] + }, + { + "forward": 0.07843250001315027, + "prefill": 0.0726477920106845, + "decode_steps": [ + 0.013435708009637892, + 0.010803416997077875, + 0.009594375005690381, + 0.016025874996557832, + 0.010953374992823228, + 0.015002250001998618, + 0.011841832994832657, + 0.01196512499882374, + 0.010534541986999102, + 0.010863208008231595, + 0.019180957999196835, + 0.010123332991497591, + 0.009951665997505188, + 0.010232875007204711, + 0.010016959000495262, + 0.009125541007961147, + 0.009140499998466112, + 0.01992308300395962, + 0.012219208001624793, + 0.012097041995730251, + 0.012011458005872555, + 0.010647750008502044, + 0.010798791990964673, + 0.011592541995923966, + 0.01083070800814312, + 0.019928665991756134, + 0.01039837500138674, + 0.010036916995886713, + 0.009349291998660192, + 0.011085707999882288, + 0.010450166009832174, + 0.012016749999020249 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 80.07452099991497, + "p95": 100.18816700321621, + "min": 74.7650419943966, + "max": 108.24429200147279, + "iqr": 11.769968245062046, + "mean": 83.95257929951185, + "cv": 0.12183290390673905 + }, + "prefill_ms": { + "count": 10, + "median": 72.6024165051058, + "p95": 75.46387690599659, + "min": 69.04687499627471, + "max": 77.42045800841879, + "iqr": 1.7441774980397895, + "mean": 72.2909082993283, + "cv": 0.030501868752876075 + }, + "decode_ms_per_step": { + "count": 320, + "median": 11.445291493146215, + "p95": 23.58382085003541, + "min": 7.870208995882422, + "max": 46.86700001184363, + "iqr": 4.835541232750984, + "mean": 13.425183715071398, + "cv": 0.4034391981691035 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 318.8740426019118, + "p95": 371.95683929958875, + "min": 176.45243399748952, + "max": 373.4067288820422, + "iqr": 56.12200527357015, + "mean": 311.9760345811802, + "cv": 0.1929288437206072 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 270.607593270277, + "p95": 308.336765083909, + "min": 160.3044865567601, + "max": 308.56744797525556, + "iqr": 42.354050729021594, + "mean": 264.203521271414, + "cv": 0.17192071151707453 + }, + "memory": { + "peak_extra_bytes": 1107558400, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "auto", + "batch": 4, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.0876528750086436 + }, + { + "forward": 0.08799658399948385 + }, + { + "forward": 0.09281083400128409 + }, + { + "forward": 0.10960974999761675 + }, + { + "forward": 0.09072720799304079 + }, + { + "forward": 0.08444900000176858 + }, + { + "forward": 0.09089420799864456 + }, + { + "forward": 0.08133916699443944 + }, + { + "forward": 0.08832533299573697 + }, + { + "forward": 0.09148441599973012 + } + ], + "forward_ms": { + "count": 10, + "median": 89.52627049438888, + "p95": 102.05023779926704, + "min": 81.33916699443944, + "max": 109.60974999761675, + "iqr": 3.598061743105063, + "mean": 90.52893749903888, + "cv": 0.0832609077015017 + }, + "memory": { + "peak_extra_bytes": 1150550016, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "sdpa", + "batch": 4, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.05159766699944157, + "prefill": 0.05264029199315701, + "decode_steps": [ + 0.006271916005061939, + 0.005113417006214149, + 0.006137750009656884, + 0.005619916002615355, + 0.0054466669971589, + 0.005391416998463683, + 0.00839658299810253, + 0.0053982090030331165, + 0.005257957993308082, + 0.005713374994229525, + 0.009989250000216998, + 0.00577562500257045, + 0.007293208996998146, + 0.005400124995503575, + 0.00497637499938719, + 0.005227457993896678, + 0.00489825000113342, + 0.005783624990726821, + 0.005949042009888217, + 0.00613899999007117, + 0.005705208008293994, + 0.005201791995204985, + 0.005423750000772998, + 0.006955333010409959, + 0.005782749998616055, + 0.005292957997880876, + 0.004815416992641985, + 0.005330458006937988, + 0.00620183399587404, + 0.005862707999767736, + 0.005792291005491279, + 0.005357083005947061 + ] + }, + { + "forward": 0.052088915996137075, + "prefill": 0.05410154200217221, + "decode_steps": [ + 0.009403084011864848, + 0.00591583400091622, + 0.0059007920062867925, + 0.006292542006121948, + 0.005795541990664788, + 0.0057015829952433705, + 0.00613020799937658, + 0.006343000000924803, + 0.005952874998911284, + 0.0058649999991757795, + 0.005847916007041931, + 0.005795374992885627, + 0.0059657499950844795, + 0.005825124986586161, + 0.006476457987446338, + 0.01002149999840185, + 0.006167499988805503, + 0.005891292006708682, + 0.0070225410017883405, + 0.006428709006286226, + 0.0058787079906323925, + 0.006306084003881551, + 0.005770959003712051, + 0.005709499993827194, + 0.005689832993084565, + 0.006187959006638266, + 0.005751457996666431, + 0.005921707997913472, + 0.007447791998856701, + 0.006248499994399026, + 0.006475375004811212, + 0.00573900000017602 + ] + }, + { + "forward": 0.0680772089981474, + "prefill": 0.07400616598897614, + "decode_steps": [ + 0.006463458004873246, + 0.006808999998611398, + 0.006218124995939434, + 0.00641416700091213, + 0.006725458995788358, + 0.005839666991960257, + 0.006282957998337224, + 0.0058686249976744875, + 0.005986042000586167, + 0.006801957992138341, + 0.00583004200598225, + 0.0062464999937219545, + 0.007559916994068772, + 0.0061634579906240106, + 0.0063655419944552705, + 0.006334707999485545, + 0.007372874999418855, + 0.005901958007598296, + 0.006174374997499399, + 0.006200374991749413, + 0.007003457998507656, + 0.006620291009312496, + 0.006078292004531249, + 0.006175374990561977, + 0.006245625001611188, + 0.005695457992260344, + 0.006203707991517149, + 0.006087582994950935, + 0.0066707089863484725, + 0.00634216598700732, + 0.005614833004074171, + 0.006173082991153933 + ] + }, + { + "forward": 0.08414245799940545, + "prefill": 0.08439308400556911, + "decode_steps": [ + 0.009136542008491233, + 0.010071459008031525, + 0.009260499995434657, + 0.009868584005744196, + 0.008492041000863537, + 0.008952375006629154, + 0.008315040991874412, + 0.008002667003893293, + 0.00930674999835901, + 0.007522333995439112, + 0.006907833012519404, + 0.0063227089995052665, + 0.006670165996183641, + 0.006636334001086652, + 0.006205624988069758, + 0.006561707996297628, + 0.006477792005171068, + 0.006332167002256028, + 0.006322500004898757, + 0.006196749993250705, + 0.00668891699751839, + 0.006294209000770934, + 0.006637165992287919, + 0.006840874993940815, + 0.006441042001824826, + 0.0069740420003654435, + 0.006829499994637445, + 0.005924832992604934, + 0.005962833005469292, + 0.005945042008534074, + 0.00602650000655558, + 0.006122666993178427 + ] + }, + { + "forward": 0.06067087499832269, + "prefill": 0.06249050000042189, + "decode_steps": [ + 0.007645249992492609, + 0.007268540997756645, + 0.008521166004356928, + 0.007062332995701581, + 0.00914224999723956, + 0.007954624990816228, + 0.008186332997865975, + 0.008839333997457288, + 0.007798417005687952, + 0.007649417006177828, + 0.007782250002492219, + 0.007006917003309354, + 0.008548833997338079, + 0.00644274998921901, + 0.007905541991931386, + 0.00579787501192186, + 0.005779416998848319, + 0.006230750004760921, + 0.006283416005317122, + 0.007214666999061592, + 0.006137666990980506, + 0.006104708008933812, + 0.006411875001504086, + 0.006132624999736436, + 0.007139791996451095, + 0.006122625010902993, + 0.0062070420099189505, + 0.005757667007856071, + 0.0059724579914473, + 0.006714249990181997, + 0.006828375000623055, + 0.0066749999969033524 + ] + }, + { + "forward": 0.06160754199663643, + "prefill": 0.06538804199954029, + "decode_steps": [ + 0.00637320798705332, + 0.0054244579951046035, + 0.005546417000005022, + 0.005402208000305109, + 0.005685249998350628, + 0.005533415998797864, + 0.005353332991944626, + 0.005141333007486537, + 0.005518582998774946, + 0.005563333004829474, + 0.005343041004380211, + 0.0056438330066157505, + 0.005302499994286336, + 0.005124707997310907, + 0.006273749997490086, + 0.005170667005586438, + 0.0052740419923793525, + 0.006194832996698096, + 0.0052385840099304914, + 0.005322165990946814, + 0.006428125008824281, + 0.0061223340017022565, + 0.008994167001219466, + 0.00794987499830313, + 0.008307374999276362, + 0.006508582999231294, + 0.0061885829927632585, + 0.006157833995530382, + 0.005866999999852851, + 0.00589154199406039, + 0.005427584008430131, + 0.00577004200022202 + ] + }, + { + "forward": 0.06952087499666959, + "prefill": 0.07036387499829289, + "decode_steps": [ + 0.010483791993465275, + 0.008251457998994738, + 0.007772832992486656, + 0.008585165996919386, + 0.006725125000230037, + 0.005859167009475641, + 0.006011290999595076, + 0.006736124996677972, + 0.006447332998504862, + 0.006634915989707224, + 0.006131709000328556, + 0.0065075829916168, + 0.006281209003645927, + 0.006045125002856366, + 0.0056646669982001185, + 0.006024500005878508, + 0.006476292008301243, + 0.005893124995054677, + 0.005845667008543387, + 0.005714957995223813, + 0.00538704200880602, + 0.005560417004744522, + 0.005340749994502403, + 0.0053569590090774, + 0.005441875007818453, + 0.005146499999682419, + 0.005368790996726602, + 0.00551749998703599, + 0.005069833001471125, + 0.005675167005392723, + 0.0051559159910539165, + 0.004997625001124106 + ] + }, + { + "forward": 0.05340566700033378, + "prefill": 0.05406424999819137, + "decode_steps": [ + 0.004903000008198433, + 0.004383916995720938, + 0.004453792003914714, + 0.004442583012860268, + 0.004472499989788048, + 0.004226832999847829, + 0.004344665998360142, + 0.004463417004444636, + 0.004294166996260174, + 0.004290791999665089, + 0.004458499999600463, + 0.004397041993797757, + 0.004238374996930361, + 0.004452749999472871, + 0.004509041988058016, + 0.004292540994356386, + 0.004251624995958991, + 0.0044525840057758614, + 0.004550958008621819, + 0.004130749992327765, + 0.0043636250047711655, + 0.0045879999961471185, + 0.0044542920077219605, + 0.0045895000075688586, + 0.004954791991622187, + 0.004854165992583148, + 0.00515524999354966, + 0.005293082998832688, + 0.005170874996110797, + 0.00595058299950324, + 0.006345792004140094, + 0.005309500003932044 + ] + }, + { + "forward": 0.06333404200267978, + "prefill": 0.06391570800042246, + "decode_steps": [ + 0.005688249992090277, + 0.004903041000943631, + 0.004607082999427803, + 0.004797458997927606, + 0.004596334008965641, + 0.004520332993706688, + 0.004736624992801808, + 0.005072708008810878, + 0.004498750000493601, + 0.004567124997265637, + 0.0047125000128289685, + 0.005059707997133955, + 0.004510291997576132, + 0.004723334001027979, + 0.004775584005983546, + 0.004429333013831638, + 0.0046191660076146945, + 0.005027375009376556, + 0.0048359579959651455, + 0.004900957996142097, + 0.005161749999388121, + 0.004964667008607648, + 0.005065541990916245, + 0.005704000010155141, + 0.005521750004845671, + 0.007961291004903615, + 0.007300500001292676, + 0.008293499995488673, + 0.008102874999167398, + 0.008139458994264714, + 0.007737708001513965, + 0.005943791009485722 + ] + }, + { + "forward": 0.07007970899576321, + "prefill": 0.07923375000245869, + "decode_steps": [ + 0.009601249999832362, + 0.00869716600573156, + 0.009865208994597197, + 0.007759166997857392, + 0.00897024999721907, + 0.008211666005081497, + 0.0088398330117343, + 0.008065833011642098, + 0.008295375009765849, + 0.0052818340045632794, + 0.006717000011121854, + 0.005879125004867092, + 0.007049042003927752, + 0.0061264579999260604, + 0.004811417005839758, + 0.0053547919960692525, + 0.00609749999421183, + 0.004894375000731088, + 0.007100208007614128, + 0.008056666993070394, + 0.005546417000005022, + 0.005943458003457636, + 0.0049009170033968985, + 0.006343208995531313, + 0.006539790992974304, + 0.005317917006323114, + 0.006860041990876198, + 0.004899666004348546, + 0.005880291995708831, + 0.006663834006758407, + 0.005126374991959892, + 0.005372457992052659 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 62.470791999658104, + "p95": 77.81422094776643, + "min": 51.59766699944157, + "max": 84.14245799940545, + "iqr": 13.937989497208036, + "mean": 63.4524959983537, + "cv": 0.15864275099774255 + }, + "prefill_ms": { + "count": 10, + "median": 64.65187499998137, + "p95": 82.07138370416942, + "min": 52.64029199315701, + "max": 84.39308400556911, + "iqr": 16.896811739570694, + "mean": 66.0597208989202, + "cv": 0.16552654866389374 + }, + "decode_ms_per_step": { + "count": 320, + "median": 5.957854002190288, + "p95": 8.839358948171139, + "min": 4.130749992327765, + "max": 10.483791993465275, + "iqr": 1.3013647439947817, + "mean": 6.173416934097986, + "cv": 0.20372427017016617 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 640.0886692970951, + "p95": 800.6035330730153, + "min": 555.9186809582911, + "max": 858.8339242566703, + "iqr": 74.87363339679064, + "mean": 657.4680441168126, + "cv": 0.13416123394173712 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 488.412372264083, + "p95": 587.2283069317897, + "min": 406.81079194485545, + "max": 630.2204223177197, + "iqr": 75.01890907259144, + "mean": 492.45064227658423, + "cv": 0.1293159710155265 + }, + "memory": { + "peak_extra_bytes": 1107558400, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "reference", + "batch": 4, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.3060350000014296, + "prefill": 0.25807908301067073, + "decode_steps": [ + 0.06937925000966061, + 0.03330320800887421, + 0.023987916007172316, + 0.027397625002777204, + 0.019780707996687852, + 0.024375958993914537, + 0.015253709003445692, + 0.020249209002940916, + 0.020107000003918074, + 0.020561375000397675, + 0.02581458300119266, + 0.027952208009082824, + 0.014872041007038206, + 0.024796207988401875, + 0.020805500011192635, + 0.021671374997822568, + 0.025240333008696325, + 0.023428707994753495, + 0.020188333000987768, + 0.018671582991373725, + 0.02215508298831992, + 0.02165145800972823, + 0.024206957998103462, + 0.02192920800007414, + 0.018314000000827946, + 0.016053415994974785, + 0.01372737500059884, + 0.013460667003528215, + 0.013606291002361104, + 0.020651416009059176, + 0.016780459001893178, + 0.0138673750043381 + ] + }, + { + "forward": 0.25180745900433976, + "prefill": 0.26433470800111536, + "decode_steps": [ + 0.02138441600254737, + 0.020380000001750886, + 0.020395832994836383, + 0.016003000011551194, + 0.01761979098955635, + 0.013105540987453423, + 0.013457625012961216, + 0.015462916999240406, + 0.014172750001307577, + 0.014652042009402066, + 0.020471124997129664, + 0.013966458005597815, + 0.012570833001518622, + 0.01259487500647083, + 0.014277666006819345, + 0.01428720899275504, + 0.01673945799120702, + 0.015859291990636848, + 0.013252625009045005, + 0.014359709006384946, + 0.012371249991701916, + 0.012524167002993636, + 0.012927749994560145, + 0.01504029099305626, + 0.012950082993484102, + 0.012911625002743676, + 0.014483999999356456, + 0.01388437500281725, + 0.013806875009322539, + 0.014986333000706509, + 0.013458874993375503, + 0.014269291001255624 + ] + }, + { + "forward": 0.26761762499518227, + "prefill": 0.28476824999961536, + "decode_steps": [ + 0.02349754099850543, + 0.02239545900374651, + 0.017242041998542845, + 0.014244874997530133, + 0.014340209003421478, + 0.014901624992489815, + 0.014484749990515411, + 0.023519415990449488, + 0.014363084002980031, + 0.016453707998152822, + 0.014743916995939799, + 0.013740624999627471, + 0.013069124994217418, + 0.016640624991850927, + 0.013913541988586076, + 0.02032120799412951, + 0.015501000001677312, + 0.01761725000687875, + 0.018100542001775466, + 0.01600450000842102, + 0.02080795900837984, + 0.014860750001389533, + 0.013640791992656887, + 0.014737042001797818, + 0.013651208006194793, + 0.016298500006087124, + 0.015480374990147538, + 0.02400083300017286, + 0.01828324999951292, + 0.014861957999528386, + 0.01570233299571555, + 0.012831166997784749 + ] + }, + { + "forward": 0.3077175000071293, + "prefill": 0.2907365419960115, + "decode_steps": [ + 0.022184415996889584, + 0.018861624994315207, + 0.014449915994191542, + 0.012074500002199784, + 0.012529457992059179, + 0.01242774999991525, + 0.012419291990227066, + 0.011584667008719407, + 0.011488040996482596, + 0.01412012500804849, + 0.010528790997341275, + 0.011313458002405241, + 0.012601790993358009, + 0.010173624992603436, + 0.010379332990851253, + 0.01145783301035408, + 0.011220333006349392, + 0.013117416005115956, + 0.012311583006521687, + 0.012480207995395176, + 0.0140705829981016, + 0.014350416997331195, + 0.019377292002900504, + 0.019908874994143844, + 0.018783541992888786, + 0.018401332999928854, + 0.018643875009729527, + 0.022472582990303636, + 0.01907108300656546, + 0.015238249994581565, + 0.015544875001069158, + 0.01696387500851415 + ] + }, + { + "forward": 0.2877560420020018, + "prefill": 0.3080432909919182, + "decode_steps": [ + 0.018007041988312267, + 0.015441042007296346, + 0.017114499991293997, + 0.01541325000289362, + 0.01566616700438317, + 0.01496837500599213, + 0.015079332995810546, + 0.014878332993248478, + 0.013025915992329828, + 0.013122333999490365, + 0.016289374994812533, + 0.015418166003655642, + 0.013344042003154755, + 0.013353832997381687, + 0.012298542002099566, + 0.01276383399090264, + 0.01248699999996461, + 0.014294915992650203, + 0.011735249994671904, + 0.011610417001065798, + 0.012280208989977837, + 0.01598412499879487, + 0.015308457994251512, + 0.012203708000015467, + 0.013040416990406811, + 0.01257166700088419, + 0.016876959009096026, + 0.014087458010180853, + 0.012778500007698312, + 0.012947374998475425, + 0.01113012499990873, + 0.010176750001846813 + ] + }, + { + "forward": 0.28359012500732206, + "prefill": 0.34152741701109335, + "decode_steps": [ + 0.020328124999650754, + 0.016618708992609754, + 0.014785542007302865, + 0.013110875006532297, + 0.014701166001032107, + 0.01425020799797494, + 0.01441808301024139, + 0.01314179199107457, + 0.012038000000757165, + 0.01199875000747852, + 0.01507175000733696, + 0.01501187500252854, + 0.01675041600537952, + 0.01525250000122469, + 0.016943583003012463, + 0.015277167010935955, + 0.012266874997294508, + 0.012415332996170036, + 0.010825582998222671, + 0.010697542005800642, + 0.010559540998656303, + 0.01148641700274311, + 0.010880457994062454, + 0.01259616699826438, + 0.010116583987837657, + 0.01090299998759292, + 0.010693999996874481, + 0.01023366700974293, + 0.010497999988729134, + 0.010182750003878027, + 0.010109499999089167, + 0.010075124999275431 + ] + }, + { + "forward": 0.27454633300658315, + "prefill": 0.26788829200086184, + "decode_steps": [ + 0.02247370799886994, + 0.01780841599975247, + 0.013090041000396013, + 0.010979540995322168, + 0.010467625004821457, + 0.00976916700892616, + 0.008853792009176686, + 0.009092749998671934, + 0.008507916005328298, + 0.00875341601204127, + 0.009426083997823298, + 0.008804833996691741, + 0.008792541004368104, + 0.007995209001819603, + 0.009101250005187467, + 0.009735958999954164, + 0.008770458996878006, + 0.00883962499210611, + 0.008731124995392747, + 0.008603375012171455, + 0.0084857909969287, + 0.008745540995732881, + 0.008267917000921443, + 0.008095415993011557, + 0.008216709000407718, + 0.008563417010009289, + 0.008237166999606416, + 0.008476584000163712, + 0.008514624991221353, + 0.00940337500651367, + 0.008890625002095476, + 0.008296833009808324 + ] + }, + { + "forward": 0.2584264170000097, + "prefill": 0.25045195799611975, + "decode_steps": [ + 0.01985933299874887, + 0.015152374995523132, + 0.01103254199551884, + 0.009534791999612935, + 0.008713332994375378, + 0.008158083001035266, + 0.008572709004511125, + 0.008025208007893525, + 0.008075624995399266, + 0.008800708994385786, + 0.008549667007173412, + 0.00794687500456348, + 0.008031457997276448, + 0.007540666003478691, + 0.007620874996064231, + 0.007975959000759758, + 0.009650167005020194, + 0.007781000007526018, + 0.009106999990763143, + 0.008233959000790492, + 0.009812833988689817, + 0.00849720899714157, + 0.00894670799607411, + 0.008200374999432825, + 0.008847208999213763, + 0.007899415999418125, + 0.007970958002260886, + 0.007869249995565042, + 0.007984583004144952, + 0.007777207996696234, + 0.008118583995383233, + 0.008547707999241538 + ] + }, + { + "forward": 0.253325625002617, + "prefill": 0.25890579201222863, + "decode_steps": [ + 0.011177874999702908, + 0.008901958004571497, + 0.008469666994642466, + 0.008314999999129213, + 0.00818333298957441, + 0.008298625005409122, + 0.008165332998032682, + 0.00804204199812375, + 0.008042666988330893, + 0.007902875004219823, + 0.007970000006025657, + 0.008300166999106295, + 0.00845024999580346, + 0.007875374998548068, + 0.007978541005286388, + 0.008269124999060296, + 0.0085975830006646, + 0.008519708993844688, + 0.008497832997818477, + 0.007948665996082127, + 0.008272459002910182, + 0.008218834002036601, + 0.00815120899642352, + 0.009293208000599407, + 0.008153750008204952, + 0.008310917008202523, + 0.007869750013924204, + 0.008356416001333855, + 0.007891000001109205, + 0.00848674999724608, + 0.007699249996221624, + 0.007972457999130711 + ] + }, + { + "forward": 0.3095427080115769, + "prefill": 0.3177390000055311, + "decode_steps": [ + 0.027037499996367842, + 0.019195917004253715, + 0.01507229199341964, + 0.015479957990464754, + 0.015391417007776909, + 0.011746417003450915, + 0.0118071670003701, + 0.012001041002804413, + 0.014777292002690956, + 0.01255595899419859, + 0.009444667011848651, + 0.009215040990966372, + 0.011061750003136694, + 0.008658833990921266, + 0.010474082999280654, + 0.008689833004609682, + 0.009471124998526648, + 0.008831875005853362, + 0.00856687499617692, + 0.009273792005842552, + 0.008088040995062329, + 0.011007749999407679, + 0.007736707993899472, + 0.008817125009954907, + 0.007507749993237667, + 0.009896041010506451, + 0.007756917009828612, + 0.007720250010606833, + 0.007603584002936259, + 0.009395250002853572, + 0.009284542000386864, + 0.007767749993945472 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 279.0682290069526, + "p95": 308.7213644095755, + "min": 251.80745900433976, + "max": 309.5427080115769, + "iqr": 40.741041502769804, + "mean": 280.03648340381915, + "cv": 0.08026252549699867 + }, + "prefill_ms": { + "count": 10, + "median": 276.3282710002386, + "p95": 330.82262935859035, + "min": 250.45195799611975, + "max": 341.52741701109335, + "iqr": 43.4535827334912, + "mean": 284.2474333025166, + "cv": 0.1058317430404973 + }, + "decode_ms_per_step": { + "count": 320, + "median": 12.58327100367751, + "p95": 22.47263924073195, + "min": 7.507749993237667, + "max": 69.37925000966061, + "iqr": 6.609832747926703, + "mean": 13.347342031283915, + "cv": 0.424933217883902 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 298.0953267383271, + "p95": 463.5054728617285, + "min": 179.2113347807614, + "max": 480.15132268791467, + "iqr": 131.39201004290226, + "mean": 325.57392549649546, + "cv": 0.2952670565214779 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 171.0514315808095, + "p95": 240.77842690650158, + "min": 131.64395468176224, + "max": 243.582914210457, + "iqr": 45.279056367929684, + "mean": 186.17353682441754, + "cv": 0.19660313520419465 + }, + "memory": { + "peak_extra_bytes": 1107558400, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "auto", + "batch": 8, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.005496167010278441 + }, + { + "forward": 0.005507708003278822 + }, + { + "forward": 0.005811166003695689 + }, + { + "forward": 0.005269916000543162 + }, + { + "forward": 0.005720334011130035 + }, + { + "forward": 0.01189579199126456 + }, + { + "forward": 0.00601229201129172 + }, + { + "forward": 0.006296167004620656 + }, + { + "forward": 0.00750799999514129 + }, + { + "forward": 0.007342500000959262 + } + ], + "forward_ms": { + "count": 10, + "median": 5.911729007493705, + "p95": 9.921285593009083, + "min": 5.269916000543162, + "max": 11.89579199126456, + "iqr": 1.5200522466329858, + "mean": 6.686004203220364, + "cv": 0.2964684707639844 + }, + "memory": { + "peak_extra_bytes": 9830400, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "sdpa", + "batch": 8, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.004889250005362555, + "prefill": 0.008977124991361052, + "decode_steps": [ + 0.0022310829954221845, + 0.0020135829981882125, + 0.0020381670037750155, + 0.002046833003987558, + 0.0045070839987602085, + 0.0019863750057993457, + 0.0019640000100480393, + 0.001946250005858019, + 0.0019582079985411838, + 0.001991541008464992, + 0.0019863750057993457, + 0.0021467499900609255, + 0.002040291001321748, + 0.002078874997096136, + 0.0020173749944660813, + 0.0019813749968307093, + 0.0019943330116802827, + 0.0053623749990947545, + 0.001957166998181492, + 0.002280166998389177, + 0.0021892919903621078, + 0.0020202909945510328, + 0.00206754200917203, + 0.004536958993412554, + 0.001997917002881877, + 0.0020838749915128574, + 0.002146250000805594, + 0.001985250011784956, + 0.0020105000003241003, + 0.005089375001261942, + 0.002003000001423061, + 0.0020926669967593625 + ] + }, + { + "forward": 0.004905499998130836, + "prefill": 0.004878874999121763, + "decode_steps": [ + 0.0025355830002808943, + 0.0019870830001309514, + 0.002214249994722195, + 0.0020669579971581697, + 0.0018814579962054268, + 0.001847999999881722, + 0.0021334170014597476, + 0.002101875012158416, + 0.0021615829900838435, + 0.0019283329893369228, + 0.0019498339970596135, + 0.0019469170074444264, + 0.004581540997605771, + 0.001976416999241337, + 0.0020374169980641454, + 0.0020473339973250404, + 0.0019412499968893826, + 0.001986999996006489, + 0.002771500003291294, + 0.0022004579950589687, + 0.0020937500084983185, + 0.002369709007325582, + 0.001970083001651801, + 0.0020656669948948547, + 0.0019783329917117953, + 0.0037564170052064583, + 0.002077833007206209, + 0.0022284159931587055, + 0.0021907500049564987, + 0.0019693330104928464, + 0.002075041993521154, + 0.005073665990494192 + ] + }, + { + "forward": 0.007602415993460454, + "prefill": 0.004807875011465512, + "decode_steps": [ + 0.0022191669995663688, + 0.0019824170012725517, + 0.004845292001846246, + 0.0022283750004135072, + 0.0020028750004712492, + 0.0019172500033164397, + 0.002038459002505988, + 0.0019232909980928525, + 0.002580291999038309, + 0.0023827080003684387, + 0.0019565419934224337, + 0.0020553340000333264, + 0.0019196250068489462, + 0.00202095799613744, + 0.001890250001451932, + 0.0019570000004023314, + 0.004093916999408975, + 0.0019911249983124435, + 0.0019998750067315996, + 0.0019422910118009895, + 0.001979583001229912, + 0.0019432920089457184, + 0.00209329099743627, + 0.0020420000073499978, + 0.0019607909925980493, + 0.001957833010237664, + 0.0019718749972525984, + 0.00193254099576734, + 0.0019878330058418214, + 0.004814957996131852, + 0.00251987500814721, + 0.002027208000072278 + ] + }, + { + "forward": 0.007524792003096081, + "prefill": 0.004794000007677823, + "decode_steps": [ + 0.002434834008454345, + 0.0019288750045234337, + 0.0020349170081317425, + 0.0038045410037739202, + 0.001992707999306731, + 0.00204320800548885, + 0.002319582999916747, + 0.002142208002624102, + 0.0020204159955028445, + 0.004770458006532863, + 0.001993875004700385, + 0.0019627919973572716, + 0.0020850839937338606, + 0.0020095410000067204, + 0.0019512500002747402, + 0.0025318749976577237, + 0.002212916995631531, + 0.00200191700423602, + 0.0020765410008607432, + 0.0020327910024207085, + 0.002107207998051308, + 0.0019831670069834217, + 0.00195162498857826, + 0.004319542000303045, + 0.0020695420098491013, + 0.0021411670022644103, + 0.002028915987466462, + 0.001931834005517885, + 0.0020236670097801834, + 0.004502208001213148, + 0.0020287090010242537, + 0.002147999999579042 + ] + }, + { + "forward": 0.004875707993051037, + "prefill": 0.007339624993619509, + "decode_steps": [ + 0.0026932089967885986, + 0.002054457989288494, + 0.0027551660023164004, + 0.0021155419963179156, + 0.004484083008719608, + 0.0020452500029932708, + 0.002011000004131347, + 0.0020372090075397864, + 0.0020322079944889992, + 0.0020185000030323863, + 0.0019842500041704625, + 0.0037597079935949296, + 0.001998625011765398, + 0.0021342910040402785, + 0.0020297080045565963, + 0.002122082994901575, + 0.00209087498660665, + 0.004820250003831461, + 0.0020680419984273612, + 0.002105833002133295, + 0.0019650419999379665, + 0.00215537499752827, + 0.0021160840115044266, + 0.0020886669954052195, + 0.0022192500036908314, + 0.0022486670059151947, + 0.0021362919942475855, + 0.002156500006094575, + 0.0021612499986076728, + 0.002133458008756861, + 0.002255333005450666, + 0.00211200000194367 + ] + }, + { + "forward": 0.007030499997199513, + "prefill": 0.009744291004608385, + "decode_steps": [ + 0.002926375003880821, + 0.0023609579948242754, + 0.0024660830094944686, + 0.007885875005740672, + 0.003029416999197565, + 0.0024272920127259567, + 0.0023040000087348744, + 0.005672500003129244, + 0.0022863749909447506, + 0.0024618750030640513, + 0.0023485830024583265, + 0.002128542007994838, + 0.002442042008624412, + 0.0025720419944263995, + 0.0021726250124629587, + 0.0022794170072302222, + 0.0022310839995043352, + 0.002114874994731508, + 0.0021196659945417196, + 0.004422458994667977, + 0.002133916990715079, + 0.0024009159969864413, + 0.002262790992972441, + 0.002127957995980978, + 0.006104124986450188, + 0.0022485830122604966, + 0.00284754199674353, + 0.002554209000663832, + 0.002745375008089468, + 0.005680624992237426, + 0.0022442500048782676, + 0.002855541999451816 + ] + }, + { + "forward": 0.008476417002384551, + "prefill": 0.005143166010384448, + "decode_steps": [ + 0.003606833008234389, + 0.002469832994393073, + 0.002288416988449171, + 0.0029310420068213716, + 0.002151374996174127, + 0.0022179170046001673, + 0.006414416988263838, + 0.002218583002104424, + 0.0029612500075018033, + 0.0023217080015456304, + 0.0021248749981168658, + 0.005654834007145837, + 0.002263749993289821, + 0.0023416249896399677, + 0.002263375004986301, + 0.0022120839857961982, + 0.0024335409980267286, + 0.002784041003906168, + 0.002809249999700114, + 0.00301970899454318, + 0.0026470420125406235, + 0.006167874991660938, + 0.002939749989309348, + 0.0023291669931495562, + 0.0024850419868016616, + 0.002309457995579578, + 0.0035887089907191694, + 0.00244679200113751, + 0.0025802499876590446, + 0.002758207992883399, + 0.0028634579939534888, + 0.003001582997967489 + ] + }, + { + "forward": 0.005310249995090999, + "prefill": 0.0053023750078864396, + "decode_steps": [ + 0.002821375004714355, + 0.002688834007130936, + 0.0022737920080544427, + 0.002367125009186566, + 0.0023197920090751722, + 0.0023072909971233457, + 0.0022352919913828373, + 0.0020850000000791624, + 0.002239167006337084, + 0.0021945839980617166, + 0.0022356669942382723, + 0.002199167007347569, + 0.002265499992063269, + 0.002241834008600563, + 0.0021982909966027364, + 0.0021556249994318932, + 0.0022455840080510825, + 0.002575916994828731, + 0.0023491250030929223, + 0.002343915999517776, + 0.002160542004276067, + 0.00218508300895337, + 0.0022916250018170103, + 0.0023552500060759485, + 0.003055625013075769, + 0.002346624998608604, + 0.0028138749912614003, + 0.0027708330017048866, + 0.002868874988052994, + 0.0022948750120121986, + 0.002419917000224814, + 0.002240166999399662 + ] + }, + { + "forward": 0.005286624989821576, + "prefill": 0.005558916993322782, + "decode_steps": [ + 0.002472292006132193, + 0.0027557499997783452, + 0.0027083330060122535, + 0.0027814999921247363, + 0.0027518340066308156, + 0.0026855420001083985, + 0.0026837089972104877, + 0.0026239999860990793, + 0.0026811250136233866, + 0.002371915994444862, + 0.0028265000000828877, + 0.0037833749956917018, + 0.002771374987787567, + 0.0026625419995980337, + 0.0024217500031227246, + 0.002738708004471846, + 0.0027144579944433644, + 0.004149500004132278, + 0.00254379199759569, + 0.002563542002462782, + 0.0024178750027203932, + 0.002682749996893108, + 0.0028593750030267984, + 0.0028890410030726343, + 0.0031674170022597536, + 0.002698166004847735, + 0.0027430409973021597, + 0.0026049580046674237, + 0.0027569999947445467, + 0.002622750005684793, + 0.0027132909890497103, + 0.0023964590072864667 + ] + }, + { + "forward": 0.005091334009193815, + "prefill": 0.005279417004203424, + "decode_steps": [ + 0.0028418750007404014, + 0.0027870409976458177, + 0.0025866659998428077, + 0.002639124999404885, + 0.0023307079973164946, + 0.002757042006123811, + 0.004094917007023469, + 0.002582167013315484, + 0.002705582999624312, + 0.002719833006267436, + 0.0023511250037699938, + 0.002751374995568767, + 0.003158959007123485, + 0.0028221249958733097, + 0.002196083005401306, + 0.002846416988177225, + 0.002774167005554773, + 0.002795124994008802, + 0.0024638339964440092, + 0.0027280830108793452, + 0.0027142500039190054, + 0.0022834580013295636, + 0.0027712500013876706, + 0.0026114579959539697, + 0.0028123750089434907, + 0.0026790419942699373, + 0.002209875005064532, + 0.002780458002234809, + 0.0027228330000070855, + 0.003491208000923507, + 0.002771749990643002, + 0.002834582992363721 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 5.298437492456287, + "p95": 8.083116548368706, + "min": 4.875707993051037, + "max": 8.47641700238455, + "iqr": 2.4492605007253587, + "mean": 6.099279198679142, + "cv": 0.22858865958523295 + }, + "prefill_ms": { + "count": 10, + "median": 5.290896006044932, + "p95": 9.399066298647085, + "min": 4.794000007677823, + "max": 9.744291004608385, + "iqr": 1.9495002416078933, + "mean": 6.182566602365114, + "cv": 0.2973229322986866 + }, + "decode_ms_per_step": { + "count": 320, + "median": 2.2590619992115535, + "p95": 4.508577748492826, + "min": 1.847999999881722, + "max": 7.885875005740672, + "iqr": 0.6685929984087124, + "mean": 2.5476216129391105, + "cv": 0.3315857116811595 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 3348.705515737497, + "p95": 3477.7383811492864, + "min": 2698.7731264600015, + "max": 3498.3132848527125, + "iqr": 496.76657137471193, + "mean": 3170.0191487476077, + "cv": 0.09981280233781724 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 3045.5869209425973, + "p95": 3263.1988764314387, + "min": 2447.367052502272, + "max": 3282.6405558470206, + "iqr": 438.6208371951984, + "mean": 2946.597355059287, + "cv": 0.09984952668401677 + }, + "memory": { + "peak_extra_bytes": 8650752, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "reference", + "batch": 8, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.013739291011006571, + "prefill": 0.007055124995531514, + "decode_steps": [ + 0.007884958991780877, + 0.005007499989005737, + 0.005111125006806105, + 0.007499208993976936, + 0.004783333002706058, + 0.004535125000984408, + 0.005644250006298535, + 0.005100958005641587, + 0.004703167011030018, + 0.006209582992596552, + 0.004531792001216672, + 0.004438082993146963, + 0.0069313330022851005, + 0.0046896249987185, + 0.0044283750030444935, + 0.006313999998383224, + 0.004411041998537257, + 0.004369708010926843, + 0.006743250007275492, + 0.004608917006407864, + 0.004364041989902034, + 0.005248834000667557, + 0.005133792001288384, + 0.0050214579969178885, + 0.005412083002738655, + 0.0050262919976376, + 0.004846665993682109, + 0.006358375001582317, + 0.004296458006137982, + 0.004679333011154085, + 0.004484624994802289, + 0.0045449590106727555 + ] + }, + { + "forward": 0.010637500003213063, + "prefill": 0.009490334006841294, + "decode_steps": [ + 0.005145458009792492, + 0.007520500003010966, + 0.004439832991920412, + 0.0044529579899972305, + 0.006830500002251938, + 0.004427957988809794, + 0.004538916997262277, + 0.007191124997916631, + 0.005156875005923212, + 0.004578416992444545, + 0.007882833000621758, + 0.004634207987692207, + 0.0042360000079497695, + 0.005800874991109595, + 0.005341000010957941, + 0.0054444160050479695, + 0.004888166004093364, + 0.004604083005688153, + 0.005065958990599029, + 0.008948750008130446, + 0.005039249997935258, + 0.004562959002214484, + 0.004323124987422489, + 0.004201875010039657, + 0.0043304999999236315, + 0.007267375010997057, + 0.005323749996023253, + 0.005146041003172286, + 0.0070324999978765845, + 0.004906707996269688, + 0.004604250003467314, + 0.005666375000146218 + ] + }, + { + "forward": 0.012461542006349191, + "prefill": 0.007278375007444993, + "decode_steps": [ + 0.007031832996290177, + 0.004736292001325637, + 0.004640250001102686, + 0.006497958995169029, + 0.004752209002617747, + 0.004518250003457069, + 0.006768792009097524, + 0.004661291008233093, + 0.004523791998508386, + 0.007041541000944562, + 0.004618749997462146, + 0.004306707996875048, + 0.007474042009562254, + 0.005067042002337985, + 0.004608624993124977, + 0.0073918749985750765, + 0.004856208004639484, + 0.00453074999677483, + 0.004515749998972751, + 0.004521374998148531, + 0.004606041999068111, + 0.007355459005339071, + 0.00504216700210236, + 0.004502875002799556, + 0.005921125004533678, + 0.004792333013028838, + 0.004688790999352932, + 0.007218499988084659, + 0.004662499995902181, + 0.0046750420005992055, + 0.00619770900812, + 0.005360333991120569 + ] + }, + { + "forward": 0.013322583006811328, + "prefill": 0.0074547080002957955, + "decode_steps": [ + 0.005832749986439012, + 0.00498329199035652, + 0.005105499993078411, + 0.005028249986935407, + 0.0049742080009309575, + 0.004938833997584879, + 0.004805584001587704, + 0.0049698749935487285, + 0.004988708009477705, + 0.004940166007145308, + 0.005988166012684815, + 0.004909708004561253, + 0.004761333009810187, + 0.004679040997871198, + 0.004619291998096742, + 0.004815666005015373, + 0.004523374998825602, + 0.004885791990091093, + 0.005231582996202633, + 0.005020999989937991, + 0.00553320899780374, + 0.005171584009076469, + 0.0048594169929856434, + 0.0050477080076234415, + 0.004247792006935924, + 0.0047986249992391095, + 0.004627667003660463, + 0.004573458005324937, + 0.004636541998479515, + 0.004701540994574316, + 0.004779832990607247, + 0.004706207997514866 + ] + }, + { + "forward": 0.013127750004059635, + "prefill": 0.007405083000776358, + "decode_steps": [ + 0.005275000003166497, + 0.00449037499492988, + 0.004677582997828722, + 0.005565625004237518, + 0.004886166992946528, + 0.00472141700447537, + 0.004835166007978842, + 0.006024667003657669, + 0.004731250010081567, + 0.004719457996543497, + 0.007183791996794753, + 0.004615500001818873, + 0.004552332989987917, + 0.007433917009620927, + 0.004640250001102686, + 0.004585540998959914, + 0.005921499992837198, + 0.004872457997407764, + 0.004781874988111667, + 0.0070129169907886535, + 0.004844084003707394, + 0.004527917000814341, + 0.0058501250023255125, + 0.005276459007291123, + 0.004935959004797041, + 0.0059806249919347465, + 0.00493879200075753, + 0.005006749997846782, + 0.006543958006659523, + 0.0055257920030271634, + 0.005014540991396643, + 0.006521832998259924 + ] + }, + { + "forward": 0.013686749996850267, + "prefill": 0.007673416999750771, + "decode_steps": [ + 0.0055927909997990355, + 0.005645958008244634, + 0.005029209001804702, + 0.005511375013156794, + 0.005424541988759302, + 0.004927458998281509, + 0.004976040989276953, + 0.0049269580049440265, + 0.005294250004226342, + 0.005525207991013303, + 0.0055638750054640695, + 0.0069174160016700625, + 0.012919291999423876, + 0.0075892079912591726, + 0.005415791005361825, + 0.007831415990949608, + 0.004951958995661698, + 0.004897584003629163, + 0.007934374996693805, + 0.005099333007819951, + 0.00483225000789389, + 0.00634237501071766, + 0.004973041010089219, + 0.004837957996642217, + 0.006530292012030259, + 0.004831250000279397, + 0.00456662499345839, + 0.00509112500003539, + 0.00565695900877472, + 0.0058389170008013025, + 0.007629042011103593, + 0.0052384169975994155 + ] + }, + { + "forward": 0.01161537499865517, + "prefill": 0.008271874990896322, + "decode_steps": [ + 0.006812124993302859, + 0.005835000003571622, + 0.005269625005894341, + 0.005802500003483146, + 0.005398666995461099, + 0.005464957997901365, + 0.005114458006573841, + 0.005536209006095305, + 0.0055290419986704364, + 0.005248667002888396, + 0.005325291000190191, + 0.004937999998219311, + 0.005154415994184092, + 0.005355083005269989, + 0.005130499994265847, + 0.0049230000004172325, + 0.004980750003596768, + 0.005116166998050176, + 0.005292583009577356, + 0.005162708999705501, + 0.005000624994863756, + 0.005010666995076463, + 0.005395290994783863, + 0.005101499991724268, + 0.005123124996316619, + 0.004966458000126295, + 0.005240375001449138, + 0.004981332996976562, + 0.005122750008013099, + 0.004589457996189594, + 0.005127500000526197, + 0.005821042010211386 + ] + }, + { + "forward": 0.011918917007278651, + "prefill": 0.007631083004525863, + "decode_steps": [ + 0.00664929099730216, + 0.005572041001869366, + 0.005841999998665415, + 0.005639041992253624, + 0.005230875001871027, + 0.005402916998718865, + 0.005539042002055794, + 0.005259042009129189, + 0.005377166991820559, + 0.005653833999531344, + 0.005430625009466894, + 0.005310375010594726, + 0.005329124993295409, + 0.005885416001547128, + 0.0051861660031136125, + 0.00588087500364054, + 0.005422916990937665, + 0.005643166005029343, + 0.004784625009051524, + 0.0051408750005066395, + 0.00530533300479874, + 0.005387875004089437, + 0.005362124997191131, + 0.005361250005080365, + 0.006001999994623475, + 0.005352916996343993, + 0.005427458003396168, + 0.005473291006637737, + 0.0054732080025132746, + 0.005260791993350722, + 0.004865166003582999, + 0.00558458300656639 + ] + }, + { + "forward": 0.010160874997382052, + "prefill": 0.007734958999208175, + "decode_steps": [ + 0.0060140830028103665, + 0.0057350830029463395, + 0.005652499996358529, + 0.0057234999985666946, + 0.005850999994436279, + 0.005880167009308934, + 0.005473957993672229, + 0.005474332996527664, + 0.005407084012404084, + 0.005289709006319754, + 0.005360915994970128, + 0.005258958000922576, + 0.00527808400511276, + 0.005278541008010507, + 0.0052872910018777475, + 0.005197582999244332, + 0.0054241249890765175, + 0.005663249990902841, + 0.005750875003286637, + 0.0054939580004429445, + 0.005289332999382168, + 0.005416083993623033, + 0.005359957998734899, + 0.00532533299701754, + 0.005143958987900987, + 0.005339249997632578, + 0.005351833999156952, + 0.0053144169942243025, + 0.005290250002872199, + 0.005257791999611072, + 0.0051663750055013224, + 0.005274583003483713 + ] + }, + { + "forward": 0.012264042001334019, + "prefill": 0.007589707995066419, + "decode_steps": [ + 0.00763287500012666, + 0.005549167006392963, + 0.006047916991519742, + 0.005338999995728955, + 0.005827999993925914, + 0.005788374997791834, + 0.006238208996364847, + 0.006511375002446584, + 0.005722333007724956, + 0.005182291992241517, + 0.005488750000949949, + 0.005008459003875032, + 0.005414833998656832, + 0.005263915998511948, + 0.007254083000589162, + 0.005616957991151139, + 0.005545709005673416, + 0.005482666994794272, + 0.005503458989551291, + 0.0054231669928412884, + 0.005311791013809852, + 0.005361499992432073, + 0.005524125008378178, + 0.005416167012299411, + 0.005503749998752028, + 0.004827625001780689, + 0.0052399170090211555, + 0.005433416998130269, + 0.004868874995736405, + 0.005309750005835667, + 0.005546000000322238, + 0.005034000001614913 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 12.362792003841605, + "p95": 13.715647554636234, + "min": 10.160874997382052, + "max": 13.739291011006571, + "iqr": 1.5826142553123645, + "mean": 12.293462503293995, + "cv": 0.10038344483641642 + }, + "prefill_ms": { + "count": 10, + "median": 7.610395499796141, + "p95": 8.942027449666055, + "min": 7.055124995531514, + "max": 9.490334006841294, + "iqr": 0.3020842486876063, + "mean": 7.75846670003375, + "cv": 0.08865311999632651 + }, + "decode_ms_per_step": { + "count": 320, + "median": 5.258375000266824, + "p95": 7.254747601109557, + "min": 4.201875010039657, + "max": 12.919291999423876, + "iqr": 0.7291560068551917, + "mean": 5.39578606590112, + "cv": 0.16649670910880565 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1481.4228026306955, + "p95": 1581.24268655201, + "min": 1359.2273863563623, + "max": 1623.4825902274001, + "iqr": 47.66482419051795, + "mean": 1485.4699776940329, + "cv": 0.04610627571246915 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1417.8590237415688, + "p95": 1513.0924167998253, + "min": 1306.01778067255, + "max": 1550.1959350158797, + "iqr": 44.22426363472073, + "mean": 1421.4442422593654, + "cv": 0.044887828976011235 + }, + "memory": { + "peak_extra_bytes": 8650752, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "auto", + "batch": 8, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.009922333003487438 + }, + { + "forward": 0.010006750002503395 + }, + { + "forward": 0.010756165997008793 + }, + { + "forward": 0.009940874995663762 + }, + { + "forward": 0.009934500005329028 + }, + { + "forward": 0.009755291001056321 + }, + { + "forward": 0.010391750009148382 + }, + { + "forward": 0.009870458001387306 + }, + { + "forward": 0.014481667007203214 + }, + { + "forward": 0.00985816601314582 + } + ], + "forward_ms": { + "count": 10, + "median": 9.937687500496395, + "p95": 12.805191552615721, + "min": 9.755291001056321, + "max": 14.481667007203214, + "iqr": 0.41207325557479635, + "mean": 10.491795603593346, + "cv": 0.13666125781667773 + }, + "memory": { + "peak_extra_bytes": 44564480, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "sdpa", + "batch": 8, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.008563417010009289, + "prefill": 0.01004300000204239, + "decode_steps": [ + 0.0024726249976083636, + 0.0021735839982284233, + 0.002343792002648115, + 0.0023006669944152236, + 0.0023937080113682896, + 0.00238679199537728, + 0.002174042005208321, + 0.0022221249964786693, + 0.0023453330068150535, + 0.002304666006239131, + 0.002325291992747225, + 0.0023616660037077963, + 0.002231582999229431, + 0.0022927080135559663, + 0.002257125001051463, + 0.0023567919997731224, + 0.002223624993348494, + 0.0026491659955354407, + 0.0022230419999687, + 0.002373415991314687, + 0.0022703750000800937, + 0.002372500006458722, + 0.0023626659967703745, + 0.0024352500040549785, + 0.002558583000791259, + 0.002262457986944355, + 0.002333249998628162, + 0.002373291994445026, + 0.0023579580010846257, + 0.002404166996711865, + 0.0022764169989386573, + 0.0022672909981338307 + ] + }, + { + "forward": 0.009246791989426129, + "prefill": 0.008575124986236915, + "decode_steps": [ + 0.003154583988361992, + 0.0025310409982921556, + 0.002657832999830134, + 0.002258249995065853, + 0.0026551249902695417, + 0.0023107499873731285, + 0.003191124997101724, + 0.0022581249941140413, + 0.0022279169934336096, + 0.002226415992481634, + 0.0026734169950941578, + 0.002294417005032301, + 0.002364083004067652, + 0.002334750010049902, + 0.0023745000071357936, + 0.002247374999569729, + 0.0025061670021386817, + 0.002278041996760294, + 0.002246999996714294, + 0.0030983339966041967, + 0.002264709008159116, + 0.0022003750054864213, + 0.002438417010125704, + 0.0023766249942127615, + 0.0023037909995764494, + 0.002543583992519416, + 0.002434249996440485, + 0.0023513339983765036, + 0.0030852919880999252, + 0.002348875001189299, + 0.002351750008529052, + 0.002333457989152521 + ] + }, + { + "forward": 0.00881962499988731, + "prefill": 0.008677541991346516, + "decode_steps": [ + 0.0024560000019846484, + 0.0023362080100923777, + 0.002920291997725144, + 0.0023511250037699938, + 0.002339083992410451, + 0.002274458995088935, + 0.0022960000060265884, + 0.0022622090036747977, + 0.0022213329939404503, + 0.002348499998333864, + 0.002362041996093467, + 0.0025154169998131692, + 0.0023533750063506886, + 0.002322042011655867, + 0.0022281660058069974, + 0.0024074160028249025, + 0.0022990830038907006, + 0.002256041989312507, + 0.002387582993833348, + 0.0023049160081427544, + 0.0022882920020492747, + 0.0022666660079266876, + 0.0023872500023571774, + 0.0023089580063242465, + 0.0024619169998914003, + 0.0023339170002145693, + 0.0023864579998189583, + 0.002322790998732671, + 0.002339333004783839, + 0.0023722919868305326, + 0.0024165409995475784, + 0.0024739170039538294 + ] + }, + { + "forward": 0.008685124994372018, + "prefill": 0.009310708002885804, + "decode_steps": [ + 0.002330792005523108, + 0.0023055829951772466, + 0.002213625004515052, + 0.002328499991563149, + 0.002291374999913387, + 0.0031112089927773923, + 0.002387000000453554, + 0.0023717090080026537, + 0.0023200409923447296, + 0.002281291992403567, + 0.002308833005372435, + 0.0023967080051079392, + 0.002436041002511047, + 0.0022587499988730997, + 0.0022659999958705157, + 0.0023056249920045957, + 0.0022962500079302117, + 0.0023697500000707805, + 0.002357291989028454, + 0.0021130840032128617, + 0.00237574998755008, + 0.002257208005175926, + 0.002320458006579429, + 0.002315416990313679, + 0.002707540988922119, + 0.0024789589951978996, + 0.002413417008938268, + 0.0024147079966496676, + 0.0023423749953508377, + 0.00230395799735561, + 0.002403374994173646, + 0.002412916990579106 + ] + }, + { + "forward": 0.008660666993819177, + "prefill": 0.009046041988767684, + "decode_steps": [ + 0.002407209001830779, + 0.002347916000871919, + 0.0023382920044241473, + 0.0022846249921713024, + 0.0024104580079438165, + 0.0023025419941404834, + 0.002308666007593274, + 0.002374834002694115, + 0.00231520899978932, + 0.0022638749942416325, + 0.0023103749990696087, + 0.0022974159946898, + 0.002703334001125768, + 0.0023116250085877255, + 0.002432499997667037, + 0.0023224160104291514, + 0.0023452910099877045, + 0.0023542089911643416, + 0.0023314160062000155, + 0.00239666600828059, + 0.0023446249979315326, + 0.0023060410021571442, + 0.002317916994797997, + 0.002406041996437125, + 0.0023665420012548566, + 0.002541583002312109, + 0.002349499991396442, + 0.0023136249947128817, + 0.002290541000547819, + 0.0022524589876411483, + 0.0025247500016121194, + 0.0023566669988213107 + ] + }, + { + "forward": 0.008790874999249354, + "prefill": 0.008585874995333143, + "decode_steps": [ + 0.002455916997860186, + 0.0022297499963315204, + 0.0024799999955575913, + 0.002251209007226862, + 0.0023280419991351664, + 0.0022189169976627454, + 0.002213084007962607, + 0.0025807500060182065, + 0.0029412500007310882, + 0.0024855840019881725, + 0.002338042002520524, + 0.002857792002032511, + 0.0023165829916251823, + 0.0021988330117892474, + 0.002395375006017275, + 0.0030632499983767048, + 0.0022850000095786527, + 0.0026921660028165206, + 0.003075040993280709, + 0.0023331670090556145, + 0.002355042000999674, + 0.0029850840073777363, + 0.0023572499922011048, + 0.0028265000000828877, + 0.0025518339971313253, + 0.0023754580033710226, + 0.002248167002107948, + 0.002648292007506825, + 0.0022796249977545813, + 0.00227537500904873, + 0.00237800000468269, + 0.002248125005280599 + ] + }, + { + "forward": 0.008717707998584956, + "prefill": 0.009993915999075398, + "decode_steps": [ + 0.002374957999563776, + 0.0023459999938495457, + 0.0023122080019675195, + 0.002420875011011958, + 0.0023745000071357936, + 0.0022349170030793175, + 0.002338042002520524, + 0.0023775410081725568, + 0.002618416998302564, + 0.0023650420043850318, + 0.002350416994886473, + 0.0023689999943599105, + 0.002609625007607974, + 0.0025188750005327165, + 0.0023647920024814084, + 0.0025432920083403587, + 0.002358833997277543, + 0.002331540992599912, + 0.0025600419903639704, + 0.002363209001487121, + 0.002359167003305629, + 0.002341084007639438, + 0.0024346660065930337, + 0.002441709002596326, + 0.002386375010246411, + 0.002381916012382135, + 0.0023830409918446094, + 0.0023816249886294827, + 0.002473667002050206, + 0.0024657500034663826, + 0.0023076670040609315, + 0.002365041000302881 + ] + }, + { + "forward": 0.009061042001121677, + "prefill": 0.008657542013679631, + "decode_steps": [ + 0.0024754999903962016, + 0.0025459589960519224, + 0.0022893749992363155, + 0.0023995829978957772, + 0.0022555830073542893, + 0.0023842919908929616, + 0.0022722499998053536, + 0.0022796659904997796, + 0.0029363749927142635, + 0.002288749994477257, + 0.0022215829958440736, + 0.0022757090046070516, + 0.0022167919960338622, + 0.002286417002324015, + 0.002449708990752697, + 0.003207208006642759, + 0.0024373330088565126, + 0.0022411250101868063, + 0.002522208000300452, + 0.0023634579993085936, + 0.003084958007093519, + 0.002396250012679957, + 0.0023029590083751827, + 0.002308499999344349, + 0.002186666999477893, + 0.0022985419927863404, + 0.002314958008355461, + 0.003341542003909126, + 0.0023147499887272716, + 0.002259041997604072, + 0.002416916991933249, + 0.002272292011184618 + ] + }, + { + "forward": 0.010064999994938262, + "prefill": 0.008928458002628759, + "decode_steps": [ + 0.0024338330113096163, + 0.0023592920042574406, + 0.0023130420013330877, + 0.00226791700697504, + 0.0028135000029578805, + 0.002521667003748007, + 0.002372875009314157, + 0.0028137079934822395, + 0.0024308749998454005, + 0.002296709004440345, + 0.0023444169928552583, + 0.0023968749883351848, + 0.0023379999911412597, + 0.0023667079949518666, + 0.002350249997107312, + 0.0022739999985788018, + 0.0023661250015720725, + 0.0026955829962389544, + 0.0027994159900117666, + 0.0025173750036628917, + 0.0024563750048400834, + 0.002319750012247823, + 0.0023360829945886508, + 0.0029452910093823448, + 0.002297083003213629, + 0.002278999992995523, + 0.0022402500035241246, + 0.0024147079966496676, + 0.002249833007226698, + 0.002259917004266754, + 0.0024499589926563203, + 0.0024771250027697533 + ] + }, + { + "forward": 0.009604375009075738, + "prefill": 0.0099891660065623, + "decode_steps": [ + 0.0029000830108998343, + 0.003310959000373259, + 0.0033024589938577265, + 0.003311874999781139, + 0.0027753750036936253, + 0.0026975420041708276, + 0.0026594170049065724, + 0.002636667006299831, + 0.002730915992287919, + 0.0026410830032546073, + 0.0023027499992167577, + 0.002368917004787363, + 0.002318792001460679, + 0.002483208998455666, + 0.002626375004183501, + 0.0024710410070838407, + 0.0023040840023895726, + 0.0023045829875627533, + 0.0024465419992338866, + 0.002386207997915335, + 0.0023809580015949905, + 0.0024042919976636767, + 0.0025378749996889383, + 0.0023112909984774888, + 0.002439916002913378, + 0.0024278329947264865, + 0.002420874996460043, + 0.0023539999965578318, + 0.0024857919925125316, + 0.0023849999997764826, + 0.0024147910007741302, + 0.0024962080060504377 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 8.805249999568332, + "p95": 9.857718751300126, + "min": 8.563417010009289, + "max": 10.064999994938262, + "iqr": 0.5070837469247635, + "mean": 9.021462599048391, + "cv": 0.05380320500998412 + }, + "prefill_ms": { + "count": 10, + "median": 8.987249995698221, + "p95": 10.020912200707244, + "min": 8.575124986236915, + "max": 10.04300000204239, + "iqr": 1.1570094975468237, + "mean": 9.180737398855854, + "cv": 0.06699334563614819 + }, + "decode_ms_per_step": { + "count": 320, + "median": 2.3583959991810843, + "p95": 2.9366187431151047, + "min": 2.1130840032128617, + "max": 3.341542003909126, + "iqr": 0.1374574967485387, + "mean": 2.4197994970108994, + "cv": 0.08822618261391849 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 3310.7954075666853, + "p95": 3410.4658655363455, + "min": 3120.516238360544, + "max": 3427.6859232048087, + "iqr": 129.14801239843155, + "mean": 3308.4763793116754, + "cv": 0.028238759428321478 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 2955.6165453867434, + "p95": 3032.779205910232, + "min": 2781.796108348502, + "max": 3037.596208745679, + "iqr": 86.76372442352022, + "mean": 2957.4341111683857, + "cv": 0.025645463428714644 + }, + "memory": { + "peak_extra_bytes": 8650752, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "reference", + "batch": 8, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.018758416990749538, + "prefill": 0.015486208998481743, + "decode_steps": [ + 0.004973417002474889, + 0.005484458000864834, + 0.004971624992322177, + 0.005028249986935407, + 0.004993624999769963, + 0.004732457993668504, + 0.004788207996170968, + 0.004915666999295354, + 0.004767625010572374, + 0.004739833006169647, + 0.005227874993579462, + 0.01603737501136493, + 0.009419207999599166, + 0.006299291999312118, + 0.006029708994901739, + 0.005427708005299792, + 0.006188208004459739, + 0.005080041999462992, + 0.004958959005307406, + 0.0056617920054122806, + 0.004674625000916421, + 0.004918667007586919, + 0.005534083000384271, + 0.005271875008475035, + 0.004746791993966326, + 0.00525954199838452, + 0.006142958009149879, + 0.004801165996468626, + 0.0052616250031860545, + 0.005699834000552073, + 0.004781542011187412, + 0.005488082999363542 + ] + }, + { + "forward": 0.019014250006875955, + "prefill": 0.015631875008693896, + "decode_steps": [ + 0.005749999996623956, + 0.005673833002219908, + 0.005433250000351109, + 0.004764208002598025, + 0.0051484580035321414, + 0.004936625002301298, + 0.0051696669979719445, + 0.0045813329925294966, + 0.004736957998829894, + 0.0047000419872347265, + 0.004622124994057231, + 0.00474608299555257, + 0.0044211670028744265, + 0.004910499992547557, + 0.004466416998184286, + 0.004849916003877297, + 0.004738165996968746, + 0.0043266249995213, + 0.00474608399963472, + 0.0046428329951595515, + 0.004953499999828637, + 0.005138459004228935, + 0.004981167003279552, + 0.004954582997015677, + 0.004611666998243891, + 0.0054728329996578395, + 0.004642083993530832, + 0.004804083000635728, + 0.004858166008489206, + 0.004849666991503909, + 0.0049130830011563376, + 0.004810083002666943 + ] + }, + { + "forward": 0.017431166997994296, + "prefill": 0.015079791992320679, + "decode_steps": [ + 0.006899207990500145, + 0.004933792006340809, + 0.005106917000375688, + 0.004961082988302223, + 0.004473833003430627, + 0.0047258749982574955, + 0.0046256659989012405, + 0.005115583000588231, + 0.004659750004066154, + 0.004802208000910468, + 0.005175708007300273, + 0.00493562500923872, + 0.004502499999944121, + 0.004775667010108009, + 0.004243083007168025, + 0.004804124997463077, + 0.004687459004344419, + 0.004826584001420997, + 0.005472958000609651, + 0.0060899590025655925, + 0.005461499997181818, + 0.0048889169993344694, + 0.005434999999124557, + 0.005711332996725105, + 0.004897999999229796, + 0.005906749996938743, + 0.005393624989665113, + 0.00547020799422171, + 0.006233833002625033, + 0.005863625003257766, + 0.0052063750044908375, + 0.004924624998238869 + ] + }, + { + "forward": 0.01877062499988824, + "prefill": 0.015292374999262393, + "decode_steps": [ + 0.005180000007385388, + 0.004862209010752849, + 0.004438415999175049, + 0.0047763749898876995, + 0.004710125009296462, + 0.004804166994290426, + 0.004916749996482395, + 0.005302082994603552, + 0.004903125009150244, + 0.004835332991206087, + 0.005027624996728264, + 0.004864084010478109, + 0.004850083001656458, + 0.004857791005633771, + 0.004717790987342596, + 0.004442040997673757, + 0.004497249989071861, + 0.004982790997019038, + 0.005559458993957378, + 0.00471691599523183, + 0.00497824999911245, + 0.0047818750026635826, + 0.004967666987795383, + 0.004778457994689234, + 0.004689125009463169, + 0.004829208992305212, + 0.005164708010852337, + 0.004999541997676715, + 0.004452374996617436, + 0.00492666699574329, + 0.004815708991372958, + 0.0047655419912189245 + ] + }, + { + "forward": 0.01801083399914205, + "prefill": 0.015216832995065488, + "decode_steps": [ + 0.005071083010989241, + 0.0048230839893221855, + 0.0048207079962594435, + 0.004770833998918533, + 0.004871457989793271, + 0.0046274589985841885, + 0.004848875003517605, + 0.0046866670018061996, + 0.004616792008164339, + 0.004734707996249199, + 0.004796334003913216, + 0.004727708001155406, + 0.004964416002621874, + 0.004778125003213063, + 0.004973125003743917, + 0.004779041992151178, + 0.004562000001897104, + 0.004707208994659595, + 0.0047903749946272, + 0.004702624995843507, + 0.005438624997623265, + 0.005075709006632678, + 0.005144333001226187, + 0.005083708994789049, + 0.004836292006075382, + 0.00479224999435246, + 0.004889417003141716, + 0.004724541999166831, + 0.0047270000068238005, + 0.005081958006485365, + 0.004788791993632913, + 0.0047153750056168064 + ] + }, + { + "forward": 0.017814457998611033, + "prefill": 0.015240916996845044, + "decode_steps": [ + 0.00534329199581407, + 0.00513933299225755, + 0.004932791998726316, + 0.004897249993518926, + 0.004833250000956468, + 0.004646375004085712, + 0.004710666005848907, + 0.004635833000065759, + 0.004365999993751757, + 0.004707750005763955, + 0.004743500001495704, + 0.004658667006879114, + 0.004608792005456053, + 0.004345791996456683, + 0.005128457996761426, + 0.005049915998824872, + 0.005571917004999705, + 0.005083166004624218, + 0.00479579099919647, + 0.004633874996216036, + 0.00501300000178162, + 0.004974332987330854, + 0.004884249996393919, + 0.004646125002182089, + 0.004790499995579012, + 0.00496158299210947, + 0.004364624997833744, + 0.004842791997361928, + 0.004360167004051618, + 0.004531625003437512, + 0.004758750001201406, + 0.0047197920066537336 + ] + }, + { + "forward": 0.019710291002411395, + "prefill": 0.016236666997428983, + "decode_steps": [ + 0.005975499996566214, + 0.005000333010684699, + 0.0050348340009804815, + 0.0048070000048028305, + 0.004862957997829653, + 0.005091707993415184, + 0.004925332992570475, + 0.004919667000649497, + 0.004825000010896474, + 0.004797083995072171, + 0.004427749998285435, + 0.004728999992948957, + 0.0047302500024670735, + 0.004773083011968993, + 0.00478291699255351, + 0.004616792008164339, + 0.004751665997901, + 0.004799291986273602, + 0.004707457992481068, + 0.004712624999228865, + 0.005384792006225325, + 0.005014374997699633, + 0.004662542007281445, + 0.005229542002780363, + 0.004940166007145308, + 0.005032250002841465, + 0.005373874999349937, + 0.004947292007273063, + 0.005067459002020769, + 0.004938375001074746, + 0.0048777910124044865, + 0.0048163749888772145 + ] + }, + { + "forward": 0.018692249999730848, + "prefill": 0.01686462499492336, + "decode_steps": [ + 0.0055010420037433505, + 0.005348750011762604, + 0.0051961249992018566, + 0.004555125007755123, + 0.005148790995008312, + 0.005416999993030913, + 0.0046571249986300245, + 0.004872624995186925, + 0.005152417012141086, + 0.005929165999987163, + 0.004886750000878237, + 0.004965666012139991, + 0.0051660000026458874, + 0.004852166995988227, + 0.005311874992912635, + 0.004846750001888722, + 0.004948750007315539, + 0.005269708010018803, + 0.005428208998637274, + 0.004924332999507897, + 0.0050369169912301, + 0.004636792000383139, + 0.005272333000903018, + 0.005306417006067932, + 0.005106500000692904, + 0.005113457998959348, + 0.005042666991357692, + 0.004932832991471514, + 0.005135792001965456, + 0.005264415987767279, + 0.005270832989481278, + 0.005608250008663163 + ] + }, + { + "forward": 0.018180666011176072, + "prefill": 0.015621875005308539, + "decode_steps": [ + 0.0053587500005960464, + 0.004956958000548184, + 0.0046467909996863455, + 0.00474720899364911, + 0.004703917002188973, + 0.00434737499745097, + 0.004913625001790933, + 0.005269665998639539, + 0.005462874993099831, + 0.005515208991710097, + 0.005141208006534725, + 0.005228666996117681, + 0.005143457994563505, + 0.005012832989450544, + 0.005177083003218286, + 0.004845082992687821, + 0.004910709001705982, + 0.004917583006317727, + 0.004946250002831221, + 0.0049591670103836805, + 0.005018832991481759, + 0.004866333998506889, + 0.004808375000720844, + 0.004785583994816989, + 0.004528334000497125, + 0.005632624990539625, + 0.0053816250001545995, + 0.01614591700490564, + 0.009546833010972477, + 0.005838166005560197, + 0.005669209000188857, + 0.005415542007540353 + ] + }, + { + "forward": 0.019607915994129144, + "prefill": 0.015951833003782667, + "decode_steps": [ + 0.005229208996752277, + 0.005083833995740861, + 0.0048873750056372955, + 0.004709583008661866, + 0.004795334010850638, + 0.004811916995095089, + 0.004694542003562674, + 0.004451999993762001, + 0.004442624995135702, + 0.0051785830000881106, + 0.004888666997430846, + 0.004800624999916181, + 0.0047814999998081475, + 0.004852167010540143, + 0.004771874999278225, + 0.0047066669940249994, + 0.004778249989612959, + 0.004725582999526523, + 0.0047110839950619265, + 0.004671165996114723, + 0.004737208000733517, + 0.0052791670022998005, + 0.004910000003292225, + 0.004895167003269307, + 0.0050820420001400635, + 0.004892041994025931, + 0.004811499995412305, + 0.004649042006349191, + 0.0046462500031339005, + 0.00515224999981001, + 0.005062249998445623, + 0.005156582992640324 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 18.725333495240193, + "p95": 19.664222248684382, + "min": 17.431166997994296, + "max": 19.710291002411395, + "iqr": 0.9000517529784702, + "mean": 18.599087400070857, + "cv": 0.040031399494766216 + }, + "prefill_ms": { + "count": 10, + "median": 15.55404200189514, + "p95": 16.58204389605089, + "min": 15.07979199232068, + "max": 16.86462499492336, + "iqr": 0.6180620075610932, + "mean": 15.66230009921128, + "cv": 0.03526126695902652 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.896208498394117, + "p95": 5.754408297070769, + "min": 4.243083007168025, + "max": 16.14591700490564, + "iqr": 0.3999270047643222, + "mean": 5.070761993601991, + "cv": 0.1997741593645705 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1631.071136470397, + "p95": 1659.6523571374557, + "min": 1404.2314735416765, + "max": 1665.799529216606, + "iqr": 96.46326085575583, + "mean": 1582.9720500925014, + "cv": 0.05920413453357035 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1480.7622833824548, + "p95": 1510.506538337616, + "min": 1294.2867576473056, + "max": 1515.502459592699, + "iqr": 82.7799879058598, + "mean": 1442.865902810147, + "cv": 0.054554231765910115 + }, + "memory": { + "peak_extra_bytes": 1082392576, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "auto", + "batch": 8, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.02394216600805521 + }, + { + "forward": 0.02529866699478589 + }, + { + "forward": 0.025362374988617375 + }, + { + "forward": 0.028508292001788504 + }, + { + "forward": 0.026933875007671304 + }, + { + "forward": 0.029393125005299225 + }, + { + "forward": 0.027537832997040823 + }, + { + "forward": 0.022333291999530047 + }, + { + "forward": 0.02136087500548456 + }, + { + "forward": 0.02299283299362287 + } + ], + "forward_ms": { + "count": 10, + "median": 25.330520991701633, + "p95": 28.9949501537194, + "min": 21.36087500548456, + "max": 29.393125005299225, + "iqr": 4.156677252467489, + "mean": 25.36633330018958, + "cv": 0.10689272284256134 + }, + "memory": { + "peak_extra_bytes": 38535168, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "sdpa", + "batch": 8, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.02004962498904206, + "prefill": 0.021500957998796366, + "decode_steps": [ + 0.00534533298923634, + 0.00482887499674689, + 0.00568145900615491, + 0.0046613329905085266, + 0.00474608299555257, + 0.004326958005549386, + 0.004350875009549782, + 0.0044323750043986365, + 0.0043131250131409615, + 0.005468333009048365, + 0.0045055000082356855, + 0.004820125002879649, + 0.0043440000008558854, + 0.004371165996417403, + 0.004377250006655231, + 0.003556125011527911, + 0.0055193329899339005, + 0.0042545000032987446, + 0.004400249992613681, + 0.004285707997041754, + 0.00433595800132025, + 0.003525416992488317, + 0.003569665990653448, + 0.004135875002248213, + 0.003529749999870546, + 0.003542916994774714, + 0.0035872909938916564, + 0.003708290998474695, + 0.003621875002863817, + 0.003631250001490116, + 0.0034147080004913732, + 0.003355499997269362 + ] + }, + { + "forward": 0.018605541990837082, + "prefill": 0.019133209003484808, + "decode_steps": [ + 0.0035665420000441372, + 0.002805791998980567, + 0.0027870000049006194, + 0.0029114170029060915, + 0.003119208005955443, + 0.00293987499026116, + 0.0030272500007413328, + 0.0033369169977959245, + 0.004934874988975935, + 0.00460945900704246, + 0.0030384580022655427, + 0.0029241669981274754, + 0.002992166002513841, + 0.0033568750077392906, + 0.0033892499923240393, + 0.002913416989031248, + 0.003088000012212433, + 0.0031315829983213916, + 0.003025916012120433, + 0.002999125004862435, + 0.0030872919887769967, + 0.003038124996237457, + 0.002995333998114802, + 0.002944084000773728, + 0.00299016700591892, + 0.003953291990910657, + 0.0030047500040382147, + 0.0030821660038782284, + 0.002830917001119815, + 0.0030062919977353886, + 0.004267249998520128, + 0.0029749999957857653 + ] + }, + { + "forward": 0.022266416999627836, + "prefill": 0.022764666995499283, + "decode_steps": [ + 0.003595791000407189, + 0.0034692079934757203, + 0.003605541991419159, + 0.005018416995881125, + 0.00335166699369438, + 0.003184832996339537, + 0.003279416006989777, + 0.005165709007997066, + 0.0032269580115098506, + 0.0034007079957518727, + 0.0031940420012688264, + 0.004688458997406997, + 0.003330791005282663, + 0.003297290997579694, + 0.0034317919926252216, + 0.0033726670080795884, + 0.0034112920111510903, + 0.0035067080025328323, + 0.003558249998604879, + 0.0037759159895358607, + 0.00489858299260959, + 0.0036237500025890768, + 0.0035122920089634135, + 0.003442290995735675, + 0.0036395000061020255, + 0.0039549170032842085, + 0.003444750007474795, + 0.003514250012813136, + 0.0036445830046432093, + 0.0038962499966146424, + 0.0034454160049790516, + 0.0034581660002004355 + ] + }, + { + "forward": 0.02301412500673905, + "prefill": 0.024571875008405186, + "decode_steps": [ + 0.0040199169889092445, + 0.004609832991263829, + 0.005047124999691732, + 0.0040282499976456165, + 0.0039562500023748726, + 0.003909541992470622, + 0.003981625006417744, + 0.006263915987801738, + 0.003791542010731064, + 0.005358666006941348, + 0.003843375001451932, + 0.0037474999990081415, + 0.003752749995328486, + 0.003800708000198938, + 0.003737125007319264, + 0.0037026659992989153, + 0.0036366249987622723, + 0.0040582910005468875, + 0.003981250003562309, + 0.0037795419921167195, + 0.0037829579960089177, + 0.0039004580030450597, + 0.003781416002311744, + 0.0038896250043762848, + 0.003761542000574991, + 0.004386582993902266, + 0.003983625007094815, + 0.003707833995576948, + 0.003944041993236169, + 0.004026167007395998, + 0.0039072920044418424, + 0.0039114579994929954 + ] + }, + { + "forward": 0.026110582999535836, + "prefill": 0.024778959006653167, + "decode_steps": [ + 0.004071458999533206, + 0.0039024580037221313, + 0.0037947079981677234, + 0.004255916996044107, + 0.0038798330060672015, + 0.005548000000999309, + 0.0037821250007255003, + 0.004102249993593432, + 0.003976874999352731, + 0.0038860000058775768, + 0.003885291996994056, + 0.003812957991613075, + 0.0037444159970618784, + 0.0040272920014103875, + 0.003907624995918013, + 0.003806249995250255, + 0.003738415995030664, + 0.0038638750120298937, + 0.0037825420004082844, + 0.003761917003430426, + 0.004659208003431559, + 0.0036995420086896047, + 0.003991458012023941, + 0.003914375003660098, + 0.005103832998429425, + 0.003858417010633275, + 0.003942791998269968, + 0.0039190829993458465, + 0.0051806670089717954, + 0.004023582994705066, + 0.0037983749934937805, + 0.0037979580083629116 + ] + }, + { + "forward": 0.023887540999567136, + "prefill": 0.025502958000288345, + "decode_steps": [ + 0.005890625005122274, + 0.005208792004850693, + 0.00481299999228213, + 0.004635707999113947, + 0.004637207995983772, + 0.004637499994714744, + 0.004627207992598414, + 0.00455337499442976, + 0.0056017919996520504, + 0.003744207991985604, + 0.0043365840101614594, + 0.005414499988546595, + 0.0044535000051837415, + 0.0037014170084148645, + 0.004326959009631537, + 0.004706791994976811, + 0.004005541995866224, + 0.003995291001047008, + 0.003803957995842211, + 0.005170834003365599, + 0.004186209000181407, + 0.0041125830030068755, + 0.004439958996954374, + 0.003943832998629659, + 0.004259291003108956, + 0.004003832989837974, + 0.003932042003725655, + 0.00441300000238698, + 0.003609542007325217, + 0.003980666006100364, + 0.0036960830038879067, + 0.004260749992681667 + ] + }, + { + "forward": 0.02154958399478346, + "prefill": 0.02162191699608229, + "decode_steps": [ + 0.0037794160016346723, + 0.003852708003250882, + 0.0036530829966068268, + 0.0037170840078033507, + 0.003391624995856546, + 0.003712207995704375, + 0.003753708995645866, + 0.0037114580045454204, + 0.0033858749957289547, + 0.004180165997240692, + 0.0037568749976344407, + 0.004015999991679564, + 0.0035998750099679455, + 0.0036235000006854534, + 0.003814666997641325, + 0.003784375003306195, + 0.003663208000943996, + 0.0037717079976573586, + 0.003792749994318001, + 0.003700207991641946, + 0.0038948330038692802, + 0.0037602919910568744, + 0.003637959001935087, + 0.003990582990809344, + 0.003716708000865765, + 0.004203417003736831, + 0.0036827500007348135, + 0.003708332995302044, + 0.003685749994474463, + 0.003853832997265272, + 0.0038124999991850927, + 0.0036252499994589016 + ] + }, + { + "forward": 0.021271167002851143, + "prefill": 0.020377334003569558, + "decode_steps": [ + 0.0044744999904651195, + 0.0034507919917814434, + 0.004099084006156772, + 0.004304874993977137, + 0.0041857080068439245, + 0.004958625009749085, + 0.003635457993368618, + 0.003859958000248298, + 0.0036803329858230427, + 0.0037673339975299314, + 0.0039219999889610335, + 0.004040874991915189, + 0.0037039169983472675, + 0.00377541700436268, + 0.003533167007844895, + 0.004609249997884035, + 0.00352549999661278, + 0.0046523749915650114, + 0.0038132500048959628, + 0.0035860420030076057, + 0.0034370000066701323, + 0.0036263750080252066, + 0.0035471669980324805, + 0.003773874996113591, + 0.0035591250052675605, + 0.004151250002905726, + 0.0034812090016203, + 0.0033589590020710602, + 0.0034216249914607033, + 0.0036601249885279685, + 0.003591667002183385, + 0.003595292000682093 + ] + }, + { + "forward": 0.018228417000500485, + "prefill": 0.019135416005156003, + "decode_steps": [ + 0.0031519579933956265, + 0.003059666996705346, + 0.0029540420073317364, + 0.0031589580030413345, + 0.003137125007924624, + 0.0029974580102134496, + 0.002896459001931362, + 0.0028705000004265457, + 0.0028525829984573647, + 0.002881832988350652, + 0.0029937499930383638, + 0.0029762089980067685, + 0.00313220800308045, + 0.0029542920092353597, + 0.0030102080054348335, + 0.0029515829955926165, + 0.0030635420116595924, + 0.0030898749973857775, + 0.002965832987683825, + 0.0030622500053141266, + 0.0030169159872457385, + 0.003105957992374897, + 0.003016417002072558, + 0.0030437500099651515, + 0.0029072079923935235, + 0.0029745829961029813, + 0.002948583001852967, + 0.003082332987105474, + 0.0030547920032404363, + 0.0030724999960511923, + 0.003145750000840053, + 0.0031051249970914796 + ] + }, + { + "forward": 0.018509500005166046, + "prefill": 0.018906166005763225, + "decode_steps": [ + 0.003104667004663497, + 0.0029908340075053275, + 0.0030589579982915893, + 0.003002540994202718, + 0.003116249994491227, + 0.0030163749906932935, + 0.0030024159932509065, + 0.003094290994340554, + 0.0030557499994756654, + 0.003074874999583699, + 0.0031247080041794106, + 0.0031496669980697334, + 0.003108000004431233, + 0.0030866249871905893, + 0.003236582997487858, + 0.003148834002786316, + 0.0030893329967511818, + 0.0030075829999987036, + 0.0031604169926140457, + 0.003190167000866495, + 0.0031117080070544034, + 0.003126374998828396, + 0.003144415997667238, + 0.003033083994523622, + 0.003169332994730212, + 0.0030906250030966476, + 0.0032582919957349077, + 0.0038955839991103858, + 0.0029617079999297857, + 0.003144832997350022, + 0.0031519580079475418, + 0.003030749998288229 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 21.410375498817302, + "p95": 25.110214099549918, + "min": 18.228417000500485, + "max": 26.110582999535836, + "iqr": 3.8606352645729203, + "mean": 21.349250098865014, + "cv": 0.12073188057294822 + }, + "prefill_ms": { + "count": 10, + "median": 21.56143749743933, + "p95": 25.177158453152515, + "min": 18.906166005763225, + "max": 25.502958000288345, + "iqr": 4.674177500419319, + "mean": 21.829345902369823, + "cv": 0.11419088410254324 + }, + "decode_ms_per_step": { + "count": 320, + "median": 3.7058754969621077, + "p95": 5.019852396071656, + "min": 2.7870000049006194, + "max": 6.263915987801738, + "iqr": 0.8523022370354738, + "mean": 3.7408049184705305, + "cv": 0.1705116048022327 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 2107.0991638057653, + "p95": 2609.7603549495375, + "min": 1814.2828625678915, + "max": 2649.164300958148, + "iqr": 436.64054381313235, + "mean": 2172.69005304053, + "cv": 0.13558932702364035 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1796.4400947242823, + "p95": 2185.548427392662, + "min": 1536.5635316769324, + "max": 2211.287406272653, + "iqr": 370.65443022248223, + "mean": 1835.3118817324564, + "cv": 0.12995534403014458 + }, + "memory": { + "peak_extra_bytes": 33816576, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "reference", + "batch": 8, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.0490903750032885, + "prefill": 0.04562679200898856, + "decode_steps": [ + 0.013504792004823685, + 0.009366583006340079, + 0.00811249999969732, + 0.00818424999306444, + 0.0073307080019731075, + 0.006796999994548969, + 0.00601845899655018, + 0.00549354100076016, + 0.006306334005785175, + 0.006445333012379706, + 0.006419709010515362, + 0.005891499997233041, + 0.005773832992417738, + 0.005856167001184076, + 0.005756499987910502, + 0.005672832994605415, + 0.005825917003676295, + 0.005906874997890554, + 0.0057846249983413145, + 0.005871750006917864, + 0.005375250009819865, + 0.0057734169968171045, + 0.005767499998910353, + 0.005599791998974979, + 0.005728749994887039, + 0.006276667001657188, + 0.007572499991510995, + 0.00723620799544733, + 0.006910249998327345, + 0.007287874992471188, + 0.005786875000922009, + 0.0062220830004662275 + ] + }, + { + "forward": 0.04935016699892003, + "prefill": 0.04899025001213886, + "decode_steps": [ + 0.014426166002522223, + 0.009499833002337255, + 0.009187957999529317, + 0.008042083005420864, + 0.00810370800900273, + 0.0072040829982142895, + 0.007627957995282486, + 0.00781287498830352, + 0.008363291999557987, + 0.007292833004612476, + 0.00808612500259187, + 0.007841708007617854, + 0.008151582995196804, + 0.007582375008496456, + 0.00848874999792315, + 0.007559916004538536, + 0.009184124995954335, + 0.0073591249965829775, + 0.008526250006980263, + 0.00730387499788776, + 0.00943945899780374, + 0.006944417007616721, + 0.00834374999976717, + 0.007226249988889322, + 0.008760791999520734, + 0.006587000010767952, + 0.008573375002015382, + 0.006894334001117386, + 0.006922082990058698, + 0.006417790995328687, + 0.006887583003845066, + 0.00845829200989101 + ] + }, + { + "forward": 0.05158241699973587, + "prefill": 0.04972883399750572, + "decode_steps": [ + 0.009435708008822985, + 0.00987754100060556, + 0.00843233299383428, + 0.00801195799431298, + 0.007879374999902211, + 0.009226124995620921, + 0.007331042012083344, + 0.00855662499088794, + 0.006632750009885058, + 0.008175583003321663, + 0.00711024999327492, + 0.008761124990996905, + 0.008946208006818779, + 0.01056958400295116, + 0.008027624993701465, + 0.0075305000063963234, + 0.007806833003996871, + 0.0075546660082181916, + 0.007856291995267384, + 0.008461624995106831, + 0.008381082996493205, + 0.007209375005913898, + 0.007094250002410263, + 0.007251458999235183, + 0.007750165998004377, + 0.008867041993653402, + 0.007952208005008288, + 0.007218124999781139, + 0.007240332997753285, + 0.007929624989628792, + 0.007491791999200359, + 0.00748033300624229 + ] + }, + { + "forward": 0.0576709590095561, + "prefill": 0.053997542010620236, + "decode_steps": [ + 0.016190791997360066, + 0.010632792007527314, + 0.010505875005037524, + 0.008765582999330945, + 0.008191041997633874, + 0.008244040989666246, + 0.007410457998048514, + 0.007581541998661123, + 0.007417290995363146, + 0.007497874990804121, + 0.00767249999626074, + 0.007668167003430426, + 0.007669333994272165, + 0.007693167004617862, + 0.007328332998440601, + 0.007503250002628192, + 0.007283790997462347, + 0.007490957999834791, + 0.00745137500052806, + 0.00753212500421796, + 0.007293374990695156, + 0.008995292009785771, + 0.007755083002848551, + 0.008419290999881923, + 0.008806459009065293, + 0.008497834001900628, + 0.007590958004584536, + 0.007974082996952347, + 0.008028749987715855, + 0.007818375001079403, + 0.010922000001301058, + 0.016766834000009112 + ] + }, + { + "forward": 0.05616929099778645, + "prefill": 0.052883374999510124, + "decode_steps": [ + 0.01110566699935589, + 0.008936499987612478, + 0.008509292005328462, + 0.008115625008940697, + 0.007687541990890168, + 0.007962584000779316, + 0.00801179200061597, + 0.007826917004422285, + 0.0075301670003682375, + 0.008417916993494146, + 0.013980040996102616, + 0.010730291993240826, + 0.008300083994981833, + 0.008660791994770989, + 0.009705000004032627, + 0.00924445800774265, + 0.010234707995550707, + 0.008908584000892006, + 0.009764499991433695, + 0.008814667002297938, + 0.011835833007353358, + 0.010149665991775692, + 0.008522124990122393, + 0.00895233399933204, + 0.009904374994221143, + 0.012091750002582557, + 0.010190083004999906, + 0.009087917002034374, + 0.00923341599991545, + 0.008969124988652766, + 0.008851292004692368, + 0.008281540998723358 + ] + }, + { + "forward": 0.052319041002192535, + "prefill": 0.04905045799387153, + "decode_steps": [ + 0.009702834009658545, + 0.008871500001987442, + 0.008243874995969236, + 0.00803233400802128, + 0.00788766700134147, + 0.008099750004475936, + 0.007950166997034103, + 0.007827749999705702, + 0.007892417008406483, + 0.00878299999749288, + 0.007905500009655952, + 0.007839166995836422, + 0.007427415999700315, + 0.007764791997033171, + 0.0076865000010002404, + 0.007827042005374096, + 0.007879666009102948, + 0.015032333001727238, + 0.07419349999690894, + 0.021010875003412366, + 0.012456958997063339, + 0.011294084004475735, + 0.01358570899174083, + 0.011822750006103888, + 0.01236195900128223, + 0.01205091699375771, + 0.015708416991401464, + 0.014354667000588961, + 0.014634999999543652, + 0.013763666996965185, + 0.01170920800359454, + 0.01099837500078138 + ] + }, + { + "forward": 0.054376832995330915, + "prefill": 0.04705862500122748, + "decode_steps": [ + 0.011060374992666766, + 0.009163374998024665, + 0.009116792003624141, + 0.008875582992914133, + 0.008377750011277385, + 0.008761207995121367, + 0.00809470799867995, + 0.008686415996635333, + 0.007403583003906533, + 0.008186583014321513, + 0.00782424998760689, + 0.008452583002508618, + 0.0078809170081513, + 0.00822129100561142, + 0.008202917000744492, + 0.007512707990827039, + 0.007456833991454914, + 0.007854416995542124, + 0.007908250001491979, + 0.008505999998305924, + 0.008271165992482565, + 0.009433125000214204, + 0.007951084000524133, + 0.007903249992523342, + 0.007376540990662761, + 0.007927416998427361, + 0.007902333003585227, + 0.008128500005113892, + 0.007704667004873045, + 0.007465916991350241, + 0.007582541991723701, + 0.007928749997518025 + ] + }, + { + "forward": 0.0539058330032276, + "prefill": 0.04564474998915102, + "decode_steps": [ + 0.008387709007365629, + 0.00767529099539388, + 0.007506499998271465, + 0.007001709003816359, + 0.006819458998506889, + 0.007387167002889328, + 0.007270292000612244, + 0.0069012919993838295, + 0.006541666996781714, + 0.006667749999905936, + 0.006238416011910886, + 0.006547707991558127, + 0.006721333003952168, + 0.006596334002097137, + 0.007138040993595496, + 0.006373833995894529, + 0.006571874997462146, + 0.006337042010272853, + 0.006439542004955001, + 0.006623209003009833, + 0.006633124998188578, + 0.006533375009894371, + 0.006727457992383279, + 0.006705917010549456, + 0.006930208008270711, + 0.006402792001608759, + 0.0067144589993404225, + 0.007137833003071137, + 0.0066652500099735335, + 0.0068725000019185245, + 0.007084666998707689, + 0.006959541991818696 + ] + }, + { + "forward": 0.04726237501017749, + "prefill": 0.042767042003106326, + "decode_steps": [ + 0.008266832999652252, + 0.007013707989244722, + 0.0072947500011650845, + 0.006825749995186925, + 0.0069778329925611615, + 0.007583415994304232, + 0.0076941669976804405, + 0.0071439589955843985, + 0.0069193329982226714, + 0.007293457994819619, + 0.0069479590019909665, + 0.006801875002565794, + 0.007080167008098215, + 0.006095958000514656, + 0.006958457990549505, + 0.006279665991314687, + 0.006278792003286071, + 0.006529167003463954, + 0.006087375004426576, + 0.006510291001177393, + 0.006385832995874807, + 0.0068629580055130646, + 0.006517624991829507, + 0.006541416994878091, + 0.006423709011869505, + 0.006440582990762778, + 0.006530416998430155, + 0.006189250008901581, + 0.007419749992550351, + 0.006550584002980031, + 0.006791416002670303, + 0.00652708399866242 + ] + }, + { + "forward": 0.04793354100547731, + "prefill": 0.043956250010523945, + "decode_steps": [ + 0.008071457996265963, + 0.007229958995594643, + 0.006821542003308423, + 0.006622791988775134, + 0.00632891699206084, + 0.006786750003811903, + 0.006658416998106986, + 0.006655750010395423, + 0.006278000000747852, + 0.006669791997410357, + 0.006633124998188578, + 0.00680145900696516, + 0.006743790989276022, + 0.006122125007095747, + 0.006636625010287389, + 0.00666929199360311, + 0.006595167011255398, + 0.006427290994906798, + 0.006588082993403077, + 0.006469957996159792, + 0.0065159999940078706, + 0.006429250002838671, + 0.006521791001432575, + 0.007156583000323735, + 0.007868083004723303, + 0.008362416992895305, + 0.00796037499094382, + 0.0075815829914063215, + 0.007111499988241121, + 0.0067235829919809476, + 0.006934334000106901, + 0.006535416992846876 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 51.9507290009642, + "p95": 56.995208404259756, + "min": 47.26237501017749, + "max": 57.6709590095561, + "iqr": 5.103759995108703, + "mean": 51.96608320256928, + "cv": 0.06835448188421223 + }, + "prefill_ms": { + "count": 10, + "median": 48.02443750668317, + "p95": 53.496166855620686, + "min": 42.767042003106326, + "max": 53.997542010620236, + "iqr": 3.9279584925679956, + "mean": 47.97039180266438, + "cv": 0.07633593809756226 + }, + "decode_ms_per_step": { + "count": 320, + "median": 7.587186999444384, + "p95": 12.052958644198952, + "min": 5.375250009819865, + "max": 74.19349999690894, + "iqr": 1.6417914994235616, + "mean": 8.20697046215173, + "cv": 0.5100710843219246 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 990.0470238165955, + "p95": 1193.615367089795, + "min": 635.8671859444498, + "max": 1208.365803519259, + "iqr": 230.4359259428985, + "mean": 1009.0522845100206, + "cv": 0.1775150495287786 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 831.3466902961263, + "p95": 989.0064370476624, + "min": 566.8102633658895, + "max": 994.2397477576504, + "iqr": 183.37240229227973, + "mean": 847.3130377316163, + "cv": 0.16161578425622145 + }, + "memory": { + "peak_extra_bytes": 1107558400, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "auto", + "batch": 8, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.06303362500329968 + }, + { + "forward": 0.06355912500293925 + }, + { + "forward": 0.05883912500576116 + }, + { + "forward": 0.06187508400762454 + }, + { + "forward": 0.054282458004308864 + }, + { + "forward": 0.05324670800473541 + }, + { + "forward": 0.059616416998323984 + }, + { + "forward": 0.05906004100688733 + }, + { + "forward": 0.06027770800574217 + }, + { + "forward": 0.06516533299873117 + } + ], + "forward_ms": { + "count": 10, + "median": 59.94706250203308, + "p95": 64.44253940062481, + "min": 53.24670800473541, + "max": 65.16533299873117, + "iqr": 3.849635748338187, + "mean": 59.895562403835356, + "cv": 0.0641116027810262 + }, + "memory": { + "peak_extra_bytes": 1116995584, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "sdpa", + "batch": 8, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.05453733399917837, + "prefill": 0.058107958000618964, + "decode_steps": [ + 0.009610791996237822, + 0.008558832996641286, + 0.007920167001429945, + 0.007957124995300546, + 0.007746415998553857, + 0.007431416001054458, + 0.009386249992530793, + 0.006517333997180685, + 0.006958750003832392, + 0.006137417003628798, + 0.008223625001846813, + 0.007789958995999768, + 0.00865116600471083, + 0.007528125002863817, + 0.00485675000527408, + 0.00536291699972935, + 0.004769791994476691, + 0.004559458000585437, + 0.005053834000136703, + 0.004908375005470589, + 0.004904166009509936, + 0.005009374988730997, + 0.005155417005880736, + 0.004723791993455961, + 0.005246500004432164, + 0.005003042009775527, + 0.0049250410083914176, + 0.004954832998919301, + 0.0052220000070519745, + 0.005123791997903027, + 0.00501599999552127, + 0.005475207988638431 + ] + }, + { + "forward": 0.054375375009840354, + "prefill": 0.05626779100566637, + "decode_steps": [ + 0.005178291990887374, + 0.005084165997686796, + 0.005041124997660518, + 0.004664957989007235, + 0.004786332996445708, + 0.004840041001443751, + 0.004545332994894125, + 0.004764375000377186, + 0.005142083013197407, + 0.004607124996255152, + 0.005400374997407198, + 0.005184458001167513, + 0.004906292000669055, + 0.00552104199596215, + 0.0050433750002412125, + 0.004835625004488975, + 0.005105749994982034, + 0.005057000002125278, + 0.005173749988898635, + 0.004860125001869164, + 0.0053944169922033325, + 0.0053299999999580905, + 0.005346124991774559, + 0.005102124996483326, + 0.0050697910046437755, + 0.0054438750084955245, + 0.005188208000618033, + 0.005099790985696018, + 0.005314334004651755, + 0.005160457993042655, + 0.005446791998110712, + 0.005280500001390465 + ] + }, + { + "forward": 0.047620042008929886, + "prefill": 0.048488125001313165, + "decode_steps": [ + 0.0049039579898817465, + 0.005742792011005804, + 0.004552500002318993, + 0.004396459000417963, + 0.0044895419996464625, + 0.004460958007257432, + 0.004810291997273453, + 0.004685584004619159, + 0.004560874993330799, + 0.004348707996541634, + 0.004352624993771315, + 0.004734834001283161, + 0.004746000005980022, + 0.0047300840087700635, + 0.004756334004923701, + 0.004868999996688217, + 0.00506083300570026, + 0.005177209008252248, + 0.005486916998052038, + 0.005504791988641955, + 0.0074802080052904785, + 0.0073721250082599, + 0.00915716600138694, + 0.007492582997656427, + 0.007435999999870546, + 0.007238499994855374, + 0.007926416001282632, + 0.0075714580016210675, + 0.007404582996969111, + 0.0068652920017484576, + 0.007460125008947216, + 0.006364749991917051 + ] + }, + { + "forward": 0.05053483300434891, + "prefill": 0.052819417003775015, + "decode_steps": [ + 0.005309832995408215, + 0.004229208992910571, + 0.004498958995100111, + 0.004331250005634502, + 0.004244291994837113, + 0.004339125007390976, + 0.004459625008166768, + 0.004335417004767805, + 0.00424850000126753, + 0.00438195800234098, + 0.004392792005091906, + 0.004288915995857678, + 0.004234209001879208, + 0.004377208999358118, + 0.004440999997314066, + 0.0043129160039825365, + 0.004481083989958279, + 0.004387332999613136, + 0.004280415989342146, + 0.00445337500423193, + 0.005746499999077059, + 0.0065426250075688586, + 0.009053000001586042, + 0.006324374990072101, + 0.004435999988345429, + 0.004702334001194686, + 0.004674708005040884, + 0.004515541004366241, + 0.004462416996830143, + 0.004516834000241943, + 0.004591374992742203, + 0.0045160000008763745 + ] + }, + { + "forward": 0.044804667006246746, + "prefill": 0.04491950001101941, + "decode_steps": [ + 0.005030458007240668, + 0.004415833987877704, + 0.004216000001179054, + 0.004301833003410138, + 0.004384790998301469, + 0.004193542001303285, + 0.004248958008247428, + 0.004364499996881932, + 0.00433691700163763, + 0.004268167002010159, + 0.00432550000550691, + 0.004254707993823104, + 0.004836624997551553, + 0.0043413329985924065, + 0.0044072080054320395, + 0.004283208996639587, + 0.004218625006615184, + 0.0042529579950496554, + 0.004424500002642162, + 0.0045628329971805215, + 0.004521292008575983, + 0.00463149999268353, + 0.005014582988223992, + 0.004961624988936819, + 0.004986749991076067, + 0.005886165992706083, + 0.005313582994858734, + 0.0053989590087439865, + 0.005318958006682806, + 0.005082292002043687, + 0.005340249990695156, + 0.005477333004819229 + ] + }, + { + "forward": 0.04456841699720826, + "prefill": 0.044115708995377645, + "decode_steps": [ + 0.004532708000624552, + 0.004260166999301873, + 0.004384208004921675, + 0.004232999999658205, + 0.0041796669975155964, + 0.004321250002249144, + 0.0042863340058829635, + 0.004334541998105124, + 0.004165207996265963, + 0.004352250005467795, + 0.004448958992725238, + 0.004132250003749505, + 0.00420604100509081, + 0.004382540995720774, + 0.004226584002026357, + 0.004494583001360297, + 0.0045979169954080135, + 0.004555833991616964, + 0.00455991699709557, + 0.004749749990878627, + 0.004783625001437031, + 0.00470270799996797, + 0.005356832989491522, + 0.005588792002527043, + 0.007109458005288616, + 0.0072492919862270355, + 0.0071412499964935705, + 0.007575084004201926, + 0.007766624999931082, + 0.008679665988893248, + 0.0068176670029060915, + 0.007155165993026458 + ] + }, + { + "forward": 0.050837332993978634, + "prefill": 0.04889083400485106, + "decode_steps": [ + 0.0046868329955032095, + 0.005066457990324125, + 0.0045557909907074645, + 0.004250416997820139, + 0.004400999998324551, + 0.004609791998518631, + 0.004785166995134205, + 0.004443707992322743, + 0.0047425830125575885, + 0.004748292005388066, + 0.0046723330015083775, + 0.004515999986324459, + 0.004969332992914133, + 0.004676709009800106, + 0.004492790991207585, + 0.0047443749936064705, + 0.004843665999942459, + 0.004692375005106442, + 0.0049255420017289, + 0.004907290989649482, + 0.004784874996403232, + 0.004824208008358255, + 0.005032166998717003, + 0.004762750002555549, + 0.004721667006378993, + 0.0051298749895067886, + 0.004883125002379529, + 0.0052888749924022704, + 0.0053269579948391765, + 0.0051667920051841065, + 0.004754166991915554, + 0.005084999997052364 + ] + }, + { + "forward": 0.044927916998858564, + "prefill": 0.04613141699519474, + "decode_steps": [ + 0.004505207994952798, + 0.004487958998652175, + 0.004115000003366731, + 0.004066374996909872, + 0.004188666993286461, + 0.004387374996440485, + 0.00408495900046546, + 0.004070958006195724, + 0.004327916001784615, + 0.004331707998062484, + 0.004020458000013605, + 0.004121541991480626, + 0.004302625005948357, + 0.004330750001827255, + 0.004183333003311418, + 0.004311999990022741, + 0.00473487499402836, + 0.004445833008503541, + 0.004502750001847744, + 0.004875250000623055, + 0.004754916008096188, + 0.004872000004979782, + 0.005891584005439654, + 0.005252417002338916, + 0.005361375006032176, + 0.007818792000762187, + 0.006366750007146038, + 0.0056650000042282045, + 0.005214791992329992, + 0.005912709006224759, + 0.005256332995486446, + 0.005386250006267801 + ] + }, + { + "forward": 0.047320084006059915, + "prefill": 0.04981450000195764, + "decode_steps": [ + 0.004734916001325473, + 0.0045666250080103055, + 0.004377584002213553, + 0.004496291992836632, + 0.004431332999956794, + 0.0045959999988554046, + 0.004413874994497746, + 0.004347166992374696, + 0.0043621249933494255, + 0.004388499990454875, + 0.004409082990605384, + 0.004346582994912751, + 0.004567333002341911, + 0.004456792012206279, + 0.0044864999945275486, + 0.00462495899409987, + 0.005065417004516348, + 0.00468645800719969, + 0.004769958002725616, + 0.004966374996001832, + 0.004988041007891297, + 0.004980209007044323, + 0.005295915994793177, + 0.005397707995143719, + 0.005479792002006434, + 0.007012416986981407, + 0.0069097090017749, + 0.007135624997317791, + 0.008366624999325722, + 0.007605374994454905, + 0.007598208001581952, + 0.007459125001332723 + ] + }, + { + "forward": 0.05343804199947044, + "prefill": 0.054907875004573725, + "decode_steps": [ + 0.009600040997611359, + 0.008257041990873404, + 0.00845287500123959, + 0.007672375009860843, + 0.007750832999590784, + 0.006855250001535751, + 0.007561125006759539, + 0.007848584005841985, + 0.006400041005690582, + 0.005882665995159186, + 0.005469833995448425, + 0.005349332990590483, + 0.005460999993374571, + 0.005491041010827757, + 0.005384542004321702, + 0.005470040996442549, + 0.00541187499766238, + 0.005609458006802015, + 0.00523991699446924, + 0.006314208003459498, + 0.005723125010263175, + 0.005437915999209508, + 0.005558167002163827, + 0.005461540989927016, + 0.005780040999525227, + 0.007842125007300638, + 0.007963749987538904, + 0.005547417007619515, + 0.006880040993564762, + 0.0047089170111576095, + 0.004456374997971579, + 0.004635250006685965 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 49.0774375066394, + "p95": 54.464452453976264, + "min": 44.56841699720826, + "max": 54.53733399917837, + "iqr": 7.261905997438589, + "mean": 49.29640440241201, + "cv": 0.08076167628009874 + }, + "prefill_ms": { + "count": 10, + "median": 49.35266700340435, + "p95": 57.279882852890296, + "min": 44.115708995377645, + "max": 58.107958000618964, + "iqr": 7.665166507649701, + "mean": 50.44631260243477, + "cv": 0.09666690748730922 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.879187501501292, + "p95": 7.852163155621384, + "min": 4.020458000013605, + "max": 9.610791996237822, + "iqr": 1.0092609882121906, + "mean": 5.303827330726563, + "cv": 0.22774310471016393 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1550.627175785673, + "p95": 1698.5162132159355, + "min": 1270.618098894479, + "max": 1711.209471752104, + "iqr": 256.61143882258625, + "mean": 1525.7505221379101, + "cv": 0.10926140551352888 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1189.4658436653276, + "p95": 1299.0233081478343, + "min": 989.1975569283826, + "max": 1316.0513176503462, + "iqr": 150.6984951027241, + "mean": 1173.409312683643, + "cv": 0.0976672236618292 + }, + "memory": { + "peak_extra_bytes": 1107558400, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "reference", + "batch": 8, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.15483458399830852, + "prefill": 0.15678308300266508, + "decode_steps": [ + 0.019104916995274834, + 0.0128574580012355, + 0.01059945800807327, + 0.012426124987541698, + 0.01006566699652467, + 0.00939908299187664, + 0.009772792007424869, + 0.008747250001761131, + 0.00996224999835249, + 0.008783583994954824, + 0.009321624995209277, + 0.008842999988701195, + 0.008652999997138977, + 0.009089416998904198, + 0.00903062499128282, + 0.00915216600697022, + 0.008273832994746044, + 0.008606083007180132, + 0.008432334012468345, + 0.00844745799258817, + 0.008469499996863306, + 0.008430582995060831, + 0.008393791998969391, + 0.008471042005112395, + 0.008447999993222766, + 0.012267541998880915, + 0.010354583006119356, + 0.008868832999723963, + 0.008487207989674062, + 0.008338832994922996, + 0.008780042000580579, + 0.008450833993265405 + ] + }, + { + "forward": 0.1537103339942405, + "prefill": 0.1560421670001233, + "decode_steps": [ + 0.012302999995881692, + 0.0107012090011267, + 0.009825665998505428, + 0.009911499990266748, + 0.009430332996998914, + 0.008688125002663583, + 0.008559125009924173, + 0.008860957997967489, + 0.008776749993558042, + 0.008821875002468005, + 0.010083291999762878, + 0.00895104200753849, + 0.009003667000797577, + 0.009090583000215702, + 0.009087457990972325, + 0.009147416989435442, + 0.008900457993149757, + 0.00888545899942983, + 0.008453209011349827, + 0.008898166997823864, + 0.008294832994579338, + 0.008770875007030554, + 0.00847737499861978, + 0.008560834001400508, + 0.00888820800173562, + 0.009070624990272336, + 0.008605082999565639, + 0.008243792006396689, + 0.009213458004524, + 0.008313666010508314, + 0.00872954199439846, + 0.008104959008051082 + ] + }, + { + "forward": 0.1477911249967292, + "prefill": 0.14421045799099375, + "decode_steps": [ + 0.022127750009531155, + 0.016898542002309114, + 0.012733999988995492, + 0.00994899999932386, + 0.010124749998794869, + 0.009508292001555674, + 0.008586957992520183, + 0.00876691599842161, + 0.008775249996688217, + 0.010510833002626896, + 0.011960582996835, + 0.01134266699955333, + 0.01124154101125896, + 0.0087633749935776, + 0.008477124996716157, + 0.008662207997986116, + 0.008423791994573548, + 0.008297499996842816, + 0.008369207993382588, + 0.008501999996951781, + 0.008051457989495248, + 0.008168291009496897, + 0.008104124994133599, + 0.008433167007751763, + 0.008791250002104789, + 0.009495624995906837, + 0.008142792008584365, + 0.008691833005286753, + 0.007860333993448876, + 0.008495041998685338, + 0.007792167001753114, + 0.008034290993236937 + ] + }, + { + "forward": 0.14076383301289752, + "prefill": 0.1402205420017708, + "decode_steps": [ + 0.021643874992150813, + 0.016592915999353863, + 0.011551499992492609, + 0.010763165992102586, + 0.009766832998138852, + 0.009233874996425584, + 0.008768457992118783, + 0.01003104199480731, + 0.009655792004195973, + 0.008457083997200243, + 0.00851737501216121, + 0.008382124986383133, + 0.008123042003717273, + 0.008201250006095506, + 0.008288458993774839, + 0.009310875000664964, + 0.00889833300607279, + 0.00890600000275299, + 0.008007707991055213, + 0.008000625006388873, + 0.008043125009862706, + 0.008040458007599227, + 0.008091249997960404, + 0.009081041993340477, + 0.008201542004826479, + 0.00789233300019987, + 0.008022583002457395, + 0.007957958994666114, + 0.007737457999610342, + 0.007809249989804812, + 0.007932082997285761, + 0.007629624989931472 + ] + }, + { + "forward": 0.1372112919925712, + "prefill": 0.13651991699589416, + "decode_steps": [ + 0.020192750002024695, + 0.014922417001798749, + 0.011137958004837856, + 0.011325041996315122, + 0.009481708009843715, + 0.009241459003533237, + 0.008410207999986596, + 0.008564792005927302, + 0.008433625000179745, + 0.007935041998280212, + 0.007691874998272397, + 0.007960333998198621, + 0.007825249995221384, + 0.007764916997984983, + 0.0078099590027704835, + 0.008326957991812378, + 0.007962582996697165, + 0.007591957997647114, + 0.00759716700122226, + 0.007688041005167179, + 0.007635375004610978, + 0.007818083991878666, + 0.0076485830068122596, + 0.007613207999384031, + 0.007795292010996491, + 0.007776916987495497, + 0.007489916999475099, + 0.007723542003077455, + 0.007568667002487928, + 0.007479000007151626, + 0.007691334001719952, + 0.00842562499747146 + ] + }, + { + "forward": 0.13348012500500772, + "prefill": 0.13292933400953189, + "decode_steps": [ + 0.019041249994188547, + 0.011457457992946729, + 0.008570999998482876, + 0.007901457996922545, + 0.007077832997310907, + 0.008057916988036595, + 0.007504041001084261, + 0.007779084000503644, + 0.008020458000828512, + 0.0077922079944983125, + 0.0075274999981047586, + 0.0076074160024290904, + 0.008284040988655761, + 0.008272874998510815, + 0.007373375003226101, + 0.0074391249945620075, + 0.007554208001238294, + 0.007252541996422224, + 0.0075412079895613715, + 0.00832558301044628, + 0.008108583991997875, + 0.00840579200303182, + 0.008005583003978245, + 0.007599541000672616, + 0.007681000002776273, + 0.007519917009631172, + 0.007702209011767991, + 0.007897709001554176, + 0.008432209011516534, + 0.00747416699596215, + 0.007446000003255904, + 0.007523042004322633 + ] + }, + { + "forward": 0.14282129200000782, + "prefill": 0.1432829999976093, + "decode_steps": [ + 0.020118291009566747, + 0.01685449999058619, + 0.012724874992272817, + 0.010793416993692517, + 0.009852542003500275, + 0.009520666004391387, + 0.008655708996229805, + 0.008684458996867761, + 0.008576666994486004, + 0.008391500014113262, + 0.008158958997228183, + 0.008228124992456287, + 0.007662582996999845, + 0.00900020799599588, + 0.008220709001761861, + 0.008234041990363039, + 0.0076085830078227445, + 0.007900207987404428, + 0.007111083003110252, + 0.007945749995997176, + 0.0076333750039339066, + 0.007687958001042716, + 0.007357666007010266, + 0.007774666999466717, + 0.007625792000908405, + 0.007686750002903864, + 0.0070690840075258166, + 0.007403625000733882, + 0.007935957997688092, + 0.008101042010821402, + 0.007170124998083338, + 0.00783233399852179 + ] + }, + { + "forward": 0.13813766700332053, + "prefill": 0.14158270800544415, + "decode_steps": [ + 0.013044082996202633, + 0.010785500009660609, + 0.009643458004575223, + 0.008721916005015373, + 0.008558750007068738, + 0.00843512499704957, + 0.008305458002723753, + 0.008202666998840868, + 0.008172041998477653, + 0.00797775000683032, + 0.009044291000464, + 0.007980416994541883, + 0.007979500005603768, + 0.007845750005799346, + 0.007851500005926937, + 0.007619666997925378, + 0.007925917001557536, + 0.007868874992709607, + 0.007631125001353212, + 0.008244125012424774, + 0.008416125012445264, + 0.008228249993408099, + 0.008572750011808239, + 0.008010499994270504, + 0.00790541700553149, + 0.008069959003478289, + 0.007313750000321306, + 0.007587291998788714, + 0.0075741670007118955, + 0.007610957996803336, + 0.007528916990850121, + 0.008661792002385482 + ] + }, + { + "forward": 0.1371355839946773, + "prefill": 0.1371639159915503, + "decode_steps": [ + 0.00960958399809897, + 0.008300791989313439, + 0.007869374996516854, + 0.007832417002646253, + 0.007951207997393794, + 0.0087072499882197, + 0.0076202919881325215, + 0.008463082995149307, + 0.007785583002259955, + 0.007872583999414928, + 0.009157666994724423, + 0.008724375002202578, + 0.008195374990464188, + 0.008068333991104737, + 0.007788415998220444, + 0.0075126250012544915, + 0.007833374998881482, + 0.008482000004732981, + 0.008324249996803701, + 0.00811495800735429, + 0.007968709003762342, + 0.007716749998508021, + 0.008079958002781495, + 0.007713208004133776, + 0.007798291990184225, + 0.00777195900445804, + 0.007883041005698033, + 0.007652542000869289, + 0.007796874997438863, + 0.007338791008805856, + 0.00806054200802464, + 0.00804387500102166 + ] + }, + { + "forward": 0.14026412500243168, + "prefill": 0.1406907499913359, + "decode_steps": [ + 0.02126824999868404, + 0.01435016599134542, + 0.011312750008073635, + 0.010252124993712641, + 0.009626459010178223, + 0.009577791002811864, + 0.009231208998244256, + 0.008977332996437326, + 0.008570999998482876, + 0.009318167009041645, + 0.008347875002073124, + 0.009084707999136299, + 0.008012667007278651, + 0.008157374992151745, + 0.007555083007900976, + 0.008069417002843693, + 0.007287542000995018, + 0.008466832994599827, + 0.00748649999150075, + 0.00792337500024587, + 0.007488917006412521, + 0.007692207989748567, + 0.007408624995150603, + 0.007511125004384667, + 0.007583250000607222, + 0.007910750005976297, + 0.007316458999412134, + 0.007574874995043501, + 0.007459165994077921, + 0.007438375003403053, + 0.007345291000092402, + 0.00740724999923259 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 140.5139790076646, + "p95": 154.3286714964779, + "min": 133.48012500500772, + "max": 154.83458399830852, + "iqr": 9.105781002290314, + "mean": 142.6149961000192, + "cv": 0.0507214279402667 + }, + "prefill_ms": { + "count": 10, + "median": 141.13672899839003, + "p95": 156.44967080152128, + "min": 132.92933400953189, + "max": 156.78308300266508, + "iqr": 6.050520998542197, + "mean": 142.94258749869186, + "cv": 0.054893271173463036 + }, + "decode_ms_per_step": { + "count": 320, + "median": 8.299145993078128, + "p95": 12.441062487778257, + "min": 7.069084007525817, + "max": 22.127750009531155, + "iqr": 1.0953330056509003, + "mean": 8.891247287147053, + "cv": 0.25275468563953135 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 900.5790596025645, + "p95": 981.728257906012, + "min": 820.2954817307112, + "max": 992.1016117879518, + "iqr": 80.13831366012482, + "mean": 903.0018906431694, + "cv": 0.06324035407557825 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 600.6024316958348, + "p95": 646.372608334044, + "min": 551.5909953444245, + "max": 647.7699995902335, + "iqr": 46.023762200075, + "mean": 600.5358664995263, + "cv": 0.05528364759238267 + }, + "memory": { + "peak_extra_bytes": 1107558400, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "auto", + "batch": 8, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.18537558299431112 + }, + { + "forward": 0.16571087499323767 + }, + { + "forward": 0.15746520800166763 + }, + { + "forward": 0.16842475000885315 + }, + { + "forward": 0.1904725829954259 + }, + { + "forward": 0.16866412499803118 + }, + { + "forward": 0.1973430840007495 + }, + { + "forward": 0.20906824999838136 + }, + { + "forward": 0.1487722500023665 + }, + { + "forward": 0.18872154199925717 + } + ], + "forward_ms": { + "count": 10, + "median": 177.01985399617115, + "p95": 203.79192529944703, + "min": 148.7722500023665, + "max": 209.06824999838136, + "iqr": 23.645478999242187, + "mean": 178.00182499922812, + "cv": 0.10708672763842567 + }, + "memory": { + "peak_extra_bytes": 1092878336, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "sdpa", + "batch": 8, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.13820345798740163, + "prefill": 0.15290187498612795, + "decode_steps": [ + 0.012022707989672199, + 0.010887917000218295, + 0.00970470899483189, + 0.011079582996899262, + 0.010379583996837027, + 0.013569125003414229, + 0.010184667000430636, + 0.010147292006877251, + 0.011973915999988094, + 0.01250566700764466, + 0.011197250001714565, + 0.010511540996958502, + 0.009271292001358233, + 0.009266207998734899, + 0.010112999996636063, + 0.009137666987953708, + 0.009301082987803966, + 0.007828666988643818, + 0.008796292007900774, + 0.00790462500299327, + 0.007805166998878121, + 0.00788837501022499, + 0.008245124990935437, + 0.00825849999091588, + 0.008311500001582317, + 0.008427792010479607, + 0.008323875008500181, + 0.00882820799597539, + 0.008568916993681341, + 0.009976790999644436, + 0.007989083998836577, + 0.009757124993484467 + ] + }, + { + "forward": 0.10570450000523124, + "prefill": 0.11246099999698345, + "decode_steps": [ + 0.008581083995522931, + 0.00664649999816902, + 0.006513707994599827, + 0.006433457994717173, + 0.006786790996557102, + 0.006439540986320935, + 0.006414582996512763, + 0.006533332998515107, + 0.006420624995371327, + 0.007507832997362129, + 0.009128082994720899, + 0.00722762499935925, + 0.008844833995681256, + 0.006807958008721471, + 0.009881000005407259, + 0.006558958004461601, + 0.00857025000732392, + 0.006477333008660935, + 0.009236875004717149, + 0.0071385409974027425, + 0.008819375012535602, + 0.006869165998068638, + 0.010801541997352615, + 0.00769904199114535, + 0.010260833994834684, + 0.008064041991019621, + 0.010461083002155647, + 0.010889083001529798, + 0.008470750006381422, + 0.01197070900525432, + 0.011343917009071447, + 0.00809441700403113 + ] + }, + { + "forward": 0.12084095799946226, + "prefill": 0.12542287500400562, + "decode_steps": [ + 0.012511459004599601, + 0.01027237500238698, + 0.010736708994954824, + 0.010452749993419275, + 0.010636749997502193, + 0.010334749997127801, + 0.01076433299749624, + 0.010627042007399723, + 0.011219250009162351, + 0.0100649169908138, + 0.010142874991288409, + 0.010016583997639827, + 0.009386292003910057, + 0.008591833000537008, + 0.007709582991083153, + 0.007864790997700766, + 0.007756457998766564, + 0.00798145800945349, + 0.00972416700096801, + 0.012098166989744641, + 0.009517416008748114, + 0.006657208999968134, + 0.006670665999990888, + 0.007897667004726827, + 0.006710750007187016, + 0.0067852500069420785, + 0.006730541994329542, + 0.006683665997115895, + 0.006751833003363572, + 0.0068037500022910535, + 0.006878457992570475, + 0.0068089160049567 + ] + }, + { + "forward": 0.1317202499922132, + "prefill": 0.13777383300475776, + "decode_steps": [ + 0.011755833009374328, + 0.011292625000351109, + 0.010905000002821907, + 0.01142358299694024, + 0.010634416990797035, + 0.013024332991335541, + 0.008935624995501712, + 0.008793000000878237, + 0.00845962499442976, + 0.008591332996729761, + 0.008113916992442682, + 0.008223249998991378, + 0.00878466600261163, + 0.00839645900123287, + 0.009103208998567425, + 0.007944417011458427, + 0.00892145799298305, + 0.008158250013366342, + 0.00859479200153146, + 0.007939457995234989, + 0.008304832997964695, + 0.007817958001396619, + 0.007913959008874372, + 0.006925875000888482, + 0.007268624991411343, + 0.0065247080055996776, + 0.006910333991982043, + 0.006935792000149377, + 0.0067385830043349415, + 0.007299832999706268, + 0.006551250000484288, + 0.007042500001261942 + ] + }, + { + "forward": 0.11334995900688227, + "prefill": 0.11768066699733026, + "decode_steps": [ + 0.007621083001140505, + 0.0065882089984370396, + 0.006942834006622434, + 0.006788667000364512, + 0.006851708996691741, + 0.007232041010865942, + 0.006459750002250075, + 0.007480000000214204, + 0.006715000010444783, + 0.007096417000866495, + 0.006939500002772547, + 0.006716624993714504, + 0.007879457989474759, + 0.006947125002625398, + 0.007992625003680587, + 0.0075387079996289685, + 0.008754832990234718, + 0.007762708992231637, + 0.00963570800377056, + 0.00980783300474286, + 0.010750500005087815, + 0.010862499999348074, + 0.011012125003617257, + 0.011508374998811632, + 0.009417624998604879, + 0.008893417005310766, + 0.008166207990143448, + 0.009080709001864307, + 0.009257708996301517, + 0.010280292000970803, + 0.00987462500052061, + 0.00791233299241867 + ] + }, + { + "forward": 0.13010125000437256, + "prefill": 0.13783579198934603, + "decode_steps": [ + 0.008518666989402846, + 0.007299208999029361, + 0.008412750001298264, + 0.00707679201150313, + 0.008191041008103639, + 0.007353084001806565, + 0.007736749990726821, + 0.00902466700063087, + 0.007951332998345606, + 0.007350500003667548, + 0.007717458996921778, + 0.007491249998565763, + 0.007729458011453971, + 0.007549874993856065, + 0.007787583002937026, + 0.007876542003941722, + 0.008482707999064587, + 0.010989250004058704, + 0.011673667002469301, + 0.01124658300250303, + 0.011242042004596442, + 0.011431959006586112, + 0.01018241599376779, + 0.04188662499655038, + 0.015379666991066188, + 0.012319375004153699, + 0.009875040996121243, + 0.009853416006080806, + 0.011117875008494593, + 0.009534417011309415, + 0.009972750005545095, + 0.009366167010739446 + ] + }, + { + "forward": 0.16787212500639725, + "prefill": 0.17855754200718366, + "decode_steps": [ + 0.01286400000390131, + 0.013670917003764771, + 0.00991612499637995, + 0.011638249998213723, + 0.012420750004821457, + 0.009889791996101849, + 0.011672666005324572, + 0.01238420800655149, + 0.011851999995997176, + 0.009707832999993116, + 0.011013041003025137, + 0.010269999998854473, + 0.009868708002613857, + 0.009278375000576489, + 0.009010457986732945, + 0.009469291995628737, + 0.008898250001948327, + 0.008855207997839898, + 0.009227167000062764, + 0.009456208004849032, + 0.009690874998341314, + 0.008980167011031881, + 0.009244999993825331, + 0.009325916995294392, + 0.008640208005090244, + 0.009465166993322782, + 0.009137916989857331, + 0.00968579199980013, + 0.008644000001368113, + 0.008155624993378296, + 0.00909233299898915, + 0.00842033298977185 + ] + }, + { + "forward": 0.10332541700336151, + "prefill": 0.10785804099577945, + "decode_steps": [ + 0.006675166994682513, + 0.006416667005396448, + 0.006557208005688153, + 0.006520916998852044, + 0.006044458001269959, + 0.007158749998779967, + 0.006171959001221694, + 0.006328292001853697, + 0.006696041004033759, + 0.006591083001694642, + 0.007810082999640144, + 0.006603583009564318, + 0.007785667010466568, + 0.007074417007970624, + 0.008421165999607183, + 0.007491791999200359, + 0.009743625007104129, + 0.008916958002373576, + 0.009927624996635132, + 0.007949792008730583, + 0.008368333001271822, + 0.010545583994826302, + 0.008441250000032596, + 0.007843415995012037, + 0.008347875002073124, + 0.008875084007740952, + 0.008367083995835856, + 0.008154333001584746, + 0.008126250002533197, + 0.007731290999799967, + 0.007955250010127202, + 0.007895874994574115 + ] + }, + { + "forward": 0.10915045799629297, + "prefill": 0.12059379200218245, + "decode_steps": [ + 0.008448167005553842, + 0.007306790997972712, + 0.008188874999177642, + 0.0069778329925611615, + 0.007502499996917322, + 0.007143458002246916, + 0.013403000004473142, + 0.009286540996981785, + 0.010440500002005138, + 0.008703999992576428, + 0.008369457995286211, + 0.008395374999963678, + 0.007806749999872409, + 0.008498582988977432, + 0.007523208987549879, + 0.007951708001201041, + 0.007534250005846843, + 0.007983750008861534, + 0.007564540996099822, + 0.008197499992093071, + 0.00743995800439734, + 0.008929499992518686, + 0.007766416005324572, + 0.008054292004089803, + 0.007789208000758663, + 0.00748145800025668, + 0.008274916996015236, + 0.007468125011655502, + 0.007702957998844795, + 0.007739583001239225, + 0.010447583001223393, + 0.007422040987876244 + ] + }, + { + "forward": 0.12272995800594799, + "prefill": 0.11998274999496061, + "decode_steps": [ + 0.011272167001152411, + 0.010837916997843422, + 0.010545041994191706, + 0.010649625008227304, + 0.010749875000328757, + 0.01027366699418053, + 0.008141333004459739, + 0.007425082992995158, + 0.007461500004865229, + 0.00717100000474602, + 0.007102332994691096, + 0.007395832988549955, + 0.006887499999720603, + 0.007134500003303401, + 0.007137374996091239, + 0.0075026660051662475, + 0.0071879169991007075, + 0.007281708996742964, + 0.007148083008360118, + 0.007236290999571793, + 0.007283832994289696, + 0.007800874998793006, + 0.0072103749989764765, + 0.007531208000727929, + 0.0072399170021526515, + 0.00728520899428986, + 0.006788208003854379, + 0.006654334007180296, + 0.007088792001013644, + 0.006668457994237542, + 0.00677154200093355, + 0.006818374997237697 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 121.78545800270513, + "p95": 154.5212248478492, + "min": 103.32541700336151, + "max": 167.87212500639725, + "iqr": 21.115166746312752, + "mean": 124.29983330075629, + "cv": 0.15475263412302448 + }, + "prefill_ms": { + "count": 10, + "median": 123.00833350309404, + "p95": 167.01249184770856, + "min": 107.85804099577945, + "max": 178.55754200718366, + "iqr": 19.564114496461116, + "mean": 131.10681669786572, + "cv": 0.16381259097203676 + }, + "decode_ms_per_step": { + "count": 320, + "median": 8.308166499773506, + "p95": 11.857935446460035, + "min": 6.044458001269959, + "max": 41.88662499655038, + "iqr": 2.51766625297023, + "mean": 8.772722774983777, + "cv": 0.28594550080460346 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 946.5813433077129, + "p95": 1026.5236137635711, + "min": 776.6497390710288, + "max": 1034.189350325139, + "iqr": 127.21850126250399, + "mean": 920.2489542790884, + "cv": 0.09803433122017235 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 645.741619803312, + "p95": 706.1352964977584, + "min": 513.6394096330733, + "max": 720.3254420209344, + "iqr": 107.63877417857896, + "mean": 628.8402164987097, + "cv": 0.11004300869353208 + }, + "memory": { + "peak_extra_bytes": 1074003968, + "method": "mps_driver_allocated_delta" + } + }, + { + "backend": "reference", + "batch": 8, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.5254731669992907, + "prefill": 0.5114068749971921, + "decode_steps": [ + 0.02424608300498221, + 0.02062195900361985, + 0.017521541012683883, + 0.016673042002366856, + 0.013379791998886503, + 0.013627834006911144, + 0.013024500003666617, + 0.011745957992388867, + 0.011281792001682334, + 0.01371487500728108, + 0.011906583997188136, + 0.011303833991405554, + 0.011489541007904336, + 0.011183124996023253, + 0.010995624994393438, + 0.0107819590048166, + 0.010838833986781538, + 0.010197584007983096, + 0.010242917007417418, + 0.009355916001368314, + 0.009488707990385592, + 0.012093458004528657, + 0.009662957992986776, + 0.01044341700617224, + 0.011032624999643303, + 0.01525170799868647, + 0.01757816699682735, + 0.010595375002594665, + 0.010066125003504567, + 0.012161791004473343, + 0.009702416995423846, + 0.00984112499281764 + ] + }, + { + "forward": 0.6392262919980567, + "prefill": 0.684493458000361, + "decode_steps": [ + 0.030319832992972806, + 0.02469849999761209, + 0.016867999991518445, + 0.015972917011822574, + 0.014634959006798454, + 0.015164707991061732, + 0.014125875008176081, + 0.012900624991743825, + 0.011671874992316589, + 0.011262375002843328, + 0.012710416005575098, + 0.01183595901238732, + 0.011044749990105629, + 0.011297542005195282, + 0.0121408330014674, + 0.01166362498770468, + 0.011099707990069874, + 0.01180212500912603, + 0.012634374987101182, + 0.011410624996642582, + 0.013219833999755792, + 0.012708583992207423, + 0.03390583299915306, + 0.012665999995078892, + 0.011100542003987357, + 0.011203207992366515, + 0.012415832999977283, + 0.011070249995100312, + 0.01033816600102, + 0.0108065410022391, + 0.010065166003187187, + 0.0105723750020843 + ] + }, + { + "forward": 0.5237182919954648, + "prefill": 0.5286792080005398, + "decode_steps": [ + 0.026483249996090308, + 0.023116665994166397, + 0.0158545000012964, + 0.014425542001845315, + 0.01606504200026393, + 0.01368062499386724, + 0.013393332992563955, + 0.012721999999484979, + 0.012678458995651454, + 0.012401084008160979, + 0.014122292006504722, + 0.012188250009785406, + 0.011712167004588991, + 0.011943249992327765, + 0.011936416005482897, + 0.011584625011892058, + 0.01082995800243225, + 0.013239542007795535, + 0.016793375005363487, + 0.011169332996360026, + 0.01105029099562671, + 0.010091749994899146, + 0.009769207987119444, + 0.009894749993691221, + 0.009514541990938596, + 0.00967274999129586, + 0.009567665998474695, + 0.010770375010906719, + 0.00928149999526795, + 0.009788499999558553, + 0.011108125007012859, + 0.009144832991296425 + ] + }, + { + "forward": 0.5477907080057776, + "prefill": 0.5160780000005616, + "decode_steps": [ + 0.028074834001017734, + 0.018690249999053776, + 0.015509500008192845, + 0.013827833012328483, + 0.013438542009680532, + 0.012282292009331286, + 0.012105750007322058, + 0.011274000004050322, + 0.011108708989922889, + 0.010759583004983142, + 0.01207129099930171, + 0.01117795800382737, + 0.011227625000174157, + 0.011047375010093674, + 0.010974332995829172, + 0.011582374994759448, + 0.011169124991283752, + 0.010541584008024074, + 0.010888500008150004, + 0.01089687499916181, + 0.010621416004141793, + 0.011564040993107483, + 0.011690917002852075, + 0.010393542004749179, + 0.01047741700313054, + 0.010072542005218565, + 0.009708292011055164, + 0.009644250007113442, + 0.009548333997372538, + 0.010168374996283092, + 0.010841082999831997, + 0.009834374999627471 + ] + }, + { + "forward": 0.5405326670006616, + "prefill": 0.692724584005191, + "decode_steps": [ + 0.030924209000659175, + 0.023698250006418675, + 0.01891179199446924, + 0.016152041993336752, + 0.015226584000629373, + 0.01509733300190419, + 0.014445875000092201, + 0.015216291008982807, + 0.01598125000600703, + 0.015486250005778857, + 0.013486249998095445, + 0.014216750001651235, + 0.011778500003856607, + 0.012544124998385087, + 0.013330292000318877, + 0.015681291988585144, + 0.01459529199928511, + 0.0138019589940086, + 0.013984292003442533, + 0.011816915997769684, + 0.0119529160001548, + 0.013852582997060381, + 0.013866374996723607, + 0.013459041991154663, + 0.013796540995826945, + 0.012111709002056159, + 0.01166829199064523, + 0.012376375001622364, + 0.010156040996662341, + 0.01084429099864792, + 0.011630292006884702, + 0.00971683299576398 + ] + }, + { + "forward": 0.5744879999983823, + "prefill": 0.603248250001343, + "decode_steps": [ + 0.01860154198948294, + 0.016642958013108, + 0.01578758300456684, + 0.01455391600029543, + 0.013898625009460375, + 0.013921874997322448, + 0.015326458000345156, + 0.013798290994600393, + 0.013710291997995228, + 0.013399249990470707, + 0.01299704099074006, + 0.012052374993800186, + 0.011924042002647184, + 0.013230459007900208, + 0.01169145799940452, + 0.012361499990220182, + 0.012326832991675474, + 0.010506332997465506, + 0.010432542010676116, + 0.010371917000156827, + 0.01049724999757018, + 0.00981312501244247, + 0.01012804199126549, + 0.01029237499460578, + 0.01078304200200364, + 0.011812541997642256, + 0.010571916005574167, + 0.009967749996576458, + 0.00972600000386592, + 0.01060808400507085, + 0.009614916009013541, + 0.010566917000687681 + ] + }, + { + "forward": 0.5551867089961888, + "prefill": 0.5119952079985524, + "decode_steps": [ + 0.029810999985784292, + 0.017878375001600944, + 0.016451291012344882, + 0.014773375005461276, + 0.013217583997175097, + 0.013160707996576093, + 0.013098417010041885, + 0.0119499580032425, + 0.012229083004058339, + 0.01169704100175295, + 0.011337709001963958, + 0.011253417003899813, + 0.011293458010186441, + 0.011293374991510063, + 0.011407958008931018, + 0.010884207993512973, + 0.009916541996062733, + 0.01087820899556391, + 0.011097958995378576, + 0.009477708008489572, + 0.011022542006685399, + 0.011193375001312234, + 0.009227041999110952, + 0.008605374998296611, + 0.010031458004959859, + 0.00912620899907779, + 0.011438290996011347, + 0.010128167006769218, + 0.00964899999962654, + 0.009579625009791926, + 0.009113583000726067, + 0.010208958003204316 + ] + }, + { + "forward": 0.49898954200034495, + "prefill": 0.558283875012421, + "decode_steps": [ + 0.01690620800945908, + 0.024446208000881597, + 0.0366317089938093, + 0.03508341699489392, + 0.026298750002752058, + 0.033639625005889684, + 0.02374554100970272, + 0.017675209004664794, + 0.01696783299848903, + 0.015788415999850258, + 0.01568783300172072, + 0.016221541998675093, + 0.015417749993503094, + 0.016858375005540438, + 0.01838762500847224, + 0.01860399999713991, + 0.019096917007118464, + 0.017194333995576017, + 0.017625124994083308, + 0.016552625005715527, + 0.01742745800584089, + 0.01696300000185147, + 0.016277041999273933, + 0.01867629200569354, + 0.01771570798882749, + 0.02423291699960828, + 0.02291733300080523, + 0.016931457997998223, + 0.016636665997793898, + 0.014448792004259303, + 0.015290541996364482, + 0.015096000002813525 + ] + }, + { + "forward": 0.8569894169922918, + "prefill": 1.0190868329955265, + "decode_steps": [ + 0.03431762500258628, + 0.04417729200213216, + 0.043967667006654665, + 0.03173058299580589, + 0.04086916599771939, + 0.0523109580099117, + 0.03877387499960605, + 0.032660667013260536, + 0.021713333000661805, + 0.025482709010248072, + 0.025928000002750196, + 0.025552749997586943, + 0.019644792002509348, + 0.0252261669957079, + 0.022472540993476287, + 0.020312042004661635, + 0.020042624993948266, + 0.018055707987514324, + 0.01729245799651835, + 0.017370333007420413, + 0.017355416988721117, + 0.01819508300104644, + 0.01718958400306292, + 0.016750916998717003, + 0.0171514169924194, + 0.016079749999335036, + 0.017694958005449735, + 0.01782866701250896, + 0.015936665993649513, + 0.01657720800722018, + 0.015467416000319645, + 0.015560707994154654 + ] + }, + { + "forward": 0.5111008329986362, + "prefill": 0.5272858329990413, + "decode_steps": [ + 0.027872250007931143, + 0.030165416988893412, + 0.0447684999962803, + 0.03761258399754297, + 0.029126042005373165, + 0.019094749994110316, + 0.016018416004953906, + 0.02056470800016541, + 0.028906291001476347, + 0.017021042003761977, + 0.015492040998651646, + 0.016580166993662715, + 0.015392583009088412, + 0.016258541989373043, + 0.0166583329992136, + 0.01579791700351052, + 0.016393584010074846, + 0.015626332999090664, + 0.017200375004904345, + 0.016352959006326273, + 0.01673245801066514, + 0.015941250007017516, + 0.01618445799977053, + 0.016004916993551888, + 0.01576408300024923, + 0.016557625000132248, + 0.01676404199679382, + 0.015752000006614253, + 0.016398207997553982, + 0.0162421659915708, + 0.015866625006310642, + 0.01675920799607411 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 544.1616875032196, + "p95": 758.9960107448858, + "min": 498.98954200034495, + "max": 856.9894169922918, + "iqr": 45.505666501412634, + "mean": 577.3495626985095, + "cv": 0.18335091631816086 + }, + "prefill_ms": { + "count": 10, + "median": 543.4815415064804, + "p95": 872.2238209498752, + "min": 511.40687499719206, + "max": 1019.0868329955265, + "iqr": 145.30219775042497, + "mean": 615.328212401073, + "cv": 0.25626245515667645 + }, + "decode_ms_per_step": { + "count": 320, + "median": 13.38656249572523, + "p95": 30.17313778909738, + "min": 8.605374998296611, + "max": 52.3109580099117, + "iqr": 5.637802016281057, + "mean": 15.34920728790894, + "cv": 0.43863712677954353 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 602.7016340337739, + "p95": 668.7871609848806, + "min": 328.33600713973414, + "max": 669.4018005002888, + "iqr": 204.70112446703763, + "mean": 552.8604205849153, + "cv": 0.2285433950023357 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 241.9136622052519, + "p95": 285.5202853389701, + "min": 142.31900586581776, + "max": 286.21701566504004, + "iqr": 57.018303877768915, + "mean": 240.9703992747347, + "cv": 0.186605204701707 + }, + "memory": { + "peak_extra_bytes": 2684616704, + "method": "mps_driver_allocated_delta" + } + } + ] +} diff --git a/experiments/records/attention/20261022T034036Z_attention_mps_float32_0502118.json b/experiments/records/attention/20261022T034036Z_attention_mps_float32_0502118.json new file mode 100644 index 0000000..af731d6 --- /dev/null +++ b/experiments/records/attention/20261022T034036Z_attention_mps_float32_0502118.json @@ -0,0 +1,14061 @@ +{ + "record_version": 1, + "created_utc": "20261022T034036Z", + "experiment": "attention_backends", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "mps", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "0502118a1a0af31c8480f6b499d9347916330a79", + "branch": "experiments/attention-kv-audit", + "dirty": false + }, + "command": "python scripts/benchmark_attention.py --device mps --dtype float32" + }, + "settings": { + "dtype": "float32", + "position": "learned", + "backends": [ + "auto", + "sdpa", + "reference" + ], + "seq_lens": [ + 128, + 256, + 512, + 1024, + 2048 + ], + "batch_sizes": [ + 1, + 4, + 8 + ], + "decode_tokens": 32, + "warmup": 3, + "repeats": 10, + "seed": 0, + "weights": "seeded random initialization (timing does not depend on trained values)", + "model_config": { + "vocab_size": 256, + "context_length": 2080, + "embedding_dim": 192, + "num_layers": 4, + "num_heads": 6, + "dropout": 0.0, + "positional_encoding": "learned", + "norm": "rmsnorm", + "activation": "swiglu", + "rope_theta": 10000.0 + }, + "inference_mode": true + }, + "results": [ + { + "backend": "auto", + "batch": 1, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.0018430000054650009 + }, + { + "forward": 0.001908749996800907 + }, + { + "forward": 0.0018293339962838218 + }, + { + "forward": 0.0018589160026749596 + }, + { + "forward": 0.0019374579860595986 + }, + { + "forward": 0.0019336249970365316 + }, + { + "forward": 0.0018779579986585304 + }, + { + "forward": 0.0020115000079385936 + }, + { + "forward": 0.001874292007414624 + }, + { + "forward": 0.0018052079976769164 + } + ], + "forward_ms": { + "count": 10, + "median": 1.8761250030365773, + "p95": 1.9781810980930459, + "min": 1.8052079976769164, + "max": 2.0115000079385936, + "iqr": 0.08042724221013486, + "mean": 1.8880040996009484, + "cv": 0.032394710185952455 + }, + "memory": { + "peak_extra_bytes": 1475584, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "sdpa", + "batch": 1, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.0015984159981599078, + "prefill": 0.001623457996174693, + "decode_steps": [ + 0.0016248750034719706, + 0.0016252079949481413, + 0.0016223749989876524, + 0.0016302910080412403, + 0.0014875829947413877, + 0.001568042003782466, + 0.0015616250020684674, + 0.0016521250072401017, + 0.0016650830075377598, + 0.0015380420081783086, + 0.0016582499956712127, + 0.0016423749912064523, + 0.0016255419905064628, + 0.0015073329996084794, + 0.0015902089944574982, + 0.0015113749977899715, + 0.0016234169888775796, + 0.0016359169967472553, + 0.0015688329876866192, + 0.0018313750042580068, + 0.0015194159932434559, + 0.0017030000017257407, + 0.0016420409956481308, + 0.0015036249969853088, + 0.001680624991422519, + 0.0016397500003222376, + 0.00166920899937395, + 0.0015243340021697804, + 0.0016199590027099475, + 0.0027642499917419627, + 0.0016303330048685893, + 0.0016123749956022948 + ] + }, + { + "forward": 0.0017815829924074933, + "prefill": 0.001615250002942048, + "decode_steps": [ + 0.0018877090042224154, + 0.0016447910020360723, + 0.0016408329975092784, + 0.0015118330047698691, + 0.0016535000031581149, + 0.001638833011384122, + 0.0015231249999487773, + 0.001618916998268105, + 0.0016481250058859587, + 0.0016487079992657527, + 0.001507082997704856, + 0.0017020000086631626, + 0.001619082991965115, + 0.0016869999963091686, + 0.0015280830120900646, + 0.0016176249919226393, + 0.0016776249976828694, + 0.0016031250124797225, + 0.0016320830036420375, + 0.0015369159955298528, + 0.0016384170012315735, + 0.0016773340030340478, + 0.001680999994277954, + 0.0015391659981105477, + 0.0016101249930216, + 0.001531082991277799, + 0.001709125004708767, + 0.0016137080092448741, + 0.001593125009094365, + 0.001582917000632733, + 0.0016715829988243058, + 0.001639790993067436 + ] + }, + { + "forward": 0.0015922919992590323, + "prefill": 0.0016246250015683472, + "decode_steps": [ + 0.001698457999737002, + 0.0015817920066183433, + 0.0015064159961184487, + 0.0021932079980615526, + 0.0016475000011269003, + 0.002082666993374005, + 0.001665749994572252, + 0.0016566249978495762, + 0.0016914169973460957, + 0.0015376669907709584, + 0.0015869999915594235, + 0.0016626669967081398, + 0.0016649169992888346, + 0.0016860829928191379, + 0.0015400419943034649, + 0.0016381669993279502, + 0.0014977090031607077, + 0.0023331669945036992, + 0.0015898330020718277, + 0.0015490829973714426, + 0.0016425830108346418, + 0.0016751249931985512, + 0.0016486249951412901, + 0.0015124579949770123, + 0.001676792002399452, + 0.0016650409961584955, + 0.001683583002886735, + 0.001615625005797483, + 0.0015048749919515103, + 0.0015845000016270205, + 0.0016210419998969883, + 0.001637374996789731 + ] + }, + { + "forward": 0.0016443339991383255, + "prefill": 0.001741166997817345, + "decode_steps": [ + 0.001693583995802328, + 0.001612958003534004, + 0.0016845830105012283, + 0.0016663330025039613, + 0.0016342499875463545, + 0.0015206670068437234, + 0.00148974999319762, + 0.001655291998758912, + 0.001553167006932199, + 0.0015589170070597902, + 0.0015673749876441434, + 0.001520790989161469, + 0.0016270000051008537, + 0.0016604580014245585, + 0.001646541990339756, + 0.0015286669950000942, + 0.0015843750006752089, + 0.0016450419934699312, + 0.0016182499966816977, + 0.0015819999971427023, + 0.001495040996815078, + 0.0016454170108772814, + 0.001636000000871718, + 0.0016407079965574667, + 0.0017399589996784925, + 0.0015239159984048456, + 0.001585584002896212, + 0.001655250001931563, + 0.0016557089984416962, + 0.0015263329987647012, + 0.0014974160003475845, + 0.0015054170071380213 + ] + }, + { + "forward": 0.0016967080009635538, + "prefill": 0.0016519170021638274, + "decode_steps": [ + 0.0017842079978436232, + 0.0017730419931467623, + 0.001731958007439971, + 0.0018094580009346828, + 0.0017407499981345609, + 0.001760333005222492, + 0.0017090000037569553, + 0.0017187080084113404, + 0.0017337500030407682, + 0.0015119999879971147, + 0.0015026670007500798, + 0.0017315410077571869, + 0.001671624995651655, + 0.0016293330118060112, + 0.001687666997895576, + 0.0016558750066906214, + 0.0017031250026775524, + 0.0015610419941367581, + 0.001512999995611608, + 0.0016603330004727468, + 0.0016953330050455406, + 0.0016727079928386956, + 0.0016576670022914186, + 0.0015303749969461933, + 0.0015575419965898618, + 0.0015476660046260804, + 0.00167350000992883, + 0.0015318750083679333, + 0.0016136250051204115, + 0.0016626249998807907, + 0.0016517500043846667, + 0.0016382499889004976 + ] + }, + { + "forward": 0.0018341669929213822, + "prefill": 0.0016744169988669455, + "decode_steps": [ + 0.0017085420113289729, + 0.0015339169913204387, + 0.0016170000017154962, + 0.0016153329925145954, + 0.0016975830076262355, + 0.0015414160006912425, + 0.0015266669943230227, + 0.0016795419942354783, + 0.0016872909909579903, + 0.0016892499988898635, + 0.001570124994032085, + 0.0015342080005211756, + 0.001560000004246831, + 0.001554125003167428, + 0.0015607499954057857, + 0.0015345830033766106, + 0.0017989170009968802, + 0.0016351669910363853, + 0.0016286669997498393, + 0.0016427910013590008, + 0.0016646250005578622, + 0.001556834002258256, + 0.0016537500050617382, + 0.0016233749920502305, + 0.001522958991699852, + 0.0018205420055892318, + 0.0016416250000474975, + 0.0016421250038547441, + 0.0016610419988865033, + 0.0015879999991739169, + 0.0016452080017188564, + 0.0015743330004625022 + ] + }, + { + "forward": 0.0016343330062227324, + "prefill": 0.0018150830001104623, + "decode_steps": [ + 0.0016775830008555204, + 0.0016011249972507358, + 0.0016173339972738177, + 0.001639249996514991, + 0.0016063750081229955, + 0.0016851669934112579, + 0.0016397079889429733, + 0.0015619579935446382, + 0.0016096249892143533, + 0.0016019999893615022, + 0.0015170830010902137, + 0.001739625004120171, + 0.0015182080096565187, + 0.0015618329925928265, + 0.0016789999936008826, + 0.001654124993365258, + 0.0015496670093853027, + 0.0015743330004625022, + 0.0015382919955300167, + 0.0021874999947613105, + 0.0016967499977909029, + 0.0017300000035902485, + 0.0016805830091470852, + 0.001655250001931563, + 0.0017622500017751008, + 0.0015252919984050095, + 0.0016152079915627837, + 0.0015369579923572019, + 0.0016528330015717074, + 0.0016352920065401122, + 0.0015244999958667904, + 0.0016358330030925572 + ] + }, + { + "forward": 0.0018122920009773225, + "prefill": 0.0018281669908901677, + "decode_steps": [ + 0.0017477090004831553, + 0.00165462501172442, + 0.0016864589997567236, + 0.0015402499993797392, + 0.0015825420123292133, + 0.0029393750010058284, + 0.0016982499946607277, + 0.0015431670035468414, + 0.0015135830035433173, + 0.001534875002107583, + 0.00157891699927859, + 0.001742833002936095, + 0.0015141249896259978, + 0.0015067089989315718, + 0.002228875004220754, + 0.0016429579991381615, + 0.0014950830081943423, + 0.001546665997011587, + 0.0015323330007959157, + 0.0015253750025294721, + 0.0016922080103540793, + 0.0016024579963413998, + 0.0016374159895349294, + 0.0017893750045914203, + 0.0017092500056605786, + 0.0016267499886453152, + 0.0015757080109324306, + 0.0014974579971749336, + 0.0016603330004727468, + 0.0016772920062066987, + 0.0015207079995889217, + 0.0015098330040927976 + ] + }, + { + "forward": 0.001658082997892052, + "prefill": 0.001673749997280538, + "decode_steps": [ + 0.001719958003377542, + 0.0015757919900352135, + 0.0015315420023398474, + 0.0017402079974999651, + 0.0016831250104587525, + 0.0016783750033937395, + 0.0017522499983897433, + 0.0017264169873669744, + 0.0016839999880176038, + 0.001712792000034824, + 0.001529749992187135, + 0.0015142089978326112, + 0.0016665839939378202, + 0.0016771249938756227, + 0.0016705419984646142, + 0.0015303330001188442, + 0.0016732919903006405, + 0.0031336669926531613, + 0.0015223749942379072, + 0.0015834590012673289, + 0.0016740829887567088, + 0.001679249995504506, + 0.001660416994127445, + 0.0015111250104382634, + 0.0015474169922526926, + 0.002203915995778516, + 0.0016805830091470852, + 0.0015001669962657616, + 0.0015904159954516217, + 0.001618458001757972, + 0.0015667080006096512, + 0.0016886669909581542 + ] + }, + { + "forward": 0.0017184579919558018, + "prefill": 0.002701042001717724, + "decode_steps": [ + 0.0016492499999003485, + 0.0015456249966518953, + 0.0014760419871890917, + 0.0015673750021960586, + 0.0016528749983990565, + 0.0016223750135395676, + 0.00150079200102482, + 0.0015175419976003468, + 0.0015597500023432076, + 0.0016359169967472553, + 0.001531292000436224, + 0.001657959001022391, + 0.0016617500077700242, + 0.0016577909991610795, + 0.0016202080005314201, + 0.0014978749968577176, + 0.002708459011046216, + 0.0016592089959885925, + 0.0015205420058919117, + 0.0014263749908423051, + 0.001599249997525476, + 0.001610334002180025, + 0.0017674580012680963, + 0.0015555840072920546, + 0.0015588340029353276, + 0.002813625003909692, + 0.001660083988099359, + 0.0017116660019382834, + 0.0017030830058502033, + 0.0016907089884625748, + 0.0016877500020200387, + 0.0015446670004166663 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 1.6773954994278029, + "p95": 1.8243232465465553, + "min": 1.5922919992590323, + "max": 1.8341669929213822, + "iqr": 0.1289684878429398, + "mean": 1.6970665979897603, + "cv": 0.05145226092562487 + }, + "prefill_ms": { + "count": 10, + "median": 1.6740834980737418, + "p95": 2.3082482468453227, + "min": 1.615250002942048, + "max": 2.701042001717724, + "iqr": 0.1651559978199657, + "mean": 1.7948875989532098, + "cv": 0.18250860805426586 + }, + "decode_ms_per_step": { + "count": 320, + "median": 1.6352294987882487, + "p95": 1.7898521044116933, + "min": 1.4263749908423051, + "max": 3.1336669926531613, + "iqr": 0.12328124284977093, + "mean": 1.649135674460922, + "cv": 0.11542917484216506 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 605.4871910559302, + "p95": 621.4422511838279, + "min": 590.1119161073269, + "max": 625.5283884473281, + "iqr": 15.042699728384946, + "mean": 606.5446734459634, + "cv": 0.017493994832661376 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 587.289433471372, + "p95": 601.449689293801, + "min": 568.6554721568847, + "max": 604.9387428099142, + "iqr": 14.254140026007349, + "mean": 586.6305923445007, + "cv": 0.01935399946292821 + }, + "memory": { + "peak_extra_bytes": 1475584, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "reference", + "batch": 1, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.005262500009848736, + "prefill": 0.0022223330015549436, + "decode_steps": [ + 0.004995916999178007, + 0.004438250005478039, + 0.004303417008486576, + 0.004270500008715317, + 0.004741958997328766, + 0.005232915995293297, + 0.004994542003259994, + 0.004717333009466529, + 0.005173833007575013, + 0.004473458000575192, + 0.0041221249994123355, + 0.004080750004504807, + 0.004071875009685755, + 0.0041265420004492626, + 0.0037381659931270406, + 0.004154667010880075, + 0.004127957989112474, + 0.0041684589959913865, + 0.004102957987925038, + 0.0038034579920349643, + 0.003696749990922399, + 0.004255833002389409, + 0.0038227080076467246, + 0.004049958006362431, + 0.004119500008528121, + 0.004251292004482821, + 0.005126707997987978, + 0.0037806669861311093, + 0.004159666990744881, + 0.0038209169870242476, + 0.004014375008409843, + 0.00409729199600406 + ] + }, + { + "forward": 0.005401166999945417, + "prefill": 0.002159374998882413, + "decode_steps": [ + 0.004391042006318457, + 0.004086250002728775, + 0.003600458992877975, + 0.004134124988922849, + 0.004217291003442369, + 0.004235000000335276, + 0.004168708008364774, + 0.004136415998800658, + 0.0041225000022677705, + 0.004101541999261826, + 0.004068791997269727, + 0.004138459000387229, + 0.003572917004930787, + 0.0041238330013584346, + 0.00408512499416247, + 0.004058332997374237, + 0.004103290993953124, + 0.004018709005322307, + 0.003992375000962056, + 0.0040825000032782555, + 0.0035894579923478886, + 0.003949207995901816, + 0.004091750000952743, + 0.004069250004249625, + 0.00402287500037346, + 0.004039790990645997, + 0.004002208996098489, + 0.003982124995673075, + 0.004650709000998177, + 0.004290208002203144, + 0.004077666002558544, + 0.004124625003896654 + ] + }, + { + "forward": 0.004146584004047327, + "prefill": 0.0022231670009205118, + "decode_steps": [ + 0.00419224999495782, + 0.004079500009538606, + 0.0040501250041415915, + 0.004082667001057416, + 0.00413504199241288, + 0.004079749996890314, + 0.004808458994375542, + 0.006147791995317675, + 0.00411095900926739, + 0.003929167010937817, + 0.004046957998070866, + 0.0039657500019529834, + 0.0039842499973019585, + 0.003997833002358675, + 0.003917041001841426, + 0.004131624998990446, + 0.004066082998178899, + 0.0043440000008558854, + 0.004328082999563776, + 0.004100916994502768, + 0.004146332998061553, + 0.003521500009810552, + 0.003736582992132753, + 0.00408587499987334, + 0.003964457995607518, + 0.004031999997096136, + 0.004074875003425404, + 0.0040356250101467595, + 0.003914499990059994, + 0.004155041999183595, + 0.004032374999951571, + 0.005308708001393825 + ] + }, + { + "forward": 0.005405417003203183, + "prefill": 0.002519749992643483, + "decode_steps": [ + 0.004286374998628162, + 0.004215749999275431, + 0.004083916996023618, + 0.004211959007079713, + 0.004162249999353662, + 0.004013083002064377, + 0.004478541988646612, + 0.005123084003571421, + 0.004147291008848697, + 0.004132875008508563, + 0.004219499998725951, + 0.004079874997842126, + 0.004083666004589759, + 0.004136208997806534, + 0.0035119170061079785, + 0.004146666993619874, + 0.004046958987601101, + 0.004217874986352399, + 0.004102875012904406, + 0.004041332998895086, + 0.004153292000410147, + 0.004142792007769458, + 0.0036869999894406646, + 0.004042291009682231, + 0.004032915996504016, + 0.0041114999912679195, + 0.00398270899313502, + 0.004096625008969568, + 0.004033792007248849, + 0.004202834010357037, + 0.005623208009637892, + 0.004313625002396293 + ] + }, + { + "forward": 0.0050800830067601055, + "prefill": 0.0021893329976592213, + "decode_steps": [ + 0.004226584002026357, + 0.004167415987467393, + 0.0042139590077567846, + 0.004106583000975661, + 0.003495999990263954, + 0.00398437499825377, + 0.0040531670092605054, + 0.004062540989252739, + 0.003989666991401464, + 0.004074124997714534, + 0.0038292499957606196, + 0.004049208000651561, + 0.004194000008283183, + 0.0037073749990668148, + 0.00441466600750573, + 0.004246042008162476, + 0.004061542000272311, + 0.0040140830096788704, + 0.00399954199383501, + 0.004081417006091215, + 0.0036684160004369915, + 0.00406400000792928, + 0.0041960000089602545, + 0.0040374169911956415, + 0.004090708011062816, + 0.0039059170085238293, + 0.004159582997090183, + 0.0040242499962914735, + 0.004142417004914023, + 0.004049167007906362, + 0.0038764579949202016, + 0.004125083010876551 + ] + }, + { + "forward": 0.004878208012087271, + "prefill": 0.002356875003897585, + "decode_steps": [ + 0.004372124996734783, + 0.004233875006320886, + 0.003603916004067287, + 0.004188291990431026, + 0.00400941600673832, + 0.0037800830032210797, + 0.004725500009953976, + 0.0043929160019615665, + 0.004208375001326203, + 0.003716209001140669, + 0.004187792001175694, + 0.004207500009215437, + 0.00415954198979307, + 0.004145083003095351, + 0.003993583013652824, + 0.004013833007775247, + 0.004033999997773208, + 0.004107250002562068, + 0.0041110829915851355, + 0.004112541995709762, + 0.004071125003974885, + 0.004141999990679324, + 0.0038716669951099902, + 0.004449584012036212, + 0.0042167920037172735, + 0.004388166998978704, + 0.00414587500563357, + 0.004122582991840318, + 0.004139708005823195, + 0.003941500006476417, + 0.004183209006441757, + 0.004037749997223727 + ] + }, + { + "forward": 0.00526629199157469, + "prefill": 0.0023855420004110783, + "decode_steps": [ + 0.004219499998725951, + 0.00416550000954885, + 0.004134415998123586, + 0.004484000004595146, + 0.004184874997008592, + 0.004090709000593051, + 0.004038458995637484, + 0.00370324999676086, + 0.00401225000678096, + 0.003974999999627471, + 0.0041129580058623105, + 0.004178834002232179, + 0.0043575839954428375, + 0.004317500002798624, + 0.004217624999000691, + 0.004984791012248024, + 0.003484207991277799, + 0.004138165997574106, + 0.004047249996801838, + 0.004154250011197291, + 0.0040206669946201146, + 0.00347466699895449, + 0.004029124989756383, + 0.0034736669913399965, + 0.004053666998515837, + 0.003475999998045154, + 0.0040907079965109006, + 0.0035391659912420437, + 0.00402645900612697, + 0.0040298339881701395, + 0.004044207991682924, + 0.0040282499976456165 + ] + }, + { + "forward": 0.005000457997084595, + "prefill": 0.0021368330053519458, + "decode_steps": [ + 0.003903374992660247, + 0.004182958000455983, + 0.003525374995660968, + 0.00408400000014808, + 0.004123916995013133, + 0.004165292004472576, + 0.003956249987822957, + 0.004102667007828131, + 0.004158625000854954, + 0.004202291995170526, + 0.0036518330016406253, + 0.004781209005159326, + 0.004348542002844624, + 0.00523004200658761, + 0.0043619170028250664, + 0.004137665993766859, + 0.004665000000386499, + 0.004289415999664925, + 0.00387766701169312, + 0.004546416996163316, + 0.004492874999414198, + 0.00375558300584089, + 0.0038337080040946603, + 0.0046072080003796145, + 0.004991207999410108, + 0.004111624992219731, + 0.004276291991118342, + 0.004260083005647175, + 0.003934084001230076, + 0.0041417500033276156, + 0.004256875006831251, + 0.0040650000009918585 + ] + }, + { + "forward": 0.004612207994796336, + "prefill": 0.0020654169929912314, + "decode_steps": [ + 0.004260874993633479, + 0.004094042000360787, + 0.0036657089949585497, + 0.0040733330097282305, + 0.00420287498855032, + 0.005041041993536055, + 0.004118333992664702, + 0.004214166998281144, + 0.004254583007423207, + 0.0054313340078806505, + 0.0037437499995576218, + 0.00406174999079667, + 0.004052292002597824, + 0.003867459003231488, + 0.003946500000893138, + 0.0041705000039655715, + 0.004049458002555184, + 0.003736624988960102, + 0.003714291990036145, + 0.004024250010843389, + 0.004059040991705842, + 0.0036813750048168004, + 0.004044166998937726, + 0.004115749994525686, + 0.0040622910019010305, + 0.0036075840034754947, + 0.0039770000003045425, + 0.004073624993907288, + 0.00404812500346452, + 0.0036994999973103404, + 0.00415841699577868, + 0.0041072919993894175 + ] + }, + { + "forward": 0.0054355000029318035, + "prefill": 0.002188333004596643, + "decode_steps": [ + 0.0039758749917382374, + 0.004406374995596707, + 0.004757999995490536, + 0.004165958001976833, + 0.004230124992318451, + 0.010077958009787835, + 0.009518292004941031, + 0.006884457994601689, + 0.004090416987310164, + 0.005250457994407043, + 0.006767124999896623, + 0.004695333002018742, + 0.004654916003346443, + 0.00430925001273863, + 0.003926624995074235, + 0.004423042002599686, + 0.004095333002624102, + 0.003510124995955266, + 0.00420474998827558, + 0.0039223329949891195, + 0.004499292001128197, + 0.0038882079970790073, + 0.0038925410044612363, + 0.004160416996455751, + 0.004174707995844074, + 0.0035825829982059076, + 0.003688041993882507, + 0.0040537500026402995, + 0.004175540991127491, + 0.004275584011338651, + 0.0035410410055192187, + 0.003981957997893915 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 5.171291508304421, + "p95": 5.421962653053924, + "min": 4.146584004047327, + "max": 5.4355000029318035, + "iqr": 0.45867773951613344, + "mean": 5.048841702227946, + "cv": 0.08168757555961129 + }, + "prefill_ms": { + "count": 10, + "median": 2.2058329996070825, + "p95": 2.459356396138901, + "min": 2.0654169929912314, + "max": 2.519749992643483, + "iqr": 0.1568335028423462, + "mean": 2.2446957998909056, + "cv": 0.06056414624378292 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.1027710103662685, + "p95": 4.994610753055895, + "min": 3.4736669913399965, + "max": 10.077958009787835, + "iqr": 0.19815699852188118, + "mean": 4.189071496921315, + "cv": 0.14357057531608364 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 241.04498831840323, + "p95": 247.4901430985399, + "min": 213.6460927523011, + "max": 247.5103112509411, + "iqr": 8.008837717982857, + "mean": 239.1291727428752, + "cv": 0.042223684294837666 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 236.95612195396643, + "p95": 243.20176555923692, + "min": 210.5696128280803, + "max": 243.34546902191929, + "iqr": 7.982083773128409, + "mean": 235.17672071188434, + "cv": 0.041520565447561955 + }, + "memory": { + "peak_extra_bytes": 1475584, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "auto", + "batch": 1, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.0032100420066853985 + }, + { + "forward": 0.0026436670013936237 + }, + { + "forward": 0.00317091699980665 + }, + { + "forward": 0.0024509169888915494 + }, + { + "forward": 0.002936958000645973 + }, + { + "forward": 0.0027982500032521784 + }, + { + "forward": 0.0024937499983934686 + }, + { + "forward": 0.002867667004466057 + }, + { + "forward": 0.0027345830021658912 + }, + { + "forward": 0.0025272920029237866 + } + ], + "forward_ms": { + "count": 10, + "median": 2.766416502709035, + "p95": 3.1924357535899617, + "min": 2.4509169888915494, + "max": 3.2100420066853985, + "iqr": 0.3632494990597479, + "mean": 2.7834043008624576, + "cv": 0.09618214798740743 + }, + "memory": { + "peak_extra_bytes": 2951168, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "sdpa", + "batch": 1, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.0023356669989880174, + "prefill": 0.0022882080083945766, + "decode_steps": [ + 0.0017201669979840517, + 0.0022263340069912374, + 0.0016549170104553923, + 0.0016107919946080074, + 0.0020048330043209717, + 0.0017964579892577603, + 0.0017351249989587814, + 0.0017174590029753745, + 0.0015790000034030527, + 0.0017689590022200719, + 0.0017692080000415444, + 0.0017136249953182414, + 0.0017027080029947683, + 0.00171050000062678, + 0.003597500006435439, + 0.005955165994237177, + 0.004665542001021095, + 0.0043847920023836195, + 0.006366041998262517, + 0.004475209003430791, + 0.003251124988310039, + 0.007133874998544343, + 0.004932708005071618, + 0.004461832999368198, + 0.006264749987167306, + 0.004362291001598351, + 0.0022438329906435683, + 0.0019522500078892335, + 0.003870499989716336, + 0.004871249999268912, + 0.0042908750037895516, + 0.0029702500032726675 + ] + }, + { + "forward": 0.002588625007774681, + "prefill": 0.0022623749973718077, + "decode_steps": [ + 0.0020863749959971756, + 0.0017601669969735667, + 0.0017839590000221506, + 0.0016804999904707074, + 0.001652417005971074, + 0.0017045420099748299, + 0.0016392090037697926, + 0.001692000005277805, + 0.0017043749976437539, + 0.001826874999096617, + 0.0016202499973587692, + 0.00156958399747964, + 0.0017507499869680032, + 0.0017145000019809231, + 0.0016964579990599304, + 0.0016431249969173223, + 0.001613542000995949, + 0.0016974589962046593, + 0.001604916003998369, + 0.0016959170025074854, + 0.0015727919962955639, + 0.0015848750044824556, + 0.0018069579964503646, + 0.0017012500029522926, + 0.0017380000062985346, + 0.0016771249938756227, + 0.002765458993962966, + 0.0016653339989716187, + 0.001621374991373159, + 0.0017403749952791259, + 0.0017553749930812046, + 0.0016679170075803995 + ] + }, + { + "forward": 0.00232658299501054, + "prefill": 0.0023388750123558566, + "decode_steps": [ + 0.0019140420045005158, + 0.001726667003822513, + 0.0017926669970620424, + 0.0016762919985922053, + 0.0016588329890510067, + 0.0016854999994393438, + 0.0018482089944882318, + 0.0015877919940976426, + 0.0016274999943561852, + 0.001698625012068078, + 0.001676582993241027, + 0.0016670839977450669, + 0.001662875001784414, + 0.0017844159883679822, + 0.0016930830024648458, + 0.0017010830051731318, + 0.0016765830077929422, + 0.0016637090011499822, + 0.0016749170026741922, + 0.00169162500242237, + 0.0016839169984450564, + 0.0015981670003384352, + 0.002203540992923081, + 0.0017370420100633055, + 0.0017110000044340268, + 0.0017029580048983917, + 0.0017266659997403622, + 0.0017489169986220077, + 0.0017090420005843043, + 0.001679624998359941, + 0.0022805829939898103, + 0.001729333002003841 + ] + }, + { + "forward": 0.002206916004070081, + "prefill": 0.002237250009784475, + "decode_steps": [ + 0.0017878750077215955, + 0.0016855829890118912, + 0.0015497090062126517, + 0.0019172079919371754, + 0.0017205420008394867, + 0.0015620420017512515, + 0.0016177090001292527, + 0.0015536249993601814, + 0.001697208994301036, + 0.001636750006582588, + 0.001559084004838951, + 0.0016636670043226331, + 0.0017734169960021973, + 0.0016967090050457045, + 0.0018732500029727817, + 0.0017359999910695478, + 0.0016471249982714653, + 0.0016712920041754842, + 0.001654334002523683, + 0.0016248749889200553, + 0.001654667008551769, + 0.0019794999971054494, + 0.001730667005176656, + 0.001663415998336859, + 0.0017140420095529407, + 0.0016315829998347908, + 0.0016595409979345277, + 0.0015802089910721406, + 0.0016924169904086739, + 0.0022089580015745014, + 0.001731042008032091, + 0.0017661669990047812 + ] + }, + { + "forward": 0.0024622089986223727, + "prefill": 0.0021781249961350113, + "decode_steps": [ + 0.001883792006992735, + 0.0017332919960608706, + 0.001664083989453502, + 0.0016720830026315525, + 0.001676999992923811, + 0.0017263329937122762, + 0.001690832999884151, + 0.0017130000051110983, + 0.0016582919924985617, + 0.0015842080028960481, + 0.001829500004532747, + 0.0016916659951675683, + 0.0016453749994980171, + 0.001806874992325902, + 0.001674290993832983, + 0.0016934169980231673, + 0.0017153329972643405, + 0.0015837499959161505, + 0.0016281249991152436, + 0.0016919589979806915, + 0.0016029999969759956, + 0.001550916989799589, + 0.0017700419994071126, + 0.0016294579982059076, + 0.0017321250052191317, + 0.0017086659936467186, + 0.001688792006461881, + 0.0017103329882957041, + 0.0016869999963091686, + 0.0016528330015717074, + 0.0027912499936064705, + 0.0016526670078746974 + ] + }, + { + "forward": 0.0023347079986706376, + "prefill": 0.00238829200679902, + "decode_steps": [ + 0.0019891660049324855, + 0.0017565839953022078, + 0.002148291008779779, + 0.0017528329917695373, + 0.0017389999993611127, + 0.0018535000126576051, + 0.0017680830060271546, + 0.001717083010589704, + 0.0016781669983174652, + 0.001738833001581952, + 0.0018592500127851963, + 0.0017257909930776805, + 0.0017014170007314533, + 0.0015536659921053797, + 0.0016166249988600612, + 0.0015632079885108396, + 0.0015869589988142252, + 0.0021160420001251623, + 0.0017287499940721318, + 0.0016818750009406358, + 0.0015832499921089038, + 0.001591459003975615, + 0.0015495419938815758, + 0.0017826249968493357, + 0.001728333008941263, + 0.001730457996018231, + 0.001705542003037408, + 0.001666500000283122, + 0.002170583000406623, + 0.0017039169906638563, + 0.001698082996881567, + 0.0018897919944720343 + ] + }, + { + "forward": 0.0021137500007171184, + "prefill": 0.0021497499983524904, + "decode_steps": [ + 0.001839999997173436, + 0.001757500009262003, + 0.0017214160034200177, + 0.0016105000104289502, + 0.0016685000009601936, + 0.0017073749913834035, + 0.0017244999908143654, + 0.0015871250070631504, + 0.0017164580058306456, + 0.0016661670088069513, + 0.0022046249941922724, + 0.0017046669963747263, + 0.0015766249998705462, + 0.0016885000077309087, + 0.0017178330017486587, + 0.0016439579922007397, + 0.0017871669988380745, + 0.001590458006830886, + 0.0025943329965230078, + 0.0017107080057030544, + 0.0016437080048490316, + 0.0016740829887567088, + 0.0016267499886453152, + 0.0016204999992623925, + 0.0016226669977186248, + 0.0016003749915398657, + 0.001655625004786998, + 0.0015929170040180907, + 0.0016171250026673079, + 0.0015704580000601709, + 0.0016555409965803847, + 0.0016644159913994372 + ] + }, + { + "forward": 0.002431041997624561, + "prefill": 0.0025396250130143017, + "decode_steps": [ + 0.002216666005551815, + 0.001749167000525631, + 0.0016641249967506155, + 0.0016360830049961805, + 0.0025429170054849237, + 0.001705125003354624, + 0.001782791005098261, + 0.0017605000030016527, + 0.0016765420004958287, + 0.0016498750046594068, + 0.0017534590006107464, + 0.001704791997326538, + 0.0015797079977346584, + 0.0017004999972414225, + 0.001626875004149042, + 0.0018219580088043585, + 0.0020935829961672425, + 0.0017892500036396086, + 0.001701583998510614, + 0.0016828329971758649, + 0.0018034579989034683, + 0.0018788339948514476, + 0.0018382919952273369, + 0.001594083005329594, + 0.0017631249938858673, + 0.0017862079985206947, + 0.001658833003602922, + 0.001550125001813285, + 0.00250604200118687, + 0.0017072920018108562, + 0.001557666008011438, + 0.0017614160024095327 + ] + }, + { + "forward": 0.0031986669928301126, + "prefill": 0.002450458996463567, + "decode_steps": [ + 0.001854916990851052, + 0.0018886670004576445, + 0.001862874996731989, + 0.00172004199703224, + 0.0016540420037927106, + 0.0016611250030109659, + 0.0017890409944811836, + 0.001706709008431062, + 0.0016658749955240637, + 0.0016463330102851614, + 0.0016837500006658956, + 0.0015909579960862175, + 0.0016073329898063093, + 0.0015472499944735318, + 0.0016455419972771779, + 0.0019612500036600977, + 0.0016791249945526943, + 0.0017939579993253574, + 0.0018294170004082844, + 0.001760957995429635, + 0.0016877919988473877, + 0.0016712500073481351, + 0.002197833004174754, + 0.001622416006284766, + 0.0016181670071091503, + 0.0017293749988311902, + 0.0016905410011531785, + 0.001672792001045309, + 0.001676957996096462, + 0.0016531250003026798, + 0.0017614159878576174, + 0.001724041998386383 + ] + }, + { + "forward": 0.002168291000998579, + "prefill": 0.002212874998804182, + "decode_steps": [ + 0.0018104589980794117, + 0.0017410419968655333, + 0.0016703339933883399, + 0.0015706249978393316, + 0.0015287919959519058, + 0.0016871670086402446, + 0.0018170840048696846, + 0.0015872920048423111, + 0.0017394580063410103, + 0.0016143329994520172, + 0.0018834999937098473, + 0.0016722910077078268, + 0.0017177919944515452, + 0.0016910419944906607, + 0.001727415990899317, + 0.001586083002621308, + 0.0016470419941470027, + 0.0015638750046491623, + 0.0016873749991646037, + 0.0015562500047963113, + 0.0017688329971861094, + 0.0017431249871151522, + 0.001708791998680681, + 0.0016841250035213307, + 0.0016780000005383044, + 0.0016655829967930913, + 0.0017070000030798838, + 0.0017561669956194237, + 0.0015827919996809214, + 0.0016848329978529364, + 0.0016731249925214797, + 0.0020629589998861775 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 2.3351874988293275, + "p95": 2.9241480995551674, + "min": 2.1137500007171184, + "max": 3.1986669928301126, + "iqr": 0.21758449656772427, + "mean": 2.41664579953067, + "cv": 0.12807751640746765 + }, + "prefill_ms": { + "count": 10, + "median": 2.275291502883192, + "p95": 2.499500305566471, + "min": 2.1497499983524904, + "max": 2.5396250130143017, + "iqr": 0.15696900663897395, + "mean": 2.304583403747529, + "cv": 0.05405453888191213 + }, + "decode_ms_per_step": { + "count": 320, + "median": 1.701333501841873, + "p95": 2.8001999940897826, + "min": 1.5287919959519058, + "max": 7.133874998544343, + "iqr": 0.11428074139985256, + "mean": 1.8848248932954448, + "cv": 0.3839430970134403 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 578.2000170800941, + "p95": 588.1780428655728, + "min": 305.46046086272065, + "max": 590.2470604577273, + "iqr": 11.507240688011734, + "mean": 550.7455144104072, + "cv": 0.15727759985112622 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 554.5856939329124, + "p95": 565.0807169644722, + "min": 298.9310859623651, + "max": 567.0997676168615, + "iqr": 12.72830605995432, + "mean": 529.3110980222868, + "cv": 0.15386598293335116 + }, + "memory": { + "peak_extra_bytes": 2951168, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "reference", + "batch": 1, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.009327040999778546, + "prefill": 0.0040969999972730875, + "decode_steps": [ + 0.005581833000178449, + 0.004477334005059674, + 0.00459579199377913, + 0.004661541999666952, + 0.004802916999324225, + 0.0044363750057527795, + 0.004779916009283625, + 0.00447037500271108, + 0.004161040997132659, + 0.004318750012316741, + 0.004533416999038309, + 0.004260665999026969, + 0.004219082999043167, + 0.004016791004687548, + 0.004482000003918074, + 0.004656416000216268, + 0.0044940410007257015, + 0.004924916996969841, + 0.004585041999234818, + 0.0038593339995713905, + 0.004321417000028305, + 0.004800875001819804, + 0.004333042001235299, + 0.0043440419976832345, + 0.005499375009094365, + 0.0043661249947035685, + 0.004270624995115213, + 0.004359958998975344, + 0.004537625005468726, + 0.0050186250009574, + 0.0042101250000996515, + 0.004403875005664304 + ] + }, + { + "forward": 0.00668633398890961, + "prefill": 0.0031308329926105216, + "decode_steps": [ + 0.005025375008699484, + 0.004475499998079613, + 0.00386850000359118, + 0.004149083004449494, + 0.003846624997095205, + 0.004174541987595148, + 0.003716541003086604, + 0.004063750006025657, + 0.00428720899799373, + 0.004323333007050678, + 0.0036734169989358634, + 0.0038712910027243197, + 0.004122458005440421, + 0.004298166997614317, + 0.003816791999270208, + 0.004600125001161359, + 0.004810124999494292, + 0.004382375002023764, + 0.0037582079967251047, + 0.0037998750049155205, + 0.004152374996920116, + 0.0045315830066101626, + 0.004267500000423752, + 0.004103041996131651, + 0.004431332999956794, + 0.004158333002123982, + 0.004321208005421795, + 0.0043632500019157305, + 0.004156249997322448, + 0.00426254200283438, + 0.004164041994954459, + 0.004091707989573479 + ] + }, + { + "forward": 0.005724665999878198, + "prefill": 0.0031732500065118074, + "decode_steps": [ + 0.0041438749904045835, + 0.004091167007572949, + 0.003652958999737166, + 0.003924042001017369, + 0.004496458001085557, + 0.004152082998189144, + 0.0037788339977851138, + 0.004148499996517785, + 0.004223916999762878, + 0.004188416991382837, + 0.004314750010962598, + 0.0038161249976838008, + 0.004872791992966086, + 0.004289165997761302, + 0.004399750003358349, + 0.004299084001104347, + 0.004263875001925044, + 0.004380124999443069, + 0.0036941250000381842, + 0.003851792003843002, + 0.004359292011940852, + 0.004238082998199388, + 0.0037704169953940436, + 0.004244084004312754, + 0.0047148329904302955, + 0.004328875002101995, + 0.0041594159993110225, + 0.004212250001728535, + 0.0036783749965252355, + 0.0041013330046553165, + 0.0042005410068668425, + 0.004110333000426181 + ] + }, + { + "forward": 0.0059922499931417406, + "prefill": 0.0030597080040024593, + "decode_steps": [ + 0.004406375010148622, + 0.004161834003753029, + 0.004435416994965635, + 0.004276416002539918, + 0.004269875003956258, + 0.00427224999293685, + 0.0042613749974407256, + 0.004309791998821311, + 0.004203000004054047, + 0.004134709000936709, + 0.0041326249920530245, + 0.0036998750001657754, + 0.004168832994764671, + 0.004122041005757637, + 0.004191707994323224, + 0.003897041009622626, + 0.003989415999967605, + 0.004290791010134853, + 0.0043016660056309775, + 0.003577541996492073, + 0.0043525829969439656, + 0.004424083002959378, + 0.004347458001575433, + 0.004123708000406623, + 0.0037206249980954453, + 0.004159707998041995, + 0.004179084004135802, + 0.0041882079967763275, + 0.003552375012077391, + 0.004236624998156913, + 0.004142541991313919, + 0.0038475000037578866 + ] + }, + { + "forward": 0.006356124999001622, + "prefill": 0.003091457998380065, + "decode_steps": [ + 0.0038035420002415776, + 0.00414962500508409, + 0.004724667000118643, + 0.004576708990498446, + 0.0038469159917440265, + 0.004178958013653755, + 0.004606957998475991, + 0.004229291997035034, + 0.0036229999968782067, + 0.0037794579984620214, + 0.004196624999167398, + 0.004039541003294289, + 0.004289250005967915, + 0.0037381670117611066, + 0.004378832993097603, + 0.004126458006794564, + 0.004836291991523467, + 0.003978334003477357, + 0.004391209004097618, + 0.004241875009029172, + 0.0038156660011736676, + 0.004145917002460919, + 0.004161624994594604, + 0.004378957994049415, + 0.005229917005635798, + 0.005606125007034279, + 0.005890624990570359, + 0.005026625003665686, + 0.004354167002020404, + 0.005145333998370916, + 0.004627583999536, + 0.005417958993348293 + ] + }, + { + "forward": 0.006446041006711312, + "prefill": 0.0036888750037178397, + "decode_steps": [ + 0.005211374998907559, + 0.0048353750025853515, + 0.005185999994864687, + 0.004587167000863701, + 0.00476420798804611, + 0.004683959006797522, + 0.0052387920004548505, + 0.005271916990750469, + 0.004898541999864392, + 0.0054023749980842695, + 0.005038917006459087, + 0.004600792002747767, + 0.004950417001964524, + 0.004523333001998253, + 0.0047519579966319725, + 0.004735916998470202, + 0.004974165989551693, + 0.00396570899465587, + 0.004557708001811989, + 0.004872750010690652, + 0.004667291999794543, + 0.004523374998825602, + 0.004611708005541004, + 0.004315083991969004, + 0.0036961250007152557, + 0.00392304100387264, + 0.004792082996573299, + 0.005003792000934482, + 0.004676832992117852, + 0.004770999992615543, + 0.0044277909910306334, + 0.005169042007764801 + ] + }, + { + "forward": 0.006723208003677428, + "prefill": 0.0037831250083399937, + "decode_steps": [ + 0.0046321670088218525, + 0.0044632919889409095, + 0.004450791995623149, + 0.004287542004021816, + 0.004267291005817242, + 0.004212292013107799, + 0.0036292500008130446, + 0.004324500012444332, + 0.004283709000446834, + 0.004157707997364923, + 0.0036020409897901118, + 0.004104082996491343, + 0.004483583994442597, + 0.005597916999249719, + 0.0045272080024005845, + 0.004541124988463707, + 0.004994832997908816, + 0.006102250001276843, + 0.0045041659905109555, + 0.005244333995506167, + 0.005303166995872743, + 0.004306832997826859, + 0.005122875008964911, + 0.005030625005019829, + 0.004416583004058339, + 0.004422124999109656, + 0.004288833006285131, + 0.004395374999148771, + 0.004227166005875915, + 0.0041377080051461235, + 0.003983291986514814, + 0.0046186250110622495 + ] + }, + { + "forward": 0.0066106250014854595, + "prefill": 0.003854291993775405, + "decode_steps": [ + 0.0046178330085240304, + 0.0044975420023547485, + 0.004375125005026348, + 0.004631791001884267, + 0.004566208997857757, + 0.005013709000195377, + 0.004362749998108484, + 0.003909708000719547, + 0.003907541002263315, + 0.004259790992364287, + 0.004427791995112784, + 0.00447054099640809, + 0.004089249996468425, + 0.004897166989394464, + 0.004999249998945743, + 0.004818624991457909, + 0.0047266250039683655, + 0.004402374994242564, + 0.004776457994012162, + 0.004288791009457782, + 0.004965624990290962, + 0.004150292006670497, + 0.004111000002012588, + 0.004911249998258427, + 0.0048690830008126795, + 0.004208374986774288, + 0.004728083004010841, + 0.004542665992630646, + 0.003663791998405941, + 0.004692832997534424, + 0.00436924998939503, + 0.00450624999939464 + ] + }, + { + "forward": 0.005973999999696389, + "prefill": 0.0036804999981541187, + "decode_steps": [ + 0.00532158299756702, + 0.004747500002849847, + 0.004477417009184137, + 0.005050209001637995, + 0.00400608399650082, + 0.004504583004745655, + 0.004339999999501742, + 0.004270333010936156, + 0.003830250003375113, + 0.004135582988965325, + 0.0036726669932249933, + 0.0043328339961590245, + 0.004475707988603972, + 0.004369916990981437, + 0.004234583000652492, + 0.003802499995799735, + 0.004259791006916203, + 0.004237667002598755, + 0.0037594579916913062, + 0.004213666994473897, + 0.004040916988742538, + 0.0037754160002805293, + 0.004241542003001086, + 0.00419408299785573, + 0.004151041008299217, + 0.004138667005463503, + 0.00534608299494721, + 0.004190250008832663, + 0.00435208399721887, + 0.0053421659977175295, + 0.004436541988980025, + 0.004978458004188724 + ] + }, + { + "forward": 0.00665266701253131, + "prefill": 0.003124625000054948, + "decode_steps": [ + 0.004084375003003515, + 0.004290416007279418, + 0.004451957996934652, + 0.003952375001972541, + 0.004480917006731033, + 0.004044250003062189, + 0.004415709001477808, + 0.004361250001238659, + 0.005384791002143174, + 0.00421391699637752, + 0.004079790989635512, + 0.004831083002500236, + 0.00416345801204443, + 0.0042137499985983595, + 0.00443095900118351, + 0.0043042079923907295, + 0.004255374995409511, + 0.0041666659963084385, + 0.004143708007177338, + 0.004147250001551583, + 0.00409708300139755, + 0.004181875003268942, + 0.00414437499421183, + 0.004200540992314927, + 0.004154791997279972, + 0.004081666993442923, + 0.004183083001407795, + 0.004030957992654294, + 0.004148291001911275, + 0.004197707996354438, + 0.004195708999759518, + 0.004051708005135879 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 6.528333004098386, + "p95": 8.15531615153304, + "min": 5.724665999878198, + "max": 9.327040999778546, + "iqr": 0.5946985002083238, + "mean": 6.6492957004811615, + "cv": 0.15079734997482705 + }, + "prefill_ms": { + "count": 10, + "median": 3.426875002332963, + "p95": 3.98778139569913, + "min": 3.0597080040024593, + "max": 4.0969999972730875, + "iqr": 0.6333855089906137, + "mean": 3.4683666002820246, + "cv": 0.1122869838281701 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.298625499359332, + "p95": 5.230360755376751, + "min": 3.552375012077391, + "max": 6.102250001276843, + "iqr": 0.43187549090362154, + "mean": 4.378829665574813, + "cv": 0.09703021263341569 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 227.9633819165066, + "p95": 241.14313058541774, + "min": 211.04205958612107, + "max": 241.7336898087723, + "iqr": 16.25110781407477, + "mean": 228.77809045808743, + "cv": 0.04421449804178329 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 222.58741079501033, + "p95": 235.62022076468236, + "min": 206.029693371777, + "max": 236.27258491754375, + "iqr": 16.527515584739575, + "mean": 223.27003859782826, + "cv": 0.0452357459691824 + }, + "memory": { + "peak_extra_bytes": 4327424, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "auto", + "batch": 1, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.004098917008377612 + }, + { + "forward": 0.0040767079917714 + }, + { + "forward": 0.004079874997842126 + }, + { + "forward": 0.0040823749877745286 + }, + { + "forward": 0.0039029170002322644 + }, + { + "forward": 0.004098917008377612 + }, + { + "forward": 0.0041009580017998815 + }, + { + "forward": 0.00407633300346788 + }, + { + "forward": 0.004124457991565578 + }, + { + "forward": 0.004112666996661574 + } + ], + "forward_ms": { + "count": 10, + "median": 4.09064599807607, + "p95": 4.119152043858776, + "min": 3.9029170002322644, + "max": 4.124457991565578, + "iqr": 0.022948010155232623, + "mean": 4.0754124987870455, + "cv": 0.015383827593590516 + }, + "memory": { + "peak_extra_bytes": 5902336, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "sdpa", + "batch": 1, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.007769541000016034, + "prefill": 0.006836082990048453, + "decode_steps": [ + 0.0025426659994991496, + 0.0017870000010589138, + 0.0017633749957894906, + 0.0017722080083331093, + 0.0017398750060237944, + 0.0018492090021027252, + 0.0017872500029625371, + 0.001723958004731685, + 0.0017589170020073652, + 0.0017130420019384474, + 0.0017732090054778382, + 0.0018377499945927411, + 0.0017772919964045286, + 0.0016391250101150945, + 0.001771042007021606, + 0.0017923330015037209, + 0.0017775410087779164, + 0.0016854580026119947, + 0.001801083009922877, + 0.0017554579972056672, + 0.0017562920111231506, + 0.001968958997167647, + 0.0018277089984621853, + 0.0017669579974608496, + 0.001790082998923026, + 0.0016982080123852938, + 0.0016987499984679744, + 0.0027516249974723905, + 0.0017367499967804179, + 0.001749125003698282, + 0.0017754580039763823, + 0.0017905829881783575 + ] + }, + { + "forward": 0.0037210419977782294, + "prefill": 0.0037543329963227734, + "decode_steps": [ + 0.001995250000618398, + 0.0017431250016670674, + 0.0017290410032728687, + 0.0017328749963780865, + 0.0017458749935030937, + 0.0017464589909650385, + 0.0017803329974412918, + 0.0017157090042019263, + 0.0017596670077182353, + 0.0017505839932709932, + 0.001756333003868349, + 0.0017656669951975346, + 0.0017486669967183843, + 0.001761500010616146, + 0.0016902499919524416, + 0.0016515839961357415, + 0.0016981659864541143, + 0.0019221670081606135, + 0.001802333994419314, + 0.001797542005078867, + 0.0016969159914879128, + 0.0017764170042937621, + 0.0017650000081630424, + 0.0019534159946488217, + 0.0018339169910177588, + 0.0018019160052062944, + 0.0017740419862093404, + 0.0017589580093044788, + 0.0017327079985989258, + 0.0017628330097068101, + 0.0017574999947100878, + 0.0017536670056870207 + ] + }, + { + "forward": 0.0034197080094600096, + "prefill": 0.003398083004867658, + "decode_steps": [ + 0.002688707987545058, + 0.0018157080048695207, + 0.0016864589997567236, + 0.001716375001706183, + 0.0017731670086504892, + 0.0017847080016508698, + 0.0016670000040903687, + 0.0017743750067893416, + 0.0017770830017980188, + 0.0017006249981932342, + 0.001961125002708286, + 0.0018108750082319602, + 0.0017690410022623837, + 0.0016389160009566694, + 0.0018277499912073836, + 0.001699625005130656, + 0.001709707998088561, + 0.0018555420101620257, + 0.0017641669983277097, + 0.001736375008476898, + 0.001752540993038565, + 0.0017662500031292439, + 0.0017572080105310306, + 0.0017740000039339066, + 0.0017662090103840455, + 0.0017182919982587919, + 0.0016616250068182126, + 0.001713208999717608, + 0.0017599170096218586, + 0.001757083009579219, + 0.0017786249954951927, + 0.0017358749901177362 + ] + }, + { + "forward": 0.0035813750000670552, + "prefill": 0.0034107499959645793, + "decode_steps": [ + 0.001844041995354928, + 0.0017113749927375466, + 0.0016487919929204509, + 0.0016537080082343891, + 0.0017297919985139742, + 0.0017514999926788732, + 0.0017493339983047917, + 0.0017503749986644834, + 0.0017576669924892485, + 0.0017523339920444414, + 0.001694334001513198, + 0.0016496669995831326, + 0.0018199170008301735, + 0.0016433330019935966, + 0.0017658750002738088, + 0.0018652919970918447, + 0.0018153329874621704, + 0.0017880000086734071, + 0.0018249999993713573, + 0.0017036670033121482, + 0.0017603329906705767, + 0.0017653749964665622, + 0.0017493750056019053, + 0.0017565830057719722, + 0.001700000007986091, + 0.0027234590088482946, + 0.0017344590014545247, + 0.0017064579878933728, + 0.0018296669877599925, + 0.0017640410078456625, + 0.0016367499920306727, + 0.001676041996688582 + ] + }, + { + "forward": 0.0035082919930573553, + "prefill": 0.0034698329982347786, + "decode_steps": [ + 0.0019937920005759224, + 0.0017720420000841841, + 0.0017846659902716056, + 0.001783790998160839, + 0.0018061669979942963, + 0.0017892080068122596, + 0.001781583996489644, + 0.0017483750125393271, + 0.001743666987749748, + 0.0017643329920247197, + 0.001759999999194406, + 0.0016335419932147488, + 0.0018765419954434037, + 0.0017974579968722537, + 0.0016531659930478781, + 0.002031416996032931, + 0.001694667007541284, + 0.0016140410007210448, + 0.0027982920000795275, + 0.0018141670007025823, + 0.0018735839985311031, + 0.001814667004509829, + 0.0017831250006565824, + 0.001754833006998524, + 0.001709666001261212, + 0.0017210829973919317, + 0.001633917010622099, + 0.0017899579979712144, + 0.0016273340006591752, + 0.0017725840007187799, + 0.001758416008669883, + 0.0017517919914098457 + ] + }, + { + "forward": 0.0033558750001247972, + "prefill": 0.003484041997580789, + "decode_steps": [ + 0.0018333330081077293, + 0.0018203340005129576, + 0.0017812500009313226, + 0.001767249996191822, + 0.0016817499999888241, + 0.001785500004189089, + 0.0017740420007612556, + 0.0016856250003911555, + 0.001749542003381066, + 0.0017443750111851841, + 0.0018972499965457246, + 0.0017996250098804012, + 0.001794290990801528, + 0.0017799170018406585, + 0.001800416997866705, + 0.0017847089911811054, + 0.0016664169961586595, + 0.001779250000254251, + 0.0017482500115875155, + 0.0017587920010555536, + 0.0017938339879037812, + 0.0017832089943112805, + 0.0017755000008037314, + 0.001778292004019022, + 0.0017890000017359853, + 0.0017021660023601726, + 0.0016914169973460957, + 0.001815459007048048, + 0.0017790419951779768, + 0.0016335410036845133, + 0.001969167002243921, + 0.001858292001998052 + ] + }, + { + "forward": 0.0034302499989280477, + "prefill": 0.004466166006750427, + "decode_steps": [ + 0.0019230829930165783, + 0.0016692920034984127, + 0.0017982499994104728, + 0.0019280829874332994, + 0.0017508339951746166, + 0.0017410409927833825, + 0.001705959002720192, + 0.0018014579982263967, + 0.0017080410034395754, + 0.001622416006284766, + 0.0016852080007083714, + 0.0018461669969838113, + 0.0017836659972090274, + 0.001767666995874606, + 0.0016803749895188957, + 0.0017759579932317138, + 0.001741583997500129, + 0.001778082994860597, + 0.0018076670094160363, + 0.001764417000231333, + 0.001688207994448021, + 0.0017742079944582656, + 0.001712458994006738, + 0.0017545839946251363, + 0.0017628339992370456, + 0.0016569170111324638, + 0.0016326670011039823, + 0.0017142500000772998, + 0.001775832992279902, + 0.001683084003161639, + 0.0017845839902292937, + 0.0017539169930387288 + ] + }, + { + "forward": 0.003555834002327174, + "prefill": 0.0037799589917995036, + "decode_steps": [ + 0.0033297910122200847, + 0.0018562909972388297, + 0.0017910410097101703, + 0.001924125012010336, + 0.0019511249993229285, + 0.001653541999985464, + 0.0016752079973230138, + 0.0016870410036062822, + 0.0016771660011727363, + 0.0017419580108253285, + 0.0016543750098207965, + 0.0017955420044017956, + 0.0016850000101840124, + 0.0018668749980861321, + 0.0017547500028740615, + 0.0015930840017972514, + 0.001718708997941576, + 0.0017473749903729185, + 0.0017313330026809126, + 0.0016917920002015308, + 0.0017665409977780655, + 0.0017568340117577463, + 0.001644582996959798, + 0.001679415989201516, + 0.0016660420078551397, + 0.001718333995086141, + 0.001685291004832834, + 0.0017216670094057918, + 0.0016294579982059076, + 0.0016542079974897206, + 0.0018030000064754859, + 0.0017849580035544932 + ] + }, + { + "forward": 0.0035113750054733828, + "prefill": 0.003530875008436851, + "decode_steps": [ + 0.001953500002855435, + 0.0018932500097434968, + 0.001719333988148719, + 0.0017871669988380745, + 0.0017879579972941428, + 0.0016427919908892363, + 0.001857583993114531, + 0.001775958007783629, + 0.0017691660032141954, + 0.0016461670020362362, + 0.0017057920049410313, + 0.00184116700256709, + 0.0017830830038292333, + 0.001736041987896897, + 0.0018605420045787469, + 0.0018311669991817325, + 0.0017927090084413067, + 0.0017826670082286, + 0.001790416004951112, + 0.001748374997987412, + 0.0017075409996323287, + 0.0017507080046925694, + 0.0016962500085355714, + 0.0016652079939376563, + 0.0018204579973826185, + 0.0017480840033385903, + 0.0016334170068148524, + 0.0017675419949227944, + 0.0017678330041235313, + 0.0018185830122092739, + 0.0017423339886590838, + 0.0017798329936340451 + ] + }, + { + "forward": 0.0034527090028859675, + "prefill": 0.003531541005941108, + "decode_steps": [ + 0.0019669579924084246, + 0.0017577499966137111, + 0.0017129169864347205, + 0.0017692499968688935, + 0.0017590829957043752, + 0.0016770840011304244, + 0.001779250000254251, + 0.001778292004019022, + 0.001748915994539857, + 0.0017521669942652807, + 0.0017227500065928325, + 0.0017183750023832545, + 0.0017138330003945157, + 0.0017010000010486692, + 0.0017664580082055181, + 0.001712750003207475, + 0.001833125003031455, + 0.0017685420025372878, + 0.0017197080014739186, + 0.0016842909972183406, + 0.001621875009732321, + 0.0017895829951157793, + 0.0016474580042995512, + 0.0016330840007867664, + 0.0017039170052157715, + 0.001713208999717608, + 0.0017287499940721318, + 0.0019262499990873039, + 0.0017254579870495945, + 0.0016978339990600944, + 0.0018155419966205955, + 0.0016310000064549968 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 3.509833499265369, + "p95": 5.947716449009017, + "min": 3.3558750001247972, + "max": 7.769541000016034, + "iqr": 0.13912500071455725, + "mean": 3.9306001010118052, + "cv": 0.3441433905898966 + }, + "prefill_ms": { + "count": 10, + "median": 3.5312080071889795, + "p95": 5.769620347564339, + "min": 3.398083004867658, + "max": 6.836082990048453, + "iqr": 0.3001672448590398, + "mean": 3.966166499594692, + "cv": 0.2664922808592717 + }, + "decode_ms_per_step": { + "count": 320, + "median": 1.759020502504427, + "p95": 1.929235088027781, + "min": 1.5930840017972514, + "max": 3.3297910122200847, + "iqr": 0.07637475209776312, + "mean": 1.776749384634968, + "cv": 0.09179438867405586 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 563.76408764742, + "p95": 573.3799011412362, + "min": 548.3368244266436, + "max": 574.7491953162494, + "iqr": 4.610720809364011, + "mean": 562.9164739734326, + "cv": 0.013364288034882021 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 529.565886631113, + "p95": 536.7277817791701, + "min": 490.839899380974, + "max": 540.4675044690181, + "iqr": 4.945312499162014, + "mean": 526.4407595647266, + "cv": 0.02518393125235755 + }, + "memory": { + "peak_extra_bytes": 5902336, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "reference", + "batch": 1, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.009805209003388882, + "prefill": 0.005849958994076587, + "decode_steps": [ + 0.004833874991163611, + 0.004600999993272126, + 0.0041893330053426325, + 0.004481208001379855, + 0.004337958991527557, + 0.004283165995730087, + 0.004302999994251877, + 0.004305249996832572, + 0.004062041989527643, + 0.004151582994381897, + 0.004217417008476332, + 0.004020416003186256, + 0.004263333001290448, + 0.004354958000476472, + 0.0042840839887503535, + 0.004141458004596643, + 0.004171541993855499, + 0.004142249992582947, + 0.004213874999550171, + 0.004568500007735565, + 0.004036707992781885, + 0.004281707995687611, + 0.004472707994864322, + 0.004086666987859644, + 0.003855000002658926, + 0.0037590410065604374, + 0.004266666001058184, + 0.004282750000129454, + 0.004291999997803941, + 0.004437209005118348, + 0.004313208002713509, + 0.004301084001781419 + ] + }, + { + "forward": 0.009288125002058223, + "prefill": 0.005748666008003056, + "decode_steps": [ + 0.004495166998822242, + 0.004292875004466623, + 0.004450666005141102, + 0.004389916997752152, + 0.004259374996763654, + 0.004284875001758337, + 0.004397416996653192, + 0.004302999994251877, + 0.004139334007049911, + 0.00425466698652599, + 0.0042647500085877255, + 0.004166749989963137, + 0.004198999988147989, + 0.004334041994297877, + 0.004150917011429556, + 0.003873499998007901, + 0.004912791991955601, + 0.0041140829998767, + 0.004483624987187795, + 0.004793542000697926, + 0.006026416987879202, + 0.004051584008266218, + 0.004767499995068647, + 0.004514250002102926, + 0.004107999993721023, + 0.00449895799101796, + 0.0044739589939126745, + 0.004217124995193444, + 0.004380291007691994, + 0.004414583003381267, + 0.004320791995269246, + 0.004128750006202608 + ] + }, + { + "forward": 0.008646292000776157, + "prefill": 0.0058619159972295165, + "decode_steps": [ + 0.004362541003501974, + 0.0041829169931588694, + 0.00408512499416247, + 0.004164708007010631, + 0.004290249999030493, + 0.004208208993077278, + 0.004014499994809739, + 0.004243791991029866, + 0.0042697080061770976, + 0.004313875004299916, + 0.004194124994683079, + 0.004298583997297101, + 0.0040864590118872, + 0.004229457990732044, + 0.00413579199812375, + 0.004155624992563389, + 0.0041469589923508465, + 0.004194084001937881, + 0.004141458004596643, + 0.004162375000305474, + 0.004152666995651089, + 0.004207417005090974, + 0.0040796670073177665, + 0.004152542009251192, + 0.0039086250035325065, + 0.004216207991703413, + 0.0042884589929599315, + 0.003917082998668775, + 0.004767624996020459, + 0.004507250007009134, + 0.004308249990572222, + 0.0043260829988867044 + ] + }, + { + "forward": 0.009535457997117192, + "prefill": 0.0058649999991757795, + "decode_steps": [ + 0.004832999999052845, + 0.0044053749879822135, + 0.004314707999583334, + 0.004313833997002803, + 0.004384666011901572, + 0.004269500001100823, + 0.004201000003376976, + 0.004262582995579578, + 0.004217959009110928, + 0.00427504199615214, + 0.004030291995150037, + 0.004616124992026016, + 0.004345042005297728, + 0.004206207988318056, + 0.004285415998310782, + 0.004265750001650304, + 0.00424566600122489, + 0.004211375009617768, + 0.004236666994984262, + 0.004193040993413888, + 0.00428587500937283, + 0.0042179170122835785, + 0.004366790992207825, + 0.004193083004793152, + 0.0042795830086106434, + 0.004299249994801357, + 0.004203416989184916, + 0.004153000001679175, + 0.004170667001744732, + 0.004103624989511445, + 0.0040186660044128075, + 0.004198458002065308 + ] + }, + { + "forward": 0.009421666996786371, + "prefill": 0.005831834001583047, + "decode_steps": [ + 0.004537708009593189, + 0.004160666008829139, + 0.004197625006781891, + 0.004617790997144766, + 0.004496291992836632, + 0.004328916998929344, + 0.004151583998464048, + 0.0042463329882593825, + 0.004211166989989579, + 0.004281916990294121, + 0.004205250006634742, + 0.004593374993419275, + 0.00491262499417644, + 0.004350125003838912, + 0.004300416010664776, + 0.004304749993025325, + 0.0042081660067196935, + 0.004318291001254693, + 0.004303333000279963, + 0.004058958002133295, + 0.004134416012675501, + 0.00406662501336541, + 0.004181667012744583, + 0.003996832994744182, + 0.0041398749890504405, + 0.004109665998839773, + 0.004175999987637624, + 0.004144916994846426, + 0.004188541992334649, + 0.004187958998954855, + 0.004203958000289276, + 0.004392957998788916 + ] + }, + { + "forward": 0.00881291700352449, + "prefill": 0.006775959001970477, + "decode_steps": [ + 0.004290416007279418, + 0.00425133400131017, + 0.004320041989558376, + 0.0038432920118793845, + 0.004276292005670257, + 0.004021083994302899, + 0.004192416003206745, + 0.004198374997940846, + 0.0042763330129673705, + 0.004299583000829443, + 0.004273375001503155, + 0.004235957996570505, + 0.004216375004034489, + 0.0041969580051954836, + 0.003982750000432134, + 0.0041819159960141405, + 0.004291916993679479, + 0.004187208003713749, + 0.00425895799708087, + 0.0041489580034976825, + 0.0040954160067485645, + 0.004105291998712346, + 0.004138040996622294, + 0.0041629170009400696, + 0.00420549999398645, + 0.004326708003645763, + 0.004824165996978991, + 0.004292124998755753, + 0.004496458001085557, + 0.0044350409880280495, + 0.0043150840065209195, + 0.0042105000029550865 + ] + }, + { + "forward": 0.009623792007914744, + "prefill": 0.005752750003011897, + "decode_steps": [ + 0.004750500011141412, + 0.004331874995841645, + 0.004357833007816225, + 0.004416291005327366, + 0.004339541992521845, + 0.004294374986784533, + 0.003993582999100909, + 0.0042680420010583475, + 0.0038955839991103858, + 0.004565208000713028, + 0.0043044170015491545, + 0.0038333330012392253, + 0.004110750000108965, + 0.004168375002336688, + 0.004157500006840564, + 0.004334540994022973, + 0.004294083992135711, + 0.004185375000815839, + 0.004271958008757792, + 0.004269291996024549, + 0.004317499988246709, + 0.004197333997581154, + 0.004235375003190711, + 0.004161999997450039, + 0.004263583992724307, + 0.0042255409935023636, + 0.004209541002637707, + 0.004132208996452391, + 0.004238958994392306, + 0.004043042004923336, + 0.005072708008810878, + 0.004411916001117788 + ] + }, + { + "forward": 0.009508832998108119, + "prefill": 0.005791291012428701, + "decode_steps": [ + 0.004486999998334795, + 0.004352208008640446, + 0.004474083005334251, + 0.004445665996172465, + 0.005455624996102415, + 0.004400707999593578, + 0.0042756670009111986, + 0.0042094169912161306, + 0.004108583991182968, + 0.004182832999504171, + 0.004186666003079154, + 0.00418270900263451, + 0.004245459000230767, + 0.004256375003024004, + 0.004304125002818182, + 0.004200791998300701, + 0.004215291992295533, + 0.004211584004224278, + 0.004263166993041523, + 0.0042735839961096644, + 0.0041177919920301065, + 0.004179583003860898, + 0.004167249993770383, + 0.004248124998412095, + 0.004174124987912364, + 0.0040912500116974115, + 0.004165167003520764, + 0.004145457991398871, + 0.004308707997552119, + 0.0039231249975273386, + 0.00480758398771286, + 0.005257540993625298 + ] + }, + { + "forward": 0.02216029200644698, + "prefill": 0.008597874999395572, + "decode_steps": [ + 0.00485208400641568, + 0.0058626660029403865, + 0.004397208002046682, + 0.004570666002109647, + 0.004835583007661626, + 0.004688166998676024, + 0.004506541998125613, + 0.00436924998939503, + 0.004176250004093163, + 0.0041400420013815165, + 0.004218500005663373, + 0.004405540996231139, + 0.005113874998642132, + 0.004704833001596853, + 0.004582666006172076, + 0.004414541996084154, + 0.004236791995936073, + 0.005042916993261315, + 0.004418500000610948, + 0.00441187500837259, + 0.004385292006190866, + 0.004565833005472086, + 0.004515541993896477, + 0.004144666992942803, + 0.004163958001299761, + 0.004378040990559384, + 0.004241000002366491, + 0.004289624994271435, + 0.004220166010782123, + 0.004148457999690436, + 0.004343541993875988, + 0.003981916990596801 + ] + }, + { + "forward": 0.00967666698852554, + "prefill": 0.006906958005856723, + "decode_steps": [ + 0.00481095899885986, + 0.004493250002269633, + 0.004556500003673136, + 0.004400915990117937, + 0.004353500000433996, + 0.00437150000652764, + 0.004266209012712352, + 0.004288832991733216, + 0.004354374992544763, + 0.00421633300720714, + 0.0042387079884065315, + 0.0042548330093268305, + 0.004226584002026357, + 0.004176749993348494, + 0.004250874990248121, + 0.004051667012390681, + 0.004470917003345676, + 0.00438941600441467, + 0.004246333002811298, + 0.004253874998539686, + 0.0042008329910459, + 0.003945666991057806, + 0.0042815829947358, + 0.00449970799672883, + 0.004467334001674317, + 0.004287457995815203, + 0.003991125005995855, + 0.004339792008977383, + 0.004199332994176075, + 0.004210415994748473, + 0.0041657499969005585, + 0.004023083005449735 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 9.522145497612655, + "p95": 16.60050465507082, + "min": 8.646292000776157, + "max": 22.16029200644698, + "iqr": 0.3419377426325809, + "mean": 10.64792520046467, + "cv": 0.3814828033867492 + }, + "prefill_ms": { + "count": 10, + "median": 5.855937495653052, + "p95": 7.836962352303088, + "min": 5.748666008003056, + "max": 8.597874999395572, + "iqr": 0.7467924915545154, + "mean": 6.298220802273136, + "cv": 0.14532927650156396 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.259166496922262, + "p95": 4.794244100048672, + "min": 3.7590410065604374, + "max": 6.026416987879202, + "iqr": 0.1810415051295422, + "mean": 4.299779277380367, + "cv": 0.05899943810609884 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 234.12276334966128, + "p95": 236.8819255809684, + "min": 223.2663957463096, + "max": 237.54305659228692, + "iqr": 2.844252085785797, + "mean": 232.64043569881443, + "cv": 0.018140651503779224 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 224.55423210374354, + "p95": 226.494296962434, + "min": 210.63105624387782, + "max": 227.63756826887428, + "iqr": 2.8447303718251646, + "mean": 222.48638143708413, + "cv": 0.021532735875838184 + }, + "memory": { + "peak_extra_bytes": 15306752, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "auto", + "batch": 1, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.00798633400700055 + }, + { + "forward": 0.007977375003974885 + }, + { + "forward": 0.00798725000640843 + }, + { + "forward": 0.008011832993361168 + }, + { + "forward": 0.008203624995076098 + }, + { + "forward": 0.008308500007842667 + }, + { + "forward": 0.008162833997630514 + }, + { + "forward": 0.00802016700617969 + }, + { + "forward": 0.0080670830066083 + }, + { + "forward": 0.008037999999942258 + } + ], + "forward_ms": { + "count": 10, + "median": 8.029083503060974, + "p95": 8.261306252097711, + "min": 7.977375003974885, + "max": 8.308500007842667, + "iqr": 0.14550049672834575, + "mean": 8.076300102402456, + "cv": 0.013830178430386952 + }, + "memory": { + "peak_extra_bytes": 11804672, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "sdpa", + "batch": 1, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.006141874997410923, + "prefill": 0.006238375004613772, + "decode_steps": [ + 0.001997500003199093, + 0.0018139580060960725, + 0.0017850409931270406, + 0.0017729170067468658, + 0.002023209002800286, + 0.0018679159984458238, + 0.00184208300197497, + 0.0017469169979449362, + 0.0018556669965619221, + 0.001851207998697646, + 0.0018032920052064583, + 0.0017462079995311797, + 0.0018024169985437766, + 0.0018732919998001307, + 0.0018721250089583918, + 0.0017417080089217052, + 0.0017516250081826001, + 0.0017236659914487973, + 0.001729333002003841, + 0.0017403749952791259, + 0.0017302500054938719, + 0.0017168750055134296, + 0.0017319579928880557, + 0.0017098340031225234, + 0.0017953330097952858, + 0.001742082997225225, + 0.0017176250112242997, + 0.0017490829923190176, + 0.0021459999989019707, + 0.001899583003250882, + 0.0017823329981183633, + 0.0019442500051809475 + ] + }, + { + "forward": 0.006180292009958066, + "prefill": 0.006142165992059745, + "decode_steps": [ + 0.0019682499987538904, + 0.0019339170103194192, + 0.001794083000277169, + 0.0017909589951159433, + 0.0017983750003622845, + 0.0018620410119183362, + 0.0018005419988185167, + 0.0018732089956756681, + 0.0017688329971861094, + 0.0018847920000553131, + 0.0017721250042086467, + 0.0019318330014357343, + 0.001997290994040668, + 0.001923750009154901, + 0.0017737500020302832, + 0.0017790410056477413, + 0.0017652920068940148, + 0.001859084004536271, + 0.001827208005124703, + 0.0017814589955378324, + 0.0019464590004645288, + 0.0018618330068420619, + 0.0018413749930914491, + 0.001853666006354615, + 0.0018954999977722764, + 0.0018711669981712475, + 0.001851958004408516, + 0.0018728750001173466, + 0.0018887920014094561, + 0.001937875000294298, + 0.0019261249981354922, + 0.0019022089982172474 + ] + }, + { + "forward": 0.006161375000374392, + "prefill": 0.0061474590038415045, + "decode_steps": [ + 0.002129665997927077, + 0.001768041998730041, + 0.0018093329999828711, + 0.001929541991557926, + 0.0018717079947236925, + 0.0017902090039569885, + 0.0022719999979017302, + 0.0018743340042419732, + 0.0019129580032313243, + 0.004530291000264697, + 0.006872084006317891, + 0.002359207996050827, + 0.0021008329931646585, + 0.003442707995418459, + 0.0019238329987274483, + 0.0019342089944984764, + 0.0018741249950835481, + 0.002236458007246256, + 0.002101125006447546, + 0.002037875005044043, + 0.0018909579957835376, + 0.0019976250041509047, + 0.0019269170006737113, + 0.0018223330116597936, + 0.0018005410092882812, + 0.001823250000597909, + 0.001876374997664243, + 0.0018435000092722476, + 0.0018637499888427556, + 0.0018670419958652928, + 0.0018741670064628124, + 0.0017822910012910143 + ] + }, + { + "forward": 0.006189541993080638, + "prefill": 0.006129250003141351, + "decode_steps": [ + 0.0020075419888598844, + 0.0017789590056054294, + 0.001841500008595176, + 0.0017471669998485595, + 0.0017939999961527064, + 0.0017817500047385693, + 0.0017787919932743534, + 0.0020023329998366535, + 0.001864749996457249, + 0.0017585000023245811, + 0.0018419170082779601, + 0.0018202909996034577, + 0.0018112910038325936, + 0.0018249999993713573, + 0.0017958750104298815, + 0.0018068750068778172, + 0.0020949579920852557, + 0.0018614159926073626, + 0.001901334006106481, + 0.0017934579955181107, + 0.0018741250096354634, + 0.0018161250045523047, + 0.0017429169965907931, + 0.0018425830057822168, + 0.0019419579912209883, + 0.001908959005959332, + 0.0018016669928329065, + 0.0017980000120587647, + 0.0018308750004507601, + 0.001792750001186505, + 0.0017955840012291446, + 0.0017680830060271546 + ] + }, + { + "forward": 0.006230042010429315, + "prefill": 0.006082333988160826, + "decode_steps": [ + 0.002084167004795745, + 0.0017985839949687943, + 0.0018305830017197877, + 0.0017608329944778234, + 0.0018099590088240802, + 0.0017236660060007125, + 0.0018220000056317076, + 0.0018049580103252083, + 0.0018917499983217567, + 0.0019871249969583005, + 0.0019913330033887178, + 0.0018504170002415776, + 0.0018045420001726598, + 0.0017430419975426048, + 0.001820333010982722, + 0.0017917919903993607, + 0.0017951249901670963, + 0.001808375003747642, + 0.0017763340001692995, + 0.0018016249960055575, + 0.001830292007070966, + 0.0018246249965159222, + 0.0019004999921889976, + 0.0018092919926857576, + 0.0018636669992702082, + 0.0018987920047948137, + 0.001895042005344294, + 0.0017741670017130673, + 0.0019978339987574145, + 0.0018279159994563088, + 0.0019385829946259037, + 0.0017879590013762936 + ] + }, + { + "forward": 0.007181542008765973, + "prefill": 0.006567875010659918, + "decode_steps": [ + 0.0021541669993894175, + 0.0018412499921396375, + 0.0017745420045685023, + 0.001983458991162479, + 0.0018986249924637377, + 0.0018034579989034683, + 0.0017616249970160425, + 0.0019191669998690486, + 0.0017407080013072118, + 0.0022075830056564882, + 0.001901124996948056, + 0.0030920830031391233, + 0.0017901669925777242, + 0.0018620000046212226, + 0.002568166994024068, + 0.001989292009966448, + 0.001933541992912069, + 0.001763834006851539, + 0.0019667080050567165, + 0.001864833990111947, + 0.0018022079893853515, + 0.0017791659920476377, + 0.0019792920065810904, + 0.001862250006524846, + 0.0018181250052293763, + 0.0017575419915374368, + 0.0021259580098558217, + 0.0018601670017233118, + 0.0017610420036362484, + 0.0018114169943146408, + 0.0019404580089030787, + 0.0018530829984229058 + ] + }, + { + "forward": 0.006042958004400134, + "prefill": 0.0062295000097947195, + "decode_steps": [ + 0.00230491699767299, + 0.0020427499985089526, + 0.0018344589916523546, + 0.001995584010728635, + 0.001949124998645857, + 0.0019673339993460104, + 0.0017634160030866042, + 0.0017358750046696514, + 0.0017957919917535037, + 0.0018149170064134523, + 0.0017507500015199184, + 0.001764540997100994, + 0.0019376669952180237, + 0.0018021669966401532, + 0.0017938749952008948, + 0.0017800839996198192, + 0.0017172499938169494, + 0.0018101660098182037, + 0.0018230830028187484, + 0.001814082992495969, + 0.001844582991907373, + 0.0019449579995125532, + 0.0017311249976046383, + 0.0018837919924408197, + 0.0017213340033777058, + 0.0018109589873347431, + 0.0018119169981218874, + 0.0017881669919006526, + 0.0018898749985964969, + 0.0018572080007288605, + 0.0017660000012256205, + 0.0018949160003103316 + ] + }, + { + "forward": 0.008699042009538971, + "prefill": 0.008141291997162625, + "decode_steps": [ + 0.0020540419936878607, + 0.001999750005779788, + 0.001876708003692329, + 0.0017488329904153943, + 0.0017704589990898967, + 0.0020114999933866784, + 0.0017788750119507313, + 0.00197387499792967, + 0.0018357499939156696, + 0.0018453749944455922, + 0.00176133299828507, + 0.001774457996361889, + 0.0018279170035384595, + 0.001795249991118908, + 0.0019040409970330074, + 0.0018200420017819852, + 0.001746250010910444, + 0.0017544589936733246, + 0.0018785830034175888, + 0.0017497919907327741, + 0.002027583002927713, + 0.0019882500055246055, + 0.0017841250082710758, + 0.0019000829925062135, + 0.001880209005321376, + 0.0018229999986942858, + 0.002398583004833199, + 0.0017610409995540977, + 0.0018068339995807037, + 0.0019078329933108762, + 0.0018730839947238564, + 0.0018413330108160153 + ] + }, + { + "forward": 0.0061280000081751496, + "prefill": 0.006196500005898997, + "decode_steps": [ + 0.0020284170022932813, + 0.0019109579880023375, + 0.0018394580110907555, + 0.0018636250024428591, + 0.001900584000395611, + 0.0018169999966630712, + 0.0019292079959996045, + 0.001918250010930933, + 0.001915041997563094, + 0.001900165996630676, + 0.0019399589946260676, + 0.0017639580037212, + 0.001858625008026138, + 0.0019600409868871793, + 0.0018347080040257424, + 0.0017726250080158934, + 0.0018127920047845691, + 0.0017816670006141067, + 0.0017435829940950498, + 0.0018319170048926026, + 0.001838333992054686, + 0.001781541999662295, + 0.0018317080102860928, + 0.0018514999974286184, + 0.0018526670028222725, + 0.0018199589976575226, + 0.0018248750129714608, + 0.0017777500033844262, + 0.0018105420022038743, + 0.0025668329908512533, + 0.0018868330080294982, + 0.001956542007974349 + ] + }, + { + "forward": 0.006165124999824911, + "prefill": 0.006147541993414052, + "decode_steps": [ + 0.002070250004180707, + 0.0017874580080388114, + 0.001857166993431747, + 0.0019097080075880513, + 0.0018925420008599758, + 0.0019088749977527186, + 0.0018893329979619011, + 0.0018900830036727712, + 0.0018610839906614274, + 0.0017656669951975346, + 0.0018077090062433854, + 0.0018555419956101105, + 0.0017593749944353476, + 0.001760750004905276, + 0.0017793749866541475, + 0.0017728749953676015, + 0.0018065829935949296, + 0.001773541996954009, + 0.001732833989080973, + 0.001773541996954009, + 0.001774457996361889, + 0.0017205000040121377, + 0.0018994169950019568, + 0.0017667920037638396, + 0.0017844999965745956, + 0.002040959006990306, + 0.0017933750059455633, + 0.0029056250059511513, + 0.0018939590081572533, + 0.0017693749978207052, + 0.001922333991387859, + 0.0018391249905107543 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 6.172708504891489, + "p95": 8.01616700919112, + "min": 6.042958004400134, + "max": 8.69904200953897, + "iqr": 0.07316700794035569, + "mean": 6.511979304195847, + "cv": 0.12820103679998499 + }, + "prefill_ms": { + "count": 10, + "median": 6.1720209996565245, + "p95": 7.433254353236405, + "min": 6.082333988160826, + "max": 8.141291997162625, + "iqr": 0.09266701090382412, + "mean": 6.402229300874751, + "cv": 0.0977529166549223 + }, + "decode_ms_per_step": { + "count": 320, + "median": 1.8387294912827201, + "p95": 2.1261434092593845, + "min": 1.7098340031225234, + "max": 6.872084006317891, + "iqr": 0.120582746603759, + "mean": 1.898882850218797, + "cv": 0.1898671986207769 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 536.4629632154601, + "p95": 548.2277818274168, + "min": 449.8217510263409, + "max": 551.6702640339541, + "iqr": 8.986193179003635, + "mean": 528.2686952792742, + "cv": 0.05526709713629554 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 486.3603604177748, + "p95": 495.662391263667, + "min": 414.0425157928686, + "max": 498.100687202091, + "iqr": 17.522432590379537, + "mean": 477.6781557231372, + "cv": 0.05125673637609462 + }, + "memory": { + "peak_extra_bytes": 11804672, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "reference", + "batch": 1, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.028168042001198046, + "prefill": 0.020586667000316083, + "decode_steps": [ + 0.006058332990505733, + 0.0051486659940565005, + 0.0052582079952117056, + 0.0051557080005295575, + 0.005290208995575085, + 0.004935790988383815, + 0.004574667007545941, + 0.004322584005421959, + 0.004047249996801838, + 0.004322666994994506, + 0.004203707998385653, + 0.0043295420036884025, + 0.004293833990232088, + 0.003963541996199638, + 0.004410416004247963, + 0.00433970800077077, + 0.004494791995966807, + 0.004341333013144322, + 0.00432945899956394, + 0.00497620900569018, + 0.0043398340058047324, + 0.004339582999818958, + 0.004310999996960163, + 0.004272833000868559, + 0.004341833002399653, + 0.00448550000146497, + 0.004507499994360842, + 0.0042783340031746775, + 0.004336874990258366, + 0.004355458004283719, + 0.004313459008699283, + 0.004419874996528961 + ] + }, + { + "forward": 0.019249457996920682, + "prefill": 0.016020374998333864, + "decode_steps": [ + 0.004862166999373585, + 0.004996999996365048, + 0.005005875005736016, + 0.004441833996679634, + 0.0040421669982606545, + 0.003771666990360245, + 0.0043082079937448725, + 0.004258750006556511, + 0.004327999995439313, + 0.004260749992681667, + 0.004225292010232806, + 0.0042716250027297065, + 0.004229124999255873, + 0.004536499996902421, + 0.004343332999269478, + 0.0042463329882593825, + 0.004079999998793937, + 0.004576999999699183, + 0.004376625001896173, + 0.004278459004126489, + 0.004302792003727518, + 0.0043588749977061525, + 0.004328291994170286, + 0.004055500001413748, + 0.0042362079984741285, + 0.004328124996391125, + 0.004244374998961575, + 0.004275499988580123, + 0.0042312919977121055, + 0.004278209002222866, + 0.004242167007760145, + 0.004103041996131651 + ] + }, + { + "forward": 0.018637250002939254, + "prefill": 0.01607787501416169, + "decode_steps": [ + 0.00514479199773632, + 0.0047139169910224155, + 0.004712082998594269, + 0.004488457998377271, + 0.004594041995005682, + 0.004582249996019527, + 0.004963917002896778, + 0.00390458399488125, + 0.004116791998967528, + 0.0044150419998914, + 0.00427316699642688, + 0.004421250006998889, + 0.004730957996798679, + 0.004398999997647479, + 0.004490708000957966, + 0.004496124995057471, + 0.004415000003064051, + 0.0037853749963687733, + 0.004411000001709908, + 0.004360459002782591, + 0.004314042002079077, + 0.004372875002445653, + 0.004267875003279187, + 0.004465582998818718, + 0.004280916997231543, + 0.004312749995733611, + 0.004422666010214016, + 0.004356709003332071, + 0.004297250008676201, + 0.004301749999285676, + 0.004313041004934348, + 0.004292500001611188 + ] + }, + { + "forward": 0.020662208000430837, + "prefill": 0.016196500000660308, + "decode_steps": [ + 0.00487954200070817, + 0.004614417004631832, + 0.004462582990527153, + 0.00477750001300592, + 0.0040387090120930225, + 0.004409291999763809, + 0.005109208999783732, + 0.003853166999761015, + 0.004428707994520664, + 0.004362000006949529, + 0.0038838330074213445, + 0.003747042006580159, + 0.004399665995151736, + 0.004281665998860262, + 0.004551209000055678, + 0.00409225000475999, + 0.004542875001789071, + 0.005123916998854838, + 0.004317292012274265, + 0.004374540993012488, + 0.004323708009906113, + 0.003869625012157485, + 0.0038536669890163466, + 0.004268457996658981, + 0.005866709005204029, + 0.004315124999266118, + 0.004339541002991609, + 0.0042569579964037985, + 0.0039036659873090684, + 0.004458875002455898, + 0.004362209001556039, + 0.0051410419982858 + ] + }, + { + "forward": 0.01920491700002458, + "prefill": 0.017136916998424567, + "decode_steps": [ + 0.005091166996862739, + 0.004825790994800627, + 0.004583333007758483, + 0.004364834006992169, + 0.0045163750037318096, + 0.004399167009978555, + 0.0044180410041008145, + 0.004261249996488914, + 0.004281792003894225, + 0.004300082990084775, + 0.0043634170142468065, + 0.004267458003596403, + 0.004288249998353422, + 0.004290708006010391, + 0.0037484170024981722, + 0.004279583998140879, + 0.004417791002197191, + 0.004329333009081893, + 0.004354375007096678, + 0.0043409999925643206, + 0.004388792003737763, + 0.005786207999335602, + 0.004919208993669599, + 0.004175708003458567, + 0.004627250003977679, + 0.0046167499967850745, + 0.004432583999005146, + 0.004389875000924803, + 0.004034750003484078, + 0.003960000001825392, + 0.004358375008450821, + 0.004484457997023128 + ] + }, + { + "forward": 0.018533209004090168, + "prefill": 0.016313000000081956, + "decode_steps": [ + 0.004508457990596071, + 0.004417083997395821, + 0.004415500006871298, + 0.004650916991522536, + 0.004619042010745034, + 0.004216458008158952, + 0.004317290993640199, + 0.0036768330028280616, + 0.004463749995920807, + 0.004199874994810671, + 0.004255083011230454, + 0.004191750005702488, + 0.004222542003844865, + 0.004340625004260801, + 0.004297791994758882, + 0.004514625004958361, + 0.0044152080081403255, + 0.004115333998925053, + 0.004326166992541403, + 0.003776375000597909, + 0.004201624993584119, + 0.004262040994944982, + 0.003794415999436751, + 0.0038340419996529818, + 0.004405083003803156, + 0.004215332999592647, + 0.0037528329994529486, + 0.0038082079990999773, + 0.0041592920024413615, + 0.004088000001502223, + 0.0041406249947613105, + 0.0038733750116080046 + ] + }, + { + "forward": 0.02032770799996797, + "prefill": 0.017122459001257084, + "decode_steps": [ + 0.005590790999121964, + 0.005012624998926185, + 0.005868833002750762, + 0.004954416988766752, + 0.004617790997144766, + 0.004306292001274414, + 0.005072708008810878, + 0.005099249989143573, + 0.004409957997268066, + 0.004927291010972112, + 0.0046775419905316085, + 0.004724250000435859, + 0.004450666994671337, + 0.004367665998870507, + 0.0047738330031279474, + 0.004519541005720384, + 0.004386791988508776, + 0.004305666996515356, + 0.004918166989227757, + 0.004640125000150874, + 0.004319041996495798, + 0.004290000011678785, + 0.004750709005747922, + 0.004743291006889194, + 0.00450141700275708, + 0.005374582993681543, + 0.005288500004098751, + 0.004878457999438979, + 0.00464608300535474, + 0.005111999998916872, + 0.004736000002594665, + 0.005112582992296666 + ] + }, + { + "forward": 0.01961970899719745, + "prefill": 0.01604600000428036, + "decode_steps": [ + 0.004950375005137175, + 0.004553999999188818, + 0.004478374990867451, + 0.004232332998071797, + 0.004362166000646539, + 0.004306207993067801, + 0.004044917004648596, + 0.004446374994586222, + 0.004842374997679144, + 0.004653208001400344, + 0.004375000004074536, + 0.003945958000258543, + 0.0038576249935431406, + 0.004414541996084154, + 0.004726541010313667, + 0.004724250000435859, + 0.004449291009223089, + 0.004415292001795024, + 0.004377290999400429, + 0.004323207991546951, + 0.004362542007584125, + 0.004342250002082437, + 0.004349458002252504, + 0.004324542009271681, + 0.00437804200919345, + 0.004261624999344349, + 0.0044003750081174076, + 0.004321542000980116, + 0.004324166991864331, + 0.004335916004492901, + 0.004252333994372748, + 0.0043326249870005995 + ] + }, + { + "forward": 0.018637958011822775, + "prefill": 0.017261749992030673, + "decode_steps": [ + 0.00519470899598673, + 0.004587415998685174, + 0.004444707999937236, + 0.004398917008074932, + 0.004521082999417558, + 0.0043125829979544505, + 0.004451749991858378, + 0.004275458006304689, + 0.004244292009389028, + 0.004325791000155732, + 0.004261875001247972, + 0.004248083001584746, + 0.004238291003275663, + 0.004177541006356478, + 0.004269750003004447, + 0.004346208006609231, + 0.0042124169995076954, + 0.0047881250065984204, + 0.004610874995705672, + 0.004544166993582621, + 0.0040689580055186525, + 0.004062082996824756, + 0.0044535420020110905, + 0.004400040998007171, + 0.0043226670095464215, + 0.004095833006431349, + 0.00441879199934192, + 0.004077582998434082, + 0.004505583012360148, + 0.0043672080064425245, + 0.004341291991295293, + 0.0043150420096935704 + ] + }, + { + "forward": 0.019499791000271216, + "prefill": 0.01608762500109151, + "decode_steps": [ + 0.0055737920047249645, + 0.004380125013994984, + 0.004768416009028442, + 0.004528417004621588, + 0.003920290997484699, + 0.00443729200924281, + 0.004233708008541726, + 0.00433970800077077, + 0.003982374997576699, + 0.004568292002659291, + 0.004936583005473949, + 0.004493957996601239, + 0.0043700829992303625, + 0.004401209007482976, + 0.004436290997546166, + 0.004390457994304597, + 0.004469916006200947, + 0.004282540990971029, + 0.004177625000011176, + 0.00434458399831783, + 0.004375375006929971, + 0.004312500008381903, + 0.004336666999734007, + 0.004427584004588425, + 0.00434083299478516, + 0.00425895799708087, + 0.004257874999893829, + 0.004342833010014147, + 0.004356749996077269, + 0.0043432919919723645, + 0.004258082990418188, + 0.0043728339951485395 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 19.37462449859595, + "p95": 24.790416700852795, + "min": 18.533209004090168, + "max": 28.168042001198046, + "iqr": 1.3710104904021136, + "mean": 20.254025001486298, + "cv": 0.14164235588562632 + }, + "prefill_ms": { + "count": 10, + "median": 16.254750000371132, + "p95": 19.090454346587645, + "min": 16.020374998333864, + "max": 20.586667000316083, + "iqr": 1.0529899882385507, + "mean": 16.88491680106381, + "cv": 0.08259422915267643 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.35672949970467, + "p95": 5.1131496926245745, + "min": 3.6768330028280616, + "max": 6.058332990505733, + "iqr": 0.238843491388252, + "mean": 4.428952853595547, + "cv": 0.07930802999039399 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 226.76691344264026, + "p95": 234.90616548850258, + "min": 208.63641034339403, + "max": 237.9614166375277, + "iqr": 2.818016165030116, + "mean": 226.0219889614989, + "cv": 0.03342249666337236 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 203.46546358964756, + "p95": 209.9565086966274, + "min": 187.68402090189352, + "max": 212.21765979803948, + "iqr": 2.557474288566027, + "mean": 201.97018549421236, + "cv": 0.034227415816356924 + }, + "memory": { + "peak_extra_bytes": 55320576, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "auto", + "batch": 1, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.020097667002119124 + }, + { + "forward": 0.020183916989481077 + }, + { + "forward": 0.02157299999089446 + }, + { + "forward": 0.0201939590042457 + }, + { + "forward": 0.0202092499966966 + }, + { + "forward": 0.020116791012696922 + }, + { + "forward": 0.02144666700041853 + }, + { + "forward": 0.021754584013251588 + }, + { + "forward": 0.020895209003356285 + }, + { + "forward": 0.024031624998315237 + } + ], + "forward_ms": { + "count": 10, + "median": 20.552229500026442, + "p95": 23.00695655503659, + "min": 20.097667002119124, + "max": 24.031624998315237, + "iqr": 1.3549892501032446, + "mean": 21.050266901147552, + "cv": 0.058843887976983505 + }, + "memory": { + "peak_extra_bytes": 29376512, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "sdpa", + "batch": 1, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.013969167004688643, + "prefill": 0.014014500004122965, + "decode_steps": [ + 0.0028363749879645184, + 0.0021545000054175034, + 0.0020382079965202138, + 0.001934625004651025, + 0.0025127919943770394, + 0.0020305420039221644, + 0.0020116670057177544, + 0.0020876670023426414, + 0.0024141670000972226, + 0.0020769160037161782, + 0.002047999994829297, + 0.00207537499954924, + 0.00211650000710506, + 0.0019811660022241995, + 0.0019844170019496232, + 0.0023120000114431605, + 0.0020223330066073686, + 0.0023885840055299923, + 0.0020473339973250404, + 0.0023769580002408475, + 0.004794374995981343, + 0.004695957992225885, + 0.005621875010547228, + 0.0031707089947303757, + 0.00350287499895785, + 0.0020687090000137687, + 0.00227554100274574, + 0.0020459580118767917, + 0.0020238330034771934, + 0.0021444169979076833, + 0.002073124996968545, + 0.0024581670004408807 + ] + }, + { + "forward": 0.012974209006642923, + "prefill": 0.013907417000154965, + "decode_steps": [ + 0.0023952500050654635, + 0.0020740830077556893, + 0.0020741670014103875, + 0.0019840840104734525, + 0.002054083000984974, + 0.002359707999858074, + 0.002162916003726423, + 0.0021390840120147914, + 0.0020745410001836717, + 0.002149874999304302, + 0.0020162500004516914, + 0.0021312500030035153, + 0.002015999998548068, + 0.002085834013996646, + 0.002003042012802325, + 0.002059749996988103, + 0.0021042500011390075, + 0.0019962499936809763, + 0.002117541996994987, + 0.002226249998784624, + 0.0020518329984042794, + 0.002066082990495488, + 0.0020812500006286427, + 0.002104915998643264, + 0.0020083750132471323, + 0.0020240420126356184, + 0.002057290999800898, + 0.002071249997243285, + 0.002031583004281856, + 0.0020191249932395294, + 0.002095042000291869, + 0.0020427079871296883 + ] + }, + { + "forward": 0.01303354199626483, + "prefill": 0.012957208004081622, + "decode_steps": [ + 0.0024689159909030423, + 0.0022772079973947257, + 0.002121708996128291, + 0.001967417003470473, + 0.002163375000236556, + 0.0023036250058794394, + 0.002152917004423216, + 0.002005500005907379, + 0.0020641660084947944, + 0.001997083003516309, + 0.0021027500042691827, + 0.0019997079944005236, + 0.0020745410001836717, + 0.0020032909960718825, + 0.002081707993056625, + 0.0020944999996572733, + 0.0020367910037748516, + 0.002099249992170371, + 0.001969125005416572, + 0.0021028750052209944, + 0.0020476670033531263, + 0.0020549999899230897, + 0.0020785000087926164, + 0.002035625002463348, + 0.001973833001102321, + 0.0020572500070557, + 0.002014083001995459, + 0.0021163750061532483, + 0.0020172499935142696, + 0.0020901249954476953, + 0.0020502910047071055, + 0.002087375003611669 + ] + }, + { + "forward": 0.013971917011076584, + "prefill": 0.012858874993980862, + "decode_steps": [ + 0.0024104590120259672, + 0.0021099160076119006, + 0.002028917006100528, + 0.0019615410128608346, + 0.0021126670035300776, + 0.0020414169994182885, + 0.0021303749963408336, + 0.002006457987590693, + 0.0020204999891575426, + 0.0021255839965306222, + 0.0019259590044384822, + 0.0020573750080075115, + 0.002039792001596652, + 0.0019930419948650524, + 0.0021039169951109216, + 0.0020150409982306883, + 0.0020743749919347465, + 0.001999250001972541, + 0.0020271669927751645, + 0.0020857920026173815, + 0.002063374995486811, + 0.0019779170106630772, + 0.002106999992975034, + 0.0020001249940833077, + 0.0020118749962421134, + 0.0019317080004839227, + 0.0020619999995687976, + 0.002104500003042631, + 0.0019783750030910596, + 0.002262458991026506, + 0.0020071670005563647, + 0.00207554099324625 + ] + }, + { + "forward": 0.012949958007084206, + "prefill": 0.012912791993585415, + "decode_steps": [ + 0.002289500000188127, + 0.0021417080133687705, + 0.002042499996605329, + 0.002100541998515837, + 0.0020053750049555674, + 0.0020082080009160563, + 0.002014457990298979, + 0.002025665991823189, + 0.0020776250021299347, + 0.0021156250004423782, + 0.001920333001180552, + 0.002178666996769607, + 0.002004750000196509, + 0.001958292006747797, + 0.002249124998343177, + 0.002136041992343962, + 0.0021695840114261955, + 0.002170249994378537, + 0.002055582997854799, + 0.0019790829974226654, + 0.0024735000042710453, + 0.0021085830085212365, + 0.002055000004475005, + 0.0020813330047531053, + 0.002068917005090043, + 0.0021635419980157167, + 0.0020193749951431528, + 0.0020640839939005673, + 0.002131207991624251, + 0.002029082999797538, + 0.002068500005407259, + 0.0020567910105455667 + ] + }, + { + "forward": 0.012918124994030222, + "prefill": 0.012943458001245745, + "decode_steps": [ + 0.002355917007662356, + 0.0022164159890962765, + 0.0019969169952673838, + 0.0020988750038668513, + 0.0020246670028427616, + 0.0020740000036312267, + 0.0020042499963892624, + 0.002159708004910499, + 0.002253667000331916, + 0.00202454200189095, + 0.001965542003745213, + 0.0020184169989079237, + 0.0020403750095283613, + 0.0019957919867010787, + 0.0019096670002909377, + 0.002015416990616359, + 0.0019247499876655638, + 0.002595875004772097, + 0.003217917008441873, + 0.002166999998735264, + 0.001993458005017601, + 0.003413915997953154, + 0.002272083002026193, + 0.003117540996754542, + 0.002201875002356246, + 0.0021712080051656812, + 0.00203145899286028, + 0.003414666003664024, + 0.002046041001449339, + 0.0021307499991962686, + 0.002416167000774294, + 0.002058208003290929 + ] + }, + { + "forward": 0.013016249999054708, + "prefill": 0.012957625003764406, + "decode_steps": [ + 0.002297832994372584, + 0.0020381250069476664, + 0.002024166999035515, + 0.0019809169898508117, + 0.0020389579876791686, + 0.002034208009717986, + 0.0021278330095810816, + 0.0019900420011254027, + 0.0024046249891398475, + 0.0020035839988850057, + 0.0027039999986300245, + 0.0022424999915529042, + 0.002075915996101685, + 0.002178874987293966, + 0.002055041986750439, + 0.002003582994802855, + 0.0019614159973571077, + 0.0019928749970858917, + 0.0019372090027900413, + 0.002100165991578251, + 0.002228875004220754, + 0.0020278750016586855, + 0.002170541003579274, + 0.001983374997507781, + 0.0021228749974397942, + 0.0021037080005044118, + 0.0021355419885367155, + 0.0023365830129478127, + 0.0022432919940911233, + 0.0022082499926909804, + 0.0022307499893940985, + 0.0023786670062690973 + ] + }, + { + "forward": 0.012940417000208981, + "prefill": 0.013824624998960644, + "decode_steps": [ + 0.002347957997699268, + 0.001997166997171007, + 0.0020979590044589713, + 0.0020038750080857426, + 0.002162415999919176, + 0.0020308750099502504, + 0.0020031669992022216, + 0.002014458004850894, + 0.002036332996794954, + 0.0026919579977402464, + 0.0019825420022243634, + 0.00208395799563732, + 0.0020332499989308417, + 0.002102833997923881, + 0.0020217499986756593, + 0.002094625000609085, + 0.0027289579884381965, + 0.0020986659947084263, + 0.0020550829940475523, + 0.002282000001287088, + 0.002277124993270263, + 0.002119208002113737, + 0.0020219579892000183, + 0.0021656669996446, + 0.0019693750073201954, + 0.0020976249943487346, + 0.0021569999953499064, + 0.0020876250055152923, + 0.0020114579965593293, + 0.0020953749917680398, + 0.0022507919929921627, + 0.0021577909938059747 + ] + }, + { + "forward": 0.014813625006354414, + "prefill": 0.014573042004485615, + "decode_steps": [ + 0.0022872089903103188, + 0.0022659999958705157, + 0.0021065000037197024, + 0.0020158330007689074, + 0.0020549170003505424, + 0.0034347080072620884, + 0.0019955419993493706, + 0.002016416998230852, + 0.0019805410120170563, + 0.0019923340005334467, + 0.001968915996258147, + 0.0020627079939004034, + 0.0031436249992111698, + 0.0020014589972561225, + 0.0020057080109836534, + 0.0020160830026725307, + 0.002037124999333173, + 0.0019718330004252493, + 0.002004041991312988, + 0.0031991669966373593, + 0.0020392500009620562, + 0.002085665997583419, + 0.0020552909991238266, + 0.002023459004703909, + 0.0020070830069016665, + 0.0020185420144116506, + 0.002050292008789256, + 0.002020707994233817, + 0.00193329100147821, + 0.00205650000134483, + 0.001920791997690685, + 0.0020035000052303076 + ] + }, + { + "forward": 0.016164165994268842, + "prefill": 0.013174291991163045, + "decode_steps": [ + 0.0031032080005388707, + 0.0030545829940820113, + 0.002998167008627206, + 0.002873207995435223, + 0.0024625000078231096, + 0.004414249997353181, + 0.003231666996725835, + 0.003131499994196929, + 0.003129958000499755, + 0.002908833004767075, + 0.0030716249893885106, + 0.003078167006606236, + 0.003260291996411979, + 0.003159875006531365, + 0.0032641249999869615, + 0.003511624992825091, + 0.0032512079924345016, + 0.003324624995002523, + 0.003410457997233607, + 0.0027589999954216182, + 0.0031972919969120994, + 0.0028763330046785995, + 0.0032323339983122423, + 0.0033031670027412474, + 0.0027275829925201833, + 0.0028231659962330014, + 0.003083667004830204, + 0.0032738750014686957, + 0.0032583749998593703, + 0.0027053749945480376, + 0.002686458989046514, + 0.003296209004474804 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 13.024895997659769, + "p95": 15.556422549707348, + "min": 12.918124994030222, + "max": 16.164165994268842, + "iqr": 1.015208752505714, + "mean": 13.675137601967435, + "cv": 0.07951598811306526 + }, + "prefill_ms": { + "count": 10, + "median": 13.065958497463726, + "p95": 14.321698104322422, + "min": 12.858874993980862, + "max": 14.573042004485615, + "iqr": 0.9398234979016706, + "mean": 13.412383399554528, + "cv": 0.04563157021164109 + }, + "decode_ms_per_step": { + "count": 320, + "median": 2.0780625054612756, + "p95": 3.2584708496870007, + "min": 1.9096670002909377, + "max": 5.621875010547228, + "iqr": 0.22453150450019166, + "mean": 2.2635484155671293, + "cv": 0.21138718101793263 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 468.3738278689534, + "p95": 483.1279151657855, + "min": 320.43993532774465, + "max": 485.94121952009493, + "iqr": 30.18606185892463, + "mean": 448.5278466511909, + "cv": 0.11550752392071785 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 391.6253101160001, + "p95": 404.3566291549558, + "min": 283.09314411595585, + "max": 406.55332979544346, + "iqr": 22.0551482179535, + "mean": 376.97104772587534, + "cv": 0.10135585215481184 + }, + "memory": { + "peak_extra_bytes": 23609344, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "reference", + "batch": 1, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.05529712499992456, + "prefill": 0.05419162500766106, + "decode_steps": [ + 0.007528041998739354, + 0.006584582995856181, + 0.006553917002747767, + 0.005509958995389752, + 0.005480041989358142, + 0.004913415992632508, + 0.00488158299413044, + 0.004824209012440406, + 0.004712416994152591, + 0.004433875001268461, + 0.004487250000238419, + 0.0045358339993981645, + 0.004316416001529433, + 0.005120334011735395, + 0.004516582994256169, + 0.004856832994846627, + 0.004802374998689629, + 0.004594041005475447, + 0.004618833001586609, + 0.0041232920048059896, + 0.004484125005546957, + 0.004447582992725074, + 0.0047552079922752455, + 0.004642708008759655, + 0.006132708003860898, + 0.006872041994938627, + 0.005402875001891516, + 0.004656208009691909, + 0.004599499996402301, + 0.004794541004230268, + 0.004767209000419825, + 0.004667540997616015 + ] + }, + { + "forward": 0.05461516699870117, + "prefill": 0.05156087499926798, + "decode_steps": [ + 0.006590959004824981, + 0.005038375005824491, + 0.005188041992369108, + 0.004995000010239892, + 0.004941249993862584, + 0.005280416997266002, + 0.0051099999982398, + 0.004914916993584484, + 0.005423000009614043, + 0.004752959008328617, + 0.0047117500071180984, + 0.005172457997105084, + 0.004672417009714991, + 0.004273708007531241, + 0.004564874994684942, + 0.005461833003209904, + 0.004820916990865953, + 0.004464625002583489, + 0.005186374997720122, + 0.0064550000097369775, + 0.0050719159917207435, + 0.005089416998089291, + 0.0055631249997531995, + 0.0046352499921340495, + 0.00605116700171493, + 0.0057010000018635765, + 0.004770083993207663, + 0.005633082997519523, + 0.0053837079904042184, + 0.004763749995618127, + 0.005041917000198737, + 0.0046539169998141006 + ] + }, + { + "forward": 0.05351029199664481, + "prefill": 0.05229616700671613, + "decode_steps": [ + 0.006112500006565824, + 0.004820375004783273, + 0.0048498329997528344, + 0.004479916009586304, + 0.0047847499954514205, + 0.004725500009953976, + 0.0047270000068238005, + 0.004619125000317581, + 0.004661082988604903, + 0.004719667005701922, + 0.0046264160046121106, + 0.004983375009032898, + 0.005219124999712221, + 0.005380749993491918, + 0.004554583996650763, + 0.004775166991748847, + 0.00523583400354255, + 0.005185292000533082, + 0.005391208003857173, + 0.005184374997043051, + 0.004576583000016399, + 0.004767249993165024, + 0.004867291005211882, + 0.0044032500009052455, + 0.0048314169980585575, + 0.005065374993137084, + 0.004535667001619004, + 0.004543874994851649, + 0.004706874999101274, + 0.004856250001466833, + 0.004779875001986511, + 0.004760875002830289 + ] + }, + { + "forward": 0.05445054099254776, + "prefill": 0.052692708995891735, + "decode_steps": [ + 0.005972041995846666, + 0.004914207995170727, + 0.004963459010468796, + 0.005114667001180351, + 0.004914374992949888, + 0.00463766600296367, + 0.004519833004451357, + 0.004620166000677273, + 0.0046017079876037315, + 0.004505832999711856, + 0.004635000004782341, + 0.004493832995649427, + 0.004644542001187801, + 0.00445525000395719, + 0.004387667009723373, + 0.0045380829978967085, + 0.004455333997611888, + 0.004461000004084781, + 0.0044944580004084855, + 0.00448254200455267, + 0.004262875008862466, + 0.00434887500887271, + 0.004555916006211191, + 0.004548834011075087, + 0.00489020899112802, + 0.0051070420013275, + 0.00447129100211896, + 0.004529584010015242, + 0.005020957993110642, + 0.0047731249942444265, + 0.004641915991669521, + 0.004722125013358891 + ] + }, + { + "forward": 0.05336570800864138, + "prefill": 0.05137595799169503, + "decode_steps": [ + 0.005463167006382719, + 0.00497712500509806, + 0.005394000007072464, + 0.004508708996581845, + 0.005033499997807667, + 0.00481841700093355, + 0.004845167000894435, + 0.0045160000008763745, + 0.005337750000762753, + 0.005927374993916601, + 0.004547791002551094, + 0.004932833006023429, + 0.004521374998148531, + 0.004728457992314361, + 0.0050534579931991175, + 0.00697566699818708, + 0.005264374995022081, + 0.005075833003502339, + 0.005734791004215367, + 0.004718958996818401, + 0.005252082992228679, + 0.0048685000074328855, + 0.004785457989783026, + 0.004481124997255392, + 0.004768957995111123, + 0.0048027909942902625, + 0.004490459003136493, + 0.004687874999945052, + 0.004973582996171899, + 0.004650083006708883, + 0.004965750005794689, + 0.00454925000667572 + ] + }, + { + "forward": 0.05600687500555068, + "prefill": 0.0510727910004789, + "decode_steps": [ + 0.006471708009485155, + 0.005009833999793045, + 0.004901166990748607, + 0.005175208003493026, + 0.004805584001587704, + 0.004707333995611407, + 0.004661333994590677, + 0.004572916004690342, + 0.004520167000009678, + 0.004405790998134762, + 0.004573749989503995, + 0.004539458008366637, + 0.004367540997918695, + 0.004521999988355674, + 0.004611124997609295, + 0.004453208006452769, + 0.0045275410084286705, + 0.0044847919925814494, + 0.004387125009088777, + 0.004609832991263829, + 0.004499207992921583, + 0.004523124996921979, + 0.004964124993421137, + 0.0046238340000854805, + 0.006021624998538755, + 0.004892084005405195, + 0.004660709004383534, + 0.004512042010901496, + 0.004666791995987296, + 0.004586625000229105, + 0.0045672080013900995, + 0.004643791005946696 + ] + }, + { + "forward": 0.05742870799440425, + "prefill": 0.05211024999152869, + "decode_steps": [ + 0.006278624990954995, + 0.004827249998925254, + 0.005225291999522597, + 0.004951040988089517, + 0.004717125004390255, + 0.0046603749942732975, + 0.00461374998849351, + 0.004484332996071316, + 0.004464124998776242, + 0.004558375003398396, + 0.004516875007539056, + 0.0043963750067632645, + 0.004800415990757756, + 0.00508383299165871, + 0.0050424170040059835, + 0.004629249990102835, + 0.004585916991345584, + 0.004597833001753315, + 0.004599584004608914, + 0.004903708992060274, + 0.004514250002102926, + 0.004471041000215337, + 0.004378833997179754, + 0.004285458999220282, + 0.006214707987965085, + 0.004725207996671088, + 0.0048985830071615055, + 0.004556416999548674, + 0.004555292005534284, + 0.004632124997442588, + 0.004740999997011386, + 0.004436874995008111 + ] + }, + { + "forward": 0.056744375004200265, + "prefill": 0.05106345799867995, + "decode_steps": [ + 0.006122708000475541, + 0.004831917001865804, + 0.005555166004342027, + 0.005193707998842001, + 0.005454958009067923, + 0.005065959005150944, + 0.004338333994382992, + 0.004729624997708015, + 0.005178915991564281, + 0.005749709001975134, + 0.0044940829975530505, + 0.004649915994377807, + 0.0047130000020843, + 0.004382915998576209, + 0.004150499997194856, + 0.004556833999231458, + 0.00455337499442976, + 0.004533583007287234, + 0.004863792011747137, + 0.0048251249972963706, + 0.004703208993305452, + 0.004616292004357092, + 0.00449745800870005, + 0.0044253329979255795, + 0.00469337499816902, + 0.004610167001374066, + 0.004649708003853448, + 0.004529583005933091, + 0.004571625002427027, + 0.004136750008910894, + 0.0058640409988584, + 0.00492441700771451 + ] + }, + { + "forward": 0.06143299999530427, + "prefill": 0.05497191699396353, + "decode_steps": [ + 0.006533250008942559, + 0.00495970799238421, + 0.004593624995322898, + 0.00457395899866242, + 0.004539959001704119, + 0.004652165996958502, + 0.0046243749966379255, + 0.004522959003224969, + 0.004531707993010059, + 0.004562582995276898, + 0.004512041996349581, + 0.004459749994566664, + 0.0049637500051176175, + 0.004586916009429842, + 0.004034500001580454, + 0.0041796669975155964, + 0.004509416990913451, + 0.004608999995980412, + 0.0044720000005327165, + 0.004699416997027583, + 0.004568082993500866, + 0.0044898750056745484, + 0.004604917005053721, + 0.004716124996775761, + 0.004716249997727573, + 0.0045447500015143305, + 0.004141500001423992, + 0.004639334001694806, + 0.004692792004789226, + 0.005017749994294718, + 0.004553499995381571, + 0.005830665992107242 + ] + }, + { + "forward": 0.061480374992243014, + "prefill": 0.0566669170075329, + "decode_steps": [ + 0.005909125000471249, + 0.004599874999257736, + 0.004792042003828101, + 0.004719666991150007, + 0.004802791998372413, + 0.004382708997582085, + 0.004536583001026884, + 0.004253416991559789, + 0.004597207996994257, + 0.014084291004110128, + 0.01763258400023915, + 0.016855791996931657, + 0.016059207991929725, + 0.009379916999023408, + 0.011301125006866641, + 0.005784125009085983, + 0.004949708003550768, + 0.004659583006286994, + 0.005771291995188221, + 0.0047992079926189035, + 0.005042374992626719, + 0.005397125001763925, + 0.006142541998997331, + 0.00435299999662675, + 0.0057589589996496215, + 0.005008915992220864, + 0.005142959009390324, + 0.004848082986427471, + 0.004579125001328066, + 0.004641624997020699, + 0.004796209002961405, + 0.004753165994770825 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 55.65200000273762, + "p95": 61.45905624362058, + "min": 53.36570800864138, + "max": 61.480374992243014, + "iqr": 2.765927252767142, + "mean": 56.433216598816216, + "cv": 0.052240408671288735 + }, + "prefill_ms": { + "count": 10, + "median": 52.20320849912241, + "p95": 55.904167001426686, + "min": 51.06345799867995, + "max": 56.6669170075329, + "iqr": 2.3947087611304596, + "mean": 52.80026669934159, + "cv": 0.03566211673297818 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.716187497251667, + "p95": 6.217903838114581, + "min": 4.034500001580454, + "max": 17.63258400023915, + "iqr": 0.48614549086778425, + "mean": 5.040657556037331, + "cv": 0.2839379144056481 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 206.52580030528094, + "p95": 213.82092412820955, + "min": 149.29945781198876, + "max": 213.8518628520408, + "iqr": 12.35116362541413, + "mean": 200.44106493842975, + "cv": 0.09550605661001585 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 154.87471807728275, + "p95": 158.06293514157522, + "min": 118.08063530219592, + "max": 158.12063736882513, + "iqr": 7.3634497187279635, + "mean": 150.496247208778, + "cv": 0.07949519578024605 + }, + "memory": { + "peak_extra_bytes": 213893120, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "auto", + "batch": 4, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.0038128330052131787 + }, + { + "forward": 0.003801166996709071 + }, + { + "forward": 0.003757500002393499 + }, + { + "forward": 0.0036942089936928824 + }, + { + "forward": 0.003764749999390915 + }, + { + "forward": 0.0037366670003393665 + }, + { + "forward": 0.00368249999883119 + }, + { + "forward": 0.004831709011341445 + }, + { + "forward": 0.0039621670002816245 + }, + { + "forward": 0.003547291998984292 + } + ], + "forward_ms": { + "count": 10, + "median": 3.761125000892207, + "p95": 4.440415106364525, + "min": 3.547291998984292, + "max": 4.831709011341445, + "iqr": 0.1050930077326484, + "mean": 3.8590794007177465, + "cv": 0.09270496776200707 + }, + "memory": { + "peak_extra_bytes": 5899264, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "sdpa", + "batch": 4, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.003227499997592531, + "prefill": 0.003344791999552399, + "decode_steps": [ + 0.002111833993694745, + 0.001986292001674883, + 0.002109708992065862, + 0.0018666660034796223, + 0.0018975830025738105, + 0.0020099579996895045, + 0.0018480829894542694, + 0.002059582999208942, + 0.0019581660017138347, + 0.001831583009334281, + 0.001917125002364628, + 0.001939958005095832, + 0.0019432499975664541, + 0.0018465839966665953, + 0.0017930419999174774, + 0.0018379999964963645, + 0.00185012498695869, + 0.0018644590018084273, + 0.0018461659929016605, + 0.0018121250031981617, + 0.003391624995856546, + 0.001774874996044673, + 0.0018649160047061741, + 0.0019502499926602468, + 0.0018943329923786223, + 0.0017779580084607005, + 0.0018609169928822666, + 0.001973082995391451, + 0.0019516250031301752, + 0.0019459580071270466, + 0.0019514160085236654, + 0.0019190420134691522 + ] + }, + { + "forward": 0.0032557499944232404, + "prefill": 0.003244999999878928, + "decode_steps": [ + 0.001975333012524061, + 0.0019508340046741068, + 0.0019000409956788644, + 0.0017412500019418076, + 0.0018523749895393848, + 0.001869665997219272, + 0.0018646249955054373, + 0.0018225419917143881, + 0.001821500001824461, + 0.0018009170016739517, + 0.001796416996512562, + 0.0018040829891106114, + 0.0018067500059260055, + 0.001817500000470318, + 0.0018012500077020377, + 0.001855249996879138, + 0.0018179159960709512, + 0.0019059580081375316, + 0.0018065000040223822, + 0.0017866669950308278, + 0.001785125001333654, + 0.0019074579904554412, + 0.0018417090032016858, + 0.001885500008938834, + 0.001855624999734573, + 0.0018484169995645061, + 0.001847459003329277, + 0.0019030419935006648, + 0.001899375012726523, + 0.0019075000018347055, + 0.0019529999990481883, + 0.001953917002538219 + ] + }, + { + "forward": 0.0032533330086153, + "prefill": 0.0033314580068690702, + "decode_steps": [ + 0.0020359170011943206, + 0.0019424170022830367, + 0.0018015419918810949, + 0.0019135839975206181, + 0.0019375420088181272, + 0.0018098750006174669, + 0.0018204579973826185, + 0.0018559579912107438, + 0.0019361660088179633, + 0.001957166998181492, + 0.0017833750025602058, + 0.002071374998195097, + 0.0019225419964641333, + 0.0019464999932097271, + 0.0019932080031139776, + 0.0018864169978769496, + 0.001924374999362044, + 0.0018403340072836727, + 0.0018532079993747175, + 0.0026367919926997274, + 0.001871208005468361, + 0.001788791996659711, + 0.0019604590052040294, + 0.001992374993278645, + 0.0019078330078627914, + 0.0018290000007255003, + 0.00190154199663084, + 0.0018678329943213612, + 0.0018770840106299147, + 0.0019802919996436685, + 0.0019053750002058223, + 0.0018707499984884635 + ] + }, + { + "forward": 0.003231249997043051, + "prefill": 0.0032972920016618446, + "decode_steps": [ + 0.0019133750029141083, + 0.0019512500002747402, + 0.0019100420031463727, + 0.0019116670009680092, + 0.0017914159980136901, + 0.0019010839896509424, + 0.0017572920041857287, + 0.001958291992195882, + 0.0018730829906417057, + 0.00188079199870117, + 0.0019516250031301752, + 0.0019231250043958426, + 0.0018364160059718415, + 0.0018900409922935069, + 0.0018846249877242371, + 0.0018954169936478138, + 0.0018115840066457167, + 0.0018425410089548677, + 0.0019120409997412935, + 0.0017969160107895732, + 0.002018208004301414, + 0.00298083400411997, + 0.0017984579899348319, + 0.0018512910028221086, + 0.001986250004847534, + 0.0019709159969352186, + 0.0019732499931706116, + 0.0018223329971078783, + 0.0018627500103320926, + 0.001938333996804431, + 0.001904959004605189, + 0.0019256250088801607 + ] + }, + { + "forward": 0.003383666989975609, + "prefill": 0.00336362500092946, + "decode_steps": [ + 0.0020740830077556893, + 0.0019380419980734587, + 0.0019429159874562174, + 0.0018024169985437766, + 0.0018120829918188974, + 0.0019009169918717816, + 0.0018591249972814694, + 0.001837875010096468, + 0.0018410000047879294, + 0.0018668749980861321, + 0.0018540000019129366, + 0.0017834580066846684, + 0.0018769169982988387, + 0.0019134590111207217, + 0.001937124994583428, + 0.0018629580008564517, + 0.001950250007212162, + 0.0019301249994896352, + 0.0018202920036856085, + 0.001881125004729256, + 0.0018696250044740736, + 0.0026666250050766394, + 0.001932958999532275, + 0.001965917006600648, + 0.0018986659997608513, + 0.001942040995345451, + 0.0019409170054132119, + 0.0018669999990379438, + 0.0018220839992864057, + 0.0018655830062925816, + 0.0018158750026486814, + 0.0019210419995943084 + ] + }, + { + "forward": 0.0032263330067507923, + "prefill": 0.003208249996532686, + "decode_steps": [ + 0.0020141249988228083, + 0.0018608340033097193, + 0.0018123339978046715, + 0.0018867499893531203, + 0.0018947080097859725, + 0.0019485419907141477, + 0.0018423750007059425, + 0.0019050830014748499, + 0.0018934169929707423, + 0.0019042080093640834, + 0.002061833001789637, + 0.0018082919996231794, + 0.001981666006031446, + 0.0018065409967675805, + 0.0018602079944685102, + 0.0019910410046577454, + 0.0019036669982597232, + 0.001865958998678252, + 0.002422958001261577, + 0.0019637910008896142, + 0.0020010839944006875, + 0.001884708006400615, + 0.0029125000000931323, + 0.001946457996382378, + 0.0019526669930201024, + 0.001938625006005168, + 0.001967417003470473, + 0.0018985839997185394, + 0.0018585000070743263, + 0.0018978329899255186, + 0.0018530420056777075, + 0.0019380000012461096 + ] + }, + { + "forward": 0.0032815830054460093, + "prefill": 0.0032419580093119293, + "decode_steps": [ + 0.0018365410069236532, + 0.00185500000952743, + 0.0017945419967873022, + 0.0018669999990379438, + 0.0018454160017427057, + 0.0018393750069662929, + 0.001898499991511926, + 0.0017815840110415593, + 0.001800750003894791, + 0.0018566249927971512, + 0.0019296249956823885, + 0.0018250830034958199, + 0.0021225000091362745, + 0.001885875011794269, + 0.001962499998626299, + 0.0018431249918648973, + 0.0018002910073846579, + 0.0023853750026319176, + 0.001866499995230697, + 0.0018156669975724071, + 0.001892375003080815, + 0.0019241669942857698, + 0.0019321250001667067, + 0.001839042000938207, + 0.0018857499962905422, + 0.0019707080064108595, + 0.001966415991773829, + 0.0018289580038981512, + 0.0019176670029992238, + 0.0019117080082651228, + 0.0019044999935431406, + 0.0018438339902786538 + ] + }, + { + "forward": 0.0032411669963039458, + "prefill": 0.004324875000747852, + "decode_steps": [ + 0.00195408400031738, + 0.001995040991459973, + 0.001883042001281865, + 0.0019373329996597022, + 0.001837542004068382, + 0.0018570840038591996, + 0.0019336249970365316, + 0.0018221250065835193, + 0.0018170000112149864, + 0.001913917003548704, + 0.0018758750084089115, + 0.0018491249938961118, + 0.0018154169956687838, + 0.0018812500056810677, + 0.0019088749977527186, + 0.001931249993504025, + 0.0018465000030118972, + 0.0019688329921336845, + 0.0019856669969158247, + 0.0019678749958984554, + 0.001897916998132132, + 0.0017851670127129182, + 0.0018390409968560562, + 0.0018357090011704713, + 0.0018782499973895028, + 0.0022054169967304915, + 0.0019437089940765873, + 0.001942457995028235, + 0.0018233330047223717, + 0.0019212920014979318, + 0.0018762920080916956, + 0.001857499999459833 + ] + }, + { + "forward": 0.0031285420118365437, + "prefill": 0.0031962909997673705, + "decode_steps": [ + 0.0020476250065257773, + 0.003203083004336804, + 0.0033641670015640557, + 0.0024301250086864457, + 0.002136916999006644, + 0.001948707998963073, + 0.0020589589985320345, + 0.0019752090011024848, + 0.0019280830019852147, + 0.002439167001284659, + 0.002368166999076493, + 0.002345499990042299, + 0.0019304169982206076, + 0.002853625002899207, + 0.0024108750076266006, + 0.0018635000014910474, + 0.00188079199870117, + 0.0023057500075083226, + 0.0025095839955611154, + 0.003004415993927978, + 0.002411209003184922, + 0.004522250004811212, + 0.0028341249999357387, + 0.003571166002075188, + 0.0030212499987101182, + 0.0036256669991416857, + 0.002828625001711771, + 0.002462459000525996, + 0.0025253750063711777, + 0.002851500001270324, + 0.002675833005923778, + 0.0019969580025644973 + ] + }, + { + "forward": 0.003132499987259507, + "prefill": 0.005023707999498583, + "decode_steps": [ + 0.002389832996414043, + 0.002333500000531785, + 0.0019363339961273596, + 0.0019381250021979213, + 0.0023353339929599315, + 0.0029935420025140047, + 0.002332874995772727, + 0.0030411250045290217, + 0.0020901669922750443, + 0.0019338330021128058, + 0.003135082995868288, + 0.0027502919983817264, + 0.002326834000996314, + 0.002477667003404349, + 0.002844832997652702, + 0.0024694580060895532, + 0.0025425000058021396, + 0.004305374997784384, + 0.002485958000761457, + 0.013369459004024975, + 0.010022583999671042, + 0.002770708000753075, + 0.00259445900155697, + 0.0024293329915963113, + 0.001959167013410479, + 0.0019015830039279535, + 0.001990082993870601, + 0.0021612080017803237, + 0.002076165998005308, + 0.0019687089952640235, + 0.0019446670048637316, + 0.0019261669949628413 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 3.2362084966734983, + "p95": 3.337729196937289, + "min": 3.1285420118365437, + "max": 3.383666989975609, + "iqr": 0.028520993510028347, + "mean": 3.236162499524653, + "cv": 0.022341847113494163 + }, + "prefill_ms": { + "count": 10, + "median": 3.3143750042654574, + "p95": 4.709233150060753, + "min": 3.1962909997673705, + "max": 5.023707999498583, + "iqr": 0.11619824363151565, + "mean": 3.5577249014750123, + "cv": 0.17250516819594663 + }, + "decode_ms_per_step": { + "count": 320, + "median": 1.910854502057191, + "p95": 2.8565687527589043, + "min": 1.7412500019418076, + "max": 13.369459004024975, + "iqr": 0.121438002679497, + "mean": 2.089421122218482, + "cv": 0.4106601536974344 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 2077.127528794237, + "p95": 2135.5130148393337, + "min": 1336.4383111459194, + "max": 2155.4082701427637, + "iqr": 59.26520173541758, + "mean": 1959.678910228628, + "cv": 0.1417185936868332 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1968.246993862696, + "p95": 2025.869527363777, + "min": 1269.8328421400633, + "max": 2043.7326532507363, + "iqr": 38.19524458751357, + "mean": 1858.9170388800862, + "cv": 0.13881497145484034 + }, + "memory": { + "peak_extra_bytes": 5899264, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "reference", + "batch": 4, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.007832957999198698, + "prefill": 0.004680792000726797, + "decode_steps": [ + 0.004910457995720208, + 0.00491675001103431, + 0.0044797920127166435, + 0.004709750006441027, + 0.004680625002947636, + 0.005972208004095592, + 0.004108374996576458, + 0.004411375004565343, + 0.00437579199206084, + 0.004345624998677522, + 0.004436124989297241, + 0.004358833000878803, + 0.004231041995808482, + 0.004315500002121553, + 0.004361665996839292, + 0.004324250010540709, + 0.004343209002399817, + 0.0043552919960347936, + 0.004329875009716488, + 0.004328500013798475, + 0.00434287499228958, + 0.004405333995237015, + 0.004376874989247881, + 0.004196124995360151, + 0.004648041998734698, + 0.004045624998980202, + 0.004515708002145402, + 0.005138833003002219, + 0.003852167006698437, + 0.004435333001310937, + 0.004323125001974404, + 0.004287666990421712 + ] + }, + { + "forward": 0.007079834002070129, + "prefill": 0.004259624998667277, + "decode_steps": [ + 0.004226084012771025, + 0.004361916988273151, + 0.0040686249994905666, + 0.0047194170037982985, + 0.004554500002996065, + 0.004466291007702239, + 0.003934583990485407, + 0.004027000002679415, + 0.00449112500064075, + 0.004301665991079062, + 0.0043552919960347936, + 0.004346083995187655, + 0.004037000006064773, + 0.004597167004249059, + 0.0048570829967502505, + 0.00475745799485594, + 0.004640250001102686, + 0.004399083991302177, + 0.004206541998428293, + 0.0045407500001601875, + 0.004465457997866906, + 0.003910832994733937, + 0.004437291994690895, + 0.00424566600122489, + 0.004787707992363721, + 0.004666958993766457, + 0.004460084004676901, + 0.004860957997152582, + 0.00408754100499209, + 0.004300125001464039, + 0.004358458987553604, + 0.004465624995646067 + ] + }, + { + "forward": 0.006702250000671484, + "prefill": 0.004315833008149639, + "decode_steps": [ + 0.004548958007944748, + 0.004298832995118573, + 0.004331834003096446, + 0.004410083012771793, + 0.004374040989205241, + 0.004321582993725315, + 0.004306082992115989, + 0.004217207999317907, + 0.004350457995315082, + 0.004290959011996165, + 0.004371458009700291, + 0.0042680420010583475, + 0.004140125005505979, + 0.00437654199777171, + 0.004277000000001863, + 0.004046417001518421, + 0.00434083299478516, + 0.004476999994949438, + 0.004517625013249926, + 0.004432832996826619, + 0.0043272500042803586, + 0.0043373749940656126, + 0.003966667005443014, + 0.004626167006790638, + 0.004412166003021412, + 0.00436608400195837, + 0.004465833000722341, + 0.00441466600750573, + 0.004441375000169501, + 0.00417020900931675, + 0.004191833999357186, + 0.004375124990474433 + ] + }, + { + "forward": 0.007803750006132759, + "prefill": 0.00415370799601078, + "decode_steps": [ + 0.004628916998626664, + 0.004451375003554858, + 0.004426041996339336, + 0.004672499999287538, + 0.004465584002900869, + 0.004526415999862365, + 0.004398582997964695, + 0.00431412500620354, + 0.004462834011064842, + 0.004374750002170913, + 0.004388667002785951, + 0.004315875004976988, + 0.004208500002278015, + 0.004261750000296161, + 0.0042459170072106645, + 0.00439654198999051, + 0.004322041000705212, + 0.004380000013043173, + 0.004216790999635123, + 0.004321708998759277, + 0.004084375003003515, + 0.004739667012472637, + 0.004531790997134522, + 0.00437600001168903, + 0.004473749999306165, + 0.004364125008578412, + 0.004413999995449558, + 0.004365415996289812, + 0.004443292011274025, + 0.004228707999573089, + 0.0042755830072565, + 0.004644125001505017 + ] + }, + { + "forward": 0.0077701669943053275, + "prefill": 0.00433804199565202, + "decode_steps": [ + 0.004869291995419189, + 0.004511749997618608, + 0.004600083993864246, + 0.004291541990824044, + 0.0043872919923160225, + 0.0043360419949749485, + 0.004383207997307181, + 0.004273708007531241, + 0.004399041994474828, + 0.004244166993885301, + 0.00429525000799913, + 0.004170041996985674, + 0.004399500001454726, + 0.004298375002690591, + 0.004275624989531934, + 0.004101250000530854, + 0.0043485829955898225, + 0.004350582996266894, + 0.0043554590083658695, + 0.004113999995752238, + 0.004801166011020541, + 0.003942666997318156, + 0.004286333001800813, + 0.004433041991433129, + 0.004390000001876615, + 0.004345042005297728, + 0.004504333002842031, + 0.0042611670069163665, + 0.0043440419976832345, + 0.004208915997878648, + 0.0043000419973395765, + 0.004302250003092922 + ] + }, + { + "forward": 0.0067098330036969855, + "prefill": 0.004359832993941382, + "decode_steps": [ + 0.004397041993797757, + 0.004313625002396293, + 0.004420917000970803, + 0.0042396250064484775, + 0.004251167003531009, + 0.004469499996048398, + 0.004164291007327847, + 0.004358333011623472, + 0.004016874998342246, + 0.004317209008149803, + 0.004439833996002562, + 0.004510792001383379, + 0.004411249989061616, + 0.004320416992413811, + 0.004236124994349666, + 0.004319375002523884, + 0.004377625009510666, + 0.0043219170038355514, + 0.003815834003034979, + 0.00427483300154563, + 0.004343791995779611, + 0.004200583003694192, + 0.0042425419960636646, + 0.004263374998117797, + 0.004055750003317371, + 0.004311124997911975, + 0.004238959008944221, + 0.00432587499381043, + 0.004208874990581535, + 0.004737541996291839, + 0.004053541997564025, + 0.004517833003774285 + ] + }, + { + "forward": 0.007748250005533919, + "prefill": 0.004309416995965876, + "decode_steps": [ + 0.004752959008328617, + 0.004420041004777886, + 0.004390708010760136, + 0.004258499990100972, + 0.004326542009948753, + 0.004679124991525896, + 0.004624249995686114, + 0.004805417003808543, + 0.004794625012436882, + 0.004512832994805649, + 0.004968415989424102, + 0.005689458004781045, + 0.00442045800446067, + 0.004433374997461215, + 0.004468791987164877, + 0.004466750004212372, + 0.004586958006257191, + 0.004408249995321967, + 0.00443091600027401, + 0.004396125004859641, + 0.004528207995463163, + 0.004196666995994747, + 0.0049752500053728, + 0.0045081249991199, + 0.0044144580024294555, + 0.004434375005075708, + 0.004498916998272762, + 0.004454166992218234, + 0.004364167005405761, + 0.004320416992413811, + 0.004109375004190952, + 0.004786124991369434 + ] + }, + { + "forward": 0.00771679199533537, + "prefill": 0.0041331659886054695, + "decode_steps": [ + 0.004625250003300607, + 0.004619250001269393, + 0.0044747089996235445, + 0.004357792000519112, + 0.004312749995733611, + 0.004375125005026348, + 0.004335874997195788, + 0.004657874989788979, + 0.004255167004885152, + 0.0048824170080479234, + 0.005676917004166171, + 0.004255000007105991, + 0.004394333009258844, + 0.004331417003413662, + 0.004286750001483597, + 0.004321000000345521, + 0.004347915994003415, + 0.0041507090063532814, + 0.004279750006389804, + 0.004305833994294517, + 0.004259374996763654, + 0.00438774999929592, + 0.004102958002476953, + 0.0042739159980556, + 0.004376208002213389, + 0.0043392919906182215, + 0.004307082999730483, + 0.0043032499961555, + 0.004323291999753565, + 0.004476125002838671, + 0.004976666998118162, + 0.0044150419998914 + ] + }, + { + "forward": 0.006950417009647936, + "prefill": 0.0043735000072047114, + "decode_steps": [ + 0.004377165998448618, + 0.004450041000382043, + 0.004325667003286071, + 0.004383791994769126, + 0.004369292000774294, + 0.004303833993617445, + 0.004379958001663908, + 0.0043603750091278926, + 0.0043274159979773685, + 0.004411708010593429, + 0.0044091669988119975, + 0.004349458002252504, + 0.0041961660026572645, + 0.00417225000273902, + 0.0042703340004663914, + 0.004321500004152767, + 0.004300542001146823, + 0.004352415999164805, + 0.004271541998605244, + 0.004120375000638887, + 0.004384333005873486, + 0.004291375007596798, + 0.0043109169928357005, + 0.004139833996305242, + 0.004669958012527786, + 0.004472999993595295, + 0.003984499999205582, + 0.0039789589936845005, + 0.004337999998824671, + 0.004438249990926124, + 0.004394167001009919, + 0.004272792008123361 + ] + }, + { + "forward": 0.010214750000159256, + "prefill": 0.004367792003904469, + "decode_steps": [ + 0.005456292012240738, + 0.00429429100768175, + 0.004851624995353632, + 0.004589292002492584, + 0.006956250013899989, + 0.004539875008049421, + 0.004341250009019859, + 0.0065022080088965595, + 0.004660582999349572, + 0.004523125011473894, + 0.004757832997711375, + 0.006327292008791119, + 0.005158958010724746, + 0.0049890829977812245, + 0.0043105419899802655, + 0.004891666991170496, + 0.004295584003557451, + 0.005084457996417768, + 0.004886708993581124, + 0.00487691699527204, + 0.00459279099595733, + 0.004684750005253591, + 0.004590083990478888, + 0.004337457998190075, + 0.004416625000885688, + 0.004701833997387439, + 0.004724583006463945, + 0.004850792000070214, + 0.00556233299721498, + 0.0057702909980434924, + 0.004828749995795079, + 0.005128958000568673 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 7.7325210004346445, + "p95": 9.142943599727003, + "min": 6.702250000671484, + "max": 10.214750000159256, + "iqr": 0.8125829954224173, + "mean": 7.652900101675186, + "cv": 0.13254468026986424 + }, + "prefill_ms": { + "count": 10, + "median": 4.326937501900829, + "p95": 4.542510603641858, + "min": 4.1331659886054695, + "max": 4.680792000726797, + "iqr": 0.09372925342177041, + "mean": 4.329170798882842, + "cv": 0.0346677253448524 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.374395495688077, + "p95": 4.968757690221537, + "min": 3.815834003034979, + "max": 6.956250013899989, + "iqr": 0.20760374900419265, + "mean": 4.4491677410860575, + "cv": 0.07865728692554134 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 908.9159263148213, + "p95": 929.0892160402485, + "min": 807.6571804694863, + "max": 931.0690383575851, + "iqr": 24.025580323646636, + "mean": 900.4419388999659, + "cv": 0.03999888478014011 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 882.553639234033, + "p95": 900.5503495444839, + "min": 785.9951625417049, + "max": 902.4493688404038, + "iqr": 23.921849539638288, + "mean": 873.8012797016472, + "cv": 0.039009710533845575 + }, + "memory": { + "peak_extra_bytes": 5899264, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "auto", + "batch": 4, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.006514792010420933 + }, + { + "forward": 0.01833204200374894 + }, + { + "forward": 0.006907166010932997 + }, + { + "forward": 0.006644207998760976 + }, + { + "forward": 0.006487167003797367 + }, + { + "forward": 0.006857958011096343 + }, + { + "forward": 0.006490542000392452 + }, + { + "forward": 0.007352042011916637 + }, + { + "forward": 0.007331042012083344 + }, + { + "forward": 0.0072617909900145605 + } + ], + "forward_ms": { + "count": 10, + "median": 6.88256201101467, + "p95": 13.391042007424392, + "min": 6.487167003797367, + "max": 18.33204200374894, + "iqr": 0.7665832490602043, + "mean": 8.017875005316455, + "cv": 0.4540303410178624 + }, + "memory": { + "peak_extra_bytes": 11798528, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "sdpa", + "batch": 4, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.0055230830039363354, + "prefill": 0.005634166998788714, + "decode_steps": [ + 0.0020951249898644164, + 0.001992290999623947, + 0.0020416250044945627, + 0.0020050839957548305, + 0.0020493330084718764, + 0.0020662909955717623, + 0.0019753749947994947, + 0.0021129160013515502, + 0.00204770801065024, + 0.0020107499876758084, + 0.001981916997465305, + 0.0020071250037290156, + 0.0019960829959018156, + 0.001985166993108578, + 0.0020868339925073087, + 0.0020644159958465025, + 0.0019729580089915544, + 0.002086916007101536, + 0.0020313330023782328, + 0.0020697080035461113, + 0.0019794999971054494, + 0.0019895830046152696, + 0.002080167003441602, + 0.002021125008468516, + 0.0019728329934878275, + 0.0020303330093156546, + 0.0020151670032646507, + 0.0020691659883596003, + 0.0020722500048577785, + 0.0021030830102972686, + 0.002068917005090043, + 0.0019779580034082755 + ] + }, + { + "forward": 0.00547974999062717, + "prefill": 0.005466417002025992, + "decode_steps": [ + 0.0020409999997355044, + 0.001994874997762963, + 0.002118792006513104, + 0.0020295830036047846, + 0.002037666999967769, + 0.002027249996899627, + 0.0020668330107582733, + 0.0021164170029805973, + 0.0021142089972272515, + 0.002014457990298979, + 0.0019894169963663444, + 0.002061499995761551, + 0.0020343749929452315, + 0.002066708999336697, + 0.002091834001475945, + 0.002077500001178123, + 0.0020766669913427904, + 0.002034709003055468, + 0.0020785410015378147, + 0.0020877079950878397, + 0.002103834005538374, + 0.002055916003882885, + 0.002108915999997407, + 0.00202191699645482, + 0.0020764999935636297, + 0.0020476670033531263, + 0.002162375007173978, + 0.0020007920102216303, + 0.0020230829977663234, + 0.0020420000073499978, + 0.0020056250068591908, + 0.0019581250089686364 + ] + }, + { + "forward": 0.0056342089956160635, + "prefill": 0.0058357090019853786, + "decode_steps": [ + 0.002717208000831306, + 0.0021520409936783835, + 0.002162292003049515, + 0.00205275000189431, + 0.002019958003074862, + 0.0020621669973479584, + 0.001991041994187981, + 0.001947541008121334, + 0.0020564159931382164, + 0.002023207998718135, + 0.0020145000016782433, + 0.0019249580072937533, + 0.002552292004111223, + 0.001987209005164914, + 0.0019730000058189034, + 0.0021089579968247563, + 0.0024415410007350147, + 0.0020248749933671206, + 0.002075165990390815, + 0.0020409999997355044, + 0.002221250004367903, + 0.0019925829983549193, + 0.002002624998567626, + 0.00213350000558421, + 0.002107958003762178, + 0.002104833009070717, + 0.0020107079908484593, + 0.002147042003343813, + 0.002109082997776568, + 0.0019957079930463806, + 0.0020875000045634806, + 0.0020685419876826927 + ] + }, + { + "forward": 0.005496665995451622, + "prefill": 0.005481375003000721, + "decode_steps": [ + 0.0020861660013906658, + 0.002231708000181243, + 0.001990083997952752, + 0.0020691250101663172, + 0.001968750002561137, + 0.0020497919904300943, + 0.00198320799972862, + 0.0020520000107353553, + 0.0019060420017922297, + 0.0020077090011909604, + 0.002050666997092776, + 0.0020103749993722886, + 0.0020225839980412275, + 0.001980500004719943, + 0.0019659999961731955, + 0.0020659159927163273, + 0.002122958001564257, + 0.002026292000664398, + 0.0019783749885391444, + 0.0022427079966291785, + 0.0022377499990398064, + 0.0020606250036507845, + 0.002154041998437606, + 0.002163750003091991, + 0.0021572499972535297, + 0.0020404579991009086, + 0.0019449169922154397, + 0.0020415830076672137, + 0.0020640420116251335, + 0.00208283300162293, + 0.0019512500002747402, + 0.002046583002083935 + ] + }, + { + "forward": 0.005817415993078612, + "prefill": 0.006072999996831641, + "decode_steps": [ + 0.005193499993765727, + 0.004556333995424211, + 0.003969417011830956, + 0.004211457999190316, + 0.0026464580005267635, + 0.0024738750071264803, + 0.0020648749923566356, + 0.0021109170047566295, + 0.0021737080096499994, + 0.0021986659994581714, + 0.0028753750084433705, + 0.0030028329929336905, + 0.0027847919991472736, + 0.0027186670049559325, + 0.002661375008756295, + 0.002417249997961335, + 0.002077707991702482, + 0.0021031670039519668, + 0.0020032920001540333, + 0.0019646669970825315, + 0.0021261250076349825, + 0.0019994590111309662, + 0.002036749996477738, + 0.0020402500085765496, + 0.0020918749942211434, + 0.0020602500007953495, + 0.0020206669869367033, + 0.00214829201286193, + 0.002008333001867868, + 0.0020044580014655367, + 0.0021075419936096296, + 0.0020467090071178973 + ] + }, + { + "forward": 0.005530166992684826, + "prefill": 0.005588083004113287, + "decode_steps": [ + 0.002446333004627377, + 0.002139959004125558, + 0.0022535419993801042, + 0.0019877499871654436, + 0.0021419580007204786, + 0.0020631670049624518, + 0.0019355000113137066, + 0.0021123750047991052, + 0.001984125003218651, + 0.0019543329981388524, + 0.0019602499960456043, + 0.002016541999182664, + 0.0020227920031175017, + 0.0020168339979136363, + 0.00213199999416247, + 0.00212154199834913, + 0.001986083996598609, + 0.0019368329958524555, + 0.002034666991676204, + 0.001977667008759454, + 0.0021429170010378584, + 0.00198058299429249, + 0.0022190420131664723, + 0.0021096249984111637, + 0.0019606670102803037, + 0.0020886249985778704, + 0.002111042005708441, + 0.0021286250121193007, + 0.0020966249867342412, + 0.0020194999960949644, + 0.0020714580023195595, + 0.0019899579929187894 + ] + }, + { + "forward": 0.005539999998291023, + "prefill": 0.005499666003743187, + "decode_steps": [ + 0.0020540839905152097, + 0.0020612499938579276, + 0.0021436669921968132, + 0.0021952080132905394, + 0.001965500006917864, + 0.0021068750065751374, + 0.001996749997488223, + 0.0020184169989079237, + 0.002148665997083299, + 0.002061416002106853, + 0.0019435409922152758, + 0.0021855829982087016, + 0.0020214169926475734, + 0.0020671670063165948, + 0.0020664170006057248, + 0.0019459580071270466, + 0.0020613750093616545, + 0.00200395799765829, + 0.0020558750111376867, + 0.002029291004873812, + 0.0020459590014070272, + 0.0019004170026164502, + 0.0020340830087661743, + 0.0020644579926738515, + 0.002016167010879144, + 0.002073292009299621, + 0.0020287090010242537, + 0.0020477079960983247, + 0.002093458009767346, + 0.002093791001243517, + 0.00203237499226816, + 0.0020873339963145554 + ] + }, + { + "forward": 0.006234166008653119, + "prefill": 0.005762041997513734, + "decode_steps": [ + 0.0025099170015892014, + 0.002173625005525537, + 0.00438458300777711, + 0.004001249995781109, + 0.003160417007165961, + 0.0031582919909851626, + 0.0025950409908546135, + 0.0023619580024387687, + 0.0022902920027263463, + 0.0022825000050943345, + 0.002361334001761861, + 0.002369624999118969, + 0.0023993340000743046, + 0.002511500002583489, + 0.0026487080031074584, + 0.0025439580058446154, + 0.002387332991929725, + 0.0024769580049905926, + 0.0023156659881351516, + 0.00279762499849312, + 0.002814374995068647, + 0.00288545798684936, + 0.0029284580086823553, + 0.002631167008075863, + 0.002608166993013583, + 0.0025892910052789375, + 0.002389166009379551, + 0.00254845799645409, + 0.002447500010021031, + 0.002720457996474579, + 0.002478249996784143, + 0.0030491249926853925 + ] + }, + { + "forward": 0.005554999996093102, + "prefill": 0.006584417002159171, + "decode_steps": [ + 0.00234266699408181, + 0.002205333992606029, + 0.0021522919996641576, + 0.0021056249970570207, + 0.0019807920034509152, + 0.0019863750057993457, + 0.002396083000348881, + 0.00213312498817686, + 0.0020355000015115365, + 0.0020638749992940575, + 0.002048999987891875, + 0.00204266699438449, + 0.0020930419996147975, + 0.002920084007200785, + 0.0022182910033734515, + 0.0019703340076375753, + 0.0023815829918021336, + 0.002364500003750436, + 0.002103000006172806, + 0.002139624994015321, + 0.0024386669974774122, + 0.0021375409996835515, + 0.002096750002237968, + 0.002314416997251101, + 0.002169250001315959, + 0.0019507909892126918, + 0.0020689999946625903, + 0.0023660410079173744, + 0.002013082994380966, + 0.0022686669981339946, + 0.002025291003519669, + 0.0020840000070165843 + ] + }, + { + "forward": 0.00546125000983011, + "prefill": 0.005521125000086613, + "decode_steps": [ + 0.002120500008459203, + 0.002159791998565197, + 0.0020629589998861775, + 0.0022727500036126003, + 0.0020821670041186735, + 0.0020155420061200857, + 0.001983791997190565, + 0.0019317919941386208, + 0.002061292005237192, + 0.0020574170048348606, + 0.002036167003097944, + 0.0019887910020770505, + 0.002050625000265427, + 0.001979375010705553, + 0.0020977919921278954, + 0.0021404589933808893, + 0.002039041995885782, + 0.002106667001498863, + 0.0019945410022046417, + 0.0019678329990711063, + 0.0019545420072972775, + 0.0028124169912189245, + 0.002106291998643428, + 0.002045500004896894, + 0.002090583002427593, + 0.002020416999584995, + 0.002042207997874357, + 0.002056958997854963, + 0.002945999993244186, + 0.0019467499951133505, + 0.002047750007477589, + 0.0021217920002527535 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 5.535083495487925, + "p95": 6.046628501644591, + "min": 5.46125000983011, + "max": 6.234166008653119, + "iqr": 0.11113649816252291, + "mean": 5.627170698426198, + "cv": 0.04205498383153886 + }, + "prefill_ms": { + "count": 10, + "median": 5.611125001451001, + "p95": 6.354279349761781, + "min": 5.466417002025992, + "max": 6.584417002159171, + "iqr": 0.3122614980384242, + "mean": 5.744600101024844, + "cv": 0.06127241220733189 + }, + "decode_ms_per_step": { + "count": 320, + "median": 2.0667710050474852, + "p95": 2.8125148914114106, + "min": 1.9004170026164502, + "max": 5.193499993765727, + "iqr": 0.1340824965154752, + "mean": 2.1855032598068647, + "cv": 0.17575364151806247 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1918.4339281853852, + "p95": 1959.4719249243608, + "min": 1491.4974912478858, + "max": 1967.4438150625583, + "iqr": 92.34496643048396, + "mean": 1846.0028034577049, + "cv": 0.09119159039899744 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1770.5995677594035, + "p95": 1805.4116284347274, + "min": 1397.6571400591797, + "max": 1810.6408073538594, + "iqr": 99.5876469376492, + "mean": 1704.3675158131475, + "cv": 0.08689198845438381 + }, + "memory": { + "peak_extra_bytes": 11798528, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "reference", + "batch": 4, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.011537958998815157, + "prefill": 0.008366583002498373, + "decode_steps": [ + 0.004369333008071408, + 0.0051720409974223, + 0.005301917000906542, + 0.004921166997519322, + 0.004638750004232861, + 0.0046982500061858445, + 0.004516582994256169, + 0.0044199999974807724, + 0.004562541013001464, + 0.004467667007702403, + 0.004472499989788048, + 0.004530166988843121, + 0.004714292008429766, + 0.00460216699866578, + 0.004427915991982445, + 0.0043916669965256006, + 0.004563332986435853, + 0.004600665997713804, + 0.00446195800032001, + 0.004494792010518722, + 0.004567582989693619, + 0.004601249995175749, + 0.004737916999147274, + 0.004989167005987838, + 0.004968458000803366, + 0.004610124989994802, + 0.004828958000871353, + 0.004571208992274478, + 0.004429749998962507, + 0.004804750002222136, + 0.004538708002655767, + 0.0045327499974519014 + ] + }, + { + "forward": 0.011169667006470263, + "prefill": 0.009100833005504683, + "decode_steps": [ + 0.0050505830004112795, + 0.004693916998803616, + 0.0046278340014396235, + 0.007099959009792656, + 0.00762258299801033, + 0.005214709002757445, + 0.005276958996546455, + 0.005538958997931331, + 0.005105916003230959, + 0.004712833004305139, + 0.004790291000972502, + 0.004281166999135166, + 0.004754041990963742, + 0.004415500006871298, + 0.004656083008740097, + 0.004997500000172295, + 0.004481791009311564, + 0.004354792006779462, + 0.0044531669991556555, + 0.00451299999258481, + 0.0050364580092718825, + 0.0049597080069361255, + 0.004520707996562123, + 0.0050892919971374795, + 0.004795832996023819, + 0.0044944160035811365, + 0.014360958011820912, + 0.03325945899996441, + 0.030741875001695007, + 0.01744683299330063, + 0.010276375003741123, + 0.0068942499929107726 + ] + }, + { + "forward": 0.01306720900174696, + "prefill": 0.010311541991541162, + "decode_steps": [ + 0.006684790991130285, + 0.005690458987373859, + 0.0060064999997848645, + 0.005539874997339211, + 0.005135166997206397, + 0.0053629579924745485, + 0.006012791011016816, + 0.0057630829978734255, + 0.005045583995524794, + 0.005071999999927357, + 0.005777166006737389, + 0.005189666990190744, + 0.005228208989137784, + 0.004873333004070446, + 0.00611508300062269, + 0.005510125003638677, + 0.0048254580033244565, + 0.004880915992544033, + 0.005662291994667612, + 0.005642290998366661, + 0.005149875010829419, + 0.005706833995645866, + 0.006101250008214265, + 0.005134625011123717, + 0.0073623749922262505, + 0.005509249996975996, + 0.005081208000774495, + 0.005169582989765331, + 0.005167249997612089, + 0.004972834009095095, + 0.005536333002964966, + 0.0040789170016068965 + ] + }, + { + "forward": 0.012294832995394245, + "prefill": 0.009387375001097098, + "decode_steps": [ + 0.005479584011482075, + 0.004607959010172635, + 0.004921083993394859, + 0.0051144169992767274, + 0.004760374999023043, + 0.004747832994326018, + 0.0049173339939443395, + 0.004544625000562519, + 0.004734165995614603, + 0.004533083993010223, + 0.004605374997481704, + 0.004275125000276603, + 0.004663625004468486, + 0.004760707990499213, + 0.004912999997031875, + 0.004772333006258123, + 0.004332124997745268, + 0.005534417010494508, + 0.004329708011937328, + 0.004758833005325869, + 0.0047414580039912835, + 0.005579499993473291, + 0.005040334013756365, + 0.0049098339950433, + 0.004782333999173716, + 0.004657958997995593, + 0.004549916004179977, + 0.004582958994433284, + 0.004455750007764436, + 0.004462875003810041, + 0.004218000001856126, + 0.004586374998325482 + ] + }, + { + "forward": 0.011898458003997803, + "prefill": 0.008447000000160187, + "decode_steps": [ + 0.005158208994544111, + 0.004489416998694651, + 0.004728833009721711, + 0.0045697079913225025, + 0.004790499995579012, + 0.004799833986908197, + 0.004600416999892332, + 0.004777167006977834, + 0.0046470420056721196, + 0.004437625000718981, + 0.0044036250037606806, + 0.00459820800460875, + 0.004624125009286217, + 0.004508666999754496, + 0.004457417002413422, + 0.004331249991082586, + 0.004258791988831945, + 0.004465249992790632, + 0.004359124999609776, + 0.0045315000024857, + 0.004633459000615403, + 0.004411333007737994, + 0.00444233299640473, + 0.004650749993743375, + 0.004465582998818718, + 0.004755708010634407, + 0.004569542012177408, + 0.0047988329897634685, + 0.004871166995144449, + 0.004546665993984789, + 0.004648250003810972, + 0.004056042002048343 + ] + }, + { + "forward": 0.010807917002239265, + "prefill": 0.008820333998301066, + "decode_steps": [ + 0.00501991699275095, + 0.004798833993845619, + 0.004760916999657638, + 0.0042575420084176585, + 0.004233083993312903, + 0.004646000001230277, + 0.00462083300226368, + 0.004208457990898751, + 0.004445833998033777, + 0.00462066700856667, + 0.0044497079943539575, + 0.004627500005881302, + 0.0051507919997675344, + 0.004378375000669621, + 0.004540624999208376, + 0.005005457991501316, + 0.004920708001009189, + 0.004893792007351294, + 0.004946957997162826, + 0.00480999999854248, + 0.0044901249930262566, + 0.00422045799496118, + 0.004304874993977137, + 0.004556624990073033, + 0.005021208999096416, + 0.004607875001966022, + 0.004874209000263363, + 0.004382707993499935, + 0.004267791999154724, + 0.004370166003354825, + 0.0046974580036476254, + 0.004743124998640269 + ] + }, + { + "forward": 0.011956250004004687, + "prefill": 0.00848166701325681, + "decode_steps": [ + 0.004765457997564226, + 0.004741249998915009, + 0.004565082999761216, + 0.004563166992738843, + 0.0045550840004580095, + 0.004477583002881147, + 0.004455416012206115, + 0.004891250006039627, + 0.0043959580070804805, + 0.004532374994596466, + 0.004627665999578312, + 0.004502334006247111, + 0.004492958993068896, + 0.004568958000163548, + 0.0044915410107932985, + 0.004449874992133118, + 0.0049622499936958775, + 0.004833625003811903, + 0.004623583008651622, + 0.004638624988729134, + 0.004571166995447129, + 0.004638208003598265, + 0.004646000001230277, + 0.004531792001216672, + 0.0044931250013178214, + 0.004620624997187406, + 0.004518125002505258, + 0.004591000004438683, + 0.004493957996601239, + 0.004130041997996159, + 0.004935959004797041, + 0.0055081249884096906 + ] + }, + { + "forward": 0.012989374998142011, + "prefill": 0.009802915999898687, + "decode_steps": [ + 0.004818375004106201, + 0.008440083009190857, + 0.007268749992363155, + 0.006485917008831166, + 0.005828125009429641, + 0.005937042005825788, + 0.006098457990447059, + 0.0049054160044761375, + 0.00470158300595358, + 0.0048602089955238625, + 0.004585208007483743, + 0.004981874997611158, + 0.004517332999967039, + 0.004991415989934467, + 0.0048312090075341985, + 0.004480292001971975, + 0.0042055840021930635, + 0.0045402090036077425, + 0.004955166994477622, + 0.006536542001413181, + 0.00534816698927898, + 0.005343417011317797, + 0.005399583009420894, + 0.004557332998956554, + 0.00493283401010558, + 0.00523937500838656, + 0.004673292001825757, + 0.004950582995661534, + 0.00454774999525398, + 0.004750583000713959, + 0.004918083010124974, + 0.004641334002371877 + ] + }, + { + "forward": 0.011043916005291976, + "prefill": 0.009129374986514449, + "decode_steps": [ + 0.00465037499088794, + 0.0043623329984257, + 0.004579250002279878, + 0.0048317920009139925, + 0.004578084000968374, + 0.004703166996478103, + 0.004796499997610226, + 0.004683457998908125, + 0.004532207996817306, + 0.004604958987329155, + 0.004757000002427958, + 0.004729583000880666, + 0.004685500010964461, + 0.004547916993033141, + 0.004547207994619384, + 0.004676166994613595, + 0.004565250012092292, + 0.004502916999626905, + 0.004473958004382439, + 0.004401999991387129, + 0.004384333995403722, + 0.004482083008042537, + 0.004288666998036206, + 0.004556832995149307, + 0.005027334002079442, + 0.005142875001183711, + 0.004610333009622991, + 0.004744459001813084, + 0.0047219169937307015, + 0.004632083000615239, + 0.0045695000007981434, + 0.00455354098812677 + ] + }, + { + "forward": 0.013052916998276487, + "prefill": 0.008431041002040729, + "decode_steps": [ + 0.00565208301122766, + 0.004961999991792254, + 0.0142597499943804, + 0.007897375005995855, + 0.005721042005461641, + 0.00471037499664817, + 0.004781249997904524, + 0.005906833990593441, + 0.0054075000080047175, + 0.00511825000285171, + 0.00662387500051409, + 0.005690499994670972, + 0.005262459002551623, + 0.005962540992186405, + 0.00535900000249967, + 0.006441582998377271, + 0.006263542003580369, + 0.0056109159922925755, + 0.005501624997123145, + 0.005526541994186118, + 0.005274834009469487, + 0.007070999999996275, + 0.0047700409922981635, + 0.005451749995700084, + 0.005358250011340715, + 0.004848875003517605, + 0.004989834007574245, + 0.005667459001415409, + 0.004788916005054489, + 0.004759250005008653, + 0.005865874991286546, + 0.005111625010613352 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 11.927354004001245, + "p95": 13.060777600185247, + "min": 10.807917002239265, + "max": 13.06720900174696, + "iqr": 1.5539994928985834, + "mean": 11.981850101437885, + "cv": 0.07125567281699899 + }, + "prefill_ms": { + "count": 10, + "median": 8.960583501902875, + "p95": 10.082660295302048, + "min": 8.366583002498373, + "max": 10.311541991541162, + "iqr": 0.8672082440170925, + "mean": 9.027866600081325, + "cv": 0.07259521003200685 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.736041497380938, + "p95": 6.540908651368228, + "min": 4.056042002048343, + "max": 33.25945899996441, + "iqr": 0.5668222547683399, + "mean": 5.193030628151973, + "cv": 0.4738711753199181 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 852.3569664472582, + "p95": 870.6028238614097, + "min": 527.7964189360729, + "max": 874.3983055824003, + "iqr": 125.6203168531008, + "mean": 788.6394830815054, + "cv": 0.14386238029975063 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 804.3795299816462, + "p95": 823.2189399920233, + "min": 508.7064653583776, + "max": 826.69513827445, + "iqr": 117.8520992143574, + "mean": 746.583578218903, + "cv": 0.1388682884510958 + }, + "memory": { + "peak_extra_bytes": 17303552, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "auto", + "batch": 4, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.012699791986960918 + }, + { + "forward": 0.012823750003008172 + }, + { + "forward": 0.012781750003341585 + }, + { + "forward": 0.012720124999759719 + }, + { + "forward": 0.013096166992909275 + }, + { + "forward": 0.012813708002795465 + }, + { + "forward": 0.012780709002981894 + }, + { + "forward": 0.012768667002092116 + }, + { + "forward": 0.012793083995347843 + }, + { + "forward": 0.012772333997418173 + } + ], + "forward_ms": { + "count": 10, + "median": 12.78122950316174, + "p95": 12.973579347453779, + "min": 12.699791986960918, + "max": 13.096166992909275, + "iqr": 0.03896825000992976, + "mean": 12.805008598661516, + "cv": 0.008520603512427314 + }, + "memory": { + "peak_extra_bytes": 23597056, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "sdpa", + "batch": 4, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.010679208004148677, + "prefill": 0.010864416006370448, + "decode_steps": [ + 0.002412665999145247, + 0.0023996249947231263, + 0.002295374986715615, + 0.0024089159996947274, + 0.0022802500025136396, + 0.002311000003828667, + 0.0023524159914813936, + 0.0022878329909872264, + 0.0022619170049438253, + 0.0023887909919722006, + 0.0022296660026768222, + 0.002351957999053411, + 0.0023435830080416054, + 0.002390583002124913, + 0.002304292007465847, + 0.002300541993463412, + 0.0023223330063046888, + 0.0022776249970775098, + 0.0023158749972935766, + 0.002314417011803016, + 0.002230583006166853, + 0.002471041996614076, + 0.0023095409997040406, + 0.0022227500012377277, + 0.0023320839973166585, + 0.0023242919996846467, + 0.002357999997911975, + 0.002294834004715085, + 0.0022937499888939783, + 0.0023396250035148114, + 0.0023377079924102873, + 0.0023314999998547137 + ] + }, + { + "forward": 0.010579459005384706, + "prefill": 0.010577124994597398, + "decode_steps": [ + 0.0023639999999431893, + 0.002301291999174282, + 0.002449791005346924, + 0.0024076670088106766, + 0.0028426670032786205, + 0.00247591600054875, + 0.002308333001565188, + 0.002252167003462091, + 0.002360541999223642, + 0.0023951660114107653, + 0.00237800000468269, + 0.002336541991098784, + 0.0023988750035641715, + 0.002282458997797221, + 0.002312708005774766, + 0.0023806669923942536, + 0.0025416660064365715, + 0.002342375009902753, + 0.00242620799690485, + 0.002507500001229346, + 0.0025152500020340085, + 0.002434249996440485, + 0.0023762079945299774, + 0.0022809160000178963, + 0.0023943330015754327, + 0.0023414169991156086, + 0.002395542003796436, + 0.0024110410013236105, + 0.002273292004247196, + 0.002361124992603436, + 0.0023429169959854335, + 0.0022840409947093576 + ] + }, + { + "forward": 0.010598583990940824, + "prefill": 0.010497875002329238, + "decode_steps": [ + 0.0025282919959863648, + 0.0023442090023308992, + 0.0024368339945795015, + 0.0022974169987719506, + 0.0022965829994063824, + 0.002317667007446289, + 0.00231783400522545, + 0.0023576249950565398, + 0.002286165996338241, + 0.0022962080111028627, + 0.002307750008185394, + 0.0022918749891687185, + 0.002317874997970648, + 0.002378958000917919, + 0.0023230840015457943, + 0.0022775410034228116, + 0.002333832992007956, + 0.002354457988985814, + 0.0022914580040378496, + 0.0023968750028871, + 0.0023053749901009724, + 0.00221462499757763, + 0.0024358340015169233, + 0.0023137499956646934, + 0.002421042008791119, + 0.0022377499990398064, + 0.0022577920026378706, + 0.0026306670042686164, + 0.002385958010563627, + 0.002359207996050827, + 0.0024270409921882674, + 0.002340833001653664 + ] + }, + { + "forward": 0.010426458000438288, + "prefill": 0.010541459007072262, + "decode_steps": [ + 0.0024128329969244078, + 0.0023831250000512227, + 0.0022677090018987656, + 0.0023143329890444875, + 0.002507791999960318, + 0.0024020830023800954, + 0.002305333997355774, + 0.002262042005895637, + 0.002301125001395121, + 0.002289292009663768, + 0.0021949169895378873, + 0.0024447500036330894, + 0.0022605830017710105, + 0.0024181250046240166, + 0.00233575000311248, + 0.002474291992257349, + 0.002374249990680255, + 0.0023722920013824478, + 0.002425417013000697, + 0.0023909159936010838, + 0.0022617919894400984, + 0.002270166005473584, + 0.0027671670104609802, + 0.0021941670129308477, + 0.0023409579880535603, + 0.0023792499996488914, + 0.0023263340117409825, + 0.002297833008924499, + 0.002308958995854482, + 0.0023290000099223107, + 0.002350999988266267, + 0.0022878750023664907 + ] + }, + { + "forward": 0.011449290992459282, + "prefill": 0.01055445900419727, + "decode_steps": [ + 0.002493250009138137, + 0.0023970000038389117, + 0.0024345839919988066, + 0.002337208003154956, + 0.0023680419981246814, + 0.002778374997433275, + 0.0024232919968198985, + 0.0023612090008100495, + 0.0023396250035148114, + 0.0023643750027986243, + 0.002347707995795645, + 0.0022987919946899638, + 0.002220499998657033, + 0.002455333000398241, + 0.0023432080051861703, + 0.0022960829955991358, + 0.003120666995528154, + 0.0023900829983176664, + 0.0023444590042345226, + 0.002402792000793852, + 0.002304875000845641, + 0.0022148330026539043, + 0.002373666997300461, + 0.002241292007965967, + 0.002321541993296705, + 0.0023092909978004172, + 0.0024192500131903216, + 0.002337874990189448, + 0.0023463749967049807, + 0.0022991669975453988, + 0.002522999988286756, + 0.0022239589889068156 + ] + }, + { + "forward": 0.010653124991222285, + "prefill": 0.01063350000185892, + "decode_steps": [ + 0.0024066659971140325, + 0.0023684159968979657, + 0.002311917007318698, + 0.00230491699767299, + 0.0024388750025536865, + 0.0023811669962015003, + 0.0022864169877720997, + 0.0023825419921195135, + 0.002394374998402782, + 0.0024047920014709234, + 0.0023744579957565293, + 0.002326750007341616, + 0.002347000001464039, + 0.002271792007377371, + 0.0023444590042345226, + 0.0022846249921713024, + 0.0023198330018203706, + 0.0024021669960347936, + 0.002369249996263534, + 0.002281082997797057, + 0.0023289579985430464, + 0.0023818750050850213, + 0.0023038340004859492, + 0.0023617080005351454, + 0.0023752920096740127, + 0.0022674589999951422, + 0.002248290998977609, + 0.0023266670032171533, + 0.002362999992328696, + 0.0022920409974176437, + 0.0023205409961519763, + 0.0024517909914720803 + ] + }, + { + "forward": 0.010524957993766293, + "prefill": 0.010580958012724295, + "decode_steps": [ + 0.002451416992698796, + 0.002328874994418584, + 0.0023707090003881603, + 0.0023541669943369925, + 0.002327958995010704, + 0.0023344999935943633, + 0.0022305409947875887, + 0.002364875006605871, + 0.0023588749900227413, + 0.002410790999419987, + 0.002319167004316114, + 0.002562167006544769, + 0.0023251670063473284, + 0.002332332995138131, + 0.0022944580123294145, + 0.0023109169997042045, + 0.0024167910014512017, + 0.002284125002915971, + 0.0024053329980233684, + 0.002379917001235299, + 0.0022017500014044344, + 0.0022875419963384047, + 0.0023700840101810172, + 0.0022951669961912557, + 0.002362750004976988, + 0.002371125010540709, + 0.002328583999769762, + 0.0022232920018723235, + 0.002275209000799805, + 0.0022838749864604324, + 0.0022638749942416325, + 0.0023183330049505457 + ] + }, + { + "forward": 0.010651665987097658, + "prefill": 0.010628624993842095, + "decode_steps": [ + 0.0024661670031491667, + 0.0023802089999662712, + 0.0023904999980004504, + 0.0022912919957889244, + 0.0023602919973200187, + 0.0023170420026872307, + 0.0022411660029320046, + 0.002352292009163648, + 0.0023060829989844933, + 0.002311582997208461, + 0.002360583996050991, + 0.002234916013549082, + 0.0024169579992303625, + 0.0022676660009892657, + 0.0024381669936701655, + 0.0022457909944932908, + 0.002436208989820443, + 0.002405416002147831, + 0.0022132080048322678, + 0.002229833000455983, + 0.002251792000606656, + 0.002402416997938417, + 0.002279833992361091, + 0.002322915999684483, + 0.0023198330018203706, + 0.0022926669917069376, + 0.002320041006896645, + 0.0023781250056345016, + 0.002311749994987622, + 0.002341499988688156, + 0.0023682500032009557, + 0.0024093749962048605 + ] + }, + { + "forward": 0.010584125004243106, + "prefill": 0.010529416002100334, + "decode_steps": [ + 0.0023972079943632707, + 0.0023236249980982393, + 0.002256749998196028, + 0.0023639169958187267, + 0.0023886249982751906, + 0.0023073339980328456, + 0.0023337500024354085, + 0.002342792009585537, + 0.002330040995730087, + 0.0026219169958494604, + 0.0022734590020263568, + 0.0022547080006916076, + 0.0022118329943623394, + 0.002339749989914708, + 0.002328499991563149, + 0.0022894169960636646, + 0.002419707991066389, + 0.0023286250070668757, + 0.0022682080016238615, + 0.002387041997280903, + 0.0022988749988144264, + 0.0023937499936437234, + 0.0023563750000903383, + 0.0022627919970545918, + 0.002352541996515356, + 0.002381832993705757, + 0.002404458005912602, + 0.002269499993417412, + 0.002369666995946318, + 0.0023779159964760765, + 0.0023556250089313835, + 0.0023803750082151964 + ] + }, + { + "forward": 0.010485584003617987, + "prefill": 0.01050845799909439, + "decode_steps": [ + 0.0024225420056609437, + 0.0022695829975418746, + 0.002238582994323224, + 0.002398625001660548, + 0.002264166992972605, + 0.0022972090082475916, + 0.002234416999272071, + 0.0024585000064689666, + 0.0023359170008916408, + 0.0025559579953551292, + 0.0023564999864902347, + 0.0022790829971199855, + 0.0023971250047907233, + 0.0024225000088335946, + 0.002257875006762333, + 0.0022955830063438043, + 0.0022582090023206547, + 0.0022414169943658635, + 0.0022911249980097637, + 0.0023947079898789525, + 0.002294791003805585, + 0.0022071249986765906, + 0.002322833999642171, + 0.002414415997918695, + 0.0024147079966496676, + 0.002708500003791414, + 0.0024257079930976033, + 0.002484500000718981, + 0.0022444589994847775, + 0.0022865840001031756, + 0.002419374999590218, + 0.0022712080099154264 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 10.591354497591965, + "p95": 11.10275364771951, + "min": 10.426458000438288, + "max": 11.449290992459282, + "iqr": 0.11417699352023192, + "mean": 10.66324579733191, + "cv": 0.026937646031769874 + }, + "prefill_ms": { + "count": 10, + "median": 10.565791999397334, + "p95": 10.76050380434026, + "min": 10.497875002329238, + "max": 10.864416006370448, + "iqr": 0.08428149521932937, + "mean": 10.591629102418665, + "cv": 0.010018348380661965 + }, + "decode_ms_per_step": { + "count": 320, + "median": 2.3377914912998676, + "p95": 2.484937501139939, + "min": 2.1941670129308477, + "max": 3.120666995528154, + "iqr": 0.09595849405741319, + "mean": 2.350631521539981, + "cv": 0.041039944623484814 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1705.4688429670255, + "p95": 1717.7528485667824, + "min": 1673.7523702926712, + "max": 1720.5371328145834, + "iqr": 9.745294743858722, + "mean": 1701.8010311832336, + "cv": 0.009196810391717103 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1495.00647899243, + "p95": 1501.0348082666987, + "min": 1470.3855500870638, + "max": 1501.293904477687, + "iqr": 6.879359099357998, + "mean": 1491.7144404636665, + "cv": 0.007624398925698557 + }, + "memory": { + "peak_extra_bytes": 23597056, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "reference", + "batch": 4, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.023986000000149943, + "prefill": 0.020314374996814877, + "decode_steps": [ + 0.005168707997654565, + 0.005015125003410503, + 0.004912042000796646, + 0.004947458000970073, + 0.00466291701013688, + 0.0049125420046038926, + 0.0047997500078054145, + 0.005050333013059571, + 0.004742124991025776, + 0.00483525000163354, + 0.00477970800420735, + 0.0045283330109668896, + 0.004814458996406756, + 0.004664624997531064, + 0.004903499997453764, + 0.004842708993237466, + 0.004809707999811508, + 0.004690999994636513, + 0.004806167009519413, + 0.004455166999832727, + 0.004917083992040716, + 0.005768292001448572, + 0.004597207996994257, + 0.0048982080043060705, + 0.005280792000121437, + 0.00504591700155288, + 0.004594375001033768, + 0.004909958006464876, + 0.004859125008806586, + 0.004836624997551553, + 0.004960417005349882, + 0.004723167003248818 + ] + }, + { + "forward": 0.0232513749942882, + "prefill": 0.02034566599468235, + "decode_steps": [ + 0.0058461669977987185, + 0.005113999999593943, + 0.005048125007306226, + 0.004998709002393298, + 0.004586208000546321, + 0.005138124994118698, + 0.005933708991506137, + 0.004839125002035871, + 0.004850707991863601, + 0.0047119579976424575, + 0.00473670799692627, + 0.005081707990029827, + 0.004745999991428107, + 0.0047628329921280965, + 0.00491695900564082, + 0.004777624999405816, + 0.004783875003340654, + 0.0047527080023428425, + 0.005298082993249409, + 0.004758124996442348, + 0.004732165994937532, + 0.004647416993975639, + 0.004722166995634325, + 0.005109207995701581, + 0.004662375009502284, + 0.005158291009138338, + 0.005265875006443821, + 0.005245583000942133, + 0.00503450000542216, + 0.0050052919978043064, + 0.004829166995477863, + 0.004993875001673587 + ] + }, + { + "forward": 0.022705791008775122, + "prefill": 0.020364292009617202, + "decode_steps": [ + 0.004762124997796491, + 0.004736333998152986, + 0.00485991699679289, + 0.0047474169987253845, + 0.004795624990947545, + 0.0045196249993750826, + 0.0045763749949401245, + 0.01403375000518281, + 0.008677750010974705, + 0.0052444580069277436, + 0.005206625006394461, + 0.005038333998527378, + 0.004957917000865564, + 0.004856166007812135, + 0.004997249998268671, + 0.004789083992363885, + 0.0048214580019703135, + 0.0047307909990195185, + 0.004945041990140453, + 0.00488591700559482, + 0.004777541995281354, + 0.004849707998801023, + 0.004982207989087328, + 0.004686375003075227, + 0.00520945800235495, + 0.004945790991769172, + 0.004729833002784289, + 0.004476583999348804, + 0.005003708996810019, + 0.005094958993140608, + 0.00459150000824593, + 0.0048786249972181395 + ] + }, + { + "forward": 0.024043040990363806, + "prefill": 0.020344832999398932, + "decode_steps": [ + 0.005077791996882297, + 0.005155082995770499, + 0.005718417000025511, + 0.005184000008739531, + 0.00488795900309924, + 0.004383499996038154, + 0.0045335000031627715, + 0.004857958003412932, + 0.004823750001378357, + 0.004731749999336898, + 0.004734542002552189, + 0.004726792001747526, + 0.004535125000984408, + 0.004668417008360848, + 0.004621416999725625, + 0.004598083003656939, + 0.004837000000406988, + 0.0048614999977871776, + 0.004731374996481463, + 0.004886333001195453, + 0.005143709000549279, + 0.004708833002950996, + 0.004695333002018742, + 0.005189749994315207, + 0.004929875009111129, + 0.004413458998897113, + 0.005117166991112754, + 0.004772750005940907, + 0.004728666986920871, + 0.005197541991947219, + 0.005046666992711835, + 0.0048910000041360036 + ] + }, + { + "forward": 0.02368199999909848, + "prefill": 0.020418041007360443, + "decode_steps": [ + 0.005378583009587601, + 0.00497900000482332, + 0.004858624990447424, + 0.004670249996706843, + 0.004836833992158063, + 0.004579084008582868, + 0.004823208000743762, + 0.00476383299974259, + 0.004789208003785461, + 0.004762374999700114, + 0.004610957999830134, + 0.004883917004917748, + 0.005148374999407679, + 0.004619374987669289, + 0.004786916004377417, + 0.004799374990398064, + 0.004790792008861899, + 0.004678541998146102, + 0.004613041994161904, + 0.004709875007392839, + 0.00469524999789428, + 0.0046068329975241795, + 0.005123124996316619, + 0.005006124993087724, + 0.00477562501328066, + 0.004862834000959992, + 0.004813583000213839, + 0.005019000003812835, + 0.0048746249958639964, + 0.004368750000139698, + 0.0044535420020110905, + 0.004793582993443124 + ] + }, + { + "forward": 0.02339183299045544, + "prefill": 0.020355707994895056, + "decode_steps": [ + 0.004909291004878469, + 0.004836041989619844, + 0.0049445000040577725, + 0.004875500002526678, + 0.00478720800310839, + 0.004857958003412932, + 0.004818291999981739, + 0.004630499999620952, + 0.004557207998004742, + 0.004736417002277449, + 0.004853416990954429, + 0.004753208995680325, + 0.00482699999702163, + 0.004604250003467314, + 0.004261665992089547, + 0.004653666997910477, + 0.004788209000253119, + 0.004743708996102214, + 0.004737625000416301, + 0.004672334005590528, + 0.005193833989324048, + 0.00525920900690835, + 0.005054124994785525, + 0.0049887079949257895, + 0.0050860420014942065, + 0.005034332993091084, + 0.004800334005267359, + 0.0047874169977148995, + 0.0046290830068755895, + 0.004926124995108694, + 0.004631000003428198, + 0.004796667009941302 + ] + }, + { + "forward": 0.02421920899359975, + "prefill": 0.020575540998834185, + "decode_steps": [ + 0.00583395900321193, + 0.0049997500027529895, + 0.004906707996269688, + 0.004822457995032892, + 0.004747457991470583, + 0.00490029199863784, + 0.004666624998208135, + 0.004794417007360607, + 0.004831666999962181, + 0.004799834001460113, + 0.004734249989269301, + 0.004590625001583248, + 0.004696084011811763, + 0.005345458994270302, + 0.005040417003328912, + 0.004263916998752393, + 0.0047231659991666675, + 0.0046439580037258565, + 0.004789790997165255, + 0.004613666998920962, + 0.004778834001626819, + 0.005110749989398755, + 0.004973750008502975, + 0.004758958995807916, + 0.004654792006476782, + 0.004681457998231053, + 0.0048608750075800344, + 0.004709875007392839, + 0.004858750005951151, + 0.004964000007021241, + 0.004869875003350899, + 0.004960707999998704 + ] + }, + { + "forward": 0.02377379100653343, + "prefill": 0.02046600000176113, + "decode_steps": [ + 0.005286083003738895, + 0.0047659169940743595, + 0.004965125001035631, + 0.004756291993544437, + 0.0048334580060327426, + 0.004730291999294423, + 0.0048565420001978055, + 0.004737042007036507, + 0.004928791997372173, + 0.00477675000729505, + 0.004765375007991679, + 0.00474216700240504, + 0.004780000002938323, + 0.004761791991768405, + 0.0048565839970251545, + 0.004571792000206187, + 0.004830332996789366, + 0.005256083997664973, + 0.005078041009255685, + 0.00469225000415463, + 0.004571833007503301, + 0.0046830000064801425, + 0.0049993749998975545, + 0.004782541989698075, + 0.004783249998581596, + 0.00479750000522472, + 0.00479599999380298, + 0.004876791004789993, + 0.004991124995285645, + 0.004719040996860713, + 0.004381833001389168, + 0.00489150000794325 + ] + }, + { + "forward": 0.022848459004308097, + "prefill": 0.02033425000263378, + "decode_steps": [ + 0.005309625004883856, + 0.004794041000423022, + 0.004784624994499609, + 0.004738083996926434, + 0.00468216699664481, + 0.004463958000997081, + 0.004782541000167839, + 0.004742375007481314, + 0.0047119579976424575, + 0.00461241599987261, + 0.004788625010405667, + 0.004542624999885447, + 0.0048324590025004, + 0.004765000005136244, + 0.0047847499954514205, + 0.004781166004249826, + 0.0046859169960953295, + 0.004761791002238169, + 0.0048799579963088036, + 0.004599541003699414, + 0.0050327079952694476, + 0.00492104199656751, + 0.004912291988148354, + 0.00493675000325311, + 0.004810665996046737, + 0.004911707990686409, + 0.004532291000941768, + 0.005342624994227663, + 0.004949291003867984, + 0.005502250001882203, + 0.004728667001472786, + 0.00504591700155288 + ] + }, + { + "forward": 0.023988666987861507, + "prefill": 0.02068366701132618, + "decode_steps": [ + 0.005804375003208406, + 0.004910167001071386, + 0.004918125006952323, + 0.00492758299515117, + 0.004790457998751663, + 0.0048081250133691356, + 0.004757624992635101, + 0.0048862080002436414, + 0.004722124998806976, + 0.004878292005741969, + 0.0045409159938571975, + 0.004739249998237938, + 0.004849916993407533, + 0.004756583002745174, + 0.004764000012073666, + 0.004835458006709814, + 0.004733458990813233, + 0.004880249995039776, + 0.004716290990472771, + 0.004668375011533499, + 0.00459187499654945, + 0.004793375002918765, + 0.005243042003712617, + 0.004636958998162299, + 0.005365916003938764, + 0.005229041009442881, + 0.005024041005526669, + 0.005080165996332653, + 0.004992249989300035, + 0.0047812920092837885, + 0.004857958003412932, + 0.0047150000027613714 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 23.727895502815954, + "p95": 24.139933392143575, + "min": 22.705791008775122, + "max": 24.21920899359975, + "iqr": 0.7015107476036064, + "mean": 23.589016597543377, + "cv": 0.022094737809549913 + }, + "prefill_ms": { + "count": 10, + "median": 20.36000000225613, + "p95": 20.635010305704782, + "min": 20.314374996814877, + "max": 20.68366701132618, + "iqr": 0.1089690049411729, + "mean": 20.420237301732413, + "cv": 0.0059476167383393005 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.808916506590322, + "p95": 5.28105655030231, + "min": 4.261665992089547, + "max": 14.03375000518281, + "iqr": 0.22190600066096522, + "mean": 4.894272940782685, + "cv": 0.12336879370647398 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 824.9668869508978, + "p95": 832.6477854430717, + "min": 760.0577673367038, + "max": 834.0118258664173, + "iqr": 9.198049788676713, + "mean": 817.8384654886928, + "cv": 0.02676214718198613 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 728.8538539052242, + "p95": 735.1395218418339, + "min": 678.0647084351875, + "max": 736.0844824261032, + "iqr": 7.829636113073775, + "mean": 723.3975054084729, + "cv": 0.0237302464790177 + }, + "memory": { + "peak_extra_bytes": 59772928, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "auto", + "batch": 4, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.028820375009672716 + }, + { + "forward": 0.027921084008994512 + }, + { + "forward": 0.027956541001913138 + }, + { + "forward": 0.028124874996137805 + }, + { + "forward": 0.0282078750024084 + }, + { + "forward": 0.02836224999919068 + }, + { + "forward": 0.028039166005328298 + }, + { + "forward": 0.0287336250039516 + }, + { + "forward": 0.028257540994673036 + }, + { + "forward": 0.027928333991440013 + } + ], + "forward_ms": { + "count": 10, + "median": 28.166374999273103, + "p95": 28.781337507098215, + "min": 27.921084008994512, + "max": 28.820375009672716, + "iqr": 0.3588754952943418, + "mean": 28.23516660137102, + "cv": 0.01138082810337151 + }, + "memory": { + "peak_extra_bytes": 47194112, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "sdpa", + "batch": 4, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.021733625006163493, + "prefill": 0.0222877089981921, + "decode_steps": [ + 0.00426625000545755, + 0.003514124997309409, + 0.003918207992683165, + 0.003979041997808963, + 0.0040825000032782555, + 0.0037860000011278316, + 0.004081125007360242, + 0.0038528750010300428, + 0.003859459000523202, + 0.0030474169907392934, + 0.003199500002665445, + 0.0032107919978443533, + 0.00308341599884443, + 0.003133958001853898, + 0.00369279200094752, + 0.002947874992969446, + 0.0030824170098640025, + 0.0030124580080155283, + 0.0030372500041266903, + 0.00297220800712239, + 0.002982707999763079, + 0.003166374997817911, + 0.0029924999980721623, + 0.0028931669949088246, + 0.0029676669946638867, + 0.003286167004262097, + 0.002966749991173856, + 0.0030121669988147914, + 0.0028941660129930824, + 0.0029873750027036294, + 0.0031315420055761933, + 0.002915334000135772 + ] + }, + { + "forward": 0.021468749997438863, + "prefill": 0.02144941700680647, + "decode_steps": [ + 0.0031477919983444735, + 0.0028016670112265274, + 0.0027940830041188747, + 0.0029095420031808317, + 0.0028605419938685372, + 0.003094166997470893, + 0.0028770829958375543, + 0.003045749996090308, + 0.002911542003857903, + 0.0034768749901559204, + 0.002602583001134917, + 0.0025697499950183555, + 0.0026152500067837536, + 0.0025572079903213307, + 0.0025447920052101836, + 0.0025854999985313043, + 0.0025398750003660098, + 0.0026319580065319315, + 0.002564499998698011, + 0.002704457991058007, + 0.0025318330008303747, + 0.0026867079868679866, + 0.0025834590051090345, + 0.002571416989667341, + 0.002508250006940216, + 0.0026931250031339005, + 0.0026592919894028455, + 0.00271483400138095, + 0.0026879159995587543, + 0.002787584002362564, + 0.002934749994892627, + 0.0028643749974435195 + ] + }, + { + "forward": 0.02192404100787826, + "prefill": 0.021351125004002824, + "decode_steps": [ + 0.002950291003799066, + 0.002980250006658025, + 0.0027704159874701872, + 0.002817083994159475, + 0.002798041998175904, + 0.003123874994344078, + 0.0027849999896716326, + 0.002834166996763088, + 0.003108000004431233, + 0.0024587499938206747, + 0.00240995800413657, + 0.0024973749968921766, + 0.0024895000096876174, + 0.002474958004313521, + 0.002431959001114592, + 0.002490583006874658, + 0.0024551669921493158, + 0.0029150420014047995, + 0.0025975000025937334, + 0.0025389590009581298, + 0.0027758340002037585, + 0.0026089159946423024, + 0.0025504999939585105, + 0.0026272499962942675, + 0.0026087080041179433, + 0.0025670420000096783, + 0.0027094999968539923, + 0.0026455420011188835, + 0.002738416995271109, + 0.002687875006813556, + 0.0027116249984828755, + 0.0027758329961216077 + ] + }, + { + "forward": 0.021755708003183827, + "prefill": 0.021508374993572943, + "decode_steps": [ + 0.00291108300734777, + 0.002884457993786782, + 0.0028781250002793968, + 0.002806707998388447, + 0.0027874169900314882, + 0.0028437089931685477, + 0.0027924589958274737, + 0.0029238339920993894, + 0.0028009170055156574, + 0.0025810830120462924, + 0.002510999998776242, + 0.0025002500042319298, + 0.0028203330002725124, + 0.0026972089981427416, + 0.0025884170026984066, + 0.002812292004819028, + 0.0026074579945998266, + 0.0025975829921662807, + 0.0028667079895967618, + 0.002864833004423417, + 0.0028424579941201955, + 0.0029064999980619177, + 0.0028442079928936437, + 0.0031098329927772284, + 0.003122584006632678, + 0.0030547500064130872, + 0.00313295800879132, + 0.0030997919966466725, + 0.0033084580063587055, + 0.0032927910069702193, + 0.003258334007114172, + 0.003246167005272582 + ] + }, + { + "forward": 0.02173562499228865, + "prefill": 0.022562500002095476, + "decode_steps": [ + 0.0030072919907979667, + 0.0029428330017253757, + 0.0028624999977182597, + 0.00293445801071357, + 0.0028447080112528056, + 0.002824958006385714, + 0.0027512500091688707, + 0.002858332998584956, + 0.002889374998630956, + 0.0025346250040456653, + 0.002534416998969391, + 0.0025662500120233744, + 0.0025034580030478537, + 0.0025603749963920563, + 0.0024948329955805093, + 0.002506833989173174, + 0.002562374997069128, + 0.002512250008294359, + 0.0025200420059263706, + 0.002562207999289967, + 0.002502083996660076, + 0.002610750001622364, + 0.002490999992005527, + 0.0028044170030625537, + 0.002481624993379228, + 0.0027021669957321137, + 0.002558042004238814, + 0.00272995799605269, + 0.0025567499978933483, + 0.0026269169902661815, + 0.002641541010234505, + 0.002681417012354359 + ] + }, + { + "forward": 0.022468416995252483, + "prefill": 0.02147133299149573, + "decode_steps": [ + 0.003062917006900534, + 0.0029333330021472648, + 0.002812541992170736, + 0.0028296250093262643, + 0.0028637080104090273, + 0.0028001669998047873, + 0.0027655419980874285, + 0.0028245840076124296, + 0.0027969579969067127, + 0.0025297079992014915, + 0.002562084002420306, + 0.002470666993758641, + 0.0025055000005522743, + 0.0025034999998752028, + 0.0024038749979808927, + 0.0025554160092724487, + 0.0024699579953448847, + 0.002769624989014119, + 0.002623542008223012, + 0.0025874579878291115, + 0.0025432919937884435, + 0.0026609579945215955, + 0.002582207991508767, + 0.002811708996887319, + 0.002593666999018751, + 0.002562333000241779, + 0.002545583003666252, + 0.0025812089879764244, + 0.0026788749964907765, + 0.002734167006565258, + 0.002672292001079768, + 0.0026412909937789664 + ] + }, + { + "forward": 0.02179154200712219, + "prefill": 0.0215587080019759, + "decode_steps": [ + 0.002955250005470589, + 0.0029425829998217523, + 0.002914917000452988, + 0.0027604170027188957, + 0.002864459005650133, + 0.002870209005777724, + 0.0027656659949570894, + 0.00283912499435246, + 0.002950750000309199, + 0.002497458001016639, + 0.0025617080100346357, + 0.0025036670122062787, + 0.002689792003366165, + 0.0024909169878810644, + 0.002498582995031029, + 0.0025495419977232814, + 0.0025064999936148524, + 0.005739832995459437, + 0.005817082987050526, + 0.004484167002374306, + 0.005505666995304637, + 0.002846167000825517, + 0.0028104999946663156, + 0.0027562090108403936, + 0.0027075830003013834, + 0.002806332995533012, + 0.002667332999408245, + 0.0029264580080052838, + 0.0028440000023692846, + 0.0029921249952167273, + 0.002882041997509077, + 0.002842749992851168 + ] + }, + { + "forward": 0.021467999991727993, + "prefill": 0.021521500006201677, + "decode_steps": [ + 0.0029563340067397803, + 0.0028710420010611415, + 0.0027834170032292604, + 0.002806791992043145, + 0.0028152919985586777, + 0.0027708749985322356, + 0.0031923329952405766, + 0.0028042909980285913, + 0.002857333005522378, + 0.00250716699520126, + 0.0024253750016214326, + 0.0027287499979138374, + 0.0024916670081438497, + 0.002492875006282702, + 0.002504875010345131, + 0.002547374999267049, + 0.002460542003973387, + 0.0025472919951425865, + 0.002510209000320174, + 0.002577999999630265, + 0.0024679580092197284, + 0.0026900829980149865, + 0.002533208011300303, + 0.0025179160002153367, + 0.0027811669860966504, + 0.002589625000837259, + 0.0025540409988025203, + 0.002536917003453709, + 0.00264604200492613, + 0.0024794160126475617, + 0.0025374999968335032, + 0.002653624993399717 + ] + }, + { + "forward": 0.02152758400188759, + "prefill": 0.021411209003417753, + "decode_steps": [ + 0.003671000013127923, + 0.002893625001888722, + 0.0029997079982422292, + 0.0028451669932110235, + 0.002821999994921498, + 0.0028238329978194088, + 0.002858540989109315, + 0.0028039590106345713, + 0.002846000003046356, + 0.0025165410042973235, + 0.0025139170029433444, + 0.0025741249992279336, + 0.0027888750046258792, + 0.002670750007382594, + 0.0028980409988434985, + 0.002566957991803065, + 0.0024432500067632645, + 0.0026603750011418015, + 0.002557457992224954, + 0.002516874999855645, + 0.0025472919951425865, + 0.002611749994684942, + 0.0025735410017659888, + 0.0025564170064171776, + 0.0025630839954828843, + 0.002532832993892953, + 0.0025286250020144507, + 0.0026073750050272793, + 0.002601625004899688, + 0.002710959000978619, + 0.002664125000592321, + 0.0027937080012634397 + ] + }, + { + "forward": 0.021812750012031756, + "prefill": 0.02188587500131689, + "decode_steps": [ + 0.004112666996661574, + 0.004020708001917228, + 0.004016833001514897, + 0.0036990829976275563, + 0.003656000000773929, + 0.0037056669971207157, + 0.0036660420009866357, + 0.003835458992398344, + 0.003072583000175655, + 0.0027985839988105, + 0.0026614579983288422, + 0.0026520419924054295, + 0.00287108299380634, + 0.002991542001836933, + 0.0028923330100951716, + 0.0028643749974435195, + 0.002845208000508137, + 0.003296832990599796, + 0.0032735830027377233, + 0.0033044580050045624, + 0.003912959000444971, + 0.003528624991304241, + 0.0036536669940687716, + 0.003420624998398125, + 0.0037272920017130673, + 0.00456366699654609, + 0.0047927080013323575, + 0.004696333009633236, + 0.004736208007670939, + 0.0042064580047735944, + 0.004429166001500562, + 0.00480254199646879 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 21.745666497736238, + "p95": 22.223447800934082, + "min": 21.467999991727993, + "max": 22.468416995252483, + "iqr": 0.228353757847799, + "mean": 21.76860420149751, + "cv": 0.013324729361266678 + }, + "prefill_ms": { + "count": 10, + "median": 21.51493749988731, + "p95": 22.438844050338957, + "min": 21.351125004002824, + "max": 22.562500002095476, + "iqr": 0.349187248502858, + "mean": 21.700775100907777, + "cv": 0.019015894549888494 + }, + "decode_ms_per_step": { + "count": 320, + "median": 2.7930834985454567, + "p95": 4.017026751535013, + "min": 2.4038749979808927, + "max": 5.817082987050526, + "iqr": 0.38644800952170044, + "mean": 2.9043101780644065, + "cv": 0.1800227217290353 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1466.1415231297983, + "p95": 1508.1381291939697, + "min": 1096.7656545100376, + "max": 1512.2992337201333, + "iqr": 179.62951139362963, + "mean": 1392.5480533970883, + "cv": 0.10428324491797754 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1177.1706519649179, + "p95": 1202.725097311766, + "min": 923.5697941837639, + "max": 1205.717731581523, + "iqr": 113.00113480422442, + "mean": 1125.04480183632, + "cv": 0.08722392959337955 + }, + "memory": { + "peak_extra_bytes": 47194112, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "reference", + "batch": 4, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.06431924999924377, + "prefill": 0.05856133298948407, + "decode_steps": [ + 0.00948562499252148, + 0.007742667003185488, + 0.00714333399082534, + 0.0074794579995796084, + 0.006387749992427416, + 0.00608258399006445, + 0.006311208999250084, + 0.0061804169963579625, + 0.00594366600853391, + 0.006137832999229431, + 0.005848832995980047, + 0.006206749996636063, + 0.005714250000892207, + 0.0055890420044306666, + 0.005326583006535657, + 0.005656833003740758, + 0.0056208749883808196, + 0.005582207988481969, + 0.005401208996772766, + 0.005338292001397349, + 0.005447916992125101, + 0.005478042003232986, + 0.00537241599522531, + 0.005610457999864593, + 0.005477291997522116, + 0.00570516700099688, + 0.005422665999503806, + 0.005288999993354082, + 0.005397000000812113, + 0.005279042001347989, + 0.0053044579981360584, + 0.005315709000569768 + ] + }, + { + "forward": 0.062088166989269666, + "prefill": 0.05957929100259207, + "decode_steps": [ + 0.014928541000699624, + 0.011469874996691942, + 0.009547833993565291, + 0.008134999996400438, + 0.007233291995362379, + 0.007366457997704856, + 0.006658582991803996, + 0.006753124995157123, + 0.0063751660054549575, + 0.005998208012897521, + 0.00609804100531619, + 0.005940040995483287, + 0.0059177080111112446, + 0.006014625003444962, + 0.005806708999443799, + 0.006151125009637326, + 0.006092499999795109, + 0.006351750009343959, + 0.006193833003635518, + 0.005253499999525957, + 0.005390249993070029, + 0.0056358750007348135, + 0.005715959006920457, + 0.005609625004581176, + 0.005264834006084129, + 0.005424834002042189, + 0.0053816250001545995, + 0.005717959007597528, + 0.005478334001963958, + 0.005527959001483396, + 0.005156749990419485, + 0.005153582998900674 + ] + }, + { + "forward": 0.0603976669954136, + "prefill": 0.05798808300460223, + "decode_steps": [ + 0.00613470800453797, + 0.00550945800205227, + 0.0054859999945620075, + 0.0053039589984109625, + 0.005117166001582518, + 0.005122791990288533, + 0.005497332997038029, + 0.005236333003267646, + 0.005259042009129189, + 0.005332000000635162, + 0.00538645799679216, + 0.006778666996979155, + 0.00586329099314753, + 0.005497749996720813, + 0.00568087499414105, + 0.00574779199087061, + 0.005904916004510596, + 0.006001874993671663, + 0.01518312499683816, + 0.01053591699746903, + 0.006148040993139148, + 0.005696958003682084, + 0.005991250000079162, + 0.005641291994834319, + 0.005687999990186654, + 0.005399082991061732, + 0.005648416001349688, + 0.005437625004560687, + 0.005409583012806252, + 0.005472749995533377, + 0.005497500009369105, + 0.005282083002384752 + ] + }, + { + "forward": 0.0652627080125967, + "prefill": 0.06164633399748709, + "decode_steps": [ + 0.01675716700265184, + 0.011625250001088716, + 0.00951754099514801, + 0.008392749994527549, + 0.007565166990389116, + 0.007067124999593943, + 0.006864167007734068, + 0.006263458009925671, + 0.00639733299612999, + 0.006451041990658268, + 0.0064613340073265135, + 0.006412707996787503, + 0.006895915998029523, + 0.006232790998183191, + 0.0054875000059837475, + 0.005890749991522171, + 0.005864541002665646, + 0.005699832996469922, + 0.0057086250017164275, + 0.005775333993369713, + 0.0056871249980758876, + 0.005946167002548464, + 0.005637333000777289, + 0.005542042010347359, + 0.005391832994064316, + 0.005593791007413529, + 0.005586875005974434, + 0.005587749998085201, + 0.0055958750017452985, + 0.005473540993989445, + 0.005546125001274049, + 0.005586208993918262 + ] + }, + { + "forward": 0.06077833399467636, + "prefill": 0.05872545900638215, + "decode_steps": [ + 0.006878208005218767, + 0.005610584004898556, + 0.005524916006834246, + 0.005446708993986249, + 0.00533879199065268, + 0.005291374996886589, + 0.005316417009453289, + 0.0053007090027676895, + 0.005223416999797337, + 0.0051672910049092025, + 0.00513374998990912, + 0.005333083012374118, + 0.0051340420031920075, + 0.005530459005967714, + 0.005219209007918835, + 0.005262458013021387, + 0.00527541599876713, + 0.005091999992146157, + 0.005408124998211861, + 0.005260374993667938, + 0.005117624998092651, + 0.0052483339968603104, + 0.0052914999978384, + 0.005271625006571412, + 0.005252125003607944, + 0.005263041006401181, + 0.005275916992104612, + 0.005195625009946525, + 0.005285124992951751, + 0.005253208000794984, + 0.005312042005243711, + 0.005114207990118302 + ] + }, + { + "forward": 0.06023295799968764, + "prefill": 0.057842667010845616, + "decode_steps": [ + 0.006214749999344349, + 0.005926792000536807, + 0.005883916994207539, + 0.00595975000760518, + 0.006714707997161895, + 0.006297499989159405, + 0.005765624999185093, + 0.00627712499408517, + 0.0058493749966146424, + 0.005476542006363161, + 0.005563333994359709, + 0.005327165999915451, + 0.005389041994931176, + 0.005150541997863911, + 0.0050365000060992315, + 0.005614833004074171, + 0.006130916008260101, + 0.005375334003474563, + 0.005428584001492709, + 0.005503834006958641, + 0.005162375004147179, + 0.0053383750055218115, + 0.005448791998787783, + 0.0052780419937334955, + 0.0052632089937105775, + 0.005341041993233375, + 0.005367916994146071, + 0.005256542004644871, + 0.005225458997301757, + 0.00520383300317917, + 0.005242917002760805, + 0.005266292006126605 + ] + }, + { + "forward": 0.06372395899961703, + "prefill": 0.05992054200032726, + "decode_steps": [ + 0.01578962500207126, + 0.01157595799304545, + 0.00930412499292288, + 0.008863290990120731, + 0.007749791999231093, + 0.006806292003602721, + 0.00697191599465441, + 0.00660575000802055, + 0.00658370899327565, + 0.0064099169976543635, + 0.006303957998170517, + 0.006126624997705221, + 0.005816250006319024, + 0.006246167002245784, + 0.006719083001371473, + 0.005854249990079552, + 0.005879541000467725, + 0.005756124999606982, + 0.006248916004551575, + 0.005375583001296036, + 0.0053080829966347665, + 0.005448166004498489, + 0.005356624998967163, + 0.005092000006698072, + 0.005346083999029361, + 0.00524125000811182, + 0.005054624998592772, + 0.00535937499080319, + 0.005274541006656364, + 0.005240457991021685, + 0.005399874993599951, + 0.00547674999688752 + ] + }, + { + "forward": 0.061030041993944906, + "prefill": 0.05968224999378435, + "decode_steps": [ + 0.010980249993735924, + 0.008484083999064751, + 0.008333792007761076, + 0.007729083998128772, + 0.007117375003872439, + 0.006758916992112063, + 0.006511874991701916, + 0.00648387499677483, + 0.006485040998086333, + 0.006116917007602751, + 0.005855209004948847, + 0.00624220899771899, + 0.005808874993817881, + 0.005170125004951842, + 0.005375791995902546, + 0.005273457994917408, + 0.005482583001139574, + 0.005530375012313016, + 0.005547374996240251, + 0.00546908300020732, + 0.0057764590019360185, + 0.005426208008429967, + 0.005515083001228049, + 0.005193459001020528, + 0.005326749989762902, + 0.0049987920065177605, + 0.005076667002867907, + 0.005330875006620772, + 0.005375875000027008, + 0.005191792006371543, + 0.00565900000219699, + 0.005466582995723002 + ] + }, + { + "forward": 0.06041425000876188, + "prefill": 0.057767417005379684, + "decode_steps": [ + 0.006357749996823259, + 0.005599625001195818, + 0.005437332991277799, + 0.005212332995142788, + 0.005467833994771354, + 0.005197458012844436, + 0.005346750011085533, + 0.005496333003975451, + 0.005145041999639943, + 0.005204790999414399, + 0.005520499995327555, + 0.005355208006221801, + 0.005228666006587446, + 0.005188000010093674, + 0.005175875005079433, + 0.005240875005256385, + 0.005327999999281019, + 0.005398875000537373, + 0.00536570799886249, + 0.005364042008295655, + 0.005302042001858354, + 0.00513074999616947, + 0.0051892500050598755, + 0.0052572499989764765, + 0.005337333001079969, + 0.005368750003981404, + 0.005245250009465963, + 0.005311209009960294, + 0.00524441699963063, + 0.0054497920064022765, + 0.005214666991378181, + 0.005239958001766354 + ] + }, + { + "forward": 0.06408275000285357, + "prefill": 0.0606276250036899, + "decode_steps": [ + 0.015258707993780263, + 0.010942207998596132, + 0.00879083399195224, + 0.008160750003298745, + 0.007755040991469286, + 0.007099707989254966, + 0.007078333001118153, + 0.006764457997633144, + 0.006384417007211596, + 0.0072835419996408746, + 0.00686124998901505, + 0.006320707994746044, + 0.006289666998782195, + 0.005850374989677221, + 0.00591283300309442, + 0.005752500001108274, + 0.00532050000037998, + 0.005412375001469627, + 0.005269542001769878, + 0.0050936670013470575, + 0.005573916001594625, + 0.005478667007992044, + 0.005233250005403534, + 0.005324375000782311, + 0.005288083993946202, + 0.006245833996217698, + 0.005519208003534004, + 0.005326541999238543, + 0.005774417004431598, + 0.01500800000212621, + 0.0092615419998765, + 0.00594124999770429 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 61.559104491607286, + "p95": 64.83815190658788, + "min": 60.23295799968764, + "max": 65.2627080125967, + "iqr": 3.4877812468039338, + "mean": 62.23300849960651, + "cv": 0.03097203243582733 + }, + "prefill_ms": { + "count": 10, + "median": 59.15237500448711, + "p95": 61.187914950278355, + "min": 57.767417005379684, + "max": 61.64633399748709, + "iqr": 1.7295734978688415, + "mean": 59.23410010145744, + "cv": 0.021724285631782844 + }, + "decode_ms_per_step": { + "count": 320, + "median": 5.530417009140365, + "p95": 9.263671149528818, + "min": 4.9987920065177605, + "max": 16.75716700265184, + "iqr": 0.8808222446532454, + "mean": 6.117201578035747, + "cv": 0.2783987275592941 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 658.1278656268934, + "p95": 749.5906604317167, + "min": 572.510972511418, + "max": 750.1712352387557, + "iqr": 98.2094890287824, + "mean": 659.3308050274263, + "cv": 0.09647657580390458 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 505.2900960793228, + "p95": 558.98261090305, + "min": 450.38051281726825, + "max": 559.7118947704222, + "iqr": 61.455423995699505, + "mean": 504.7208388935087, + "cv": 0.07784209082510925 + }, + "memory": { + "peak_extra_bytes": 220209152, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "auto", + "batch": 4, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.07620374999532942 + }, + { + "forward": 0.07067662499321159 + }, + { + "forward": 0.07216541600064375 + }, + { + "forward": 0.07278395901084878 + }, + { + "forward": 0.07140633299422916 + }, + { + "forward": 0.07158674999664072 + }, + { + "forward": 0.07336745799693745 + }, + { + "forward": 0.07123095799761359 + }, + { + "forward": 0.07134783299989067 + }, + { + "forward": 0.07316991699917708 + } + ], + "forward_ms": { + "count": 10, + "median": 71.87608299864223, + "p95": 74.92741859605303, + "min": 70.67662499321159, + "max": 76.20374999532942, + "iqr": 1.7109695036197081, + "mean": 72.39389989845222, + "cv": 0.022221765914284968 + }, + "memory": { + "peak_extra_bytes": 94388224, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "sdpa", + "batch": 4, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.05388554099772591, + "prefill": 0.05264037499728147, + "decode_steps": [ + 0.007656333007616922, + 0.008583959002862684, + 0.007142416012356989, + 0.00742129200079944, + 0.006262291004532017, + 0.006121124999481253, + 0.006439832999603823, + 0.006039958985638805, + 0.005344999997760169, + 0.00531691599462647, + 0.004751375003252178, + 0.004724457990960218, + 0.004715917006251402, + 0.004772500004037283, + 0.0053296669939300045, + 0.0043476669961819425, + 0.00440179200086277, + 0.0043599580094451085, + 0.004421207995619625, + 0.004421541001647711, + 0.004283124988432974, + 0.0045405840064631775, + 0.004413957998622209, + 0.0045587089989567176, + 0.00440458299999591, + 0.004286625000531785, + 0.0041920420044334605, + 0.004169625011854805, + 0.0049685839912854135, + 0.004186500009382144, + 0.004196082998532802, + 0.004006209012004547 + ] + }, + { + "forward": 0.04909720900468528, + "prefill": 0.04934395800228231, + "decode_steps": [ + 0.0037326669989852235, + 0.0036493750085355714, + 0.003438500003539957, + 0.0034711670014075935, + 0.003585166996344924, + 0.003492541000014171, + 0.0037000839947722852, + 0.003606500002206303, + 0.003774791010073386, + 0.003714624996064231, + 0.0037121249915799126, + 0.0037482920015463606, + 0.003968417004216462, + 0.003969208002672531, + 0.003996750005171634, + 0.003992500001913868, + 0.004080416998476721, + 0.004136375006055459, + 0.004130250003072433, + 0.004189375002169982, + 0.004591832999722101, + 0.004756583002745174, + 0.005137375002959743, + 0.0047514589969068766, + 0.00499412500357721, + 0.00477895799849648, + 0.004771625011926517, + 0.0046531669941032305, + 0.004680375001044013, + 0.004636916011804715, + 0.004606041999068111, + 0.00477600000158418 + ] + }, + { + "forward": 0.04914779200043995, + "prefill": 0.04883908400370274, + "decode_steps": [ + 0.0037237499927869067, + 0.0034777920081978664, + 0.0035728330112760887, + 0.0033907909964909777, + 0.0033867080055642873, + 0.0034626660053618252, + 0.0033681669883662835, + 0.0036221250047674403, + 0.0034244999988004565, + 0.0034262919944012538, + 0.0034116249880753458, + 0.003496207995340228, + 0.0034195000043837354, + 0.003408042000955902, + 0.003446499991696328, + 0.003575375012587756, + 0.00351391700678505, + 0.003555292001692578, + 0.0035183750005671754, + 0.003727333009010181, + 0.0036890409974148497, + 0.003745666006579995, + 0.003719957996509038, + 0.004003167006885633, + 0.003962292001233436, + 0.003976374995545484, + 0.004050624993396923, + 0.004165457998169586, + 0.0041205830057151616, + 0.004187541999272071, + 0.004658958001527935, + 0.005320625001331791 + ] + }, + { + "forward": 0.052820749988313764, + "prefill": 0.05186408398731146, + "decode_steps": [ + 0.008804291996057145, + 0.007263374995090999, + 0.0067264170065755025, + 0.007068999999319203, + 0.006552709004608914, + 0.006148208995000459, + 0.0062707089964533225, + 0.006416667005396448, + 0.006437042000470683, + 0.0060719169996446, + 0.005723624999518506, + 0.00644966700929217, + 0.005191000003833324, + 0.005726999996113591, + 0.004735541995614767, + 0.005334542001946829, + 0.0046283749979920685, + 0.005857791999005713, + 0.004611458003637381, + 0.004653083000448532, + 0.005354500011890195, + 0.004648542002541944, + 0.005496459009009413, + 0.004374749987618998, + 0.0043222920066909865, + 0.004653291005524807, + 0.004614292003680021, + 0.004570791992591694, + 0.004476083995541558, + 0.004310750009608455, + 0.004498582988162525, + 0.00420233300246764 + ] + }, + { + "forward": 0.04931600000418257, + "prefill": 0.04868487500061747, + "decode_steps": [ + 0.0040021669992711395, + 0.0037288329913280904, + 0.0035318330046720803, + 0.003518291996442713, + 0.0033826250000856817, + 0.0034995000023627654, + 0.0035132079938193783, + 0.00337016700359527, + 0.0036625420034397393, + 0.0033946250041481107, + 0.0037920409959042445, + 0.003467249989625998, + 0.003518917001201771, + 0.0035528329899534583, + 0.003501458006212488, + 0.003450916992733255, + 0.0036287910043029115, + 0.0037434590049088, + 0.0038758750015404075, + 0.0037131670105736703, + 0.003934957989258692, + 0.004040250001708046, + 0.003950667000026442, + 0.0039931659994181246, + 0.004187124999589287, + 0.004197833011858165, + 0.004150042004766874, + 0.004762749988003634, + 0.005140415989444591, + 0.0048134579992620274, + 0.004607082999427803, + 0.004497415997320786 + ] + }, + { + "forward": 0.04890529200201854, + "prefill": 0.049118209004518576, + "decode_steps": [ + 0.003674499996122904, + 0.0035852500004693866, + 0.0035126669972669333, + 0.0034256249928148463, + 0.003414208986214362, + 0.0034128750121453777, + 0.0034860839950852096, + 0.0035019579954678193, + 0.003487249996396713, + 0.0036901660059811547, + 0.003458166989730671, + 0.003449499999987893, + 0.0034254999918630347, + 0.0035557499941205606, + 0.003554833005182445, + 0.0035574170033214614, + 0.0035707079950952902, + 0.003529499997966923, + 0.003768750000745058, + 0.004036124999402091, + 0.003718457999639213, + 0.0037834160029888153, + 0.003951958991819993, + 0.0039926250028656796, + 0.003947959005017765, + 0.004018040999653749, + 0.004141790996072814, + 0.004143292011576705, + 0.0041548750014044344, + 0.004895749996649101, + 0.004700500008766539, + 0.004761208008858375 + ] + }, + { + "forward": 0.05264737500692718, + "prefill": 0.05225633399095386, + "decode_steps": [ + 0.008159749995684251, + 0.007076708003296517, + 0.006323499997961335, + 0.006374208998749964, + 0.006455041002482176, + 0.00555954099399969, + 0.005864457998541184, + 0.006413540992070921, + 0.006422958002076484, + 0.0064687499980209395, + 0.006364582994137891, + 0.006852709004306234, + 0.005826417007483542, + 0.004808292011148296, + 0.004740667005535215, + 0.004633624994312413, + 0.0056602499971631914, + 0.0051845419948222116, + 0.004724707992863841, + 0.005803790991194546, + 0.004609416995663196, + 0.004754374996991828, + 0.004762000011396594, + 0.00433970800077077, + 0.004645791006623767, + 0.00444216700270772, + 0.004381833001389168, + 0.004650833012419753, + 0.004346500005340204, + 0.004619291998096742, + 0.004355958997621201, + 0.004380707992822863 + ] + }, + { + "forward": 0.049166667013196275, + "prefill": 0.04882999999972526, + "decode_steps": [ + 0.0036711249995278195, + 0.003778250000323169, + 0.0034822080051526427, + 0.0037005000049248338, + 0.003466708993073553, + 0.0036623749911086634, + 0.003361583992955275, + 0.003514916999847628, + 0.003427250005188398, + 0.0034182079980382696, + 0.0033699999912641943, + 0.00350474999868311, + 0.0035408340045250952, + 0.003547667001839727, + 0.003519250007229857, + 0.003711583005497232, + 0.00370420899707824, + 0.003762834006920457, + 0.003726666996954009, + 0.003996708997874521, + 0.004003624999313615, + 0.004106666994630359, + 0.0039842080004746094, + 0.004087207998964004, + 0.004149917003815062, + 0.00412941598915495, + 0.004731209002784453, + 0.004703665996203199, + 0.0046723330015083775, + 0.005521125000086613, + 0.0056375410058535635, + 0.0053483340016100556 + ] + }, + { + "forward": 0.04912675000377931, + "prefill": 0.048837875001481734, + "decode_steps": [ + 0.0037057909939903766, + 0.003575916009140201, + 0.003507082990836352, + 0.0035028340062126517, + 0.0034152089938288555, + 0.003385208998224698, + 0.003685667004901916, + 0.003414291000808589, + 0.0033651249977992848, + 0.003451625001616776, + 0.0034473340056138113, + 0.0035418339975876734, + 0.0033904589945450425, + 0.003746832997421734, + 0.0036277500039432198, + 0.003544042003341019, + 0.0036040830018464476, + 0.0037620000075548887, + 0.0037580000062007457, + 0.0037721249973401427, + 0.0037804160092491657, + 0.004025625006761402, + 0.003956041997298598, + 0.004026082999189384, + 0.003947707999031991, + 0.004168290994130075, + 0.0041367920057382435, + 0.004186000005574897, + 0.004110791996936314, + 0.004762458993354812, + 0.004801917006261647, + 0.00564795799436979 + ] + }, + { + "forward": 0.05264579100185074, + "prefill": 0.05388054200739134, + "decode_steps": [ + 0.00853908300632611, + 0.007199541010777466, + 0.006629291994613595, + 0.006774291003239341, + 0.006352041993523017, + 0.0063923330017132685, + 0.006496667003375478, + 0.006130415989900939, + 0.006180165990372188, + 0.005867083003977314, + 0.005958958005066961, + 0.006075124998460524, + 0.006211333005921915, + 0.006021916997269727, + 0.005994792009005323, + 0.00583487500261981, + 0.0056602499971631914, + 0.005682832997990772, + 0.004720500000985339, + 0.004717874995549209, + 0.004683291001128964, + 0.004785916986293159, + 0.00470158300595358, + 0.00474870900507085, + 0.004729958003736101, + 0.004734041998744942, + 0.004693042006692849, + 0.004570707998936996, + 0.004435374998138286, + 0.004705291998106986, + 0.004377416000352241, + 0.004607957991538569 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 49.24133350868942, + "p95": 53.406385043490445, + "min": 48.90529200201854, + "max": 53.88554099772591, + "iqr": 3.5149685027136, + "mean": 50.67591670231195, + "cv": 0.04009662347637667 + }, + "prefill_ms": { + "count": 10, + "median": 49.23108350340044, + "p95": 53.3224668528419, + "min": 48.68487500061747, + "max": 53.88054200739134, + "iqr": 3.320094238006277, + "mean": 50.42953359952662, + "cv": 0.03952450918389576 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.186812504485715, + "p95": 6.499469103437151, + "min": 3.361583992955275, + "max": 8.804291996057145, + "iqr": 1.102864509448409, + "mean": 4.506482553415481, + "cv": 0.2404378870438363 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 984.5591933370547, + "p95": 1063.8124063847442, + "min": 714.2352435116834, + "max": 1070.8780640245031, + "iqr": 293.96451820711195, + "mean": 912.223611570588, + "cv": 0.16810406649564144 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 714.6614236553116, + "p95": 756.1125326960132, + "min": 549.136556745803, + "max": 760.2432456557424, + "iqr": 173.1812181650888, + "mean": 669.2117422170492, + "cv": 0.13526830382301236 + }, + "memory": { + "peak_extra_bytes": 94388224, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "reference", + "batch": 4, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.1973761670087697, + "prefill": 0.19404441700316966, + "decode_steps": [ + 0.020648040997912176, + 0.0149997499975143, + 0.010194125003181398, + 0.010230540996417403, + 0.009348457999294624, + 0.009594832998118363, + 0.008309959011967294, + 0.007869541994296014, + 0.007651583000551909, + 0.007565459003672004, + 0.007666082994546741, + 0.0073781250102911144, + 0.007364125005551614, + 0.00812374999804888, + 0.007365833007497713, + 0.007341000004089437, + 0.007235207987832837, + 0.007289082990610041, + 0.007558665995020419, + 0.007144540999433957, + 0.007045584003208205, + 0.0069334579893620685, + 0.006958541998756118, + 0.006905332993483171, + 0.006891625002026558, + 0.0069818329939153045, + 0.007015625000349246, + 0.007249542002682574, + 0.007797124999342486, + 0.0070615830045426264, + 0.0070297909987857565, + 0.006759708994650282 + ] + }, + { + "forward": 0.19758041700697504, + "prefill": 0.19141937499807682, + "decode_steps": [ + 0.01989333299570717, + 0.01290983401122503, + 0.010454125003889203, + 0.00976095799705945, + 0.00953049999952782, + 0.008667667003464885, + 0.00815733300987631, + 0.007884666993049905, + 0.007591166999191046, + 0.007557791002909653, + 0.0075809999980265275, + 0.007690457990975119, + 0.008468499989248812, + 0.007511292002163827, + 0.007739541004411876, + 0.006735957998898812, + 0.007109791011316702, + 0.006938332997378893, + 0.007025290993624367, + 0.0071690839977236465, + 0.007096375004039146, + 0.0069016250054119155, + 0.006839541005319916, + 0.006801457988331094, + 0.006676999997580424, + 0.0063777090108487755, + 0.0064287499990314245, + 0.006675916985841468, + 0.006596042003366165, + 0.006442458994570188, + 0.007648041995707899, + 0.006544250005390495 + ] + }, + { + "forward": 0.19323408299533185, + "prefill": 0.19070195801032241, + "decode_steps": [ + 0.008546167009626515, + 0.007349166990024969, + 0.006757458992069587, + 0.00662858400028199, + 0.006532917002914473, + 0.006449749998864718, + 0.006590290999156423, + 0.0063409590075025335, + 0.006250708000152372, + 0.006353791992296465, + 0.0064991660037776455, + 0.007022166988463141, + 0.006854999999632128, + 0.00643845900776796, + 0.006319375010207295, + 0.006334124991553836, + 0.006356249999953434, + 0.0064573329873383045, + 0.006196707996423356, + 0.006432499998481944, + 0.006410000001778826, + 0.006120499994722195, + 0.0064079580042744055, + 0.006248624995350838, + 0.006242625007871538, + 0.006327333991066553, + 0.006767457991372794, + 0.006686041000648402, + 0.006430666995584033, + 0.006751458000508137, + 0.006800041999667883, + 0.006741541001247242 + ] + }, + { + "forward": 0.19412066700169817, + "prefill": 0.19392954200156964, + "decode_steps": [ + 0.019392834001337178, + 0.014827291990513913, + 0.011106624995591119, + 0.010737290998804383, + 0.008845499993185513, + 0.008097666999674402, + 0.007730749988695607, + 0.0076109170040581375, + 0.007494040997698903, + 0.0075425830000313, + 0.007391457998892292, + 0.007415707994368859, + 0.007719707995420322, + 0.007273999988683499, + 0.007218250000732951, + 0.007525916007580236, + 0.0070532499958062544, + 0.007259542006067932, + 0.006929374998435378, + 0.006950333001441322, + 0.006898499996168539, + 0.007007499996689148, + 0.006972499992116354, + 0.006852792008430697, + 0.006971416994929314, + 0.007791124997311272, + 0.006973083000048064, + 0.006794791988795623, + 0.0067917919950559735, + 0.006869208998978138, + 0.006520124996313825, + 0.007005666004261002 + ] + }, + { + "forward": 0.195237416002783, + "prefill": 0.19946604101278353, + "decode_steps": [ + 0.011081958000431769, + 0.008534374996088445, + 0.007746749994112179, + 0.007268707995535806, + 0.007099207999999635, + 0.007463415997335687, + 0.007669458005693741, + 0.018022666990873404, + 0.008995999989565462, + 0.007416584005113691, + 0.007143750000977889, + 0.006830875005107373, + 0.006615499994950369, + 0.006614374986384064, + 0.006334666002658196, + 0.006469250001828186, + 0.006429874993045814, + 0.006564958006492816, + 0.006807749989093281, + 0.00752837500476744, + 0.006794750006520189, + 0.006753708992619067, + 0.006575374995009042, + 0.006622707995120436, + 0.006497000009403564, + 0.006430000008549541, + 0.006423958999221213, + 0.00649266698746942, + 0.00655100001313258, + 0.0065925420058192685, + 0.0065077920007752255, + 0.006479124989709817 + ] + }, + { + "forward": 0.1926087500032736, + "prefill": 0.1926797920023091, + "decode_steps": [ + 0.011853625008370727, + 0.012076625003828667, + 0.011779249995015562, + 0.010017666005296633, + 0.008545541990315542, + 0.008051833996432833, + 0.007261333012138493, + 0.007278332996065728, + 0.006805915996665135, + 0.006645082990871742, + 0.006778582988772541, + 0.006574582992470823, + 0.006364499990013428, + 0.006346583002596162, + 0.006291874989983626, + 0.00652141599857714, + 0.006833541992818937, + 0.0068828340008622035, + 0.006451416003983468, + 0.00656275000073947, + 0.006341917003737763, + 0.0064279999933205545, + 0.006411875001504086, + 0.006358917002216913, + 0.006411416994524188, + 0.006364917004248127, + 0.006436458992538974, + 0.0063609159988118336, + 0.006333249999443069, + 0.006356166995828971, + 0.006415957992430776, + 0.006907333998242393 + ] + }, + { + "forward": 0.19614820799324661, + "prefill": 0.1949307499889983, + "decode_steps": [ + 0.019043916996452026, + 0.015054208008223213, + 0.01042041699110996, + 0.009422082992387004, + 0.008902875008061528, + 0.01017158300965093, + 0.008353540993994102, + 0.008228375008911826, + 0.007295584000530653, + 0.00763062501209788, + 0.007430915997247212, + 0.00737904199922923, + 0.007254875003127381, + 0.007291166999493726, + 0.007169625008828007, + 0.007285707994014956, + 0.007116541994037107, + 0.007336374997976236, + 0.007088166996254586, + 0.007287041997187771, + 0.008371999996597879, + 0.007154207996791229, + 0.007012458998360671, + 0.007084124998073094, + 0.007062125005177222, + 0.00692462500592228, + 0.00690612499602139, + 0.007062749995384365, + 0.006771917003788985, + 0.006425582992960699, + 0.006418459000997245, + 0.006339290994219482 + ] + }, + { + "forward": 0.19570974999805912, + "prefill": 0.19384949999221135, + "decode_steps": [ + 0.021365583001170307, + 0.014664333997643553, + 0.01072341600956861, + 0.009448458004044369, + 0.009466832998441532, + 0.009066500002518296, + 0.008040250002522953, + 0.007762082997942343, + 0.0079552910028724, + 0.007358916991506703, + 0.007915916998172179, + 0.007387457997538149, + 0.007326166989514604, + 0.007097791996784508, + 0.007055583002511412, + 0.0070675419992767274, + 0.007547500004875474, + 0.006996249998337589, + 0.007014834001893178, + 0.007666417004656978, + 0.006929042006959207, + 0.006857040993054397, + 0.006838125002104789, + 0.00682724999205675, + 0.006778666996979155, + 0.006900374995893799, + 0.007331624990911223, + 0.006839499998022802, + 0.006794916000217199, + 0.00683208298869431, + 0.0068283329892437905, + 0.0066491249890532345 + ] + }, + { + "forward": 0.19390325000858866, + "prefill": 0.19467733301280532, + "decode_steps": [ + 0.014148166999802925, + 0.00930016599886585, + 0.009405999997397885, + 0.008005167008377612, + 0.007992041995748878, + 0.007431582998833619, + 0.0075781670020660385, + 0.007895709000877105, + 0.007470000011380762, + 0.007263292005518451, + 0.0070308339927578345, + 0.0070241249923128635, + 0.007012208006926812, + 0.0069109999894862995, + 0.006984750012634322, + 0.007628334002220072, + 0.006890083008329384, + 0.007046042010188103, + 0.007007042004261166, + 0.0068656250077765435, + 0.006806000004871748, + 0.006675958997220732, + 0.006850916004623286, + 0.0067682910012081265, + 0.006863625007099472, + 0.006829750011092983, + 0.006790749990614131, + 0.0068536670005414635, + 0.0068145830009598285, + 0.006480666997958906, + 0.006303874994046055, + 0.007431958001689054 + ] + }, + { + "forward": 0.1947775000007823, + "prefill": 0.19408170899259858, + "decode_steps": [ + 0.01977016699675005, + 0.013138416994479485, + 0.009650207997765392, + 0.008905916998628527, + 0.008649417010019533, + 0.008341375010786578, + 0.007264957996085286, + 0.007260041995323263, + 0.007078707989421673, + 0.0070612499985145405, + 0.007078374997945502, + 0.007028082996839657, + 0.00697733300330583, + 0.006996625001193024, + 0.006926833011675626, + 0.0069570409978041425, + 0.0069713330012746155, + 0.006817708999733441, + 0.006937708996701986, + 0.006930916992132552, + 0.0072621659928699955, + 0.007312665999052115, + 0.006822334005846642, + 0.0072504169947933406, + 0.00679583300370723, + 0.006807291996665299, + 0.006714749993989244, + 0.006809374986914918, + 0.0067666249960893765, + 0.006718457996612415, + 0.00679100000706967, + 0.00683716700586956 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 195.00745800178265, + "p95": 197.48850450778264, + "min": 192.6087500032736, + "max": 197.58041700697504, + "iqr": 2.080989237583708, + "mean": 195.0696208019508, + "cv": 0.008543424387496074 + }, + "prefill_ms": { + "count": 10, + "median": 193.98697950236965, + "p95": 197.42516005208017, + "min": 190.70195801032241, + "max": 199.46604101278353, + "iqr": 1.5562080079689622, + "mean": 193.97804170148447, + "cv": 0.012213245822223944 + }, + "decode_ms_per_step": { + "count": 320, + "median": 7.023145990388002, + "p95": 11.78296874568332, + "min": 6.120499994722195, + "max": 21.365583001170307, + "iqr": 0.8561247595935129, + "mean": 7.729877724113976, + "cv": 0.298592198496469 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 505.01358293894197, + "p95": 581.3682096472988, + "min": 482.0938753574194, + "max": 607.6570697579428, + "iqr": 47.176317559951656, + "mean": 519.9193916101686, + "cv": 0.07498980367333624 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 286.83213551496675, + "p95": 310.70622560766, + "min": 278.53160714342476, + "max": 318.92594991241884, + "iqr": 13.479106234495532, + "mean": 290.4777162111344, + "cv": 0.04239774186817564 + }, + "memory": { + "peak_extra_bytes": 843071488, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "auto", + "batch": 8, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.0073631249979371205 + }, + { + "forward": 0.0060292910056887195 + }, + { + "forward": 0.006059416991774924 + }, + { + "forward": 0.006018000000040047 + }, + { + "forward": 0.006161375000374392 + }, + { + "forward": 0.006256917011342011 + }, + { + "forward": 0.006125250001787208 + }, + { + "forward": 0.006122042002971284 + }, + { + "forward": 0.006142250000266358 + }, + { + "forward": 0.0059996250056428835 + } + ], + "forward_ms": { + "count": 10, + "median": 6.123646002379246, + "p95": 6.8653314039693205, + "min": 5.9996250056428835, + "max": 7.3631249979371205, + "iqr": 0.1197712481371127, + "mean": 6.227729201782495, + "cv": 0.06527041977414114 + }, + "memory": { + "peak_extra_bytes": 11797504, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "sdpa", + "batch": 8, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.008174957998562604, + "prefill": 0.007117291999747977, + "decode_steps": [ + 0.0024938339920481667, + 0.002462624994223006, + 0.0024126249918481335, + 0.002453833003528416, + 0.002542624992202036, + 0.0022385420015780255, + 0.0021303329995134845, + 0.0021670829883078113, + 0.002204749995144084, + 0.002090166002744809, + 0.0023097079974832013, + 0.002244167000753805, + 0.0021723750105593354, + 0.0020834160095546395, + 0.0021504160104086623, + 0.002059207996353507, + 0.002136542010703124, + 0.0021710830042138696, + 0.0020736250007757917, + 0.002219374990090728, + 0.0020585839956765994, + 0.002126000006683171, + 0.0020610000065062195, + 0.002200208997237496, + 0.0022847500076750293, + 0.0021740829979535192, + 0.0022344580065691844, + 0.002138833995559253, + 0.002244083007099107, + 0.0022788750065956265, + 0.0022080420021666214, + 0.002221250004367903 + ] + }, + { + "forward": 0.00539508399378974, + "prefill": 0.005431375000625849, + "decode_steps": [ + 0.0022838750010123476, + 0.002276875005918555, + 0.002143749996321276, + 0.0022052910062484443, + 0.0020933330088155344, + 0.002232124999864027, + 0.002188749989727512, + 0.002154292000341229, + 0.0021129579981788993, + 0.0020662910101236776, + 0.002215957996668294, + 0.0021078329882584512, + 0.0021754159970441833, + 0.002293875004397705, + 0.002169250001315959, + 0.0021057499980088323, + 0.0021266660041874275, + 0.002276208993862383, + 0.0021172090055188164, + 0.002017125007114373, + 0.0026051250024465844, + 0.002143915990018286, + 0.005995750005240552, + 0.004753041997901164, + 0.004789333004737273, + 0.0030797920044278726, + 0.0035812919959425926, + 0.002311042000656016, + 0.00222491699969396, + 0.0022299580014077947, + 0.0022505420056404546, + 0.002565583010436967 + ] + }, + { + "forward": 0.005337041991879232, + "prefill": 0.005315667003742419, + "decode_steps": [ + 0.002282041998114437, + 0.0022107080003479496, + 0.0022307500039460137, + 0.0021952499955659732, + 0.0020654169929912314, + 0.0021468750055646524, + 0.0022207919973880053, + 0.002190291997976601, + 0.002207291006925516, + 0.002097832999425009, + 0.002173667002352886, + 0.0020577499963110313, + 0.002425416008918546, + 0.0020804999949177727, + 0.0021703329985029995, + 0.0021095420088386163, + 0.0021547500073211268, + 0.002419166994513944, + 0.002156708011170849, + 0.0022002920013619587, + 0.002244167000753805, + 0.002176750000216998, + 0.0023722080077277496, + 0.002177083006245084, + 0.0021290409931680188, + 0.0020351659914013, + 0.0022761659929528832, + 0.00213558399991598, + 0.002183333010179922, + 0.0023572909994982183, + 0.0023159999982453883, + 0.0021791249891975895 + ] + }, + { + "forward": 0.00534012500429526, + "prefill": 0.005409709003288299, + "decode_steps": [ + 0.0022667500015813857, + 0.002149665990145877, + 0.002034874996752478, + 0.0021232909930404276, + 0.0021482079901034012, + 0.0020429589931154624, + 0.00213350000558421, + 0.0020358750043669716, + 0.002272500001708977, + 0.002154791000066325, + 0.0021188749960856512, + 0.0022334169916575775, + 0.002135208997060545, + 0.002099249992170371, + 0.0021146250073798, + 0.002211458995589055, + 0.0022044999932404608, + 0.002159583003958687, + 0.0021614169963868335, + 0.0021646250097546726, + 0.0023107080050976947, + 0.0020655830012401566, + 0.0021726659906562418, + 0.002150667001842521, + 0.002122416000929661, + 0.0022962499933782965, + 0.00211329098965507, + 0.0021616250014631078, + 0.002282624991494231, + 0.0022177920036483556, + 0.0023179999989224598, + 0.00206995899497997 + ] + }, + { + "forward": 0.005404750001616776, + "prefill": 0.005240624988800846, + "decode_steps": [ + 0.002318457991350442, + 0.002030249990639277, + 0.0021117090072948486, + 0.0021029170020483434, + 0.002181040996219963, + 0.0021219589980319142, + 0.002018166007474065, + 0.002125500002875924, + 0.002160500007448718, + 0.0020013330067740753, + 0.0025437920121476054, + 0.002220750000560656, + 0.0022516250028274953, + 0.00245349999750033, + 0.0022141250083222985, + 0.0021895829995628446, + 0.00224491699191276, + 0.0022634580091107637, + 0.002290040996740572, + 0.002146874991012737, + 0.002260875000501983, + 0.0020935420034220442, + 0.0023280419991351664, + 0.0021269169956212863, + 0.0021669169946108013, + 0.00223333400208503, + 0.0020859579963143915, + 0.0021262910013319924, + 0.0021335420024115592, + 0.0021984999912092462, + 0.0021702920057578012, + 0.00212416700378526 + ] + }, + { + "forward": 0.0054216249991441146, + "prefill": 0.00538050000614021, + "decode_steps": [ + 0.0024489160132361576, + 0.002121665995218791, + 0.0022352910018526018, + 0.002316917001735419, + 0.002160125004593283, + 0.0024291669978993014, + 0.0021317079954314977, + 0.0020577500108629465, + 0.0022097500041127205, + 0.0021084579930175096, + 0.0022507499961648136, + 0.0022690000041620806, + 0.0021657920005964115, + 0.0021012090001022443, + 0.0022213340125745162, + 0.0021915419929428026, + 0.002190832994529046, + 0.0023245830088853836, + 0.0022598329960601404, + 0.0022669159952783957, + 0.0021855410013813525, + 0.0021794999920530245, + 0.0022044590004952624, + 0.0021406250016298145, + 0.0021899580024182796, + 0.002229874997283332, + 0.0022370420047082007, + 0.00221891600813251, + 0.0022039589966880158, + 0.0021522500028368086, + 0.0020926249999320135, + 0.002055792007013224 + ] + }, + { + "forward": 0.0053664169972762465, + "prefill": 0.005280332989059389, + "decode_steps": [ + 0.0021281249937601388, + 0.0021990829991409555, + 0.0020085830037714913, + 0.002160290998290293, + 0.002162625009077601, + 0.002042750013060868, + 0.002105791005305946, + 0.0021175840083742514, + 0.0021343330008676276, + 0.00201162499433849, + 0.0022897500020917505, + 0.0020027499995194376, + 0.00217500000144355, + 0.002189541992265731, + 0.0021925409964751452, + 0.0021999169985065237, + 0.002023749999352731, + 0.002422999998088926, + 0.0022234579955693334, + 0.0020797079923795536, + 0.0022173750039655715, + 0.0023080410028342158, + 0.002319207997061312, + 0.0024101669987430796, + 0.0021463749872054905, + 0.002226291995611973, + 0.002078417004668154, + 0.0022922080097487196, + 0.002263458998640999, + 0.002126624996890314, + 0.0021811670012539253, + 0.0022540840000147 + ] + }, + { + "forward": 0.005313707995810546, + "prefill": 0.005419875000370666, + "decode_steps": [ + 0.002310458992724307, + 0.0022798330028308555, + 0.002082958002574742, + 0.002166083999327384, + 0.0022674579959129915, + 0.0021888749906793237, + 0.0021257909975247458, + 0.002123792000929825, + 0.002391041998635046, + 0.0021089579968247563, + 0.0021560840104939416, + 0.002108959000906907, + 0.0022013749985489994, + 0.00209687500318978, + 0.0022262920101638883, + 0.0022463749919552356, + 0.0021752500033471733, + 0.0021535409905482084, + 0.0022674999927403405, + 0.0021585419890470803, + 0.002237166991108097, + 0.002182625001296401, + 0.00221291599154938, + 0.002263749993289821, + 0.0020633329986594617, + 0.002227541001047939, + 0.0021684999956050888, + 0.002247750002425164, + 0.002270833996590227, + 0.0021649999980581924, + 0.0021821669943165034, + 0.0022081659990362823 + ] + }, + { + "forward": 0.005333458000677638, + "prefill": 0.0053376249998109415, + "decode_steps": [ + 0.0021432920038932934, + 0.002509333993657492, + 0.0022252499911701307, + 0.002627916997880675, + 0.002207792000262998, + 0.0020121249981457368, + 0.0022312920045806095, + 0.0022147499985294417, + 0.002217458008090034, + 0.0021824160066898912, + 0.0025673339987406507, + 0.0021117079886607826, + 0.0022152500023366883, + 0.0021233340085018426, + 0.0022004579950589687, + 0.0021459999989019707, + 0.0022911249980097637, + 0.002140750002581626, + 0.002114292001351714, + 0.0021502089948626235, + 0.002146833998267539, + 0.0021827500022482127, + 0.0022615000052610412, + 0.002245707990368828, + 0.0022179579973453656, + 0.002200959002948366, + 0.0022617079957854003, + 0.002252500009490177, + 0.0021973330003675073, + 0.0021184580109547824, + 0.0022188750008353963, + 0.0021244579984340817 + ] + }, + { + "forward": 0.005375166001613252, + "prefill": 0.0053431669948622584, + "decode_steps": [ + 0.002170583000406623, + 0.0021804580028401688, + 0.001965874995221384, + 0.002173833010601811, + 0.0020076250075362623, + 0.0020709170057671145, + 0.002234583007520996, + 0.0020502910047071055, + 0.0022299999982351437, + 0.002103916005580686, + 0.002165875004720874, + 0.002070832997560501, + 0.002231917009339668, + 0.0021166249935049564, + 0.0020692090038210154, + 0.0021526250056922436, + 0.002142042008927092, + 0.0021485829929588363, + 0.0028226249996805564, + 0.0022888329986017197, + 0.002133250003680587, + 0.002160584001103416, + 0.0021774170018034056, + 0.0021400840050773695, + 0.002382417005719617, + 0.0023910829913802445, + 0.002175249988795258, + 0.0024585840001236647, + 0.0020840420038439333, + 0.0022835419949842617, + 0.0022019159951014444, + 0.002198249989305623 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 5.370791499444749, + "p95": 6.935958148824281, + "min": 5.313707995810546, + "max": 8.174957998562604, + "iqr": 0.06452075467677787, + "mean": 5.646233298466541, + "cv": 0.15748193207031702 + }, + "prefill_ms": { + "count": 10, + "median": 5.361833500501234, + "p95": 6.358629350143017, + "min": 5.240624988800846, + "max": 7.117291999747977, + "iqr": 0.09617699834052473, + "mean": 5.527616798644885, + "cv": 0.10166133260437768 + }, + "decode_ms_per_step": { + "count": 320, + "median": 2.1822915005031973, + "p95": 2.4540705533581786, + "min": 1.9658749952213839, + "max": 5.995750005240552, + "iqr": 0.11921898840228096, + "mean": 2.2323719875657844, + "cv": 0.14624250794539687 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 3640.4231075219723, + "p95": 3686.251258766078, + "min": 3126.0256114741246, + "max": 3696.9133567484087, + "iqr": 43.64101470213063, + "mean": 3591.4021482553026, + "cv": 0.04622876130741747 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 3382.247712499757, + "p95": 3422.5004351794614, + "min": 2931.594292198349, + "max": 3429.0307072951014, + "iqr": 41.8930686868207, + "mean": 3332.7248986814534, + "cv": 0.04413902003723666 + }, + "memory": { + "peak_extra_bytes": 11797504, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "reference", + "batch": 8, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.010726791006163694, + "prefill": 0.00735191599233076, + "decode_steps": [ + 0.005608834006125107, + 0.004956666991347447, + 0.004268625008990057, + 0.004747917002532631, + 0.0046743340062676, + 0.004581250002956949, + 0.0045187089999672025, + 0.004493499989621341, + 0.004545249990769662, + 0.00435375000233762, + 0.0044355830032145604, + 0.00454216699290555, + 0.004565249997540377, + 0.004859249995206483, + 0.004364916996564716, + 0.004537957996944897, + 0.005002167003112845, + 0.005011958011891693, + 0.00484195799799636, + 0.004790999999386258, + 0.0049157500034198165, + 0.004766500002006069, + 0.004716459006885998, + 0.004539708999800496, + 0.0047416659945156425, + 0.0046971669944468886, + 0.0048178750002989545, + 0.004650207993108779, + 0.004378750003525056, + 0.004902041997411288, + 0.004935459000989795, + 0.005180833002668805 + ] + }, + { + "forward": 0.01273600000422448, + "prefill": 0.007159375003539026, + "decode_steps": [ + 0.0052564160141628236, + 0.004746416001580656, + 0.004965000000083819, + 0.004561790992738679, + 0.004956666991347447, + 0.004714958005934022, + 0.004643708001822233, + 0.004707915999460965, + 0.004610290998243727, + 0.004636541998479515, + 0.004996249990654178, + 0.00483429100131616, + 0.004474749992368743, + 0.004643667009077035, + 0.004651291994377971, + 0.004443915997399017, + 0.004817624998395331, + 0.004469082996365614, + 0.004972082999302074, + 0.004773958993609995, + 0.004559584005619399, + 0.005373125008190982, + 0.004920040999422781, + 0.0047188329917844385, + 0.005111459002364427, + 0.00532233300327789, + 0.004968082997947931, + 0.004851749996305443, + 0.004675124990171753, + 0.004596500002662651, + 0.0041762499895412475, + 0.004707000000053085 + ] + }, + { + "forward": 0.009716374988784082, + "prefill": 0.00712220800051, + "decode_steps": [ + 0.004765958001371473, + 0.0052151250129099935, + 0.00433879200136289, + 0.0044970829912927, + 0.004838499997276813, + 0.00470458299969323, + 0.004780207993462682, + 0.004250042009516619, + 0.0046576659951824695, + 0.004513791005592793, + 0.004550541998469271, + 0.004688375003752299, + 0.004566542003885843, + 0.004667291999794543, + 0.0046963749919086695, + 0.004578249994665384, + 0.004731291992357001, + 0.004563957991194911, + 0.004684457991970703, + 0.004450874999747612, + 0.005647375000989996, + 0.005237165998551063, + 0.004799292000825517, + 0.0045211669930722564, + 0.004888791008852422, + 0.004977958000381477, + 0.004743333993246779, + 0.004694958988693543, + 0.004688790999352932, + 0.0046735419891774654, + 0.00473620901175309, + 0.004548832992441021 + ] + }, + { + "forward": 0.010616125000524335, + "prefill": 0.007051958004012704, + "decode_steps": [ + 0.004939666003338061, + 0.0047881670034257695, + 0.00477895799849648, + 0.004642208004952408, + 0.004506917000981048, + 0.004673250004998408, + 0.004645625012926757, + 0.004613833007169887, + 0.004548917000647634, + 0.0044760000018868595, + 0.004692792004789226, + 0.004464291996555403, + 0.0045067910104990005, + 0.004248124998412095, + 0.004766416997881606, + 0.005404249997809529, + 0.005048250008258037, + 0.0049009170033968985, + 0.004452124994713813, + 0.004856416999245994, + 0.004698999997344799, + 0.004609582989360206, + 0.004688125001848675, + 0.004748750012367964, + 0.004627917005564086, + 0.004785666998941451, + 0.004474208006286062, + 0.004723709003883414, + 0.00447225000243634, + 0.004634791999706067, + 0.004647875000955537, + 0.004540666996035725 + ] + }, + { + "forward": 0.010453041002620012, + "prefill": 0.007215584002551623, + "decode_steps": [ + 0.004778749993420206, + 0.004851792007684708, + 0.004525083000771701, + 0.004792457999428734, + 0.004465833000722341, + 0.004606208007317036, + 0.00476458300545346, + 0.004986500003724359, + 0.004740832999232225, + 0.004777416004799306, + 0.004485582991037518, + 0.004627707996405661, + 0.004435125010786578, + 0.004645459004677832, + 0.0046950000105425715, + 0.004640042010578327, + 0.004347667010733858, + 0.004389458001242019, + 0.004745750004076399, + 0.004659832993638702, + 0.004560041008517146, + 0.004585375005262904, + 0.00444104200869333, + 0.004683999999542721, + 0.0045114590029697865, + 0.004698291988461278, + 0.0047642910067224875, + 0.004429375010658987, + 0.004613333003362641, + 0.004612082993844524, + 0.00453691701113712, + 0.004534874999080785 + ] + }, + { + "forward": 0.00950449999072589, + "prefill": 0.007214625002234243, + "decode_steps": [ + 0.00494870801048819, + 0.004616042002453469, + 0.0045935840025777, + 0.004552667000098154, + 0.004644041997380555, + 0.004578457999741659, + 0.004435374998138286, + 0.004555666004307568, + 0.00453166599618271, + 0.004612292003002949, + 0.004627374990377575, + 0.004564375005429611, + 0.0044979170052101836, + 0.004426458006491885, + 0.0048125419998541474, + 0.004482250005821697, + 0.004854915998294018, + 0.0048643749905750155, + 0.0046712079929420725, + 0.004793125001015142, + 0.004423000005772337, + 0.004895584002952091, + 0.004937749996315688, + 0.004771124993567355, + 0.004991541994968429, + 0.005091999992146157, + 0.005212249991018325, + 0.0048022499977378175, + 0.005484832989168353, + 0.005043124998337589, + 0.005226416993536986, + 0.004888333001872525 + ] + }, + { + "forward": 0.010611707999487408, + "prefill": 0.007139041001209989, + "decode_steps": [ + 0.005047083002864383, + 0.004858624990447424, + 0.004720625001937151, + 0.004605499998433515, + 0.004511374994763173, + 0.00461841699143406, + 0.004556292013148777, + 0.004522542003542185, + 0.00461558299139142, + 0.004366790992207825, + 0.005161624998436309, + 0.004712499998277053, + 0.004529915997409262, + 0.005166165996342897, + 0.0047590000031050295, + 0.004750083986436948, + 0.004629584000213072, + 0.004716790994280018, + 0.0049072090041590855, + 0.004740832999232225, + 0.004625792003935203, + 0.004749624989926815, + 0.00469170899305027, + 0.004615124998963438, + 0.004699500001152046, + 0.004549082994344644, + 0.004719874996226281, + 0.0045337079936871305, + 0.004606375005096197, + 0.00464158300019335, + 0.004565124996588565, + 0.0045956249959999695 + ] + }, + { + "forward": 0.009972833999199793, + "prefill": 0.0073559169977670535, + "decode_steps": [ + 0.005212375006522052, + 0.005262708000373095, + 0.0047557500074617565, + 0.00441670800501015, + 0.004511374994763173, + 0.0049374579975847155, + 0.005013542002416216, + 0.004833166996832006, + 0.0051875420031137764, + 0.004808792000403628, + 0.004846915995585732, + 0.0051877919904654846, + 0.004579583008307964, + 0.004754958994453773, + 0.0051120830030413345, + 0.004912375006824732, + 0.004279166008927859, + 0.004653291995055042, + 0.004678209006669931, + 0.004707917003543116, + 0.004771082996740006, + 0.004678249999415129, + 0.004496875000768341, + 0.0047640419943491, + 0.004630958006600849, + 0.005145458009792492, + 0.004233792002196424, + 0.004944041997077875, + 0.004848832992138341, + 0.004706249994342215, + 0.004611709009623155, + 0.004268541990313679 + ] + }, + { + "forward": 0.00967066599696409, + "prefill": 0.007097833004081622, + "decode_steps": [ + 0.004781416995683685, + 0.0046542500058421865, + 0.004621250001946464, + 0.00469541699567344, + 0.004659709011320956, + 0.004610000003594905, + 0.004584958995110355, + 0.0045506669994210824, + 0.00455708299705293, + 0.004537417000392452, + 0.004472707994864322, + 0.004740375006804243, + 0.00475729200115893, + 0.005076208006357774, + 0.004837083004531451, + 0.004837917003897019, + 0.004779042006703094, + 0.004635125005734153, + 0.004457957998965867, + 0.00492866599233821, + 0.004802916999324225, + 0.004594415993778966, + 0.004602415996487252, + 0.004634958997485228, + 0.004796666995389387, + 0.004563958995277062, + 0.0046904159971745685, + 0.004688375003752299, + 0.004540458001429215, + 0.00461804200313054, + 0.004612915989127941, + 0.00422420899849385 + ] + }, + { + "forward": 0.010856792010599747, + "prefill": 0.007231167008285411, + "decode_steps": [ + 0.004927457994199358, + 0.00493862500297837, + 0.0047127080033533275, + 0.00463241599209141, + 0.00465504199382849, + 0.004538124994724058, + 0.005663374991854653, + 0.004440667005837895, + 0.004509042002609931, + 0.00457858300069347, + 0.0045371669984888285, + 0.005098833993542939, + 0.004519374997471459, + 0.0045919580006739125, + 0.004892042008577846, + 0.004582957990351133, + 0.004747208004118875, + 0.004679250007029623, + 0.004769000006490387, + 0.004724333994090557, + 0.004658541001845151, + 0.004593999998178333, + 0.004607124996255152, + 0.004706458988948725, + 0.004689915993367322, + 0.004355124998255633, + 0.0053052500006742775, + 0.005000584002118558, + 0.004825875003007241, + 0.0052324999996926636, + 0.004661540995584801, + 0.004885542002739385 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 10.53237450105371, + "p95": 11.890356407093348, + "min": 9.50449999072589, + "max": 12.73600000422448, + "iqr": 0.9186347633658443, + "mean": 10.486483199929353, + "cv": 0.08863693162350032 + }, + "prefill_ms": { + "count": 10, + "median": 7.187000002886634, + "p95": 7.354116545320721, + "min": 7.051958004012704, + "max": 7.3559169977670535, + "iqr": 0.10085500616696663, + "mean": 7.193962401652243, + "cv": 0.01407129950210364 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.681625003286172, + "p95": 5.189014890493127, + "min": 4.1762499895412475, + "max": 5.663374991854653, + "iqr": 0.2563340021879412, + "mean": 4.715354034215125, + "cv": 0.05040390378868294 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1696.2596678625332, + "p95": 1724.1939607296092, + "min": 1674.771163848907, + "max": 1730.5229161416069, + "iqr": 29.86265689885363, + "mean": 1696.7772579511056, + "cv": 0.011234538507595877 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1618.6423642253153, + "p95": 1644.8391905474195, + "min": 1598.9261210592656, + "max": 1650.0400627137428, + "iqr": 28.710394547011674, + "mean": 1619.5550073478944, + "cv": 0.011006630262878724 + }, + "memory": { + "peak_extra_bytes": 11797504, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "auto", + "batch": 8, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.011790749995270744 + }, + { + "forward": 0.011872791990754195 + }, + { + "forward": 0.011804458001279272 + }, + { + "forward": 0.011752958002034575 + }, + { + "forward": 0.012202583995531313 + }, + { + "forward": 0.012000540998997167 + }, + { + "forward": 0.011691874999087304 + }, + { + "forward": 0.01184495899360627 + }, + { + "forward": 0.0153427500044927 + }, + { + "forward": 0.01228679199994076 + } + ], + "forward_ms": { + "count": 10, + "median": 11.858875492180232, + "p95": 13.967568902444324, + "min": 11.691874999087304, + "max": 15.3427500044927, + "iqr": 0.3578962496249005, + "mean": 12.25904589809943, + "cv": 0.08978909848515836 + }, + "memory": { + "peak_extra_bytes": 23595008, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "sdpa", + "batch": 8, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.010173624992603436, + "prefill": 0.01012545901176054, + "decode_steps": [ + 0.007096750006894581, + 0.00506741700519342, + 0.00525937500060536, + 0.005808041998534463, + 0.00273912500415463, + 0.00267966600949876, + 0.002589291994809173, + 0.0024813750060275197, + 0.002475375003996305, + 0.0025335410027764738, + 0.002395500006969087, + 0.0025554590101819485, + 0.0024839169927872717, + 0.0025330000062240288, + 0.0024882500001695007, + 0.0025129999994533136, + 0.002483541000401601, + 0.002556040999479592, + 0.0025137080083368346, + 0.0024850000045262277, + 0.002472666004905477, + 0.002493499996489845, + 0.0024479169951518998, + 0.0025332910008728504, + 0.002512166989617981, + 0.0025312079960713163, + 0.0024082090094452724, + 0.0025470830005360767, + 0.002493292005965486, + 0.0023619580024387687, + 0.00253687500662636, + 0.002545208000810817 + ] + }, + { + "forward": 0.010188666987232864, + "prefill": 0.010230458996375091, + "decode_steps": [ + 0.0025885409995680675, + 0.002562584006227553, + 0.002461041003698483, + 0.002477415997418575, + 0.0026011669979197904, + 0.0025360839936183766, + 0.0023534590000053868, + 0.0028105829987907782, + 0.0024802499974612147, + 0.002382415987085551, + 0.0028091659914935008, + 0.0024639999901410192, + 0.0024067090125754476, + 0.0025432920083403587, + 0.0024882090074243024, + 0.0024623329954920337, + 0.0025522089999867603, + 0.0024370419996557757, + 0.002440334006678313, + 0.0024941669980762526, + 0.0024611669941805303, + 0.002556291001383215, + 0.002451791995554231, + 0.0024025000020628795, + 0.0024882500001695007, + 0.0024660829949425533, + 0.002517708999221213, + 0.002468291000695899, + 0.0025061250053113326, + 0.0025217919901479036, + 0.002539041990530677, + 0.0026042079989565536 + ] + }, + { + "forward": 0.01027966701076366, + "prefill": 0.010307457996532321, + "decode_steps": [ + 0.0026267089997418225, + 0.003073582993238233, + 0.0025440419994993135, + 0.002478791997418739, + 0.00237558399385307, + 0.002421625002170913, + 0.0025210000021615997, + 0.0024612909910501912, + 0.0024671249993843958, + 0.002439500007312745, + 0.0025507079990347847, + 0.002492583997081965, + 0.0025705000007292256, + 0.0023265420022653416, + 0.002635542012285441, + 0.0026517500082263723, + 0.0025022920017363504, + 0.0025036250008270144, + 0.002607959002489224, + 0.0024879169941414148, + 0.0024541250022593886, + 0.0024303329992108047, + 0.002498750007362105, + 0.0024585420032963157, + 0.0025114169984590262, + 0.00253800000064075, + 0.002576667000539601, + 0.0024882500001695007, + 0.0025760839926078916, + 0.002476499998010695, + 0.0025981250073527917, + 0.0025452910049352795 + ] + }, + { + "forward": 0.010187416002736427, + "prefill": 0.010314416998880915, + "decode_steps": [ + 0.0027129999944008887, + 0.002394958006334491, + 0.0027597919979598373, + 0.002418624993879348, + 0.0023161659919423983, + 0.0025459159951424226, + 0.0025706669985083863, + 0.0026531250041443855, + 0.0025606660055927932, + 0.002465333993313834, + 0.0024046250036917627, + 0.0023757089948048815, + 0.002508916993974708, + 0.002423207988613285, + 0.002738791998126544, + 0.00247349998971913, + 0.002526541007682681, + 0.0025192500033881515, + 0.002481625007931143, + 0.0024619590112706646, + 0.002482249998138286, + 0.002548541990108788, + 0.0024935840046964586, + 0.002478790993336588, + 0.0024813750060275197, + 0.002444500001729466, + 0.002555166996899061, + 0.00257666599645745, + 0.0025150419969577342, + 0.002550999997765757, + 0.0025813329993980005, + 0.002493624997441657 + ] + }, + { + "forward": 0.010271249993820675, + "prefill": 0.010249124999972992, + "decode_steps": [ + 0.0026643749879440293, + 0.002474166001775302, + 0.002500832997611724, + 0.0025337499973829836, + 0.002485709002939984, + 0.00243499998759944, + 0.0025180829979944974, + 0.0024555839918321, + 0.0026087919977726415, + 0.0023972079943632707, + 0.002395125004113652, + 0.0024821249971864745, + 0.0024413329956587404, + 0.0025283750001108274, + 0.0024638329923618585, + 0.0024601660115877166, + 0.002371166003285907, + 0.0027723749954020604, + 0.002384958992479369, + 0.0025516669993521646, + 0.0024620410113129765, + 0.002406415995210409, + 0.0029004169919062406, + 0.00248620800266508, + 0.0023896659986348823, + 0.0024861249985406175, + 0.0025316249957541004, + 0.002538083994295448, + 0.0024852909991750494, + 0.0026584580045891926, + 0.0025149589928332716, + 0.0026122079871129245 + ] + }, + { + "forward": 0.010241124997264706, + "prefill": 0.01024391601094976, + "decode_steps": [ + 0.0026912080065812916, + 0.0024965829943539575, + 0.0024663749936735258, + 0.0024547919892938808, + 0.002368874993408099, + 0.002636290999362245, + 0.00249070800782647, + 0.0024225830129580572, + 0.0024124159972416237, + 0.0025568750133970752, + 0.0025070830015465617, + 0.0025327920011477545, + 0.002482915995642543, + 0.002510582999093458, + 0.002453082997817546, + 0.00245816599635873, + 0.0025358330021845177, + 0.002442708006128669, + 0.002458999995724298, + 0.0025386670022271574, + 0.0024610419932287186, + 0.0024699159985175356, + 0.002537541004130617, + 0.002373208000790328, + 0.0024735409970162436, + 0.0024715410108910874, + 0.002814542007399723, + 0.0024638329923618585, + 0.002488332989742048, + 0.0024868750042514876, + 0.0024674580054124817, + 0.0025017080042744055 + ] + }, + { + "forward": 0.010182166995946318, + "prefill": 0.011174874991411343, + "decode_steps": [ + 0.002500709000742063, + 0.002515709013096057, + 0.0024604169884696603, + 0.0025756250106496736, + 0.002430207998258993, + 0.0024951249943114817, + 0.0025097079924307764, + 0.0024467089970130473, + 0.002398374999756925, + 0.002595707992441021, + 0.0024661249917699024, + 0.0024784580018604174, + 0.002573791003669612, + 0.0024857079988578334, + 0.0025344580062665045, + 0.0024243749940069392, + 0.002409875000012107, + 0.0025362500018673018, + 0.002484416007064283, + 0.0024348750011995435, + 0.0025206249993061647, + 0.00245274999178946, + 0.0024901249998947605, + 0.0025645840069046244, + 0.0024668749974807724, + 0.0023004159884294495, + 0.002500292001059279, + 0.0023923749977257103, + 0.002582959001301788, + 0.002440416006720625, + 0.0025785420002648607, + 0.0025606249982956797 + ] + }, + { + "forward": 0.010795082998811267, + "prefill": 0.010470374996657483, + "decode_steps": [ + 0.0026091670006280765, + 0.002472874999511987, + 0.0024766249989625067, + 0.0024584579950897023, + 0.002429291998851113, + 0.002402458994765766, + 0.0025245420110877603, + 0.002392290989519097, + 0.0023954580101417378, + 0.002623916996526532, + 0.0026506249996600673, + 0.0024783330009086058, + 0.002565709000919014, + 0.0024464999878546223, + 0.00253520900150761, + 0.002415791997918859, + 0.002529874996980652, + 0.0025749169872142375, + 0.002517874992918223, + 0.0025480000040261075, + 0.002581666994956322, + 0.0025432499969610944, + 0.0024672500003362074, + 0.002443000004859641, + 0.002470749997883104, + 0.002420667005935684, + 0.0024980830057756975, + 0.0024819169921102002, + 0.0025398329889867455, + 0.002490792001481168, + 0.0026257499994244426, + 0.0025344999885419384 + ] + }, + { + "forward": 0.01078362500993535, + "prefill": 0.01056154198886361, + "decode_steps": [ + 0.003168832990922965, + 0.0025577500055078417, + 0.002594417004729621, + 0.0024578750017099082, + 0.0024685410025995225, + 0.00245258399809245, + 0.002431916000205092, + 0.0025175829941872507, + 0.0024845830048434436, + 0.0024754170008236542, + 0.00247687500086613, + 0.002467582991812378, + 0.002469457991537638, + 0.002466042002197355, + 0.002518416993552819, + 0.0026045420090667903, + 0.0025015419960254803, + 0.0025378749996889383, + 0.0024687919940333813, + 0.0025156670017167926, + 0.002418417003354989, + 0.00242529199749697, + 0.0023757909948471934, + 0.0023939999955473468, + 0.0024072920059552416, + 0.002388125009019859, + 0.002515542000764981, + 0.0024897080002119765, + 0.002435582995531149, + 0.0025354999961564317, + 0.0025487499951850623, + 0.002600082996650599 + ] + }, + { + "forward": 0.010205124999629334, + "prefill": 0.011309416004223749, + "decode_steps": [ + 0.0025638749939389527, + 0.0024316250055562705, + 0.002474042004905641, + 0.0024667079997016117, + 0.0025061670021386817, + 0.002389874993241392, + 0.0024737500061746687, + 0.0025158749922411516, + 0.002623542008223012, + 0.0025267090095439926, + 0.002529249992221594, + 0.002399875011178665, + 0.002532458005589433, + 0.002419707991066389, + 0.002472916996339336, + 0.0025034999998752028, + 0.0024916249967645854, + 0.0024733750033192337, + 0.0025589169963495806, + 0.0024883750011213124, + 0.002466624995577149, + 0.0024467089970130473, + 0.002466958001605235, + 0.0024651669955346733, + 0.0024579589953646064, + 0.0024651249987073243, + 0.0025446249928791076, + 0.0024745000118855387, + 0.00249237498792354, + 0.002452458007610403, + 0.002332874995772727, + 0.0025141670048469678 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 10.22312499844702, + "p95": 10.789926903817104, + "min": 10.173624992603436, + "max": 10.795082998811267, + "iqr": 0.08983400766737759, + "mean": 10.330774998874404, + "cv": 0.023670230012953822 + }, + "prefill_ms": { + "count": 10, + "median": 10.310937497706618, + "p95": 11.248872548458166, + "min": 10.12545901176054, + "max": 11.309416004223749, + "iqr": 0.2935319826065097, + "mean": 10.49870419956278, + "cv": 0.03924970426741755 + }, + "decode_ms_per_step": { + "count": 320, + "median": 2.4890415006666444, + "p95": 2.6802431093528867, + "min": 2.3004159884294495, + "max": 7.096750006894581, + "iqr": 0.08338599218404852, + "mean": 2.546962102633188, + "cv": 0.15303105329191238 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 3189.9425175720776, + "p95": 3219.9201345899787, + "min": 2734.4365995529115, + "max": 3223.2999868091433, + "iqr": 20.009267379031826, + "mean": 3147.9082685462213, + "cv": 0.046468857141575226 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 2821.5111398468907, + "p95": 2833.4964271772515, + "min": 2467.560089193129, + "max": 2839.0505315530113, + "iqr": 6.031054431226039, + "mean": 2787.115914616343, + "cv": 0.04038550460845792 + }, + "memory": { + "peak_extra_bytes": 23595008, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "reference", + "batch": 8, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.01976154200383462, + "prefill": 0.015735958004370332, + "decode_steps": [ + 0.005484541994519532, + 0.004978041994036175, + 0.004895667007076554, + 0.00492366700200364, + 0.004868625008384697, + 0.004952082992531359, + 0.004862749992753379, + 0.004783208001754247, + 0.00482120800006669, + 0.004751208995003253, + 0.004808458004845306, + 0.0048150000075111166, + 0.005031291992054321, + 0.004958292003720999, + 0.004771624997374602, + 0.005515958997420967, + 0.006211874992004596, + 0.005304042002535425, + 0.005342040996765718, + 0.005194083001697436, + 0.0048972920048981905, + 0.005197667007450946, + 0.0047849170077824965, + 0.005268708991934545, + 0.005236375000094995, + 0.005084165997686796, + 0.004667917004553601, + 0.005465124995680526, + 0.004790874998434447, + 0.004813916006241925, + 0.005170791991986334, + 0.005048000006354414 + ] + }, + { + "forward": 0.01884175000304822, + "prefill": 0.015566165995551273, + "decode_steps": [ + 0.006350709008984268, + 0.0051290840056026354, + 0.005345500001567416, + 0.0057339579943800345, + 0.005416542000602931, + 0.005146916999365203, + 0.004999416996724904, + 0.004684416999225505, + 0.004882207998889498, + 0.0049178749904967844, + 0.004841542002395727, + 0.004668458001106046, + 0.004857040999922901, + 0.004828916993574239, + 0.0048502499994356185, + 0.004942832994856872, + 0.004771958003402688, + 0.004628249997040257, + 0.004879374988377094, + 0.004844374998356216, + 0.004932374999043532, + 0.005426624993560836, + 0.005431416007922962, + 0.004737167007988319, + 0.005505083012394607, + 0.005250249989330769, + 0.005248707995633595, + 0.00481075000425335, + 0.005061499992734753, + 0.005267458007438108, + 0.004753542001708411, + 0.005231999995885417 + ] + }, + { + "forward": 0.01817041600588709, + "prefill": 0.015614584001014009, + "decode_steps": [ + 0.005567375003010966, + 0.005236625001998618, + 0.004762791999382898, + 0.0049441250012023374, + 0.004884542009676807, + 0.004850334007642232, + 0.005007290994399227, + 0.004879083004198037, + 0.004650542003219016, + 0.005053834000136703, + 0.004857332998653874, + 0.004989832988940179, + 0.004798916008439846, + 0.004824333009310067, + 0.00480254199646879, + 0.0046212920133257285, + 0.004884208014118485, + 0.0048460840043844655, + 0.004716875002486631, + 0.004722708006738685, + 0.00505337500362657, + 0.0048858749942155555, + 0.004873207988566719, + 0.005684832998667844, + 0.00533749999885913, + 0.005297458003042266, + 0.005222459003562108, + 0.004794207998202182, + 0.004579667001962662, + 0.004968042005202733, + 0.005001749988878146, + 0.004798041001777165 + ] + }, + { + "forward": 0.01896895900426898, + "prefill": 0.01572258300438989, + "decode_steps": [ + 0.005295916998875327, + 0.004631542004062794, + 0.004979999997885898, + 0.00523508399783168, + 0.005021917007979937, + 0.0049253339966526255, + 0.004900125000858679, + 0.004877290994045325, + 0.004892708995612338, + 0.00500570799340494, + 0.0047940840013325214, + 0.0048530829953961074, + 0.004983250008081086, + 0.004857624997384846, + 0.004963250001310371, + 0.004776457994012162, + 0.004825291005545296, + 0.004750708001665771, + 0.004785917000845075, + 0.0049679160001687706, + 0.004869334006798454, + 0.004488083999603987, + 0.005464041998493485, + 0.005628124999930151, + 0.005462749992148019, + 0.006007208998198621, + 0.005096917011542246, + 0.005761249994975515, + 0.0063224579935194924, + 0.005042584001785144, + 0.005023875011829659, + 0.005162250003195368 + ] + }, + { + "forward": 0.018694041995331645, + "prefill": 0.016233583010034636, + "decode_steps": [ + 0.005853124996065162, + 0.005010833003325388, + 0.004830790989217348, + 0.004897000006167218, + 0.004672166003729217, + 0.004834583000047132, + 0.0047516249906038865, + 0.005131625002832152, + 0.005791999996290542, + 0.005249958994681947, + 0.004856457991991192, + 0.004935167002258822, + 0.005560749996220693, + 0.005733250000048429, + 0.004732874993351288, + 0.005035417008912191, + 0.004586540992022492, + 0.0048833750042831525, + 0.004916999998386018, + 0.005110333004267886, + 0.004925041997921653, + 0.0050774160044966266, + 0.00536216699401848, + 0.0051785830000881106, + 0.004899249994195998, + 0.005067165999207646, + 0.004968749999534339, + 0.005091541999718174, + 0.004967249988112599, + 0.004918582999380305, + 0.004505624994635582, + 0.005091832994366996 + ] + }, + { + "forward": 0.018251500005135313, + "prefill": 0.015978249997715466, + "decode_steps": [ + 0.004945540989865549, + 0.004795875007403083, + 0.00488779200532008, + 0.004753500004881062, + 0.004787291996763088, + 0.0047344159975182265, + 0.004890541997156106, + 0.004799499991349876, + 0.004960250007570721, + 0.0048891249898588285, + 0.005341666997992434, + 0.004846499999985099, + 0.005018833995563909, + 0.004920332998153754, + 0.004843583010369912, + 0.004610999996657483, + 0.004915625002468005, + 0.0048893339990172535, + 0.004692125003202818, + 0.005119416004163213, + 0.00506854199920781, + 0.004950290996930562, + 0.004926082998281345, + 0.004939291990012862, + 0.005099667003378272, + 0.004776083005708642, + 0.004827875003684312, + 0.004872833000263199, + 0.004677624994656071, + 0.004928458991344087, + 0.004965915999491699, + 0.004888041992671788 + ] + }, + { + "forward": 0.019012832999578677, + "prefill": 0.0156156250013737, + "decode_steps": [ + 0.004983457998605445, + 0.004764708995935507, + 0.004998374992283061, + 0.00502608300303109, + 0.005117249995237216, + 0.005547250009840354, + 0.005566207997617312, + 0.005022207988076843, + 0.005110375001095235, + 0.005255416996078566, + 0.005017125004087575, + 0.005455000005895272, + 0.004921832995023578, + 0.00442549999570474, + 0.005247624998446554, + 0.00503166600537952, + 0.004928540991386399, + 0.004546750002191402, + 0.00510029100405518, + 0.004774291999638081, + 0.004957875004038215, + 0.005009792002965696, + 0.004924624998238869, + 0.004652291987440549, + 0.004880499996943399, + 0.005065207995357923, + 0.0050201669946545735, + 0.004738458999781869, + 0.0052549160027410835, + 0.004976833006367087, + 0.005376500004786067, + 0.005197917009354569 + ] + }, + { + "forward": 0.018999749998329207, + "prefill": 0.016273166998871602, + "decode_steps": [ + 0.005602083998383023, + 0.004996624993509613, + 0.005001917001209222, + 0.004892416996881366, + 0.005647500001941808, + 0.005032208995544352, + 0.0050245000020368025, + 0.005105166012072004, + 0.005225416010944173, + 0.004914958000881597, + 0.004908625007374212, + 0.004956208009389229, + 0.004896500002359971, + 0.004635332996258512, + 0.004789708997122943, + 0.005520125007024035, + 0.0051907919987570494, + 0.004961250000633299, + 0.004835583007661626, + 0.004927082991343923, + 0.005007707994082011, + 0.0048645420029060915, + 0.005257374999928288, + 0.005010541004594415, + 0.004538875000434928, + 0.004925249988446012, + 0.0052188340050634, + 0.005036832997575402, + 0.005076166999060661, + 0.004797959001734853, + 0.004990500005078502, + 0.005083791998913512 + ] + }, + { + "forward": 0.017894749995321035, + "prefill": 0.016165082997758873, + "decode_steps": [ + 0.005563957995036617, + 0.004991875000996515, + 0.004865374998189509, + 0.004934917000355199, + 0.004807625009561889, + 0.004787625002791174, + 0.0049953749985434115, + 0.004897999999229796, + 0.005453749996377155, + 0.005075583001598716, + 0.005050082996604033, + 0.0048641670000506565, + 0.0053632909985026345, + 0.005178417006391101, + 0.005178874998819083, + 0.005389290992752649, + 0.006201167008839548, + 0.004971583999576978, + 0.0054294170113280416, + 0.005483417000505142, + 0.005380374990636483, + 0.005202125001233071, + 0.005749250005465001, + 0.00640762499824632, + 0.005516125005669892, + 0.005549416004214436, + 0.005277832999126986, + 0.004857541993260384, + 0.005730000004405156, + 0.005386999997426756, + 0.005199582999921404, + 0.005720209010178223 + ] + }, + { + "forward": 0.01900625000416767, + "prefill": 0.0160610420134617, + "decode_steps": [ + 0.0055850000062491745, + 0.00486458299565129, + 0.00486908400489483, + 0.005141500005265698, + 0.005044417004683055, + 0.004930833994876593, + 0.004868749994784594, + 0.004840916997636668, + 0.005018874988309108, + 0.004824999996344559, + 0.004822124989004806, + 0.004803499992704019, + 0.0049254999903496355, + 0.004843999995500781, + 0.004820582995307632, + 0.004666875000111759, + 0.004775540990522131, + 0.005196833997615613, + 0.0050717919948510826, + 0.0048435409989906475, + 0.005072375002782792, + 0.005091832994366996, + 0.005333290988346562, + 0.004928374997689389, + 0.004956000004312955, + 0.004959124999004416, + 0.004952207993483171, + 0.005129541998030618, + 0.005101999995531514, + 0.005022540994104929, + 0.004890707990853116, + 0.0046032079990254715 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 18.9053545036586, + "p95": 19.424622951919446, + "min": 17.894749995321035, + "max": 19.76154200383462, + "iqr": 0.6424895000236575, + "mean": 18.760179201490246, + "cv": 0.02862084886449855 + }, + "prefill_ms": { + "count": 10, + "median": 15.857104001042899, + "p95": 16.255354203894967, + "min": 15.566165995551273, + "max": 16.273166998871602, + "iqr": 0.496708249556832, + "mean": 15.896604102454148, + "cv": 0.017353607029125695 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.96075000410201, + "p95": 5.629093750030734, + "min": 4.42549999570474, + "max": 6.40762499824632, + "iqr": 0.3320625037304126, + "mean": 5.041077733994825, + "cv": 0.06186067113774136 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1588.4724632222967, + "p95": 1626.6048469224108, + "min": 1510.6731861320397, + "max": 1635.1249575113202, + "iqr": 28.66220059483885, + "mean": 1587.602283026282, + "cv": 0.02094433720996358 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1443.9590885080474, + "p95": 1478.0615190422056, + "min": 1379.117461403015, + "max": 1483.7035338629278, + "iqr": 21.36166657611011, + "mean": 1445.1046143807466, + "cv": 0.019396417167985493 + }, + "memory": { + "peak_extra_bytes": 34605056, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "auto", + "batch": 8, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.024908082996262237 + }, + { + "forward": 0.024108042009174824 + }, + { + "forward": 0.024202624990721233 + }, + { + "forward": 0.024778374994639307 + }, + { + "forward": 0.024509166993084364 + }, + { + "forward": 0.02399420899746474 + }, + { + "forward": 0.02465412500896491 + }, + { + "forward": 0.023914667006465606 + }, + { + "forward": 0.024008124993997626 + }, + { + "forward": 0.025138333992799744 + } + ], + "forward_ms": { + "count": 10, + "median": 24.3558959919028, + "p95": 25.034721044357866, + "min": 23.914667006465606, + "max": 25.138333992799744, + "iqr": 0.7142082504287828, + "mean": 24.42157519835746, + "cv": 0.0177767683341414 + }, + "memory": { + "peak_extra_bytes": 47190016, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "sdpa", + "batch": 8, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.021236415996099822, + "prefill": 0.022695500010740943, + "decode_steps": [ + 0.005819707992486656, + 0.005093665997264907, + 0.0053263750014593825, + 0.005140666995430365, + 0.004739875002996996, + 0.004350250004790723, + 0.004756249996717088, + 0.004275290993973613, + 0.004354375007096678, + 0.0046242090029409155, + 0.004798916997970082, + 0.0037392080121207982, + 0.00391204199695494, + 0.0037709169992012903, + 0.00426754099316895, + 0.0037909579987172037, + 0.003627250000135973, + 0.0036467500030994415, + 0.003492624993668869, + 0.003504958003759384, + 0.0034132080036215484, + 0.0034307080059079453, + 0.003389125005924143, + 0.003470833005849272, + 0.003453334007645026, + 0.003445708003710024, + 0.0034743750002235174, + 0.0034757499961415306, + 0.0033898749970830977, + 0.003394125000340864, + 0.0034875419951276854, + 0.0034480829926906154 + ] + }, + { + "forward": 0.019996041999547742, + "prefill": 0.019998833013232797, + "decode_steps": [ + 0.0028252909978618845, + 0.0027445419982541353, + 0.002652709008543752, + 0.0027261669893050566, + 0.0026222089945804328, + 0.0027119580045109615, + 0.002701834004255943, + 0.0026255839911755174, + 0.0027685829991241917, + 0.0026976660010404885, + 0.002701958001125604, + 0.0027239999908488244, + 0.0027420840051490813, + 0.0027025830058846623, + 0.002700707991607487, + 0.0026855420001083985, + 0.002724791003856808, + 0.002861250002752058, + 0.0027479999989736825, + 0.002787374993204139, + 0.002838792002876289, + 0.002922542000305839, + 0.003175583013216965, + 0.003029458996024914, + 0.0031157920020632446, + 0.0031043330091051757, + 0.0029232079978100955, + 0.0032798750034999102, + 0.0033309580030618235, + 0.0036634999996749684, + 0.003358457994181663, + 0.0033770830050343648 + ] + }, + { + "forward": 0.020113749997108243, + "prefill": 0.02002204100426752, + "decode_steps": [ + 0.0028095419984310865, + 0.002806375006912276, + 0.002670375004527159, + 0.0026654589892132208, + 0.002641792001668364, + 0.0027120419981656596, + 0.0027457090036477894, + 0.0026498750085011125, + 0.0027007080061594024, + 0.0026975000073434785, + 0.0027774579939432442, + 0.0027245840028626844, + 0.002771500003291294, + 0.002686375009943731, + 0.0027109580114483833, + 0.0028005000058328733, + 0.0026912920002359897, + 0.0028866250067949295, + 0.0027411670016590506, + 0.0032957920047920197, + 0.0031656250066589564, + 0.003065750002861023, + 0.0030580409948015586, + 0.003010875007021241, + 0.0030205410002963617, + 0.003325917001347989, + 0.0033176249999087304, + 0.0033083339949371293, + 0.0032476249907631427, + 0.0034482500050216913, + 0.003488041998934932, + 0.00349183299113065 + ] + }, + { + "forward": 0.021979416997055523, + "prefill": 0.02215629199054092, + "decode_steps": [ + 0.0059562079986790195, + 0.005023040997912176, + 0.004829541008803062, + 0.004269416996976361, + 0.004391541006043553, + 0.004286790994228795, + 0.004460625001229346, + 0.004375541000626981, + 0.004393333001644351, + 0.004345875000581145, + 0.004469124993192963, + 0.004423749996931292, + 0.004537000000709668, + 0.0045044169964967296, + 0.004365832995972596, + 0.004298250001738779, + 0.003672959006507881, + 0.0036754580069100484, + 0.0036229170073056594, + 0.00439395901048556, + 0.004501832998357713, + 0.0036417500086827204, + 0.0037078750028740615, + 0.004193832995952107, + 0.003802499995799735, + 0.0037189589929766953, + 0.003951332997530699, + 0.003712458987138234, + 0.003878292001900263, + 0.0037721249973401427, + 0.0036823330010520294, + 0.003715999991982244 + ] + }, + { + "forward": 0.020313875007559545, + "prefill": 0.0200230839982396, + "decode_steps": [ + 0.0029150420014047995, + 0.002747124992311001, + 0.0027125839988002554, + 0.002620166997076012, + 0.002670375004527159, + 0.0027591660036705434, + 0.002693832997465506, + 0.0026329170068493113, + 0.0027126249915454537, + 0.002730583000811748, + 0.0027246249956078827, + 0.00277633398945909, + 0.0026693749969126657, + 0.002791499995510094, + 0.0027094169927295297, + 0.0028698329988401383, + 0.0028509580006357282, + 0.002856999999494292, + 0.0027557919966056943, + 0.002915290999226272, + 0.0033125000045401976, + 0.003134583996143192, + 0.00300174999574665, + 0.0030926660110708326, + 0.0030115409899735823, + 0.0031389999930979684, + 0.0032485419942531735, + 0.003366167002241127, + 0.0031606660049874336, + 0.003513915988150984, + 0.00323033299355302, + 0.003468041992164217 + ] + }, + { + "forward": 0.019981082994490862, + "prefill": 0.02034191699931398, + "decode_steps": [ + 0.002948000008473173, + 0.0028027910011587664, + 0.0027759580116253346, + 0.002790207989164628, + 0.0026107909943675622, + 0.002602332999231294, + 0.002643125000759028, + 0.002622166008222848, + 0.002753624998149462, + 0.002695125003810972, + 0.002688874999876134, + 0.002779499991447665, + 0.0027269999991403893, + 0.0026363749930169433, + 0.002755541994702071, + 0.0027918340056203306, + 0.002688499997020699, + 0.0027612090052571148, + 0.002751958993030712, + 0.0028664999990724027, + 0.0028068749961676076, + 0.002770166000118479, + 0.002824334005708806, + 0.002834374987287447, + 0.003075665998039767, + 0.0030350419983733445, + 0.0029691250092582777, + 0.003049083999940194, + 0.0030263330118032172, + 0.0033749579888535663, + 0.003299250005511567, + 0.003254499999457039 + ] + }, + { + "forward": 0.021180166993872263, + "prefill": 0.020490666007390246, + "decode_steps": [ + 0.003107416006969288, + 0.002889208000851795, + 0.0028390420047799125, + 0.0029652910016011447, + 0.0028693749918602407, + 0.0028663339908234775, + 0.002742957993177697, + 0.00263499999709893, + 0.0027576660068007186, + 0.0028368339990265667, + 0.002758792004897259, + 0.0029256670095492154, + 0.002925457985838875, + 0.0029502919933293015, + 0.003013167006429285, + 0.0028735000087181106, + 0.0030496670078719035, + 0.0031110420095501468, + 0.003488582995487377, + 0.0031305830052588135, + 0.003083709001657553, + 0.0034701659897109494, + 0.0033876660017995164, + 0.0034254589991178364, + 0.003456916005234234, + 0.003638875001342967, + 0.0036874999932479113, + 0.0037029160012025386, + 0.003660750007838942, + 0.004925208995700814, + 0.004908708011498675, + 0.004932750001898967 + ] + }, + { + "forward": 0.020071584003744647, + "prefill": 0.02031054100370966, + "decode_steps": [ + 0.002776541994535364, + 0.0026191659999312833, + 0.00285858400457073, + 0.0026513750053709373, + 0.0028893339913338423, + 0.00274008400447201, + 0.002661624996108003, + 0.0026522499974817038, + 0.002758542002993636, + 0.0026072500040754676, + 0.0030542499880539253, + 0.002756874993792735, + 0.002626792003866285, + 0.002837000007275492, + 0.0026517920050537214, + 0.0027375420031603426, + 0.0028249169990886003, + 0.0027461669960757717, + 0.002768667007330805, + 0.0028876669966848567, + 0.002873582998290658, + 0.0030996249988675117, + 0.0029826670070178807, + 0.0030513749952660874, + 0.002963874998386018, + 0.003084708994720131, + 0.003288958003395237, + 0.0032580420083831996, + 0.003276540999650024, + 0.0033417500089854, + 0.003516167009365745, + 0.003456291990005411 + ] + }, + { + "forward": 0.020319667004514486, + "prefill": 0.020206291999784298, + "decode_steps": [ + 0.002965834006317891, + 0.002780874987365678, + 0.0027034160011680797, + 0.0026214999961666763, + 0.0026525840075919405, + 0.002721416996791959, + 0.002641499988385476, + 0.002673291994142346, + 0.0026696670101955533, + 0.002711916997213848, + 0.002695167000638321, + 0.0027756670024245977, + 0.002863499990780838, + 0.002815875006490387, + 0.002777250003418885, + 0.002757916008704342, + 0.0030251249991124496, + 0.003039500006707385, + 0.0030190420075086877, + 0.0029123749991413206, + 0.002975708994199522, + 0.003208624999388121, + 0.0034675420029088855, + 0.00332324999908451, + 0.0032163750001927838, + 0.003690208002808504, + 0.003454292003880255, + 0.0035026249970542267, + 0.003445583992288448, + 0.003657458000816405, + 0.004516207991400734, + 0.004511040999204852 + ] + }, + { + "forward": 0.02139779200660996, + "prefill": 0.022190583011251874, + "decode_steps": [ + 0.0065773340029409155, + 0.0053505830001086, + 0.005049166997196153, + 0.004915458004688844, + 0.00477616599528119, + 0.005506957997567952, + 0.004375125005026348, + 0.004828583012567833, + 0.004412291993503459, + 0.00452216700068675, + 0.004365583008620888, + 0.004413667003973387, + 0.004484083998249844, + 0.004699457989772782, + 0.0038351249968400225, + 0.0037917089939583093, + 0.0036841249966528267, + 0.003624917007982731, + 0.00364550000813324, + 0.003763957996852696, + 0.004479082999750972, + 0.00448083299852442, + 0.0037168749986449257, + 0.0036879590043099597, + 0.004433041991433129, + 0.003649915990536101, + 0.003701707988511771, + 0.0036183749907650054, + 0.003811000002315268, + 0.0037977920001139864, + 0.0037025829951744527, + 0.0036800410016439855 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 20.316771006037015, + "p95": 21.71768575135502, + "min": 19.981082994490862, + "max": 21.979416997055523, + "iqr": 1.1402282434573863, + "mean": 20.65897930006031, + "cv": 0.0348682414292255 + }, + "prefill_ms": { + "count": 10, + "median": 20.32622900151182, + "p95": 22.468287360970862, + "min": 19.998833013232797, + "max": 22.695500010740943, + "iqr": 1.6709994961274788, + "mean": 20.843574903847184, + "cv": 0.05081237585355678 + }, + "decode_ms_per_step": { + "count": 320, + "median": 3.084208998188842, + "p95": 4.800400298699969, + "min": 2.602332999231294, + "max": 6.5773340029409155, + "iqr": 0.9159055007330608, + "mean": 3.3402601656234765, + "cv": 0.21654204940565835 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 2655.409561163806, + "p95": 2791.4679035395056, + "min": 1863.4286453178263, + "max": 2812.8429665126887, + "iqr": 634.3541303653737, + "mean": 2456.0901397339844, + "cv": 0.15702918553483433 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 2196.914050089454, + "p95": 2287.7805286019625, + "min": 1604.2940033598397, + "max": 2298.994241352933, + "iqr": 470.8342274286904, + "mean": 2044.2179774356057, + "cv": 0.14011292752511906 + }, + "memory": { + "peak_extra_bytes": 47190016, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "reference", + "batch": 8, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.04631904199777637, + "prefill": 0.04090991600241978, + "decode_steps": [ + 0.011517417005961761, + 0.009093915999983437, + 0.007983625007909723, + 0.0074406250059837475, + 0.007490832998882979, + 0.007279583995114081, + 0.006682332998025231, + 0.0066110000043408945, + 0.00628349999897182, + 0.006206666002981365, + 0.006406249987776391, + 0.006215625006007031, + 0.005740332999266684, + 0.005710500001441687, + 0.005667125005857088, + 0.005565708997892216, + 0.005471374999615364, + 0.005437416999484412, + 0.005437333005829714, + 0.005348959006369114, + 0.005230124996160157, + 0.005532250012038276, + 0.005422208996606059, + 0.005484875000547618, + 0.005430875011370517, + 0.005769249997683801, + 0.005548749992158264, + 0.005688042001565918, + 0.005243874998996034, + 0.005341500000213273, + 0.0065810829983092844, + 0.005855708004673943 + ] + }, + { + "forward": 0.04571183399821166, + "prefill": 0.04040004099078942, + "decode_steps": [ + 0.012609875004272908, + 0.009289833993534558, + 0.008075166988419369, + 0.007895582995843142, + 0.007515708988648839, + 0.006987750006373972, + 0.007572625007014722, + 0.007293708011275157, + 0.006657207995885983, + 0.006734540991601534, + 0.006182333003380336, + 0.005844957995577715, + 0.0059326249902369455, + 0.005471042008139193, + 0.0054530839988728985, + 0.00527658399369102, + 0.005356749999918975, + 0.005443333997391164, + 0.005149792006704956, + 0.005369417005567811, + 0.005355834000511095, + 0.005287459003739059, + 0.005498083002748899, + 0.005444667011033744, + 0.005611417000181973, + 0.005571750007220544, + 0.005664917000103742, + 0.0058752920012921095, + 0.005532208000659011, + 0.005672499988577329, + 0.005396959008066915, + 0.005265083003905602 + ] + }, + { + "forward": 0.041933959000743926, + "prefill": 0.03926937500364147, + "decode_steps": [ + 0.005906416990910657, + 0.006310917000519112, + 0.0060305419901851565, + 0.0057442919933237135, + 0.005472584001836367, + 0.005883707999601029, + 0.004944083993905224, + 0.00554983300389722, + 0.00541825000254903, + 0.005263999992166646, + 0.0053270830103429034, + 0.005435041995951906, + 0.005144834009115584, + 0.005546957996557467, + 0.00547750000259839, + 0.005406249998486601, + 0.00547170800564345, + 0.005523250001715496, + 0.005051000000094064, + 0.005320583004504442, + 0.0055092080001486465, + 0.005210374991293065, + 0.005229500005953014, + 0.0054920410038903356, + 0.0054064169962657616, + 0.005190332987695001, + 0.005420917004812509, + 0.005442292007501237, + 0.0059230829938314855, + 0.006050041993148625, + 0.005795958000817336, + 0.005617457994958386 + ] + }, + { + "forward": 0.046696999997948296, + "prefill": 0.041375084008905105, + "decode_steps": [ + 0.010945582995191216, + 0.010611290999804623, + 0.008264916992629878, + 0.00839345900749322, + 0.007900041993707418, + 0.007049749998259358, + 0.006812374995206483, + 0.006741666991729289, + 0.005832458002259955, + 0.005732749996241182, + 0.005616291004116647, + 0.005541749997064471, + 0.005436499995994382, + 0.005470125004649162, + 0.0053081249934621155, + 0.005145750008523464, + 0.005367999998270534, + 0.005477124999742955, + 0.005286665997118689, + 0.015483582988963462, + 0.00870654100435786, + 0.00618833300541155, + 0.005913166998652741, + 0.005637959009618498, + 0.005857791999005713, + 0.005796875004307367, + 0.005419292006990872, + 0.005495874996995553, + 0.0056532499875174835, + 0.005248417000984773, + 0.005767499998910353, + 0.0057410000008530915 + ] + }, + { + "forward": 0.04466670801048167, + "prefill": 0.04007337499933783, + "decode_steps": [ + 0.00999908299127128, + 0.008926499998779036, + 0.007956040994031355, + 0.006939375001820736, + 0.007079625007463619, + 0.006394500000169501, + 0.00658425000438001, + 0.006481374992290512, + 0.006316707993391901, + 0.006359667007927783, + 0.0062084589881123975, + 0.006144541999674402, + 0.006313542005955242, + 0.005863541999133304, + 0.006571791003807448, + 0.006657541991444305, + 0.0063382920052390546, + 0.006266791999223642, + 0.005358958005672321, + 0.0055294170015258715, + 0.005430290999356657, + 0.005372958010411821, + 0.0053745409968541935, + 0.0055048329959390685, + 0.005399083995143883, + 0.005320708005456254, + 0.005315959002473392, + 0.00552070900448598, + 0.005506834000698291, + 0.005321624994394369, + 0.005260958001599647, + 0.00539079200825654 + ] + }, + { + "forward": 0.04209225000522565, + "prefill": 0.040044790992396884, + "decode_steps": [ + 0.006625708003412001, + 0.00554833299247548, + 0.005170332995476201, + 0.006625333000556566, + 0.0066613340022740886, + 0.007816625002305955, + 0.009491875011008233, + 0.007470333002856933, + 0.0061123750056140125, + 0.00661920799757354, + 0.00644445899524726, + 0.005743583009461872, + 0.005730249991756864, + 0.005617540999082848, + 0.005537332996027544, + 0.005561583006056026, + 0.005418708999059163, + 0.005377000008593313, + 0.005693708997569047, + 0.005722333007724956, + 0.006263750008656643, + 0.005642417003400624, + 0.006366250003338791, + 0.0054655830026604235, + 0.005746209004428238, + 0.005387292010709643, + 0.0055311250034719706, + 0.005443916990770958, + 0.005478125007357448, + 0.00553004200628493, + 0.005593875001068227, + 0.005502665997482836 + ] + }, + { + "forward": 0.05402920900087338, + "prefill": 0.044506124992039986, + "decode_steps": [ + 0.01130341699172277, + 0.008397666999371722, + 0.007932249995064922, + 0.00752837500476744, + 0.007573874987429008, + 0.007428583005093969, + 0.006619375009904616, + 0.0066791659919545054, + 0.006163500002003275, + 0.006039416010025889, + 0.006423541999538429, + 0.006103334002546035, + 0.006091875009587966, + 0.006002666996209882, + 0.006025291993864812, + 0.005772000004071742, + 0.0058096669963561, + 0.00589133299945388, + 0.00583691599604208, + 0.005841999998665415, + 0.005952374995104037, + 0.0062794159894110635, + 0.0056859999895095825, + 0.0056979579967446625, + 0.005991250000079162, + 0.005605665995972231, + 0.00579991600534413, + 0.0056613750057294965, + 0.0054639169975416735, + 0.005543249993934296, + 0.005408457989688031, + 0.005250665999483317 + ] + }, + { + "forward": 0.045281375001650304, + "prefill": 0.040492083004210144, + "decode_steps": [ + 0.011516749989823438, + 0.009044415986863896, + 0.008444374994724058, + 0.007504791996325366, + 0.007650291008758359, + 0.00809983401268255, + 0.007178291998570785, + 0.0065342080051777884, + 0.006288208998739719, + 0.006465208003646694, + 0.005824791005579755, + 0.005864916995051317, + 0.005535832999157719, + 0.005489625007612631, + 0.005350499995984137, + 0.005383583004004322, + 0.005422374990303069, + 0.005451249991892837, + 0.005514707998372614, + 0.005353374988771975, + 0.005118208006024361, + 0.005334166009561159, + 0.005345209006918594, + 0.005373207997763529, + 0.005345290992408991, + 0.00543808400107082, + 0.0054966669995337725, + 0.005778958002338186, + 0.005331458000000566, + 0.005487041009473614, + 0.00539095800195355, + 0.005365124990930781 + ] + }, + { + "forward": 0.04119287501089275, + "prefill": 0.0394114999944577, + "decode_steps": [ + 0.006023292007739656, + 0.005373207997763529, + 0.005512458010343835, + 0.005305500002577901, + 0.005416332991444506, + 0.005429124998045154, + 0.005409332996350713, + 0.005302207995555364, + 0.005281124991597608, + 0.005428875010693446, + 0.005366500001400709, + 0.005401916001574136, + 0.005627666003420018, + 0.005389166995882988, + 0.005482499997015111, + 0.005350833002012223, + 0.005402583003160544, + 0.005663791991537437, + 0.005382624993217178, + 0.005310542008373886, + 0.0054887080041226, + 0.0052885420009261, + 0.0052060419984627515, + 0.005426207993878052, + 0.005288208005367778, + 0.0052680829976452515, + 0.005201457999646664, + 0.005301291996147484, + 0.005445124988909811, + 0.00525541600654833, + 0.00535920899710618, + 0.005419667009846307 + ] + }, + { + "forward": 0.04695124999852851, + "prefill": 0.04141870899184141, + "decode_steps": [ + 0.007510333001846448, + 0.00601937499595806, + 0.00587637499847915, + 0.005537167002330534, + 0.006048415991244838, + 0.011103124998044223, + 0.0072692079993430525, + 0.008440416000667028, + 0.007246708002639934, + 0.00666800000180956, + 0.007445790994097479, + 0.00607379199936986, + 0.005816167002194561, + 0.005619292001938447, + 0.00711437501013279, + 0.006063209002604708, + 0.006009125005220994, + 0.005818041987367906, + 0.005458041996462271, + 0.005537541990634054, + 0.005669499994837679, + 0.005636125002638437, + 0.005397749991971068, + 0.005407499993452802, + 0.0070996660069795325, + 0.008044791000429541, + 0.008653500000946224, + 0.010432291994220577, + 0.007342750002862886, + 0.007976082997629419, + 0.008034625003347173, + 0.006613749996176921 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 45.49660449993098, + "p95": 50.84412744981818, + "min": 41.19287501089275, + "max": 54.02920900087338, + "iqr": 3.866645991365658, + "mean": 45.48755020223325, + "cv": 0.08048259731020611 + }, + "prefill_ms": { + "count": 10, + "median": 40.44606199749978, + "p95": 43.11678779195063, + "min": 39.26937500364147, + "max": 44.506124992039986, + "iqr": 1.206855013151653, + "mean": 40.79009989800397, + "cv": 0.03662244287324485 + }, + "decode_ms_per_step": { + "count": 320, + "median": 5.664354495820589, + "p95": 8.656152051116806, + "min": 4.944083993905224, + "max": 15.483582988963462, + "iqr": 0.9899684846459422, + "mean": 6.18635154992262, + "cv": 0.21591212794611422 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1280.8945441700648, + "p95": 1467.4096263061858, + "min": 1169.0414161670526, + "max": 1481.4168495420722, + "iqr": 58.73075318625183, + "mean": 1299.6374375163184, + "cv": 0.07608217885816787 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1064.82486591515, + "p95": 1197.3281122009191, + "min": 983.097100948414, + "max": 1206.3008164348553, + "iqr": 54.84274534020528, + "mean": 1076.3833867765454, + "cv": 0.06673333410144039 + }, + "memory": { + "peak_extra_bytes": 119541760, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "auto", + "batch": 8, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.05635454200091772 + }, + { + "forward": 0.05525741699966602 + }, + { + "forward": 0.05549770800280385 + }, + { + "forward": 0.05725404099212028 + }, + { + "forward": 0.05451958300545812 + }, + { + "forward": 0.0553302500047721 + }, + { + "forward": 0.05622241699893493 + }, + { + "forward": 0.05479837500024587 + }, + { + "forward": 0.05460725000011735 + }, + { + "forward": 0.05661333400348667 + } + ], + "forward_ms": { + "count": 10, + "median": 55.41397900378797, + "p95": 56.965722847235156, + "min": 54.51958300545812, + "max": 57.25404099212028, + "iqr": 1.4083752503211144, + "mean": 55.64549170085229, + "cv": 0.01661678726790499 + }, + "memory": { + "peak_extra_bytes": 94380032, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "sdpa", + "batch": 8, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.045443416995112784, + "prefill": 0.04616879099921789, + "decode_steps": [ + 0.008227124999393709, + 0.0070100000011734664, + 0.006932750009582378, + 0.007494207995478064, + 0.006415124997147359, + 0.006416750009520911, + 0.0063854580075712875, + 0.00633991599897854, + 0.0059118329954799265, + 0.005844708997756243, + 0.0059261249989503995, + 0.005838041004608385, + 0.004899291991023347, + 0.005406334006693214, + 0.004893249992164783, + 0.004880000007688068, + 0.005171042008441873, + 0.005620541007374413, + 0.004911124997306615, + 0.0049370829947292805, + 0.004504333002842031, + 0.004808999990927987, + 0.005017749994294718, + 0.004635791992768645, + 0.0045904999860795215, + 0.004562791000353172, + 0.0045777079940307885, + 0.004507125006057322, + 0.004553292004857212, + 0.00458216700644698, + 0.004605499998433515, + 0.004544166993582621 + ] + }, + { + "forward": 0.043666124998708256, + "prefill": 0.042422583006555215, + "decode_steps": [ + 0.0037835000111954287, + 0.0036804170085815713, + 0.0035910830047214404, + 0.0035112089972244576, + 0.0036358330107759684, + 0.0037138340121600777, + 0.0036282919900259003, + 0.0035800829937215894, + 0.003588207997381687, + 0.003935583998099901, + 0.0036478750116657466, + 0.003675250001833774, + 0.0036818329972447827, + 0.003918625006917864, + 0.003913458000170067, + 0.003918999995221384, + 0.004071417002705857, + 0.004156708004302345, + 0.004173624998657033, + 0.004124833008972928, + 0.004262832997483201, + 0.004325082991272211, + 0.004287667004973628, + 0.004382500002975576, + 0.005000749995815568, + 0.0049525000067660585, + 0.005509374997927807, + 0.0048133340023923665, + 0.004877166997175664, + 0.0051580419967649505, + 0.0048614999977871776, + 0.0045840419916203246 + ] + }, + { + "forward": 0.042440500008524396, + "prefill": 0.04284191699116491, + "decode_steps": [ + 0.0038365419895853847, + 0.0036743750097230077, + 0.003550499997800216, + 0.003591790999053046, + 0.00364475000242237, + 0.0035555840004235506, + 0.0035142910055583343, + 0.0035895409964723513, + 0.003595417001633905, + 0.003618040995206684, + 0.0035560420074034482, + 0.0035632499930216, + 0.003571624998585321, + 0.0036214590072631836, + 0.003656792003312148, + 0.003654250002000481, + 0.003674333987873979, + 0.0036473749933065847, + 0.004525791999185458, + 0.004143334008404054, + 0.004019290994619951, + 0.004172375003690831, + 0.004247207994922064, + 0.0042028750031022355, + 0.00424962499528192, + 0.004742666991660371, + 0.0045438329980243, + 0.004371959003037773, + 0.005301083001540974, + 0.005502792002516799, + 0.005515458004083484, + 0.006513334010378458 + ] + }, + { + "forward": 0.04505899999639951, + "prefill": 0.04590658399683889, + "decode_steps": [ + 0.009316375013440847, + 0.007436291998601519, + 0.007814791999408044, + 0.0064207499963231385, + 0.006661709005129524, + 0.006221834002644755, + 0.006327709008473903, + 0.006654500000877306, + 0.006123750004917383, + 0.005880791999516077, + 0.0058359999966342, + 0.004923625005176291, + 0.005445250004413538, + 0.004894833007710986, + 0.004926333989715204, + 0.005380999995395541, + 0.00488441699417308, + 0.005410958008724265, + 0.004992417001631111, + 0.00491037500614766, + 0.005425625000498258, + 0.00550162501167506, + 0.005151000004843809, + 0.0047810830001253635, + 0.0047711660008644685, + 0.0047810830001253635, + 0.004700624995166436, + 0.004574999999022111, + 0.004567416995996609, + 0.00460233299236279, + 0.00462849999894388, + 0.004618791994289495 + ] + }, + { + "forward": 0.04281687500770204, + "prefill": 0.04287950000434648, + "decode_steps": [ + 0.003936667009838857, + 0.0039087909972295165, + 0.0036387919972185045, + 0.0035730840027099475, + 0.003654708998510614, + 0.0036687910032924265, + 0.003645499993581325, + 0.003745458001503721, + 0.0038870830030646175, + 0.0037270830071065575, + 0.003988791999290697, + 0.003888917010044679, + 0.003953375009587035, + 0.003925375000108033, + 0.0038693329988745973, + 0.004135416995268315, + 0.004448833002243191, + 0.004153667003265582, + 0.0042405410058563575, + 0.004276125007891096, + 0.004486208010348491, + 0.004427958992891945, + 0.004841583999223076, + 0.005125041992869228, + 0.004607416994986124, + 0.004815374995814636, + 0.0046709159942111, + 0.004596124999807216, + 0.0045740829955320805, + 0.004575500002829358, + 0.004693208000389859, + 0.004604083005688153 + ] + }, + { + "forward": 0.043440125009510666, + "prefill": 0.04310899999109097, + "decode_steps": [ + 0.0043440000008558854, + 0.004072458992595784, + 0.00410833400383126, + 0.004063957996550016, + 0.004462041993974708, + 0.004660000005969778, + 0.004404416991746984, + 0.0043332089990144596, + 0.0046363330038730055, + 0.004934958997182548, + 0.0055728329898556694, + 0.006875292005133815, + 0.005896208007470705, + 0.0059270829951856285, + 0.005807749999803491, + 0.004944499989505857, + 0.004908041999442503, + 0.004966916996636428, + 0.004734125002869405, + 0.0046270839957287535, + 0.004547374992398545, + 0.004519582987995818, + 0.00459450000198558, + 0.004536583990557119, + 0.00459841699921526, + 0.004576499995891936, + 0.004539792003924958, + 0.004584790993249044, + 0.00462737500492949, + 0.004570874996716157, + 0.004554208993795328, + 0.004582624998874962 + ] + }, + { + "forward": 0.04439525000634603, + "prefill": 0.045577791999676265, + "decode_steps": [ + 0.005426749994512647, + 0.005490125011419877, + 0.005136542007676326, + 0.006377375000738539, + 0.005929958002525382, + 0.007525042004999705, + 0.007258124998770654, + 0.006735500006470829, + 0.006353040997055359, + 0.0061547090008389205, + 0.006271916994592175, + 0.0061120419995859265, + 0.006042416993295774, + 0.006072749994928017, + 0.00589208300516475, + 0.005951790997642092, + 0.00594179100880865, + 0.004859000007854775, + 0.0049256670026807114, + 0.005628708997392096, + 0.004605416004778817, + 0.004610125004546717, + 0.004550459008896723, + 0.004647666006349027, + 0.004806875003851019, + 0.0045702089992119, + 0.00455691700335592, + 0.004771834006533027, + 0.004586499999277294, + 0.004553708000457846, + 0.004586750001180917, + 0.004562875008559786 + ] + }, + { + "forward": 0.04280304099665955, + "prefill": 0.04233983300218824, + "decode_steps": [ + 0.0037234160117805004, + 0.004004375005024485, + 0.0036530409997794777, + 0.0037245410057948902, + 0.00352866698813159, + 0.003840125005808659, + 0.0035442920052446425, + 0.003621500000008382, + 0.0036732500011567026, + 0.003609708001022227, + 0.0036273329897085205, + 0.003601791991968639, + 0.003928499994799495, + 0.003714624996064231, + 0.003682249996927567, + 0.003701167006511241, + 0.0038635420060018077, + 0.003975582993007265, + 0.004012958001112565, + 0.0038515000051120296, + 0.0041630840132711455, + 0.00421391699637752, + 0.0041855409945128486, + 0.004422500001965091, + 0.004448334002518095, + 0.004302792003727518, + 0.004369207992567681, + 0.004980874989996664, + 0.005451291988720186, + 0.006469457992352545, + 0.006019791006110609, + 0.005204166998737492 + ] + }, + { + "forward": 0.04245316699962132, + "prefill": 0.042479042007471435, + "decode_steps": [ + 0.0037759589904453605, + 0.0038560839893762022, + 0.0036679159966297448, + 0.003643708012532443, + 0.0036901669955113903, + 0.00356308399932459, + 0.0035557919909479097, + 0.003584417005185969, + 0.003641666000476107, + 0.0035890829894924536, + 0.003514874988468364, + 0.0036296670004958287, + 0.003651582999737002, + 0.003670334001071751, + 0.003657292007119395, + 0.003851999994367361, + 0.004029125004308298, + 0.003914125001756474, + 0.003932416992029175, + 0.004159375006565824, + 0.00418158300453797, + 0.004112874987185933, + 0.0045306670072022825, + 0.004378915997222066, + 0.004515749998972751, + 0.004315083002438769, + 0.0052284999983385205, + 0.005475333004142158, + 0.006518665992189199, + 0.006524374999571592, + 0.006149250009912066, + 0.007031207991531119 + ] + }, + { + "forward": 0.04614216600020882, + "prefill": 0.045719834000919946, + "decode_steps": [ + 0.008262000003014691, + 0.0071812079986557364, + 0.006791249994421378, + 0.006362332991557196, + 0.006578957996680401, + 0.006295208993833512, + 0.0060729160031769425, + 0.0061691669980064034, + 0.007097875000908971, + 0.006856833002530038, + 0.005836791999172419, + 0.004723082995042205, + 0.004852375001064502, + 0.005596125003648922, + 0.004687167005613446, + 0.00495000000228174, + 0.0056543329992564395, + 0.0048361250082962215, + 0.004592208992107771, + 0.004690332993050106, + 0.0046274170017568395, + 0.004559708992019296, + 0.004486000005272217, + 0.004574834005325101, + 0.004665792002924718, + 0.004530542006250471, + 0.00460141699295491, + 0.004662750012357719, + 0.004541667003650218, + 0.0045058750110911205, + 0.0045110840001143515, + 0.0045034590002615005 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 43.55312500410946, + "p95": 45.8277289479156, + "min": 42.440500008524396, + "max": 46.14216600020882, + "iqr": 2.0865629994659685, + "mean": 43.865966601879336, + "cv": 0.03025523321606528 + }, + "prefill_ms": { + "count": 10, + "median": 42.994249997718725, + "p95": 46.05079784814734, + "min": 42.33983300218824, + "max": 46.16879099921789, + "iqr": 3.1145627472142223, + "mean": 43.944487599947024, + "cv": 0.03770100621255762 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.575250000925735, + "p95": 6.738287505868358, + "min": 3.5112089972244576, + "max": 9.316375013440847, + "iqr": 1.2445310094335582, + "mean": 4.762604206416654, + "cv": 0.21576513705274192 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1766.6208474218752, + "p95": 1939.9566927072235, + "min": 1433.6278384941413, + "max": 1951.7017104329877, + "iqr": 451.2848308251523, + "mean": 1707.6755257166624, + "cv": 0.13325457972751378 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1363.3575612858235, + "p95": 1466.055392317814, + "min": 1140.441113115997, + "max": 1471.1840277054996, + "iqr": 294.26203234033414, + "mean": 1318.9151635222715, + "cv": 0.11159169555772776 + }, + "memory": { + "peak_extra_bytes": 94380032, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "reference", + "batch": 8, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.11687937499664258, + "prefill": 0.11397004200262018, + "decode_steps": [ + 0.01902841599076055, + 0.015911540991510265, + 0.011485750001156703, + 0.010886125004617497, + 0.009340208998764865, + 0.008455415998469107, + 0.00858762499410659, + 0.007980082998983562, + 0.0077025000064168125, + 0.008370707990252413, + 0.007809291986632161, + 0.007673042011447251, + 0.007638333001523279, + 0.0076693750015692785, + 0.007510458992328495, + 0.007428041004459374, + 0.007197625003755093, + 0.007205667003290728, + 0.008018834007089026, + 0.007664167002076283, + 0.007423957998980768, + 0.00724554099724628, + 0.007050165993859991, + 0.006988083012402058, + 0.0073416249942965806, + 0.006948249996639788, + 0.006801417010137811, + 0.007045791993732564, + 0.006866707990411669, + 0.006936707999557257, + 0.007013374997768551, + 0.007290207999176346 + ] + }, + { + "forward": 0.11581649999425281, + "prefill": 0.11308904198813252, + "decode_steps": [ + 0.01764604198979214, + 0.016174250005860813, + 0.01633791599306278, + 0.010465833998750895, + 0.008239999995566905, + 0.008256375003838912, + 0.007916125003248453, + 0.007404291012790054, + 0.0077153339952928945, + 0.007305833001737483, + 0.006783125005313195, + 0.006796749992645346, + 0.007079750008415431, + 0.006808207996073179, + 0.006881708002765663, + 0.007047209001029842, + 0.006668375004664995, + 0.007134625004255213, + 0.007960416987771168, + 0.006915916004800238, + 0.007178166997618973, + 0.007105166994733736, + 0.007074167006067, + 0.007004832994425669, + 0.007251583010656759, + 0.007054958987282589, + 0.007243000000016764, + 0.0068820419983239844, + 0.006750250002369285, + 0.006934457996976562, + 0.006547665994730778, + 0.006902624998474494 + ] + }, + { + "forward": 0.11359125000308268, + "prefill": 0.11351837500114925, + "decode_steps": [ + 0.007960874994751066, + 0.00652741699013859, + 0.006156999996164814, + 0.006743542006006464, + 0.0064142500050365925, + 0.006451624998589978, + 0.006398916011676192, + 0.006787000005715527, + 0.006830917001934722, + 0.006577624997589737, + 0.006432416994357482, + 0.006337792001431808, + 0.006464499994763173, + 0.006491874999483116, + 0.006653000004007481, + 0.007041500008199364, + 0.006602541994652711, + 0.006606791008380242, + 0.00646612499258481, + 0.0064797920058481395, + 0.006507124999188818, + 0.006430666006053798, + 0.006536874992889352, + 0.006502666990854777, + 0.006421917001716793, + 0.006484041994553991, + 0.006535249995067716, + 0.006579332999535836, + 0.006343415996525437, + 0.0065237500093644485, + 0.006592292003915645, + 0.006369915994582698 + ] + }, + { + "forward": 0.11705604099552147, + "prefill": 0.11374950000026729, + "decode_steps": [ + 0.019860207990859635, + 0.014634541003033519, + 0.010680166989914142, + 0.009887041000183672, + 0.009116665998590179, + 0.008550250000553206, + 0.00846000001183711, + 0.007985042000655085, + 0.007247999994433485, + 0.007729125005425885, + 0.007822291008778848, + 0.007731459001661278, + 0.007723333008470945, + 0.007637917005922645, + 0.007463916001142934, + 0.007568415996502154, + 0.007998250002856366, + 0.009059584001079202, + 0.007891459012171254, + 0.007910208005341701, + 0.0074447910010349005, + 0.007485041001928039, + 0.007292791997315362, + 0.007328334002522752, + 0.007464707989129238, + 0.0072014589968603104, + 0.007020082994131371, + 0.00694574999215547, + 0.0065013329876819625, + 0.006478208000771701, + 0.006689957997878082, + 0.0064038749987958 + ] + }, + { + "forward": 0.11569533300644252, + "prefill": 0.1153825420042267, + "decode_steps": [ + 0.01847233300213702, + 0.01421525000478141, + 0.010121874991455115, + 0.010206541992374696, + 0.009529165996355005, + 0.009027584004797973, + 0.008018666005227715, + 0.008341249995282851, + 0.008062833003350534, + 0.008057582992478274, + 0.00859324999328237, + 0.007453124999301508, + 0.008055541009525768, + 0.00730912500876002, + 0.006955916993319988, + 0.007373832995654084, + 0.006713332986691967, + 0.006585750001249835, + 0.006840083995484747, + 0.006732624999131076, + 0.006871709003462456, + 0.006565542003954761, + 0.006792165993829258, + 0.006992624999838881, + 0.006668207992333919, + 0.006793040985940024, + 0.007998417000635527, + 0.00715233400114812, + 0.006952374998945743, + 0.007508499998948537, + 0.006822625000495464, + 0.0066829999996116385 + ] + }, + { + "forward": 0.11483799999405164, + "prefill": 0.113180583997746, + "decode_steps": [ + 0.008409834001213312, + 0.007648083003005013, + 0.006837791996076703, + 0.006684167005005293, + 0.006308208001428284, + 0.006506417004857212, + 0.0064162089984165505, + 0.006436124996980652, + 0.006715916999382898, + 0.006547874989337288, + 0.006311083008768037, + 0.0063558340043528005, + 0.006517625006381422, + 0.006530208003823645, + 0.006395290998625569, + 0.006475166999734938, + 0.006500957999378443, + 0.00694129201292526, + 0.00682799999776762, + 0.006133875009254552, + 0.006516042005387135, + 0.006579458000487648, + 0.006404166997526772, + 0.0071092080033849925, + 0.00674500000604894, + 0.007013124995864928, + 0.007002625003224239, + 0.006439000004320405, + 0.006668041998636909, + 0.006636874997639097, + 0.006867417003377341, + 0.006625374997383915 + ] + }, + { + "forward": 0.1173924589966191, + "prefill": 0.1141992500051856, + "decode_steps": [ + 0.019401958998059854, + 0.015227583004161716, + 0.011700292001478374, + 0.00965566599916201, + 0.008828666992485523, + 0.008020542009035125, + 0.00936858300701715, + 0.008220457995776087, + 0.008853791994624771, + 0.007932332999189384, + 0.007918125003925525, + 0.007632375010871328, + 0.007554625000921078, + 0.007472332988982089, + 0.007658916001673788, + 0.007335834001423791, + 0.007782833999954164, + 0.00774966599419713, + 0.007132041995646432, + 0.006950750001124106, + 0.007148542004870251, + 0.007514291995903477, + 0.0070442920114146546, + 0.007025124999927357, + 0.006932208008947782, + 0.006862374997581355, + 0.006992624999838881, + 0.006986625012359582, + 0.00680462499440182, + 0.006790208994061686, + 0.0071224589919438586, + 0.007701916998485103 + ] + }, + { + "forward": 0.11612099999911152, + "prefill": 0.11507641700154636, + "decode_steps": [ + 0.019134916001348756, + 0.014694625002448447, + 0.010635833998094313, + 0.009328749991254881, + 0.009933749999618158, + 0.008848333003697917, + 0.008657250000396743, + 0.008858250002958812, + 0.009080625008209608, + 0.008335833001183346, + 0.00896541700058151, + 0.00779845799843315, + 0.007554290990810841, + 0.007092542000464164, + 0.0073424169968348, + 0.007590166002046317, + 0.00724370899843052, + 0.007142082991776988, + 0.007186500006355345, + 0.007117833010852337, + 0.007957249996252358, + 0.007634540990693495, + 0.007280041012563743, + 0.007164750000811182, + 0.006928417002200149, + 0.007399041001917794, + 0.007061292009893805, + 0.007133499995688908, + 0.007204374996945262, + 0.00675974998739548, + 0.00697304199275095, + 0.006904459005454555 + ] + }, + { + "forward": 0.11477695799840149, + "prefill": 0.1138457089982694, + "decode_steps": [ + 0.019241582995164208, + 0.015940166005748324, + 0.011564041997189634, + 0.010449167006299831, + 0.009432374994503334, + 0.00894245901145041, + 0.008221958996728063, + 0.008502500000759028, + 0.007989249992533587, + 0.007850915993913077, + 0.007454957987647504, + 0.007258709010784514, + 0.007545833999756724, + 0.007406207994790748, + 0.007360833013080992, + 0.007381583011010662, + 0.00731958300457336, + 0.007158417007303797, + 0.007114792009815574, + 0.006865208008093759, + 0.0070896249962970614, + 0.0070866249880054966, + 0.007587083004182205, + 0.007377292000455782, + 0.00663229200290516, + 0.006616750004468486, + 0.006994875002419576, + 0.0068652920017484576, + 0.006835666004917584, + 0.006422499995096587, + 0.006639290993916802, + 0.006460541990236379 + ] + }, + { + "forward": 0.11615387500205543, + "prefill": 0.113462042005267, + "decode_steps": [ + 0.019185166995157488, + 0.015120042007765733, + 0.011517249993630685, + 0.010452292000991292, + 0.009574249997967854, + 0.009060458993189968, + 0.008563458002754487, + 0.008551999999326654, + 0.007863874998292886, + 0.00799045799067244, + 0.008004165996680968, + 0.007626667007571086, + 0.007817459001671523, + 0.007858708006097004, + 0.007553542003734037, + 0.007717374988715164, + 0.007543291008914821, + 0.007665917000849731, + 0.007586791994981468, + 0.007560875004855916, + 0.0075299579912098125, + 0.008183250000001863, + 0.0170612919901032, + 0.01001570800144691, + 0.007645874997251667, + 0.007324167003389448, + 0.007095666995155625, + 0.0071060000045690686, + 0.0068936250027036294, + 0.006644875000347383, + 0.006508249993203208, + 0.007179500011261553 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 115.96874999668216, + "p95": 117.24107089612517, + "min": 113.59125000308268, + "max": 117.3924589966191, + "iqr": 1.6456667508464307, + "mean": 115.83207909861812, + "cv": 0.010116539872847401 + }, + "prefill_ms": { + "count": 10, + "median": 113.79760449926835, + "p95": 115.24478575302055, + "min": 113.08904198813252, + "max": 115.3825420042267, + "iqr": 0.6658227503066882, + "mean": 113.94735030044103, + "cv": 0.006665754665677077 + }, + "decode_ms_per_step": { + "count": 320, + "median": 7.255146010720637, + "p95": 14.637545203004265, + "min": 6.133875009254552, + "max": 19.860207990859635, + "iqr": 1.1787817529693712, + "mean": 8.018427465549394, + "cv": 0.30826659583774285 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 964.5852416486338, + "p95": 1207.7207012716983, + "min": 901.401436326864, + "max": 1217.5821719592466, + "iqr": 27.457738131161705, + "mean": 1006.8679773297356, + "cv": 0.107012397174689 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 675.1041072011893, + "p95": 786.8598796013017, + "min": 644.0830809775912, + "max": 790.6819979780497, + "iqr": 13.158560828043278, + "mean": 693.8701186832625, + "cv": 0.07208205993642203 + }, + "memory": { + "peak_extra_bytes": 440410112, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "auto", + "batch": 8, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.14020087500102818 + }, + { + "forward": 0.1383600419940194 + }, + { + "forward": 0.13828095801000018 + }, + { + "forward": 0.14081187499687076 + }, + { + "forward": 0.13966891600284725 + }, + { + "forward": 0.1388743749994319 + }, + { + "forward": 0.14082058299391065 + }, + { + "forward": 0.13986870800727047 + }, + { + "forward": 0.13906499999575317 + }, + { + "forward": 0.14112470899999607 + } + ], + "forward_ms": { + "count": 10, + "median": 139.76881200505886, + "p95": 140.98785229725763, + "min": 138.28095801000018, + "max": 141.12470899999607, + "iqr": 1.7370937493979, + "mean": 139.7076041001128, + "cv": 0.007429246667248755 + }, + "memory": { + "peak_extra_bytes": 188760064, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "sdpa", + "batch": 8, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.10055408300831914, + "prefill": 0.10082095800316893, + "decode_steps": [ + 0.01053366700944025, + 0.00913024999317713, + 0.009023332997458056, + 0.00813908399140928, + 0.008547917008399963, + 0.008905291993869469, + 0.008379249993595295, + 0.00866841699462384, + 0.00779754199902527, + 0.007843542000046, + 0.00831458299944643, + 0.006876250001369044, + 0.0069635420077247545, + 0.006815874992753379, + 0.00685491701005958, + 0.0063914170023053885, + 0.006420542005798779, + 0.006838874993263744, + 0.006470042004366405, + 0.0068064590013818815, + 0.006880582994199358, + 0.006824374999268912, + 0.007941375006339513, + 0.00666912499582395, + 0.006481916003394872, + 0.006570833997102454, + 0.006447041989304125, + 0.006345499990857206, + 0.007369250000920147, + 0.006481542004621588, + 0.006937958998605609, + 0.006446917002904229 + ] + }, + { + "forward": 0.09733616700395942, + "prefill": 0.0972979590005707, + "decode_steps": [ + 0.00540158299554605, + 0.005421167006716132, + 0.005196250000153668, + 0.005143208007211797, + 0.00509674999921117, + 0.005210333998547867, + 0.005142750000231899, + 0.005140834007761441, + 0.0051439590024529025, + 0.005145832998096012, + 0.005219209007918835, + 0.005190124997170642, + 0.005300125005305745, + 0.00530945899663493, + 0.005255374999251217, + 0.005354208013159223, + 0.0055372499919030815, + 0.005479915998876095, + 0.005679542009602301, + 0.005674665997503325, + 0.005741291999584064, + 0.005866583000170067, + 0.006422125006793067, + 0.006396791999577545, + 0.0066576670069480315, + 0.00820279099571053, + 0.007386375000351109, + 0.008977292003692128, + 0.008540958006051369, + 0.008429957990301773, + 0.008505833000526763, + 0.008900082990294322 + ] + }, + { + "forward": 0.097041041997727, + "prefill": 0.09703245900163893, + "decode_steps": [ + 0.00537508400157094, + 0.0055782910058042035, + 0.005532708004466258, + 0.005229708011029288, + 0.005298624993884005, + 0.0053327089990489185, + 0.005144416994880885, + 0.005164290996617638, + 0.005150332988705486, + 0.005177917002583854, + 0.005067124991910532, + 0.005198375001782551, + 0.005307291998178698, + 0.005314291993272491, + 0.005305667000357062, + 0.005421292007667944, + 0.00550945800205227, + 0.005502041996805929, + 0.005725792012526654, + 0.005800375001854263, + 0.006612499986658804, + 0.005868874999578111, + 0.006032332996255718, + 0.0065664579888107255, + 0.006565542003954761, + 0.008394125004997477, + 0.006970458998694085, + 0.009773375000804663, + 0.008776374990702607, + 0.008865166993928142, + 0.008393750002142042, + 0.008163582999259233 + ] + }, + { + "forward": 0.1000724170007743, + "prefill": 0.10094562500307802, + "decode_steps": [ + 0.010328499993192963, + 0.008637249993626028, + 0.008646084010251798, + 0.008915124999475665, + 0.00857137500133831, + 0.008710457987035625, + 0.00908674999664072, + 0.008392292002099566, + 0.008754458001931198, + 0.008051458004047163, + 0.008055332989897579, + 0.006849416997283697, + 0.0069237090065144, + 0.006955208009458147, + 0.006888042000355199, + 0.007569958004751243, + 0.006571707999682985, + 0.00653058300667908, + 0.006843541996204294, + 0.006927417009137571, + 0.0064641670032870024, + 0.006923042004927993, + 0.006495624998933636, + 0.006784583994885907, + 0.0065418750018579885, + 0.006540332993608899, + 0.0064464590104762465, + 0.006355915989843197, + 0.006397999997716397, + 0.006516749999718741, + 0.006706916989060119, + 0.006493458000477403 + ] + }, + { + "forward": 0.09725129099388141, + "prefill": 0.09730966600182, + "decode_steps": [ + 0.0057882080000126734, + 0.005622874989057891, + 0.005222749998210929, + 0.006088249996537343, + 0.005138792010257021, + 0.005406458003562875, + 0.005626540994853713, + 0.005176332997507416, + 0.005738334002671763, + 0.005717792009818368, + 0.005329457999323495, + 0.005806791989016347, + 0.005892667002626695, + 0.005777667000074871, + 0.006026792005286552, + 0.0057231249957112595, + 0.005923749995417893, + 0.0059214159991825, + 0.006714415998430923, + 0.006637250000494532, + 0.008307374999276362, + 0.007857791002606973, + 0.008663917004014365, + 0.008390417002374306, + 0.009675124994828366, + 0.009870584006421268, + 0.008255416993051767, + 0.008342916989931837, + 0.008019540997338481, + 0.00769587499962654, + 0.007760249995044433, + 0.007721000001765788 + ] + }, + { + "forward": 0.09830033300386276, + "prefill": 0.0971948340011295, + "decode_steps": [ + 0.005588125000940636, + 0.005306124992785044, + 0.005196416997932829, + 0.005171791999600828, + 0.005165124995983206, + 0.005183999994187616, + 0.005149958000401966, + 0.005127166004967876, + 0.005121499998494983, + 0.0056570000015199184, + 0.00523599999723956, + 0.0055878749990370125, + 0.005428874996141531, + 0.005239709003944881, + 0.005524749998585321, + 0.0055362919956678525, + 0.0055307080037891865, + 0.005792707990622148, + 0.005783417000202462, + 0.005858792006620206, + 0.006508834005217068, + 0.006397874996764585, + 0.006820416994742118, + 0.006536834000144154, + 0.0075741670007118955, + 0.007110125006875023, + 0.008999792000395246, + 0.008973541989689693, + 0.009403249991009943, + 0.008453290996840224, + 0.007991666992893443, + 0.007400874994345941 + ] + }, + { + "forward": 0.10103150000213645, + "prefill": 0.10101316700456664, + "decode_steps": [ + 0.00945645799220074, + 0.008495624992065132, + 0.008361625004909001, + 0.00838708299852442, + 0.008584750001318753, + 0.008427916996879503, + 0.008726332991500385, + 0.008271707993117161, + 0.008350541000254452, + 0.008378624988836236, + 0.008380915998714045, + 0.00814479099062737, + 0.008419916004640982, + 0.007844332998502068, + 0.007726416995865293, + 0.007134416999178939, + 0.006894667007145472, + 0.007657624999410473, + 0.007715875006397255, + 0.006919041989021935, + 0.006984541003475897, + 0.006902875000378117, + 0.006958750003832392, + 0.006640582985710353, + 0.006368333997670561, + 0.007076915993820876, + 0.006185790989547968, + 0.006044125009793788, + 0.005968124998616986, + 0.005593249996309169, + 0.0054472909978358075, + 0.0055040840088622645 + ] + }, + { + "forward": 0.09914579200267326, + "prefill": 0.09742712500155903, + "decode_steps": [ + 0.008321916990098543, + 0.006922708009369671, + 0.006679250000161119, + 0.006488708007964306, + 0.00662558400654234, + 0.006557833999977447, + 0.006450542001402937, + 0.0065123749955091625, + 0.00716183299664408, + 0.006600416003493592, + 0.006519875008962117, + 0.006887042007292621, + 0.006491834006737918, + 0.006488083003205247, + 0.006522458003018983, + 0.0064070000080391765, + 0.006525500008137897, + 0.006524709009681828, + 0.006487499995273538, + 0.0064730830054031685, + 0.006448250001994893, + 0.006457458992372267, + 0.006419042008928955, + 0.0065271659987047315, + 0.006440375000238419, + 0.006702874990878627, + 0.00650441600009799, + 0.006401916005415842, + 0.006453750000218861, + 0.0064050830114865676, + 0.006775917005143128, + 0.006422457998269238 + ] + }, + { + "forward": 0.0975949169951491, + "prefill": 0.09722629201132804, + "decode_steps": [ + 0.005547292006667703, + 0.005353750006179325, + 0.00526870800240431, + 0.005219707993092015, + 0.005228583991993219, + 0.005176666993065737, + 0.0051376670016907156, + 0.005294874994433485, + 0.005260584002826363, + 0.005175458005396649, + 0.005197749997023493, + 0.005592416011495516, + 0.005325916994479485, + 0.005353917003958486, + 0.005288665997795761, + 0.005466792004881427, + 0.005516790988622233, + 0.005454541998915374, + 0.005713666992960498, + 0.005732249992433935, + 0.005925459001446143, + 0.005851790992892347, + 0.0059189169987803325, + 0.0064571249968139455, + 0.006485249992692843, + 0.008288624987471849, + 0.008228082995628938, + 0.008563625000533648, + 0.008438250006292947, + 0.008217833994422108, + 0.00817812500463333, + 0.007831666007405147 + ] + }, + { + "forward": 0.1035759579972364, + "prefill": 0.10303937501157634, + "decode_steps": [ + 0.006592958001419902, + 0.0066290830000070855, + 0.006221083996933885, + 0.006698750003124587, + 0.006239458001800813, + 0.006485082994913682, + 0.006431750007322989, + 0.0062245840090326965, + 0.006509540995466523, + 0.006514874999993481, + 0.006074250006349757, + 0.0062308749911608174, + 0.006218374997843057, + 0.006210749997990206, + 0.006163291996927001, + 0.00616049999371171, + 0.006122459002654068, + 0.006100625003455207, + 0.006221165996976197, + 0.0061472919915104285, + 0.006186333994264714, + 0.006117624987382442, + 0.0062472499994328246, + 0.006073333992389962, + 0.006107292007072829, + 0.006121915997937322, + 0.006717707990901545, + 0.006207291997270659, + 0.006106625005486421, + 0.0060249579983064905, + 0.006086125009460375, + 0.006168249994516373 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 98.72306250326801, + "p95": 102.43095189944142, + "min": 97.041041997727, + "max": 103.5759579972364, + "iqr": 3.0328120046760887, + "mean": 99.19035000057193, + "cv": 0.02136474752164218 + }, + "prefill_ms": { + "count": 10, + "median": 97.36839550168952, + "p95": 102.12758140842197, + "min": 97.03245900163893, + "max": 103.03937501157634, + "iqr": 3.670249494462041, + "mean": 98.93074600404361, + "cv": 0.022833915639459752 + }, + "decode_ms_per_step": { + "count": 320, + "median": 6.467104503826704, + "p95": 8.866912793746451, + "min": 5.067124991910532, + "max": 10.53366700944025, + "iqr": 1.7406147489964496, + "mean": 6.656221902630932, + "cv": 0.18220666817459943 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1243.740460953919, + "p95": 1307.0469683293825, + "min": 1075.8412212420387, + "max": 1308.1865072570795, + "iqr": 181.48010251931692, + "mean": 1209.3473558281007, + "cv": 0.08128995208637707 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 836.077936387297, + "p95": 873.3626995022773, + "min": 755.2368834873546, + "max": 873.9675827795535, + "iqr": 92.88068646274644, + "mean": 823.4523927902483, + "cv": 0.06009637824978083 + }, + "memory": { + "peak_extra_bytes": 188760064, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "reference", + "batch": 8, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.39761416699911933, + "prefill": 0.3941425410012016, + "decode_steps": [ + 0.02121616700605955, + 0.019306499991216697, + 0.01374125000438653, + 0.01240358299401123, + 0.01139891600178089, + 0.010912249999819323, + 0.010754916991572827, + 0.010423582993098535, + 0.010260874987579882, + 0.011799958010669798, + 0.010881291003897786, + 0.009943875003955327, + 0.009829041009652428, + 0.009664042008807883, + 0.00975549999566283, + 0.009740916997543536, + 0.009413167004822753, + 0.009820334002142772, + 0.00958041699777823, + 0.009383667013025843, + 0.009901583005557768, + 0.010368000002927147, + 0.01166362498770468, + 0.0102276669931598, + 0.009723082999698818, + 0.008996792006655596, + 0.008823625001241453, + 0.0086235419876175, + 0.00919399999838788, + 0.008928249997552484, + 0.009039291995577514, + 0.008837291999952868 + ] + }, + { + "forward": 0.386740625006496, + "prefill": 0.38389570800063666, + "decode_steps": [ + 0.02166229199792724, + 0.017014209006447345, + 0.012846375000663102, + 0.011704458011081442, + 0.011296249998849817, + 0.011083833000157028, + 0.010477500007255003, + 0.010441749996971339, + 0.010313083010260016, + 0.010152250004466623, + 0.01015533399186097, + 0.010021458001574501, + 0.009984582997276448, + 0.009971207997296005, + 0.009821792002185248, + 0.009871542002656497, + 0.011164457988343202, + 0.009922791999997571, + 0.008982957995613106, + 0.009026957995956764, + 0.008638833998702466, + 0.008592749989475124, + 0.008640250001917593, + 0.008662833002745174, + 0.008490167005220428, + 0.008568083008867688, + 0.009075875001144595, + 0.008730541987461038, + 0.00856762500188779, + 0.008505958001478575, + 0.008757042000070214, + 0.010409291004179977 + ] + }, + { + "forward": 0.37898491699888837, + "prefill": 0.3790699160017539, + "decode_steps": [ + 0.010043625006801449, + 0.008811583000351675, + 0.008609750002506189, + 0.008553374995244667, + 0.008546874989406206, + 0.008545916003640741, + 0.008467916006338783, + 0.008575957996072248, + 0.0089125420054188, + 0.008073750010225922, + 0.009692957988590933, + 0.008884416005457751, + 0.008574291001423262, + 0.008227458005421795, + 0.008511833002557978, + 0.008565916999941692, + 0.00853062501118984, + 0.008284166004159488, + 0.008485582991852425, + 0.008808124999632128, + 0.008826624994981103, + 0.008522124990122393, + 0.008608624993939884, + 0.008448624997981824, + 0.00836800000979565, + 0.0085491249919869, + 0.008372666998184286, + 0.00844337500166148, + 0.008503832999849692, + 0.00853741700120736, + 0.00852241599932313, + 0.008479791998979636 + ] + }, + { + "forward": 0.3946215830073925, + "prefill": 0.38620774999435525, + "decode_steps": [ + 0.020276791998185217, + 0.017945332991075702, + 0.012453874995117076, + 0.011464666007668711, + 0.01223375000699889, + 0.010703958992962725, + 0.010580083006061614, + 0.011058666990720667, + 0.010247250000247732, + 0.009786082999198698, + 0.009777917002793401, + 0.009780084001249634, + 0.010489790991414338, + 0.009788291994482279, + 0.009573333998559974, + 0.009673874999862164, + 0.00967629200022202, + 0.009863917002803646, + 0.00963270801003091, + 0.010477792005985975, + 0.010601916001178324, + 0.008976500001153909, + 0.008844249998219311, + 0.008617125000455417, + 0.008684540996910073, + 0.00855174999742303, + 0.008379999999306165, + 0.009592083006282337, + 0.009164791990770027, + 0.010266582990880124, + 0.010419333993922919, + 0.00921758299227804 + ] + }, + { + "forward": 0.3856578749982873, + "prefill": 0.3809957920020679, + "decode_steps": [ + 0.010338542007957585, + 0.00866449999739416, + 0.00850766699295491, + 0.008368082999368198, + 0.00889587499841582, + 0.008819416994811036, + 0.008640790998470038, + 0.008412790994043462, + 0.008606667004642077, + 0.008773166991886683, + 0.008308624994242564, + 0.008541000002878718, + 0.007962292002048343, + 0.008669625007314608, + 0.008501417003571987, + 0.00842262500373181, + 0.009162084010313265, + 0.008524666001903825, + 0.008447166997939348, + 0.008524707998731174, + 0.008421792008448392, + 0.008410958005697466, + 0.008395041004405357, + 0.008678041995153762, + 0.008332166995387524, + 0.00835608299530577, + 0.00840708299074322, + 0.00926337500277441, + 0.009168832999421284, + 0.009553165989927948, + 0.008523208001861349, + 0.008404541993513703 + ] + }, + { + "forward": 0.3791535409982316, + "prefill": 0.4020684999995865, + "decode_steps": [ + 0.014954625003156252, + 0.011698458998580463, + 0.011937791990931146, + 0.010605957999359816, + 0.011587583008804359, + 0.011069667001720518, + 0.00976262500626035, + 0.009302750011556782, + 0.010077874991111457, + 0.009721249996800907, + 0.009819583996431902, + 0.010584791001747362, + 0.009257625002646819, + 0.010289500001817942, + 0.009787083996343426, + 0.010344333000830375, + 0.010773165995487943, + 0.009336917006294243, + 0.009946499994839542, + 0.00940229200932663, + 0.009864583989838138, + 0.010900792010943405, + 0.010021333990152925, + 0.010835999986738898, + 0.010120875012944452, + 0.009857415992883034, + 0.010755082999821752, + 0.009968332989956252, + 0.00983979100419674, + 0.009641666998504661, + 0.008908749994589016, + 0.009110708007938229 + ] + }, + { + "forward": 0.3947240830020746, + "prefill": 0.38004845900286455, + "decode_steps": [ + 0.020712124998681247, + 0.018297500006156042, + 0.01419095900200773, + 0.011839083002996631, + 0.010762584002804942, + 0.00981629099987913, + 0.009973624997655861, + 0.009693625004729256, + 0.009601167010259815, + 0.009791832999326289, + 0.009668332990258932, + 0.009512750009889714, + 0.010481208009878173, + 0.0198432920005871, + 0.011033250004402362, + 0.009593042006599717, + 0.009205000009387732, + 0.009355957998195663, + 0.00970745799713768, + 0.009087208003620617, + 0.009176250008749776, + 0.009051124987308867, + 0.009336457995232195, + 0.009397583999088965, + 0.009917709001456387, + 0.009246040994185023, + 0.009231124990037642, + 0.009094875000300817, + 0.009187458999804221, + 0.009254749995307066, + 0.008996917007607408, + 0.008873208993463777 + ] + }, + { + "forward": 0.3867412920080824, + "prefill": 0.3823616250010673, + "decode_steps": [ + 0.020938209010637365, + 0.01772966700082179, + 0.014685624992125668, + 0.012302249990170822, + 0.011652542001684196, + 0.012468124987208284, + 0.010692374999052845, + 0.0106234169943491, + 0.01003012499131728, + 0.009961957999621518, + 0.009802000000490807, + 0.00963775000127498, + 0.009686792007414624, + 0.009653625005739741, + 0.009836416007601656, + 0.009902957986923866, + 0.010768250009277835, + 0.009660833995440044, + 0.009658582988777198, + 0.009895958006381989, + 0.00903550001385156, + 0.009086249992833473, + 0.008669542003190145, + 0.008371375006390736, + 0.008536541994544677, + 0.008741459008888341, + 0.009588333006831817, + 0.008674540993524715, + 0.008606417002738453, + 0.008669250004459172, + 0.008788166000158526, + 0.008472083005472086 + ] + }, + { + "forward": 0.38410599999770056, + "prefill": 0.3790021250024438, + "decode_steps": [ + 0.023336832993663847, + 0.017855250000138767, + 0.013708208003663458, + 0.012617665997822769, + 0.011732708007912152, + 0.011587125001824461, + 0.010827582998899743, + 0.010024582996265963, + 0.009961584000848234, + 0.010805792000610381, + 0.01074566699389834, + 0.010079250001581386, + 0.01015612499031704, + 0.009726875010528602, + 0.010294499996234663, + 0.010167583008296788, + 0.009686959005193785, + 0.009154832994681783, + 0.008856708998791873, + 0.009052499997778796, + 0.009200291009619832, + 0.009351875007268973, + 0.009453042002860457, + 0.008868125005392358, + 0.00939700000162702, + 0.009089459010283463, + 0.009609999993699603, + 0.009199958003591746, + 0.009232124997652136, + 0.00939608400221914, + 0.00928958400618285, + 0.009171207988401875 + ] + }, + { + "forward": 0.3936781670054188, + "prefill": 0.3956650409963913, + "decode_steps": [ + 0.015067209009430371, + 0.012678333005169407, + 0.012336250001681037, + 0.011500416992930695, + 0.010742208003648557, + 0.0112983750004787, + 0.01066262500535231, + 0.009998416004236788, + 0.009853166993707418, + 0.010362582994275726, + 0.009511500000371598, + 0.010502958000870422, + 0.009701208997284994, + 0.009357165996334516, + 0.010225749996607192, + 0.010527792008360848, + 0.009326750005129725, + 0.009620666998671368, + 0.009324541999376379, + 0.009669166000094265, + 0.008655041994643398, + 0.009348667008453049, + 0.00839270900178235, + 0.009171875004540198, + 0.008679708000272512, + 0.008017875006771646, + 0.008483166006044485, + 0.009167833006358705, + 0.00901137500477489, + 0.008751250003115274, + 0.009335583003121428, + 0.009195999999064952 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 386.7409585072892, + "p95": 396.3136292004492, + "min": 378.98491699888837, + "max": 397.61416699911933, + "iqr": 9.891760259051807, + "mean": 388.20222500216914, + "cv": 0.017101637877259636 + }, + "prefill_ms": { + "count": 10, + "median": 383.128666500852, + "p95": 399.18694344814867, + "min": 379.0021250024438, + "max": 402.0684999995865, + "iqr": 11.873550996824633, + "mean": 386.3457457002369, + "cv": 0.020990400922271904 + }, + "decode_ms_per_step": { + "count": 320, + "median": 9.615333496185485, + "p95": 14.215692301513633, + "min": 7.962292002048343, + "max": 23.336832993663847, + "iqr": 1.5608019966748543, + "mean": 10.105516268959036, + "cv": 0.2307111746309839 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 768.3474715871236, + "p95": 926.2251160777829, + "min": 742.9830166324668, + "max": 927.9084475507702, + "iqr": 44.73299499177301, + "mean": 796.706762742411, + "cv": 0.08834456875297364 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 356.09477460849564, + "p95": 390.0507332074588, + "min": 346.55497322865523, + "max": 390.8640398135674, + "iqr": 4.228122851440901, + "mean": 361.27734753912074, + "cv": 0.043054038112138704 + }, + "memory": { + "peak_extra_bytes": 1686126592, + "method": "mps_live_allocated_peak_per_op" + } + } + ] +} diff --git a/experiments/records/attention/20261022T034304Z_attention_mps_float16_0502118.json b/experiments/records/attention/20261022T034304Z_attention_mps_float16_0502118.json new file mode 100644 index 0000000..58aec25 --- /dev/null +++ b/experiments/records/attention/20261022T034304Z_attention_mps_float16_0502118.json @@ -0,0 +1,14061 @@ +{ + "record_version": 1, + "created_utc": "20261022T034304Z", + "experiment": "attention_backends", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "mps", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "0502118a1a0af31c8480f6b499d9347916330a79", + "branch": "experiments/attention-kv-audit", + "dirty": false + }, + "command": "python scripts/benchmark_attention.py --device mps --dtype float16" + }, + "settings": { + "dtype": "float16", + "position": "learned", + "backends": [ + "auto", + "sdpa", + "reference" + ], + "seq_lens": [ + 128, + 256, + 512, + 1024, + 2048 + ], + "batch_sizes": [ + 1, + 4, + 8 + ], + "decode_tokens": 32, + "warmup": 3, + "repeats": 10, + "seed": 0, + "weights": "seeded random initialization (timing does not depend on trained values)", + "model_config": { + "vocab_size": 256, + "context_length": 2080, + "embedding_dim": 192, + "num_layers": 4, + "num_heads": 6, + "dropout": 0.0, + "positional_encoding": "learned", + "norm": "rmsnorm", + "activation": "swiglu", + "rope_theta": 10000.0 + }, + "inference_mode": true + }, + "results": [ + { + "backend": "auto", + "batch": 1, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.0019347080087754875 + }, + { + "forward": 0.0020462910033529624 + }, + { + "forward": 0.001927375007653609 + }, + { + "forward": 0.0018364999996265396 + }, + { + "forward": 0.002185499994084239 + }, + { + "forward": 0.002184500001021661 + }, + { + "forward": 0.0019892499985871837 + }, + { + "forward": 0.0020790830021724105 + }, + { + "forward": 0.002271750010550022 + }, + { + "forward": 0.0018959169974550605 + } + ], + "forward_ms": { + "count": 10, + "median": 2.017770500970073, + "p95": 2.2329375031404197, + "min": 1.8364999996265396, + "max": 2.271750010550022, + "iqr": 0.2289374933752697, + "mean": 2.0350874023279175, + "cv": 0.07056477259072569 + }, + "memory": { + "peak_extra_bytes": 738304, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "sdpa", + "batch": 1, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.0016258749965345487, + "prefill": 0.0020619580027414486, + "decode_steps": [ + 0.0017376249888911843, + 0.0015634169976692647, + 0.001541542005725205, + 0.0015565840003546327, + 0.0016562919918214902, + 0.0015369579923572019, + 0.0015106250066310167, + 0.0015484159957850352, + 0.001529333007056266, + 0.0015233750018524006, + 0.001584999990882352, + 0.0015762919938424602, + 0.001699082989944145, + 0.0015112499968381599, + 0.001582875003805384, + 0.0015296670026145875, + 0.0016993330063996837, + 0.0015752499894006178, + 0.00154208300227765, + 0.0015480829897569492, + 0.001533333008410409, + 0.0017507080046925694, + 0.0015404590085381642, + 0.0015800409892108291, + 0.0015448750054929405, + 0.0015227500116452575, + 0.0015450420032721013, + 0.002281459004734643, + 0.0015343750128522515, + 0.0015227500116452575, + 0.0015631659916834906, + 0.0016448329988634214 + ] + }, + { + "forward": 0.001883083998109214, + "prefill": 0.0018742079992080107, + "decode_steps": [ + 0.0018904169992310926, + 0.002022415996179916, + 0.0019375420088181272, + 0.001816791991586797, + 0.00167012499878183, + 0.0018684999959077686, + 0.0018560839962447062, + 0.0017466249992139637, + 0.0015227920084726065, + 0.0015998750022845343, + 0.0015168329991865903, + 0.001531415997305885, + 0.001512417002231814, + 0.001475291996030137, + 0.0015079170116223395, + 0.0015260419895639643, + 0.0015482919989153743, + 0.0015318330115405843, + 0.0016431669937446713, + 0.0015416670066770166, + 0.0015474170068046078, + 0.0015570839896099642, + 0.0015193750004982576, + 0.0015391659981105477, + 0.0014821660006418824, + 0.001684041999396868, + 0.0015665000100852922, + 0.001539832999696955, + 0.0017094999930122867, + 0.0015802499983692542, + 0.0017848749994300306, + 0.0015847089962335303 + ] + }, + { + "forward": 0.0016761250008130446, + "prefill": 0.0016784170002210885, + "decode_steps": [ + 0.001798458004486747, + 0.0015764580020913854, + 0.001564417005283758, + 0.0015785840078024194, + 0.0015721670060884207, + 0.0016610830061836168, + 0.0016834999987622723, + 0.00156974999117665, + 0.0015989579987945035, + 0.0016012919950298965, + 0.0016137920028995723, + 0.0016199999954551458, + 0.0016709160117898136, + 0.0017707500082906336, + 0.0016435419966001064, + 0.0015826250019017607, + 0.0015303749969461933, + 0.001630833008675836, + 0.0015682919911341742, + 0.0019621669925982133, + 0.001606709003681317, + 0.0015602500061504543, + 0.0015650000132154673, + 0.0016008749953471124, + 0.0016053750005085021, + 0.0016634169878670946, + 0.001570124994032085, + 0.0015143329947022721, + 0.0017001249943859875, + 0.0015640409983461723, + 0.0017070000030798838, + 0.0017241669993381947 + ] + }, + { + "forward": 0.0024410420010099187, + "prefill": 0.0016886249941308051, + "decode_steps": [ + 0.0016252499917754903, + 0.001602709002327174, + 0.0015443750016856939, + 0.0015506250056205317, + 0.001524459003121592, + 0.0014903750125085935, + 0.00152658399019856, + 0.0016384170012315735, + 0.001523040991742164, + 0.0015289579896489158, + 0.0016254160000244156, + 0.0015429589984705672, + 0.001580833995831199, + 0.0015462080045836046, + 0.0015826250019017607, + 0.0015278750070137903, + 0.0016382080066250637, + 0.0015268330025719479, + 0.00154824998753611, + 0.0015556670114165172, + 0.0015570409887004644, + 0.0015681669901823625, + 0.0015333750052377582, + 0.001574875001097098, + 0.0016128749994095415, + 0.0015926659980323166, + 0.001571832995978184, + 0.0015592500130878761, + 0.0015630419948138297, + 0.0015221250068861991, + 0.0015259999927366152, + 0.0015248330018948764 + ] + }, + { + "forward": 0.0020559169934131205, + "prefill": 0.0020623749878723174, + "decode_steps": [ + 0.0018898749985964969, + 0.001817583994125016, + 0.0018820409895852208, + 0.0018754169868770987, + 0.0018629170081112534, + 0.0018812089983839542, + 0.0016055419982876629, + 0.0016287080070469528, + 0.001720165993901901, + 0.0015419580013258383, + 0.0016214170027524233, + 0.0015930420049699023, + 0.0016220840043388307, + 0.0016294170054607093, + 0.0015582500054733828, + 0.0015565420035272837, + 0.0015513329999521375, + 0.0016500829951837659, + 0.001553708003484644, + 0.00155929199536331, + 0.0016524999955436215, + 0.0015443339943885803, + 0.0015467909979633987, + 0.001552000001538545, + 0.0016207909939112142, + 0.0015550420066574588, + 0.0016745829925639555, + 0.0015630000125383958, + 0.0016286669997498393, + 0.001553374997456558, + 0.001559333992190659, + 0.0015853750082897022 + ] + }, + { + "forward": 0.0018764999986160547, + "prefill": 0.0019805410120170563, + "decode_steps": [ + 0.001826667008572258, + 0.001987541007110849, + 0.0017447090067435056, + 0.0017658330034464598, + 0.0018350419995840639, + 0.0017924580024555326, + 0.0019413749978411943, + 0.0017841249937191606, + 0.0017456250061513856, + 0.001783625004463829, + 0.0017701670003589243, + 0.001757291000103578, + 0.0018062910094158724, + 0.0015535419952357188, + 0.0015511660021729767, + 0.0015582079940941185, + 0.0016284160083159804, + 0.001584458994329907, + 0.001682916990830563, + 0.0015774999919813126, + 0.0015909579960862175, + 0.0015567499940516427, + 0.0015667080006096512, + 0.0015464169991901144, + 0.0016605839919066057, + 0.0015612500137649477, + 0.001596083995536901, + 0.0015531669923802838, + 0.0015618330071447417, + 0.001579625008162111, + 0.001570292006363161, + 0.001523832994280383 + ] + }, + { + "forward": 0.0017630410002311692, + "prefill": 0.001762041007168591, + "decode_steps": [ + 0.0016164160042535514, + 0.0015462500014109537, + 0.0016132919990923256, + 0.0015474579995498061, + 0.001566124992677942, + 0.0016942499933065847, + 0.0015110829990589991, + 0.0015929580113152042, + 0.0015392079949378967, + 0.0015400419943034649, + 0.0016499159974046052, + 0.0015511660021729767, + 0.0015832920034881681, + 0.0015486670017708093, + 0.0016053339932113886, + 0.0015445830067619681, + 0.0015378750103991479, + 0.0015295000048354268, + 0.0015422089927596971, + 0.0015424579905811697, + 0.0016018750029616058, + 0.0015629999979864806, + 0.0015640410128980875, + 0.0015467499906662852, + 0.0016261669952655211, + 0.001557374998810701, + 0.0016776249976828694, + 0.0015399590047309175, + 0.0016503749939147383, + 0.001560999997309409, + 0.0015727909922134131, + 0.0015488329954678193 + ] + }, + { + "forward": 0.0018847079918486997, + "prefill": 0.0017176249966723844, + "decode_steps": [ + 0.0018376250081928447, + 0.0018567090010037646, + 0.0016242919955402613, + 0.0015694169997004792, + 0.0015239170024869964, + 0.0015146249934332445, + 0.0015301670064218342, + 0.00153929200314451, + 0.0015271250013029203, + 0.0016377920110244304, + 0.0015842499997233972, + 0.0015110420063138008, + 0.0017227499920409173, + 0.0015287499991245568, + 0.0015844580047996715, + 0.0017187079938594252, + 0.0015789589961059391, + 0.0015610830014338717, + 0.001691042009042576, + 0.0015612919960403815, + 0.0016730000061215833, + 0.0015587079979013652, + 0.001515666997875087, + 0.0015653749869670719, + 0.0015710830048192292, + 0.0015739579976070672, + 0.0015595409931847826, + 0.0016867079975781962, + 0.0015072499954840168, + 0.0016411249962402508, + 0.0016395410057157278, + 0.0015242920053424314 + ] + }, + { + "forward": 0.001953541999682784, + "prefill": 0.001991166005609557, + "decode_steps": [ + 0.001830833003623411, + 0.0017385409882990643, + 0.001741500003845431, + 0.0017448330036131665, + 0.0016989579889923334, + 0.0015397919923998415, + 0.0015263330133166164, + 0.0015605420048814267, + 0.0015342499973485246, + 0.0015717919886810705, + 0.0015485839976463467, + 0.0015688749990658835, + 0.0015237920015351847, + 0.0015260829968610778, + 0.0016367079952033237, + 0.0015442090079886839, + 0.0015255420003086329, + 0.0015075420087669045, + 0.0015975419955793768, + 0.0016088749980553985, + 0.0016936659958446398, + 0.0015524160116910934, + 0.001560000004246831, + 0.0015553749981336296, + 0.001584792000357993, + 0.0015500829904340208, + 0.0015513749967794865, + 0.00172079200274311, + 0.0015724169934401289, + 0.001560083997901529, + 0.0015530840028077364, + 0.00159891699149739 + ] + }, + { + "forward": 0.0016499159974046052, + "prefill": 0.0017289999959757552, + "decode_steps": [ + 0.0017098749958677217, + 0.0015438749978784472, + 0.0015233330050250515, + 0.0015183750074356794, + 0.00162958400323987, + 0.001491208007792011, + 0.001553333000629209, + 0.0016419169987784699, + 0.0015226660034386441, + 0.0015555829886579886, + 0.0016202499973587692, + 0.0015192910068435594, + 0.0014940830005798489, + 0.0016070419951574877, + 0.0015435000095749274, + 0.0015364159917226061, + 0.0015239999920595437, + 0.001494667012593709, + 0.001865500002168119, + 0.0015261660009855404, + 0.0015370829933090135, + 0.0015359999961219728, + 0.0016784579929662868, + 0.0015442090079886839, + 0.0017227909993380308, + 0.0015359170065494254, + 0.0015816660015843809, + 0.00153070900705643, + 0.0015297079953597859, + 0.0015224579983623698, + 0.001535082992631942, + 0.0015303749969461933 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 1.8797919983626343, + "p95": 2.267735747591359, + "min": 1.6258749965345487, + "max": 2.4410420010099187, + "iqr": 0.23847949705668725, + "mean": 1.880974997766316, + "cv": 0.1282709632515728 + }, + "prefill_ms": { + "count": 10, + "median": 1.8181245031883009, + "p95": 2.0621873445634264, + "min": 1.6784170002210885, + "max": 2.0623749878723174, + "iqr": 0.26804101071320474, + "mean": 1.8545956001617014, + "cv": 0.08482907223783481 + }, + "decode_ms_per_step": { + "count": 320, + "median": 1.5666040053474717, + "p95": 1.838547957595438, + "min": 1.475291996030137, + "max": 2.281459004734643, + "iqr": 0.09681274968897924, + "mean": 1.6085576874047547, + "cv": 0.06661004265143418 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 625.3200592912619, + "p95": 639.2762322214465, + "min": 597.634212493036, + "max": 640.8234582220934, + "iqr": 20.90838534437478, + "mean": 621.9626471623116, + "cv": 0.022600301338990395 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 602.0804232198036, + "p95": 618.1985812049909, + "min": 576.3169844203057, + "max": 619.8622260787462, + "iqr": 20.065934373917116, + "mean": 600.3531655798565, + "cv": 0.023427013580951894 + }, + "memory": { + "peak_extra_bytes": 738304, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "reference", + "batch": 1, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.00569133298995439, + "prefill": 0.0022516669996548444, + "decode_steps": [ + 0.004563124995911494, + 0.004408708002301864, + 0.00421295799606014, + 0.004194583991193213, + 0.004072583993547596, + 0.004182000004220754, + 0.004172291999566369, + 0.0040173750021494925, + 0.003951417005737312, + 0.004147375002503395, + 0.004101667000213638, + 0.004104082996491343, + 0.003834666989860125, + 0.00412791600683704, + 0.004143290992942639, + 0.004197208007099107, + 0.0041975839994847775, + 0.004521584007306956, + 0.005119499997817911, + 0.004334333003498614, + 0.004340875006164424, + 0.004163666002568789, + 0.004304083995521069, + 0.004051708005135879, + 0.004104000006918795, + 0.004190749998087995, + 0.004182374992524274, + 0.003887582992319949, + 0.004876624996541068, + 0.004317625003750436, + 0.004200124996714294, + 0.0041885829996317625 + ] + }, + { + "forward": 0.005513500000233762, + "prefill": 0.0023301670007640496, + "decode_steps": [ + 0.004328583003371023, + 0.00418737500149291, + 0.004208041995298117, + 0.004399292010930367, + 0.004146957988268696, + 0.004014874997665174, + 0.003944791998947039, + 0.004123791994061321, + 0.004145708997384645, + 0.003970375008066185, + 0.004113041999517009, + 0.004040291009005159, + 0.004141040990361944, + 0.0038768329977756366, + 0.003928125006495975, + 0.004176083006314002, + 0.004090625006938353, + 0.004302624991396442, + 0.004045417008455843, + 0.004552999991574325, + 0.0042488339968258515, + 0.00425916699168738, + 0.004196832989691757, + 0.004156791997957043, + 0.0040901250031311065, + 0.00418287499633152, + 0.0041473339952062815, + 0.004162457989878021, + 0.004106249994947575, + 0.003990500001236796, + 0.004153334011789411, + 0.004146958002820611 + ] + }, + { + "forward": 0.004801249990123324, + "prefill": 0.0023520420072600245, + "decode_steps": [ + 0.0041695420077303424, + 0.004173249995801598, + 0.004092292001587339, + 0.004004792004707269, + 0.003928791993530467, + 0.0040432500099996105, + 0.004023333007353358, + 0.00410362500406336, + 0.0041047920094570145, + 0.004055083001730964, + 0.004078999991179444, + 0.003905916993971914, + 0.004076375000295229, + 0.004171249995124526, + 0.004347833004430868, + 0.004063999993377365, + 0.0042876250081462786, + 0.004365834000054747, + 0.004265791998477653, + 0.004179124996881001, + 0.003982249996624887, + 0.003933250001864508, + 0.004044083005283028, + 0.004217875000904314, + 0.004111834001378156, + 0.004046457994263619, + 0.004110790992854163, + 0.004211167004541494, + 0.004148834006628022, + 0.004143707992625423, + 0.0040539159963373095, + 0.004105750005692244 + ] + }, + { + "forward": 0.005785417000879534, + "prefill": 0.0024042500008363277, + "decode_steps": [ + 0.004666499997256324, + 0.004759875009767711, + 0.004301833003410138, + 0.004380667000077665, + 0.004332541997428052, + 0.004133875001571141, + 0.004043291002744809, + 0.004181666998192668, + 0.0036865419970126823, + 0.004100791993550956, + 0.0040827079938026145, + 0.004190499996184371, + 0.004135834009503014, + 0.0041869169945130125, + 0.004104666993953288, + 0.004044375004014, + 0.00398866600880865, + 0.004100875012227334, + 0.004227375000482425, + 0.004093249997822568, + 0.0039410000026691705, + 0.004133333000936545, + 0.00414341599389445, + 0.00407279199862387, + 0.00387804199999664, + 0.004196875001071021, + 0.003990249999333173, + 0.003982082998845726, + 0.004194250010186806, + 0.004323000001022592, + 0.004291332996217534, + 0.003877416005707346 + ] + }, + { + "forward": 0.005020208991481923, + "prefill": 0.0022570419969270006, + "decode_steps": [ + 0.004322750013670884, + 0.00394345801032614, + 0.004426750005222857, + 0.00428999999712687, + 0.003697499996633269, + 0.004106832988327369, + 0.0038146670121932402, + 0.00405095900350716, + 0.004202415992040187, + 0.004194333989289589, + 0.004174166999291629, + 0.004054959004861303, + 0.004070374998264015, + 0.004114124996704049, + 0.00412362499628216, + 0.004051542011438869, + 0.004042000000481494, + 0.004031958000268787, + 0.004012584002339281, + 0.004088250003405847, + 0.00407445800374262, + 0.003966417003539391, + 0.004114167008083314, + 0.004082915998878889, + 0.00411970799905248, + 0.004098624995094724, + 0.004133499998715706, + 0.00395354199281428, + 0.004201500007184222, + 0.004579083004500717, + 0.004263417009497061, + 0.004149291999056004 + ] + }, + { + "forward": 0.00484945900097955, + "prefill": 0.0025767500046640635, + "decode_steps": [ + 0.004263541995896958, + 0.004087499997694977, + 0.0039268750115297735, + 0.004070416995091364, + 0.004132457994273864, + 0.004013042009319179, + 0.0038124159909784794, + 0.004085082997335121, + 0.004133458001888357, + 0.00404158300079871, + 0.003982709007686935, + 0.004014207996078767, + 0.004071125003974885, + 0.004015333004645072, + 0.003975625004386529, + 0.004029832998639904, + 0.003942041992559098, + 0.0040905000059865415, + 0.004107957996893674, + 0.0041621249984018505, + 0.004195417001028545, + 0.004391000009491108, + 0.004044416011311114, + 0.00433595800132025, + 0.0036327910056570545, + 0.0041702909948071465, + 0.004153250003582798, + 0.0040829170029610395, + 0.004048916001920588, + 0.004184625009656884, + 0.0044855419982923195, + 0.004233250001561828 + ] + }, + { + "forward": 0.0056509159912820905, + "prefill": 0.002327915994101204, + "decode_steps": [ + 0.0044304999901214615, + 0.004346542002167553, + 0.004203999997116625, + 0.004197875008685514, + 0.004247332995873876, + 0.00400104200525675, + 0.003925625002011657, + 0.003970459001720883, + 0.004184374993201345, + 0.00391654200211633, + 0.004976208001608029, + 0.0041017500043381006, + 0.004252917002304457, + 0.004128667002078146, + 0.004077875011716969, + 0.004069707996677607, + 0.004280958004528657, + 0.004260333997081034, + 0.004123958991840482, + 0.003946708005969413, + 0.00400512499618344, + 0.004084166997927241, + 0.004016874998342246, + 0.004104666993953288, + 0.004161165998084471, + 0.00415783301286865, + 0.004137500000069849, + 0.0042312500008847564, + 0.004096500008017756, + 0.004180874995654449, + 0.004055500001413748, + 0.0043447500065667555 + ] + }, + { + "forward": 0.005558625009143725, + "prefill": 0.0022399589943233877, + "decode_steps": [ + 0.0045324169914238155, + 0.004112832990358584, + 0.003962750008213334, + 0.004198082999209873, + 0.004284292008378543, + 0.004515458000241779, + 0.004307124996557832, + 0.004436791001353413, + 0.003953665989683941, + 0.004167417006101459, + 0.004027041999506764, + 0.004226875011227094, + 0.004848625001613982, + 0.00474120800208766, + 0.0044727500062435865, + 0.004445583006599918, + 0.004579041007673368, + 0.004606957998475991, + 0.004400125006213784, + 0.00425487500615418, + 0.004384749991004355, + 0.004343625012552366, + 0.00423445799970068, + 0.004183416007435881, + 0.0038454579917015508, + 0.004215666005620733, + 0.004297041989048012, + 0.004264084011083469, + 0.004674458003137261, + 0.004271458994480781, + 0.004401166996103711, + 0.004066708002937958 + ] + }, + { + "forward": 0.0051376250048633665, + "prefill": 0.0026590420020511374, + "decode_steps": [ + 0.004325458008679561, + 0.004646834000595845, + 0.00444529099331703, + 0.004559499997412786, + 0.00422383300610818, + 0.004705915998783894, + 0.004165249993093312, + 0.004186999998637475, + 0.004198249996989034, + 0.00421916700724978, + 0.004299125008401461, + 0.004211792009300552, + 0.004563124995911494, + 0.004145792001509108, + 0.004279834000044502, + 0.004083290987182409, + 0.004144709004322067, + 0.003906375000951812, + 0.003991750010754913, + 0.004036500002257526, + 0.00497166599961929, + 0.005686583011993207, + 0.0045235410070745274, + 0.00435879199358169, + 0.004195875008008443, + 0.004440749995410442, + 0.003942374998587184, + 0.004681125006754883, + 0.004743208002764732, + 0.004324666995671578, + 0.00436158299271483, + 0.00437020898971241 + ] + }, + { + "forward": 0.005581250006798655, + "prefill": 0.002316166996024549, + "decode_steps": [ + 0.004169332998571917, + 0.004096791992196813, + 0.004168542000115849, + 0.004138917007367127, + 0.004094207994057797, + 0.004360125007224269, + 0.004178999995929189, + 0.00424008299887646, + 0.004101416998310015, + 0.0041364170028828084, + 0.0039293750014621764, + 0.00412908400176093, + 0.0040712920017540455, + 0.003981499990914017, + 0.004012042001704685, + 0.004202957992674783, + 0.0041375840082764626, + 0.00419520799187012, + 0.004307166993385181, + 0.0036276659957366064, + 0.004170874992269091, + 0.004270167002687231, + 0.004170750005869195, + 0.004237624991219491, + 0.004137790994718671, + 0.004210166996927001, + 0.004184874997008592, + 0.004156499999226071, + 0.0041880420030793175, + 0.004195165995042771, + 0.004141334007726982, + 0.0040025420021265745 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 5.5360625046887435, + "p95": 5.743079195963219, + "min": 4.801249990123324, + "max": 5.785417000879534, + "iqr": 0.5839365003339481, + "mean": 5.358958398574032, + "cv": 0.06884715103079679 + }, + "prefill_ms": { + "count": 10, + "median": 2.329041497432627, + "p95": 2.622010603226954, + "min": 2.2399589943233877, + "max": 2.6590420020511374, + "iqr": 0.1193747557408642, + "mean": 2.371500199660659, + "cv": 0.05925023962446324 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.149063002842013, + "p95": 4.579043107514735, + "min": 3.6276659957366064, + "max": 5.686583011993207, + "iqr": 0.18699000429478474, + "mean": 4.182919144204789, + "cv": 0.05282031284190881 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 241.19565888594113, + "p95": 243.90132299220096, + "min": 228.67079252601826, + "max": 244.43116105329773, + "iqr": 5.598669572229056, + "mean": 239.17656977789306, + "cv": 0.02226125643374368 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 236.96766846129827, + "p95": 239.45991673421875, + "min": 224.40674706297503, + "max": 239.71302446588956, + "iqr": 5.284651114915903, + "mean": 235.01120748601434, + "cv": 0.022078068115239588 + }, + "memory": { + "peak_extra_bytes": 1229824, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "auto", + "batch": 1, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.002571666002040729 + }, + { + "forward": 0.0029697079880861565 + }, + { + "forward": 0.002776082998025231 + }, + { + "forward": 0.002533457998652011 + }, + { + "forward": 0.0027378329978091642 + }, + { + "forward": 0.0026135410007555038 + }, + { + "forward": 0.0025935000012395903 + }, + { + "forward": 0.0024411250051343814 + }, + { + "forward": 0.002559374988777563 + }, + { + "forward": 0.002426291990559548 + } + ], + "forward_ms": { + "count": 10, + "median": 2.5825830016401596, + "p95": 2.88257674255874, + "min": 2.426291990559548, + "max": 2.9697079880861565, + "iqr": 0.16682275236235, + "mean": 2.622258097107988, + "cv": 0.06282396199540999 + }, + "memory": { + "peak_extra_bytes": 1476608, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "sdpa", + "batch": 1, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.002208125006291084, + "prefill": 0.0022260830010054633, + "decode_steps": [ + 0.0016355409898096696, + 0.0016754160023992881, + 0.0016366659983759746, + 0.0015762499970151111, + 0.0015623749932274222, + 0.001577791990712285, + 0.0015700840012868866, + 0.001624375014216639, + 0.001596458998392336, + 0.0015988340019248426, + 0.0016025420045480132, + 0.0016448749956907704, + 0.0015951249952195212, + 0.0015847500035306439, + 0.001557167008286342, + 0.0017887080030050129, + 0.0017249170050490648, + 0.0016011249972507358, + 0.0015837079990888014, + 0.0017120420088758692, + 0.0015848750044824556, + 0.0015695420006522909, + 0.001592207991052419, + 0.0016337090055458248, + 0.0016784579929662868, + 0.0015904159954516217, + 0.0016533329908270389, + 0.0016823749901959673, + 0.0015968330117175356, + 0.0016229169996222481, + 0.0016346250049537048, + 0.0016896670131245628 + ] + }, + { + "forward": 0.002415916998870671, + "prefill": 0.0023352080024778843, + "decode_steps": [ + 0.001805708001484163, + 0.0018111249955836684, + 0.0017179579881485552, + 0.0016732499934732914, + 0.0016020000039134175, + 0.0015741250099381432, + 0.0016850840038387105, + 0.0017021670064423233, + 0.0016346250049537048, + 0.0015920830046525225, + 0.0016109580028569326, + 0.001683624999714084, + 0.0015657910116715357, + 0.001609291008207947, + 0.0015817500097909942, + 0.0015972920082276687, + 0.001622416006284766, + 0.0015812499914318323, + 0.0016023749922169372, + 0.0015873750089667737, + 0.0015599159960402176, + 0.0015838749968679622, + 0.0015932079986669123, + 0.0015827919996809214, + 0.0015478330024052411, + 0.0016618749941699207, + 0.001782416002242826, + 0.0015886250039329752, + 0.001604958000825718, + 0.0016404160123784095, + 0.001808417000574991, + 0.001557374998810701 + ] + }, + { + "forward": 0.0022386659984476864, + "prefill": 0.002251375000923872, + "decode_steps": [ + 0.0017947500018635765, + 0.0018399169930489734, + 0.001587416001711972, + 0.0015776660002302378, + 0.0015335000061895698, + 0.0017395000031683594, + 0.0017859589861473069, + 0.0015312089963117614, + 0.0016986249975161627, + 0.0016791249945526943, + 0.001567082988913171, + 0.0016887499950826168, + 0.0015728330035926774, + 0.0015263329987647012, + 0.0015667500119889155, + 0.001554875008878298, + 0.0016402920009568334, + 0.0015247919945977628, + 0.0016708750044927, + 0.0015595419972669333, + 0.0015353750059148297, + 0.0016039579932112247, + 0.0015514580009039491, + 0.001668915996560827, + 0.0016875409928616136, + 0.001549374996102415, + 0.0015880840073805302, + 0.0016822920006234199, + 0.0015711250016465783, + 0.0015784160059411079, + 0.0017369580018566921, + 0.0015853330114623532 + ] + }, + { + "forward": 0.0022018339950591326, + "prefill": 0.002129416010575369, + "decode_steps": [ + 0.0016011249972507358, + 0.0016135829937411472, + 0.001629291000426747, + 0.0015710839943494648, + 0.0017077499942388386, + 0.0015684999962104484, + 0.0016015419969335198, + 0.0017229160002898425, + 0.0015727079880889505, + 0.0015393750072689727, + 0.0015664160018786788, + 0.001552041998365894, + 0.0016552089946344495, + 0.0016657090018270537, + 0.0015548749943263829, + 0.0015538329898845404, + 0.0015458330017281696, + 0.0015207499964162707, + 0.00154974999895785, + 0.001575250003952533, + 0.0016075000021373853, + 0.0015592910058330745, + 0.0016950419958448038, + 0.0015729589940747246, + 0.0015701660013291985, + 0.001581917007570155, + 0.0016660420078551397, + 0.0015651660069124773, + 0.0015923340106382966, + 0.0015918749995762482, + 0.0016714579978724942, + 0.0015586250083288178 + ] + }, + { + "forward": 0.002393625007243827, + "prefill": 0.002382208011113107, + "decode_steps": [ + 0.0018169579998357221, + 0.0017141250136774033, + 0.0017294170102104545, + 0.0017510420002508909, + 0.001763832988217473, + 0.001602541989996098, + 0.0017057920049410313, + 0.0017977910029003397, + 0.0017410830041626468, + 0.0016055000014603138, + 0.0016669169999659061, + 0.0016413749981438741, + 0.0016136250051204115, + 0.0015937910065986216, + 0.0015919169964035973, + 0.0016160410013981164, + 0.001603249998879619, + 0.0016150830051628873, + 0.0015912079979898408, + 0.0016017089947126806, + 0.0015902080049272627, + 0.0015985840000212193, + 0.0017895409982884303, + 0.001644708012463525, + 0.0016177910001715645, + 0.0015885419998085126, + 0.0015918339922791347, + 0.0016411669930675998, + 0.0016392079996876419, + 0.0015835829981369898, + 0.001709208998363465, + 0.0016230839974014089 + ] + }, + { + "forward": 0.0021976249990984797, + "prefill": 0.0022265000006882474, + "decode_steps": [ + 0.0019162499957019463, + 0.0017801670037442818, + 0.0016154580080183223, + 0.0015647910040570423, + 0.001716792001388967, + 0.0017152079963125288, + 0.0016349580109817907, + 0.0016327909979736432, + 0.0015722499956609681, + 0.001519459008704871, + 0.0015378749958472326, + 0.0015543330082437024, + 0.0016656670049997047, + 0.001541749996249564, + 0.001662082999246195, + 0.0015819580003153533, + 0.0016503749939147383, + 0.0016137080092448741, + 0.001553750000311993, + 0.001685125011135824, + 0.0015990420070011169, + 0.0016969580028671771, + 0.0017242920002900064, + 0.0016834580019349232, + 0.0015849170013098046, + 0.001535791001515463, + 0.0017179169954033569, + 0.0015732500032754615, + 0.001602541989996098, + 0.001574082998558879, + 0.0016265419981209561, + 0.001663166010985151 + ] + }, + { + "forward": 0.002186249999795109, + "prefill": 0.0022009579988662153, + "decode_steps": [ + 0.0017608339985599741, + 0.0016837500006658956, + 0.001552834000904113, + 0.001571790999150835, + 0.0015450409991899505, + 0.0015602500061504543, + 0.0016462499916087836, + 0.001541707999422215, + 0.0015654159942641854, + 0.0015463750023627654, + 0.0015296670026145875, + 0.001716207989375107, + 0.0016092080040834844, + 0.0015940419980324805, + 0.0019642080005723983, + 0.0015902500017546117, + 0.0017590419884072617, + 0.0016082080110209063, + 0.0016063750081229955, + 0.0015601250051986426, + 0.0015866670000832528, + 0.0016185830027097836, + 0.0015886250039329752, + 0.0015894590032985434, + 0.0016017500020097941, + 0.001668791999691166, + 0.0015749589947517961, + 0.0015714170003775507, + 0.0015599589969497174, + 0.0015801670087967068, + 0.001598333998117596, + 0.001574541995069012 + ] + }, + { + "forward": 0.0021547919895965606, + "prefill": 0.0020952920021954924, + "decode_steps": [ + 0.0017527090094517916, + 0.0017493750056019053, + 0.001745749992551282, + 0.0019922079954994842, + 0.0016860829928191379, + 0.0017701250035315752, + 0.0016870419931365177, + 0.001707791001535952, + 0.0016734999953769147, + 0.0016727080073906109, + 0.0016586249985266477, + 0.0019433330016909167, + 0.0027365829882910475, + 0.0021501669980352744, + 0.002121500001521781, + 0.0021379999961936846, + 0.00223445899609942, + 0.0021514580002985895, + 0.002561541012255475, + 0.0026708330115070567, + 0.0026392080035293475, + 0.002597708997200243, + 0.0026585419982438907, + 0.0026287920045433566, + 0.0016060839989222586, + 0.001855459006037563, + 0.001808916000300087, + 0.0018166250083595514, + 0.0017107500025304034, + 0.001690082994173281, + 0.001819249999243766, + 0.0018117080035153776 + ] + }, + { + "forward": 0.0022897080052644014, + "prefill": 0.002240083005744964, + "decode_steps": [ + 0.0017659580043982714, + 0.0016625830030534416, + 0.001577584000187926, + 0.0017199169960804284, + 0.001579250005306676, + 0.0016348339995602146, + 0.0016341670125257224, + 0.001601708005182445, + 0.0015910420042928308, + 0.0016974590107565746, + 0.0015618329925928265, + 0.0016099579952424392, + 0.001590500003658235, + 0.0015686249971622601, + 0.001589708001120016, + 0.0016294999950332567, + 0.0016933339938987046, + 0.001565749989822507, + 0.0015838749968679622, + 0.0017842920060502365, + 0.0019370000081835315, + 0.0018252080044476315, + 0.0016182499966816977, + 0.0015902080049272627, + 0.0016420409956481308, + 0.001570833002915606, + 0.0015967499930411577, + 0.0017063749983208254, + 0.001614499997231178, + 0.001601208990905434, + 0.0016182079998543486, + 0.0015734159969724715 + ] + }, + { + "forward": 0.0022079170012148097, + "prefill": 0.002202834002673626, + "decode_steps": [ + 0.0017648750072112307, + 0.0016301669966196641, + 0.0017383330123266205, + 0.001570083011756651, + 0.0016005000070435926, + 0.0015373750065919012, + 0.0016491660062456504, + 0.0015879169950494543, + 0.0015806250012246892, + 0.001564624995808117, + 0.001597333000972867, + 0.0016135000041685998, + 0.0016094580059871078, + 0.0015846669994061813, + 0.00154191700858064, + 0.0015416250098496675, + 0.0016072079888544977, + 0.0015474590036319569, + 0.0015512499958276749, + 0.0015527909999946132, + 0.0015947920037433505, + 0.0015902080049272627, + 0.0015837079990888014, + 0.001686499992501922, + 0.0015405839949380606, + 0.0015653340087737888, + 0.001589542007423006, + 0.0015870420029386878, + 0.0015656250034226105, + 0.0015902919985819608, + 0.0015134159912122414, + 0.0015840840060263872 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 2.208021003752947, + "p95": 2.405885602638591, + "min": 2.1547919895965606, + "max": 2.415916998870671, + "iqr": 0.07827025547157973, + "mean": 2.249445900088176, + "cv": 0.03964456643280686 + }, + "prefill_ms": { + "count": 10, + "median": 2.2262915008468553, + "p95": 2.361058007227257, + "min": 2.0952920021954924, + "max": 2.382208011113107, + "iqr": 0.04712500231107697, + "mean": 2.228995703626424, + "cv": 0.038025670876134024 + }, + "decode_ms_per_step": { + "count": 320, + "median": 1.605229001143016, + "p95": 1.9172874963260258, + "min": 1.5134159912122414, + "max": 2.7365829882910475, + "iqr": 0.10893674334511161, + "mean": 1.6632968569410878, + "cv": 0.10850981038414287 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 613.7654621555089, + "p95": 627.1562102491101, + "min": 496.53268111549386, + "max": 627.918828296943, + "iqr": 9.162090224900112, + "mean": 603.8063127653666, + "cv": 0.06361405765934418 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 588.4926864289154, + "p95": 601.5734227601744, + "min": 480.89777829929085, + "max": 601.9016132716026, + "iqr": 9.038980924012321, + "mean": 579.3183397174016, + "cv": 0.06105604863275308 + }, + "memory": { + "peak_extra_bytes": 1476608, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "reference", + "batch": 1, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.006667541994829662, + "prefill": 0.003387874996406026, + "decode_steps": [ + 0.004805083008250222, + 0.004443750003702007, + 0.003901250005583279, + 0.00429525000799913, + 0.00479187500604894, + 0.004257207998307422, + 0.00424550000752788, + 0.004023415996925905, + 0.0042899159889202565, + 0.00422699999762699, + 0.004272916994523257, + 0.004259749999619089, + 0.004180666001047939, + 0.003980207999120466, + 0.004079500009538606, + 0.0041407920070923865, + 0.004086375003680587, + 0.003907167003490031, + 0.0041316250135423616, + 0.004284333001123741, + 0.004149624990532175, + 0.004122667000046931, + 0.004185208003036678, + 0.0042162499885307625, + 0.004042833999847062, + 0.004143750004004687, + 0.004137291005463339, + 0.004500125010963529, + 0.004202125011943281, + 0.004233792002196424, + 0.0042824999982258305, + 0.0041842499922495335 + ] + }, + { + "forward": 0.006842667004093528, + "prefill": 0.00327408300654497, + "decode_steps": [ + 0.004695707990322262, + 0.0043920840107603, + 0.004320292006013915, + 0.004538624998531304, + 0.004302540997741744, + 0.004296458006137982, + 0.0037138340121600777, + 0.004316042002756149, + 0.0044392499985406175, + 0.004167000006418675, + 0.004087708992301486, + 0.004337916994700208, + 0.0039073340012691915, + 0.004267500000423752, + 0.004228166013490409, + 0.0044570830068551, + 0.004237790999468416, + 0.004222042000037618, + 0.004291167002520524, + 0.004234958003507927, + 0.004272542006219737, + 0.00411095900926739, + 0.004185292011243291, + 0.004324374996940605, + 0.004383125007734634, + 0.004277291998732835, + 0.004302540997741744, + 0.004036999991512857, + 0.004438666001078673, + 0.004327541988459416, + 0.004352666990598664, + 0.004275042010704055 + ] + }, + { + "forward": 0.006429417000617832, + "prefill": 0.0031879589951131493, + "decode_steps": [ + 0.00431729199772235, + 0.004197250003926456, + 0.004123958002310246, + 0.004125708001083694, + 0.004151749992161058, + 0.004033584002172574, + 0.004143749989452772, + 0.004175791997113265, + 0.0040358330006711185, + 0.0038702089950675145, + 0.00494158299989067, + 0.004543417002423666, + 0.004338167011155747, + 0.0043342919962015, + 0.004840834008064121, + 0.00536741700489074, + 0.00520029199833516, + 0.005081166003947146, + 0.004858125001192093, + 0.005575208997470327, + 0.005698207998648286, + 0.0048911249905359, + 0.0052152910066070035, + 0.005480583000462502, + 0.006062874992494471, + 0.005488499999046326, + 0.006364292014040984, + 0.0046799160045338795, + 0.005155290986294858, + 0.004474500005017035, + 0.004403915998409502, + 0.004333125005359761 + ] + }, + { + "forward": 0.0063519579998683184, + "prefill": 0.0031888749945210293, + "decode_steps": [ + 0.004691791997174732, + 0.004470583007787354, + 0.004324167006416246, + 0.004291167002520524, + 0.004136832998483442, + 0.00412625000171829, + 0.004195374989649281, + 0.004147458006627858, + 0.0040911250107456, + 0.004205249992082827, + 0.004191874992102385, + 0.004157582996413112, + 0.004053208002005704, + 0.004119416989851743, + 0.004165791004197672, + 0.004048167000291869, + 0.0040533749997848645, + 0.004057874990394339, + 0.004066207999130711, + 0.004169834006461315, + 0.004197165995719843, + 0.00390533400059212, + 0.004258125001797453, + 0.004148666994296946, + 0.0042280000052414834, + 0.0040790829953039065, + 0.004214250002405606, + 0.003912000000127591, + 0.004117582997423597, + 0.004317290993640199, + 0.004166499988059513, + 0.004031999997096136 + ] + }, + { + "forward": 0.006443166988901794, + "prefill": 0.003221582999685779, + "decode_steps": [ + 0.004456583992578089, + 0.004186500009382144, + 0.004542750000837259, + 0.004351499999756925, + 0.004191542000626214, + 0.004347791997133754, + 0.004423665988724679, + 0.00493970800016541, + 0.004429791995789856, + 0.004000916000222787, + 0.00429020899173338, + 0.003920000002835877, + 0.0040620830113766715, + 0.004180874995654449, + 0.004258624991052784, + 0.004326166003011167, + 0.0042415830102982, + 0.004073249991051853, + 0.004083666994119994, + 0.004176832997472957, + 0.004195208995952271, + 0.004112458991585299, + 0.004222624993417412, + 0.0040750829939497635, + 0.004072041003382765, + 0.0041148750024149194, + 0.003998125001089647, + 0.004118041004403494, + 0.004088000001502223, + 0.004088583998964168, + 0.003962290997151285, + 0.004241292001097463 + ] + }, + { + "forward": 0.005695792002370581, + "prefill": 0.003198915990651585, + "decode_steps": [ + 0.004199749993858859, + 0.0040048340015346184, + 0.004041542008053511, + 0.004023832996608689, + 0.004041791005874984, + 0.0041445829992881045, + 0.00413391699839849, + 0.003983749993494712, + 0.004000375003670342, + 0.0039914999942993745, + 0.004099208003026433, + 0.00381520799419377, + 0.004087957990122959, + 0.004247999997460283, + 0.004194749999442138, + 0.004174416011665016, + 0.004220125003485009, + 0.004137707990594208, + 0.0041518749931128696, + 0.004214208005578257, + 0.004227790996083058, + 0.00404216701281257, + 0.004314749996410683, + 0.004316292004659772, + 0.00426383399462793, + 0.004088542002136819, + 0.004235541011439636, + 0.0041089580045081675, + 0.004099915997358039, + 0.00411858300503809, + 0.003981166999437846, + 0.004065624991199002 + ] + }, + { + "forward": 0.006520833005197346, + "prefill": 0.0031395420082844794, + "decode_steps": [ + 0.00460329199268017, + 0.004227792000165209, + 0.004258999993908219, + 0.0042436250078026205, + 0.004015083992271684, + 0.004295082995668054, + 0.004197540998575278, + 0.004170542000792921, + 0.004214583997963928, + 0.004319540996220894, + 0.004152708002948202, + 0.004219082999043167, + 0.004066709007020108, + 0.004156124996370636, + 0.004045082998345606, + 0.004193957996903919, + 0.0040290000033564866, + 0.004136749994358979, + 0.0041013750014826655, + 0.0040057500009424984, + 0.003993082995293662, + 0.004158750001806766, + 0.003848541993647814, + 0.0042356250050943345, + 0.003943457995774224, + 0.0041482499946141616, + 0.004105041996808723, + 0.004210832994431257, + 0.003892332999384962, + 0.004145250000874512, + 0.004080250000697561, + 0.004133834008825943 + ] + }, + { + "forward": 0.006976667005801573, + "prefill": 0.00343295800848864, + "decode_steps": [ + 0.00453037500847131, + 0.00443000000086613, + 0.004382957995403558, + 0.0044312499958323315, + 0.004568500007735565, + 0.0042939170089084655, + 0.004101208003703505, + 0.004296291997889057, + 0.004246584008797072, + 0.004331207994255237, + 0.004062540989252739, + 0.00415370799601078, + 0.004200540992314927, + 0.0041434999875491485, + 0.004140874996664934, + 0.004036124999402091, + 0.004243666000547819, + 0.004139249998843297, + 0.00407541700406, + 0.004114167008083314, + 0.004198958005872555, + 0.004023082990897819, + 0.004028416005894542, + 0.003920375005691312, + 0.004088292000233196, + 0.0041174580110237, + 0.004135959010454826, + 0.004074833006598055, + 0.004065707995323464, + 0.0040320420084754005, + 0.004152334004174918, + 0.0040642499952809885 + ] + }, + { + "forward": 0.005726290997699834, + "prefill": 0.0033244579972233623, + "decode_steps": [ + 0.004324833003920503, + 0.0040828750061336905, + 0.00407745799748227, + 0.004072499999892898, + 0.004088957997737452, + 0.004149250002228655, + 0.004096750009921379, + 0.004271375000826083, + 0.004161165998084471, + 0.0041030409920495, + 0.004007208000984974, + 0.0040902909968281165, + 0.0040819590067258105, + 0.004075000004377216, + 0.004122207988984883, + 0.004037416001665406, + 0.004198833004920743, + 0.004069624992553145, + 0.004059124999912456, + 0.003970124991610646, + 0.004318209001212381, + 0.004268666001735255, + 0.004092208997462876, + 0.004031541990116239, + 0.004131042005610652, + 0.004077290999703109, + 0.004150875000050291, + 0.004068416004884057, + 0.004087624998646788, + 0.004110041001695208, + 0.0038781670009484515, + 0.004038291997858323 + ] + }, + { + "forward": 0.006758333009202033, + "prefill": 0.0034280830004718155, + "decode_steps": [ + 0.004630458002793603, + 0.004150416993070394, + 0.004299124993849546, + 0.004304333997424692, + 0.004318000006605871, + 0.004106124993995763, + 0.0042679160105763, + 0.0040438330033794045, + 0.004173334004008211, + 0.004261334004695527, + 0.004208833008306101, + 0.00417541699425783, + 0.004113458999199793, + 0.004119499993976206, + 0.004083291991264559, + 0.004135916009545326, + 0.00411187500867527, + 0.004106917011085898, + 0.004069374990649521, + 0.004014916004962288, + 0.0041806249937508255, + 0.0044570830068551, + 0.003962957998737693, + 0.004480500007048249, + 0.004227166995406151, + 0.00428512500366196, + 0.004080458005773835, + 0.00419879199762363, + 0.004324333989643492, + 0.004247333010425791, + 0.004148374995565973, + 0.004257459004293196 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 6.48199999704957, + "p95": 6.916367005032953, + "min": 5.695792002370581, + "max": 6.976667005801573, + "iqr": 0.3643125055532437, + "mean": 6.44126670085825, + "cv": 0.06709280264527542 + }, + "prefill_ms": { + "count": 10, + "median": 3.2478330031153746, + "p95": 3.430764254881069, + "min": 3.1395420082844794, + "max": 3.43295800848864, + "iqr": 0.18063550305669196, + "mean": 3.2784331997390836, + "cv": 0.03301523293014667 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.168417006439995, + "p95": 4.806870558240917, + "min": 3.7138340121600777, + "max": 6.364292014040984, + "iqr": 0.20726074944832362, + "mean": 4.238877837724431, + "cv": 0.07280297892687679 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 238.47989416574435, + "p95": 243.2291911017787, + "min": 209.83159374369762, + "max": 243.54534252820608, + "iqr": 3.932222558872752, + "mean": 236.30776403427686, + "cv": 0.041199730749546165 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 232.53485745452264, + "p95": 237.33366395966462, + "min": 205.53504725900595, + "max": 237.535282258078, + "iqr": 4.079055025400379, + "mean": 230.71159737077522, + "cv": 0.04020982516500392 + }, + "memory": { + "peak_extra_bytes": 4425728, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "auto", + "batch": 1, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.0037995000020600855 + }, + { + "forward": 0.00408400000014808 + }, + { + "forward": 0.004471667009056546 + }, + { + "forward": 0.003813709001406096 + }, + { + "forward": 0.0041152920020977035 + }, + { + "forward": 0.004243417002726346 + }, + { + "forward": 0.003828166998573579 + }, + { + "forward": 0.00482420799380634 + }, + { + "forward": 0.0037634580075973645 + }, + { + "forward": 0.003731999997398816 + } + ], + "forward_ms": { + "count": 10, + "median": 3.9560834993608296, + "p95": 4.6655645506689325, + "min": 3.731999997398816, + "max": 4.82420799380634, + "iqr": 0.40833350067259744, + "mean": 4.067541801487096, + "cv": 0.08856535457117198 + }, + "memory": { + "peak_extra_bytes": 2953216, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "sdpa", + "batch": 1, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.003158541992888786, + "prefill": 0.0032397920003859326, + "decode_steps": [ + 0.0018406250019324943, + 0.0015972919936757535, + 0.0017017920035868883, + 0.0016353749961126596, + 0.0016854159912327304, + 0.0017452920001232997, + 0.0017221659945789725, + 0.001615042012417689, + 0.0017251249955734238, + 0.0017124159930972382, + 0.0017244999908143654, + 0.0017077080119634047, + 0.0016077079926617444, + 0.0017577499966137111, + 0.0017062499973690137, + 0.0017149580089608207, + 0.0016027499950723723, + 0.0017797089967643842, + 0.0017064170097000897, + 0.0016954999882727861, + 0.0016754999960539863, + 0.0017148750048363581, + 0.001674458006164059, + 0.0016881670017028227, + 0.0015590420080116019, + 0.0015629160043317825, + 0.0018576250004116446, + 0.0016053339932113886, + 0.0016465420048916712, + 0.0015828329924261197, + 0.0017172500083688647, + 0.001747040994814597 + ] + }, + { + "forward": 0.003188625007169321, + "prefill": 0.0032835410092957318, + "decode_steps": [ + 0.0018642920040292665, + 0.0017111659981310368, + 0.0017127909959526733, + 0.0016525419923709705, + 0.0016274170047836378, + 0.0017094159993575886, + 0.001720375003060326, + 0.0017143330042017624, + 0.0015810000040801242, + 0.0017032910109264776, + 0.001691457990091294, + 0.0017990000051213428, + 0.0016008329985197634, + 0.001727625000057742, + 0.0018246249965159222, + 0.001638665999053046, + 0.0016793749964563176, + 0.0016627500008326024, + 0.0016248340107267722, + 0.0017054580093827099, + 0.0017293329874519259, + 0.0016048749967012554, + 0.0017418329953216016, + 0.0016252909990726039, + 0.001579166011651978, + 0.0017234999977517873, + 0.0016194169875234365, + 0.0017185409960802644, + 0.0017087089945562184, + 0.0017097080126404762, + 0.00171050000062678, + 0.0017375000024912879 + ] + }, + { + "forward": 0.0033968329953495413, + "prefill": 0.0034633330069482327, + "decode_steps": [ + 0.0020300410105846822, + 0.0017275840073125437, + 0.0017243340116692707, + 0.0018406660092296079, + 0.001650874997721985, + 0.0017402910016244277, + 0.0017072089976863936, + 0.0017050000024028122, + 0.0016985420079436153, + 0.0017380839999532327, + 0.0017094999930122867, + 0.0017114170041168109, + 0.001686499992501922, + 0.0016613750049145892, + 0.0017943749990081415, + 0.001749707997078076, + 0.001742166990879923, + 0.0016101660003187135, + 0.0017047089932020754, + 0.0017150420026155189, + 0.0017215419939020649, + 0.001596834001247771, + 0.0017559169937158003, + 0.0016090830031316727, + 0.0016559590003453195, + 0.0016084160015452653, + 0.0015945839986670762, + 0.0017652079986874014, + 0.0017185409960802644, + 0.001599459006683901, + 0.001634416010347195, + 0.0017519590037409216 + ] + }, + { + "forward": 0.0032150830083992332, + "prefill": 0.0032462090020999312, + "decode_steps": [ + 0.00171929199132137, + 0.0017484590061940253, + 0.0018134580022888258, + 0.001818457996705547, + 0.0017297909944318235, + 0.0017175410030176863, + 0.002170625011785887, + 0.001662833004957065, + 0.0015886670007603243, + 0.0017190409998875111, + 0.0015770419995533302, + 0.0016985000111162663, + 0.0016953750018728897, + 0.0016789159999461845, + 0.0015732500032754615, + 0.0017261659959331155, + 0.0016971660079434514, + 0.001687208001385443, + 0.0017853750032372773, + 0.001597167007275857, + 0.0015798750100657344, + 0.0019115420000161976, + 0.0017009580042213202, + 0.001698667008895427, + 0.0017015840130625293, + 0.0016997499915305525, + 0.0017034579941537231, + 0.0017003329994622618, + 0.0018122500041499734, + 0.0017747499950928614, + 0.0016772500093793496, + 0.0016195000061998144 + ] + }, + { + "forward": 0.0034921250044135377, + "prefill": 0.00337987499369774, + "decode_steps": [ + 0.0018298749928362668, + 0.0017456249915994704, + 0.0017057920049410313, + 0.001766832996509038, + 0.0016298340051434934, + 0.0018304159893887118, + 0.0016375419945688918, + 0.0016402920009568334, + 0.0019146249978803098, + 0.0018452500080456957, + 0.0016219999961322173, + 0.0016752500087022781, + 0.0016207909939112142, + 0.0018372500053374097, + 0.0017191250080941245, + 0.001724417001241818, + 0.0016972499870462343, + 0.0018103749898727983, + 0.0017145419988082722, + 0.0016334579995600507, + 0.0016654170030960813, + 0.0015887090121395886, + 0.0017432079912396148, + 0.0016969580028671771, + 0.0017041249957401305, + 0.001583416000357829, + 0.001857124996604398, + 0.0017657499993219972, + 0.001773541996954009, + 0.0016500830097356811, + 0.0017890419985633343, + 0.0017167089972645044 + ] + }, + { + "forward": 0.0037280420074239373, + "prefill": 0.004006916991784237, + "decode_steps": [ + 0.0021356250072130933, + 0.0017902499967021868, + 0.0018229580018669367, + 0.0018745419947663322, + 0.0017296669975621626, + 0.001707999996142462, + 0.0017157499969471246, + 0.001818333999835886, + 0.0017096669907914475, + 0.0017710829997668043, + 0.0017877500067697838, + 0.0017395829927409068, + 0.0017212919919984415, + 0.0018701250082813203, + 0.0017882919928524643, + 0.0017666659987298772, + 0.0017272500117542222, + 0.0016797499993117526, + 0.0017665000050328672, + 0.001737332990160212, + 0.00159087500651367, + 0.0017827080009737983, + 0.001768582995282486, + 0.001771208000718616, + 0.0017297089943895116, + 0.0016139159997692332, + 0.0017440420051570982, + 0.001922791008837521, + 0.0018092090031132102, + 0.001776375007466413, + 0.0017271250108024105, + 0.0016269580082735047 + ] + }, + { + "forward": 0.003250666006351821, + "prefill": 0.0032809999975143, + "decode_steps": [ + 0.0017407499981345609, + 0.0016655829967930913, + 0.0016965420072665438, + 0.001868915991508402, + 0.0017492499900981784, + 0.0017250840028282255, + 0.0016798750002635643, + 0.0028710409969789907, + 0.0017092079942813143, + 0.0017611249932087958, + 0.001575457994476892, + 0.0018872920045396313, + 0.001748957991367206, + 0.0016277909890050068, + 0.001731666998239234, + 0.001778457997716032, + 0.0017574999947100878, + 0.0017820830107666552, + 0.0016018339956644922, + 0.0017106250015785918, + 0.0016121670050779358, + 0.001617749992874451, + 0.0016125419933814555, + 0.0015912079979898408, + 0.0017874580080388114, + 0.0017190420039696619, + 0.0017225830088136718, + 0.0018129590025637299, + 0.0017254999984288588, + 0.0017087909945985302, + 0.0017242909962078556, + 0.0016669580072630197 + ] + }, + { + "forward": 0.0033183330087922513, + "prefill": 0.0032614579977234825, + "decode_steps": [ + 0.0018472090014256537, + 0.0017677919968264177, + 0.0016238329990301281, + 0.001612916006706655, + 0.0017047080036718398, + 0.0015940000012051314, + 0.0017036659992299974, + 0.0016960409993771464, + 0.0016976670012809336, + 0.0017006669950205833, + 0.0018010829953709617, + 0.0017265000060433522, + 0.001563832993269898, + 0.0017909589951159433, + 0.001826416002586484, + 0.0017403329984517768, + 0.0017256249993806705, + 0.0017350419948343188, + 0.0018558750016381964, + 0.0017271659889956936, + 0.0017355000018142164, + 0.001585332996910438, + 0.0016757920093368739, + 0.0016530410066479817, + 0.001698250009212643, + 0.0016901250055525452, + 0.0017750410042935982, + 0.001719249994494021, + 0.0016890419938135892, + 0.0016934589948505163, + 0.0016052919963840395, + 0.0016380409942939878 + ] + }, + { + "forward": 0.0033408750023227185, + "prefill": 0.0032262500026263297, + "decode_steps": [ + 0.0019213749910704792, + 0.0017399589996784925, + 0.0016619999951217324, + 0.0016594999906374142, + 0.0016058340115705505, + 0.0015917920100037009, + 0.0017088749882532284, + 0.001706124996417202, + 0.0017760829941835254, + 0.0017717919981805608, + 0.0016321250004693866, + 0.0017407499981345609, + 0.0018212910072179511, + 0.0016475829906994477, + 0.0018191670096712187, + 0.0017154579982161522, + 0.001716957995085977, + 0.0017119590047514066, + 0.0017030000017257407, + 0.0017291249969275668, + 0.001762708998285234, + 0.0017434999899705872, + 0.0017302500054938719, + 0.0017469170124968514, + 0.0017289580137003213, + 0.001743583008646965, + 0.0017193749954458326, + 0.0017262500041397288, + 0.0017256249993806705, + 0.001698833002592437, + 0.001699208005447872, + 0.0017042079998645931 + ] + }, + { + "forward": 0.003247667002142407, + "prefill": 0.0033099160064011812, + "decode_steps": [ + 0.0018834579968824983, + 0.00169162500242237, + 0.001577584000187926, + 0.0015609170077368617, + 0.0017246249917661771, + 0.0017301670013694093, + 0.001698499996564351, + 0.0017171249928651378, + 0.0017057090008165687, + 0.0017037499928846955, + 0.0017032089963322505, + 0.0016317499976139516, + 0.001585707999765873, + 0.0015826250019017607, + 0.001746499998262152, + 0.0016744169988669455, + 0.0016962499939836562, + 0.001587250008014962, + 0.0017909159942064434, + 0.0016184169944608584, + 0.0016012919950298965, + 0.0019088749977527186, + 0.0016047910030465573, + 0.0017263329937122762, + 0.0017123340076068416, + 0.0017139160045189783, + 0.0017065829888451844, + 0.001565749989822507, + 0.001776333010639064, + 0.001723666995530948, + 0.0017683749902062118, + 0.0016041250055423006 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 3.2844995075720362, + "p95": 3.621879356069257, + "min": 3.158541992888786, + "max": 3.7280420074239373, + "iqr": 0.15961449025780894, + "mean": 3.3336791035253555, + "cv": 0.05146998409790352 + }, + "prefill_ms": { + "count": 10, + "median": 3.282270503405016, + "p95": 3.7623041986080348, + "min": 3.2262500026263297, + "max": 4.006916991784237, + "iqr": 0.11236399586778134, + "mean": 3.36982910084771, + "cv": 0.06981896381691235 + }, + "decode_ms_per_step": { + "count": 320, + "median": 1.7121465061791241, + "p95": 1.8571499967947602, + "min": 1.5590420080116019, + "max": 2.8710409969789907, + "iqr": 0.08438550867140293, + "mean": 1.7161878593924484, + "cv": 0.06206185034922504 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 583.7137216561648, + "p95": 592.3869069570208, + "min": 566.1320420259071, + "max": 592.4239274626292, + "iqr": 9.19415318518088, + "mean": 582.8041684563067, + "cv": 0.014887605896427146 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 550.0493676375597, + "p95": 558.5610051070463, + "min": 528.6561973051536, + "max": 558.9015322695834, + "iqr": 8.776451718112298, + "mean": 549.1411715924821, + "cv": 0.016778665265284005 + }, + "memory": { + "peak_extra_bytes": 2953216, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "reference", + "batch": 1, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.010027666998212226, + "prefill": 0.0063299169996753335, + "decode_steps": [ + 0.0047746669879416, + 0.004369292000774294, + 0.004158708004979417, + 0.00444512499962002, + 0.004506666999077424, + 0.00483787500706967, + 0.004922791005810723, + 0.005193582997890189, + 0.004533625004114583, + 0.004556457992293872, + 0.004438125004526228, + 0.004582209003274329, + 0.00442045800446067, + 0.004469417006475851, + 0.004376917000627145, + 0.004178417002549395, + 0.00433445798989851, + 0.004238791996613145, + 0.004394750008941628, + 0.004311917000450194, + 0.004400334000820294, + 0.004311750002671033, + 0.00412812500144355, + 0.004233332991134375, + 0.003930207996745594, + 0.004317500002798624, + 0.004287541989469901, + 0.004408084001624957, + 0.004463416000362486, + 0.0042669169924920425, + 0.004268959004548378, + 0.004099374986253679 + ] + }, + { + "forward": 0.009279999998398125, + "prefill": 0.006428207998396829, + "decode_steps": [ + 0.004108915993128903, + 0.004851791993132792, + 0.004450832988368347, + 0.004359792001196183, + 0.0041688339988468215, + 0.004231917002471164, + 0.004073834003065713, + 0.004583375004585832, + 0.004339833001722582, + 0.004064666005433537, + 0.004106458000023849, + 0.004157583010965027, + 0.004163333011092618, + 0.00408999998762738, + 0.004183833007118665, + 0.004231041995808482, + 0.00430850000702776, + 0.00416458300605882, + 0.004142250007134862, + 0.004160458003752865, + 0.004280749999452382, + 0.004141124998568557, + 0.004220832997816615, + 0.0040738339885137975, + 0.004141874989727512, + 0.0039051670028129593, + 0.003993040998466313, + 0.004099375000805594, + 0.0040526669909013435, + 0.0039475419907830656, + 0.00407520800945349, + 0.0040751249907771125 + ] + }, + { + "forward": 0.008544333002646454, + "prefill": 0.007196666003437713, + "decode_steps": [ + 0.005075833993032575, + 0.004662333012674935, + 0.004314041987527162, + 0.004263750000973232, + 0.0036899580009048805, + 0.004286625000531785, + 0.004195041008642875, + 0.004159957999945618, + 0.004131499998038635, + 0.004218749993015081, + 0.004230332997394726, + 0.004386125001474284, + 0.004336541998782195, + 0.004230541002471, + 0.004125457999180071, + 0.004034415993373841, + 0.004193125001620501, + 0.004178458999376744, + 0.004366624998510815, + 0.00392662500962615, + 0.004147875006310642, + 0.0042354159959359095, + 0.004159500007517636, + 0.0038735420093871653, + 0.004165625010500662, + 0.0040097500022966415, + 0.004364915992482565, + 0.003928082995116711, + 0.0043476249993545935, + 0.00448233300994616, + 0.004310499993152916, + 0.0043762079876614735 + ] + }, + { + "forward": 0.010204125006566755, + "prefill": 0.006403667008271441, + "decode_steps": [ + 0.004765082994708791, + 0.004390500005683862, + 0.004249792007612996, + 0.004192915992462076, + 0.004273499987903051, + 0.0047651669883634895, + 0.0042967920016963035, + 0.004369375004898757, + 0.004122250000364147, + 0.0043889579974347726, + 0.004262166999978945, + 0.0039008340099826455, + 0.004791374987689778, + 0.004390207992400974, + 0.004214041997329332, + 0.0044817919988418, + 0.004282374997274019, + 0.0043748330062953755, + 0.004156041002715938, + 0.0041697499982547015, + 0.004203542004688643, + 0.004121292004128918, + 0.0041977919900091365, + 0.004130958011955954, + 0.004219207999994978, + 0.004173665991402231, + 0.0041053750028368086, + 0.0040903329936554655, + 0.004312374992878176, + 0.004195375004201196, + 0.004620499996235594, + 0.004347999987658113 + ] + }, + { + "forward": 0.009711332997540012, + "prefill": 0.006395332995452918, + "decode_steps": [ + 0.004866083996603265, + 0.004488792008487508, + 0.004196875001071021, + 0.0046207499981392175, + 0.004305666996515356, + 0.004158708004979417, + 0.004303749999962747, + 0.004182749995379709, + 0.004337791004218161, + 0.004339874998549931, + 0.004098541001440026, + 0.00432270800229162, + 0.004109708999749273, + 0.004134041999350302, + 0.004172458007815294, + 0.004118541008210741, + 0.004218292000587098, + 0.0043308750027790666, + 0.004153249989030883, + 0.004148541003814898, + 0.0042200419993605465, + 0.004107207991182804, + 0.004064832988660783, + 0.004122042009839788, + 0.00412812500144355, + 0.0039263329963432625, + 0.004189957995549776, + 0.004157333998591639, + 0.004134000002522953, + 0.00386587499815505, + 0.004235166998114437, + 0.004265416995622218 + ] + }, + { + "forward": 0.009557832992868498, + "prefill": 0.006771624990506098, + "decode_steps": [ + 0.004100500009371899, + 0.0046007080090930685, + 0.0041451249999227, + 0.0047597500088158995, + 0.0041802499908953905, + 0.004417375006596558, + 0.004346667003119364, + 0.004325457994127646, + 0.004362000006949529, + 0.004123291000723839, + 0.004273875005310401, + 0.004201499992632307, + 0.003849832995911129, + 0.0038038329948903993, + 0.004134749993681908, + 0.004187458005617373, + 0.004172332992311567, + 0.004333667005994357, + 0.004540167006780393, + 0.004423750011483207, + 0.0044716669945046306, + 0.004450167005416006, + 0.00423337500251364, + 0.004139750002650544, + 0.004151375003857538, + 0.004654916003346443, + 0.004222792005748488, + 0.004229875004966743, + 0.004062457999680191, + 0.004280624998500571, + 0.004107792003196664, + 0.00390604100539349 + ] + }, + { + "forward": 0.010206666993326508, + "prefill": 0.006335124999168329, + "decode_steps": [ + 0.004677417004131712, + 0.004287624993594363, + 0.004288459007511847, + 0.004278834006981924, + 0.0041348750091856346, + 0.004404083010740578, + 0.0042682079947553575, + 0.004637708989321254, + 0.004492167005082592, + 0.004302208006265573, + 0.004310917007387616, + 0.004490417006309144, + 0.004486791003728285, + 0.00426420800795313, + 0.004274708000593819, + 0.004200915995170362, + 0.004218834001221694, + 0.004058542006532662, + 0.004248915996868163, + 0.004093583993380889, + 0.004143708007177338, + 0.0042495840025367215, + 0.0042577499989420176, + 0.0041507079877192155, + 0.004085750013473444, + 0.004261874986696057, + 0.0041436669998802245, + 0.004256375003024004, + 0.004156250011874363, + 0.004073332995176315, + 0.004186457998002879, + 0.004167709004832432 + ] + }, + { + "forward": 0.009761082998011261, + "prefill": 0.006489624996902421, + "decode_steps": [ + 0.0048243749915855005, + 0.004364666005130857, + 0.004320167005062103, + 0.004237832996295765, + 0.0041120420064544305, + 0.004304916990804486, + 0.004282416004571132, + 0.004260625006281771, + 0.004175084002781659, + 0.004134417002205737, + 0.004233458006638102, + 0.004384250001749024, + 0.004211542007396929, + 0.004324125009588897, + 0.004273250000551343, + 0.004292082987376489, + 0.00422699999762699, + 0.004270833000191487, + 0.004116124997381121, + 0.0044072500022593886, + 0.004484125005546957, + 0.004578749998472631, + 0.004414832990732975, + 0.0038603339926339686, + 0.004277000000001863, + 0.004405667001265101, + 0.00426662499376107, + 0.004250250000040978, + 0.0042444589926162735, + 0.004557709005894139, + 0.004267333002644591, + 0.004223916999762878 + ] + }, + { + "forward": 0.011987667006906122, + "prefill": 0.007381167000858113, + "decode_steps": [ + 0.004355666998890229, + 0.0041019159980351105, + 0.004207666992442682, + 0.004611457989085466, + 0.004512250001425855, + 0.005143332993611693, + 0.003978749999077991, + 0.004193417000351474, + 0.00438962499902118, + 0.004328708004322834, + 0.004481834010221064, + 0.004314791993238032, + 0.00417241700051818, + 0.004350584000349045, + 0.004269540993846022, + 0.004376875003799796, + 0.004260500005329959, + 0.0042037910025101155, + 0.004130790999624878, + 0.004328708004322834, + 0.004320499996538274, + 0.004375874996185303, + 0.004110709007363766, + 0.004228833000524901, + 0.004213417007122189, + 0.004270582998287864, + 0.004297249994124286, + 0.004106042004423216, + 0.0041956670029321685, + 0.004279083004803397, + 0.004034666999359615, + 0.00417074999131728 + ] + }, + { + "forward": 0.010020000001532026, + "prefill": 0.006392875002347864, + "decode_steps": [ + 0.004692167000030167, + 0.004770125000504777, + 0.00442683300934732, + 0.004457166010979563, + 0.0042895840015262365, + 0.004214042011881247, + 0.004220791990519501, + 0.004192709006019868, + 0.0042632919939933345, + 0.004109166999114677, + 0.004052167001646012, + 0.004350666000391357, + 0.00431337500049267, + 0.004840709007112309, + 0.0044707089982694015, + 0.0042792919994099066, + 0.004483541997615248, + 0.004333708988269791, + 0.004350458009866998, + 0.004286542010959238, + 0.004421165998792276, + 0.004186334001133218, + 0.004205833000014536, + 0.004204832992400043, + 0.004300540997064672, + 0.004224584001349285, + 0.004190250008832663, + 0.0040325830050278455, + 0.0041804590000538155, + 0.004274958002497442, + 0.004105916988919489, + 0.004211540988762863 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 9.890541499771643, + "p95": 11.186217000795294, + "min": 8.544333002646454, + "max": 11.987667006906122, + "iqr": 0.5638025104417466, + "mean": 9.930070799600799, + "cv": 0.08856343418658157 + }, + "prefill_ms": { + "count": 10, + "median": 6.415937503334135, + "p95": 7.298141552018933, + "min": 6.3299169996753335, + "max": 7.381167000858113, + "iqr": 0.30763549148105085, + "mean": 6.612420799501706, + "cv": 0.05752681501064708 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.249688005074859, + "p95": 4.678154503926635, + "min": 3.6899580009048805, + "max": 5.193582997890189, + "iqr": 0.2050007533398457, + "mean": 4.27712904083819, + "cv": 0.04846951964114676 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 233.69053917064028, + "p95": 238.04134961049422, + "min": 226.69344503843732, + "max": 238.89801008850955, + "iqr": 3.3904348885160402, + "mean": 233.8454318637814, + "cv": 0.014353090230697528 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 223.05781386755464, + "p95": 227.2016311000059, + "min": 216.9642871069659, + "max": 227.95825021163975, + "iqr": 2.4734797653710245, + "mean": 223.0635273448009, + "cv": 0.013502502884872288 + }, + "memory": { + "peak_extra_bytes": 16715776, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "auto", + "batch": 1, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.007279874989762902 + }, + { + "forward": 0.007248665991937742 + }, + { + "forward": 0.007267292006872594 + }, + { + "forward": 0.007272000002558343 + }, + { + "forward": 0.007803249987773597 + }, + { + "forward": 0.007260750004206784 + }, + { + "forward": 0.0071852499968372285 + }, + { + "forward": 0.007372791995294392 + }, + { + "forward": 0.007838250006898306 + }, + { + "forward": 0.007206458001746796 + } + ], + "forward_ms": { + "count": 10, + "median": 7.269646004715469, + "p95": 7.822499998292187, + "min": 7.1852499968372285, + "max": 7.838250006898306, + "iqr": 0.0978757489065174, + "mean": 7.373458298388869, + "cv": 0.03268274516951385 + }, + "memory": { + "peak_extra_bytes": 6430720, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "sdpa", + "batch": 1, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.005558582997764461, + "prefill": 0.005541125006857328, + "decode_steps": [ + 0.0019804580078925937, + 0.0018488749919924885, + 0.001803249993827194, + 0.001779208003426902, + 0.0017583329899935052, + 0.0017646659980528057, + 0.0016479589976370335, + 0.001829458007705398, + 0.0016975829930743203, + 0.001680375004070811, + 0.001791040995158255, + 0.0017900420061778277, + 0.0017869590083137155, + 0.0017212079983437434, + 0.0017846249975264072, + 0.0017873750039143488, + 0.001770082992152311, + 0.0019771660008700565, + 0.0018446250032866374, + 0.0017717090086080134, + 0.0016387920040870085, + 0.00178208299621474, + 0.0018084579933201894, + 0.0017416670016245916, + 0.001777500001480803, + 0.001778124991687946, + 0.001768957998137921, + 0.0017734169960021973, + 0.0017773750005289912, + 0.0017240829911315814, + 0.0017442919925088063, + 0.0017572499928064644 + ] + }, + { + "forward": 0.005652250009006821, + "prefill": 0.0056638749956618994, + "decode_steps": [ + 0.002286000002641231, + 0.001998250008909963, + 0.001977500010980293, + 0.0017902499967021868, + 0.0019497920002322644, + 0.0019320410065120086, + 0.0018925420008599758, + 0.0016717499966034666, + 0.00173049999284558, + 0.0017430000007152557, + 0.0017280829924857244, + 0.001711040997179225, + 0.0018964580085594207, + 0.0017557079991092905, + 0.0017510829929960892, + 0.0017007080023176968, + 0.001766916990163736, + 0.0017429590079700574, + 0.0016698330000508577, + 0.001705125003354624, + 0.001694292004685849, + 0.001745500005199574, + 0.0017895419878186658, + 0.0017734579887473956, + 0.0017927090084413067, + 0.0016877079906407744, + 0.0017768750112736598, + 0.0018760829989332706, + 0.0018812090129358694, + 0.0019375409901840612, + 0.0018159169994760305, + 0.001749666000250727 + ] + }, + { + "forward": 0.005514582997420803, + "prefill": 0.005597082999884151, + "decode_steps": [ + 0.002109458000632003, + 0.001835999995819293, + 0.00170283300394658, + 0.0017957079980988055, + 0.0017285000067204237, + 0.0017936669901246205, + 0.0017865419940790161, + 0.0017388750129612163, + 0.0018147499940823764, + 0.0018118330044671893, + 0.001799249992473051, + 0.0018157919985242188, + 0.0018189590045949444, + 0.001817875003325753, + 0.001807249995181337, + 0.0018157089943997562, + 0.0018126669892808422, + 0.0018050000071525574, + 0.0017986250022659078, + 0.0017934170027729124, + 0.0017693749978207052, + 0.0018002089927904308, + 0.0018024579912889749, + 0.0018058339919662103, + 0.0017307079979218543, + 0.001894291999633424, + 0.0018129589880118147, + 0.0016981250082608312, + 0.0017595839890418574, + 0.001746458001434803, + 0.0017919590027304366, + 0.0017921249964274466 + ] + }, + { + "forward": 0.005464916990604252, + "prefill": 0.005605915997875854, + "decode_steps": [ + 0.0019760409923037514, + 0.0018017919937847182, + 0.001775625001755543, + 0.0016704169975128025, + 0.0017175410030176863, + 0.0017717500013532117, + 0.0017951670015463606, + 0.001664041992626153, + 0.001677332998951897, + 0.0017852500022854656, + 0.001773166994098574, + 0.001768041998730041, + 0.0017504170100437477, + 0.0016312499938067049, + 0.0019384589977562428, + 0.001768582995282486, + 0.0018595840083435178, + 0.0017101669945986941, + 0.002005125003051944, + 0.0018206670065410435, + 0.0016561670054215938, + 0.0019093750015599653, + 0.0018297920032637194, + 0.0018207910034107044, + 0.001775958007783629, + 0.001802291997591965, + 0.001764375003403984, + 0.0017822080117184669, + 0.0018406250019324943, + 0.0017260000022361055, + 0.0016691669879946858, + 0.001712957993731834 + ] + }, + { + "forward": 0.005668916011927649, + "prefill": 0.005586374987615272, + "decode_steps": [ + 0.002047792004304938, + 0.001720999993267469, + 0.0017165000026579946, + 0.0018630410049809143, + 0.0018420839915052056, + 0.0018755409982986748, + 0.0018276659975526854, + 0.001723249995848164, + 0.0017459999944549054, + 0.0016987089911708608, + 0.0017358749901177362, + 0.0017812080041039735, + 0.0017921250109793618, + 0.0017987500032177195, + 0.0017336669989163056, + 0.0018772079929476604, + 0.0017304580105701461, + 0.0017582089931238443, + 0.0017464589909650385, + 0.0018122499895980582, + 0.001694667007541284, + 0.0017837500054156408, + 0.001785707994713448, + 0.0016614590131212026, + 0.0017880420055007562, + 0.0016794999974081293, + 0.0017608750058570877, + 0.0017420000076526776, + 0.0016845000063767657, + 0.0019080000056419522, + 0.001967624993994832, + 0.0018716670019784942 + ] + }, + { + "forward": 0.005525625005248003, + "prefill": 0.005596041009994224, + "decode_steps": [ + 0.0019402500038268045, + 0.001846499988459982, + 0.0018299590010428801, + 0.0017725840007187799, + 0.0017608339985599741, + 0.0017768340039765462, + 0.001789500005543232, + 0.001766124987625517, + 0.0018083340110024437, + 0.0016746670007705688, + 0.0017770420090528205, + 0.0017415830079698935, + 0.0016747500048950315, + 0.0018790830072248355, + 0.0018313749897060916, + 0.0018189580005127937, + 0.001813541995943524, + 0.0018066250049741939, + 0.0018071669910568744, + 0.0017328329995507374, + 0.0018113330006599426, + 0.001793500006897375, + 0.0017979160038521513, + 0.0017829590069595724, + 0.0017102080018958077, + 0.0018320420058444142, + 0.0016931249992921948, + 0.0018125000060535967, + 0.0017905830027302727, + 0.00175275000219699, + 0.0017993340006796643, + 0.0017886659916257486 + ] + }, + { + "forward": 0.005523041996639222, + "prefill": 0.005627458987873979, + "decode_steps": [ + 0.001955166007974185, + 0.0017515000072307885, + 0.0017473330080974847, + 0.0017303750064456835, + 0.0016805000050226226, + 0.001701791989034973, + 0.001782874998752959, + 0.0017328749963780865, + 0.0016563329991186038, + 0.0016755419928813353, + 0.0017934159986907616, + 0.0016823329933686182, + 0.0017950000037671998, + 0.0017279999883612618, + 0.0017707499937387183, + 0.0016898750036489218, + 0.0016282500000670552, + 0.0018814579962054268, + 0.0018012090004049242, + 0.0016833749978104606, + 0.001708458992652595, + 0.001974832994164899, + 0.0018078749999403954, + 0.0017876659985631704, + 0.0017837079940363765, + 0.0016947089898167178, + 0.0017703329940559343, + 0.0017687920044409111, + 0.0017886249988805503, + 0.0017085839936044067, + 0.0017485410062363371, + 0.0017696659924695268 + ] + }, + { + "forward": 0.0068662080011563376, + "prefill": 0.0063107089954428375, + "decode_steps": [ + 0.0020129580079810694, + 0.0018086670024786144, + 0.0018304999975953251, + 0.0021648340043611825, + 0.0017492500046500936, + 0.0016942499933065847, + 0.0018735409976216033, + 0.0017077920056181028, + 0.0017478329973528162, + 0.001740916006383486, + 0.0017727499944157898, + 0.0017848330026026815, + 0.0017303749918937683, + 0.0016993329918477684, + 0.001932915998622775, + 0.0018465000030118972, + 0.001817583994125016, + 0.001646832999540493, + 0.0018847909959731624, + 0.001794083000277169, + 0.0016532080044271424, + 0.0016863330092746764, + 0.0016990830044960603, + 0.0018044589960481972, + 0.0017609170026844367, + 0.001647457989747636, + 0.0016962499939836562, + 0.0017760829941835254, + 0.0017779999907361344, + 0.001778292004019022, + 0.001818625009036623, + 0.0017789580015232787 + ] + }, + { + "forward": 0.00566870799229946, + "prefill": 0.005573166999965906, + "decode_steps": [ + 0.0021912500087637454, + 0.0017820420034695417, + 0.0018257920019095764, + 0.0016937500040512532, + 0.001816916003008373, + 0.0017812080041039735, + 0.001851084001827985, + 0.001708750001853332, + 0.0016812079993542284, + 0.0017400420038029552, + 0.0018024590099230409, + 0.0017109170003095642, + 0.001906208010041155, + 0.0018217909964732826, + 0.001713332996587269, + 0.0017892920004669577, + 0.0018055420077871531, + 0.0017405409889761358, + 0.0017008340073516592, + 0.001799624995328486, + 0.0016491669957758859, + 0.0018930410005850717, + 0.0017883340042317286, + 0.001989124997635372, + 0.0017194579995702952, + 0.001677332998951897, + 0.0016688749892637134, + 0.0019065410015173256, + 0.0017653749964665622, + 0.0017434170003980398, + 0.0017624159954721108, + 0.0016515419993083924 + ] + }, + { + "forward": 0.0056402080081170425, + "prefill": 0.005562874997849576, + "decode_steps": [ + 0.002002417008043267, + 0.0017829999997047707, + 0.0018000830023083836, + 0.0017777920002117753, + 0.001781500002834946, + 0.0017957079980988055, + 0.001721332999295555, + 0.0017494999920018017, + 0.001808041997719556, + 0.0017851659940788522, + 0.001745959001709707, + 0.001797666001948528, + 0.00185425000381656, + 0.0018101250025210902, + 0.0017814580060075969, + 0.0017758750036591664, + 0.0017927090084413067, + 0.001690667006187141, + 0.0019259170076111332, + 0.001976332991034724, + 0.001745290996041149, + 0.0018152919947169721, + 0.002016583996010013, + 0.0017844999965745956, + 0.0016740420105634257, + 0.001809083012631163, + 0.0017313330026809126, + 0.0017926250002346933, + 0.001798458004486747, + 0.0017562499997438863, + 0.0017448340076953173, + 0.0017515000072307885 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 5.599395502940752, + "p95": 6.327426606003427, + "min": 5.464916990604252, + "max": 6.866208001156338, + "iqr": 0.14090574768488295, + "mean": 5.708304001018405, + "cv": 0.07243067540970295 + }, + "prefill_ms": { + "count": 10, + "median": 5.596562004939187, + "p95": 6.0196336955414145, + "min": 5.541125006857328, + "max": 6.3107089954428375, + "iqr": 0.04560424349620007, + "mean": 5.666462497902103, + "cv": 0.040396477866269084 + }, + "decode_ms_per_step": { + "count": 320, + "median": 1.7812080041039735, + "p95": 1.9748933940718416, + "min": 1.6282500000670552, + "max": 2.286000002641231, + "iqr": 0.07985449337866157, + "mean": 1.7865108124624385, + "cv": 0.050111357068467755 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 560.0584950548384, + "p95": 566.4135663214043, + "min": 552.4858062010804, + "max": 569.6002771467946, + "iqr": 4.909676246326512, + "mean": 559.7853938643493, + "cv": 0.008359896375801987 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 509.75128031740417, + "p95": 515.4217091184447, + "min": 503.27193594894544, + "max": 517.7389755584252, + "iqr": 4.918977780566365, + "mean": 509.3040153625446, + "cv": 0.008388691160357467 + }, + "memory": { + "peak_extra_bytes": 5906432, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "reference", + "batch": 1, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.02079258298908826, + "prefill": 0.0174135830020532, + "decode_steps": [ + 0.0045532910007750615, + 0.005076458997791633, + 0.004433124995557591, + 0.004514458996709436, + 0.004355875003966503, + 0.004274208011338487, + 0.0041474590107100084, + 0.004181041003903374, + 0.00425133400131017, + 0.004084291998879053, + 0.004686374988523312, + 0.004421334000653587, + 0.004344708009739406, + 0.004086582994204946, + 0.004302041998016648, + 0.004561500012641773, + 0.004396999996970408, + 0.004318125007557683, + 0.004334583005402237, + 0.004313875004299916, + 0.004294708007364534, + 0.004288750002160668, + 0.004137332987738773, + 0.004247417004080489, + 0.004265458002919331, + 0.004196541995042935, + 0.004567625001072884, + 0.00445558299543336, + 0.004156249997322448, + 0.00439170899335295, + 0.0039263329963432625, + 0.004419249991769902 + ] + }, + { + "forward": 0.020988084012060426, + "prefill": 0.01746154199645389, + "decode_steps": [ + 0.005085207987576723, + 0.00447337499645073, + 0.004428624990396202, + 0.004059250000864267, + 0.004159541000262834, + 0.004239792004227638, + 0.004192708001937717, + 0.00419095900724642, + 0.004340833998867311, + 0.004227790996083058, + 0.004260000001522712, + 0.004091875001904555, + 0.004139249998843297, + 0.004751041007693857, + 0.005091042010462843, + 0.004576291990815662, + 0.00437112498912029, + 0.0044268749916227534, + 0.004396792006446049, + 0.004342625004937872, + 0.004507874997216277, + 0.004321375003200956, + 0.004262957998435013, + 0.004327125003328547, + 0.004351458992459811, + 0.0043701249960577115, + 0.004317750004702248, + 0.004303624999010935, + 0.0041989169985754415, + 0.004130250003072433, + 0.004012458011857234, + 0.004117208998650312 + ] + }, + { + "forward": 0.019620167004177347, + "prefill": 0.01763116699294187, + "decode_steps": [ + 0.004956333999871276, + 0.0040653750038472936, + 0.004401500002131797, + 0.004320583990192972, + 0.004300000000512227, + 0.004819292007596232, + 0.0040821250004228204, + 0.004458791998331435, + 0.004689749999670312, + 0.0044037909974576905, + 0.004337666992796585, + 0.004430458997376263, + 0.00431729199772235, + 0.004232874998706393, + 0.004252915998222306, + 0.004266792006092146, + 0.004413458998897113, + 0.00421295799606014, + 0.00431279199256096, + 0.004392457994981669, + 0.004155749993515201, + 0.004262250004103407, + 0.004134540999075398, + 0.00433783300104551, + 0.004193999993731268, + 0.004356582998298109, + 0.004388041008496657, + 0.004252374987117946, + 0.004363333006040193, + 0.0041478329949313775, + 0.004204625001875684, + 0.00430962500104215 + ] + }, + { + "forward": 0.020528749999357387, + "prefill": 0.018043625008431263, + "decode_steps": [ + 0.00496120800380595, + 0.004448666993994266, + 0.004525791009655222, + 0.004222417002893053, + 0.004312249991926365, + 0.004182166987447999, + 0.004124750004848465, + 0.0045031660120002925, + 0.004408000007970259, + 0.004208832993754186, + 0.0041413750004721805, + 0.004298166008084081, + 0.004390666988911107, + 0.00427037499321159, + 0.004181707990937866, + 0.004468833009013906, + 0.004402500009746291, + 0.0041188750037690625, + 0.004145832994254306, + 0.004229458994814195, + 0.004234708001604304, + 0.0042481660057092085, + 0.0041638330003479496, + 0.0042276250023860484, + 0.004597000006469898, + 0.004313291996368207, + 0.00431187498907093, + 0.004015790997073054, + 0.004269208991900086, + 0.004201457995804958, + 0.004188917009741999, + 0.004225374999805354 + ] + }, + { + "forward": 0.02076112500799354, + "prefill": 0.018226250002044253, + "decode_steps": [ + 0.005330667001544498, + 0.004550291996565647, + 0.004244249998009764, + 0.004573708996758796, + 0.0038650839996989816, + 0.004409667002619244, + 0.004288583004381508, + 0.003971750003984198, + 0.0041779159946599975, + 0.004010582997580059, + 0.0042864580027526245, + 0.00395895799738355, + 0.004393625000375323, + 0.0043807500042021275, + 0.004328708004322834, + 0.0044600419932976365, + 0.004717499992693774, + 0.005213041004026309, + 0.004520374990534037, + 0.0043147090036654845, + 0.004128707994823344, + 0.004639957987819798, + 0.00428812499740161, + 0.004368832989712246, + 0.004271208003046922, + 0.004843749993597157, + 0.004566290997900069, + 0.004575499988277443, + 0.003995125007349998, + 0.004372249997686595, + 0.003839750002953224, + 0.004234167005051859 + ] + }, + { + "forward": 0.019832041012705304, + "prefill": 0.017513458005851135, + "decode_steps": [ + 0.0046327080053742975, + 0.004249165998771787, + 0.004165042002568953, + 0.0048915410006884485, + 0.0044996669894317165, + 0.004442666991963051, + 0.0044001659989589825, + 0.004260916990460828, + 0.004270707999239676, + 0.0042022920097224414, + 0.00414308400650043, + 0.004531083002802916, + 0.004375042000901885, + 0.0042781249940162525, + 0.004274000006262213, + 0.004170708009041846, + 0.0043988329998683184, + 0.00431150000076741, + 0.004097875003935769, + 0.003956083004595712, + 0.004178834002232179, + 0.0041882499936036766, + 0.004263875001925044, + 0.004250833997502923, + 0.004168083003605716, + 0.004447250001248904, + 0.004212875006487593, + 0.004150416003540158, + 0.004213249994791113, + 0.004135124996537343, + 0.004332124997745268, + 0.0042823340045288205 + ] + }, + { + "forward": 0.020630250000976957, + "prefill": 0.01795875000243541, + "decode_steps": [ + 0.011575708005693741, + 0.005991416997858323, + 0.00468479200208094, + 0.00488887500250712, + 0.004386291999253444, + 0.004422875004820526, + 0.004765000005136244, + 0.004750708001665771, + 0.004704124992713332, + 0.004582790992571972, + 0.004527790995780379, + 0.004992582995328121, + 0.004268292002961971, + 0.004838958993786946, + 0.004513040999881923, + 0.0043364169978303835, + 0.004263167007593438, + 0.004169874999206513, + 0.004432583999005146, + 0.004580875000101514, + 0.0044144580024294555, + 0.004482042000745423, + 0.0044567919976543635, + 0.004442083998583257, + 0.0041565410065231845, + 0.004429790991707705, + 0.004415167000843212, + 0.00443879199156072, + 0.004372583003714681, + 0.004183541008387692, + 0.004104666993953288, + 0.004295083999750204 + ] + }, + { + "forward": 0.020793500007130206, + "prefill": 0.017377374999341555, + "decode_steps": [ + 0.004899915991700254, + 0.004368750000139698, + 0.004158167008426972, + 0.004287999996449798, + 0.004333042001235299, + 0.004313792000175454, + 0.004081624996615574, + 0.0045315830066101626, + 0.004150790991843678, + 0.004390999994939193, + 0.003963041992392391, + 0.004286166004021652, + 0.00414737498795148, + 0.00426362500002142, + 0.004116708994843066, + 0.004224167001666501, + 0.0044971250026719645, + 0.004438916992512532, + 0.004051458992762491, + 0.004223416995955631, + 0.004217709007207304, + 0.004254082989064045, + 0.004173292007180862, + 0.004228125006193295, + 0.004274417005944997, + 0.004302415996789932, + 0.004248541998094879, + 0.004090916001587175, + 0.004014749996713363, + 0.004111374990316108, + 0.0049226249975617975, + 0.004453959001693875 + ] + }, + { + "forward": 0.019708459003595635, + "prefill": 0.01727479101100471, + "decode_steps": [ + 0.005191125004785135, + 0.004315833008149639, + 0.0041992920014308766, + 0.004406709005706944, + 0.004368249996332452, + 0.004213709005853161, + 0.004178082992439158, + 0.004126416999497451, + 0.00438587499957066, + 0.004284499998902902, + 0.004003540991107002, + 0.004363207990536466, + 0.004376832992420532, + 0.004197208007099107, + 0.004329040995799005, + 0.0041516670025885105, + 0.00474666700756643, + 0.004164207988651469, + 0.0045737090113107115, + 0.004288457988877781, + 0.004313582991017029, + 0.0046475830022245646, + 0.004154374997597188, + 0.00480495800729841, + 0.003918125003110617, + 0.004620207997504622, + 0.00435691699385643, + 0.004377042001578957, + 0.004244291994837113, + 0.004469625011552125, + 0.004256667001754977, + 0.004437750001670793 + ] + }, + { + "forward": 0.02079562500875909, + "prefill": 0.017415583002730273, + "decode_steps": [ + 0.004832874998101033, + 0.004250041005434468, + 0.004183958997600712, + 0.004163707999396138, + 0.004170958010945469, + 0.004272666003089398, + 0.004603834007866681, + 0.0042744999955175444, + 0.004288042007829063, + 0.004409999994095415, + 0.00414437499421183, + 0.0041062089876504615, + 0.004284082999220118, + 0.004201249990728684, + 0.004215624998323619, + 0.004822457995032892, + 0.004505999997491017, + 0.004295625010854565, + 0.004161750010098331, + 0.004213750013150275, + 0.004265875002602115, + 0.004288333002477884, + 0.004244791998644359, + 0.004302999994251877, + 0.004339041988714598, + 0.004112833004910499, + 0.004042208005557768, + 0.004153582995058969, + 0.004420666009536944, + 0.004250250000040978, + 0.004329249990405515, + 0.004371957998955622 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 20.69568750448525, + "p95": 20.901477460574824, + "min": 19.620167004177347, + "max": 20.988084012060426, + "iqr": 0.7870524932513945, + "mean": 20.445058404584415, + "cv": 0.02525767999639271 + }, + "prefill_ms": { + "count": 10, + "median": 17.48750000115251, + "p95": 18.144068754918408, + "min": 17.27479101100471, + "max": 18.226250002044253, + "iqr": 0.46277124783955514, + "mean": 17.631612402328756, + "cv": 0.01851783003061253 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.299083004298154, + "p95": 4.8391985437774565, + "min": 3.839750002953224, + "max": 11.575708005693741, + "iqr": 0.23140599660109729, + "mean": 4.367935540267354, + "cv": 0.10835145555697248 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 230.41601886123, + "p95": 233.27055913003687, + "min": 210.7097039421556, + "max": 233.54105383063236, + "iqr": 3.0673489731966868, + "mean": 229.1294967420531, + "cv": 0.02921756164527095 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 204.60137245304037, + "p95": 206.96138268759702, + "min": 188.42764660787083, + "max": 207.25623247492078, + "iqr": 2.0893775546823576, + "mean": 203.4379050968766, + "cv": 0.026949173377162874 + }, + "memory": { + "peak_extra_bytes": 64888832, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "auto", + "batch": 1, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.017454291999456473 + }, + { + "forward": 0.01751695800339803 + }, + { + "forward": 0.01749237500189338 + }, + { + "forward": 0.017485582997323945 + }, + { + "forward": 0.01739287500095088 + }, + { + "forward": 0.01745183400635142 + }, + { + "forward": 0.017412625005817972 + }, + { + "forward": 0.01758466700266581 + }, + { + "forward": 0.017423042008886114 + }, + { + "forward": 0.01741629200114403 + } + ], + "forward_ms": { + "count": 10, + "median": 17.453063002903946, + "p95": 17.55419795299531, + "min": 17.39287500095088, + "max": 17.58466700266581, + "iqr": 0.07269749767147005, + "mean": 17.463054302788805, + "cv": 0.003334715452114283 + }, + "memory": { + "peak_extra_bytes": 14696448, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "sdpa", + "batch": 1, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.011217042003409006, + "prefill": 0.01125629199668765, + "decode_steps": [ + 0.0021270830038702115, + 0.00181200000224635, + 0.0019019590108655393, + 0.0019210829923395067, + 0.0019321669969940558, + 0.0019419580057729036, + 0.0019296249956823885, + 0.0019149590079905465, + 0.0018922500021290034, + 0.0018184170039603487, + 0.0018096249987138435, + 0.001818874996388331, + 0.001824209000915289, + 0.0018925829936051741, + 0.0018678749911487103, + 0.0018517499993322417, + 0.001813541995943524, + 0.0017912500043166801, + 0.0020476670033531263, + 0.002013625009567477, + 0.001863208002760075, + 0.0018062499875668436, + 0.0024024580052355304, + 0.0019372090027900413, + 0.0019718330004252493, + 0.0021090000082040206, + 0.0019314999954076484, + 0.0017721250042086467, + 0.0018847499886760488, + 0.001973708000150509, + 0.0019326250039739534, + 0.0019294590019853786 + ] + }, + { + "forward": 0.011264959000982344, + "prefill": 0.011112749998574145, + "decode_steps": [ + 0.0023289160017156973, + 0.002102250000461936, + 0.0019612500036600977, + 0.00199554099526722, + 0.001981583991437219, + 0.0018522909958846867, + 0.002072916002362035, + 0.0019437079899944365, + 0.0019525410025380552, + 0.0018685000104596838, + 0.0019034580036532134, + 0.0019343329913681373, + 0.0018857079994631931, + 0.0019541250076144934, + 0.0021315830090316013, + 0.002003375004278496, + 0.0017663750040810555, + 0.0019795839907601476, + 0.0017862090026028454, + 0.002015541002037935, + 0.0020390840072650462, + 0.001824749997467734, + 0.0019193749903934076, + 0.0018740420055110008, + 0.0018619580077938735, + 0.0019390829984331504, + 0.0018154589924961329, + 0.0018572919943835586, + 0.001964416995178908, + 0.0018391659978078678, + 0.0019340410071890801, + 0.0019420000026002526 + ] + }, + { + "forward": 0.011492124991491437, + "prefill": 0.011116499998024665, + "decode_steps": [ + 0.002887791008106433, + 0.0018748340080492198, + 0.0018874999950639904, + 0.0018793750059558079, + 0.0017569170013302937, + 0.0020188750058878213, + 0.001954290986759588, + 0.0019481250055832788, + 0.0017996659880736843, + 0.0018742499960353598, + 0.0018750830058706924, + 0.0019276250095572323, + 0.0018408750038361177, + 0.005446708993986249, + 0.00625920899619814, + 0.0036609999951906502, + 0.004909208000754006, + 0.0026717919972725213, + 0.0019296670070616528, + 0.0018873329972848296, + 0.0019447920058155432, + 0.0018347919976804405, + 0.0019710830092662945, + 0.0019467079982860014, + 0.0019477500027278438, + 0.001822874997742474, + 0.0018700840009842068, + 0.0019335830002091825, + 0.0019401249883230776, + 0.002173625005525537, + 0.0020124580041738227, + 0.0019394999981159344 + ] + }, + { + "forward": 0.011180458997841924, + "prefill": 0.011269833004917018, + "decode_steps": [ + 0.002243374998215586, + 0.0019264169968664646, + 0.001938040993991308, + 0.0019040839979425073, + 0.0019036250014323741, + 0.0018677909974940121, + 0.0019439169991528615, + 0.001941665992490016, + 0.0019004580099135637, + 0.001852333007263951, + 0.0018693750025704503, + 0.0019232080085203052, + 0.0019552910089259967, + 0.0018469170026946813, + 0.001888333004899323, + 0.001946124990354292, + 0.0018365830037510023, + 0.001917542002047412, + 0.0017484580021118745, + 0.0018769169982988387, + 0.0018968329968629405, + 0.001851207998697646, + 0.0017724579956848174, + 0.001892375003080815, + 0.0019373329996597022, + 0.001820584002416581, + 0.0019804589974228293, + 0.001960333000170067, + 0.001954624996869825, + 0.0018370000034337863, + 0.0018702910019783303, + 0.0019358330027898774 + ] + }, + { + "forward": 0.011290042006294243, + "prefill": 0.011203249989193864, + "decode_steps": [ + 0.002217667002696544, + 0.001995999991777353, + 0.0019757080008275807, + 0.002488292011548765, + 0.0020817500044358894, + 0.00184304200229235, + 0.0019077080069109797, + 0.0019465000077616423, + 0.0019456250010989606, + 0.0018057919951388612, + 0.0022543329978361726, + 0.001817875003325753, + 0.0018453749944455922, + 0.0018133330013370141, + 0.0019021250045625493, + 0.0018667499971343204, + 0.0018694589962251484, + 0.001957166998181492, + 0.0018580410105641931, + 0.0017812080041039735, + 0.001975999999558553, + 0.001821124998969026, + 0.0019555420003598556, + 0.0019272089994046837, + 0.0019446249934844673, + 0.0018362499977229163, + 0.0018351250037085265, + 0.0019642080005723983, + 0.0018339999951422215, + 0.0018355420033913106, + 0.001831166009651497, + 0.0019036669982597232 + ] + }, + { + "forward": 0.011175833002198488, + "prefill": 0.011257457997999154, + "decode_steps": [ + 0.002249083001515828, + 0.001913166997837834, + 0.001881125004729256, + 0.0017555829981574789, + 0.0018656250031199306, + 0.0019459589966572821, + 0.0018372089980402961, + 0.0018982089968631044, + 0.0018552909896243364, + 0.0018778749945340678, + 0.001859916010289453, + 0.001913083004183136, + 0.0018450420029694214, + 0.0019133750029141083, + 0.0019388750079087913, + 0.0018563749908935279, + 0.00183275000017602, + 0.0019317499973112717, + 0.0017696249997243285, + 0.002304541994817555, + 0.001907541009131819, + 0.0019265000009909272, + 0.001880125011666678, + 0.0018715829937718809, + 0.0018406250019324943, + 0.001938040993991308, + 0.0018201670027337968, + 0.0019753749947994947, + 0.0018019590061157942, + 0.0019122079975204542, + 0.0019265419978182763, + 0.0018786250002449378 + ] + }, + { + "forward": 0.01119241699052509, + "prefill": 0.011230165997403674, + "decode_steps": [ + 0.0021930840011918917, + 0.0018465419998392463, + 0.0018693750025704503, + 0.0019701670098584145, + 0.001922666997415945, + 0.0019419999880483374, + 0.0019345000036992133, + 0.001933874998940155, + 0.0018493329989723861, + 0.0019049580005230382, + 0.0018818329990608618, + 0.0018395000079181045, + 0.0018223329971078783, + 0.002006250011618249, + 0.0018674169987207279, + 0.0018637500033946708, + 0.0018732919998001307, + 0.0019086249958490953, + 0.001851792010711506, + 0.0018487920024199411, + 0.0018177909951191396, + 0.0019932499999413267, + 0.0019358749996172264, + 0.0018597919988678768, + 0.00183183400076814, + 0.001907000012579374, + 0.0019154999899910763, + 0.0018286669946974143, + 0.0018010000057984143, + 0.00183199999446515, + 0.0018296250054845586, + 0.0018115409911843017 + ] + }, + { + "forward": 0.011261041989200749, + "prefill": 0.011171791004016995, + "decode_steps": [ + 0.0022674999927403405, + 0.0020127089956076816, + 0.001898292000987567, + 0.0018526670028222725, + 0.0018347499863011762, + 0.0018549999949755147, + 0.0019129169959342107, + 0.0019188750011380762, + 0.0019513339939294383, + 0.0019326249894220382, + 0.001936915999976918, + 0.0018772499897750095, + 0.0018535830022301525, + 0.0019583329994929954, + 0.001844041995354928, + 0.0018314579938305542, + 0.0018360410031164065, + 0.0018009589985013008, + 0.001916167006129399, + 0.0019128749991068617, + 0.0018522499885875732, + 0.0018717909988481551, + 0.0018671249999897555, + 0.0018224590021418408, + 0.0018609999970067292, + 0.00193237500207033, + 0.0019394580012885854, + 0.0018333750049350783, + 0.0017766660021152347, + 0.0023777500027790666, + 0.001822250007535331, + 0.001788957990356721 + ] + }, + { + "forward": 0.011239750005188398, + "prefill": 0.011155083004268818, + "decode_steps": [ + 0.0023249579971889034, + 0.0020139590051257983, + 0.0019089999987045303, + 0.0019979999924544245, + 0.0019536249892553315, + 0.0019524580129655078, + 0.0018951249949168414, + 0.001901916999486275, + 0.0019331249932292849, + 0.0019474169966997579, + 0.0018324999982723966, + 0.0018398329993942752, + 0.0019248329917900264, + 0.0018789159948937595, + 0.0018665410025278106, + 0.0018773330084513873, + 0.0018636250024428591, + 0.0017461250099586323, + 0.0019115000031888485, + 0.0018673750018933788, + 0.0018709580035647377, + 0.0019357080018380657, + 0.0018444159941282123, + 0.0019223330018576235, + 0.001930625003296882, + 0.001939458990818821, + 0.0018220830097561702, + 0.0018205409869551659, + 0.001836083989473991, + 0.0018397090025246143, + 0.001806667001801543, + 0.0018346249999012798 + ] + }, + { + "forward": 0.011205791990505531, + "prefill": 0.011197291998541914, + "decode_steps": [ + 0.0021624999935738742, + 0.001976208994165063, + 0.0019437080045463517, + 0.0017506669973954558, + 0.002223291012342088, + 0.0018674580060178414, + 0.001796750002540648, + 0.0018469579954398796, + 0.0019195000058971345, + 0.001984582995646633, + 0.001849583990406245, + 0.0018330840102862567, + 0.0018917919951491058, + 0.001931042002979666, + 0.00179312500404194, + 0.0019482079951558262, + 0.0018197079916717485, + 0.0019317910046083853, + 0.0018442500004312024, + 0.001803834005841054, + 0.0019217090011807159, + 0.0018340839887969196, + 0.0018073749961331487, + 0.002009832998737693, + 0.0018665829993551597, + 0.0018745419947663322, + 0.0018669169949134812, + 0.0018035000102827325, + 0.00193516600120347, + 0.0017317500023636967, + 0.00249650000478141, + 0.004989416993339546 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 11.228396004298702, + "p95": 11.4011876481527, + "min": 11.175833002198488, + "max": 11.492124991491437, + "iqr": 0.06821900751674548, + "mean": 11.251946097763721, + "cv": 0.008238267274556352 + }, + "prefill_ms": { + "count": 10, + "median": 11.200270993867889, + "p95": 11.26426425180398, + "min": 11.112749998574145, + "max": 11.269833004917018, + "iqr": 0.09050049266079441, + "mean": 11.19704149896279, + "cv": 0.0051205137092188565 + }, + "decode_ms_per_step": { + "count": 320, + "median": 1.9011875046999194, + "p95": 2.2493455013318453, + "min": 1.7317500023636967, + "max": 6.25920899619814, + "iqr": 0.1008022554742638, + "mean": 1.9673025938118371, + "cv": 0.21572869344717066 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 522.9422953373091, + "p95": 527.6191858976854, + "min": 423.1265081211257, + "max": 528.9784301018511, + "iqr": 10.179053545344004, + "mean": 510.40121944870623, + "cv": 0.06258019113638379 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 441.68109099974436, + "p95": 445.3945011331445, + "min": 368.90159543674633, + "max": 446.15392028537457, + "iqr": 6.8482480688548435, + "mean": 432.8427115746661, + "cv": 0.05393521187428928 + }, + "memory": { + "peak_extra_bytes": 12599296, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "reference", + "batch": 1, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.060158249994856305, + "prefill": 0.05644687499443535, + "decode_steps": [ + 0.005634957997244783, + 0.004664583000703715, + 0.004561541005386971, + 0.004492207997827791, + 0.004404999999678694, + 0.004396666001412086, + 0.004362583000329323, + 0.004313458004617132, + 0.0043603330123005435, + 0.004408249995321967, + 0.004273458005627617, + 0.004259249995811842, + 0.004149958011112176, + 0.004318374994909391, + 0.00437450000026729, + 0.004374166994239204, + 0.00438366700836923, + 0.004271708006854169, + 0.004348583999671973, + 0.004281415996956639, + 0.004754792011226527, + 0.004523541996604763, + 0.004445500002475455, + 0.004458540992345661, + 0.0045740829955320805, + 0.004362165986094624, + 0.004379375008284114, + 0.0043741249974118546, + 0.004516749992035329, + 0.004575333005050197, + 0.004356709003332071, + 0.004320416992413811 + ] + }, + { + "forward": 0.06013483299466316, + "prefill": 0.056759665996651165, + "decode_steps": [ + 0.0060933749919058755, + 0.004957041994202882, + 0.004554958999506198, + 0.004623374989023432, + 0.004448750012670644, + 0.004416124997078441, + 0.00433016600436531, + 0.004403041995828971, + 0.004382667000754736, + 0.004409832996316254, + 0.004074417010997422, + 0.0043994170118821785, + 0.0043347910104785115, + 0.0044457079929998145, + 0.004332458003773354, + 0.0042059159895870835, + 0.004681584003265016, + 0.0043594590097200125, + 0.004511749997618608, + 0.004417959004058503, + 0.004188124992651865, + 0.004160083990427665, + 0.004981334001058713, + 0.004539292000117712, + 0.005277500007650815, + 0.004849791992455721, + 0.004537959001027048, + 0.00476720898586791, + 0.004115999996429309, + 0.004728582993266173, + 0.004527625002083369, + 0.004599499996402301 + ] + }, + { + "forward": 0.05911787498916965, + "prefill": 0.05638583299878519, + "decode_steps": [ + 0.00515991699649021, + 0.004572707999614067, + 0.004539457993814722, + 0.0043946670048171654, + 0.004666374996304512, + 0.004084291998879053, + 0.0043811670038849115, + 0.004461084012291394, + 0.004421749996254221, + 0.004429332999279723, + 0.0038704580074409023, + 0.004273792001185939, + 0.0042926249880110845, + 0.0042992080125259236, + 0.004398834003950469, + 0.004403750004712492, + 0.004331167001510039, + 0.0043517909944057465, + 0.004294333004509099, + 0.0043070840038126335, + 0.004326374997617677, + 0.004959624988259748, + 0.004397540993522853, + 0.004528792007477023, + 0.004975082993041724, + 0.004646125002182089, + 0.0043661249947035685, + 0.004527875003986992, + 0.0041524159896653146, + 0.004456666996702552, + 0.004406209001899697, + 0.004295708000427112 + ] + }, + { + "forward": 0.05995158299629111, + "prefill": 0.05713724999804981, + "decode_steps": [ + 0.005368917001760565, + 0.004688208005973138, + 0.004545416988548823, + 0.004383042003610171, + 0.004380458005471155, + 0.00438141699123662, + 0.004310624994104728, + 0.004402500009746291, + 0.004417791002197191, + 0.004312791003030725, + 0.004345916997408494, + 0.0042505409946898, + 0.0043228330032434314, + 0.0043442499882075936, + 0.004277958010789007, + 0.004414334005559795, + 0.004366207998828031, + 0.004823583003599197, + 0.00460270800977014, + 0.004885041998932138, + 0.004101541009731591, + 0.00475337500392925, + 0.004834541003219783, + 0.004460208001546562, + 0.004719250006019138, + 0.004018374995212071, + 0.004341166000813246, + 0.004389291992993094, + 0.004285292001441121, + 0.004452458000741899, + 0.0045015000068815425, + 0.004295749997254461 + ] + }, + { + "forward": 0.06090908400074113, + "prefill": 0.05699500000628177, + "decode_steps": [ + 0.006441499994252808, + 0.005322708995663561, + 0.004638541999156587, + 0.004514415995799936, + 0.004321415995946154, + 0.004392082992126234, + 0.0046931669930927455, + 0.00438683399988804, + 0.004131166992010549, + 0.004549584002234042, + 0.004443582991370931, + 0.004837959000724368, + 0.004575750004732981, + 0.00441879199934192, + 0.005024459009291604, + 0.004494249995332211, + 0.004314416000852361, + 0.0043404169991845265, + 0.0044308749929768965, + 0.004104708001250401, + 0.004433875001268461, + 0.004034249999676831, + 0.004392166010802612, + 0.004289250005967915, + 0.004331999996793456, + 0.0046642090019304305, + 0.004188791004708037, + 0.004581165994750336, + 0.004293000005418435, + 0.004250666999723762, + 0.004350416988017969, + 0.004295000006095506 + ] + }, + { + "forward": 0.05983958300203085, + "prefill": 0.05704425000294577, + "decode_steps": [ + 0.005034791000070982, + 0.004704457998741418, + 0.004446207996807061, + 0.004335959005402401, + 0.004329208008130081, + 0.004305959009798244, + 0.004292917001293972, + 0.00470270799996797, + 0.004155916001764126, + 0.004528667006525211, + 0.004485625002416782, + 0.004298625004594214, + 0.004443582991370931, + 0.004402332997415215, + 0.004311583004891872, + 0.00434104198939167, + 0.004326958005549386, + 0.00399558299977798, + 0.004962708000675775, + 0.004419958000653423, + 0.0050285420002182946, + 0.004115750009077601, + 0.004595457998220809, + 0.004573749989503995, + 0.004328709008404985, + 0.004438374991877936, + 0.00446833400928881, + 0.004422166995937005, + 0.004320332998759113, + 0.004244499999913387, + 0.0041440839995630085, + 0.004469749997952022 + ] + }, + { + "forward": 0.06034137500682846, + "prefill": 0.05856520800443832, + "decode_steps": [ + 0.006814707987359725, + 0.00596275000134483, + 0.005993375001708046, + 0.006127166998339817, + 0.006311541001196019, + 0.006644291992415674, + 0.006623249995755032, + 0.006371458002831787, + 0.005954290987574495, + 0.006066124988137744, + 0.005136208012118004, + 0.004982917002053, + 0.004940457991324365, + 0.004557374995783903, + 0.0046715829957975075, + 0.004545250005321577, + 0.004400916994200088, + 0.004300165994209237, + 0.004392916991491802, + 0.004239665999193676, + 0.004387458000564948, + 0.004314167003030889, + 0.0043923750054091215, + 0.004400417004944757, + 0.004322541994042695, + 0.00427091600431595, + 0.004366749999462627, + 0.004298749990994111, + 0.0043226670095464215, + 0.004405832994962111, + 0.004199957998935133, + 0.004412124995724298 + ] + }, + { + "forward": 0.060853207993204705, + "prefill": 0.05711233400506899, + "decode_steps": [ + 0.005771333002485335, + 0.004680334008298814, + 0.004444375008461066, + 0.004414542010636069, + 0.004475167006603442, + 0.004398875011247583, + 0.004343584005255252, + 0.004656666002119891, + 0.0041017080075107515, + 0.004648749993066303, + 0.004456124996067956, + 0.004321540996897966, + 0.004390292000607587, + 0.004489083992666565, + 0.004377791992737912, + 0.004356417004601099, + 0.004390458998386748, + 0.004740999997011386, + 0.004409209010191262, + 0.00428141700103879, + 0.004909957991912961, + 0.004641750012524426, + 0.004535665997536853, + 0.00440458299999591, + 0.004591749995597638, + 0.0044369160023052245, + 0.004223208001349121, + 0.004509874997893348, + 0.00432866599294357, + 0.004364833002910018, + 0.004416040988871828, + 0.0044862920040031895 + ] + }, + { + "forward": 0.05932637500518467, + "prefill": 0.056556458002887666, + "decode_steps": [ + 0.004942416999256238, + 0.004563540991512127, + 0.004413417002069764, + 0.0044154160132166, + 0.004360916995210573, + 0.004280499997548759, + 0.004334415993071161, + 0.00433654201333411, + 0.004348959002527408, + 0.004307291004806757, + 0.004386291999253444, + 0.0042316250037401915, + 0.004348667003796436, + 0.004796416003955528, + 0.004209582999465056, + 0.005048749997513369, + 0.0050490829889895394, + 0.004881665998254903, + 0.004571249999571592, + 0.004664834006689489, + 0.004124083003262058, + 0.004484332996071316, + 0.004535082989605144, + 0.004484000004595146, + 0.004407542000990361, + 0.004237834000377916, + 0.004323000001022592, + 0.004273124999599531, + 0.004389957990497351, + 0.004429458000231534, + 0.004374208001536317, + 0.004390707996208221 + ] + }, + { + "forward": 0.06784879199403804, + "prefill": 0.05741308299184311, + "decode_steps": [ + 0.006342042004689574, + 0.004883959001745097, + 0.004556458006845787, + 0.004624667009920813, + 0.004399124998599291, + 0.004505834003794007, + 0.004693666996899992, + 0.004373874995508231, + 0.0043632500019157305, + 0.004227541998261586, + 0.005856417003087699, + 0.0046108329988783225, + 0.004648916990845464, + 0.004419832999701612, + 0.00442720900173299, + 0.004327458009356633, + 0.0043769999901996925, + 0.004367832996649668, + 0.004288584008463658, + 0.0043857080017914996, + 0.004535916988970712, + 0.004569791999529116, + 0.004498999987845309, + 0.004295666003599763, + 0.004224292002618313, + 0.004335125006036833, + 0.004387000008136965, + 0.004434750007931143, + 0.0045439999958034605, + 0.004611582990037277, + 0.004418875003466383, + 0.004333541000960395 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 60.14654149475973, + "p95": 64.72592339705442, + "min": 59.11787498916965, + "max": 67.84879199403804, + "iqr": 0.8576667460147291, + "mean": 60.84809579770081, + "cv": 0.04149836894773207 + }, + "prefill_ms": { + "count": 10, + "median": 57.01962500461377, + "p95": 58.046751748770475, + "min": 56.38583299878519, + "max": 58.56520800443832, + "iqr": 0.5237609984760638, + "mean": 57.041595700138714, + "cv": 0.011047956149269435 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.406875501445029, + "p95": 5.382219051534778, + "min": 3.8704580074409023, + "max": 6.814707987359725, + "iqr": 0.24388523888774216, + "mean": 4.541501934363623, + "cv": 0.09756190055785975 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 223.01837914969386, + "p95": 225.3944500893494, + "min": 198.59694739547965, + "max": 225.39602681700686, + "iqr": 4.509371041183755, + "mean": 220.47512607001119, + "cv": 0.03626212664254473 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 159.5124471800836, + "p95": 161.19110717538456, + "min": 145.65609380070313, + "max": 161.32433811659544, + "iqr": 2.4167052318303206, + "mean": 158.2556686870317, + "cv": 0.0291400302613928 + }, + "memory": { + "peak_extra_bytes": 255606784, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "auto", + "batch": 4, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.003525540989357978 + }, + { + "forward": 0.0036997499992139637 + }, + { + "forward": 0.004350792005425319 + }, + { + "forward": 0.0034246659924974665 + }, + { + "forward": 0.0035730829986277968 + }, + { + "forward": 0.005125040988787077 + }, + { + "forward": 0.003533833005349152 + }, + { + "forward": 0.00377633300377056 + }, + { + "forward": 0.005084666001494043 + }, + { + "forward": 0.003535291994921863 + } + ], + "forward_ms": { + "count": 10, + "median": 3.6364164989208803, + "p95": 5.106872244505212, + "min": 3.4246659924974665, + "max": 5.125040988787077, + "iqr": 0.6729795022692997, + "mean": 3.962899697944522, + "cv": 0.16532804129954246 + }, + "memory": { + "peak_extra_bytes": 2950144, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "sdpa", + "batch": 4, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.003001374992891215, + "prefill": 0.003111500001978129, + "decode_steps": [ + 0.0019257079984527081, + 0.0018717500061029568, + 0.001878458002465777, + 0.0018874169909395278, + 0.0017734159919200465, + 0.0017838749918155372, + 0.0017943749990081415, + 0.001776416989741847, + 0.0017849580035544932, + 0.0017541249981150031, + 0.0017883330001495779, + 0.0018567499937489629, + 0.0018200830090790987, + 0.0018131249962607399, + 0.0017677499999990687, + 0.0018505830084905028, + 0.001825500003178604, + 0.0019034579891012982, + 0.0018195419979747385, + 0.0018974160047946498, + 0.0019042920030187815, + 0.0019092500006081536, + 0.001787083994713612, + 0.0019110419962089509, + 0.0019308750052005053, + 0.0018548330117482692, + 0.001913458007038571, + 0.001808000000892207, + 0.0018125000060535967, + 0.0017814999882830307, + 0.0017729590035742149, + 0.001869290994363837 + ] + }, + { + "forward": 0.003226708999136463, + "prefill": 0.0032357499876525253, + "decode_steps": [ + 0.0019893749995389953, + 0.001824209000915289, + 0.0019475419976515695, + 0.001846791012212634, + 0.0019262090063421056, + 0.001747250003973022, + 0.0018787500011967495, + 0.0019220000103814527, + 0.001990249991649762, + 0.0019119170028716326, + 0.0018536249990575016, + 0.001934625004651025, + 0.0019530419958755374, + 0.001837500007241033, + 0.0018764999986160547, + 0.0018101669993484393, + 0.0017506669973954558, + 0.0018060409929603338, + 0.0018035000102827325, + 0.0017980840057134628, + 0.0018379579996690154, + 0.001828749998821877, + 0.0018167909875046462, + 0.001831208006478846, + 0.0018519590084906667, + 0.001851958004408516, + 0.0019014160061487928, + 0.001911583007313311, + 0.0018029170023510233, + 0.0019806249911198393, + 0.0019253329955972731, + 0.0019379579898668453 + ] + }, + { + "forward": 0.0035094999911962077, + "prefill": 0.003121917005046271, + "decode_steps": [ + 0.0020271249959478155, + 0.0019004579953616485, + 0.0018480839935364202, + 0.0018092920072376728, + 0.0017945419967873022, + 0.0020035839988850057, + 0.001854499991168268, + 0.0017670829984126613, + 0.0018377500091446564, + 0.0019212500046705827, + 0.0019258749962318689, + 0.001895084002171643, + 0.0018789579917211086, + 0.0018966669886140153, + 0.001887582999188453, + 0.0017511250043753535, + 0.001774124990333803, + 0.0018828339962055907, + 0.0018564579950179905, + 0.0020046249992446974, + 0.0019489999976940453, + 0.001793917006580159, + 0.0019460000039543957, + 0.0018333750049350783, + 0.0017759579932317138, + 0.0018815830117091537, + 0.0018110829987563193, + 0.0019391249952604994, + 0.001825707993702963, + 0.0019165419944329187, + 0.001898040995001793, + 0.0017736250010784715 + ] + }, + { + "forward": 0.0030515410035150126, + "prefill": 0.003109750003204681, + "decode_steps": [ + 0.0019326249894220382, + 0.0018861249991459772, + 0.0018077910062856972, + 0.0018769580055959523, + 0.0019100000063190237, + 0.0019110419962089509, + 0.0017781250062398612, + 0.0019258749962318689, + 0.0018700420041568577, + 0.001760624989401549, + 0.0017719579918775707, + 0.0018387500022072345, + 0.0017544579895911738, + 0.0017625000036787242, + 0.001811666996218264, + 0.0018140839965781197, + 0.0017932910122908652, + 0.001757958991220221, + 0.0018094169936375692, + 0.0018509590008761734, + 0.001994999998714775, + 0.0018451249925419688, + 0.001797458011424169, + 0.0019396670104470104, + 0.0018844170117517933, + 0.006670583999948576, + 0.004509082995355129, + 0.003870249987812713, + 0.003977542000939138, + 0.0028089170082239434, + 0.002142083991202526, + 0.0024912500084610656 + ] + }, + { + "forward": 0.003092625003773719, + "prefill": 0.003104834002442658, + "decode_steps": [ + 0.0019511659920681268, + 0.0017840829968918115, + 0.0019506659882608801, + 0.0017985000013140962, + 0.0018752079922705889, + 0.0019105420069536194, + 0.0018202919891336933, + 0.0019475000008242205, + 0.001924915995914489, + 0.0019075839954894036, + 0.0018982080073328689, + 0.0017733749991748482, + 0.001884708006400615, + 0.001993541998672299, + 0.001835291986935772, + 0.0018482080049579963, + 0.0018433750083204359, + 0.0019446250080363825, + 0.0018696669867495075, + 0.0018130000098608434, + 0.0019053750002058223, + 0.001859249998233281, + 0.00188079199870117, + 0.0019185830024071038, + 0.001909833008539863, + 0.0019264580041635782, + 0.0019383750041015446, + 0.0017166250036098063, + 0.0019034580036532134, + 0.0018262079975102097, + 0.00182079199294094, + 0.0018096249987138435 + ] + }, + { + "forward": 0.004319041996495798, + "prefill": 0.003926541001419537, + "decode_steps": [ + 0.0022296249953797087, + 0.0023105420114006847, + 0.0017075000068871304, + 0.0017585000023245811, + 0.001772000003256835, + 0.0017838749918155372, + 0.0017584169982001185, + 0.0019314579985802993, + 0.0019452080014161766, + 0.0018769169982988387, + 0.0017737919988576323, + 0.0018581250042188913, + 0.0018096249987138435, + 0.0018687920091906562, + 0.0018999159947270527, + 0.001754083001287654, + 0.0019035000004805624, + 0.0018935830012196675, + 0.0017727499944157898, + 0.0018089170043822378, + 0.0018878749979194254, + 0.0018739590013865381, + 0.0017978749965550378, + 0.0018869170016841963, + 0.0017552079953020439, + 0.0019081659993389621, + 0.0017835420003393665, + 0.0017632500093895942, + 0.002247374999569729, + 0.002045665998593904, + 0.002478957991115749, + 0.001954291990841739 + ] + }, + { + "forward": 0.0032586250017629936, + "prefill": 0.003106542004388757, + "decode_steps": [ + 0.0018612499989103526, + 0.001685666007688269, + 0.002022541011683643, + 0.001930291997268796, + 0.0019719170086318627, + 0.0018064159958157688, + 0.0018290000007255003, + 0.0018176669982494786, + 0.0017656249983701855, + 0.0018209580011898652, + 0.0018932079983642325, + 0.0017919159872690216, + 0.0019520000059856102, + 0.001812375005101785, + 0.0019114590104436502, + 0.0019140420045005158, + 0.0018137080041924492, + 0.001891624997369945, + 0.0017812920123105869, + 0.0019886250083800405, + 0.002025499998126179, + 0.0017586250032763928, + 0.001832374997320585, + 0.0018097919964930043, + 0.0018349579913774505, + 0.0018773750052787364, + 0.001855208000051789, + 0.0018584170029498637, + 0.0019133339956169948, + 0.0017892500036396086, + 0.0019035829900531098, + 0.0019397500000195578 + ] + }, + { + "forward": 0.003285416998551227, + "prefill": 0.0030492920050164685, + "decode_steps": [ + 0.0018827500025508925, + 0.002066415996523574, + 0.0020634999964386225, + 0.0017714580026222393, + 0.0019044580112677068, + 0.0018362910050200298, + 0.0018040840077446774, + 0.0017486660071881488, + 0.0019373749964870512, + 0.001856584000051953, + 0.0018366250005783513, + 0.0017810419958550483, + 0.0018430000054650009, + 0.001888708007754758, + 0.0019031250121770427, + 0.0017462079995311797, + 0.0019739160052267835, + 0.0019430840038694441, + 0.001931249993504025, + 0.0018220419879071414, + 0.0019073339935857803, + 0.0018697500054258853, + 0.0019212080078432336, + 0.001906292003695853, + 0.0018212090071756393, + 0.0018383329879725352, + 0.001819459008402191, + 0.0017665420018602163, + 0.0019370419904589653, + 0.0018852920038625598, + 0.001759999999194406, + 0.0018469170026946813 + ] + }, + { + "forward": 0.0032818750041769817, + "prefill": 0.0030948339990573004, + "decode_steps": [ + 0.0020365410018712282, + 0.0018549579981481656, + 0.0019958750053774565, + 0.001896959001896903, + 0.0018152089905925095, + 0.0018543749902164564, + 0.0018510830122977495, + 0.0018922500021290034, + 0.0018949579971376806, + 0.0017833750025602058, + 0.0020034579938510433, + 0.0019049999973503873, + 0.0018814159993780777, + 0.0018218749901279807, + 0.0018896670080721378, + 0.0018847909959731624, + 0.0018860419950215146, + 0.0018348340090597048, + 0.0019059580081375316, + 0.0019241660047555342, + 0.0019076669996138662, + 0.0017971659981412813, + 0.0018808750028256327, + 0.001891583000542596, + 0.0018851670029107481, + 0.0017467919969931245, + 0.00192304200027138, + 0.0017984589940169826, + 0.0017947500018635765, + 0.001752791999024339, + 0.001832416994147934, + 0.0019162499957019463 + ] + }, + { + "forward": 0.0030641670018667355, + "prefill": 0.0030218750034691766, + "decode_steps": [ + 0.0019875839934684336, + 0.0017772499995771796, + 0.001806832995498553, + 0.0017959580000024289, + 0.0018429170013405383, + 0.0019172080064890906, + 0.001855249996879138, + 0.0019007920054718852, + 0.0017817500047385693, + 0.0018120419990736991, + 0.001818832999560982, + 0.0018910840008175, + 0.001905458004330285, + 0.0019034580036532134, + 0.0019126660045003518, + 0.0019196659995941445, + 0.0018634579901117831, + 0.0017903749976539984, + 0.0018814999930327758, + 0.0018005419988185167, + 0.001782124993042089, + 0.0018010000057984143, + 0.0017943749990081415, + 0.001793500006897375, + 0.0018203750078100711, + 0.0017831659934017807, + 0.0017754580039763823, + 0.0018008749902946874, + 0.0019046250090468675, + 0.001796540993382223, + 0.0018427499890094623, + 0.0017600830033188686 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 3.242667000449728, + "p95": 3.9547480941109816, + "min": 3.001374992891215, + "max": 4.319041996495798, + "iqr": 0.2132499976141844, + "mean": 3.3090875993366353, + "cv": 0.11654847534793421 + }, + "prefill_ms": { + "count": 10, + "median": 3.108146003796719, + "p95": 3.6156850452243807, + "min": 3.0218750034691766, + "max": 3.9265410014195368, + "iqr": 0.0219787543755956, + "mean": 3.1882835013675503, + "cv": 0.08318969150941015 + }, + "decode_ms_per_step": { + "count": 320, + "median": 1.8582710035843775, + "p95": 2.025581248017261, + "min": 1.685666007688269, + "max": 6.670583999948576, + "iqr": 0.105594503111206, + "mean": 1.9067506406827306, + "cv": 0.18972282454425737 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 2137.832516977803, + "p95": 2176.005587159643, + "min": 1764.1636291890895, + "max": 2176.1491775870372, + "iqr": 9.922964236973257, + "mean": 2105.1490714603656, + "cv": 0.05774708999038784 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 2031.464871725334, + "p95": 2068.3343295576938, + "min": 1691.6587663742025, + "max": 2069.8117760768096, + "iqr": 10.443497941404985, + "mean": 2000.0264331531484, + "cv": 0.055601049633415146 + }, + "memory": { + "peak_extra_bytes": 2950144, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "reference", + "batch": 4, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.007322999997995794, + "prefill": 0.004335791993071325, + "decode_steps": [ + 0.004676750002545305, + 0.004302500004996546, + 0.004188167004031129, + 0.00431204198685009, + 0.004275125000276603, + 0.004173124994849786, + 0.0041951249877456576, + 0.004085708002094179, + 0.004870707998634316, + 0.004070124996360391, + 0.004390792004414834, + 0.00456179199682083, + 0.004505165998125449, + 0.004670541995437816, + 0.0049199159984709695, + 0.004665000000386499, + 0.004157125003985129, + 0.004811708000488579, + 0.004473124994547106, + 0.004323582994402386, + 0.004840917012188584, + 0.004614708988810889, + 0.004482999996980652, + 0.004461040996829979, + 0.004393875002278946, + 0.004268749995389953, + 0.004716208000900224, + 0.004739209005492739, + 0.004422875004820526, + 0.004307291004806757, + 0.004189333994872868, + 0.004583875008393079 + ] + }, + { + "forward": 0.007060292002279311, + "prefill": 0.00428420800017193, + "decode_steps": [ + 0.004392332994029857, + 0.00415429200802464, + 0.004522041999734938, + 0.004383332998258993, + 0.00435579199984204, + 0.004278790991520509, + 0.00443187500059139, + 0.004355541997938417, + 0.004228457997669466, + 0.004205082994303666, + 0.0041244589956477284, + 0.004444459002115764, + 0.004328167007770389, + 0.0042640419997042045, + 0.00427616699016653, + 0.004291665987693705, + 0.004319375002523884, + 0.0041141669935313985, + 0.004331708012614399, + 0.00430233299266547, + 0.004376500000944361, + 0.004070833005243912, + 0.004815084001165815, + 0.004368082998553291, + 0.004711917004897259, + 0.0045593750110128894, + 0.004985917010344565, + 0.004354042001068592, + 0.004638083992176689, + 0.004488167003728449, + 0.004561708992696367, + 0.0044432499998947605 + ] + }, + { + "forward": 0.00683454200043343, + "prefill": 0.004345333989476785, + "decode_steps": [ + 0.004391082999063656, + 0.0043613749876385555, + 0.004468875005841255, + 0.0042211250110995024, + 0.004241832997649908, + 0.004419832999701612, + 0.00440366700058803, + 0.0042680000042309985, + 0.0043597909971140325, + 0.004334249999374151, + 0.0044179159885970876, + 0.004236084001604468, + 0.004328834009356797, + 0.004178749994025566, + 0.004345666995504871, + 0.004272292004316114, + 0.00438291700265836, + 0.00424812501296401, + 0.004260333997081034, + 0.004132582995225675, + 0.00389316699875053, + 0.004617625003447756, + 0.004213833002722822, + 0.004421999998157844, + 0.0043548749963520095, + 0.0043142920039827, + 0.004374625001219101, + 0.004482958000153303, + 0.004183124998235144, + 0.004353708005510271, + 0.004287207993911579, + 0.0044099170045228675 + ] + }, + { + "forward": 0.009651874992414378, + "prefill": 0.005295665992889553, + "decode_steps": [ + 0.005207458001677878, + 0.0051769169949693605, + 0.004146499995840713, + 0.004519124995567836, + 0.0046642090019304305, + 0.004456166992895305, + 0.004482624994125217, + 0.0042947910114889964, + 0.004683124992880039, + 0.003991042001871392, + 0.004458791998331435, + 0.004390292000607587, + 0.004426166997291148, + 0.0044280000001890585, + 0.004085499997017905, + 0.004446541002835147, + 0.004386250002426095, + 0.004238084002281539, + 0.004251374994055368, + 0.004349458002252504, + 0.004435499999090098, + 0.004270624995115213, + 0.004306582995923236, + 0.004276874999050051, + 0.004301417007809505, + 0.004298082989407703, + 0.004444457998033613, + 0.004285999995772727, + 0.004225666998536326, + 0.004242083989083767, + 0.004282750000129454, + 0.004206999990856275 + ] + }, + { + "forward": 0.006971374998101965, + "prefill": 0.004297999999835156, + "decode_steps": [ + 0.004134000002522953, + 0.004299166001146659, + 0.004383375009638257, + 0.004361250001238659, + 0.004384667001431808, + 0.0043045829952461645, + 0.004258208995452151, + 0.004264291012077592, + 0.004146417006268166, + 0.004220667004119605, + 0.004185457990388386, + 0.004217416993924417, + 0.004311166005209088, + 0.00427279100404121, + 0.004019291998702101, + 0.005154667000169866, + 0.004528541001491249, + 0.004999416996724904, + 0.004226792007102631, + 0.0045472919882740825, + 0.004813832987565547, + 0.004475166992051527, + 0.004356417004601099, + 0.004341166990343481, + 0.00455616699764505, + 0.004368834008346312, + 0.004775250010425225, + 0.0044249159982427955, + 0.004220749993692152, + 0.00405370800581295, + 0.004601250009727664, + 0.004423625010531396 + ] + }, + { + "forward": 0.007142415997805074, + "prefill": 0.004294208993087523, + "decode_steps": [ + 0.004616792008164339, + 0.004094749994692393, + 0.004308667004806921, + 0.0042508750048000365, + 0.004218875008518808, + 0.004273499987903051, + 0.004457250004634261, + 0.004379499994684011, + 0.004247792006935924, + 0.004184708013781346, + 0.004044999994221143, + 0.0047110839950619265, + 0.004393750001327135, + 0.004797834000783041, + 0.005122292001033202, + 0.0044800000032410026, + 0.004566040995996445, + 0.0045261249906616285, + 0.004403166996780783, + 0.004062166990479454, + 0.004496500012464821, + 0.0042493749933782965, + 0.004371249990072101, + 0.004218084010062739, + 0.004327292001107708, + 0.004301374996430241, + 0.00422400000388734, + 0.004232999999658205, + 0.004198416005237959, + 0.004765083009260707, + 0.004380958009278402, + 0.00484216699260287 + ] + }, + { + "forward": 0.008077250007772818, + "prefill": 0.004288541997084394, + "decode_steps": [ + 0.005137957996339537, + 0.0047490830038441345, + 0.004539874993497506, + 0.004311374999815598, + 0.00439429200196173, + 0.004762750002555549, + 0.004260375004378147, + 0.0045348330022534356, + 0.004298750005546026, + 0.004396749995066784, + 0.004360625011031516, + 0.0041348750091856346, + 0.004830083998967893, + 0.004393166993395425, + 0.004979834004188888, + 0.004657375000533648, + 0.005178083010832779, + 0.004491416999371722, + 0.0044925840047653764, + 0.004656999997678213, + 0.004687499997089617, + 0.004557041000225581, + 0.004407875007018447, + 0.004581084009259939, + 0.004568000003928319, + 0.004184124991297722, + 0.004467334001674317, + 0.004336874990258366, + 0.004473875000257976, + 0.004343167005572468, + 0.004152416993747465, + 0.004018332998384722 + ] + }, + { + "forward": 0.0074394999974174425, + "prefill": 0.004034291006973945, + "decode_steps": [ + 0.005061959003796801, + 0.00439520800136961, + 0.004726125000161119, + 0.00455504099954851, + 0.004863166992436163, + 0.00447820799308829, + 0.0043878340075025335, + 0.004303374997107312, + 0.0045591669913847, + 0.004510500002652407, + 0.00434700000914745, + 0.0043209160066908225, + 0.004243500006850809, + 0.004263166993041523, + 0.004422958008944988, + 0.004360332997748628, + 0.004678500001318753, + 0.004326916998252273, + 0.005227707995800301, + 0.004561959009151906, + 0.004727709005237557, + 0.004439292009919882, + 0.0047283339954447, + 0.004651082999771461, + 0.00470645799941849, + 0.004561624999041669, + 0.004333125005359761, + 0.004270834004273638, + 0.004294915997888893, + 0.004256542000803165, + 0.00441037499695085, + 0.004205207995255478 + ] + }, + { + "forward": 0.00683137500891462, + "prefill": 0.004077624995261431, + "decode_steps": [ + 0.004843208007514477, + 0.004277000000001863, + 0.004635290999431163, + 0.004687958004069515, + 0.004607625000062399, + 0.0043272500042803586, + 0.004407207990880124, + 0.004235125001287088, + 0.0043060830066679046, + 0.00433208400500007, + 0.004404499995871447, + 0.004291917008231394, + 0.004352832998847589, + 0.00443954199727159, + 0.0041634579974925146, + 0.004297042003599927, + 0.004310042000724934, + 0.00420345900056418, + 0.004773332999320701, + 0.004863624999416061, + 0.004836958003579639, + 0.004629791001207195, + 0.004387082997709513, + 0.004555125007755123, + 0.004460541997104883, + 0.0043577499891398475, + 0.015214833008940332, + 0.010093916993355379, + 0.006363292006426491, + 0.006603542002267204, + 0.005364249998820014, + 0.004814832995180041 + ] + }, + { + "forward": 0.007374415989033878, + "prefill": 0.0043312080088071525, + "decode_steps": [ + 0.004691542009823024, + 0.004427457999554463, + 0.004451125001651235, + 0.004361375002190471, + 0.004532667007879354, + 0.004436958013684489, + 0.004346208006609231, + 0.004346832996816374, + 0.004276833002222702, + 0.004256583008100279, + 0.004362832987681031, + 0.004298125000786968, + 0.0042872080084634945, + 0.004198333001113497, + 0.00423133400909137, + 0.004346540998085402, + 0.004315708996728063, + 0.004385583990369923, + 0.004225541008054279, + 0.004286625000531785, + 0.0042835830099647865, + 0.0042292910075047985, + 0.004296125000109896, + 0.004201917006867006, + 0.004323917004512623, + 0.004179915995337069, + 0.004306333008571528, + 0.004427542007761076, + 0.004128334010601975, + 0.004334083991125226, + 0.0043501669861143455, + 0.004350417002569884 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 7.232707997900434, + "p95": 8.943293749325674, + "min": 6.83137500891462, + "max": 9.651874992414378, + "iqr": 0.42962474617525004, + "mean": 7.470604099216871, + "cv": 0.1137263474258177 + }, + "prefill_ms": { + "count": 10, + "median": 4.296104496461339, + "p95": 4.868016591353807, + "min": 4.034291006973945, + "max": 5.295665992889553, + "iqr": 0.04935449760523625, + "mean": 4.358487497665919, + "cv": 0.07955017807965055 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.362103994935751, + "p95": 4.922911898756866, + "min": 3.89316699875053, + "max": 15.214833008940332, + "iqr": 0.2605422523629386, + "mean": 4.483687141009796, + "cv": 0.16439311896104497 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 909.7140719153507, + "p95": 925.4313570487994, + "min": 778.3970540180552, + "max": 926.3235065306168, + "iqr": 20.76695118919656, + "mean": 894.1751018828903, + "cv": 0.04785825716771133 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 879.7428081523419, + "p95": 897.2830059788131, + "min": 759.5622176928159, + "max": 898.0817472029514, + "iqr": 18.53505575742315, + "mean": 867.6516756838573, + "cv": 0.045976088614042565 + }, + "memory": { + "peak_extra_bytes": 4916224, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "auto", + "batch": 4, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.00575262498750817 + }, + { + "forward": 0.005792291995021515 + }, + { + "forward": 0.005753459001425654 + }, + { + "forward": 0.005620750001980923 + }, + { + "forward": 0.00615937499969732 + }, + { + "forward": 0.0057596670085331425 + }, + { + "forward": 0.00562795800215099 + }, + { + "forward": 0.005611209009657614 + }, + { + "forward": 0.005745125003159046 + }, + { + "forward": 0.005690124991815537 + } + ], + "forward_ms": { + "count": 10, + "median": 5.748874995333608, + "p95": 5.994187647593208, + "min": 5.611209009657614, + "max": 6.15937499969732, + "iqr": 0.11461525718914345, + "mean": 5.751258500094991, + "cv": 0.02742748973221074 + }, + "memory": { + "peak_extra_bytes": 5900288, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "sdpa", + "batch": 4, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.0049808750045485795, + "prefill": 0.005009540996979922, + "decode_steps": [ + 0.0020642919989768416, + 0.0019278330000815913, + 0.0020056250068591908, + 0.0019092919974355027, + 0.0020035420020576566, + 0.001998624997213483, + 0.002003708010306582, + 0.001970625002286397, + 0.0019485410011839122, + 0.0019507079996401444, + 0.001953541999682784, + 0.0019382089958526194, + 0.0018071669910568744, + 0.0019968330016126856, + 0.0020630409999284893, + 0.0019296249956823885, + 0.001872791995992884, + 0.002318375001777895, + 0.00195237499428913, + 0.0023794170119799674, + 0.0019302080036140978, + 0.0020012080058222637, + 0.0020559170079650357, + 0.0019237090018577874, + 0.002080749996821396, + 0.001945709009305574, + 0.002012125012697652, + 0.001994291989831254, + 0.002070625007036142, + 0.002062333995127119, + 0.0018944160110550001, + 0.0022162499953992665 + ] + }, + { + "forward": 0.005432791993371211, + "prefill": 0.005030250002164394, + "decode_steps": [ + 0.0023000410001259297, + 0.0021198329923208803, + 0.002094416006002575, + 0.0019298750121379271, + 0.0019905000081053004, + 0.0019801660091616213, + 0.0020156660029897466, + 0.0018654999876162037, + 0.00201237500004936, + 0.002021292006247677, + 0.0019638329977169633, + 0.0020329169929027557, + 0.0019227919983677566, + 0.0019676670053740963, + 0.001947000011568889, + 0.0021571670076809824, + 0.0020278750016586855, + 0.0020229159999871626, + 0.0019263330032117665, + 0.0020758750033564866, + 0.0019723339937627316, + 0.001985458002309315, + 0.001964582988875918, + 0.0019777909910771996, + 0.001897916998132132, + 0.0024772079923423007, + 0.002218708992586471, + 0.0020084999996470287, + 0.0020271669927751645, + 0.0021096249984111637, + 0.002034666991676204, + 0.002078791003441438 + ] + }, + { + "forward": 0.005027874998631887, + "prefill": 0.004987000007531606, + "decode_steps": [ + 0.0021386659936979413, + 0.00196229200810194, + 0.001943541006767191, + 0.0019610000017564744, + 0.002060124999843538, + 0.0020581249991664663, + 0.001949917001184076, + 0.0020069169986527413, + 0.0020151670032646507, + 0.0018921669980045408, + 0.002017167003941722, + 0.001952416991116479, + 0.0019737499969778582, + 0.0019486250093905255, + 0.002007249990128912, + 0.0019065829983446747, + 0.0018733339966274798, + 0.0020078329980606213, + 0.0019404160120757297, + 0.001998457999434322, + 0.001991750003071502, + 0.0019078749901382253, + 0.001987957992241718, + 0.0020056250068591908, + 0.0021350829920265824, + 0.002059791993815452, + 0.0019941670034313574, + 0.0019465410005068406, + 0.0019497500034049153, + 0.001939375011716038, + 0.00197441600903403, + 0.0020482079999055713 + ] + }, + { + "forward": 0.005053834000136703, + "prefill": 0.005017916992073879, + "decode_steps": [ + 0.0020076660002814606, + 0.0019889999966835603, + 0.00194283299788367, + 0.00203312499797903, + 0.0020172499935142696, + 0.001853875000961125, + 0.0020227500062901527, + 0.001973291000467725, + 0.001955250001628883, + 0.0019402500038268045, + 0.0019017080048797652, + 0.0020410420111147687, + 0.001968041993677616, + 0.0019170419982401654, + 0.0020364999945741147, + 0.001958958004252054, + 0.0019860419997712597, + 0.002009292002185248, + 0.0020354580046841875, + 0.0020284579950384796, + 0.0019397919968469068, + 0.0018970829987665638, + 0.0020187500049360096, + 0.00190491700777784, + 0.0019255840015830472, + 0.001975207997020334, + 0.0019861670007230714, + 0.00207704100466799, + 0.002027249996899627, + 0.002050291994237341, + 0.0020477079960983247, + 0.0020406659896252677 + ] + }, + { + "forward": 0.00644875000580214, + "prefill": 0.005900332995224744, + "decode_steps": [ + 0.0022191669995663688, + 0.002026499991188757, + 0.0019880409963661805, + 0.0024407919991062954, + 0.002036167003097944, + 0.0020377080072648823, + 0.0019334160024300218, + 0.0019504590018186718, + 0.002122999998391606, + 0.0020170839998172596, + 0.0019372499955352396, + 0.0020108750031795353, + 0.002017499995417893, + 0.001984125003218651, + 0.0019077920005656779, + 0.0018934999970952049, + 0.0019489580008666962, + 0.0018856669921660796, + 0.0019047920068260282, + 0.0018891660001827404, + 0.0019357090059202164, + 0.0024731669982429594, + 0.002051792005659081, + 0.00205275000189431, + 0.0021154580026632175, + 0.0020888330036541447, + 0.002039042010437697, + 0.0021160829928703606, + 0.002015624995692633, + 0.0019238329987274483, + 0.0019758749986067414, + 0.001927999997860752 + ] + }, + { + "forward": 0.004929707996780053, + "prefill": 0.005055417001130991, + "decode_steps": [ + 0.002035792000242509, + 0.0019148749997839332, + 0.0019861659966409206, + 0.0019164580007782206, + 0.002002958004595712, + 0.0020261250028852373, + 0.002016500002355315, + 0.0020506249857135117, + 0.0019212079932913184, + 0.0020000420045107603, + 0.001991791999898851, + 0.0019485829980112612, + 0.002030084011494182, + 0.0019033329881494865, + 0.0019043329957639799, + 0.0021802499977638945, + 0.002068540998152457, + 0.0020692499965662137, + 0.001955208004801534, + 0.0020734169956995174, + 0.0020691659883596003, + 0.0020616249967133626, + 0.0020720419997815043, + 0.0019391670066397637, + 0.0019840829918393865, + 0.0019446249934844673, + 0.002003165995120071, + 0.001953500002855435, + 0.0020772499992744997, + 0.002071165989036672, + 0.0020553749927785248, + 0.0019567910057958215 + ] + }, + { + "forward": 0.005027416002121754, + "prefill": 0.005049917002907023, + "decode_steps": [ + 0.001985666007385589, + 0.001887792008346878, + 0.0019291670032544062, + 0.001915041997563094, + 0.001967834003153257, + 0.001887624996015802, + 0.0020082499977434054, + 0.0019848749943776056, + 0.0019790829974226654, + 0.002013124991208315, + 0.0020055830100318417, + 0.0019474580039968714, + 0.001887207996333018, + 0.002344874999835156, + 0.0020824999955948442, + 0.002096541997161694, + 0.0019608329894253984, + 0.0019401669997023419, + 0.0019718749972525984, + 0.0019436669972492382, + 0.001906833000248298, + 0.002045041008386761, + 0.002031625001109205, + 0.0020086659933440387, + 0.001903249998576939, + 0.0019008340022992343, + 0.0020077910012332723, + 0.0019511669961502776, + 0.002138250012649223, + 0.0019773749954765663, + 0.001906708988826722, + 0.001984707996598445 + ] + }, + { + "forward": 0.005078458008938469, + "prefill": 0.0050877080066129565, + "decode_steps": [ + 0.0021404579892987385, + 0.0020322080090409145, + 0.0020440000080270693, + 0.0020423340029083192, + 0.002014499987126328, + 0.0019199590024072677, + 0.0019889579998562112, + 0.0019743750017369166, + 0.002009832998737693, + 0.0019027499947696924, + 0.0019369999936316162, + 0.0019143329991493374, + 0.002014457990298979, + 0.0020197080011712387, + 0.001873958986834623, + 0.002038499995251186, + 0.002007624992984347, + 0.0019318330014357343, + 0.0020040410017827526, + 0.002028540999162942, + 0.0019991250010207295, + 0.0020187500049360096, + 0.0020197090052533895, + 0.0019235409999964759, + 0.0019381669990252703, + 0.0019602079992182553, + 0.001963957998668775, + 0.002067832989268936, + 0.001982499990845099, + 0.002066667002509348, + 0.0020704170019598678, + 0.0019864169880747795 + ] + }, + { + "forward": 0.004992582995328121, + "prefill": 0.005059166011051275, + "decode_steps": [ + 0.001999041996896267, + 0.0021131250105099753, + 0.0020326669909991324, + 0.0020181250001769513, + 0.0019121249933959916, + 0.0020837919873883948, + 0.00198133300000336, + 0.0019355420081410557, + 0.0020279169984860346, + 0.002013707999140024, + 0.001899583003250882, + 0.0020734159916173667, + 0.0019496249879011884, + 0.0019216670043533668, + 0.0020381250069476664, + 0.001913542000693269, + 0.0019711670029209927, + 0.002011999997193925, + 0.00190245799603872, + 0.0021526249911403283, + 0.002072707997285761, + 0.0019592089956859127, + 0.0019806669879471883, + 0.001981625013286248, + 0.0019804579933406785, + 0.0019407500076340511, + 0.0020620830036932603, + 0.0020558749965857714, + 0.0020545000006677583, + 0.0020379590132506564, + 0.0020012080058222637, + 0.0020120830013183877 + ] + }, + { + "forward": 0.005060583003796637, + "prefill": 0.00500666699372232, + "decode_steps": [ + 0.0020718750020023435, + 0.0020090000034542754, + 0.0020284589991206303, + 0.002008167008170858, + 0.002022708998993039, + 0.0018965000053867698, + 0.0018749169976217672, + 0.00198491700575687, + 0.001966833006008528, + 0.0018955840059788898, + 0.001995250000618398, + 0.002046624998911284, + 0.002038499995251186, + 0.0019591669988585636, + 0.002021500011323951, + 0.0019351250084582716, + 0.0019878750026691705, + 0.001990791002754122, + 0.002020416999584995, + 0.0020797089964617044, + 0.0019252079946454614, + 0.001975416991626844, + 0.0020369999983813614, + 0.0019261669949628413, + 0.0019341670122230425, + 0.0019586660055210814, + 0.0019769169884966686, + 0.0020117499952903017, + 0.0019760830036830157, + 0.0020084579882677644, + 0.0019742089934879914, + 0.002023374996497296 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 5.040854499384295, + "p95": 5.991568900208221, + "min": 4.929707996780053, + "max": 6.44875000580214, + "iqr": 0.07269801062648185, + "mean": 5.203287400945555, + "cv": 0.08812999080509144 + }, + "prefill_ms": { + "count": 10, + "median": 5.040083502535708, + "p95": 5.534651750349439, + "min": 4.987000007531606, + "max": 5.900332995224744, + "iqr": 0.046593762817792594, + "mean": 5.120391600939911, + "cv": 0.05383804158660264 + }, + "decode_ms_per_step": { + "count": 320, + "median": 1.9942294966313057, + "p95": 2.1352413430577144, + "min": 1.8071669910568744, + "max": 2.4772079923423007, + "iqr": 0.09044774560607038, + "mean": 2.001290087173402, + "cv": 0.04396959847499672 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 2001.143318406023, + "p95": 2015.604904425948, + "min": 1965.3970659672837, + "max": 2015.703083604312, + "iqr": 19.67427589939166, + "mean": 1998.855454736642, + "cv": 0.008938899077799758 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1854.0798639867767, + "p95": 1867.7135406467376, + "min": 1808.7153769617278, + "max": 1867.8985110060298, + "iqr": 18.110590945675085, + "mean": 1850.9329517450667, + "cv": 0.010811133041828964 + }, + "memory": { + "peak_extra_bytes": 5900288, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "reference", + "batch": 4, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.012246709011378698, + "prefill": 0.008777625000220723, + "decode_steps": [ + 0.004962625011103228, + 0.004474582994589582, + 0.004850708995945752, + 0.0046899169974494725, + 0.004534166000667028, + 0.004509582999162376, + 0.0044531250023283064, + 0.004503417003434151, + 0.004331999996793456, + 0.004424166996614076, + 0.004601916007231921, + 0.004534041989245452, + 0.0045584579929709435, + 0.004444833000889048, + 0.004612666991306469, + 0.0053799169982085004, + 0.004786584002431482, + 0.004587916992022656, + 0.004664624997531064, + 0.004800166992936283, + 0.0048689159884816036, + 0.015363541999249719, + 0.008505290999892168, + 0.005464834001031704, + 0.005094457999803126, + 0.004778749993420206, + 0.004853666992858052, + 0.0050102499953936785, + 0.004740499993204139, + 0.004862875008257106, + 0.004578958003548905, + 0.004459375006263144 + ] + }, + { + "forward": 0.011858083002152853, + "prefill": 0.008271792001323774, + "decode_steps": [ + 0.00485429099353496, + 0.004448291991138831, + 0.004586499999277294, + 0.0047490410070167854, + 0.004775459005031735, + 0.004614999998011626, + 0.004499207992921583, + 0.004590458993334323, + 0.004715916991699487, + 0.004620292005711235, + 0.004569208002067171, + 0.004481875002966262, + 0.004883166999206878, + 0.004572000005282462, + 0.004568832999211736, + 0.004459249990759417, + 0.004387708991998807, + 0.0045226670044939965, + 0.00448233300994616, + 0.004464790996280499, + 0.00422158399305772, + 0.004378291007014923, + 0.004366500012110919, + 0.004459957999642938, + 0.004433791007613763, + 0.00463112500438001, + 0.0042941249994328246, + 0.004457417002413422, + 0.0044634590012719855, + 0.004607249997206964, + 0.004509959006099962, + 0.004445999991730787 + ] + }, + { + "forward": 0.011110624996945262, + "prefill": 0.008347750001121312, + "decode_steps": [ + 0.004357208003057167, + 0.0051759589987341315, + 0.004960499994922429, + 0.004632208001567051, + 0.004528958001174033, + 0.004549667006358504, + 0.004626499998266809, + 0.0044767080107703805, + 0.0044939170038560405, + 0.004426749990670942, + 0.004282624999177642, + 0.004421833000378683, + 0.004425959006766789, + 0.004522999995970167, + 0.0043660829978762195, + 0.004435374998138286, + 0.004310249991249293, + 0.004352874995674938, + 0.004313249999540858, + 0.0043223340035183355, + 0.004327917005866766, + 0.004360749997431412, + 0.004441500001121312, + 0.004501917006564327, + 0.004476624992094003, + 0.00442345799820032, + 0.004398167002364062, + 0.004468458006158471, + 0.0044399580074241385, + 0.004407999993418343, + 0.004436415998497978, + 0.004528833000222221 + ] + }, + { + "forward": 0.011773833000916056, + "prefill": 0.008484582998789847, + "decode_steps": [ + 0.005213874988839962, + 0.004619291998096742, + 0.004601667009410448, + 0.004758833994856104, + 0.004754791996674612, + 0.004331541989813559, + 0.004344499990111217, + 0.004693666996899992, + 0.004440208009327762, + 0.004549582998151891, + 0.004186374993878417, + 0.004633332995581441, + 0.004381334001664072, + 0.00449395900068339, + 0.004366459004813805, + 0.0045927080063847825, + 0.004539374989690259, + 0.004381458013085648, + 0.0044784169876948, + 0.004361499988590367, + 0.004437583003891632, + 0.004436917006387375, + 0.004219790993374772, + 0.01016654199338518, + 0.00565824999648612, + 0.004670124995755032, + 0.004679040997871198, + 0.005221624989644624, + 0.004800291993888095, + 0.004943417006870732, + 0.004725291000795551, + 0.004337999998824671 + ] + }, + { + "forward": 0.012326333002420142, + "prefill": 0.008788708000793122, + "decode_steps": [ + 0.005647166995913722, + 0.0046202090015867725, + 0.005033624998759478, + 0.004788584003108554, + 0.00455245899502188, + 0.004442375007783994, + 0.00516066700220108, + 0.004560083005344495, + 0.004435833005118184, + 0.0045238330058055, + 0.004639209000742994, + 0.004469167004572228, + 0.004307124996557832, + 0.004983000006177463, + 0.004519124995567836, + 0.004873332989518531, + 0.004682708997279406, + 0.004672709008445963, + 0.004608834002283402, + 0.004633749995264225, + 0.004551292004180141, + 0.004617375001544133, + 0.004293582998798229, + 0.00447633299336303, + 0.004395083000417799, + 0.004454916008398868, + 0.004202958996756934, + 0.004724084006738849, + 0.004497250003623776, + 0.004475584006286226, + 0.004792124993400648, + 0.004682874990976416 + ] + }, + { + "forward": 0.011345916005666368, + "prefill": 0.008350999996764585, + "decode_steps": [ + 0.0049652919988147914, + 0.004590040989569388, + 0.004543333008768968, + 0.0044906250113854185, + 0.004406749998452142, + 0.0045279580081114545, + 0.004281834000721574, + 0.004368375011836179, + 0.004404709005029872, + 0.004370916009065695, + 0.004164792000665329, + 0.004855500010307878, + 0.004798374997335486, + 0.004486666992306709, + 0.004622249995009042, + 0.004547667005681433, + 0.004561792011372745, + 0.004236416993080638, + 0.004514957996434532, + 0.004567041003610939, + 0.0043236250057816505, + 0.004406542007927783, + 0.004320459003793076, + 0.004380458005471155, + 0.004379707999760285, + 0.004535542000667192, + 0.004504208991420455, + 0.004390250003780238, + 0.004484290999243967, + 0.004353333002654836, + 0.004433624999364838, + 0.004489791986998171 + ] + }, + { + "forward": 0.01170550000097137, + "prefill": 0.008312916004797444, + "decode_steps": [ + 0.004771417006850243, + 0.004550541998469271, + 0.0042224999924656, + 0.004474415996810421, + 0.0044280419970164075, + 0.004327582995756529, + 0.0043362079886719584, + 0.004377624994958751, + 0.004373999996460043, + 0.004283916990971193, + 0.004434999995282851, + 0.004341416992247105, + 0.004195582994725555, + 0.004349875001935288, + 0.004364292006357573, + 0.00427745800698176, + 0.005110250000143424, + 0.004716625000583008, + 0.004380500002298504, + 0.004520332993706688, + 0.004482625008677132, + 0.004460042007849552, + 0.004434374990523793, + 0.004416958996444009, + 0.0042696660093497485, + 0.004264834002242424, + 0.004435458002262749, + 0.004421292003826238, + 0.0044213329965714365, + 0.004478584000025876, + 0.0044172909983899444, + 0.004397750002681278 + ] + }, + { + "forward": 0.011758041990105994, + "prefill": 0.008266499993624166, + "decode_steps": [ + 0.004857333013205789, + 0.004532042003120296, + 0.004754417008371092, + 0.004632958007277921, + 0.004562332993373275, + 0.00455858300847467, + 0.004513916006544605, + 0.004473665991099551, + 0.004378249999717809, + 0.004445500002475455, + 0.004447291008546017, + 0.004404667008202523, + 0.004412125010276213, + 0.004426500003319234, + 0.004451499989954755, + 0.004257249995134771, + 0.004884417008724995, + 0.004667665998567827, + 0.004421541991177946, + 0.004490624996833503, + 0.004482583011849783, + 0.004456374997971579, + 0.004099208003026433, + 0.0044354160054354, + 0.004390665999380872, + 0.004359917002147995, + 0.004441082986886613, + 0.004503374992054887, + 0.00443283400090877, + 0.004557249994832091, + 0.00455204100580886, + 0.004493041997193359 + ] + }, + { + "forward": 0.011863917010487057, + "prefill": 0.008382583007914945, + "decode_steps": [ + 0.00452420899819117, + 0.004877457991824485, + 0.0045700829941779375, + 0.004971834001480602, + 0.004840541005250998, + 0.00477524999587331, + 0.00445504100935068, + 0.004721125005744398, + 0.004910167001071386, + 0.004852874990319833, + 0.004713415997684933, + 0.004668458990636282, + 0.004917500002193265, + 0.004639749997295439, + 0.00727529200958088, + 0.011698625006829388, + 0.007565624997369014, + 0.005012833004002459, + 0.005322874989360571, + 0.005008624997572042, + 0.004995625000447035, + 0.0051195409905631095, + 0.004792207997525111, + 0.004846499999985099, + 0.004626124995411374, + 0.004608083007042296, + 0.004562250003800727, + 0.004462333003175445, + 0.004435291994013824, + 0.00431933299114462, + 0.004437124996911734, + 0.004431499997735955 + ] + }, + { + "forward": 0.011632459005340934, + "prefill": 0.008418166995397769, + "decode_steps": [ + 0.004789792001247406, + 0.004869291995419189, + 0.004566167001030408, + 0.004451625005458482, + 0.004477542010135949, + 0.0043994159932481125, + 0.004281708010239527, + 0.004592583994963206, + 0.004469000006793067, + 0.004357207988505252, + 0.004364541993709281, + 0.004306584000005387, + 0.0043609160056803375, + 0.0044363750057527795, + 0.004640125000150874, + 0.004573042009724304, + 0.004534959007287398, + 0.004316416991059668, + 0.004533792001893744, + 0.0044580829999176785, + 0.0042617080034688115, + 0.004438375006429851, + 0.004718124997452833, + 0.004512917003012262, + 0.004522041010204703, + 0.004382416998851113, + 0.004452916997252032, + 0.0044207500031916425, + 0.0045020830002613366, + 0.004532000006292947, + 0.004829707992030308, + 0.0045997920096851885 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 11.765937495511025, + "p95": 12.290502206451492, + "min": 11.110624996945262, + "max": 12.326333002420142, + "iqr": 0.21173925415496342, + "mean": 11.762141702638473, + "cv": 0.030922255803042357 + }, + "prefill_ms": { + "count": 10, + "median": 8.366791502339765, + "p95": 8.783720650535543, + "min": 8.266499993624166, + "max": 8.788708000793122, + "iqr": 0.1463544940634165, + "mean": 8.440162400074769, + "cv": 0.022767234373991044 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.502000003412832, + "p95": 5.110714549664408, + "min": 4.099208003026433, + "max": 15.363541999249719, + "iqr": 0.2440624994051177, + "mean": 4.661451059337196, + "cv": 0.1880164944652746 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 885.6198815519672, + "p95": 898.9434158457417, + "min": 774.4113487173574, + "max": 903.0760125408532, + "iqr": 49.33407920763818, + "mean": 860.6275956388101, + "cv": 0.05550001097480181 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 837.273515095949, + "p95": 849.2681072754499, + "min": 735.3597671585737, + "max": 853.0449587761248, + "iqr": 45.3741584684974, + "mean": 814.3477759787559, + "cv": 0.053314076867452764 + }, + "memory": { + "peak_extra_bytes": 17696768, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "auto", + "batch": 4, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.010640750013408251 + }, + { + "forward": 0.013468666991684586 + }, + { + "forward": 0.010673249998944812 + }, + { + "forward": 0.010612459009280428 + }, + { + "forward": 0.010672333010006696 + }, + { + "forward": 0.010655208010575734 + }, + { + "forward": 0.010566375000053085 + }, + { + "forward": 0.01064941700315103 + }, + { + "forward": 0.01061558400397189 + }, + { + "forward": 0.010547832993324846 + } + ], + "forward_ms": { + "count": 10, + "median": 10.64508350827964, + "p95": 12.210729344951684, + "min": 10.547832993324846, + "max": 13.468666991684586, + "iqr": 0.05481150219566189, + "mean": 10.910187603440136, + "cv": 0.0824865305153186 + }, + "memory": { + "peak_extra_bytes": 11800576, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "sdpa", + "batch": 4, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.009008084001834504, + "prefill": 0.008855375010170974, + "decode_steps": [ + 0.0024084589967969805, + 0.002476250010658987, + 0.0025018750020535663, + 0.002337666999665089, + 0.002179000002797693, + 0.00227537500904873, + 0.0023122919956222177, + 0.002225250005722046, + 0.002312083001015708, + 0.0023610000062035397, + 0.0021580830070888624, + 0.0022076250024838373, + 0.0022510830021928996, + 0.0022144999966258183, + 0.0022500419872812927, + 0.0022299999982351437, + 0.0022412079997593537, + 0.00218958398909308, + 0.0022297079995041713, + 0.0023011669982224703, + 0.002251375000923872, + 0.0023537079978268594, + 0.0023012079909676686, + 0.0021902909938944504, + 0.002202000003308058, + 0.0023411669972119853, + 0.002327416994376108, + 0.002197291993070394, + 0.002294040998094715, + 0.002342833991860971, + 0.0022225419961614534, + 0.002110624991473742 + ] + }, + { + "forward": 0.009024583996506408, + "prefill": 0.009095957997487858, + "decode_steps": [ + 0.0023834999883547425, + 0.0022982910013524815, + 0.0023261669994099066, + 0.002289625001139939, + 0.002187833990319632, + 0.002231749997008592, + 0.0023752499982947484, + 0.0022297079995041713, + 0.002112917005433701, + 0.002419750002445653, + 0.002292499993927777, + 0.002230208003311418, + 0.0025314999948022887, + 0.0024007909960346296, + 0.0022224580025067553, + 0.002196916990214959, + 0.002348541995161213, + 0.0022237090015551075, + 0.0023707499931333587, + 0.002290042000822723, + 0.0022389590012608096, + 0.0022882500052219257, + 0.0021765410056104884, + 0.002285124995978549, + 0.0022172079916344956, + 0.00228770800458733, + 0.002407791995210573, + 0.002208791993325576, + 0.0023082499974407256, + 0.0023084160056896508, + 0.0022517499892273918, + 0.0023782909993315116 + ] + }, + { + "forward": 0.009007291999296285, + "prefill": 0.008964167005615309, + "decode_steps": [ + 0.002625624998472631, + 0.0022840829915367067, + 0.0022262500133365393, + 0.002285499998833984, + 0.002208959005656652, + 0.0023444159887731075, + 0.002259792003314942, + 0.002162791002774611, + 0.0022137500054668635, + 0.0022133750026114285, + 0.0022414589911932126, + 0.002233791004982777, + 0.0024041670112637803, + 0.00226529200153891, + 0.002181791001930833, + 0.002381292011705227, + 0.002202625008067116, + 0.0022918749891687185, + 0.0023364579974440858, + 0.002187749996664934, + 0.0023609579948242754, + 0.0023080410028342158, + 0.002227167002274655, + 0.0022800000006100163, + 0.0022629579907516018, + 0.002277082996442914, + 0.0023424999963026494, + 0.0023384580126730725, + 0.0022849579981993884, + 0.0022312500077532604, + 0.0022975829924689606, + 0.002250250006909482 + ] + }, + { + "forward": 0.008710333990165964, + "prefill": 0.008942624990595505, + "decode_steps": [ + 0.0023715830029686913, + 0.0023325829970417544, + 0.002245625000796281, + 0.0022737079998478293, + 0.002324374989257194, + 0.002114415998221375, + 0.0022013330017216504, + 0.002260458000819199, + 0.0022570000000996515, + 0.0022242080012802035, + 0.0023611670039827004, + 0.002353041010792367, + 0.002381625003181398, + 0.0025860419991659, + 0.002329875002033077, + 0.0022457080049207434, + 0.002324125001905486, + 0.002176750000216998, + 0.0022339999995892867, + 0.002286250004544854, + 0.0022950000129640102, + 0.002246250005555339, + 0.0022374169930117205, + 0.002212708001025021, + 0.0024057500122580677, + 0.0022946660028537735, + 0.002189457998611033, + 0.0023306670045712963, + 0.002191167004639283, + 0.00224566699762363, + 0.002292124991072342, + 0.0023140830016927794 + ] + }, + { + "forward": 0.00903249999100808, + "prefill": 0.00905158300884068, + "decode_steps": [ + 0.0023937499936437234, + 0.0023664160107728094, + 0.002341208004509099, + 0.002276458006235771, + 0.002233916995464824, + 0.0022998749918770045, + 0.0023237089917529374, + 0.0022424999915529042, + 0.0022872080007800832, + 0.002338916005101055, + 0.0023373750009341165, + 0.0022821249876869842, + 0.002218291992903687, + 0.0024028750049183145, + 0.0021842919959453866, + 0.002160707997973077, + 0.0022122079972177744, + 0.00231687500490807, + 0.00218266699812375, + 0.002234999992651865, + 0.0022438749874709174, + 0.00221558399789501, + 0.002362166007515043, + 0.002231916005257517, + 0.002293875004397705, + 0.0022986250114627182, + 0.0023569580080220476, + 0.002285875001689419, + 0.0022872080007800832, + 0.002242417001980357, + 0.0022596669878112152, + 0.0022413330007111654 + ] + }, + { + "forward": 0.008953708005719818, + "prefill": 0.0091560830041999, + "decode_steps": [ + 0.002794332991470583, + 0.002281667009810917, + 0.002307332993950695, + 0.002318583006854169, + 0.0026145000010728836, + 0.0021998750016791746, + 0.0023594590020366013, + 0.0023028750001685694, + 0.0022104169911472127, + 0.002278667001519352, + 0.0023873339960118756, + 0.002252749996841885, + 0.0021976249990984797, + 0.0023695420095464215, + 0.0023283330083359033, + 0.0022390840022126213, + 0.0022975829924689606, + 0.002260209002997726, + 0.0022271669877227396, + 0.0022929999977350235, + 0.002275791994179599, + 0.0021953340037725866, + 0.002293875004397705, + 0.0022796659904997796, + 0.002253832994028926, + 0.0023480419913539663, + 0.0022366250050254166, + 0.0022385000047506765, + 0.002198792004492134, + 0.002326040994375944, + 0.002174666995415464, + 0.002204125004936941 + ] + }, + { + "forward": 0.008966957990196534, + "prefill": 0.00900787499267608, + "decode_steps": [ + 0.0023576249950565398, + 0.0022246250009629875, + 0.002273457997944206, + 0.0023007919953670353, + 0.002180916999350302, + 0.0022289999906206504, + 0.0022148749994812533, + 0.0022774170065531507, + 0.0023118749959394336, + 0.002236291009467095, + 0.0022634160122834146, + 0.002206165998359211, + 0.0022935419983696193, + 0.0022416249994421378, + 0.0022400419984478503, + 0.002311042000656016, + 0.0021971670066704974, + 0.0022392499959096313, + 0.002194667002186179, + 0.0023095420037861913, + 0.0023015409969957545, + 0.0022152919991640374, + 0.002173707995098084, + 0.002229082994745113, + 0.0023708330118097365, + 0.002230083991889842, + 0.0022560000070370734, + 0.002219333007815294, + 0.0022774169920012355, + 0.002246957999886945, + 0.002250082994578406, + 0.002241875001345761 + ] + }, + { + "forward": 0.008956541991210543, + "prefill": 0.00902712500828784, + "decode_steps": [ + 0.002599500003270805, + 0.002412290996289812, + 0.002378624994889833, + 0.002402999991318211, + 0.0022832089889561757, + 0.0022038750030333176, + 0.0022364169999491423, + 0.002163750003091991, + 0.002182790994993411, + 0.002716457995120436, + 0.0023522080009570345, + 0.002309625007910654, + 0.0022381250018952414, + 0.0022030420077499002, + 0.0023040420055622235, + 0.002137500006938353, + 0.002257125001051463, + 0.0022239169920794666, + 0.0022376249980879948, + 0.0021893750090384856, + 0.002282000001287088, + 0.002342166000744328, + 0.0023665000044275075, + 0.0022773329983465374, + 0.002292124991072342, + 0.0021891670039622113, + 0.0022384999901987612, + 0.0023112920025596395, + 0.002201875002356246, + 0.00272716699691955, + 0.002437832998111844, + 0.002314207988092676 + ] + }, + { + "forward": 0.008944791989051737, + "prefill": 0.009038917007273994, + "decode_steps": [ + 0.0024064580065896735, + 0.002470958002959378, + 0.002309625007910654, + 0.002286208007717505, + 0.0026822920044651255, + 0.0022083750081947073, + 0.002248374992632307, + 0.0023085830034688115, + 0.0021406669984571636, + 0.00281741599610541, + 0.002887041002395563, + 0.002254958002595231, + 0.002442541008349508, + 0.0026685829943744466, + 0.0021852089994354174, + 0.00230679199739825, + 0.002385166008025408, + 0.002370750007685274, + 0.0022596669878112152, + 0.0023274999985005707, + 0.0023141249985201284, + 0.0021342090039979666, + 0.0023707919899607077, + 0.0023415000032400712, + 0.0021358749945648015, + 0.0024519580038031563, + 0.0022976249892963096, + 0.0022997079940978438, + 0.002288125004270114, + 0.0022813749965280294, + 0.0023400000063702464, + 0.0021823330025654286 + ] + }, + { + "forward": 0.008970499999122694, + "prefill": 0.009020333003718406, + "decode_steps": [ + 0.003027124999789521, + 0.002614416996948421, + 0.0023202500015031546, + 0.0022702909918734804, + 0.0022663749987259507, + 0.002401334000751376, + 0.0023575829982291907, + 0.002192875006585382, + 0.002245166993816383, + 0.0023305000067921355, + 0.0022727909963577986, + 0.0023468750005122274, + 0.0022754999954486266, + 0.0022768749913666397, + 0.002218457986600697, + 0.002277708990732208, + 0.002291000011609867, + 0.002327750000404194, + 0.002224958996521309, + 0.0022624999983236194, + 0.00230695799109526, + 0.00220529199577868, + 0.0022179170046001673, + 0.0022506660025101155, + 0.0023606250033481047, + 0.0021693749877158552, + 0.002248916993266903, + 0.0022587919957004488, + 0.002333083000849001, + 0.002370624992181547, + 0.002234416999272071, + 0.0027668749971780926 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 8.968728994659614, + "p95": 9.028937793482328, + "min": 8.710333990165964, + "max": 9.03249999100808, + "iqr": 0.05346949910745025, + "mean": 8.957529395411257, + "cv": 0.010305249384606986 + }, + "prefill_ms": { + "count": 10, + "median": 9.023729006003123, + "p95": 9.129026751179481, + "min": 8.855375010170974, + "max": 9.1560830041999, + "iqr": 0.07332250606850721, + "mean": 9.016004102886654, + "cv": 0.009197411922977145 + }, + "decode_ms_per_step": { + "count": 320, + "median": 2.27818799612578, + "p95": 2.4775312602287163, + "min": 2.110624991473742, + "max": 3.027124999789521, + "iqr": 0.10208375897491351, + "mean": 2.2963296620673646, + "cv": 0.051269668957670024 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1747.9425991735789, + "p95": 1767.4553205778718, + "min": 1697.5049152669692, + "max": 1774.929484358763, + "iqr": 21.967211653147388, + "mean": 1742.1675815846709, + "cv": 0.012759363648557067 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1556.2685933428522, + "p95": 1573.2452787953869, + "min": 1515.8027284495572, + "max": 1577.8428266973574, + "iqr": 18.271949812915864, + "mean": 1551.735084020199, + "cv": 0.011706566919447575 + }, + "memory": { + "peak_extra_bytes": 12587008, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "reference", + "batch": 4, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.024881374993128702, + "prefill": 0.021096249998663552, + "decode_steps": [ + 0.005600249991402961, + 0.004826584001420997, + 0.005039457988459617, + 0.004785500001162291, + 0.004702541002188809, + 0.004934167009196244, + 0.004688875007559545, + 0.005181708009331487, + 0.005123999988427386, + 0.004860000000917353, + 0.004800792012247257, + 0.004380832993774675, + 0.005292500005452894, + 0.004429957989486866, + 0.004747209008201025, + 0.004804833006346598, + 0.005068790997029282, + 0.0051738329930230975, + 0.004988417000276968, + 0.004830791993299499, + 0.004550707992166281, + 0.0049658749921945855, + 0.004818249988602474, + 0.004777207999723032, + 0.005222834006417543, + 0.005317540999385528, + 0.004820500005735084, + 0.004875833998085, + 0.004971542002749629, + 0.004869708005571738, + 0.004720625001937151, + 0.004794750013388693 + ] + }, + { + "forward": 0.02403174999926705, + "prefill": 0.020572417008224875, + "decode_steps": [ + 0.005881916993530467, + 0.005229832997429185, + 0.004862417001277208, + 0.0048022499977378175, + 0.0048260829935316, + 0.004862417001277208, + 0.0046410830109380186, + 0.004631000003428198, + 0.004631874995538965, + 0.004541667003650218, + 0.004659666999941692, + 0.004637708989321254, + 0.004617333004716784, + 0.005179792002309114, + 0.0049822080036392435, + 0.005115042004035786, + 0.004893167002592236, + 0.004768041995703243, + 0.005209208000451326, + 0.004855624996707775, + 0.004956374992616475, + 0.004742874996736646, + 0.004713707996415906, + 0.0046826250036247075, + 0.004816708009457216, + 0.004823292008950375, + 0.004438999996636994, + 0.004679457997553982, + 0.004698125005234033, + 0.00511187499796506, + 0.004986582993296906, + 0.007314082991797477 + ] + }, + { + "forward": 0.026930000007268973, + "prefill": 0.021385583007941023, + "decode_steps": [ + 0.005803417006973177, + 0.005349250001017936, + 0.005368250000174157, + 0.0050431670097168535, + 0.00472329200420063, + 0.004652582996641286, + 0.0047818750026635826, + 0.00470587499148678, + 0.00474479200784117, + 0.0046822500007692724, + 0.004791082988958806, + 0.0048122910084202886, + 0.004658292004023679, + 0.004626041991286911, + 0.004735417009214871, + 0.004803625008207746, + 0.004973040995537303, + 0.005104666997794993, + 0.005526790992007591, + 0.005059792005340569, + 0.005019249991164543, + 0.004901583000901155, + 0.004662916995584965, + 0.005311667002388276, + 0.004921833009575494, + 0.0047582919942215085, + 0.004716291994554922, + 0.004838916996959597, + 0.005089208993013017, + 0.004807999997865409, + 0.004659625003114343, + 0.00465616698784288 + ] + }, + { + "forward": 0.024543084000470117, + "prefill": 0.021578750005573966, + "decode_steps": [ + 0.005884750004042871, + 0.005062541997176595, + 0.004626833004294895, + 0.004721584002254531, + 0.004735042006359436, + 0.004636957994080149, + 0.004700250006862916, + 0.004535584012046456, + 0.004944749991409481, + 0.005117249995237216, + 0.004804665994015522, + 0.004396584001369774, + 0.0046249590086517856, + 0.00459112499083858, + 0.004488792008487508, + 0.004557000007480383, + 0.004697791999205947, + 0.004556833999231458, + 0.004596500002662651, + 0.004690957997809164, + 0.005199292005272582, + 0.004976125012035482, + 0.004784583012224175, + 0.004797625006176531, + 0.004592541998135857, + 0.004628583003068343, + 0.00476495799375698, + 0.004711082990979776, + 0.004824250005185604, + 0.004740541990031488, + 0.0046457079879473895, + 0.004620624997187406 + ] + }, + { + "forward": 0.02343670799746178, + "prefill": 0.020581000004312955, + "decode_steps": [ + 0.005053542001405731, + 0.004763416000059806, + 0.004527583994786255, + 0.004727417006506585, + 0.004525541007751599, + 0.004571792000206187, + 0.004742582998005673, + 0.004558458997053094, + 0.004631542004062794, + 0.004595666003297083, + 0.0045637909934157506, + 0.004586374998325482, + 0.004444874997716397, + 0.004664124993723817, + 0.004525042008026503, + 0.004692750007961877, + 0.004702874997747131, + 0.004673333998653106, + 0.004631666000932455, + 0.004615750003722496, + 0.004587707997416146, + 0.004712541995104402, + 0.004908499991870485, + 0.004794541993760504, + 0.0046500409953296185, + 0.0047249589988496155, + 0.00473579099343624, + 0.004742000004625879, + 0.004723584002931602, + 0.004756207999889739, + 0.004703917002188973, + 0.004649750000680797 + ] + }, + { + "forward": 0.022948750003706664, + "prefill": 0.021072916002594866, + "decode_steps": [ + 0.005543999999645166, + 0.004768082988448441, + 0.004717874995549209, + 0.004476874993997626, + 0.004700709003373049, + 0.004632833006326109, + 0.004655417011235841, + 0.004662333012674935, + 0.004572499994537793, + 0.0046915410057408735, + 0.004528290999587625, + 0.004584375012200326, + 0.004613750003045425, + 0.00472904100024607, + 0.0048205420025624335, + 0.0046051669924054295, + 0.004688208005973138, + 0.004807999997865409, + 0.005547249995288439, + 0.00539750000461936, + 0.005067792008048855, + 0.004707790998509154, + 0.004708249995019287, + 0.00482887499674689, + 0.005231999995885417, + 0.005194083001697436, + 0.004917333004414104, + 0.004814667001483031, + 0.00477412500185892, + 0.004626916008419357, + 0.004394791991217062, + 0.004703166996478103 + ] + }, + { + "forward": 0.023743500001728535, + "prefill": 0.02052187499066349, + "decode_steps": [ + 0.00490029199863784, + 0.004698332995758392, + 0.00458945898571983, + 0.004607458002283238, + 0.0046558339963667095, + 0.004648999994969927, + 0.004619041996193118, + 0.004539916000794619, + 0.004552667000098154, + 0.0046535419969586655, + 0.004632958996808156, + 0.004569957993226126, + 0.00455412500014063, + 0.004585082991980016, + 0.004524250005488284, + 0.004596667000441812, + 0.004537333006737754, + 0.004602999993949197, + 0.004571416997350752, + 0.004593083998770453, + 0.004588166993926279, + 0.004631749994587153, + 0.005173500001546927, + 0.0049270000017713755, + 0.0050728750065900385, + 0.004451750006410293, + 0.004675667005358264, + 0.004824666990316473, + 0.004765417004819028, + 0.00471583400212694, + 0.0048178750002989545, + 0.0046862500021234155 + ] + }, + { + "forward": 0.024555416996008717, + "prefill": 0.020965625008102506, + "decode_steps": [ + 0.005489332994329743, + 0.004945165987010114, + 0.004782957999850623, + 0.004793415995663963, + 0.004836832988075912, + 0.004598540996084921, + 0.005200540996156633, + 0.004793375002918765, + 0.004699083001469262, + 0.004956167002092116, + 0.004653666997910477, + 0.004585834001773037, + 0.0048502499994356185, + 0.00470366700028535, + 0.0045439999958034605, + 0.004692916991189122, + 0.004616083999280818, + 0.0044475830072769895, + 0.0052324999996926636, + 0.0050158339872723445, + 0.004827124997973442, + 0.004823084003874101, + 0.00453245900280308, + 0.005416582993348129, + 0.0052109579992247745, + 0.0050277499976800755, + 0.004919665996567346, + 0.004522666000411846, + 0.004619625004124828, + 0.00475412500964012, + 0.0047179169923765585, + 0.004904208995867521 + ] + }, + { + "forward": 0.022913041000720114, + "prefill": 0.02067574999819044, + "decode_steps": [ + 0.005824709005537443, + 0.004973416987922974, + 0.004652750008972362, + 0.004683666993514635, + 0.004689458000939339, + 0.004889750009169802, + 0.004650207993108779, + 0.004663499988964759, + 0.0046714170021004975, + 0.004752583001391031, + 0.0044217079994268715, + 0.004666291992180049, + 0.004442542005563155, + 0.005395874992245808, + 0.005325250007444993, + 0.0047999579983297735, + 0.0052755000069737434, + 0.0044639999978244305, + 0.0048146250046556816, + 0.004912792006507516, + 0.004742457997053862, + 0.004758083989145234, + 0.004756082998937927, + 0.004746332997456193, + 0.004591833989252336, + 0.004429708991665393, + 0.013117666996549815, + 0.010475583010702394, + 0.00587004199041985, + 0.005377541994675994, + 0.00537879100011196, + 0.005059375005657785 + ] + }, + { + "forward": 0.02433662499242928, + "prefill": 0.020767167006852105, + "decode_steps": [ + 0.005732040997827426, + 0.005010832988773473, + 0.004834500010474585, + 0.004762541997479275, + 0.0046963749919086695, + 0.004668458001106046, + 0.004539833011222072, + 0.005728499992983416, + 0.004886041991994716, + 0.004867083000135608, + 0.004784000004292466, + 0.0048200830060523, + 0.004716415991424583, + 0.004390999994939193, + 0.004892750002909452, + 0.004851709003560245, + 0.004587124989484437, + 0.004793958010850474, + 0.004926667010295205, + 0.005029374995501712, + 0.004928333990392275, + 0.004800624999916181, + 0.0047812920092837885, + 0.004946875007590279, + 0.005160125001566485, + 0.005159791995538399, + 0.004889374991762452, + 0.004829708006582223, + 0.004345207998994738, + 0.0048216659924946725, + 0.004641291001462378, + 0.0047922910016495734 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 24.184187495848164, + "p95": 26.008118750905847, + "min": 22.913041000720114, + "max": 26.930000007268973, + "iqr": 1.0389277485955972, + "mean": 24.232024999218993, + "cv": 0.04806858810546153 + }, + "prefill_ms": { + "count": 10, + "median": 20.866396007477306, + "p95": 21.49182485663914, + "min": 20.52187499066349, + "max": 21.578750005573966, + "iqr": 0.48572899686405435, + "mean": 20.921733303111978, + "cv": 0.017339779771660763 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.753354005515575, + "p95": 5.398454154055798, + "min": 4.345207998994738, + "max": 13.117666996549815, + "iqr": 0.27996824428555556, + "mean": 4.867733999617485, + "cv": 0.1314429846671552 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 824.9358344418786, + "p95": 856.0608339512218, + "min": 751.7317640343115, + "max": 856.2579280138127, + "iqr": 24.488341813564602, + "mean": 822.7517840769955, + "cv": 0.036167332775440805 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 727.1463562598651, + "p95": 752.6025624887164, + "min": 670.3351737930628, + "max": 752.6371686367065, + "iqr": 17.393658487348375, + "mean": 725.1333665787096, + "cv": 0.03208998622220909 + }, + "memory": { + "peak_extra_bytes": 66850816, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "auto", + "batch": 4, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.024328790997969918 + }, + { + "forward": 0.023795083005097695 + }, + { + "forward": 0.023743707992252894 + }, + { + "forward": 0.023959999991348013 + }, + { + "forward": 0.023903374996734783 + }, + { + "forward": 0.023783917000400834 + }, + { + "forward": 0.023816667002392933 + }, + { + "forward": 0.02387529201223515 + }, + { + "forward": 0.023774792003678158 + }, + { + "forward": 0.024787041998934 + } + ], + "forward_ms": { + "count": 10, + "median": 23.84597950731404, + "p95": 24.580829048500163, + "min": 23.743707992252894, + "max": 24.787041998934, + "iqr": 0.1591352411196567, + "mean": 23.976866700104438, + "cv": 0.013814242859502383 + }, + "memory": { + "peak_extra_bytes": 24125440, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "sdpa", + "batch": 4, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.01946541700453963, + "prefill": 0.020018292008899152, + "decode_steps": [ + 0.0032003329979488626, + 0.0029240000003483146, + 0.0029228749917820096, + 0.002886540998588316, + 0.0028449579986045137, + 0.002921791994594969, + 0.0026653329987311736, + 0.0027163749909959733, + 0.0026924170088022947, + 0.00274679200083483, + 0.0026970830076606944, + 0.0028554579912452027, + 0.0028572089941008016, + 0.0029199999989941716, + 0.002798750007059425, + 0.0028342919977148995, + 0.0029226250044303015, + 0.00325949999387376, + 0.0032349579996662214, + 0.0032861660001799464, + 0.003262166996137239, + 0.004255458989064209, + 0.00395804199797567, + 0.0065139169892063364, + 0.005673457999364473, + 0.005516749995877035, + 0.00438249998842366, + 0.0053050420101499185, + 0.0035319169983267784, + 0.003991999998106621, + 0.003927290992578492, + 0.003280792006989941 + ] + }, + { + "forward": 0.018660292000276968, + "prefill": 0.018619915994349867, + "decode_steps": [ + 0.0025514160079183057, + 0.0024072500091278926, + 0.002457208000123501, + 0.0024554159899707884, + 0.002507500001229346, + 0.0023966669978108257, + 0.002430832988466136, + 0.0025001659960253164, + 0.0023876250052126125, + 0.002470624996931292, + 0.002478042006259784, + 0.00242883300234098, + 0.0024339579977095127, + 0.002341708997846581, + 0.0025432080001337454, + 0.0025181249948218465, + 0.0023411250003846362, + 0.002644499996677041, + 0.0025014999991981313, + 0.0025379169965162873, + 0.002453833003528416, + 0.0025804169999901205, + 0.002564833004726097, + 0.0026986669981852174, + 0.0024897500115912408, + 0.0026727919903350994, + 0.0026432500017108396, + 0.0026436250045662746, + 0.0028270420007174835, + 0.002803125011269003, + 0.0028306659951340407, + 0.0027793750050477684 + ] + }, + { + "forward": 0.018649834004463628, + "prefill": 0.01868191700486932, + "decode_steps": [ + 0.00259816600009799, + 0.0024815409997245297, + 0.0024549169902456924, + 0.002507875004084781, + 0.0024422919959761202, + 0.002440166994347237, + 0.002409833003184758, + 0.002375291005591862, + 0.002478833994246088, + 0.0025318340049125254, + 0.002425458005745895, + 0.002439332994981669, + 0.002483875010511838, + 0.002490792001481168, + 0.0024897499970393255, + 0.0024082080053631216, + 0.0024327089922735468, + 0.0025574169994797558, + 0.0025110829883487895, + 0.002500124988728203, + 0.00242866700864397, + 0.0025096659956034273, + 0.0026163330039707944, + 0.002500083006452769, + 0.0024731250014156103, + 0.0025624999980209395, + 0.002602417007437907, + 0.0026682090101530775, + 0.002653999996255152, + 0.0026283330080332235, + 0.0025546670076437294, + 0.002741291988058947 + ] + }, + { + "forward": 0.019019165993086062, + "prefill": 0.018780749989673495, + "decode_steps": [ + 0.0028757919935742393, + 0.0029562090057879686, + 0.002715665992582217, + 0.0027256660105194896, + 0.002697082993108779, + 0.0029393750010058284, + 0.0027390420000301674, + 0.0028486660012276843, + 0.0028632090106839314, + 0.003094542000326328, + 0.003204459004336968, + 0.003203708998626098, + 0.0031625830015400425, + 0.0032150420011021197, + 0.0031753750081406906, + 0.00413075000687968, + 0.0040388750057900324, + 0.004174208006588742, + 0.004354042001068592, + 0.004624250010238029, + 0.004492624997510575, + 0.005289916996844113, + 0.005969541001832113, + 0.005467250011861324, + 0.005417874999693595, + 0.005425499999546446, + 0.005257707991404459, + 0.00514083300367929, + 0.004835040992475115, + 0.004928833004669286, + 0.0046218750067055225, + 0.004461749995243736 + ] + }, + { + "forward": 0.018678415988688357, + "prefill": 0.018570874992292374, + "decode_steps": [ + 0.002842166999471374, + 0.0026064999983645976, + 0.002598249993752688, + 0.002817000000504777, + 0.0024330419983016327, + 0.0024754579935688525, + 0.0023939999955473468, + 0.0025419169978704304, + 0.002463333003106527, + 0.0024363749980693683, + 0.0024691669968888164, + 0.0024263749946840107, + 0.00238733300648164, + 0.0024486670008627698, + 0.00246641700505279, + 0.0024733329919399694, + 0.002474374996381812, + 0.0025070830015465617, + 0.002485958000761457, + 0.002417083000182174, + 0.0025446669897064567, + 0.0026348329993197694, + 0.0026424590032547712, + 0.002608166993013583, + 0.0025782909942790866, + 0.002580125001259148, + 0.0027756250055972487, + 0.0028178330103401095, + 0.0028275829972699285, + 0.0027194170106668025, + 0.0028729579935315996, + 0.0028768750053131953 + ] + }, + { + "forward": 0.018672207996132784, + "prefill": 0.018628125006216578, + "decode_steps": [ + 0.0025952500000130385, + 0.002477333997376263, + 0.0024712919985176995, + 0.0024056660040514544, + 0.0025280409900005907, + 0.002550374993006699, + 0.0024287499982165173, + 0.0024055419926298782, + 0.0024761670065345243, + 0.0024613749992568046, + 0.002493833002517931, + 0.0024050420033745468, + 0.0024118750006891787, + 0.0024493749951943755, + 0.0025167919957311824, + 0.0024550829984946176, + 0.00242341699777171, + 0.002450375002808869, + 0.002482791998772882, + 0.0025475410075159743, + 0.002486625002347864, + 0.002505666998331435, + 0.0025225419958587736, + 0.0025170000008074567, + 0.002525624993722886, + 0.002555500002927147, + 0.0024931660009315237, + 0.0026510000025155023, + 0.0027211669948883355, + 0.002664457992068492, + 0.002518416993552819, + 0.00261274998774752 + ] + }, + { + "forward": 0.019685124992975034, + "prefill": 0.019743500000913627, + "decode_steps": [ + 0.003407625001273118, + 0.0032277919963235036, + 0.003161458997055888, + 0.0032563749991822988, + 0.0031655000057071447, + 0.004157916991971433, + 0.0041775409918045625, + 0.004176916001597419, + 0.005061584000941366, + 0.004951291994075291, + 0.004532834005658515, + 0.0044281659938860685, + 0.005060166004113853, + 0.004915999990771525, + 0.004999500000849366, + 0.005102709008497186, + 0.0041855409945128486, + 0.004692666989285499, + 0.0042872500052908435, + 0.0038867080002091825, + 0.0035582080017775297, + 0.0034454999986337498, + 0.003690625002491288, + 0.0033496249961899593, + 0.003539999990607612, + 0.0035462499945424497, + 0.003430708995438181, + 0.0034980000054929405, + 0.003488957998342812, + 0.0034949999972013757, + 0.0032845419918885455, + 0.0032487080025020987 + ] + }, + { + "forward": 0.018686625000555068, + "prefill": 0.018630749997100793, + "decode_steps": [ + 0.0025243749987566844, + 0.0025680000107968226, + 0.002538207991165109, + 0.0024611249973531812, + 0.002438791998429224, + 0.0025301249988842756, + 0.002437457995256409, + 0.0024937499983934686, + 0.0023518329981015995, + 0.0024295000039273873, + 0.002781957999104634, + 0.0024699159985175356, + 0.002510750011424534, + 0.002579708001576364, + 0.0024266670079668984, + 0.00258100000792183, + 0.002544625007431023, + 0.002613417003885843, + 0.0025361660082126036, + 0.002443749996018596, + 0.0025057500024558976, + 0.0025882080080918968, + 0.002516916996682994, + 0.002783167001325637, + 0.0025939170009223744, + 0.002661040998646058, + 0.002551333003793843, + 0.002928415997303091, + 0.0028177499916637316, + 0.0028274159994907677, + 0.0027392920019337907, + 0.0028734999941661954 + ] + }, + { + "forward": 0.018567874998552725, + "prefill": 0.018841208002413623, + "decode_steps": [ + 0.0027315830084262416, + 0.0024055840040091425, + 0.00253274998976849, + 0.0025349170027766377, + 0.002428666004561819, + 0.002435332993627526, + 0.002464666002197191, + 0.0023606670001754537, + 0.00245633399754297, + 0.0025227089936379343, + 0.0024363749980693683, + 0.002424874997814186, + 0.002490624989150092, + 0.0024730830045882612, + 0.0024908750056056306, + 0.0024410419864580035, + 0.0024366669968003407, + 0.0028294579969951883, + 0.002590749994851649, + 0.002597042010165751, + 0.002567290997831151, + 0.0026390419952804223, + 0.00254904200846795, + 0.0027611249970505014, + 0.0027356249920558184, + 0.0027653750003082678, + 0.0027689999988069758, + 0.00264791700465139, + 0.002975458002765663, + 0.003135833001579158, + 0.0033900419948622584, + 0.0030104170000413433 + ] + }, + { + "forward": 0.01988562500628177, + "prefill": 0.02042749999964144, + "decode_steps": [ + 0.003335415996843949, + 0.003240999998524785, + 0.0031652920006308705, + 0.0031955419981386513, + 0.003361333001521416, + 0.0028855420096078888, + 0.0028605829866137356, + 0.0028890410030726343, + 0.0028811249940190464, + 0.0029544999997597188, + 0.003198332997271791, + 0.0031722909916425124, + 0.0031225830025505275, + 0.0032149580074474216, + 0.0031707919988548383, + 0.003371625003637746, + 0.00337333299103193, + 0.003457665996393189, + 0.00340662500821054, + 0.004564457995002158, + 0.004672291994211264, + 0.004863750000367872, + 0.006038708001142368, + 0.005965999996988103, + 0.005271041998639703, + 0.004662291001295671, + 0.004889791001915, + 0.0046279579983092844, + 0.0047090830048546195, + 0.004656709003029391, + 0.00420437499997206, + 0.004155332993832417 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 18.682520494621713, + "p95": 19.79540000029374, + "min": 18.567874998552725, + "max": 19.88562500628177, + "iqr": 0.6905832524353173, + "mean": 18.997058298555203, + "cv": 0.026052023119933005 + }, + "prefill_ms": { + "count": 10, + "median": 18.731333497271407, + "p95": 20.24335640380741, + "min": 18.570874992292374, + "max": 20.42749999964144, + "iqr": 0.8891457473509945, + "mean": 19.094283299637027, + "cv": 0.0362685692163609 + }, + "decode_ms_per_step": { + "count": 320, + "median": 2.6705005002440885, + "p95": 5.060236903955229, + "min": 2.3411250003846362, + "max": 6.513916989206336, + "iqr": 0.750333747419063, + "mean": 3.062610649567432, + "cv": 0.2814636389115183 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1536.001226058986, + "p95": 1594.1265395240239, + "min": 1007.4988528875219, + "max": 1595.8142998114656, + "iqr": 504.37666727926944, + "mean": 1358.3933912651391, + "cv": 0.19772496056686129 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1254.0565246501842, + "p95": 1293.6211957739686, + "min": 875.7815592290308, + "max": 1295.0490009518085, + "iqr": 367.12796451281906, + "mean": 1125.419993539195, + "cv": 0.1718388168652095 + }, + "memory": { + "peak_extra_bytes": 23601152, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "reference", + "batch": 4, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.06750158399518114, + "prefill": 0.0636743750073947, + "decode_steps": [ + 0.008724541999981739, + 0.006820750000770204, + 0.006419208002625965, + 0.005694166000466794, + 0.0056347499921685085, + 0.005658042005961761, + 0.005211832991335541, + 0.005125791998580098, + 0.0051764999952865764, + 0.005246458007604815, + 0.005111791004310362, + 0.005150875003891997, + 0.0051486670126905665, + 0.0052044580079382285, + 0.005156375002115965, + 0.005209000009926967, + 0.005089582991786301, + 0.004965415995684452, + 0.0051729169936152175, + 0.005113833001814783, + 0.005052416992839426, + 0.005120166009874083, + 0.005144082999322563, + 0.005179666986805387, + 0.005125333002069965, + 0.005155957987881266, + 0.005117042004712857, + 0.005223375002969988, + 0.0049993330030702055, + 0.005188958995859139, + 0.00538254200364463, + 0.005113999999593943 + ] + }, + { + "forward": 0.06643437501043081, + "prefill": 0.06346329201187473, + "decode_steps": [ + 0.0061828330071875826, + 0.005419708002591506, + 0.005183625005884096, + 0.005112957995152101, + 0.005444624999654479, + 0.005077207999420352, + 0.005067916994448751, + 0.005195249992539175, + 0.005368124999222346, + 0.005145582996192388, + 0.0051709580002352595, + 0.0050785829953383654, + 0.005093542000395246, + 0.0050009589904220775, + 0.005176999999093823, + 0.005020374999730848, + 0.005255707990727387, + 0.005152833997271955, + 0.005024165991926566, + 0.005160125001566485, + 0.005025708000175655, + 0.005178042003535666, + 0.0052064170013181865, + 0.005194750003283843, + 0.005171083001187071, + 0.005358124995836988, + 0.005422209011157975, + 0.005407040996942669, + 0.005248250003205612, + 0.005161624998436309, + 0.004793542000697926, + 0.005042333999881521 + ] + }, + { + "forward": 0.06400912499520928, + "prefill": 0.061669292001170106, + "decode_steps": [ + 0.006311250006547198, + 0.005273041999316774, + 0.005474042001878843, + 0.006366208996041678, + 0.005754333004006185, + 0.005314916998031549, + 0.005040167001425289, + 0.005354624998290092, + 0.005229583999607712, + 0.005192917000385933, + 0.00525733400718309, + 0.005171791999600828, + 0.0051220839959569275, + 0.005158250001841225, + 0.004975542004103772, + 0.0051622499886434525, + 0.005250709000392817, + 0.0051414579938864335, + 0.005158332991413772, + 0.005225916000199504, + 0.005225708999205381, + 0.0051305829983903095, + 0.005096833003335632, + 0.005034208006691188, + 0.005172916004084982, + 0.0055242500093299896, + 0.005586583007243462, + 0.005300707998685539, + 0.0053558750078082085, + 0.005271125002764165, + 0.00516816700110212, + 0.005075458000646904 + ] + }, + { + "forward": 0.07057962499675341, + "prefill": 0.06334258300194051, + "decode_steps": [ + 0.010367208000388928, + 0.008455582996248268, + 0.006999040997470729, + 0.006777583999792114, + 0.00623191699560266, + 0.0061040419968776405, + 0.005622750002657995, + 0.005629124992992729, + 0.005484833993250504, + 0.005301833996782079, + 0.005722875008359551, + 0.005629249993944541, + 0.005107457996928133, + 0.0054248329979600385, + 0.0053431669948622584, + 0.005175541999051347, + 0.0052688330033561215, + 0.005179833009606227, + 0.005180791995371692, + 0.005160291999345645, + 0.005205458001000807, + 0.005196040990995243, + 0.005205499997828156, + 0.00515224999981001, + 0.005363249991205521, + 0.005155083010322414, + 0.005220166000071913, + 0.005157749998033978, + 0.00520233299175743, + 0.005661167000653222, + 0.005448083000374027, + 0.005248250003205612 + ] + }, + { + "forward": 0.06777891698584426, + "prefill": 0.06507245899410918, + "decode_steps": [ + 0.015507792006246746, + 0.011065458005759865, + 0.009606583000277169, + 0.008420208003371954, + 0.007515541001339443, + 0.006993917006184347, + 0.006371124996803701, + 0.006420791003620252, + 0.006290334000368603, + 0.00612983301107306, + 0.0059122499951627105, + 0.006004124996252358, + 0.006138457989436574, + 0.005942208008491434, + 0.006576416999450885, + 0.006211165993590839, + 0.006080667008063756, + 0.005416583007900044, + 0.005263333994662389, + 0.005211875002714805, + 0.005172292003408074, + 0.0054802500089863315, + 0.005311083004926331, + 0.005116957996506244, + 0.0051380420045461506, + 0.0054250000102911144, + 0.005118500004755333, + 0.005081291004898958, + 0.005231832998106256, + 0.00518591700529214, + 0.005114875006256625, + 0.005181708009331487 + ] + }, + { + "forward": 0.07299645899911411, + "prefill": 0.062436459003947675, + "decode_steps": [ + 0.008250375001807697, + 0.00725362500816118, + 0.006054249999579042, + 0.0062468749965773895, + 0.005476042002555914, + 0.005424083006801084, + 0.0053734590037493035, + 0.005932665997534059, + 0.00538066599983722, + 0.005489208007929847, + 0.005257791999611072, + 0.005135583007358946, + 0.00513620900164824, + 0.005283209000481293, + 0.00518533299327828, + 0.005173208002815954, + 0.005303375000949018, + 0.004998292002710514, + 0.005164458998478949, + 0.005082791001768783, + 0.004980290992534719, + 0.005502208994585089, + 0.005298792006215081, + 0.005140458000823855, + 0.005166959002963267, + 0.005141125002410263, + 0.005118750006658956, + 0.005079624999780208, + 0.0051414159970590845, + 0.005045999991125427, + 0.005154167010914534, + 0.0051582499872893095 + ] + }, + { + "forward": 0.06885541600058787, + "prefill": 0.06414354201115202, + "decode_steps": [ + 0.015262874992913567, + 0.010021415990195237, + 0.00834949999989476, + 0.007707958007813431, + 0.007060374991851859, + 0.006342708002193831, + 0.006707666994770989, + 0.006778333001420833, + 0.0063563750009052455, + 0.005619333998765796, + 0.005841707999934442, + 0.005821124999783933, + 0.0055844579910626635, + 0.005926333004026674, + 0.005889583990210667, + 0.0056691659992793575, + 0.005896749993553385, + 0.0055460830044467, + 0.005582666999544017, + 0.005710083001758903, + 0.0058273339964216575, + 0.005608708001091145, + 0.005527667002752423, + 0.005627832986647263, + 0.005529916001250967, + 0.005474040997796692, + 0.0053208330064080656, + 0.0052440830040723085, + 0.0052129169926047325, + 0.005061958989244886, + 0.005108416997245513, + 0.005159000007552095 + ] + }, + { + "forward": 0.06652183400001377, + "prefill": 0.06383674999233335, + "decode_steps": [ + 0.007680791008169763, + 0.006414249990484677, + 0.0059872919955523685, + 0.006895249985973351, + 0.005724750008084811, + 0.005925999997998588, + 0.005334125002264045, + 0.004856415995163843, + 0.005164625006727874, + 0.005292167013976723, + 0.005261583995888941, + 0.00517841600230895, + 0.0051772919978247955, + 0.005108915996970609, + 0.005219457991188392, + 0.005190958007005975, + 0.005116582993650809, + 0.005148416996235028, + 0.005197958002099767, + 0.005202625005040318, + 0.005075624998426065, + 0.005206166999414563, + 0.005120667003211565, + 0.005160375003470108, + 0.005023290999815799, + 0.005022500001359731, + 0.0051461660041240975, + 0.00522254200768657, + 0.005128207994857803, + 0.005161458990187384, + 0.005190208001295105, + 0.005058416005340405 + ] + }, + { + "forward": 0.06462558299244847, + "prefill": 0.06242637500690762, + "decode_steps": [ + 0.006817916000727564, + 0.0059688339970307425, + 0.004933999996865168, + 0.005212999996729195, + 0.005285082996124402, + 0.00503841599856969, + 0.00533262500539422, + 0.005178124993108213, + 0.005016625000280328, + 0.005159582986379974, + 0.005120916990563273, + 0.005123750001075678, + 0.005142041001818143, + 0.005014082998968661, + 0.005120582995004952, + 0.005051125001045875, + 0.00514029199257493, + 0.005158125000889413, + 0.0050172920018667355, + 0.005168208008399233, + 0.005075833003502339, + 0.00509020900062751, + 0.005211208001128398, + 0.005062583994003944, + 0.00497112498851493, + 0.005451625009300187, + 0.005134624996571802, + 0.00518795799871441, + 0.005136625011800788, + 0.005118375003803521, + 0.005110375001095235, + 0.005064458004198968 + ] + }, + { + "forward": 0.0694589999911841, + "prefill": 0.0662954580038786, + "decode_steps": [ + 0.010808541992446408, + 0.006913208999321796, + 0.006754415997420438, + 0.006396041993866675, + 0.006638541992288083, + 0.006036749997292645, + 0.005766832997323945, + 0.005761083011748269, + 0.005657416011672467, + 0.005543874998693354, + 0.0055048329959390685, + 0.005435540995677002, + 0.0055570839904248714, + 0.005390416001318954, + 0.0054497920064022765, + 0.005458832994918339, + 0.0053553749894490466, + 0.005338417002349161, + 0.005266874999506399, + 0.005324540994479321, + 0.00532816699706018, + 0.005457375009427778, + 0.005128708988195285, + 0.005420916990260594, + 0.00517937500262633, + 0.005308958003297448, + 0.005383833995438181, + 0.005400333000579849, + 0.005113000006531365, + 0.005291792011121288, + 0.005924916986259632, + 0.0054151250078575686 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 67.6402504905127, + "p95": 71.9088836980518, + "min": 64.00912499520928, + "max": 72.99645899911411, + "iqr": 2.85186423570849, + "mean": 67.87619179667672, + "cv": 0.040069297844375476 + }, + "prefill_ms": { + "count": 10, + "median": 63.568833509634715, + "p95": 65.74510844948236, + "min": 61.669292001170106, + "max": 66.2954580038786, + "iqr": 1.4038540030014701, + "mean": 63.63605850347085, + "cv": 0.021161913228061786 + }, + "decode_ms_per_step": { + "count": 320, + "median": 5.227749999903608, + "p95": 7.070037492667327, + "min": 4.793542000697926, + "max": 15.507792006246746, + "iqr": 0.4677920005633496, + "mean": 5.622192284272387, + "cv": 0.20741394783939185 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 736.9894001962565, + "p95": 768.4240786578491, + "min": 622.4578490518533, + "max": 768.5785491568887, + "iqr": 56.29087957600518, + "mean": 715.1816189822209, + "cv": 0.0737452678364183 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 540.6800806229136, + "p95": 557.797093375243, + "min": 472.832672762596, + "max": 558.8494735763392, + "iqr": 37.7904713694312, + "mean": 527.2750722917364, + "cv": 0.05865586782009011 + }, + "memory": { + "peak_extra_bytes": 259530752, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "auto", + "batch": 4, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.061431207999703474 + }, + { + "forward": 0.06052395899314433 + }, + { + "forward": 0.060395249995053746 + }, + { + "forward": 0.06193341698963195 + }, + { + "forward": 0.06062375000328757 + }, + { + "forward": 0.06058070799917914 + }, + { + "forward": 0.061178499992820434 + }, + { + "forward": 0.06085437499859836 + }, + { + "forward": 0.06033987499540672 + }, + { + "forward": 0.062462041998514906 + } + ], + "forward_ms": { + "count": 10, + "median": 60.739062500942964, + "p95": 62.224160744517576, + "min": 60.33987499540672, + "max": 62.462041998514906, + "iqr": 0.8298847533296794, + "mean": 61.03230839653406, + "cv": 0.011691980200045282 + }, + "memory": { + "peak_extra_bytes": 47202304, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "sdpa", + "batch": 4, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.043695166998077184, + "prefill": 0.04321204200095963, + "decode_steps": [ + 0.003351417006342672, + 0.003369958998518996, + 0.003234125004382804, + 0.0031516660092165694, + 0.003225167005439289, + 0.0031301660055760294, + 0.0032677080016583204, + 0.0033884160075103864, + 0.003208834008546546, + 0.0032079999946290627, + 0.0032232079975074157, + 0.0032957079965854064, + 0.0033532499946886674, + 0.0034160409995820373, + 0.00341258299886249, + 0.003590916996472515, + 0.0035189159971196204, + 0.0036215830041328445, + 0.003918000002158806, + 0.0038703330064890906, + 0.004014707999886014, + 0.003908291997504421, + 0.0037615409964928403, + 0.004200124996714294, + 0.004169708001427352, + 0.004189582992694341, + 0.004262500006007031, + 0.004516541986959055, + 0.0049403329903725535, + 0.005216999998083338, + 0.006492875007097609, + 0.006039375002728775 + ] + }, + { + "forward": 0.04258845899312291, + "prefill": 0.04202820800128393, + "decode_steps": [ + 0.0034286250011064112, + 0.0032907499989960343, + 0.0032246670016320422, + 0.0031673749908804893, + 0.0032066249987110496, + 0.003238374993088655, + 0.003212458992493339, + 0.003203667001798749, + 0.003193791999365203, + 0.0032207079930230975, + 0.003216457989765331, + 0.0031400000007124618, + 0.003216416997020133, + 0.003184458997566253, + 0.0032072499889181927, + 0.0031801670120330527, + 0.0032035839976742864, + 0.003214458003640175, + 0.003206708002835512, + 0.003267665990279056, + 0.0033353329927194864, + 0.0032732079998822883, + 0.003279459007899277, + 0.0033437499951105565, + 0.003341458010254428, + 0.003499291997286491, + 0.0035534579947125167, + 0.0034679589880397543, + 0.003513290997943841, + 0.003761875006603077, + 0.003749916999367997, + 0.003688791999593377 + ] + }, + { + "forward": 0.042323541987570934, + "prefill": 0.042119541991269216, + "decode_steps": [ + 0.003542709004250355, + 0.003343292002682574, + 0.003183082997566089, + 0.0032375830051023513, + 0.003246332998969592, + 0.0032324999920092523, + 0.003222542000003159, + 0.003178124999976717, + 0.0032614170049782842, + 0.0032239580032182857, + 0.003171874996041879, + 0.003320708012324758, + 0.003340416995342821, + 0.0033203749917447567, + 0.0033555840054759756, + 0.0033102080051321536, + 0.0035158749960828573, + 0.0035890419967472553, + 0.003518666999298148, + 0.003460958003415726, + 0.00380304099235218, + 0.003763209009775892, + 0.003796500008320436, + 0.0037524170038523152, + 0.003973500002757646, + 0.003965625001001172, + 0.003971334008383565, + 0.00394758400216233, + 0.004534957988653332, + 0.00498479200177826, + 0.005052334003266878, + 0.006143000005977228 + ] + }, + { + "forward": 0.04583787500450853, + "prefill": 0.04327379199094139, + "decode_steps": [ + 0.003435834005358629, + 0.0033644580107647926, + 0.0033064590097637847, + 0.003391625010408461, + 0.0031787089974386618, + 0.003206833003787324, + 0.003151249999064021, + 0.0032007080008042976, + 0.0033992910030065104, + 0.0032094579946715385, + 0.0031437919969903305, + 0.0031948340038070455, + 0.003217332996428013, + 0.0031437919969903305, + 0.0032456670014653355, + 0.0031794169917702675, + 0.003225541004212573, + 0.0031671659962739795, + 0.0033117499988293275, + 0.003299250005511567, + 0.0033518339914735407, + 0.003286332997959107, + 0.003284624996013008, + 0.0034932499984279275, + 0.003516791999572888, + 0.0035277919960208237, + 0.0035337919980520383, + 0.003718333988217637, + 0.0037680420064134523, + 0.0037660419911844656, + 0.00371633299801033, + 0.004058958002133295 + ] + }, + { + "forward": 0.04243500001030043, + "prefill": 0.042646542002330534, + "decode_steps": [ + 0.0034012079995591193, + 0.003336750000016764, + 0.0031980829953681678, + 0.00324062499566935, + 0.003222917002858594, + 0.0032073750044219196, + 0.0032088750012917444, + 0.0031839170114835724, + 0.003217042001779191, + 0.003173374992911704, + 0.0032011669973144308, + 0.003236791002564132, + 0.0032336670119548216, + 0.0032410830026492476, + 0.0032189159974223003, + 0.0031571250001434237, + 0.0032109159947140142, + 0.0032823750079842284, + 0.003318292001495138, + 0.0032700000010663643, + 0.003339583010529168, + 0.0032856250036275014, + 0.00353129199356772, + 0.0035192909999750555, + 0.0035071669990429655, + 0.0035210420028306544, + 0.0037623750104103237, + 0.0037365830066846684, + 0.003729167001438327, + 0.003782250001677312, + 0.003959416993893683, + 0.003936041000997648 + ] + }, + { + "forward": 0.04205987499153707, + "prefill": 0.04203770900494419, + "decode_steps": [ + 0.0034617500059539452, + 0.003400333007448353, + 0.0032147089950740337, + 0.0032194170053116977, + 0.0032208750053541735, + 0.003316624992294237, + 0.00321033300133422, + 0.003220291997422464, + 0.0032352090056519955, + 0.0031978749902918935, + 0.0032463749957969412, + 0.0032241670123767108, + 0.003158874998916872, + 0.0032361250050598755, + 0.0031580409995513037, + 0.0032128750026458874, + 0.003224415995646268, + 0.003194332995917648, + 0.0032765000069048256, + 0.0033079170098062605, + 0.0033186250075232238, + 0.0032840829953784123, + 0.003330250008730218, + 0.0035039590002270415, + 0.003520458994898945, + 0.0035247499909019098, + 0.0035107920120935887, + 0.003764167006011121, + 0.0038067919958848506, + 0.003762333988561295, + 0.0037949580000713468, + 0.003956667002057657 + ] + }, + { + "forward": 0.052970832999562845, + "prefill": 0.04509191699617077, + "decode_steps": [ + 0.004646125002182089, + 0.004206541998428293, + 0.0051656669966178015, + 0.0041773749981075525, + 0.004399375000502914, + 0.004265583993401378, + 0.004011583994724788, + 0.003931875005946495, + 0.0038981670077191666, + 0.004222375006065704, + 0.0041377080051461235, + 0.003926916993805207, + 0.004083500010892749, + 0.0040244170086225495, + 0.003969292010879144, + 0.004036750004161149, + 0.004112292008358054, + 0.003990875004092231, + 0.003966000003856607, + 0.003935749991796911, + 0.003991000005044043, + 0.004060375009430572, + 0.003976292005972937, + 0.004130999994231388, + 0.003943666000850499, + 0.003976457999669947, + 0.003946208002162166, + 0.00393241700658109, + 0.003906332989572547, + 0.003939583999454044, + 0.004012290999526158, + 0.004034959012642503 + ] + }, + { + "forward": 0.04203799999959301, + "prefill": 0.042029625008581206, + "decode_steps": [ + 0.00348233399563469, + 0.0032753749983385205, + 0.0032517499930690974, + 0.0032460840011481196, + 0.0032222499867202714, + 0.0031940409971866757, + 0.0031729170004837215, + 0.003247291999286972, + 0.0032107919978443533, + 0.0032071669993456453, + 0.0031904579955153167, + 0.0031793329981155694, + 0.0032454579923069105, + 0.0031660410022595897, + 0.003496542005450465, + 0.0033200409961864352, + 0.003387624994502403, + 0.0033190000103786588, + 0.003525916996295564, + 0.0035044160031247884, + 0.003565416991477832, + 0.003535708994604647, + 0.0036225829971954226, + 0.0037814170063938946, + 0.003753041004529223, + 0.003723750007338822, + 0.0038985419960226864, + 0.00388474999635946, + 0.003899999996065162, + 0.004221707989927381, + 0.004941166989738122, + 0.005267332991934381 + ] + }, + { + "forward": 0.042036749990074895, + "prefill": 0.04214804199000355, + "decode_steps": [ + 0.003579249998438172, + 0.0034098329924745485, + 0.003259832999901846, + 0.00315970899828244, + 0.003259292003349401, + 0.003165750007610768, + 0.0032690829975763336, + 0.0032012919982662424, + 0.003193125012330711, + 0.003168791998177767, + 0.0032475829939357936, + 0.0032161669951165095, + 0.0033751250011846423, + 0.003295457994681783, + 0.0037010840023867786, + 0.003376624998054467, + 0.003724374997545965, + 0.00380399999266956, + 0.0035235409886809066, + 0.0035063749965047464, + 0.0037634579930454493, + 0.003815459000179544, + 0.0037888329970883206, + 0.00371354199887719, + 0.004149000000325032, + 0.003993582999100909, + 0.003996999992523342, + 0.0038770840037614107, + 0.005034499990870245, + 0.004958749996148981, + 0.005090167003800161, + 0.006110415997682139 + ] + }, + { + "forward": 0.045480040993425064, + "prefill": 0.04494200000772253, + "decode_steps": [ + 0.007487625000067055, + 0.006324583999230526, + 0.006041750006261282, + 0.006057375008822419, + 0.006039500003680587, + 0.006097416000557132, + 0.006514041000627913, + 0.005970833997707814, + 0.00596312498964835, + 0.005982625007163733, + 0.00591545800853055, + 0.005745290996856056, + 0.005563416008953936, + 0.005529583009774797, + 0.004980832993169315, + 0.005238292011199519, + 0.004890416006674059, + 0.005125041992869228, + 0.004978375000064261, + 0.004989500012015924, + 0.004968000008375384, + 0.0049917500000447035, + 0.005050541993114166, + 0.004997666997951455, + 0.005037458002334461, + 0.004519458001595922, + 0.005027874998631887, + 0.005032291999668814, + 0.004988458997104317, + 0.0050029999983962625, + 0.004981999998562969, + 0.004904874993371777 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 42.51172950171167, + "p95": 49.7610019017884, + "min": 42.036749990074895, + "max": 52.970832999562845, + "iqr": 2.9080307540425565, + "mean": 44.14655419677729, + "cv": 0.07723883061214301 + }, + "prefill_ms": { + "count": 10, + "median": 42.39729199616704, + "p95": 45.02445435136906, + "min": 42.02820800128393, + "max": 45.09191699617077, + "iqr": 1.2001872419205029, + "mean": 42.95294189942069, + "cv": 0.02763764153359109 + }, + "decode_ms_per_step": { + "count": 320, + "median": 3.504187501675915, + "p95": 5.572509758349044, + "min": 3.1301660055760294, + "max": 7.487625000067055, + "iqr": 0.7339065050473437, + "mean": 3.7872192938721128, + "cv": 0.2066891380444949 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1102.8621353922308, + "p95": 1196.057066345443, + "min": 731.6858863601276, + "max": 1204.909986268985, + "iqr": 139.2521699641777, + "mean": 1077.6911162004196, + "cv": 0.1325364885067901 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 809.3802199336519, + "p95": 858.7062466447435, + "min": 582.1345003410521, + "max": 863.3469495953763, + "iqr": 73.60030492482076, + "mean": 789.6764984688901, + "cv": 0.10719239075772596 + }, + "memory": { + "peak_extra_bytes": 47202304, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "reference", + "batch": 4, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.22192645800532773, + "prefill": 0.21407949998683762, + "decode_steps": [ + 0.018928667006548494, + 0.014214041002560407, + 0.010456749994773418, + 0.009186291994410567, + 0.008479333991999738, + 0.008202500001061708, + 0.008100124992779456, + 0.007899000003817491, + 0.007561958002042957, + 0.00740145800227765, + 0.007605207996675745, + 0.007178917003329843, + 0.00687679200200364, + 0.006435125003918074, + 0.0068005000066477805, + 0.00683154200669378, + 0.006697624994558282, + 0.006687582994345576, + 0.006728916996507905, + 0.007375375003903173, + 0.00708741600101348, + 0.006951999996090308, + 0.006459458003519103, + 0.006563959002960473, + 0.006374541990226135, + 0.006627041002502665, + 0.006459208991145715, + 0.006472916997154243, + 0.006434416005504318, + 0.006351625008392148, + 0.006376000004820526, + 0.006415249998099171 + ] + }, + { + "forward": 0.2163966249936493, + "prefill": 0.21634591701149475, + "decode_steps": [ + 0.018600124996737577, + 0.014230415996280499, + 0.010331666009733453, + 0.009184791008010507, + 0.008964957989519462, + 0.008838000008836389, + 0.00838016701163724, + 0.007909166990430094, + 0.007846249995054677, + 0.00759845798893366, + 0.007381499992334284, + 0.007449875003658235, + 0.007042333003482781, + 0.0069745829969178885, + 0.006846583986771293, + 0.006863625007099472, + 0.006684292005957104, + 0.006641957996180281, + 0.007796124991727993, + 0.007805750006809831, + 0.007315584007301368, + 0.006635333003941923, + 0.006602417008252814, + 0.006501957992441021, + 0.006446542000048794, + 0.006480583004304208, + 0.006306832990958355, + 0.006528749989229254, + 0.006358500002534129, + 0.006424499995773658, + 0.00638612499460578, + 0.0062760000000707805 + ] + }, + { + "forward": 0.21373241700348444, + "prefill": 0.21435466599359643, + "decode_steps": [ + 0.015593292002449743, + 0.013341292011318728, + 0.010349624993978068, + 0.00849416600249242, + 0.007930624997243285, + 0.007399249996524304, + 0.007208499999251217, + 0.006802832998801023, + 0.006967292007175274, + 0.006234792002942413, + 0.006440583005314693, + 0.0061887500050943345, + 0.006080624996684492, + 0.005899707990465686, + 0.006107582987169735, + 0.006067750000511296, + 0.006081957995775156, + 0.00597974999982398, + 0.006096124998293817, + 0.00606066599721089, + 0.005960791997495107, + 0.0059895409940509126, + 0.0060872079920955, + 0.006037667000782676, + 0.006151999987196177, + 0.005957999994279817, + 0.006041374988853931, + 0.005889290987397544, + 0.006421208003303036, + 0.006221667004865594, + 0.006139916004030965, + 0.0060615420079557225 + ] + }, + { + "forward": 0.22179595900524873, + "prefill": 0.2151376249967143, + "decode_steps": [ + 0.01925104099791497, + 0.014567833000910468, + 0.01113012499990873, + 0.009464625007240102, + 0.008684333006385714, + 0.008185958009562455, + 0.00835475001076702, + 0.007633291999809444, + 0.00730091700097546, + 0.007332375011174008, + 0.0071176669880514964, + 0.0069662079913541675, + 0.006933290991582908, + 0.006973708004807122, + 0.007899208008893766, + 0.0073650839913170785, + 0.007912540997494943, + 0.007239958009449765, + 0.006863249989692122, + 0.006664708998869173, + 0.0065959159983322024, + 0.006968292000237852, + 0.006560458001331426, + 0.0064512079989071935, + 0.0065597920038271695, + 0.006409249996067956, + 0.0064220409985864535, + 0.0064282090024789795, + 0.0062347079947358, + 0.006207166996318847, + 0.006288250006036833, + 0.00662595899484586 + ] + }, + { + "forward": 0.21531770900764968, + "prefill": 0.212939874996664, + "decode_steps": [ + 0.007944874989334494, + 0.0069005419936729595, + 0.006900124993990175, + 0.0061636250029550865, + 0.006086958994274028, + 0.0062400000024354085, + 0.006153249996714294, + 0.006460417003836483, + 0.006199499999638647, + 0.006228999991435558, + 0.006199000010383315, + 0.007236290999571793, + 0.0069336250016931444, + 0.006981167010962963, + 0.006260791007662192, + 0.00614441599464044, + 0.006078458987758495, + 0.006187124992720783, + 0.006106875007390045, + 0.006168499996419996, + 0.006356917001539841, + 0.006180374999530613, + 0.006159874988952652, + 0.00600145899807103, + 0.006013332997099496, + 0.006041291999281384, + 0.005940542003372684, + 0.006100500002503395, + 0.006141334000858478, + 0.005977624998195097, + 0.006177957999170758, + 0.00614612500066869 + ] + }, + { + "forward": 0.2144958749995567, + "prefill": 0.2139877500012517, + "decode_steps": [ + 0.008151667003403418, + 0.006910707990755327, + 0.006848207995062694, + 0.0068437500012805685, + 0.007601791003253311, + 0.008010624995222315, + 0.007868290995247662, + 0.008302292000735179, + 0.007851249989471398, + 0.007620541990036145, + 0.007106416000169702, + 0.00679266698716674, + 0.0066066250001313165, + 0.0068569589930120856, + 0.00640891699003987, + 0.006130958994617686, + 0.006064707995392382, + 0.005847332999110222, + 0.006134583003586158, + 0.006118125005741604, + 0.005961457994999364, + 0.006105833002948202, + 0.006002082998747937, + 0.006029124997439794, + 0.005998749998980202, + 0.00594108400400728, + 0.006227208999916911, + 0.006058167011360638, + 0.006069958995794877, + 0.00588779199460987, + 0.0064203749934677035, + 0.006250165999517776 + ] + }, + { + "forward": 0.21689666699967347, + "prefill": 0.21640712500084192, + "decode_steps": [ + 0.019028458002139814, + 0.015188709003268741, + 0.011140374990645796, + 0.009209917014231905, + 0.008599167005741037, + 0.008456000010482967, + 0.007974749998538755, + 0.007730249999440275, + 0.007443624999723397, + 0.007336209004279226, + 0.007194624995463528, + 0.007538791993283667, + 0.007220416999189183, + 0.007119583999156021, + 0.006883916998049244, + 0.0068652500049211085, + 0.006931124997208826, + 0.00674629099376034, + 0.00666724999609869, + 0.006647499991231598, + 0.006577416992513463, + 0.0065422080078860745, + 0.006699167002807371, + 0.006718708988046274, + 0.0066534999932628125, + 0.00689304199477192, + 0.006729999993694946, + 0.006697666991385631, + 0.006658624988631345, + 0.006554833002155647, + 0.006281458001467399, + 0.006093292002333328 + ] + }, + { + "forward": 0.21991475000686478, + "prefill": 0.21723741599998903, + "decode_steps": [ + 0.018423292000079527, + 0.01349354199192021, + 0.010140250000404194, + 0.009335249997093342, + 0.00840579200303182, + 0.008087416994385421, + 0.007638124996447004, + 0.00740166699688416, + 0.007306790997972712, + 0.00689845799934119, + 0.0065918339969357476, + 0.006483499993919395, + 0.006181124990689568, + 0.006221916002687067, + 0.0060477910010376945, + 0.00600629199470859, + 0.006003583999699913, + 0.006182666998938657, + 0.006038500010618009, + 0.00611804099753499, + 0.006056166996131651, + 0.006119166995631531, + 0.006059000006644055, + 0.006055624995497055, + 0.006119416997535154, + 0.00609695800812915, + 0.00594608299434185, + 0.006121792001067661, + 0.005867540996405296, + 0.006814167005359195, + 0.006742875004420057, + 0.0066766660020221025 + ] + }, + { + "forward": 0.21485691699490417, + "prefill": 0.21346433299186174, + "decode_steps": [ + 0.0078125, + 0.006544750009197742, + 0.006324000001768582, + 0.006203458993695676, + 0.006140124998637475, + 0.006912167009431869, + 0.006301916000666097, + 0.0061876249965280294, + 0.006315542006632313, + 0.0066905830026371405, + 0.006881332999910228, + 0.006275791994994506, + 0.005980667003314011, + 0.006134540992206894, + 0.00611770800605882, + 0.006061833002604544, + 0.0059505419922061265, + 0.0060495840007206425, + 0.008748666994506493, + 0.014027249999344349, + 0.0071860000025480986, + 0.0067342499969527125, + 0.006215332992724143, + 0.006466833001468331, + 0.006478665993199684, + 0.006205041994689964, + 0.006535749998874962, + 0.00654124999709893, + 0.006121375001384877, + 0.006152625006507151, + 0.006900749998749234, + 0.006456540999352001 + ] + }, + { + "forward": 0.21632354200119153, + "prefill": 0.21353483300481457, + "decode_steps": [ + 0.018311708001419902, + 0.01380779201281257, + 0.010010375001002103, + 0.009055208007339388, + 0.008347166003659368, + 0.008066750000580214, + 0.008265041004051454, + 0.008013500002562068, + 0.0074326250032754615, + 0.008021625006222166, + 0.007472458994016051, + 0.00693320800201036, + 0.006913541001267731, + 0.006488749990239739, + 0.006781165997381322, + 0.0067858330003218725, + 0.006530915998155251, + 0.006544499992742203, + 0.006658875005086884, + 0.006514332999358885, + 0.006243083000299521, + 0.007488833012757823, + 0.006913332996191457, + 0.006634916004259139, + 0.006537583001772873, + 0.006467750004958361, + 0.006593458994757384, + 0.006347500006086193, + 0.006294499995419756, + 0.006387166999047622, + 0.006409790992620401, + 0.0063064160058274865 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 216.36008349742042, + "p95": 221.86773345529218, + "min": 213.73241700348444, + "max": 221.92645800532773, + "iqr": 4.188114256976405, + "mean": 217.16569190175505, + "cv": 0.013788339955681074 + }, + "prefill_ms": { + "count": 10, + "median": 214.21708299021702, + "p95": 216.86378505037283, + "min": 212.939874996664, + "max": 217.23741599998903, + "iqr": 2.395781753875781, + "mean": 214.7489039984066, + "cv": 0.0068055626059908255 + }, + "decode_ms_per_step": { + "count": 320, + "median": 6.630978503380902, + "p95": 11.130637499445584, + "min": 5.847332999110222, + "max": 19.25104099791497, + "iqr": 1.2004269956378266, + "mean": 7.304722502431105, + "cv": 0.3028038908799584 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 534.4851519438332, + "p95": 617.5061726057744, + "min": 508.6758077156825, + "max": 631.1267461964015, + "iqr": 76.47733845826195, + "mean": 550.8217330370119, + "cv": 0.08233955811191244 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 281.33122350313727, + "p95": 304.2213244513116, + "min": 273.5161580226711, + "max": 307.8760327419348, + "iqr": 21.761303610499283, + "mean": 285.88681869698723, + "cv": 0.04415056223370962 + }, + "memory": { + "peak_extra_bytes": 1022377984, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "auto", + "batch": 8, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.006893500001751818 + }, + { + "forward": 0.005728916003135964 + }, + { + "forward": 0.007642540993401781 + }, + { + "forward": 0.0054579999996349216 + }, + { + "forward": 0.005630917003145441 + }, + { + "forward": 0.00563004199648276 + }, + { + "forward": 0.005429915996501222 + }, + { + "forward": 0.006620375002967194 + }, + { + "forward": 0.005474041012348607 + }, + { + "forward": 0.005372375002480112 + } + ], + "forward_ms": { + "count": 10, + "median": 5.630479499814101, + "p95": 7.3054725471592965, + "min": 5.372375002480112, + "max": 7.642540993401781, + "iqr": 0.9355000001960434, + "mean": 5.988062301184982, + "cv": 0.13073415995613416 + }, + "memory": { + "peak_extra_bytes": 5899264, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "sdpa", + "batch": 8, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.006063542008632794, + "prefill": 0.006008208991261199, + "decode_steps": [ + 0.0021525829943129793, + 0.00208658300107345, + 0.002080707999994047, + 0.002114624992827885, + 0.002018832994508557, + 0.0022994159953668714, + 0.0019325419998494908, + 0.001924041993333958, + 0.00197295899852179, + 0.0019896669982699677, + 0.0019953329901909456, + 0.0020330419938545674, + 0.0020688329968834296, + 0.0020503340056166053, + 0.002051041999948211, + 0.001958082997589372, + 0.0019768339989241213, + 0.0019912919960916042, + 0.00205220899078995, + 0.0020434579928405583, + 0.0020656249980675057, + 0.0020584169978974387, + 0.0020439590007299557, + 0.001974417013116181, + 0.0020309579995227978, + 0.0020202500018058345, + 0.002042499996605329, + 0.0020667919889092445, + 0.0020763750071637332, + 0.0020729169918922707, + 0.002009083007578738, + 0.0019386669882806018 + ] + }, + { + "forward": 0.004968375011230819, + "prefill": 0.0049944589991355315, + "decode_steps": [ + 0.0021133749978616834, + 0.0019741249998332933, + 0.001987375013413839, + 0.001989083000808023, + 0.0020467499998630956, + 0.0020383749942993745, + 0.0019441670010564849, + 0.0020318330061854795, + 0.0019230829930165783, + 0.002056957993772812, + 0.0020003750105388463, + 0.0020589999912772328, + 0.002074999996693805, + 0.0020360829948913306, + 0.001955582993105054, + 0.0019597919890657067, + 0.0019512500002747402, + 0.001998457999434322, + 0.0019749579951167107, + 0.0021305419941199943, + 0.002060957995126955, + 0.002040584004134871, + 0.0019607089925557375, + 0.00190395800746046, + 0.0020401249930728227, + 0.00199387499014847, + 0.0021077080018585548, + 0.0020843750098720193, + 0.0020947079901816323, + 0.002059375008684583, + 0.001998499996261671, + 0.0020311249973019585 + ] + }, + { + "forward": 0.006412000002455898, + "prefill": 0.00577658400288783, + "decode_steps": [ + 0.002130958004272543, + 0.001977665990125388, + 0.0020065839926246554, + 0.002429625004879199, + 0.002061874998616986, + 0.002061916995444335, + 0.0019642500119516626, + 0.0020324999932199717, + 0.0019550830038497224, + 0.0019924579974031076, + 0.002124708000337705, + 0.0021422910067485645, + 0.002079708006931469, + 0.0020185829926049337, + 0.001965916992048733, + 0.001966291994904168, + 0.002035540994256735, + 0.002096166994306259, + 0.0020477910002227873, + 0.0019273329962743446, + 0.0020718330051749945, + 0.0020039170049130917, + 0.002040916995611042, + 0.0020067079894943163, + 0.002071583003271371, + 0.00201779100461863, + 0.0020229999936418608, + 0.002060583996353671, + 0.001985250011784956, + 0.0020781670027645305, + 0.0020407920092111453, + 0.002039582992438227 + ] + }, + { + "forward": 0.0048932920035440475, + "prefill": 0.004960417005349882, + "decode_steps": [ + 0.0020152920042164624, + 0.0020069589954800904, + 0.00197387499792967, + 0.002048582988209091, + 0.001976166997337714, + 0.0018913749954663217, + 0.002109916997142136, + 0.001952917009475641, + 0.0019443750061327592, + 0.0020104169961996377, + 0.0020073340128874406, + 0.0020364579977467656, + 0.001941665992490016, + 0.001973332997295074, + 0.001968999989912845, + 0.001955583007656969, + 0.001946084012161009, + 0.0019465000077616423, + 0.002013916993746534, + 0.001990624994505197, + 0.0019589169969549403, + 0.002076542004942894, + 0.002075458993203938, + 0.0020814159943256527, + 0.0019877079903380945, + 0.0020679169974755496, + 0.0019743749871850014, + 0.001979958004085347, + 0.002048707989160903, + 0.0020322499913163483, + 0.002042249994701706, + 0.002083416999084875 + ] + }, + { + "forward": 0.005249124995316379, + "prefill": 0.0049989999970421195, + "decode_steps": [ + 0.0021169169922359288, + 0.002037707992712967, + 0.002043375003268011, + 0.001931792008690536, + 0.0019642080005723983, + 0.00191966698912438, + 0.0019191669998690486, + 0.0019387500069569796, + 0.0019890420080628246, + 0.001977874999283813, + 0.0019524999952409416, + 0.0019088749977527186, + 0.0020370420097606257, + 0.002041749990894459, + 0.0020303329947637394, + 0.0021579580061370507, + 0.002049541988526471, + 0.002084125007968396, + 0.001997916988329962, + 0.0019726250029634684, + 0.001977082996745594, + 0.0020959170069545507, + 0.002139249991159886, + 0.002021583990426734, + 0.002025208988925442, + 0.0023891669989097863, + 0.002176959009375423, + 0.002031000010902062, + 0.002081334008835256, + 0.0020108339958824217, + 0.0020882920071016997, + 0.0020821250072913244 + ] + }, + { + "forward": 0.004736916002002545, + "prefill": 0.004990332992747426, + "decode_steps": [ + 0.0020625409961212426, + 0.002227207995019853, + 0.0021094589901622385, + 0.0020592500077327713, + 0.002008916999329813, + 0.001991582990740426, + 0.002059624996036291, + 0.0019019159954041243, + 0.0019972500012954697, + 0.0019749170023715124, + 0.0019916659948648885, + 0.001941124995937571, + 0.001959083994734101, + 0.0019737919938052073, + 0.0020459580118767917, + 0.002075791999232024, + 0.0020796669996343553, + 0.0020172080112388358, + 0.001949083001818508, + 0.002017708000494167, + 0.0020979170076316223, + 0.002099207995343022, + 0.0019934579904656857, + 0.001988374991924502, + 0.002120374992955476, + 0.0021389159956015646, + 0.001989833006518893, + 0.0020509579917415977, + 0.0020797079923795536, + 0.002055999997537583, + 0.002074333999189548, + 0.002034832999925129 + ] + }, + { + "forward": 0.0048910000041360036, + "prefill": 0.004957083991030231, + "decode_steps": [ + 0.00209329099743627, + 0.0020773750002263114, + 0.00197275000391528, + 0.0020413750025909394, + 0.002048750000540167, + 0.0020338329923106357, + 0.002056333003565669, + 0.0019976670009782538, + 0.0019819170120172203, + 0.0019907079986296594, + 0.001993124998989515, + 0.002038457998423837, + 0.002043624990619719, + 0.0020532079943222925, + 0.001997125000343658, + 0.0019897909951396286, + 0.0019453330023679882, + 0.0019183749973308295, + 0.0021625840017804876, + 0.0020848749991273507, + 0.0021063329913886264, + 0.0019520420028129593, + 0.0020040829986101016, + 0.0020393340091686696, + 0.00200304199825041, + 0.0020244589977664873, + 0.002093666000291705, + 0.0020403750095283613, + 0.002005166999879293, + 0.0019814590050373226, + 0.002017500009969808, + 0.0020474999910220504 + ] + }, + { + "forward": 0.004924749999190681, + "prefill": 0.004938291996950284, + "decode_steps": [ + 0.0022080000053392723, + 0.0021221250062808394, + 0.001964540992048569, + 0.0020099579996895045, + 0.0020530000037979335, + 0.002050249997409992, + 0.002082249993691221, + 0.002126084000337869, + 0.001959957997314632, + 0.0019581669912440702, + 0.00197108399879653, + 0.00198583300516475, + 0.0020610840001609176, + 0.002038582999375649, + 0.0019871249969583005, + 0.0020446249982342124, + 0.002088666005874984, + 0.002020582993282005, + 0.0020290419925004244, + 0.001960749999852851, + 0.0020749169925693423, + 0.0021038330014562234, + 0.0020834580063819885, + 0.001984749993425794, + 0.0019797499990090728, + 0.002010459007578902, + 0.001996542006963864, + 0.0020052500040037557, + 0.0019607910071499646, + 0.002100334007991478, + 0.0021099590085214004, + 0.0021498340065591037 + ] + }, + { + "forward": 0.004871249999268912, + "prefill": 0.004853958991589025, + "decode_steps": [ + 0.0021065839973744005, + 0.0019654579955386, + 0.0021170420077396557, + 0.002189457998611033, + 0.002059582999208942, + 0.002142082987120375, + 0.001936708009452559, + 0.0019806670024991035, + 0.0019585419940995052, + 0.002230792000773363, + 0.00202379199618008, + 0.0020455830090213567, + 0.0019429999956628308, + 0.00203312499797903, + 0.0020082919945707545, + 0.001959624991286546, + 0.0019712089997483417, + 0.002053791002254002, + 0.002051624993328005, + 0.0020588330080499873, + 0.002078249992337078, + 0.001948917008121498, + 0.0020322079944889992, + 0.0019878750026691705, + 0.0020027080026920885, + 0.001995250000618398, + 0.0019832920079352334, + 0.0019644999993033707, + 0.00199350000184495, + 0.001971207995666191, + 0.0021322499960660934, + 0.0019825829949695617 + ] + }, + { + "forward": 0.004804374999366701, + "prefill": 0.004852666999795474, + "decode_steps": [ + 0.002037500002188608, + 0.001921666000271216, + 0.0019105839892290533, + 0.001916416993481107, + 0.0019330000068293884, + 0.0019217090011807159, + 0.0019252920028520748, + 0.00197725000907667, + 0.001924750002217479, + 0.0019457909947959706, + 0.001964750001206994, + 0.0018647909892024472, + 0.001959375003934838, + 0.002018625003984198, + 0.0019519170018611476, + 0.002129290995071642, + 0.00203200000396464, + 0.001961292000487447, + 0.0019877080048900098, + 0.001980166998691857, + 0.002186666999477893, + 0.0019791670056292787, + 0.002048709007794969, + 0.0019505829986883327, + 0.002050041002803482, + 0.002012333003222011, + 0.001968541997484863, + 0.0020048330043209717, + 0.001962290989467874, + 0.002017167003941722, + 0.0019993339956272393, + 0.0020496660144999623 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 4.909021001367364, + "p95": 6.255193905235501, + "min": 4.736916002002545, + "max": 6.412000002455898, + "iqr": 0.3027499988093041, + "mean": 5.181462502514478, + "cv": 0.11164192423797395 + }, + "prefill_ms": { + "count": 10, + "median": 4.975374999048654, + "p95": 5.903977746493183, + "min": 4.852666999795474, + "max": 6.008208991261199, + "iqr": 0.054874752095201984, + "mean": 5.1331003967789, + "cv": 0.0793410581363776 + }, + "decode_ms_per_step": { + "count": 320, + "median": 2.0195414981571957, + "p95": 2.1389326953794807, + "min": 1.8647909892024472, + "max": 2.429625004879199, + "iqr": 0.08633400284452364, + "mean": 2.025835558606559, + "cv": 0.03512322184356094 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 3936.176137332912, + "p95": 4014.2111255481577, + "min": 3910.9422247663906, + "max": 4031.92731867619, + "iqr": 31.354569623853422, + "mean": 3949.3025814271086, + "cv": 0.00946046047051649 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 3658.4603050408077, + "p95": 3727.748407917045, + "min": 3593.79170322108, + "max": 3745.6539973257936, + "iqr": 29.269667422174734, + "mean": 3659.7583484729425, + "cv": 0.012531674532096889 + }, + "memory": { + "peak_extra_bytes": 5899264, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "reference", + "batch": 8, + "seq_len": 128, + "samples_seconds": [ + { + "forward": 0.010671208001440391, + "prefill": 0.006981665996136144, + "decode_steps": [ + 0.004777750000357628, + 0.004588291005347855, + 0.004526000004261732, + 0.00468512499355711, + 0.004452167006093077, + 0.004393791998154484, + 0.00447225000243634, + 0.004340250001405366, + 0.004391583002870902, + 0.0043186659895582125, + 0.004374916999950074, + 0.004395083000417799, + 0.004391584006953053, + 0.0043672500032698736, + 0.004365041997516528, + 0.004422750003868714, + 0.004817417007870972, + 0.004592957993736491, + 0.004435499999090098, + 0.0044499170035123825, + 0.0044375839934218675, + 0.0044948330032639205, + 0.004555874998914078, + 0.004270707999239676, + 0.0043955830042250454, + 0.00438366700836923, + 0.004481958007090725, + 0.004487250000238419, + 0.004577042011078447, + 0.00441691699961666, + 0.004424707993166521, + 0.004549958001007326 + ] + }, + { + "forward": 0.00969237499521114, + "prefill": 0.006975374999456108, + "decode_steps": [ + 0.0045358339993981645, + 0.004441750003024936, + 0.00529095801175572, + 0.004563291993690655, + 0.005047500002547167, + 0.004584082998917438, + 0.004206750003504567, + 0.004619458006345667, + 0.00465358400833793, + 0.004462666998733766, + 0.004543666000245139, + 0.004446209000889212, + 0.004461999997147359, + 0.004531499987933785, + 0.0044832080020569265, + 0.004528708013822325, + 0.00443016599456314, + 0.0044803330092690885, + 0.004362333987955935, + 0.0044213329965714365, + 0.004480167001020163, + 0.0045635830028913915, + 0.004376875003799796, + 0.0052724999986821786, + 0.004940457991324365, + 0.0047043749946169555, + 0.004619958010152914, + 0.0046131249982863665, + 0.004716833995189518, + 0.004930709008476697, + 0.004495083005167544, + 0.004553749997285195 + ] + }, + { + "forward": 0.01035516599949915, + "prefill": 0.007074084001942538, + "decode_steps": [ + 0.004662749997805804, + 0.004546833006315865, + 0.004227083991281688, + 0.004510583006776869, + 0.004469874998903833, + 0.004518791000009514, + 0.004620292005711235, + 0.004466166996280663, + 0.004372707990114577, + 0.0044218750117579475, + 0.004543792005279101, + 0.004230999998981133, + 0.004449125000974163, + 0.004432041998370551, + 0.004405416009831242, + 0.004420332988956943, + 0.004437583003891632, + 0.004464249999728054, + 0.0044647920003626496, + 0.004447416999028064, + 0.004453457993804477, + 0.0044028749980498105, + 0.00435020899749361, + 0.00444329199672211, + 0.004384250001749024, + 0.004463290999410674, + 0.004203917007544078, + 0.017089207991375588, + 0.008304500006488524, + 0.005697125001461245, + 0.00498383300146088, + 0.005034207992139272 + ] + }, + { + "forward": 0.010050750002847053, + "prefill": 0.006867542004329152, + "decode_steps": [ + 0.005243916995823383, + 0.004825249998248182, + 0.0047751660022186115, + 0.00458608299959451, + 0.004588625000906177, + 0.004884542009676807, + 0.004708541993750259, + 0.00492104199656751, + 0.0048787920095492154, + 0.004793665997567587, + 0.004722124998806976, + 0.004635833989595994, + 0.004492542007938027, + 0.0045695000007981434, + 0.004617582992068492, + 0.004649666996556334, + 0.004463374993065372, + 0.004532875012955628, + 0.004554500002996065, + 0.0043440419976832345, + 0.004542374997981824, + 0.004312667006161064, + 0.00467075000051409, + 0.0045514170051319525, + 0.004370166992885061, + 0.0047595829964848235, + 0.004679707999457605, + 0.004431957990163937, + 0.0044250830105738714, + 0.004459333998966031, + 0.004473792010685429, + 0.004582499997923151 + ] + }, + { + "forward": 0.01010491700435523, + "prefill": 0.0074406250059837475, + "decode_steps": [ + 0.004851416000747122, + 0.004617458005668595, + 0.004528708988800645, + 0.004322041990235448, + 0.004423542006406933, + 0.004366749999462627, + 0.004429749998962507, + 0.00449933300842531, + 0.004453708999790251, + 0.004429833992617205, + 0.0043221249943599105, + 0.00511354100308381, + 0.004711084009613842, + 0.0052755420038010925, + 0.0044571250036824495, + 0.004531915998086333, + 0.004655750002712011, + 0.004493457992793992, + 0.004664583000703715, + 0.004561416993965395, + 0.004567665993818082, + 0.004429166001500562, + 0.004381165999802761, + 0.004618458988261409, + 0.004542124996078201, + 0.00450833399372641, + 0.0045439999958034605, + 0.0044348749943310395, + 0.004537208005785942, + 0.004527666009380482, + 0.0043870840017916635, + 0.004467167003895156 + ] + }, + { + "forward": 0.00938112499716226, + "prefill": 0.007072583990520798, + "decode_steps": [ + 0.004569540993543342, + 0.004396624994114973, + 0.004417208008817397, + 0.004543792005279101, + 0.004505792006966658, + 0.004551165999146178, + 0.00444587500533089, + 0.0042792500025825575, + 0.004404332998092286, + 0.004551625010208227, + 0.004331207994255237, + 0.004382790997624397, + 0.004211209001368843, + 0.0044904160022269934, + 0.004432792004081421, + 0.004450916007044725, + 0.004473208013223484, + 0.004549667006358504, + 0.004864832997554913, + 0.004602500004693866, + 0.004571707991999574, + 0.00439429200196173, + 0.004844958006287925, + 0.004974833005690016, + 0.004428958010976203, + 0.004565041992464103, + 0.00493508399813436, + 0.004762750002555549, + 0.0043126249947818, + 0.004535499989287928, + 0.004535915999440476, + 0.004430375003721565 + ] + }, + { + "forward": 0.010421750004752539, + "prefill": 0.006947166999452747, + "decode_steps": [ + 0.0047742500028107315, + 0.00468570800148882, + 0.0045011250040261075, + 0.004496624998864718, + 0.004442834004294127, + 0.004614000004949048, + 0.004479875002289191, + 0.004450541993719526, + 0.004479958995943889, + 0.004286583003704436, + 0.004423958002007566, + 0.0045559580030385405, + 0.004262624992406927, + 0.00480533299560193, + 0.00450549999368377, + 0.004291834004106931, + 0.004964499996276572, + 0.00452645801124163, + 0.004464249999728054, + 0.0042899590043816715, + 0.004416291005327366, + 0.00427208399923984, + 0.004521875001955777, + 0.004560834000585601, + 0.004454916997929104, + 0.004503000003751367, + 0.004403167011332698, + 0.0045647919905604795, + 0.004479250012082048, + 0.004330041992943734, + 0.004524499992839992, + 0.0044323750043986365 + ] + }, + { + "forward": 0.010219417003099807, + "prefill": 0.006943041997146793, + "decode_steps": [ + 0.004706958992755972, + 0.004662459003156982, + 0.0045463339920388535, + 0.0044719580037053674, + 0.00444329199672211, + 0.004562959002214484, + 0.0044935000041732565, + 0.004444999998668209, + 0.004331874995841645, + 0.004471541993552819, + 0.004418291006004438, + 0.004499291011597961, + 0.004478624992771074, + 0.00443129098857753, + 0.004488250007852912, + 0.004353832991910167, + 0.004470542000490241, + 0.004344457993283868, + 0.004342833999544382, + 0.00458029200672172, + 0.005309542000759393, + 0.005271667003398761, + 0.005055082991020754, + 0.004963625004165806, + 0.004811540988157503, + 0.004670041991630569, + 0.0049694999906932935, + 0.00526779200299643, + 0.005079666996607557, + 0.004841999994823709, + 0.004834041988942772, + 0.004737291994388215 + ] + }, + { + "forward": 0.00950429200020153, + "prefill": 0.006998165990808047, + "decode_steps": [ + 0.00453262499650009, + 0.00444233299640473, + 0.004428084008395672, + 0.004498333000810817, + 0.0044677920086542144, + 0.004342250002082437, + 0.004622000007657334, + 0.004463542005396448, + 0.004455583999515511, + 0.0043341249984223396, + 0.004447207989869639, + 0.0045747910044156015, + 0.004387332999613136, + 0.004324250010540709, + 0.004342500003986061, + 0.005102750001242384, + 0.004755874993861653, + 0.004391750000650063, + 0.004213958003674634, + 0.004683332997956313, + 0.00484383299772162, + 0.004469541003345512, + 0.004569292010273784, + 0.00446029199520126, + 0.004569208002067171, + 0.004544040988548659, + 0.004519165988313034, + 0.004368458001408726, + 0.004477374997804873, + 0.004379915990284644, + 0.004446207996807061, + 0.004476833011722192 + ] + }, + { + "forward": 0.010217875009402633, + "prefill": 0.007467165996786207, + "decode_steps": [ + 0.005021749995648861, + 0.004615958008798771, + 0.0044577079970622435, + 0.004720875003840774, + 0.004583667003316805, + 0.004599749998305924, + 0.004440667005837895, + 0.004521000009845011, + 0.004553082995698787, + 0.004419749995577149, + 0.00448908400721848, + 0.004438916992512532, + 0.004210457991575822, + 0.00448662499547936, + 0.00516758298908826, + 0.004997916999855079, + 0.004430832996149547, + 0.004613208002410829, + 0.0045259589969646186, + 0.004515416992944665, + 0.00459729200520087, + 0.004611083000781946, + 0.00453600000764709, + 0.004430625005625188, + 0.004365041997516528, + 0.004546708005364053, + 0.004416374998982064, + 0.0044102500105509534, + 0.004454666006495245, + 0.004461083997739479, + 0.004459083007532172, + 0.004469583000172861 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 10.161396006878931, + "p95": 10.558951902930858, + "min": 9.38112499716226, + "max": 10.671208001440391, + "iqr": 0.5392600032791961, + "mean": 10.061887501797173, + "cv": 0.041228416285463754 + }, + "prefill_ms": { + "count": 10, + "median": 6.989915993472096, + "p95": 7.4552225509251, + "min": 6.867542004329152, + "max": 7.467165996786207, + "iqr": 0.11948999963351525, + "mean": 7.076741698256228, + "cv": 0.02936974364393234 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.493478998483624, + "p95": 5.034872592659667, + "min": 4.203917007544078, + "max": 17.089207991375588, + "iqr": 0.17483275951235555, + "mean": 4.60122109088843, + "cv": 0.16562535484922028 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1758.109087397057, + "p95": 1785.6864275904704, + "min": 1590.8241857716566, + "max": 1789.7795542411277, + "iqr": 44.90293511026175, + "mean": 1740.5120153852313, + "cv": 0.03325228723818139 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1672.4932834471833, + "p95": 1702.9372157721918, + "min": 1523.8371161458465, + "max": 1706.4841748394902, + "iqr": 39.57640443083665, + "mean": 1660.5356139128714, + "cv": 0.03176421350806092 + }, + "memory": { + "peak_extra_bytes": 9831424, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "auto", + "batch": 8, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.009796750004170462 + }, + { + "forward": 0.009985083001083694 + }, + { + "forward": 0.009807334005017765 + }, + { + "forward": 0.00992004199360963 + }, + { + "forward": 0.00983033400552813 + }, + { + "forward": 0.014049167002667673 + }, + { + "forward": 0.009756333005498163 + }, + { + "forward": 0.00977558401064016 + }, + { + "forward": 0.009844792002695613 + }, + { + "forward": 0.009874709008727223 + } + ], + "forward_ms": { + "count": 10, + "median": 9.837563004111871, + "p95": 12.220329201954879, + "min": 9.756333005498163, + "max": 14.049167002667673, + "iqr": 0.10931274300673977, + "mean": 10.264012803963851, + "cv": 0.1297496916294092 + }, + "memory": { + "peak_extra_bytes": 11798528, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "sdpa", + "batch": 8, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.008553584004403092, + "prefill": 0.008919458006857894, + "decode_steps": [ + 0.0026293749979231507, + 0.0022569580032723024, + 0.0023347079986706376, + 0.0023575840023113415, + 0.002288542003952898, + 0.0025897919986164197, + 0.0024505000037606806, + 0.002329249997274019, + 0.0025268329918617383, + 0.002398208001977764, + 0.0023527919984189793, + 0.0023302920017158613, + 0.002311207994353026, + 0.0024305419938173145, + 0.0022683750139549375, + 0.0022865000064484775, + 0.0023903330002212897, + 0.002263999995193444, + 0.0023337079910561442, + 0.002338207996217534, + 0.0024152499972842634, + 0.0023252079990925267, + 0.0023175000096671283, + 0.002283333000377752, + 0.0023971670016180724, + 0.002317082995432429, + 0.0023192500084405765, + 0.002271167002618313, + 0.0023981250124052167, + 0.002351207993342541, + 0.0024244170053862035, + 0.0022859999880893156 + ] + }, + { + "forward": 0.008770792002906092, + "prefill": 0.008660999999847263, + "decode_steps": [ + 0.002370417001657188, + 0.0024505839974153787, + 0.0023492910113418475, + 0.002311375006684102, + 0.00239516599685885, + 0.002346999986912124, + 0.0024045419995673, + 0.002397292002569884, + 0.002413458001683466, + 0.0022392499959096313, + 0.002306624999619089, + 0.0023554999934276566, + 0.002242375005153008, + 0.002335375000257045, + 0.0023261669994099066, + 0.0023401660000672564, + 0.005548125001951121, + 0.00510404200758785, + 0.0037926669901935384, + 0.005209290990023874, + 0.0025967920082621276, + 0.002735166999627836, + 0.0024560000019846484, + 0.002460667004925199, + 0.0023219580034492537, + 0.0024089999933494255, + 0.002389499990385957, + 0.002308875002199784, + 0.0023699170124018565, + 0.002378459001192823, + 0.0023112080089049414, + 0.0023467090068152174 + ] + }, + { + "forward": 0.00899924999976065, + "prefill": 0.008717208009329624, + "decode_steps": [ + 0.0024318329960806295, + 0.0023050000017974526, + 0.002252167003462091, + 0.002358541009016335, + 0.002373208000790328, + 0.0023592499928781763, + 0.0023398330085910857, + 0.0023036250058794394, + 0.002358750003622845, + 0.002214041000115685, + 0.0023292080004466698, + 0.002281084001879208, + 0.002530708006815985, + 0.002258624997921288, + 0.002254207996884361, + 0.0022847500076750293, + 0.0023160420096246526, + 0.00232566699560266, + 0.0023822909861337394, + 0.002263916001538746, + 0.0023120000114431605, + 0.002274416998261586, + 0.002439374991809018, + 0.002358832993195392, + 0.0023594579979544505, + 0.0024572090042056516, + 0.002320083003723994, + 0.00244770800054539, + 0.0024339999945368618, + 0.0022667080047540367, + 0.0024145000061253086, + 0.0024101669987430796 + ] + }, + { + "forward": 0.008712958006071858, + "prefill": 0.008700291000423022, + "decode_steps": [ + 0.0023718750016996637, + 0.002333084004931152, + 0.0022250420006457716, + 0.0022734170051990077, + 0.0022745000023860484, + 0.00231591600459069, + 0.0023643750027986243, + 0.002377291995799169, + 0.002162250006222166, + 0.00257670899736695, + 0.0025792499945964664, + 0.0023236670094775036, + 0.0023438329953933135, + 0.0024647499958518893, + 0.0023787910031387582, + 0.002279749998706393, + 0.0022802500025136396, + 0.002382666993071325, + 0.0023171670036390424, + 0.0022692079946864396, + 0.002398832992184907, + 0.0023563750000903383, + 0.0022742909932276234, + 0.002302249995409511, + 0.002371333001065068, + 0.002283125009853393, + 0.0024122919858200476, + 0.0023810839920770377, + 0.0023494170018238947, + 0.0022508749971166253, + 0.0022808339999755844, + 0.0023449160071322694 + ] + }, + { + "forward": 0.008821875002468005, + "prefill": 0.008991374998004176, + "decode_steps": [ + 0.0023378750047413632, + 0.0025291659985668957, + 0.002349499991396442, + 0.002336292003747076, + 0.0023418750060955063, + 0.0023496249923482537, + 0.0023125830048229545, + 0.002247374999569729, + 0.002840667002601549, + 0.002431166998576373, + 0.002637000012327917, + 0.0023640839935978875, + 0.002390458990703337, + 0.0023352080024778843, + 0.0023979580000741407, + 0.002328125003259629, + 0.0024257500044768676, + 0.002434583002468571, + 0.0024044169986154884, + 0.0022762919979868457, + 0.00264770801004488, + 0.0023206249898066744, + 0.0022918330068932846, + 0.0023007089912425727, + 0.0024078749993350357, + 0.002263834001496434, + 0.0022723750007571653, + 0.0023197910049930215, + 0.002431000000797212, + 0.002347707995795645, + 0.0023939160018926486, + 0.0023244159965543076 + ] + }, + { + "forward": 0.009156542000710033, + "prefill": 0.00896337500307709, + "decode_steps": [ + 0.0028636250062845647, + 0.0028737499960698187, + 0.0025410419912077487, + 0.0023268749937415123, + 0.0023573749931529164, + 0.0023671249946346506, + 0.0023267079959623516, + 0.0023315419966820627, + 0.0027916249964619055, + 0.002767125013633631, + 0.002476041001500562, + 0.0023082080006133765, + 0.0023735410068184137, + 0.0023257499997271225, + 0.0023862500092945993, + 0.002377457989496179, + 0.0023257080028997734, + 0.0023567500029457733, + 0.00234491701121442, + 0.0023105000000214204, + 0.002335333003429696, + 0.002399374992819503, + 0.0024591249966761097, + 0.002361457998631522, + 0.0022012080007698387, + 0.00253033300396055, + 0.0023159999982453883, + 0.0024122090107994154, + 0.0022615000052610412, + 0.0023906660062493756, + 0.0023834159947000444, + 0.002391916001215577 + ] + }, + { + "forward": 0.008743124999455176, + "prefill": 0.008689666996360756, + "decode_steps": [ + 0.0024337920040125027, + 0.0022601660020882264, + 0.002357916993787512, + 0.002372875009314157, + 0.0023948750022100285, + 0.002192166997701861, + 0.0024331669992534444, + 0.002318999991985038, + 0.0022731670032953843, + 0.0023127500026021153, + 0.0023685829946771264, + 0.0023019590007606894, + 0.0022017500014044344, + 0.002369458001339808, + 0.0023794580047251657, + 0.002341250001336448, + 0.0025725839950609952, + 0.0023066670109983534, + 0.0022507079993374646, + 0.0024720000074012205, + 0.0023790420091245323, + 0.002274415994179435, + 0.0023775410081725568, + 0.00240566699358169, + 0.0023056659993017092, + 0.0023352499993052334, + 0.0023107080050976947, + 0.0023904999980004504, + 0.0023565410083392635, + 0.002355333010200411, + 0.0024257500044768676, + 0.002300332998856902 + ] + }, + { + "forward": 0.00870654100435786, + "prefill": 0.008715458985534497, + "decode_steps": [ + 0.002421208002488129, + 0.0023570000048493966, + 0.002377208002144471, + 0.002303542001754977, + 0.0023722920013824478, + 0.0023470420128433034, + 0.002399916003923863, + 0.0022956659959163517, + 0.002383333005127497, + 0.0023668329959036782, + 0.0022400829911930487, + 0.0023099160025594756, + 0.0024382920091738924, + 0.002308708004420623, + 0.0023897499922895804, + 0.0023851659934734926, + 0.0023413330054609105, + 0.002345583008718677, + 0.002478291993611492, + 0.002467332989908755, + 0.0023275830026250333, + 0.0023819170019123703, + 0.0022846249921713024, + 0.002363374995184131, + 0.0023537920060334727, + 0.002421416007564403, + 0.0023071250034263358, + 0.0023273329861694947, + 0.002396416006376967, + 0.0023570830089738593, + 0.0023984580038813874, + 0.002427542000077665 + ] + }, + { + "forward": 0.008753500005695969, + "prefill": 0.008735334005905315, + "decode_steps": [ + 0.0023588750045746565, + 0.002432125009363517, + 0.0023022500099614263, + 0.0021899580024182796, + 0.0024043750017881393, + 0.0022915830049896613, + 0.00255408399971202, + 0.002312500000698492, + 0.0022356249974109232, + 0.0023548330063931644, + 0.002380125006311573, + 0.0022246250009629875, + 0.002387166998232715, + 0.002613042001030408, + 0.002309666990186088, + 0.0022841249883640558, + 0.00219049998850096, + 0.0023419579956680536, + 0.002317416001460515, + 0.002385415995377116, + 0.0024421249981969595, + 0.002355750009883195, + 0.002332874995772727, + 0.0023741670011077076, + 0.0023226670018630102, + 0.0022792920062784106, + 0.002452542001265101, + 0.002395125004113652, + 0.002322915999684483, + 0.0023533330095233396, + 0.0023753339919494465, + 0.0022747500042896718 + ] + }, + { + "forward": 0.008785375001025386, + "prefill": 0.008695750002516434, + "decode_steps": [ + 0.0023433329915860668, + 0.002322792002814822, + 0.0023958749952726066, + 0.0023184999881777912, + 0.0023465419944841415, + 0.0022680420079268515, + 0.002238417000626214, + 0.0055675000039627776, + 0.0030410830077016726, + 0.005021499993745238, + 0.003014542002347298, + 0.004703708997112699, + 0.0024591249966761097, + 0.0023273339902516454, + 0.002281624998431653, + 0.002521292000892572, + 0.0023412090085912496, + 0.00231779100431595, + 0.0025150830042548478, + 0.002396500000031665, + 0.002369334004470147, + 0.0023739999887766317, + 0.002290291988174431, + 0.0023686670028837398, + 0.002421790995867923, + 0.0023096660006558523, + 0.002359500009333715, + 0.002311375006684102, + 0.002496125001925975, + 0.002656917000422254, + 0.0023342920030700043, + 0.002409249995253049 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 8.76214600430103, + "p95": 9.08576060028281, + "min": 8.553584004403092, + "max": 9.156542000710033, + "iqr": 0.0922502476896625, + "mean": 8.800354202685412, + "cv": 0.018968749300316484 + }, + "prefill_ms": { + "count": 10, + "median": 8.71633349743206, + "p95": 8.978775000286987, + "min": 8.660999999847263, + "max": 8.991374998004176, + "iqr": 0.176541754626669, + "mean": 8.778891700785607, + "cv": 0.014389431698755527 + }, + "decode_ms_per_step": { + "count": 320, + "median": 2.355416501814034, + "p95": 2.648168459563749, + "min": 2.162250006222166, + "max": 5.5675000039627776, + "iqr": 0.08969749615062028, + "mean": 2.4286861661494186, + "cv": 0.1683556111058717 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 3385.1573464431685, + "p95": 3415.502115573313, + "min": 2965.6072953867874, + "max": 3417.9170796672834, + "iqr": 94.32434860587045, + "mean": 3302.7589054761943, + "cv": 0.052504231094567776 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 3031.6769003459776, + "p95": 3059.9948272905367, + "min": 2695.1919033664803, + "max": 3062.211616477633, + "iqr": 85.29429051050556, + "mean": 2965.9807796039518, + "cv": 0.0472283102935585 + }, + "memory": { + "peak_extra_bytes": 12584960, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "reference", + "batch": 8, + "seq_len": 256, + "samples_seconds": [ + { + "forward": 0.018578125003841706, + "prefill": 0.015072333000716753, + "decode_steps": [ + 0.005240790997049771, + 0.005303791011101566, + 0.00493879200075753, + 0.004713625006843358, + 0.004891374992439523, + 0.0046426670014625415, + 0.004827291006222367, + 0.00473858299665153, + 0.004666915992856957, + 0.0048677919985493645, + 0.004720208991784602, + 0.0047765420022187755, + 0.004699249999248423, + 0.004672458002460189, + 0.004671791990404017, + 0.004625291010597721, + 0.004744125006254762, + 0.00472645899571944, + 0.004688875007559545, + 0.004698125005234033, + 0.005303582991473377, + 0.004960834005032666, + 0.005125042007421143, + 0.004757333008456044, + 0.005329916995833628, + 0.005037249997258186, + 0.004913666009088047, + 0.004776207992108539, + 0.004709209009888582, + 0.0048744590021669865, + 0.0048030420002760366, + 0.004888374998699874 + ] + }, + { + "forward": 0.018469000002369285, + "prefill": 0.015035958000225946, + "decode_steps": [ + 0.005099749992950819, + 0.0050330410012975335, + 0.0051720410119742155, + 0.004926000008708797, + 0.004940583006828092, + 0.004944041997077875, + 0.005258749995846301, + 0.004960874997777864, + 0.00480737499310635, + 0.004819209003471769, + 0.0044280000001890585, + 0.004701833997387439, + 0.004835499988985248, + 0.004817749999347143, + 0.004866208997555077, + 0.004558666987577453, + 0.0048557910049567, + 0.004742374992929399, + 0.004723125006421469, + 0.00473991601029411, + 0.004818916990188882, + 0.004771332998643629, + 0.004772125001181848, + 0.004714584007160738, + 0.004693167007644661, + 0.004711707995738834, + 0.00485729199135676, + 0.004749750005430542, + 0.004786042001796886, + 0.004838374996325001, + 0.0053507919947151095, + 0.004958458011969924 + ] + }, + { + "forward": 0.017793209000956267, + "prefill": 0.01565845801087562, + "decode_steps": [ + 0.005504375003511086, + 0.004958749996148981, + 0.004599916996085085, + 0.0047982919932110235, + 0.004708250009571202, + 0.004551457997877151, + 0.004892291995929554, + 0.004426458006491885, + 0.004761582997161895, + 0.005205458001000807, + 0.004903207998722792, + 0.00499916699482128, + 0.004694916991866194, + 0.004552124999463558, + 0.005188958995859139, + 0.004897207996691577, + 0.005392125007347204, + 0.004907541995635256, + 0.005137334010214545, + 0.004774750006617978, + 0.00481487499200739, + 0.004901708991383202, + 0.0045887080050306395, + 0.004793500003870577, + 0.004826042000786401, + 0.00482741701125633, + 0.004924875000142492, + 0.004805541000678204, + 0.004740415999549441, + 0.004725999999209307, + 0.004632124997442588, + 0.004862292000325397 + ] + }, + { + "forward": 0.018671209007152356, + "prefill": 0.015165667005931027, + "decode_steps": [ + 0.005642042000545189, + 0.004957249999279156, + 0.004833582992432639, + 0.004728333005914465, + 0.004751208995003253, + 0.004661374987335876, + 0.004785041004652157, + 0.004697625001426786, + 0.004693750001024455, + 0.004693416005466133, + 0.00469616700138431, + 0.004694375005783513, + 0.004697208001744002, + 0.004713709000498056, + 0.004703625003458001, + 0.0047188329917844385, + 0.005023666002671234, + 0.005216459001530893, + 0.005353125001420267, + 0.004932291994919069, + 0.005363499993109144, + 0.005048582999734208, + 0.00493037499836646, + 0.004913833006867208, + 0.004867833005846478, + 0.004698000004282221, + 0.004948834000970237, + 0.004898375002085231, + 0.004881000000750646, + 0.0048357500054407865, + 0.00471375000779517, + 0.004845540999667719 + ] + }, + { + "forward": 0.018458124992321245, + "prefill": 0.015133167005842552, + "decode_steps": [ + 0.005320708005456254, + 0.004953791998559609, + 0.0052024169999640435, + 0.0048366660048486665, + 0.0048498329997528344, + 0.00474999999278225, + 0.004732457993668504, + 0.0050472910079406574, + 0.004749291008920409, + 0.004928125010337681, + 0.0047845420049270615, + 0.004648208006983623, + 0.004772625004989095, + 0.004549417004454881, + 0.004768332990352064, + 0.004775708002853207, + 0.004798499998287298, + 0.004884167006821372, + 0.005247375011094846, + 0.004840458001126535, + 0.005005749990232289, + 0.0048837500071385875, + 0.004967708999174647, + 0.004887167000561021, + 0.004981959005817771, + 0.005361417002859525, + 0.005037750001065433, + 0.004963332990882918, + 0.004476083006011322, + 0.0048174999974435195, + 0.004902125001535751, + 0.004850292010814883 + ] + }, + { + "forward": 0.01746070799708832, + "prefill": 0.015335584001149982, + "decode_steps": [ + 0.004830500009120442, + 0.004793083993718028, + 0.004704958002548665, + 0.004784750010003336, + 0.004708375010523014, + 0.005075500012026168, + 0.004760250012623146, + 0.005009542001062073, + 0.004793375002918765, + 0.004755499991006218, + 0.004687500011641532, + 0.004651666997233406, + 0.004727082996396348, + 0.004811708000488579, + 0.004696415999205783, + 0.004601791995810345, + 0.004682374987169169, + 0.004741166005260311, + 0.005035375012084842, + 0.004770083993207663, + 0.004790042003151029, + 0.004706916006398387, + 0.004900792002445087, + 0.004776375004439615, + 0.004818000001250766, + 0.004707084008259699, + 0.004808249999769032, + 0.004771167004946619, + 0.00798479099466931, + 0.015527791998465545, + 0.007381917006568983, + 0.006587750001926906 + ] + }, + { + "forward": 0.018537583004217595, + "prefill": 0.015399916999740526, + "decode_steps": [ + 0.005213083000853658, + 0.005122749993461184, + 0.004957208002451807, + 0.004827042008400895, + 0.004751625005155802, + 0.004788459002156742, + 0.00457858300069347, + 0.004936083001666702, + 0.005701874993974343, + 0.004822957998840138, + 0.004895541002042592, + 0.0049079579912358895, + 0.004817207998712547, + 0.0046685420093126595, + 0.00438029199722223, + 0.00472812500083819, + 0.004732832996523939, + 0.004817207998712547, + 0.004996584000764415, + 0.004858332991716452, + 0.004722291007055901, + 0.004908208997221664, + 0.004745958998682909, + 0.004893208999419585, + 0.004659708996769041, + 0.004700124991359189, + 0.004734542002552189, + 0.004989166001905687, + 0.004894332989351824, + 0.0046079160092631355, + 0.004858332991716452, + 0.004749208004795946 + ] + }, + { + "forward": 0.01843574999656994, + "prefill": 0.015699957992183045, + "decode_steps": [ + 0.005564750012126751, + 0.004932499999995343, + 0.00483541599533055, + 0.00480812499881722, + 0.004702249992988072, + 0.004847791002248414, + 0.004612958000507206, + 0.004701708996435627, + 0.004733958005090244, + 0.0047081670054467395, + 0.00469708300079219, + 0.004604667003150098, + 0.00482699999702163, + 0.004698749995441176, + 0.005157250008778647, + 0.005149749995325692, + 0.0050737089914036915, + 0.004741999990073964, + 0.004535083993687294, + 0.0050538749928819016, + 0.004931707997457124, + 0.004705583996837959, + 0.004837083994061686, + 0.004825416996027343, + 0.004938333004247397, + 0.004877582992776297, + 0.004833415994653478, + 0.004900625004665926, + 0.00481595798919443, + 0.00466829098877497, + 0.00506983399100136, + 0.004830000005313195 + ] + }, + { + "forward": 0.01770770900475327, + "prefill": 0.01556704199174419, + "decode_steps": [ + 0.005403957999078557, + 0.004956541990395635, + 0.0047566660068696365, + 0.004754875000799075, + 0.004830833000596613, + 0.004638665996026248, + 0.004772790998686105, + 0.004666000008000992, + 0.004679624995333143, + 0.004699292010627687, + 0.004718833006336354, + 0.004980750003596768, + 0.004912874996080063, + 0.004776708010467701, + 0.004614541001501493, + 0.004633541000657715, + 0.004718166994280182, + 0.004728542000520974, + 0.005052874999819323, + 0.004824458999792114, + 0.00509524998778943, + 0.004991541994968429, + 0.0052612500003306195, + 0.004842583002755418, + 0.004970416994183324, + 0.004850333003560081, + 0.004799125003046356, + 0.0048580409929854795, + 0.004753791989060119, + 0.004847332995268516, + 0.004821541995625012, + 0.004713875008746982 + ] + }, + { + "forward": 0.01901874999748543, + "prefill": 0.0151378750015283, + "decode_steps": [ + 0.005207082998822443, + 0.00523508399783168, + 0.004936666999128647, + 0.005264583000098355, + 0.00496608299727086, + 0.00524816699908115, + 0.005072833999292925, + 0.004986084008123726, + 0.004724124999484047, + 0.004874292004387826, + 0.004838167005800642, + 0.004885083006229252, + 0.004810665996046737, + 0.0047890840069158, + 0.004877208004472777, + 0.004724375001387671, + 0.004732542001875117, + 0.004775541994604282, + 0.0047595829964848235, + 0.0047913750022416934, + 0.004942582992953248, + 0.004860417000600137, + 0.005258332996163517, + 0.005274249997455627, + 0.005026625003665686, + 0.005456582992337644, + 0.005217374986386858, + 0.004782292002346367, + 0.004820875008590519, + 0.004852209007367492, + 0.004757374990731478, + 0.004463125005713664 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 18.463562497345265, + "p95": 18.862356551835546, + "min": 17.46070799708832, + "max": 19.01874999748543, + "iqr": 0.6141452540759929, + "mean": 18.31301680067554, + "cv": 0.02681661386513084 + }, + "prefill_ms": { + "count": 10, + "median": 15.250625503540505, + "p95": 15.681283000594703, + "min": 15.035958000225946, + "max": 15.699957992183045, + "iqr": 0.3909167389792856, + "mean": 15.320595900993794, + "cv": 0.0162887404098647 + }, + "decode_ms_per_step": { + "count": 320, + "median": 4.822249997232575, + "p95": 5.321168454975123, + "min": 4.38029199722223, + "max": 15.527791998465545, + "iqr": 0.21660349739249796, + "mean": 4.9165687471941055, + "cv": 0.13713514166965005 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1648.2561452744412, + "p95": 1652.2154814229295, + "min": 1493.7402125646568, + "max": 1652.4058236163303, + "iqr": 13.03440322273218, + "mean": 1628.5491923234756, + "cv": 0.02979144088513785 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1497.6159304159498, + "p95": 1503.0224683048557, + "min": 1371.0554968654703, + "max": 1503.3237490048411, + "iqr": 10.321143786364928, + "mean": 1483.8206312657335, + "cv": 0.027235424309534344 + }, + "memory": { + "peak_extra_bytes": 35391488, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "auto", + "batch": 8, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.02374308300204575 + }, + { + "forward": 0.022597875009523705 + }, + { + "forward": 0.022726666007656604 + }, + { + "forward": 0.023229124999488704 + }, + { + "forward": 0.02970524999545887 + }, + { + "forward": 0.02105570799903944 + }, + { + "forward": 0.021070083006634377 + }, + { + "forward": 0.020202917003189214 + }, + { + "forward": 0.02029479200427886 + }, + { + "forward": 0.020954624997102655 + } + ], + "forward_ms": { + "count": 10, + "median": 21.83397900807904, + "p95": 27.02227484842296, + "min": 20.202917003189214, + "max": 29.70524999545887, + "iqr": 2.123614503943827, + "mean": 22.55801240244182, + "cv": 0.12422830330560924 + }, + "memory": { + "peak_extra_bytes": 23597056, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "sdpa", + "batch": 8, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.023457207993487827, + "prefill": 0.02158583300479222, + "decode_steps": [ + 0.00523320899810642, + 0.005000999997719191, + 0.004327832997660153, + 0.003369666010257788, + 0.00656162500672508, + 0.003987500007497147, + 0.0036023330030729994, + 0.003410416000406258, + 0.0047512079909211025, + 0.004014667007140815, + 0.0035002909862669185, + 0.003437708001001738, + 0.005600083997705951, + 0.004098582998267375, + 0.004130624991375953, + 0.0034117500035790727, + 0.004644915999961086, + 0.00328266600263305, + 0.003325291007058695, + 0.0032092920009745285, + 0.005935958004556596, + 0.0033928329939953983, + 0.003280333010479808, + 0.0035454999888315797, + 0.005837042001076043, + 0.0034303330030525103, + 0.003312540997285396, + 0.0032445000106235966, + 0.0055039169965311885, + 0.0034388339990982786, + 0.0033425420115236193, + 0.003274959002737887 + ] + }, + { + "forward": 0.019535874991561286, + "prefill": 0.019626082998001948, + "decode_steps": [ + 0.003977457992732525, + 0.0025737499963724986, + 0.0025756250106496736, + 0.0025169580039801076, + 0.002460167001117952, + 0.004603749999660067, + 0.002394750001258217, + 0.0028185830014990643, + 0.0026129999896511436, + 0.002481834002537653, + 0.0047733340034028515, + 0.0024160000029951334, + 0.002616874990053475, + 0.002602332999231294, + 0.002581375010777265, + 0.0025255420041503385, + 0.003971832993556745, + 0.0026071659958688542, + 0.0026236249977955595, + 0.0028064159996574745, + 0.0028067910025129095, + 0.004495125001994893, + 0.002635500000906177, + 0.0028775840037269518, + 0.0029774579888908193, + 0.0029072079923935235, + 0.004236417007632554, + 0.002662957995198667, + 0.0029540420073317364, + 0.0029752499976893887, + 0.002896874997531995, + 0.005376041997806169 + ] + }, + { + "forward": 0.021880000000237487, + "prefill": 0.019771125007537194, + "decode_steps": [ + 0.002757040987489745, + 0.0025707079912535846, + 0.002469041006406769, + 0.0041129580058623105, + 0.0024882919969968498, + 0.00248454199754633, + 0.002495708002243191, + 0.0023682080063736066, + 0.004822875009267591, + 0.0024954999971669167, + 0.0025329159980174154, + 0.0025290000048698857, + 0.0024876249954104424, + 0.0025150829897029325, + 0.003810333990259096, + 0.0025725410087034106, + 0.002795290987705812, + 0.0026464999973541126, + 0.002626208995934576, + 0.0024622080090921372, + 0.0027590000099735335, + 0.00266025000018999, + 0.0026515829958952963, + 0.0025777079863473773, + 0.002657958000781946, + 0.002897416998166591, + 0.0030958340066717938, + 0.002914041993790306, + 0.00282116599555593, + 0.004265832991222851, + 0.0031131669966271147, + 0.003466499998467043 + ] + }, + { + "forward": 0.0228543340053875, + "prefill": 0.02130300000135321, + "decode_steps": [ + 0.005483415996422991, + 0.005555249997996725, + 0.004948250003508292, + 0.004274916005670093, + 0.0066256670106668025, + 0.004344250002759509, + 0.004186500009382144, + 0.0035618330002762377, + 0.003930624996428378, + 0.004661459010094404, + 0.00396191599429585, + 0.006531124992761761, + 0.004074708005646244, + 0.0036418329982552677, + 0.005879917007405311, + 0.003613292006775737, + 0.0033447079913457856, + 0.0032132920023286715, + 0.005431792000308633, + 0.0031029999954625964, + 0.0033484579907963052, + 0.0030218329920899123, + 0.003063457988901064, + 0.00468029199691955, + 0.003207499990821816, + 0.0031621660018572584, + 0.002992583002196625, + 0.0031884999916655943, + 0.0030763749964535236, + 0.003112082995357923, + 0.0030230420088628307, + 0.0030889170011505485 + ] + }, + { + "forward": 0.01965274999383837, + "prefill": 0.019517458000336774, + "decode_steps": [ + 0.002711499997531064, + 0.0025887920055538416, + 0.004444332997081801, + 0.0025725840096129104, + 0.002468625010806136, + 0.0024427090102108195, + 0.0026127919991267845, + 0.004578124993713573, + 0.0025769589992705733, + 0.0025789169885683805, + 0.002540833011153154, + 0.0024815420038066804, + 0.00477658401359804, + 0.002485250006429851, + 0.002562958005000837, + 0.002551290992414579, + 0.002492334009730257, + 0.0024734170001465827, + 0.004033250006614253, + 0.0026194999954896048, + 0.002589458992588334, + 0.002704832993913442, + 0.0026287499931640923, + 0.004857749998336658, + 0.00280883300001733, + 0.002810000005410984, + 0.0028283749998081475, + 0.003048999991733581, + 0.0033434169890824705, + 0.0034157500049332157, + 0.0031412499956786633, + 0.0032184169976972044 + ] + }, + { + "forward": 0.018299792005564086, + "prefill": 0.01803104100690689, + "decode_steps": [ + 0.0027017499960493296, + 0.0026171249919570982, + 0.002522333001252264, + 0.0026344580110162497, + 0.0026289579982403666, + 0.002499374997569248, + 0.002680833000340499, + 0.0024634589935885742, + 0.002547750002122484, + 0.0026015000039478764, + 0.002501458002370782, + 0.002467416998115368, + 0.0025055419973796234, + 0.002573750010924414, + 0.0025836669956333935, + 0.0026370419946033508, + 0.0025845839991234243, + 0.0024512499949196354, + 0.0028060829936293885, + 0.0025788329949136823, + 0.0025755410024430603, + 0.002523208997445181, + 0.002559708009357564, + 0.002617457997985184, + 0.002695332994335331, + 0.0026765840011648834, + 0.0026188750052824616, + 0.0027072499942732975, + 0.002850000004400499, + 0.0027958339924225584, + 0.0028226249996805564, + 0.0028237920050742105 + ] + }, + { + "forward": 0.018294042005436495, + "prefill": 0.017775208005332388, + "decode_steps": [ + 0.0027592499973252416, + 0.002741375006735325, + 0.00267554099264089, + 0.0027122089959448203, + 0.0025056670128833503, + 0.002540874993428588, + 0.0024782909895293415, + 0.002514666994102299, + 0.002547500000218861, + 0.0027798339870059863, + 0.002715082999202423, + 0.0026189169875578955, + 0.0029675830010091886, + 0.0027357500075595453, + 0.002770500010228716, + 0.0028932089917361736, + 0.002929209003923461, + 0.0029243329918244854, + 0.0029815000016242266, + 0.002940499995020218, + 0.0032315830030711368, + 0.0033494999952381477, + 0.003274374990724027, + 0.0033110419899458066, + 0.003529166991938837, + 0.0035029170103371143, + 0.0035117090010317042, + 0.0035319579910719767, + 0.004410958994412795, + 0.004715167000540532, + 0.0046904579940019175, + 0.0062716249958612025 + ] + }, + { + "forward": 0.01791791699361056, + "prefill": 0.017550374992424622, + "decode_steps": [ + 0.0028732909995596856, + 0.0027966249908786267, + 0.003039833012735471, + 0.002635374999954365, + 0.0026799999905051664, + 0.002570208001998253, + 0.00250679200689774, + 0.0024548749934183434, + 0.0025150830042548478, + 0.002627209003549069, + 0.0026752080011647195, + 0.0026832919975277036, + 0.0026638330018613487, + 0.0032161249982891604, + 0.0026749580138130113, + 0.002938959005405195, + 0.002821792004397139, + 0.0027915829996345565, + 0.0028106249956181273, + 0.0028532089927466586, + 0.0029990000039106235, + 0.003059458002098836, + 0.003381500006071292, + 0.0029298750014277175, + 0.003541083002346568, + 0.0032407079997938126, + 0.0029458329954650253, + 0.0030239580082707107, + 0.003032624998013489, + 0.00306175000150688, + 0.003005791993928142, + 0.0031435839919140562 + ] + }, + { + "forward": 0.017488709010649472, + "prefill": 0.0173082080000313, + "decode_steps": [ + 0.0026253329997416586, + 0.002507375000277534, + 0.0025382919993717223, + 0.0024061250005615875, + 0.0025674579956103116, + 0.002511875005438924, + 0.0024356670037377626, + 0.0024276670010294765, + 0.0025151660083793104, + 0.002500707996659912, + 0.002576541999587789, + 0.0025293749931734055, + 0.0024962500028777868, + 0.0024597089941380545, + 0.0027224999939789996, + 0.002614291006466374, + 0.0025592500023776665, + 0.002511124999728054, + 0.0025379159924341366, + 0.002537082997150719, + 0.002617292004288174, + 0.0026082499971380457, + 0.002873958001146093, + 0.0026829999987967312, + 0.003015166992554441, + 0.002931124996393919, + 0.0028589169960469007, + 0.0028454580024117604, + 0.0028113750013289973, + 0.0032030419970396906, + 0.0030692080035805702, + 0.0031220419914461672 + ] + }, + { + "forward": 0.01839362499595154, + "prefill": 0.018438374987454154, + "decode_steps": [ + 0.002875334001146257, + 0.0029720419988734648, + 0.002831125006196089, + 0.0030331250018207356, + 0.0031755829986650497, + 0.0036572919925674796, + 0.0031914169958326966, + 0.003153375000692904, + 0.003716333012562245, + 0.004268791002687067, + 0.004329000003053807, + 0.0042427090083947405, + 0.004207833000691608, + 0.004112166992854327, + 0.0035515839990694076, + 0.0034697910014074296, + 0.003573709007469006, + 0.0035465830005705357, + 0.0035782499908236787, + 0.003308250001282431, + 0.003390458005014807, + 0.0034321669954806566, + 0.0036115840048296377, + 0.0034802500013029203, + 0.0033249169937334955, + 0.0034070839901687577, + 0.0032494580082129687, + 0.003385917007108219, + 0.003358874993864447, + 0.003468209004495293, + 0.0033877079986268654, + 0.0034048750094370916 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 18.964749993756413, + "p95": 23.18591469884268, + "min": 17.488709010649472, + "max": 23.457207993487827, + "iqr": 3.027707993169315, + "mean": 19.777425199572463, + "cv": 0.1098730561546262 + }, + "prefill_ms": { + "count": 10, + "median": 18.977916493895464, + "p95": 21.458558153244667, + "min": 17.3082080000313, + "max": 21.58583300479222, + "iqr": 1.8956982494273689, + "mean": 19.09067060041707, + "cv": 0.07978623915400297 + }, + "decode_ms_per_step": { + "count": 320, + "median": 2.902312495280057, + "p95": 4.8622749985952405, + "min": 2.3682080063736066, + "max": 6.6256670106668025, + "iqr": 0.8134380077535752, + "mean": 3.1907803371723276, + "cv": 0.2607580971063479 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 2635.0834047767494, + "p95": 3030.939920249346, + "min": 1962.5888397830686, + "max": 3052.948041785351, + "iqr": 436.86898709277784, + "mean": 2563.103509051053, + "cv": 0.14938900259044438 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 2193.2142091514716, + "p95": 2505.567361585799, + "min": 1683.9248348087685, + "max": 2512.6511744084873, + "iqr": 313.0557160031317, + "mean": 2150.7468213164393, + "cv": 0.13546875637526484 + }, + "memory": { + "peak_extra_bytes": 23597056, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "reference", + "batch": 8, + "seq_len": 512, + "samples_seconds": [ + { + "forward": 0.05129350000061095, + "prefill": 0.04725508300180081, + "decode_steps": [ + 0.0143638750014361, + 0.007242917010444216, + 0.006318750005448237, + 0.006001917005050927, + 0.005600082993623801, + 0.0057314580044476315, + 0.007903208999778144, + 0.005829874993651174, + 0.005797292003990151, + 0.005335875001037493, + 0.00557525000476744, + 0.005540375001146458, + 0.005258042001514696, + 0.005415082996478304, + 0.005339999988791533, + 0.006223874996067025, + 0.005876666997210123, + 0.0054711249977117404, + 0.005049208004493266, + 0.005506583998794667, + 0.005254707997664809, + 0.004942500003380701, + 0.005155708990059793, + 0.005157959007192403, + 0.005098084002383985, + 0.005289000007905997, + 0.0049443329917266965, + 0.005965792006463744, + 0.006203584009199403, + 0.005636417001369409, + 0.005039791998569854, + 0.005263915998511948 + ] + }, + { + "forward": 0.05228287499630824, + "prefill": 0.04881133399612736, + "decode_steps": [ + 0.00974429199413862, + 0.008789957995759323, + 0.007541583006968722, + 0.007018250005785376, + 0.006482458004029468, + 0.006790542000089772, + 0.005942625008174218, + 0.0053792910039192066, + 0.005009167012758553, + 0.005976917003863491, + 0.005990999998175539, + 0.005828709006891586, + 0.005715916995541193, + 0.007196499995188788, + 0.005607291008345783, + 0.005701625006622635, + 0.00610833300743252, + 0.005762959001003765, + 0.005284875005600043, + 0.00498887499270495, + 0.005499416001839563, + 0.0053082920057931915, + 0.005132333011715673, + 0.005468749994179234, + 0.005410709010902792, + 0.005067999998573214, + 0.0056930839928099886, + 0.0055934589909156784, + 0.005306542007019743, + 0.005459542007884011, + 0.005279083998175338, + 0.005043417011620477 + ] + }, + { + "forward": 0.04991049999080133, + "prefill": 0.04757008299930021, + "decode_steps": [ + 0.0062979170033941045, + 0.005677791996276937, + 0.005310582986567169, + 0.005173375000595115, + 0.005056791997049004, + 0.004906333007966168, + 0.005237125005805865, + 0.005684957999619655, + 0.00534383300691843, + 0.005326875005266629, + 0.006313875011983328, + 0.00551449999329634, + 0.005233624993707053, + 0.005168749994481914, + 0.005248290995950811, + 0.005063709002570249, + 0.005757584003731608, + 0.005253082999843173, + 0.005118207991472445, + 0.00694054200721439, + 0.005550834001041949, + 0.007958625006722286, + 0.005731124998419546, + 0.005378250003559515, + 0.006306499999482185, + 0.005408957993495278, + 0.005120375004480593, + 0.0051949579938082024, + 0.005463875000714324, + 0.005079582988400944, + 0.005722125002648681, + 0.005367584002669901 + ] + }, + { + "forward": 0.05165833300270606, + "prefill": 0.046236249996582046, + "decode_steps": [ + 0.011436791988671757, + 0.00912970800709445, + 0.008136000004014932, + 0.010217290997388773, + 0.007191042008344084, + 0.008945374996983446, + 0.007045082995318808, + 0.007060582996928133, + 0.0055726669961586595, + 0.005627040998660959, + 0.00697379199846182, + 0.005321792006725445, + 0.005093957995995879, + 0.005727790994569659, + 0.005252415998256765, + 0.005140582987223752, + 0.006098500001826324, + 0.005667207995429635, + 0.005659458009176888, + 0.005103499992401339, + 0.0053334590047597885, + 0.005292374989949167, + 0.004903875000309199, + 0.005295124996337108, + 0.005158708998351358, + 0.006038749997969717, + 0.005267083004582673, + 0.005194625002332032, + 0.00574425001104828, + 0.0052358750108396634, + 0.005139250002685003, + 0.005763041001046076 + ] + }, + { + "forward": 0.059760167001513764, + "prefill": 0.06298908300232142, + "decode_steps": [ + 0.015126042009796947, + 0.007835665994207375, + 0.006504958000732586, + 0.00595900000189431, + 0.006803124997531995, + 0.00645004199759569, + 0.006479499992565252, + 0.006330415999400429, + 0.008653374999994412, + 0.0074404159968253225, + 0.007149083001422696, + 0.006555208994541317, + 0.006305624992819503, + 0.006617624996579252, + 0.006838500004960224, + 0.006409624998923391, + 0.008955375000368804, + 0.007021917001111433, + 0.006343125001876615, + 0.005451124990941025, + 0.006467583996709436, + 0.006072374992072582, + 0.0059762499877251685, + 0.006023207999533042, + 0.005882833997020498, + 0.009882832993753254, + 0.008775958005571738, + 0.006203375000040978, + 0.01013645798957441, + 0.006344875000650063, + 0.0098123750067316, + 0.006187875012983568 + ] + }, + { + "forward": 0.05587825000111479, + "prefill": 0.04313799999363255, + "decode_steps": [ + 0.007087625010171905, + 0.006340958003420383, + 0.005837832999532111, + 0.007823167004971765, + 0.006470542008173652, + 0.005266917010885663, + 0.006324709000182338, + 0.005540290992939845, + 0.005227292000199668, + 0.00617712500388734, + 0.005827083994518034, + 0.00601004199415911, + 0.00585079200391192, + 0.00573449999501463, + 0.00589524999668356, + 0.005526624998310581, + 0.005463374996907078, + 0.005199042003368959, + 0.005547624998143874, + 0.005113000006531365, + 0.005660040988004766, + 0.005550750007387251, + 0.005278833996271715, + 0.005230292008491233, + 0.005196666999836452, + 0.005286250001518056, + 0.005438958003651351, + 0.005160166998393834, + 0.005309207990649156, + 0.0054105840099509805, + 0.005250125002930872, + 0.005168916002730839 + ] + }, + { + "forward": 0.05303216700849589, + "prefill": 0.04347841700655408, + "decode_steps": [ + 0.017430958003387786, + 0.014348249998874962, + 0.010058415995445102, + 0.007667208003113046, + 0.006722332997014746, + 0.006103916995925829, + 0.006836624990683049, + 0.00594758300576359, + 0.005925250006839633, + 0.005726290997699834, + 0.005711290999897756, + 0.005697374988812953, + 0.0057757910108193755, + 0.005547624998143874, + 0.005646165998768993, + 0.005585667007835582, + 0.005461292006657459, + 0.00559620899730362, + 0.005451374992844649, + 0.005244500003755093, + 0.005593624999164604, + 0.005990625009872019, + 0.006180874988785945, + 0.005605333994026296, + 0.005090708000352606, + 0.005074749991763383, + 0.005358458001865074, + 0.004901125008473173, + 0.005382207993534394, + 0.005254249990684912, + 0.004980415993486531, + 0.005408832992543466 + ] + }, + { + "forward": 0.047658791998401284, + "prefill": 0.0421714159892872, + "decode_steps": [ + 0.009848832996794954, + 0.008071749994996935, + 0.007483666006010026, + 0.006852833001175895, + 0.006624582994845696, + 0.0062502500077243894, + 0.006179459000122733, + 0.006143667007563636, + 0.006090166003559716, + 0.0058939159935107455, + 0.006116624994319864, + 0.005851624999195337, + 0.00556420799694024, + 0.005636125002638437, + 0.006223042000783607, + 0.005630667001241818, + 0.005571958012296818, + 0.005459082996821962, + 0.005443333007860929, + 0.005153457997948863, + 0.0060933749919058755, + 0.005150416996912099, + 0.0050649170007091016, + 0.006315375008853152, + 0.005279334000078961, + 0.005284667000523768, + 0.005056667010649107, + 0.005056791997049004, + 0.005203000007895753, + 0.0053307080088416114, + 0.005206625006394461, + 0.005084000003989786 + ] + }, + { + "forward": 0.042408583001815714, + "prefill": 0.04029662499669939, + "decode_steps": [ + 0.005908666003961116, + 0.005352375010261312, + 0.005226374996709637, + 0.005137374988407828, + 0.005250250003882684, + 0.005321542004821822, + 0.005184375011594966, + 0.005336125002941117, + 0.005173207988264039, + 0.005113333012559451, + 0.005235290998825803, + 0.005120625006384216, + 0.0051311670104041696, + 0.005380542002967559, + 0.017090000008465722, + 0.007125499992980622, + 0.005522124993149191, + 0.00555920900660567, + 0.005391041006078012, + 0.005340000003343448, + 0.005372917003114708, + 0.00530816700484138, + 0.005266332998871803, + 0.005338125003618188, + 0.005157166000572033, + 0.005175792000954971, + 0.005545291991438717, + 0.005385291995480657, + 0.005236333992797881, + 0.005218416001298465, + 0.005280957993818447, + 0.0051961249992018566 + ] + }, + { + "forward": 0.04784370800189208, + "prefill": 0.04242450000310782, + "decode_steps": [ + 0.010370000003604218, + 0.008545333999791183, + 0.008027666000998579, + 0.00703100000100676, + 0.007108249992597848, + 0.006106707995058969, + 0.00640762499824632, + 0.006138915996416472, + 0.006120791993453167, + 0.006067624999559484, + 0.005943416996160522, + 0.005864375008968636, + 0.005877124989638105, + 0.005836708005517721, + 0.005651125000440516, + 0.005635000008624047, + 0.0055726669961586595, + 0.005576625000685453, + 0.005653542000800371, + 0.005549749999772757, + 0.005412832993897609, + 0.005599666998023167, + 0.005569625005591661, + 0.005508916001417674, + 0.005642625008476898, + 0.005543332998058759, + 0.005985084004350938, + 0.005881874996703118, + 0.005443917005322874, + 0.005459791995235719, + 0.005168208997929469, + 0.005080458999145776 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 51.4759165016585, + "p95": 58.01330435133422, + "min": 42.408583001815714, + "max": 59.760167001513764, + "iqr": 4.484438006329583, + "mean": 51.17268750036601, + "cv": 0.09273090471668605 + }, + "prefill_ms": { + "count": 10, + "median": 44.857333501568064, + "p95": 56.60909594953408, + "min": 40.29662499669939, + "max": 62.98908300232142, + "iqr": 4.8884579991863575, + "mean": 46.43707909854129, + "cv": 0.13867784615707412 + }, + "decode_ms_per_step": { + "count": 320, + "median": 5.597937997663394, + "p95": 8.945874997152714, + "min": 4.901125008473173, + "max": 17.430958003387786, + "iqr": 0.9033534952322952, + "mean": 6.095675006645251, + "cv": 0.26609865038358677 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1342.6452990578482, + "p95": 1423.6652909979437, + "min": 1098.7324925307732, + "max": 1438.9254123075946, + "iqr": 101.13600142683322, + "mean": 1319.6350458757051, + "cv": 0.07473409759422944 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1079.2800752055887, + "p95": 1140.7016664739274, + "min": 864.909185737108, + "max": 1144.5091905991637, + "iqr": 82.8611360898849, + "mean": 1066.5170438263012, + "cv": 0.07711039266842912 + }, + "memory": { + "peak_extra_bytes": 133697536, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "auto", + "batch": 8, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.0474194160051411 + }, + { + "forward": 0.045409958009258844 + }, + { + "forward": 0.045896790994447656 + }, + { + "forward": 0.04699245799565688 + }, + { + "forward": 0.04615879199991468 + }, + { + "forward": 0.04600229200150352 + }, + { + "forward": 0.04717562500445638 + }, + { + "forward": 0.0464767500088783 + }, + { + "forward": 0.046516292000887915 + }, + { + "forward": 0.047376709000673145 + } + ], + "forward_ms": { + "count": 10, + "median": 46.49652100488311, + "p95": 47.40019785313052, + "min": 45.409958009258844, + "max": 47.4194160051411, + "iqr": 1.0884162511501927, + "mean": 46.54250830208184, + "cv": 0.014698248986656509 + }, + "memory": { + "peak_extra_bytes": 47194112, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "sdpa", + "batch": 8, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.03978304199699778, + "prefill": 0.039826832988183014, + "decode_steps": [ + 0.008812874992145225, + 0.006985791987972334, + 0.00657395800226368, + 0.006530124999699183, + 0.006374375006998889, + 0.006107209002948366, + 0.006330958989565261, + 0.005703040995285846, + 0.0058600839984137565, + 0.006808207996073179, + 0.006342499997117557, + 0.005718000000342727, + 0.0055604160006623715, + 0.005043166995164938, + 0.005028333005611785, + 0.0053844999929424375, + 0.005061458999989554, + 0.00505900000280235, + 0.005025874997954816, + 0.005008084001019597, + 0.005141750007169321, + 0.005064750002929941, + 0.005021374992793426, + 0.005066708996309899, + 0.0051117080001858994, + 0.00456720900547225, + 0.004349166993051767, + 0.005033875000663102, + 0.0057160420110449195, + 0.004505999997491017, + 0.0043087500089313835, + 0.0040637089987285435 + ] + }, + { + "forward": 0.03591929101094138, + "prefill": 0.03596237499732524, + "decode_steps": [ + 0.003534791001584381, + 0.0034201250091427937, + 0.003445041991653852, + 0.0032774999999674037, + 0.0032197499967878684, + 0.0033350419980706647, + 0.003272916001151316, + 0.0033263749937759712, + 0.003296208000392653, + 0.0032777090091258287, + 0.0033296250039711595, + 0.003304625002783723, + 0.0032281250023515895, + 0.0034170839935541153, + 0.0033602920011617243, + 0.0033502500009490177, + 0.003416624997043982, + 0.003458040999248624, + 0.003675083993584849, + 0.0035902499948861077, + 0.0035992909979540855, + 0.0036146249913144857, + 0.003868167012115009, + 0.003829458000836894, + 0.0037732500059064478, + 0.003942665993236005, + 0.0044607500021811575, + 0.004310290998546407, + 0.004647124995244667, + 0.0052105829963693395, + 0.005022459008614533, + 0.005101666989503428 + ] + }, + { + "forward": 0.035861332988133654, + "prefill": 0.03594050000538118, + "decode_steps": [ + 0.0035812499991152436, + 0.003404458999284543, + 0.0031679579988121986, + 0.0032559579994995147, + 0.0033066670002881438, + 0.0032979579991661012, + 0.0032353749993490055, + 0.0032344170031137764, + 0.0032007080008042976, + 0.0034298329992452636, + 0.0034066249936586246, + 0.003249166999012232, + 0.0033040830021491274, + 0.003322707998449914, + 0.003267334002885036, + 0.003338875001645647, + 0.0033616669970797375, + 0.003369333004229702, + 0.003347292004036717, + 0.0035107909934595227, + 0.0035815830051433295, + 0.003600374999223277, + 0.003669083002023399, + 0.003558041003998369, + 0.003975500003434718, + 0.0040510830003768206, + 0.0040065000066533685, + 0.003989624994574115, + 0.004138958000112325, + 0.004053666998515837, + 0.004048791000968777, + 0.0050530829903436825 + ] + }, + { + "forward": 0.03875966700434219, + "prefill": 0.03991929099720437, + "decode_steps": [ + 0.007912082990515046, + 0.008427084001596086, + 0.0067710000002989545, + 0.0065203330013900995, + 0.006191124994074926, + 0.005725541996071115, + 0.005799375008791685, + 0.0038435000024037436, + 0.0037705419963458553, + 0.0054056669905548915, + 0.0034972080029547215, + 0.004159874995821156, + 0.005529250003746711, + 0.004818625006009825, + 0.005326708007487468, + 0.004912333009997383, + 0.004604000001563691, + 0.004549207995296456, + 0.00435408299381379, + 0.004254999992554076, + 0.0042924169974867254, + 0.004273291997378692, + 0.0043215839978074655, + 0.004208459009532817, + 0.004256208005244844, + 0.004334375000325963, + 0.004247249991749413, + 0.004250708996551111, + 0.00427954200131353, + 0.00425916699168738, + 0.004250625002896413, + 0.004196333000436425 + ] + }, + { + "forward": 0.03613337500428315, + "prefill": 0.03589079200173728, + "decode_steps": [ + 0.0034039999882224947, + 0.003321666008559987, + 0.0033284999954048544, + 0.0033752079907571897, + 0.003234874995541759, + 0.0034079170000040904, + 0.003529916997649707, + 0.003259040997363627, + 0.0032938749936874956, + 0.003273540991358459, + 0.0035191250062780455, + 0.00321391700708773, + 0.003138041007332504, + 0.0032623750012135133, + 0.0034045420034090057, + 0.0034769580088322982, + 0.0033669999975245446, + 0.003402500005904585, + 0.003608333005104214, + 0.0035393750004004687, + 0.003583666999475099, + 0.00367654200817924, + 0.003892374996212311, + 0.0037718329986091703, + 0.003820792000624351, + 0.003803165993303992, + 0.004013958998257294, + 0.003999374996055849, + 0.0040170419961214066, + 0.003973874991061166, + 0.004985499996109866, + 0.004648832997190766 + ] + }, + { + "forward": 0.035917707995395176, + "prefill": 0.03607750000082888, + "decode_steps": [ + 0.003580165997846052, + 0.0033808330044848844, + 0.003264375001890585, + 0.0032749999954830855, + 0.003290125008788891, + 0.003533124996465631, + 0.0033134999976027757, + 0.0032348329987144098, + 0.0032645419996697456, + 0.003219667007215321, + 0.003366292003192939, + 0.003261957986978814, + 0.0032618330005789176, + 0.0032922500104177743, + 0.0033218750031664968, + 0.003365791999385692, + 0.0033801250101532787, + 0.0033681250060908496, + 0.0033854579960461706, + 0.0036236660089343786, + 0.003599792005843483, + 0.0035234589886385947, + 0.0036191669933032244, + 0.003709541997523047, + 0.0038229579949984327, + 0.0038971659960225224, + 0.003839582990622148, + 0.003979457993409596, + 0.004083332998561673, + 0.004077833000337705, + 0.005728125004679896, + 0.005308749998221174 + ] + }, + { + "forward": 0.03999029200349469, + "prefill": 0.03922954200243112, + "decode_steps": [ + 0.008329082993441261, + 0.006857792002847418, + 0.006321375010884367, + 0.0063931250042514876, + 0.00676575000397861, + 0.0060289579996606335, + 0.005724167000153102, + 0.006065415989723988, + 0.005762957996921614, + 0.005622458003927022, + 0.005929167004069313, + 0.005606708989944309, + 0.00499787500302773, + 0.0052948750089854, + 0.005115249994560145, + 0.004635833000065759, + 0.005102042006910779, + 0.005020374999730848, + 0.005072583997389302, + 0.004237500004819594, + 0.004286916999262758, + 0.004435666996869259, + 0.004265791998477653, + 0.004247207994922064, + 0.004235624990542419, + 0.0042977499979315326, + 0.004285083006834611, + 0.004503582997131161, + 0.004278790991520509, + 0.00425579100556206, + 0.004575333005050197, + 0.0042454169888515025 + ] + }, + { + "forward": 0.03653370800020639, + "prefill": 0.03686987499531824, + "decode_steps": [ + 0.004358207996119745, + 0.0035069590085186064, + 0.00353745800384786, + 0.003824207989964634, + 0.003875708003761247, + 0.00357616700057406, + 0.00383054198755417, + 0.0037445830093929544, + 0.00402345800830517, + 0.0036842499976046383, + 0.0036617500009015203, + 0.0037015830021118745, + 0.0036670830013463274, + 0.0037174579920247197, + 0.003752542004804127, + 0.003715625003678724, + 0.0037943329953122884, + 0.0038893750024726614, + 0.003843416998279281, + 0.003841332989395596, + 0.0038246250041993335, + 0.00388304099033121, + 0.0038330419920384884, + 0.003863624995574355, + 0.0037585419922834262, + 0.0043149169941898435, + 0.0038311670068651438, + 0.003896584006724879, + 0.003829499997664243, + 0.0038764159980928525, + 0.003762874999665655, + 0.0039878329989733174 + ] + }, + { + "forward": 0.03674125000543427, + "prefill": 0.03697862499393523, + "decode_steps": [ + 0.0034051249967887998, + 0.0032986250007525086, + 0.0032483329996466637, + 0.003206916997442022, + 0.00334024999756366, + 0.003311290987767279, + 0.0033536249975441024, + 0.0032424590026494116, + 0.0035577500093495473, + 0.003216750003048219, + 0.0032938749936874956, + 0.0032822080102050677, + 0.003255416013416834, + 0.0036864169960608706, + 0.0033304159878753126, + 0.0032048330031102523, + 0.0033357499924022704, + 0.0036655829899245873, + 0.003390458005014807, + 0.0033306249970337376, + 0.003766583002288826, + 0.0033953750098589808, + 0.004072124997037463, + 0.0036256669991416857, + 0.0035802500060526654, + 0.003579249998438172, + 0.00417037500301376, + 0.003910375002305955, + 0.003824416999123059, + 0.0038657919940305874, + 0.004367625006125309, + 0.003986166993854567 + ] + }, + { + "forward": 0.03966591600328684, + "prefill": 0.04112029199313838, + "decode_steps": [ + 0.007578874996397644, + 0.007161708999774419, + 0.006644499997491948, + 0.007197917002486065, + 0.007183875000919215, + 0.0060298329917714, + 0.006594584003323689, + 0.0058832500071730465, + 0.005724624992581084, + 0.004871458993875422, + 0.005600917007541284, + 0.005236958008026704, + 0.0049485000054119155, + 0.005080749993794598, + 0.005065334000391886, + 0.005482249995111488, + 0.005126000003656372, + 0.00428999999712687, + 0.004201875010039657, + 0.0040542909991927445, + 0.004310167001676746, + 0.004362250008853152, + 0.0045348330022534356, + 0.003906540994648822, + 0.004113875009352341, + 0.004052584001328796, + 0.004977624994353391, + 0.004161624994594604, + 0.0040825000032782555, + 0.004015999991679564, + 0.004435082999407314, + 0.0039295840106206015 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 36.63747900282033, + "p95": 39.89702950057108, + "min": 35.861332988133654, + "max": 39.99029200349469, + "iqr": 3.4665417442738544, + "mean": 37.53055820125155, + "cv": 0.04762011013107486 + }, + "prefill_ms": { + "count": 10, + "median": 36.924249994626734, + "p95": 40.579841544968076, + "min": 35.89079200173728, + "max": 41.12029199313838, + "iqr": 3.6863539935438894, + "mean": 37.78156249754829, + "cv": 0.05341195854583217 + }, + "decode_ms_per_step": { + "count": 320, + "median": 3.901853495335672, + "p95": 6.532316649827409, + "min": 3.138041007332504, + "max": 8.812874992145225, + "iqr": 1.4647082534793299, + "mean": 4.270974209021006, + "cv": 0.2522158798939845 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 2123.784780849739, + "p95": 2252.616957206575, + "min": 1444.132895501858, + "max": 2263.469493757035, + "iqr": 641.5516328081899, + "mean": 1932.6460847800197, + "cv": 0.1781947122103682 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 1631.0923720047117, + "p95": 1704.845432582553, + "min": 1179.2027133059166, + "max": 1705.764522079865, + "iqr": 431.8713447933226, + "mean": 1500.4745708316912, + "cv": 0.15149459834415396 + }, + "memory": { + "peak_extra_bytes": 47194112, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "reference", + "batch": 8, + "seq_len": 1024, + "samples_seconds": [ + { + "forward": 0.12416870800370816, + "prefill": 0.12621733400737867, + "decode_steps": [ + 0.01299583399668336, + 0.009896832998492755, + 0.00923250000050757, + 0.008414500000071712, + 0.008292416998301633, + 0.007815707998815924, + 0.007651708001503721, + 0.00813983399712015, + 0.00737229100195691, + 0.007436083003995009, + 0.007072332999086939, + 0.007177957988460548, + 0.006983459010371007, + 0.007025875005638227, + 0.0066622080048546195, + 0.006481290998635814, + 0.006152083005872555, + 0.00622445899352897, + 0.006084833003114909, + 0.006022458997904323, + 0.006143749997136183, + 0.006185791993630119, + 0.006150458997581154, + 0.006124042003648356, + 0.0061346670117927715, + 0.006252749997656792, + 0.00660162499116268, + 0.0065561250085011125, + 0.00611416700121481, + 0.006018040992785245, + 0.006247167009860277, + 0.006063166991225444 + ] + }, + { + "forward": 0.12482791600632481, + "prefill": 0.12220633300603367, + "decode_steps": [ + 0.01923829199222382, + 0.013427666999632493, + 0.01064304199826438, + 0.009246249988791533, + 0.0086422079912154, + 0.008098375008557923, + 0.007994708008482121, + 0.008258166999439709, + 0.00758612499339506, + 0.007400374990538694, + 0.007312166009796783, + 0.007150167002691887, + 0.00694054200721439, + 0.006983832994592376, + 0.006984749998082407, + 0.006948082998860627, + 0.006888417003210634, + 0.00665679200028535, + 0.006630541000049561, + 0.006907457995112054, + 0.006866041992907412, + 0.007004750004853122, + 0.0066978329996345565, + 0.006477834001998417, + 0.006562708003912121, + 0.006479499992565252, + 0.006723084006807767, + 0.006666665998636745, + 0.006526707991724834, + 0.006674958000076003, + 0.0062167919968487695, + 0.006135083007393405 + ] + }, + { + "forward": 0.12218083300103899, + "prefill": 0.12138329200388398, + "decode_steps": [ + 0.00799733301391825, + 0.006305375005467795, + 0.006065916997613385, + 0.006176166993100196, + 0.006090832990594208, + 0.005934832995990291, + 0.006063541004550643, + 0.006338749997667037, + 0.00605712499236688, + 0.005928624988882802, + 0.0060329580010147765, + 0.0060537499957717955, + 0.005870666995178908, + 0.0061417079996317625, + 0.006401708000339568, + 0.005783542001154274, + 0.006336459002341144, + 0.006060125000658445, + 0.006032208009855822, + 0.0061096660065231845, + 0.006130209003458731, + 0.006057666993001476, + 0.0061237909976625815, + 0.00621379200310912, + 0.00676499999826774, + 0.0061148749955464154, + 0.006090167007641867, + 0.006712832997436635, + 0.006530458005727269, + 0.006781541989766993, + 0.006477832997916266, + 0.006195875001139939 + ] + }, + { + "forward": 0.12397937499918044, + "prefill": 0.12653304199920967, + "decode_steps": [ + 0.011334666007314809, + 0.00892675000068266, + 0.008386833011172712, + 0.007677208996028639, + 0.007467832998372614, + 0.007449459008057602, + 0.0071404999907827005, + 0.007257833989569917, + 0.0069736669975100085, + 0.007023792000836693, + 0.007082292009727098, + 0.007033583999145776, + 0.006989540997892618, + 0.006683833998977207, + 0.006720749996020459, + 0.006615540987695567, + 0.006546125005115755, + 0.006630375006352551, + 0.006561165995663032, + 0.0064622500067343935, + 0.006587167008547112, + 0.006836667002062313, + 0.006662999992840923, + 0.006335583995678462, + 0.0063487500010523945, + 0.006795042005251162, + 0.00722204199701082, + 0.007059874988044612, + 0.006904207999468781, + 0.006353417003992945, + 0.0064933340036077425, + 0.00678537500789389 + ] + }, + { + "forward": 0.12404974999662954, + "prefill": 0.12419058299565222, + "decode_steps": [ + 0.018533875001594424, + 0.013674417001311667, + 0.009942083997884765, + 0.009047790998010896, + 0.008382917003473267, + 0.008119417005218565, + 0.007747916999505833, + 0.007578749995445833, + 0.007288917011464946, + 0.0072366250096820295, + 0.007415875006699935, + 0.00750612499541603, + 0.0071362080052495, + 0.007248958994750865, + 0.007108707999577746, + 0.006653208009083755, + 0.006935040990356356, + 0.006707332999212667, + 0.006776084002922289, + 0.006727125000907108, + 0.006569999997736886, + 0.006577958993148059, + 0.006566166994161904, + 0.00644612500036601, + 0.006702333004795946, + 0.006893040990689769, + 0.006874124999740161, + 0.006648082999163307, + 0.006309208998573013, + 0.006086167006287724, + 0.006282250004005618, + 0.00610087500535883 + ] + }, + { + "forward": 0.1266610419988865, + "prefill": 0.12368787500599865, + "decode_steps": [ + 0.008163583988789469, + 0.007251791990711354, + 0.006860250010504387, + 0.00749250000808388, + 0.007575958996312693, + 0.00802579200535547, + 0.008416917000431567, + 0.008155125004122965, + 0.007742708010482602, + 0.00729604200751055, + 0.006726666004396975, + 0.006592833989998326, + 0.00626520901278127, + 0.0063404169923160225, + 0.006022417001076974, + 0.006238207992282696, + 0.00604741700226441, + 0.0063123750005615875, + 0.005700541005353443, + 0.005942207993939519, + 0.006555040992680006, + 0.006264334006118588, + 0.006238166999537498, + 0.006391791990608908, + 0.006177458999445662, + 0.00614070899609942, + 0.006129000001237728, + 0.006213290995219722, + 0.006145916995592415, + 0.005978749992209487, + 0.006092957992223091, + 0.006312917001196183 + ] + }, + { + "forward": 0.12563545900047757, + "prefill": 0.12310858299315441, + "decode_steps": [ + 0.035717625010875054, + 0.011806832990259863, + 0.008141583006363362, + 0.007727916003204882, + 0.0074585000111255795, + 0.0073852079949574545, + 0.007398665999062359, + 0.006839875000878237, + 0.007507207992603071, + 0.007647541002370417, + 0.007263833002070896, + 0.0069231670058798045, + 0.006856915992102586, + 0.006875624996609986, + 0.006657583988271654, + 0.006700084006297402, + 0.006793834007112309, + 0.006711583002470434, + 0.00696666601288598, + 0.006716709001921117, + 0.007131874997867271, + 0.006724166989442892, + 0.006720667006447911, + 0.006362790998537093, + 0.006682332998025231, + 0.00680333400669042, + 0.0066062919941032305, + 0.006691207992844284, + 0.005897583003388718, + 0.0062676249945070595, + 0.0064317920041503385, + 0.006090374998166226 + ] + }, + { + "forward": 0.1274371670006076, + "prefill": 0.12572095800715033, + "decode_steps": [ + 0.01798091699311044, + 0.014875792010570876, + 0.010375375000876375, + 0.009937916998751462, + 0.008842458002618514, + 0.008799749994068407, + 0.008825375000014901, + 0.007186250004451722, + 0.007463207992259413, + 0.008029832999454811, + 0.0074094169976888224, + 0.007922916003735736, + 0.007597124989842996, + 0.007662249990971759, + 0.007243667001603171, + 0.007090333005180582, + 0.006943707994651049, + 0.007012375004705973, + 0.007011749999946915, + 0.0067130410025129095, + 0.007148999997298233, + 0.006816332999733277, + 0.006885916998726316, + 0.006812917010392994, + 0.006646167006692849, + 0.0062824999913573265, + 0.006585291004739702, + 0.007508707989472896, + 0.006974999996600673, + 0.006711499998345971, + 0.005961208007647656, + 0.006179167001391761 + ] + }, + { + "forward": 0.12656808299652766, + "prefill": 0.12471575000381563, + "decode_steps": [ + 0.007811750008841045, + 0.0063731669943081215, + 0.0062603749975096434, + 0.005727375013520941, + 0.006252874998608604, + 0.006258916007936932, + 0.006476250011473894, + 0.006233333988348022, + 0.006536459011840634, + 0.005781332991318777, + 0.0061020830034976825, + 0.005709917008061893, + 0.006015916995238513, + 0.006617166000069119, + 0.006086167006287724, + 0.005979958004900254, + 0.006077000012737699, + 0.0061444169987225905, + 0.006029625001247041, + 0.005974790998152457, + 0.006061292006052099, + 0.0063635830010753125, + 0.006081500003347173, + 0.006106166998506524, + 0.005727416995796375, + 0.006246583012398332, + 0.006268958008149639, + 0.0062036250019446015, + 0.006238083995413035, + 0.0068404580088099465, + 0.005893709007068537, + 0.006100625003455207 + ] + }, + { + "forward": 0.13179345900425687, + "prefill": 0.12930666700412985, + "decode_steps": [ + 0.018156707999878563, + 0.01305375000811182, + 0.01196512499882374, + 0.012150042006396689, + 0.016553124994970858, + 0.010621667010127567, + 0.007433542006765492, + 0.007903249992523342, + 0.006932167001650669, + 0.0075037910137325525, + 0.006908374998602085, + 0.006606958995689638, + 0.006726541992975399, + 0.006659916005446576, + 0.006992874987190589, + 0.007108584002708085, + 0.007840958001906984, + 0.006687917004455812, + 0.0067749160079983994, + 0.006780458003049716, + 0.006714499992085621, + 0.00636950001353398, + 0.006937749989447184, + 0.007172583005740307, + 0.007032249995972961, + 0.006360082988976501, + 0.007097709007211961, + 0.0065883749921340495, + 0.0064343339909100905, + 0.0068427920050453395, + 0.006346791997202672, + 0.0062902500067139044 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 125.23168750340119, + "p95": 129.8331276026147, + "min": 122.18083300103899, + "max": 131.79345900425687, + "iqr": 2.5583127498975955, + "mean": 125.73017920076381, + "cv": 0.02104208941990658 + }, + "prefill_ms": { + "count": 10, + "median": 124.45316649973392, + "p95": 128.05853575191577, + "min": 121.38329200388398, + "max": 129.30666700412985, + "iqr": 2.8398340109561104, + "mean": 124.70704170264071, + "cv": 0.018703122247788088 + }, + "decode_ms_per_step": { + "count": 320, + "median": 6.713770497299265, + "p95": 10.622735759534407, + "min": 5.700541005353443, + "max": 35.717625010875054, + "iqr": 1.0306880103598814, + "mean": 7.292892994018985, + "cv": 0.3391885791406309 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1082.776307133698, + "p95": 1285.1020669619993, + "min": 978.7893854282853, + "max": 1289.1472987459135, + "iqr": 168.04383683791116, + "mean": 1107.330698541005, + "cv": 0.10433496522186492 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 707.5719653845504, + "p95": 794.4689028695987, + "min": 654.9756045266693, + "max": 796.617799839137, + "iqr": 68.86673375616124, + "mean": 717.9785372878182, + "cv": 0.0697069579309342 + }, + "memory": { + "peak_extra_bytes": 519053312, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "auto", + "batch": 8, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.12175366700103041 + }, + { + "forward": 0.11919062500237487 + }, + { + "forward": 0.11729883300722577 + }, + { + "forward": 0.12030954199144617 + }, + { + "forward": 0.11781537499336991 + }, + { + "forward": 0.11778808300732635 + }, + { + "forward": 0.11868825000419747 + }, + { + "forward": 0.11817758399411105 + }, + { + "forward": 0.11694920800800901 + }, + { + "forward": 0.11938829100108705 + } + ], + "forward_ms": { + "count": 10, + "median": 118.43291699915426, + "p95": 121.1038107467175, + "min": 116.94920800800901, + "max": 121.75366700103041, + "iqr": 1.5439684975717682, + "mean": 118.7359458010178, + "cv": 0.01240734153214498 + }, + "memory": { + "peak_extra_bytes": 94388224, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "sdpa", + "batch": 8, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.0889047499949811, + "prefill": 0.08714033299474977, + "decode_steps": [ + 0.009636167000280693, + 0.008994874995551072, + 0.007939834002172574, + 0.008720665995497257, + 0.008116083001368679, + 0.00850841699866578, + 0.008210082989535294, + 0.008202375000109896, + 0.008063500004936941, + 0.009412416999111883, + 0.008981040999060497, + 0.007928833001642488, + 0.008289541001431644, + 0.007109708996722475, + 0.007604915997944772, + 0.007009082997683436, + 0.0067361669935053214, + 0.006090166003559716, + 0.00607750000199303, + 0.0065090830030385405, + 0.006088499998440966, + 0.00615749999997206, + 0.00609320800867863, + 0.006049708987120539, + 0.00641545899270568, + 0.00625995900190901, + 0.006077457990613766, + 0.006409542009350844, + 0.006136167008662596, + 0.006429458997445181, + 0.006247666999115609, + 0.006102084007579833 + ] + }, + { + "forward": 0.08265712500724476, + "prefill": 0.08270533299946692, + "decode_steps": [ + 0.005040042000473477, + 0.004952792005497031, + 0.004753041997901164, + 0.00467712500540074, + 0.004689042005338706, + 0.0047374999994644895, + 0.00471208400267642, + 0.004686915999627672, + 0.004669749992899597, + 0.004724207989056595, + 0.005078583999420516, + 0.004834042003494687, + 0.0047268329944927245, + 0.004978166994987987, + 0.004875250000623055, + 0.0048209590022452176, + 0.00518908399681095, + 0.0051049169996986166, + 0.005117707987665199, + 0.005302125005982816, + 0.005319415999110788, + 0.005360374998417683, + 0.005525875007151626, + 0.00547862499661278, + 0.005525000000488944, + 0.006065417008358054, + 0.006060250001610257, + 0.007512124997447245, + 0.006805041994084604, + 0.007619916999829002, + 0.008494166992022656, + 0.008145125000737607 + ] + }, + { + "forward": 0.08271641700412147, + "prefill": 0.0827144579961896, + "decode_steps": [ + 0.005122292001033202, + 0.004951875002007, + 0.004914250006549992, + 0.0046712499897694215, + 0.004772999993292615, + 0.004749875006382354, + 0.004706417006673291, + 0.0047010830021463335, + 0.004695208001066931, + 0.004690874993684702, + 0.004736041999422014, + 0.004843458998948336, + 0.00477675000729505, + 0.004909583003609441, + 0.005072792002465576, + 0.005031916996813379, + 0.005040375006501563, + 0.0054806669941172, + 0.005860500008566305, + 0.005437334009911865, + 0.0056190830073319376, + 0.005671624996466562, + 0.006129540997790173, + 0.00605508299486246, + 0.00718241700087674, + 0.007420415990054607, + 0.007250666996696964, + 0.008382125000935048, + 0.008329792006406933, + 0.008821415991405956, + 0.008334167010616511, + 0.00872658399748616 + ] + }, + { + "forward": 0.08660487498855218, + "prefill": 0.08645195799181238, + "decode_steps": [ + 0.008617084007710218, + 0.008545332995709032, + 0.007616709001013078, + 0.006365624998579733, + 0.0062072500004433095, + 0.006088416994316503, + 0.006464083009632304, + 0.006019958003889769, + 0.006093875010265037, + 0.006336332997307181, + 0.006110999995144084, + 0.006162707999465056, + 0.006012333004036918, + 0.006007542004226707, + 0.006105957989348099, + 0.006422542006475851, + 0.006088082998758182, + 0.006073334006941877, + 0.006063958004233427, + 0.006042708992026746, + 0.006044583002221771, + 0.006031915996572934, + 0.006150749992229976, + 0.006012124998960644, + 0.00609320800867863, + 0.006110041998908855, + 0.0060608749918174, + 0.006016125000314787, + 0.00585620800848119, + 0.0060542909923242405, + 0.006090915994718671, + 0.0060664160118903965 + ] + }, + { + "forward": 0.08314575000258628, + "prefill": 0.0827197500038892, + "decode_steps": [ + 0.005051959000411443, + 0.004936957993777469, + 0.004746625010739081, + 0.004689292007242329, + 0.004724792001070455, + 0.004652665986213833, + 0.00477691600099206, + 0.004777583002578467, + 0.004848957993090153, + 0.004790749997482635, + 0.0047322910104412585, + 0.00482741701125633, + 0.004919583006994799, + 0.004880082997260615, + 0.005153916994458996, + 0.005094833002658561, + 0.005134625011123717, + 0.005591084001935087, + 0.005300791992340237, + 0.0054819999932078645, + 0.0055520829919259995, + 0.00551600000471808, + 0.006047750008292496, + 0.006510916995466687, + 0.007495166995795444, + 0.007486250004149042, + 0.007932541004265659, + 0.008561041002394632, + 0.00818295800127089, + 0.008575292013119906, + 0.008733916998608038, + 0.007283083992660977 + ] + }, + { + "forward": 0.08313054199970793, + "prefill": 0.08779204198799562, + "decode_steps": [ + 0.006353832999593578, + 0.006532790997880511, + 0.006205541008966975, + 0.007583166006952524, + 0.0067964580084662884, + 0.007243582993396558, + 0.007900374999735504, + 0.007825708002201281, + 0.007976541994139552, + 0.007884375008870848, + 0.008217666007112712, + 0.008150374997057952, + 0.007623540994245559, + 0.007437874999595806, + 0.007295999996131286, + 0.006549166995682754, + 0.006905707996338606, + 0.006478541996330023, + 0.007381749994237907, + 0.007580667006550357, + 0.007998374989256263, + 0.0072680829907767475, + 0.007267833992955275, + 0.007017000010819174, + 0.0061525409983005375, + 0.0061797089874744415, + 0.006609582997043617, + 0.007608915999298915, + 0.006070875009754673, + 0.006136583993793465, + 0.0060868330037919804, + 0.0065653750061756 + ] + }, + { + "forward": 0.08714129200961906, + "prefill": 0.08557475000270642, + "decode_steps": [ + 0.008180333999916911, + 0.006815083004767075, + 0.006135500007076189, + 0.006427333006286062, + 0.006807374986237846, + 0.006200916992384009, + 0.0068027920060558245, + 0.0057398749922867864, + 0.00939737498993054, + 0.009924750003847294, + 0.00535150000359863, + 0.005184333000215702, + 0.005095458007417619, + 0.005408208002336323, + 0.005114374987897463, + 0.005006541003240272, + 0.005442667010356672, + 0.005184417008422315, + 0.00517412499175407, + 0.0055918750003911555, + 0.005184083012863994, + 0.005325457997969352, + 0.0053149579907767475, + 0.005092749997857027, + 0.005299500000546686, + 0.005472707998706028, + 0.005208417001995258, + 0.005270916997687891, + 0.00527995900483802, + 0.005278875003568828, + 0.005342083997675218, + 0.005473791999975219 + ] + }, + { + "forward": 0.08337700000265613, + "prefill": 0.08417958399513736, + "decode_steps": [ + 0.005577333009568974, + 0.005238665995420888, + 0.005411916994489729, + 0.005363250005757436, + 0.005516334000276402, + 0.005901666998397559, + 0.005603790996246971, + 0.0060996669926680624, + 0.007182250003097579, + 0.006077041995013133, + 0.006200791991432197, + 0.005856457995832898, + 0.006150707995402627, + 0.0061847500037401915, + 0.006048583003575914, + 0.005980375004583038, + 0.006121582991909236, + 0.006097124991356395, + 0.005998416992952116, + 0.006017916995915584, + 0.005806084009236656, + 0.005912334003369324, + 0.0057817500055534765, + 0.006101958002545871, + 0.006092416995670646, + 0.006103374995291233, + 0.00610779199632816, + 0.006114540999988094, + 0.008674541997606866, + 0.008201707998523489, + 0.006200083007570356, + 0.006990334004512988 + ] + }, + { + "forward": 0.0829557919932995, + "prefill": 0.08270516600168776, + "decode_steps": [ + 0.00499020800634753, + 0.004912000003969297, + 0.004749749990878627, + 0.004711917004897259, + 0.004699082986917347, + 0.005021916003897786, + 0.005038374991272576, + 0.004675999996834435, + 0.0047399999893968925, + 0.0049153750005643815, + 0.0048049589968286455, + 0.00485937501071021, + 0.00509825001063291, + 0.005052749998867512, + 0.005097458997624926, + 0.005290125001920387, + 0.00536495799315162, + 0.0055514999985462055, + 0.005520917009562254, + 0.005636125002638437, + 0.0057697919983183965, + 0.0061377079982776195, + 0.006216290988959372, + 0.007511207993957214, + 0.007626625010743737, + 0.00789308300591074, + 0.007946167010231875, + 0.008262540999567136, + 0.00814729199919384, + 0.007930166000733152, + 0.007700041998759843, + 0.007668000005651265 + ] + }, + { + "forward": 0.0884720419999212, + "prefill": 0.08698362500581425, + "decode_steps": [ + 0.009871832997305319, + 0.008097957994323224, + 0.00792337500024587, + 0.00788591599848587, + 0.0075825000094482675, + 0.007474500001990236, + 0.007700208007008769, + 0.007382083000265993, + 0.007451792000210844, + 0.011056542003643699, + 0.009434249994228594, + 0.00769866700284183, + 0.007498709004721604, + 0.005432874997495674, + 0.005531540999072604, + 0.005310666994773783, + 0.005296625007758848, + 0.005168000003322959, + 0.005192707991227508, + 0.0054032909974921495, + 0.00543824999476783, + 0.005401041998993605, + 0.005346332996850833, + 0.0051814579928759485, + 0.005313125002430752, + 0.005287249994580634, + 0.005266874999506399, + 0.005375499997171573, + 0.005427416996099055, + 0.0054042500123614445, + 0.005360541996196844, + 0.0055231250007636845 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 83.26137500262121, + "p95": 88.71003139720415, + "min": 82.65712500724476, + "max": 88.9047499949811, + "iqr": 4.007708259450737, + "mean": 84.91055850026896, + "cv": 0.030111100516333443 + }, + "prefill_ms": { + "count": 10, + "median": 84.8771669989219, + "p95": 87.49877294103499, + "min": 82.70516600168776, + "max": 87.79204198799562, + "iqr": 4.134927254199283, + "mean": 84.89669989794493, + "cv": 0.024916391790023355 + }, + "decode_ms_per_step": { + "count": 320, + "median": 6.057666498236358, + "p95": 8.561753552930895, + "min": 4.652665986213833, + "max": 11.056542003643699, + "iqr": 1.9666047519422136, + "mean": 6.233399865595857, + "cv": 0.2034439026786498 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 1326.0050938891231, + "p95": 1417.9821059751644, + "min": 1100.5679756870645, + "max": 1458.0123429726477, + "iqr": 129.04679936089474, + "mean": 1292.7845009709063, + "cv": 0.08763872740263585 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 922.663681310247, + "p95": 972.2560043462889, + "min": 800.6317460347793, + "max": 991.1461342519716, + "iqr": 75.5631050526523, + "mean": 904.2105469039732, + "cv": 0.06849850321120309 + }, + "memory": { + "peak_extra_bytes": 94388224, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "reference", + "batch": 8, + "seq_len": 2048, + "samples_seconds": [ + { + "forward": 0.4634996250097174, + "prefill": 0.43780649999098387, + "decode_steps": [ + 0.012231832995894365, + 0.012411250005243346, + 0.011338333002640866, + 0.010591124999336898, + 0.009943500001099892, + 0.00978637499792967, + 0.009640750009566545, + 0.009493457997450605, + 0.010130167007446289, + 0.00952487500035204, + 0.009556666002026759, + 0.009250458009773865, + 0.009125374999712221, + 0.009431291997316293, + 0.009105334000196308, + 0.008630416996311396, + 0.008098417005385272, + 0.008342749992152676, + 0.008001790993148461, + 0.008550832993933, + 0.007973790998221375, + 0.00814558299316559, + 0.007984084004419856, + 0.008312584002851509, + 0.008081708001554944, + 0.008030332988710143, + 0.007898750001913868, + 0.007922124990727752, + 0.008138499993947335, + 0.00788145799015183, + 0.008244750002631918, + 0.008147832995746285 + ] + }, + { + "forward": 0.44315033400198445, + "prefill": 0.4321321659954265, + "decode_steps": [ + 0.022682041002553888, + 0.016294082990498282, + 0.01246470799378585, + 0.011306208005407825, + 0.01108883299457375, + 0.010318208005628549, + 0.009902791993226856, + 0.009714375002658926, + 0.009825208006077446, + 0.010952500000712462, + 0.009386374993482605, + 0.009408541998709552, + 0.009052167006302625, + 0.009109584003454074, + 0.009006750013213605, + 0.008980250000604428, + 0.009021792007843032, + 0.00886258301034104, + 0.008964790991740301, + 0.008895374994608574, + 0.008905499998945743, + 0.008836584005621262, + 0.008698375007952563, + 0.00900016599916853, + 0.008202124998206273, + 0.008189291998860426, + 0.008052624994888902, + 0.008073624994722195, + 0.008065625006565824, + 0.007979166999575682, + 0.008013832994038239, + 0.0081381249910919 + ] + }, + { + "forward": 0.44105858300463296, + "prefill": 0.4286675409966847, + "decode_steps": [ + 0.020835332994465716, + 0.016686125003616326, + 0.012881542003015056, + 0.011961666998104192, + 0.011218750005355105, + 0.010037542000645772, + 0.009759542008396238, + 0.00963962500100024, + 0.010459208002430387, + 0.009636707996833138, + 0.009293084003729746, + 0.0092398340057116, + 0.009006125008454546, + 0.009367083999677561, + 0.0092521670012502, + 0.009016207986860536, + 0.008931667005526833, + 0.008953374999691732, + 0.009072625005501322, + 0.00899083400145173, + 0.009003583007142879, + 0.009111916995607316, + 0.009785332993487827, + 0.009798040991881862, + 0.008711625006981194, + 0.008712084003491327, + 0.008467166990158148, + 0.008226249992731027, + 0.008142749997205101, + 0.008032375000766478, + 0.0080620409862604, + 0.008031459001358598 + ] + }, + { + "forward": 0.4529145830019843, + "prefill": 0.4301450409984682, + "decode_steps": [ + 0.020654000007198192, + 0.017117291994509287, + 0.013078707997919992, + 0.011383457996998914, + 0.010994417010806501, + 0.010217458009719849, + 0.010100625004270114, + 0.010262750001857057, + 0.009463458001846448, + 0.009583999999449588, + 0.009535249992040917, + 0.009030790999531746, + 0.009139040994341485, + 0.00853345901123248, + 0.008436875010374933, + 0.008121249993564561, + 0.008115415999782272, + 0.008831125000142492, + 0.008397499987040646, + 0.008121208011289127, + 0.00802133399702143, + 0.008078041006228887, + 0.008028457988984883, + 0.008082208994892426, + 0.00800333300139755, + 0.008054250007262453, + 0.0079939999995986, + 0.008056959006353281, + 0.008228708000387996, + 0.008497457994963042, + 0.00813266698969528, + 0.008154832990840077 + ] + }, + { + "forward": 0.43323720899934415, + "prefill": 0.4280367920000572, + "decode_steps": [ + 0.018661375012015924, + 0.01221037500363309, + 0.011359916999936104, + 0.010744458995759487, + 0.010302416994818486, + 0.00983387501037214, + 0.009564333988237195, + 0.009569875008310191, + 0.009567417000653222, + 0.010159041004953906, + 0.009729249999509193, + 0.009316791998571716, + 0.00865341599273961, + 0.008442792008281685, + 0.008367542002815753, + 0.008162541998899542, + 0.008151791000273079, + 0.00810450001154095, + 0.008122749990434386, + 0.008146084001054987, + 0.008061749991611578, + 0.008029666991205886, + 0.008046417002333328, + 0.008181166995200329, + 0.008499791991198435, + 0.00817458299570717, + 0.007974457999807782, + 0.008029707998503, + 0.007997042004717514, + 0.00792070799798239, + 0.008105332992272452, + 0.008018833003006876 + ] + }, + { + "forward": 0.4335014999960549, + "prefill": 0.45289950000005774, + "decode_steps": [ + 0.01586779099307023, + 0.013156250002793968, + 0.011201665998669341, + 0.010608583004795946, + 0.009843833002378233, + 0.01056845800485462, + 0.010359541993238963, + 0.009502166998572648, + 0.009240375002264045, + 0.008811292005702853, + 0.009242665997589938, + 0.009138833003817126, + 0.010287083001458086, + 0.010017374996095896, + 0.009147791002760641, + 0.009690333012258634, + 0.008707582994247787, + 0.008772083005169407, + 0.008892708996427245, + 0.00937512500968296, + 0.008988500005216338, + 0.008796333000645973, + 0.008689666996360756, + 0.008535250002751127, + 0.008571625003241934, + 0.009538000012980774, + 0.008901957990019582, + 0.008352583012310788, + 0.008746792009333149, + 0.00889566600380931, + 0.00849495800503064, + 0.008365000001504086 + ] + }, + { + "forward": 0.4475934999936726, + "prefill": 0.43083958300121594, + "decode_steps": [ + 0.020788207999430597, + 0.01790991700545419, + 0.012832332999096252, + 0.01254083299136255, + 0.012166832995717414, + 0.01084529199579265, + 0.010000333000789396, + 0.00983387501037214, + 0.009495792008237913, + 0.009378582995850593, + 0.009215791986207478, + 0.00931441699503921, + 0.009164125003735535, + 0.009127666999120265, + 0.010135124990483746, + 0.01017162500647828, + 0.008946166999521665, + 0.009288624991313554, + 0.009081374999368563, + 0.008981958002550527, + 0.009377875001518987, + 0.010310416007996537, + 0.009066083002835512, + 0.00825066700053867, + 0.00873666700499598, + 0.008573792001698166, + 0.008558790999813937, + 0.00811604200862348, + 0.008095084005617537, + 0.008103125001071021, + 0.00791120799840428, + 0.008153499991749413 + ] + }, + { + "forward": 0.43088179199548904, + "prefill": 0.4332028330100002, + "decode_steps": [ + 0.020826583000598475, + 0.017182458002935164, + 0.012779292010236531, + 0.012089416006347165, + 0.011966875012149103, + 0.010239416995318606, + 0.00975237500097137, + 0.009717292006826028, + 0.01001945800089743, + 0.009742624999489635, + 0.009956290989066474, + 0.009424625008250587, + 0.019872958000632934, + 0.011539041996002197, + 0.009945792000507936, + 0.009438875000341795, + 0.009524583001621068, + 0.010153458002605475, + 0.008681834005983546, + 0.008730041998205706, + 0.008389042006456293, + 0.008573165992856957, + 0.008545041011529975, + 0.008548500001779757, + 0.008795083005679771, + 0.009486625000135973, + 0.009303916012868285, + 0.00836058400454931, + 0.009991875005653128, + 0.008591916994191706, + 0.009383624987094663, + 0.00890600000275299 + ] + }, + { + "forward": 0.4350517909915652, + "prefill": 0.4297022920072777, + "decode_steps": [ + 0.019756958994548768, + 0.017045542001142167, + 0.013394958004937507, + 0.012227708997670561, + 0.01116675000230316, + 0.010792665998451412, + 0.009773708996362984, + 0.010045332994195633, + 0.009891875000903383, + 0.009445708987186663, + 0.00924445800774265, + 0.008460042008664459, + 0.008353625002200715, + 0.008490916996379383, + 0.008245292003266513, + 0.008976874989457428, + 0.009137542001553811, + 0.00829716699081473, + 0.00828787501086481, + 0.00869154100655578, + 0.009688207996077836, + 0.008294458006275818, + 0.008325125003466383, + 0.008748457999899983, + 0.008345167007064447, + 0.00848674999724608, + 0.008271667000371963, + 0.008313584010466002, + 0.008422917002462782, + 0.009109874998102896, + 0.008282000009785406, + 0.008260874994448386 + ] + }, + { + "forward": 0.44073849999404047, + "prefill": 0.4300881670060335, + "decode_steps": [ + 0.022129165998194367, + 0.01639695800258778, + 0.012428582995198667, + 0.011645209000562318, + 0.0107415420061443, + 0.010196250004810281, + 0.009843541993177496, + 0.009608291991753504, + 0.009648167004343122, + 0.010200999997323379, + 0.009777083003427833, + 0.009228249997249804, + 0.009168082993710414, + 0.009048291001818143, + 0.009053749992744997, + 0.008905750000849366, + 0.008937292004702613, + 0.0089869590010494, + 0.008863374998327345, + 0.008827958998153917, + 0.009266833003493957, + 0.009211707991198637, + 0.008587333999457769, + 0.008306625008117408, + 0.007838958001229912, + 0.00814691701089032, + 0.008008790988242254, + 0.008053124998696148, + 0.00857437499507796, + 0.008359542000107467, + 0.008258415997261181, + 0.008042332992772572 + ] + } + ], + "forward_ms": { + "count": 10, + "median": 440.8985414993367, + "p95": 458.7363561062375, + "min": 430.88179199548904, + "max": 463.4996250097174, + "iqr": 12.593635750818066, + "mean": 442.16274169884855, + "cv": 0.023077401121622845 + }, + "prefill_ms": { + "count": 10, + "median": 430.49231199984206, + "p95": 446.1076499959745, + "min": 428.0367920000572, + "max": 452.89950000005774, + "iqr": 3.1364054993900936, + "mean": 433.35204150062054, + "cv": 0.017100127906693154 + }, + "decode_ms_per_step": { + "count": 320, + "median": 9.052958499523811, + "p95": 15.889105592941638, + "min": 7.838958001229912, + "max": 22.682041002553888, + "iqr": 1.550041495647747, + "mean": 9.74727329403322, + "cv": 0.25567288923344783 + }, + "decode_tokens_per_second": { + "count": 10, + "median": 817.1608803196166, + "p95": 877.1638793070784, + "min": 756.3700576293028, + "max": 882.9215106631182, + "iqr": 28.70752600671574, + "mean": 822.1920578585808, + "cv": 0.04424484722710234 + }, + "generation_tokens_per_second": { + "count": 10, + "median": 343.3095928903861, + "p95": 353.2426796486239, + "min": 331.75168210254867, + "max": 354.4494792244316, + "iqr": 6.4241361413652385, + "mean": 343.62034765597286, + "cv": 0.019538649891093295 + }, + "memory": { + "peak_extra_bytes": 2044739584, + "method": "mps_live_allocated_peak_per_op" + } + } + ] +} diff --git a/experiments/records/attention_quality/20261022T033711Z_attention_quality_615767d.json b/experiments/records/attention_quality/20261022T033711Z_attention_quality_615767d.json new file mode 100644 index 0000000..60258e4 --- /dev/null +++ b/experiments/records/attention_quality/20261022T033711Z_attention_quality_615767d.json @@ -0,0 +1,977 @@ +{ + "record_version": 1, + "created_utc": "20261022T033711Z", + "experiment": "attention_quality", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "cpu,mps", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "615767d83416226055484fb1f7336bdf4a40eaa0", + "branch": "experiments/attention-kv-audit", + "dirty": true + }, + "command": "python scripts/compare_attention_quality.py artifacts/runs/science-v2-s42/best_checkpoint.pt artifacts/runs/science-v2-s43/best_checkpoint.pt artifacts/runs/science-v2-s44/best_checkpoint.pt" + }, + "settings": { + "validation_text": "/Users/anvit/Downloads/Projects/MiniLM-Lab/data/science/validation.txt", + "validation_sha256": "e25d4e45706bbf4251129dbb079f3487660c453cf8b34407901350538b23e2e7", + "validation_targets": 112111, + "oracle": "cpu float64, reference backend", + "loss_precision": "logits upcast to float32 before cross-entropy", + "greedy": { + "prompts": 8, + "prompt_tokens": 32, + "generated_tokens": 64, + "cached": true + }, + "skipped_targets": [] + }, + "results": [ + { + "checkpoint": "artifacts/runs/science-v2-s42/best_checkpoint.pt", + "checkpoint_sha256": "657c3c616443474aa04c407c870c50a653446522eda677b15064a8639772501d", + "seed": 42, + "device": "cpu", + "dtype": "float32", + "backend": "auto", + "max_abs_logit_error": 3.654949019749054e-05, + "mean_abs_logit_error": 1.0375581694966199e-06, + "top1_agreement": 1.0, + "validation_loss": 2.8669173297000294, + "perplexity": 17.58273280263234, + "bits_per_byte": 4.1360874141968615, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s42/best_checkpoint.pt", + "checkpoint_sha256": "657c3c616443474aa04c407c870c50a653446522eda677b15064a8639772501d", + "seed": 42, + "device": "cpu", + "dtype": "float32", + "backend": "sdpa", + "max_abs_logit_error": 3.347584891333355e-05, + "mean_abs_logit_error": 1.0386070915819935e-06, + "top1_agreement": 1.0, + "validation_loss": 2.8669173297000294, + "perplexity": 17.58273280263234, + "bits_per_byte": 4.1360874141968615, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s42/best_checkpoint.pt", + "checkpoint_sha256": "657c3c616443474aa04c407c870c50a653446522eda677b15064a8639772501d", + "seed": 42, + "device": "cpu", + "dtype": "float32", + "backend": "reference", + "max_abs_logit_error": 3.657529044165386e-05, + "mean_abs_logit_error": 1.0373159225101448e-06, + "top1_agreement": 1.0, + "validation_loss": 2.8669173297000294, + "perplexity": 17.58273280263234, + "bits_per_byte": 4.1360874141968615, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s42/best_checkpoint.pt", + "checkpoint_sha256": "657c3c616443474aa04c407c870c50a653446522eda677b15064a8639772501d", + "seed": 42, + "device": "mps", + "dtype": "float32", + "backend": "auto", + "max_abs_logit_error": 3.267033399900754e-05, + "mean_abs_logit_error": 1.0308867831769693e-06, + "top1_agreement": 1.0, + "validation_loss": 2.8669173297000294, + "perplexity": 17.58273280263234, + "bits_per_byte": 4.1360874141968615, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s42/best_checkpoint.pt", + "checkpoint_sha256": "657c3c616443474aa04c407c870c50a653446522eda677b15064a8639772501d", + "seed": 42, + "device": "mps", + "dtype": "float32", + "backend": "sdpa", + "max_abs_logit_error": 3.267033399900754e-05, + "mean_abs_logit_error": 1.0308867831769693e-06, + "top1_agreement": 1.0, + "validation_loss": 2.8669173297000294, + "perplexity": 17.58273280263234, + "bits_per_byte": 4.1360874141968615, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s42/best_checkpoint.pt", + "checkpoint_sha256": "657c3c616443474aa04c407c870c50a653446522eda677b15064a8639772501d", + "seed": 42, + "device": "mps", + "dtype": "float32", + "backend": "reference", + "max_abs_logit_error": 4.110524344458355e-05, + "mean_abs_logit_error": 1.0309519384724672e-06, + "top1_agreement": 1.0, + "validation_loss": 2.8669173297000294, + "perplexity": 17.58273280263234, + "bits_per_byte": 4.1360874141968615, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s42/best_checkpoint.pt", + "checkpoint_sha256": "657c3c616443474aa04c407c870c50a653446522eda677b15064a8639772501d", + "seed": 42, + "device": "mps", + "dtype": "float16", + "backend": "auto", + "max_abs_logit_error": 0.07358595400162127, + "mean_abs_logit_error": 0.002181009015840955, + "top1_agreement": 0.9987958073616028, + "validation_loss": 2.866908409968692, + "perplexity": 17.582575970079013, + "bits_per_byte": 4.1360745457446955, + "greedy_identical_sequences": 6, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 62, + 64, + 62, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s42/best_checkpoint.pt", + "checkpoint_sha256": "657c3c616443474aa04c407c870c50a653446522eda677b15064a8639772501d", + "seed": 42, + "device": "mps", + "dtype": "float16", + "backend": "sdpa", + "max_abs_logit_error": 0.07358595400162127, + "mean_abs_logit_error": 0.002181009015840955, + "top1_agreement": 0.9987958073616028, + "validation_loss": 2.866908409968692, + "perplexity": 17.582575970079013, + "bits_per_byte": 4.1360745457446955, + "greedy_identical_sequences": 6, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 62, + 64, + 62, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s42/best_checkpoint.pt", + "checkpoint_sha256": "657c3c616443474aa04c407c870c50a653446522eda677b15064a8639772501d", + "seed": 42, + "device": "mps", + "dtype": "float16", + "backend": "reference", + "max_abs_logit_error": 0.08143464674639045, + "mean_abs_logit_error": 0.002186594494940434, + "top1_agreement": 0.9987958073616028, + "validation_loss": 2.866903392619814, + "perplexity": 17.582487752382512, + "bits_per_byte": 4.136067307240351, + "greedy_identical_sequences": 6, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 62, + 64, + 62, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s42/best_checkpoint.pt", + "checkpoint_sha256": "657c3c616443474aa04c407c870c50a653446522eda677b15064a8639772501d", + "seed": 42, + "device": "mps", + "dtype": "bfloat16", + "backend": "auto", + "max_abs_logit_error": 0.5995575399191182, + "mean_abs_logit_error": 0.017492001637517338, + "top1_agreement": 0.9878602623939514, + "validation_loss": 2.8671219260375875, + "perplexity": 17.586330533396737, + "bits_per_byte": 4.136382584318441, + "greedy_identical_sequences": 4, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 57, + 64, + 62, + 64, + 62, + 62, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s42/best_checkpoint.pt", + "checkpoint_sha256": "657c3c616443474aa04c407c870c50a653446522eda677b15064a8639772501d", + "seed": 42, + "device": "mps", + "dtype": "bfloat16", + "backend": "sdpa", + "max_abs_logit_error": 0.5995575399191182, + "mean_abs_logit_error": 0.017492001637517338, + "top1_agreement": 0.9878602623939514, + "validation_loss": 2.8671219260375875, + "perplexity": 17.586330533396737, + "bits_per_byte": 4.136382584318441, + "greedy_identical_sequences": 4, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 57, + 64, + 62, + 64, + 62, + 62, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s42/best_checkpoint.pt", + "checkpoint_sha256": "657c3c616443474aa04c407c870c50a653446522eda677b15064a8639772501d", + "seed": 42, + "device": "mps", + "dtype": "bfloat16", + "backend": "reference", + "max_abs_logit_error": 0.5995575399191182, + "mean_abs_logit_error": 0.01753400831410292, + "top1_agreement": 0.9875480532646179, + "validation_loss": 2.8670912644611146, + "perplexity": 17.585791317044865, + "bits_per_byte": 4.1363383490141175, + "greedy_identical_sequences": 3, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 57, + 64, + 62, + 64, + 62, + 62, + 29 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s43/best_checkpoint.pt", + "checkpoint_sha256": "5ef10871bf409763d01ab42d66f6c013ced19da345baa3b8f1181dc76830a205", + "seed": 43, + "device": "cpu", + "dtype": "float32", + "backend": "auto", + "max_abs_logit_error": 1.8184877605076366e-05, + "mean_abs_logit_error": 7.865472163673178e-07, + "top1_agreement": 1.0, + "validation_loss": 2.943878722872867, + "perplexity": 18.989358111792892, + "bits_per_byte": 4.24711923446722, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s43/best_checkpoint.pt", + "checkpoint_sha256": "5ef10871bf409763d01ab42d66f6c013ced19da345baa3b8f1181dc76830a205", + "seed": 43, + "device": "cpu", + "dtype": "float32", + "backend": "sdpa", + "max_abs_logit_error": 1.6992784709568554e-05, + "mean_abs_logit_error": 7.871928740223922e-07, + "top1_agreement": 1.0, + "validation_loss": 2.9438790016144716, + "perplexity": 18.989363404917775, + "bits_per_byte": 4.247119636606351, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s43/best_checkpoint.pt", + "checkpoint_sha256": "5ef10871bf409763d01ab42d66f6c013ced19da345baa3b8f1181dc76830a205", + "seed": 43, + "device": "cpu", + "dtype": "float32", + "backend": "reference", + "max_abs_logit_error": 2.005820226136379e-05, + "mean_abs_logit_error": 7.879261010533779e-07, + "top1_agreement": 1.0, + "validation_loss": 2.9438790016144716, + "perplexity": 18.989363404917775, + "bits_per_byte": 4.247119636606351, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s43/best_checkpoint.pt", + "checkpoint_sha256": "5ef10871bf409763d01ab42d66f6c013ced19da345baa3b8f1181dc76830a205", + "seed": 43, + "device": "mps", + "dtype": "float32", + "backend": "auto", + "max_abs_logit_error": 1.6005086416637226e-05, + "mean_abs_logit_error": 7.810520410704364e-07, + "top1_agreement": 1.0, + "validation_loss": 2.9438790016144716, + "perplexity": 18.989363404917775, + "bits_per_byte": 4.247119636606351, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s43/best_checkpoint.pt", + "checkpoint_sha256": "5ef10871bf409763d01ab42d66f6c013ced19da345baa3b8f1181dc76830a205", + "seed": 43, + "device": "mps", + "dtype": "float32", + "backend": "sdpa", + "max_abs_logit_error": 1.6005086416637226e-05, + "mean_abs_logit_error": 7.810520410704364e-07, + "top1_agreement": 1.0, + "validation_loss": 2.9438790016144716, + "perplexity": 18.989363404917775, + "bits_per_byte": 4.247119636606351, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s43/best_checkpoint.pt", + "checkpoint_sha256": "5ef10871bf409763d01ab42d66f6c013ced19da345baa3b8f1181dc76830a205", + "seed": 43, + "device": "mps", + "dtype": "float32", + "backend": "reference", + "max_abs_logit_error": 1.7946459025974804e-05, + "mean_abs_logit_error": 7.804728863020003e-07, + "top1_agreement": 1.0, + "validation_loss": 2.9438790016144716, + "perplexity": 18.989363404917775, + "bits_per_byte": 4.247119636606351, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s43/best_checkpoint.pt", + "checkpoint_sha256": "5ef10871bf409763d01ab42d66f6c013ced19da345baa3b8f1181dc76830a205", + "seed": 43, + "device": "mps", + "dtype": "float16", + "backend": "auto", + "max_abs_logit_error": 0.04851824365975732, + "mean_abs_logit_error": 0.001783361968591014, + "top1_agreement": 0.9985104203224182, + "validation_loss": 2.9438541936116884, + "perplexity": 18.988892322580906, + "bits_per_byte": 4.247083846223761, + "greedy_identical_sequences": 6, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 25, + 64, + 64, + 0, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s43/best_checkpoint.pt", + "checkpoint_sha256": "5ef10871bf409763d01ab42d66f6c013ced19da345baa3b8f1181dc76830a205", + "seed": 43, + "device": "mps", + "dtype": "float16", + "backend": "sdpa", + "max_abs_logit_error": 0.04851824365975732, + "mean_abs_logit_error": 0.001783361968591014, + "top1_agreement": 0.9985104203224182, + "validation_loss": 2.9438541936116884, + "perplexity": 18.988892322580906, + "bits_per_byte": 4.247083846223761, + "greedy_identical_sequences": 6, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 25, + 64, + 64, + 0, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s43/best_checkpoint.pt", + "checkpoint_sha256": "5ef10871bf409763d01ab42d66f6c013ced19da345baa3b8f1181dc76830a205", + "seed": 43, + "device": "mps", + "dtype": "float16", + "backend": "reference", + "max_abs_logit_error": 0.04851824365975732, + "mean_abs_logit_error": 0.0017833137948247, + "top1_agreement": 0.9985460638999939, + "validation_loss": 2.943849455004415, + "perplexity": 18.98880234189083, + "bits_per_byte": 4.247077009858548, + "greedy_identical_sequences": 6, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 25, + 64, + 64, + 0, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s43/best_checkpoint.pt", + "checkpoint_sha256": "5ef10871bf409763d01ab42d66f6c013ced19da345baa3b8f1181dc76830a205", + "seed": 43, + "device": "mps", + "dtype": "bfloat16", + "backend": "auto", + "max_abs_logit_error": 0.4578779846110539, + "mean_abs_logit_error": 0.014313540390667307, + "top1_agreement": 0.9870842099189758, + "validation_loss": 2.943952310656403, + "perplexity": 18.990755547983596, + "bits_per_byte": 4.247225399197598, + "greedy_identical_sequences": 3, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 26, + 9, + 64, + 19, + 64, + 64, + 0, + 9 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s43/best_checkpoint.pt", + "checkpoint_sha256": "5ef10871bf409763d01ab42d66f6c013ced19da345baa3b8f1181dc76830a205", + "seed": 43, + "device": "mps", + "dtype": "bfloat16", + "backend": "sdpa", + "max_abs_logit_error": 0.4578779846110539, + "mean_abs_logit_error": 0.014313540390667307, + "top1_agreement": 0.9870842099189758, + "validation_loss": 2.943952310656403, + "perplexity": 18.990755547983596, + "bits_per_byte": 4.247225399197598, + "greedy_identical_sequences": 3, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 26, + 9, + 64, + 19, + 64, + 64, + 0, + 9 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s43/best_checkpoint.pt", + "checkpoint_sha256": "5ef10871bf409763d01ab42d66f6c013ced19da345baa3b8f1181dc76830a205", + "seed": 43, + "device": "mps", + "dtype": "bfloat16", + "backend": "reference", + "max_abs_logit_error": 0.42891336658846146, + "mean_abs_logit_error": 0.014283214894333058, + "top1_agreement": 0.9870128631591797, + "validation_loss": 2.943881231547306, + "perplexity": 18.98940574996995, + "bits_per_byte": 4.247122853719393, + "greedy_identical_sequences": 4, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 26, + 9, + 64, + 19, + 64, + 64, + 0, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s44/best_checkpoint.pt", + "checkpoint_sha256": "2f325db359957aaec71472bd94c9444df65e4ef3638aa70da13fbeb512aa68a4", + "seed": 44, + "device": "cpu", + "dtype": "float32", + "backend": "auto", + "max_abs_logit_error": 1.1338876370281525e-05, + "mean_abs_logit_error": 7.668247586234577e-07, + "top1_agreement": 1.0, + "validation_loss": 3.0477406320521627, + "perplexity": 21.067690943787593, + "bits_per_byte": 4.3969602957774505, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s44/best_checkpoint.pt", + "checkpoint_sha256": "2f325db359957aaec71472bd94c9444df65e4ef3638aa70da13fbeb512aa68a4", + "seed": 44, + "device": "cpu", + "dtype": "float32", + "backend": "sdpa", + "max_abs_logit_error": 9.265895823951098e-06, + "mean_abs_logit_error": 7.678921484076295e-07, + "top1_agreement": 1.0, + "validation_loss": 3.0477406320521627, + "perplexity": 21.067690943787593, + "bits_per_byte": 4.3969602957774505, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s44/best_checkpoint.pt", + "checkpoint_sha256": "2f325db359957aaec71472bd94c9444df65e4ef3638aa70da13fbeb512aa68a4", + "seed": 44, + "device": "cpu", + "dtype": "float32", + "backend": "reference", + "max_abs_logit_error": 9.204858675282424e-06, + "mean_abs_logit_error": 7.673975437292323e-07, + "top1_agreement": 1.0, + "validation_loss": 3.0477406320521627, + "perplexity": 21.067690943787593, + "bits_per_byte": 4.3969602957774505, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s44/best_checkpoint.pt", + "checkpoint_sha256": "2f325db359957aaec71472bd94c9444df65e4ef3638aa70da13fbeb512aa68a4", + "seed": 44, + "device": "mps", + "dtype": "float32", + "backend": "auto", + "max_abs_logit_error": 1.0966976371484805e-05, + "mean_abs_logit_error": 7.675022009917108e-07, + "top1_agreement": 1.0, + "validation_loss": 3.0477406320521627, + "perplexity": 21.067690943787593, + "bits_per_byte": 4.3969602957774505, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s44/best_checkpoint.pt", + "checkpoint_sha256": "2f325db359957aaec71472bd94c9444df65e4ef3638aa70da13fbeb512aa68a4", + "seed": 44, + "device": "mps", + "dtype": "float32", + "backend": "sdpa", + "max_abs_logit_error": 1.0966976371484805e-05, + "mean_abs_logit_error": 7.675022009917108e-07, + "top1_agreement": 1.0, + "validation_loss": 3.0477406320521627, + "perplexity": 21.067690943787593, + "bits_per_byte": 4.3969602957774505, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s44/best_checkpoint.pt", + "checkpoint_sha256": "2f325db359957aaec71472bd94c9444df65e4ef3638aa70da13fbeb512aa68a4", + "seed": 44, + "device": "mps", + "dtype": "float32", + "backend": "reference", + "max_abs_logit_error": 1.144381352968793e-05, + "mean_abs_logit_error": 7.676573937342432e-07, + "top1_agreement": 1.0, + "validation_loss": 3.0477406320521627, + "perplexity": 21.067690943787593, + "bits_per_byte": 4.3969602957774505, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s44/best_checkpoint.pt", + "checkpoint_sha256": "2f325db359957aaec71472bd94c9444df65e4ef3638aa70da13fbeb512aa68a4", + "seed": 44, + "device": "mps", + "dtype": "float16", + "backend": "auto", + "max_abs_logit_error": 0.03329051149986384, + "mean_abs_logit_error": 0.001564766364116153, + "top1_agreement": 0.9980376362800598, + "validation_loss": 3.0477300398711993, + "perplexity": 21.067467792174465, + "bits_per_byte": 4.396945014490502, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s44/best_checkpoint.pt", + "checkpoint_sha256": "2f325db359957aaec71472bd94c9444df65e4ef3638aa70da13fbeb512aa68a4", + "seed": 44, + "device": "mps", + "dtype": "float16", + "backend": "sdpa", + "max_abs_logit_error": 0.03329051149986384, + "mean_abs_logit_error": 0.001564766364116153, + "top1_agreement": 0.9980376362800598, + "validation_loss": 3.0477300398711993, + "perplexity": 21.067467792174465, + "bits_per_byte": 4.396945014490502, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s44/best_checkpoint.pt", + "checkpoint_sha256": "2f325db359957aaec71472bd94c9444df65e4ef3638aa70da13fbeb512aa68a4", + "seed": 44, + "device": "mps", + "dtype": "float16", + "backend": "reference", + "max_abs_logit_error": 0.03329051149986384, + "mean_abs_logit_error": 0.0015677312119954292, + "top1_agreement": 0.998019814491272, + "validation_loss": 3.047726137488739, + "perplexity": 21.067385579018087, + "bits_per_byte": 4.396939384542679, + "greedy_identical_sequences": 7, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 30, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s44/best_checkpoint.pt", + "checkpoint_sha256": "2f325db359957aaec71472bd94c9444df65e4ef3638aa70da13fbeb512aa68a4", + "seed": 44, + "device": "mps", + "dtype": "bfloat16", + "backend": "auto", + "max_abs_logit_error": 0.28835909893385026, + "mean_abs_logit_error": 0.0123753888803735, + "top1_agreement": 0.9842209815979004, + "validation_loss": 3.0479639040772093, + "perplexity": 21.07239529496313, + "bits_per_byte": 4.397282409220754, + "greedy_identical_sequences": 5, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 34, + 64, + 0, + 64, + 30, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s44/best_checkpoint.pt", + "checkpoint_sha256": "2f325db359957aaec71472bd94c9444df65e4ef3638aa70da13fbeb512aa68a4", + "seed": 44, + "device": "mps", + "dtype": "bfloat16", + "backend": "sdpa", + "max_abs_logit_error": 0.28835909893385026, + "mean_abs_logit_error": 0.0123753888803735, + "top1_agreement": 0.9842209815979004, + "validation_loss": 3.0479639040772093, + "perplexity": 21.07239529496313, + "bits_per_byte": 4.397282409220754, + "greedy_identical_sequences": 5, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 34, + 64, + 0, + 64, + 30, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s44/best_checkpoint.pt", + "checkpoint_sha256": "2f325db359957aaec71472bd94c9444df65e4ef3638aa70da13fbeb512aa68a4", + "seed": 44, + "device": "mps", + "dtype": "bfloat16", + "backend": "reference", + "max_abs_logit_error": 0.2749941325493195, + "mean_abs_logit_error": 0.012380473170180505, + "top1_agreement": 0.9841585755348206, + "validation_loss": 3.047937981108009, + "perplexity": 21.071849042989193, + "bits_per_byte": 4.397245010281644, + "greedy_identical_sequences": 6, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 0, + 64, + 30, + 64, + 64 + ] + } + ] +} diff --git a/experiments/records/attention_quality/20261022T034349Z_attention_quality_0502118.json b/experiments/records/attention_quality/20261022T034349Z_attention_quality_0502118.json new file mode 100644 index 0000000..2516994 --- /dev/null +++ b/experiments/records/attention_quality/20261022T034349Z_attention_quality_0502118.json @@ -0,0 +1,977 @@ +{ + "record_version": 1, + "created_utc": "20261022T034349Z", + "experiment": "attention_quality", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "cpu,mps", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "0502118a1a0af31c8480f6b499d9347916330a79", + "branch": "experiments/attention-kv-audit", + "dirty": false + }, + "command": "python scripts/compare_attention_quality.py artifacts/runs/science-v2-s42/best_checkpoint.pt artifacts/runs/science-v2-s43/best_checkpoint.pt artifacts/runs/science-v2-s44/best_checkpoint.pt" + }, + "settings": { + "validation_text": "/Users/anvit/Downloads/Projects/MiniLM-Lab/data/science/validation.txt", + "validation_sha256": "e25d4e45706bbf4251129dbb079f3487660c453cf8b34407901350538b23e2e7", + "validation_targets": 112111, + "oracle": "cpu float64, reference backend", + "loss_precision": "logits upcast to float32 before cross-entropy", + "greedy": { + "prompts": 8, + "prompt_tokens": 32, + "generated_tokens": 64, + "cached": true + }, + "skipped_targets": [] + }, + "results": [ + { + "checkpoint": "artifacts/runs/science-v2-s42/best_checkpoint.pt", + "checkpoint_sha256": "657c3c616443474aa04c407c870c50a653446522eda677b15064a8639772501d", + "seed": 42, + "device": "cpu", + "dtype": "float32", + "backend": "auto", + "max_abs_logit_error": 3.654949019749054e-05, + "mean_abs_logit_error": 1.0375581694966199e-06, + "top1_agreement": 1.0, + "validation_loss": 2.8669173297000294, + "perplexity": 17.58273280263234, + "bits_per_byte": 4.1360874141968615, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s42/best_checkpoint.pt", + "checkpoint_sha256": "657c3c616443474aa04c407c870c50a653446522eda677b15064a8639772501d", + "seed": 42, + "device": "cpu", + "dtype": "float32", + "backend": "sdpa", + "max_abs_logit_error": 3.347584891333355e-05, + "mean_abs_logit_error": 1.0386070915819935e-06, + "top1_agreement": 1.0, + "validation_loss": 2.8669173297000294, + "perplexity": 17.58273280263234, + "bits_per_byte": 4.1360874141968615, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s42/best_checkpoint.pt", + "checkpoint_sha256": "657c3c616443474aa04c407c870c50a653446522eda677b15064a8639772501d", + "seed": 42, + "device": "cpu", + "dtype": "float32", + "backend": "reference", + "max_abs_logit_error": 3.657529044165386e-05, + "mean_abs_logit_error": 1.0373159225101448e-06, + "top1_agreement": 1.0, + "validation_loss": 2.8669173297000294, + "perplexity": 17.58273280263234, + "bits_per_byte": 4.1360874141968615, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s42/best_checkpoint.pt", + "checkpoint_sha256": "657c3c616443474aa04c407c870c50a653446522eda677b15064a8639772501d", + "seed": 42, + "device": "mps", + "dtype": "float32", + "backend": "auto", + "max_abs_logit_error": 3.267033399900754e-05, + "mean_abs_logit_error": 1.0308867831769693e-06, + "top1_agreement": 1.0, + "validation_loss": 2.8669173297000294, + "perplexity": 17.58273280263234, + "bits_per_byte": 4.1360874141968615, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s42/best_checkpoint.pt", + "checkpoint_sha256": "657c3c616443474aa04c407c870c50a653446522eda677b15064a8639772501d", + "seed": 42, + "device": "mps", + "dtype": "float32", + "backend": "sdpa", + "max_abs_logit_error": 3.267033399900754e-05, + "mean_abs_logit_error": 1.0308867831769693e-06, + "top1_agreement": 1.0, + "validation_loss": 2.8669173297000294, + "perplexity": 17.58273280263234, + "bits_per_byte": 4.1360874141968615, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s42/best_checkpoint.pt", + "checkpoint_sha256": "657c3c616443474aa04c407c870c50a653446522eda677b15064a8639772501d", + "seed": 42, + "device": "mps", + "dtype": "float32", + "backend": "reference", + "max_abs_logit_error": 4.110524344458355e-05, + "mean_abs_logit_error": 1.0309519384724672e-06, + "top1_agreement": 1.0, + "validation_loss": 2.8669173297000294, + "perplexity": 17.58273280263234, + "bits_per_byte": 4.1360874141968615, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s42/best_checkpoint.pt", + "checkpoint_sha256": "657c3c616443474aa04c407c870c50a653446522eda677b15064a8639772501d", + "seed": 42, + "device": "mps", + "dtype": "float16", + "backend": "auto", + "max_abs_logit_error": 0.07358595400162127, + "mean_abs_logit_error": 0.002181009015840955, + "top1_agreement": 0.9987958073616028, + "validation_loss": 2.866908409968692, + "perplexity": 17.582575970079013, + "bits_per_byte": 4.1360745457446955, + "greedy_identical_sequences": 6, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 62, + 64, + 62, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s42/best_checkpoint.pt", + "checkpoint_sha256": "657c3c616443474aa04c407c870c50a653446522eda677b15064a8639772501d", + "seed": 42, + "device": "mps", + "dtype": "float16", + "backend": "sdpa", + "max_abs_logit_error": 0.07358595400162127, + "mean_abs_logit_error": 0.002181009015840955, + "top1_agreement": 0.9987958073616028, + "validation_loss": 2.866908409968692, + "perplexity": 17.582575970079013, + "bits_per_byte": 4.1360745457446955, + "greedy_identical_sequences": 6, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 62, + 64, + 62, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s42/best_checkpoint.pt", + "checkpoint_sha256": "657c3c616443474aa04c407c870c50a653446522eda677b15064a8639772501d", + "seed": 42, + "device": "mps", + "dtype": "float16", + "backend": "reference", + "max_abs_logit_error": 0.08143464674639045, + "mean_abs_logit_error": 0.002186594494940434, + "top1_agreement": 0.9987958073616028, + "validation_loss": 2.866903392619814, + "perplexity": 17.582487752382512, + "bits_per_byte": 4.136067307240351, + "greedy_identical_sequences": 6, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 62, + 64, + 62, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s42/best_checkpoint.pt", + "checkpoint_sha256": "657c3c616443474aa04c407c870c50a653446522eda677b15064a8639772501d", + "seed": 42, + "device": "mps", + "dtype": "bfloat16", + "backend": "auto", + "max_abs_logit_error": 0.5995575399191182, + "mean_abs_logit_error": 0.017492001637517338, + "top1_agreement": 0.9878602623939514, + "validation_loss": 2.8671219260375875, + "perplexity": 17.586330533396737, + "bits_per_byte": 4.136382584318441, + "greedy_identical_sequences": 4, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 57, + 64, + 62, + 64, + 62, + 62, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s42/best_checkpoint.pt", + "checkpoint_sha256": "657c3c616443474aa04c407c870c50a653446522eda677b15064a8639772501d", + "seed": 42, + "device": "mps", + "dtype": "bfloat16", + "backend": "sdpa", + "max_abs_logit_error": 0.5995575399191182, + "mean_abs_logit_error": 0.017492001637517338, + "top1_agreement": 0.9878602623939514, + "validation_loss": 2.8671219260375875, + "perplexity": 17.586330533396737, + "bits_per_byte": 4.136382584318441, + "greedy_identical_sequences": 4, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 57, + 64, + 62, + 64, + 62, + 62, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s42/best_checkpoint.pt", + "checkpoint_sha256": "657c3c616443474aa04c407c870c50a653446522eda677b15064a8639772501d", + "seed": 42, + "device": "mps", + "dtype": "bfloat16", + "backend": "reference", + "max_abs_logit_error": 0.5995575399191182, + "mean_abs_logit_error": 0.01753400831410292, + "top1_agreement": 0.9875480532646179, + "validation_loss": 2.8670912644611146, + "perplexity": 17.585791317044865, + "bits_per_byte": 4.1363383490141175, + "greedy_identical_sequences": 3, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 57, + 64, + 62, + 64, + 62, + 62, + 29 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s43/best_checkpoint.pt", + "checkpoint_sha256": "5ef10871bf409763d01ab42d66f6c013ced19da345baa3b8f1181dc76830a205", + "seed": 43, + "device": "cpu", + "dtype": "float32", + "backend": "auto", + "max_abs_logit_error": 1.8184877605076366e-05, + "mean_abs_logit_error": 7.865472163673178e-07, + "top1_agreement": 1.0, + "validation_loss": 2.943878722872867, + "perplexity": 18.989358111792892, + "bits_per_byte": 4.24711923446722, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s43/best_checkpoint.pt", + "checkpoint_sha256": "5ef10871bf409763d01ab42d66f6c013ced19da345baa3b8f1181dc76830a205", + "seed": 43, + "device": "cpu", + "dtype": "float32", + "backend": "sdpa", + "max_abs_logit_error": 1.6992784709568554e-05, + "mean_abs_logit_error": 7.871928740223922e-07, + "top1_agreement": 1.0, + "validation_loss": 2.9438790016144716, + "perplexity": 18.989363404917775, + "bits_per_byte": 4.247119636606351, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s43/best_checkpoint.pt", + "checkpoint_sha256": "5ef10871bf409763d01ab42d66f6c013ced19da345baa3b8f1181dc76830a205", + "seed": 43, + "device": "cpu", + "dtype": "float32", + "backend": "reference", + "max_abs_logit_error": 2.005820226136379e-05, + "mean_abs_logit_error": 7.879261010533779e-07, + "top1_agreement": 1.0, + "validation_loss": 2.9438790016144716, + "perplexity": 18.989363404917775, + "bits_per_byte": 4.247119636606351, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s43/best_checkpoint.pt", + "checkpoint_sha256": "5ef10871bf409763d01ab42d66f6c013ced19da345baa3b8f1181dc76830a205", + "seed": 43, + "device": "mps", + "dtype": "float32", + "backend": "auto", + "max_abs_logit_error": 1.6005086416637226e-05, + "mean_abs_logit_error": 7.810520410704364e-07, + "top1_agreement": 1.0, + "validation_loss": 2.9438790016144716, + "perplexity": 18.989363404917775, + "bits_per_byte": 4.247119636606351, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s43/best_checkpoint.pt", + "checkpoint_sha256": "5ef10871bf409763d01ab42d66f6c013ced19da345baa3b8f1181dc76830a205", + "seed": 43, + "device": "mps", + "dtype": "float32", + "backend": "sdpa", + "max_abs_logit_error": 1.6005086416637226e-05, + "mean_abs_logit_error": 7.810520410704364e-07, + "top1_agreement": 1.0, + "validation_loss": 2.9438790016144716, + "perplexity": 18.989363404917775, + "bits_per_byte": 4.247119636606351, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s43/best_checkpoint.pt", + "checkpoint_sha256": "5ef10871bf409763d01ab42d66f6c013ced19da345baa3b8f1181dc76830a205", + "seed": 43, + "device": "mps", + "dtype": "float32", + "backend": "reference", + "max_abs_logit_error": 1.7946459025974804e-05, + "mean_abs_logit_error": 7.804728863020003e-07, + "top1_agreement": 1.0, + "validation_loss": 2.9438790016144716, + "perplexity": 18.989363404917775, + "bits_per_byte": 4.247119636606351, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s43/best_checkpoint.pt", + "checkpoint_sha256": "5ef10871bf409763d01ab42d66f6c013ced19da345baa3b8f1181dc76830a205", + "seed": 43, + "device": "mps", + "dtype": "float16", + "backend": "auto", + "max_abs_logit_error": 0.04851824365975732, + "mean_abs_logit_error": 0.001783361968591014, + "top1_agreement": 0.9985104203224182, + "validation_loss": 2.9438541936116884, + "perplexity": 18.988892322580906, + "bits_per_byte": 4.247083846223761, + "greedy_identical_sequences": 6, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 25, + 64, + 64, + 0, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s43/best_checkpoint.pt", + "checkpoint_sha256": "5ef10871bf409763d01ab42d66f6c013ced19da345baa3b8f1181dc76830a205", + "seed": 43, + "device": "mps", + "dtype": "float16", + "backend": "sdpa", + "max_abs_logit_error": 0.04851824365975732, + "mean_abs_logit_error": 0.001783361968591014, + "top1_agreement": 0.9985104203224182, + "validation_loss": 2.9438541936116884, + "perplexity": 18.988892322580906, + "bits_per_byte": 4.247083846223761, + "greedy_identical_sequences": 6, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 25, + 64, + 64, + 0, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s43/best_checkpoint.pt", + "checkpoint_sha256": "5ef10871bf409763d01ab42d66f6c013ced19da345baa3b8f1181dc76830a205", + "seed": 43, + "device": "mps", + "dtype": "float16", + "backend": "reference", + "max_abs_logit_error": 0.04851824365975732, + "mean_abs_logit_error": 0.0017833137948247, + "top1_agreement": 0.9985460638999939, + "validation_loss": 2.943849455004415, + "perplexity": 18.98880234189083, + "bits_per_byte": 4.247077009858548, + "greedy_identical_sequences": 6, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 25, + 64, + 64, + 0, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s43/best_checkpoint.pt", + "checkpoint_sha256": "5ef10871bf409763d01ab42d66f6c013ced19da345baa3b8f1181dc76830a205", + "seed": 43, + "device": "mps", + "dtype": "bfloat16", + "backend": "auto", + "max_abs_logit_error": 0.4578779846110539, + "mean_abs_logit_error": 0.014313540390667307, + "top1_agreement": 0.9870842099189758, + "validation_loss": 2.943952310656403, + "perplexity": 18.990755547983596, + "bits_per_byte": 4.247225399197598, + "greedy_identical_sequences": 3, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 26, + 9, + 64, + 19, + 64, + 64, + 0, + 9 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s43/best_checkpoint.pt", + "checkpoint_sha256": "5ef10871bf409763d01ab42d66f6c013ced19da345baa3b8f1181dc76830a205", + "seed": 43, + "device": "mps", + "dtype": "bfloat16", + "backend": "sdpa", + "max_abs_logit_error": 0.4578779846110539, + "mean_abs_logit_error": 0.014313540390667307, + "top1_agreement": 0.9870842099189758, + "validation_loss": 2.943952310656403, + "perplexity": 18.990755547983596, + "bits_per_byte": 4.247225399197598, + "greedy_identical_sequences": 3, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 26, + 9, + 64, + 19, + 64, + 64, + 0, + 9 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s43/best_checkpoint.pt", + "checkpoint_sha256": "5ef10871bf409763d01ab42d66f6c013ced19da345baa3b8f1181dc76830a205", + "seed": 43, + "device": "mps", + "dtype": "bfloat16", + "backend": "reference", + "max_abs_logit_error": 0.42891336658846146, + "mean_abs_logit_error": 0.014283214894333058, + "top1_agreement": 0.9870128631591797, + "validation_loss": 2.943881231547306, + "perplexity": 18.98940574996995, + "bits_per_byte": 4.247122853719393, + "greedy_identical_sequences": 4, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 26, + 9, + 64, + 19, + 64, + 64, + 0, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s44/best_checkpoint.pt", + "checkpoint_sha256": "2f325db359957aaec71472bd94c9444df65e4ef3638aa70da13fbeb512aa68a4", + "seed": 44, + "device": "cpu", + "dtype": "float32", + "backend": "auto", + "max_abs_logit_error": 1.1338876370281525e-05, + "mean_abs_logit_error": 7.668247586234577e-07, + "top1_agreement": 1.0, + "validation_loss": 3.0477406320521627, + "perplexity": 21.067690943787593, + "bits_per_byte": 4.3969602957774505, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s44/best_checkpoint.pt", + "checkpoint_sha256": "2f325db359957aaec71472bd94c9444df65e4ef3638aa70da13fbeb512aa68a4", + "seed": 44, + "device": "cpu", + "dtype": "float32", + "backend": "sdpa", + "max_abs_logit_error": 9.265895823951098e-06, + "mean_abs_logit_error": 7.678921484076295e-07, + "top1_agreement": 1.0, + "validation_loss": 3.0477406320521627, + "perplexity": 21.067690943787593, + "bits_per_byte": 4.3969602957774505, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s44/best_checkpoint.pt", + "checkpoint_sha256": "2f325db359957aaec71472bd94c9444df65e4ef3638aa70da13fbeb512aa68a4", + "seed": 44, + "device": "cpu", + "dtype": "float32", + "backend": "reference", + "max_abs_logit_error": 9.204858675282424e-06, + "mean_abs_logit_error": 7.673975437292323e-07, + "top1_agreement": 1.0, + "validation_loss": 3.0477406320521627, + "perplexity": 21.067690943787593, + "bits_per_byte": 4.3969602957774505, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s44/best_checkpoint.pt", + "checkpoint_sha256": "2f325db359957aaec71472bd94c9444df65e4ef3638aa70da13fbeb512aa68a4", + "seed": 44, + "device": "mps", + "dtype": "float32", + "backend": "auto", + "max_abs_logit_error": 1.0966976371484805e-05, + "mean_abs_logit_error": 7.675022009917108e-07, + "top1_agreement": 1.0, + "validation_loss": 3.0477406320521627, + "perplexity": 21.067690943787593, + "bits_per_byte": 4.3969602957774505, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s44/best_checkpoint.pt", + "checkpoint_sha256": "2f325db359957aaec71472bd94c9444df65e4ef3638aa70da13fbeb512aa68a4", + "seed": 44, + "device": "mps", + "dtype": "float32", + "backend": "sdpa", + "max_abs_logit_error": 1.0966976371484805e-05, + "mean_abs_logit_error": 7.675022009917108e-07, + "top1_agreement": 1.0, + "validation_loss": 3.0477406320521627, + "perplexity": 21.067690943787593, + "bits_per_byte": 4.3969602957774505, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s44/best_checkpoint.pt", + "checkpoint_sha256": "2f325db359957aaec71472bd94c9444df65e4ef3638aa70da13fbeb512aa68a4", + "seed": 44, + "device": "mps", + "dtype": "float32", + "backend": "reference", + "max_abs_logit_error": 1.144381352968793e-05, + "mean_abs_logit_error": 7.676573937342432e-07, + "top1_agreement": 1.0, + "validation_loss": 3.0477406320521627, + "perplexity": 21.067690943787593, + "bits_per_byte": 4.3969602957774505, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s44/best_checkpoint.pt", + "checkpoint_sha256": "2f325db359957aaec71472bd94c9444df65e4ef3638aa70da13fbeb512aa68a4", + "seed": 44, + "device": "mps", + "dtype": "float16", + "backend": "auto", + "max_abs_logit_error": 0.03329051149986384, + "mean_abs_logit_error": 0.001564766364116153, + "top1_agreement": 0.9980376362800598, + "validation_loss": 3.0477300398711993, + "perplexity": 21.067467792174465, + "bits_per_byte": 4.396945014490502, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s44/best_checkpoint.pt", + "checkpoint_sha256": "2f325db359957aaec71472bd94c9444df65e4ef3638aa70da13fbeb512aa68a4", + "seed": 44, + "device": "mps", + "dtype": "float16", + "backend": "sdpa", + "max_abs_logit_error": 0.03329051149986384, + "mean_abs_logit_error": 0.001564766364116153, + "top1_agreement": 0.9980376362800598, + "validation_loss": 3.0477300398711993, + "perplexity": 21.067467792174465, + "bits_per_byte": 4.396945014490502, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s44/best_checkpoint.pt", + "checkpoint_sha256": "2f325db359957aaec71472bd94c9444df65e4ef3638aa70da13fbeb512aa68a4", + "seed": 44, + "device": "mps", + "dtype": "float16", + "backend": "reference", + "max_abs_logit_error": 0.03329051149986384, + "mean_abs_logit_error": 0.0015677312119954292, + "top1_agreement": 0.998019814491272, + "validation_loss": 3.047726137488739, + "perplexity": 21.067385579018087, + "bits_per_byte": 4.396939384542679, + "greedy_identical_sequences": 7, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 30, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s44/best_checkpoint.pt", + "checkpoint_sha256": "2f325db359957aaec71472bd94c9444df65e4ef3638aa70da13fbeb512aa68a4", + "seed": 44, + "device": "mps", + "dtype": "bfloat16", + "backend": "auto", + "max_abs_logit_error": 0.28835909893385026, + "mean_abs_logit_error": 0.0123753888803735, + "top1_agreement": 0.9842209815979004, + "validation_loss": 3.0479639040772093, + "perplexity": 21.07239529496313, + "bits_per_byte": 4.397282409220754, + "greedy_identical_sequences": 5, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 34, + 64, + 0, + 64, + 30, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s44/best_checkpoint.pt", + "checkpoint_sha256": "2f325db359957aaec71472bd94c9444df65e4ef3638aa70da13fbeb512aa68a4", + "seed": 44, + "device": "mps", + "dtype": "bfloat16", + "backend": "sdpa", + "max_abs_logit_error": 0.28835909893385026, + "mean_abs_logit_error": 0.0123753888803735, + "top1_agreement": 0.9842209815979004, + "validation_loss": 3.0479639040772093, + "perplexity": 21.07239529496313, + "bits_per_byte": 4.397282409220754, + "greedy_identical_sequences": 5, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 34, + 64, + 0, + 64, + 30, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/science-v2-s44/best_checkpoint.pt", + "checkpoint_sha256": "2f325db359957aaec71472bd94c9444df65e4ef3638aa70da13fbeb512aa68a4", + "seed": 44, + "device": "mps", + "dtype": "bfloat16", + "backend": "reference", + "max_abs_logit_error": 0.2749941325493195, + "mean_abs_logit_error": 0.012380473170180505, + "top1_agreement": 0.9841585755348206, + "validation_loss": 3.047937981108009, + "perplexity": 21.071849042989193, + "bits_per_byte": 4.397245010281644, + "greedy_identical_sequences": 6, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 0, + 64, + 30, + 64, + 64 + ] + } + ] +} diff --git a/experiments/records/attention_quality/20261022T041752Z_attention_quality_48e95f8.json b/experiments/records/attention_quality/20261022T041752Z_attention_quality_48e95f8.json new file mode 100644 index 0000000..751cd6e --- /dev/null +++ b/experiments/records/attention_quality/20261022T041752Z_attention_quality_48e95f8.json @@ -0,0 +1,977 @@ +{ + "record_version": 1, + "created_utc": "20261022T041752Z", + "experiment": "attention_quality", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "cpu,mps", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "48e95f8ea09889e6875fb6129026662f8a699cdb", + "branch": "experiments/sdpa-training-v2", + "dirty": false + }, + "command": "python scripts/compare_attention_quality.py artifacts/runs/sdpa-v1/auto-s42/best_checkpoint.pt artifacts/runs/sdpa-v1/auto-s43/best_checkpoint.pt artifacts/runs/sdpa-v1/auto-s44/best_checkpoint.pt" + }, + "settings": { + "validation_text": "/Users/anvit/Downloads/Projects/MiniLM-Lab-dev/data/science/validation.txt", + "validation_sha256": "e25d4e45706bbf4251129dbb079f3487660c453cf8b34407901350538b23e2e7", + "validation_targets": 112111, + "oracle": "cpu float64, reference backend", + "loss_precision": "logits upcast to float32 before cross-entropy", + "greedy": { + "prompts": 8, + "prompt_tokens": 32, + "generated_tokens": 64, + "cached": true + }, + "skipped_targets": [] + }, + "results": [ + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s42/best_checkpoint.pt", + "checkpoint_sha256": "23f0aa813704e0cfc02d0780386fe76d1ee26f70aaa674ce83a849a85fb93546", + "seed": 42, + "device": "cpu", + "dtype": "float32", + "backend": "auto", + "max_abs_logit_error": 3.603674140517654e-05, + "mean_abs_logit_error": 1.035222313048979e-06, + "top1_agreement": 1.0, + "validation_loss": 2.8669173297000294, + "perplexity": 17.58273280263234, + "bits_per_byte": 4.1360874141968615, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s42/best_checkpoint.pt", + "checkpoint_sha256": "23f0aa813704e0cfc02d0780386fe76d1ee26f70aaa674ce83a849a85fb93546", + "seed": 42, + "device": "cpu", + "dtype": "float32", + "backend": "sdpa", + "max_abs_logit_error": 3.400610464021625e-05, + "mean_abs_logit_error": 1.036835568198049e-06, + "top1_agreement": 1.0, + "validation_loss": 2.8669173297000294, + "perplexity": 17.58273280263234, + "bits_per_byte": 4.1360874141968615, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s42/best_checkpoint.pt", + "checkpoint_sha256": "23f0aa813704e0cfc02d0780386fe76d1ee26f70aaa674ce83a849a85fb93546", + "seed": 42, + "device": "cpu", + "dtype": "float32", + "backend": "reference", + "max_abs_logit_error": 3.710554616853656e-05, + "mean_abs_logit_error": 1.0357951049534412e-06, + "top1_agreement": 1.0, + "validation_loss": 2.8669173297000294, + "perplexity": 17.58273280263234, + "bits_per_byte": 4.1360874141968615, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s42/best_checkpoint.pt", + "checkpoint_sha256": "23f0aa813704e0cfc02d0780386fe76d1ee26f70aaa674ce83a849a85fb93546", + "seed": 42, + "device": "mps", + "dtype": "float32", + "backend": "auto", + "max_abs_logit_error": 3.311288880691876e-05, + "mean_abs_logit_error": 1.0283335180344067e-06, + "top1_agreement": 1.0, + "validation_loss": 2.8669173297000294, + "perplexity": 17.58273280263234, + "bits_per_byte": 4.1360874141968615, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s42/best_checkpoint.pt", + "checkpoint_sha256": "23f0aa813704e0cfc02d0780386fe76d1ee26f70aaa674ce83a849a85fb93546", + "seed": 42, + "device": "mps", + "dtype": "float32", + "backend": "sdpa", + "max_abs_logit_error": 3.311288880691876e-05, + "mean_abs_logit_error": 1.0283335180344067e-06, + "top1_agreement": 1.0, + "validation_loss": 2.8669173297000294, + "perplexity": 17.58273280263234, + "bits_per_byte": 4.1360874141968615, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s42/best_checkpoint.pt", + "checkpoint_sha256": "23f0aa813704e0cfc02d0780386fe76d1ee26f70aaa674ce83a849a85fb93546", + "seed": 42, + "device": "mps", + "dtype": "float32", + "backend": "reference", + "max_abs_logit_error": 4.163549917146625e-05, + "mean_abs_logit_error": 1.0287371241117412e-06, + "top1_agreement": 1.0, + "validation_loss": 2.8669173297000294, + "perplexity": 17.58273280263234, + "bits_per_byte": 4.1360874141968615, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s42/best_checkpoint.pt", + "checkpoint_sha256": "23f0aa813704e0cfc02d0780386fe76d1ee26f70aaa674ce83a849a85fb93546", + "seed": 42, + "device": "mps", + "dtype": "float16", + "backend": "auto", + "max_abs_logit_error": 0.07358631104276281, + "mean_abs_logit_error": 0.002181010033254489, + "top1_agreement": 0.9987958073616028, + "validation_loss": 2.866908409968692, + "perplexity": 17.582575970079013, + "bits_per_byte": 4.1360745457446955, + "greedy_identical_sequences": 6, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 62, + 64, + 62, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s42/best_checkpoint.pt", + "checkpoint_sha256": "23f0aa813704e0cfc02d0780386fe76d1ee26f70aaa674ce83a849a85fb93546", + "seed": 42, + "device": "mps", + "dtype": "float16", + "backend": "sdpa", + "max_abs_logit_error": 0.07358631104276281, + "mean_abs_logit_error": 0.002181010033254489, + "top1_agreement": 0.9987958073616028, + "validation_loss": 2.866908409968692, + "perplexity": 17.582575970079013, + "bits_per_byte": 4.1360745457446955, + "greedy_identical_sequences": 6, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 62, + 64, + 62, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s42/best_checkpoint.pt", + "checkpoint_sha256": "23f0aa813704e0cfc02d0780386fe76d1ee26f70aaa674ce83a849a85fb93546", + "seed": 42, + "device": "mps", + "dtype": "float16", + "backend": "reference", + "max_abs_logit_error": 0.08143548359960029, + "mean_abs_logit_error": 0.0021865955885589784, + "top1_agreement": 0.9987958073616028, + "validation_loss": 2.866903392619814, + "perplexity": 17.582487752382512, + "bits_per_byte": 4.136067307240351, + "greedy_identical_sequences": 6, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 62, + 64, + 62, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s42/best_checkpoint.pt", + "checkpoint_sha256": "23f0aa813704e0cfc02d0780386fe76d1ee26f70aaa674ce83a849a85fb93546", + "seed": 42, + "device": "mps", + "dtype": "bfloat16", + "backend": "auto", + "max_abs_logit_error": 0.5995577751900609, + "mean_abs_logit_error": 0.017492003908500038, + "top1_agreement": 0.9878602623939514, + "validation_loss": 2.8671219260375875, + "perplexity": 17.586330533396737, + "bits_per_byte": 4.136382584318441, + "greedy_identical_sequences": 4, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 57, + 64, + 62, + 64, + 62, + 62, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s42/best_checkpoint.pt", + "checkpoint_sha256": "23f0aa813704e0cfc02d0780386fe76d1ee26f70aaa674ce83a849a85fb93546", + "seed": 42, + "device": "mps", + "dtype": "bfloat16", + "backend": "sdpa", + "max_abs_logit_error": 0.5995577751900609, + "mean_abs_logit_error": 0.017492003908500038, + "top1_agreement": 0.9878602623939514, + "validation_loss": 2.8671219260375875, + "perplexity": 17.586330533396737, + "bits_per_byte": 4.136382584318441, + "greedy_identical_sequences": 4, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 57, + 64, + 62, + 64, + 62, + 62, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s42/best_checkpoint.pt", + "checkpoint_sha256": "23f0aa813704e0cfc02d0780386fe76d1ee26f70aaa674ce83a849a85fb93546", + "seed": 42, + "device": "mps", + "dtype": "bfloat16", + "backend": "reference", + "max_abs_logit_error": 0.5995577751900609, + "mean_abs_logit_error": 0.017534010877599676, + "top1_agreement": 0.9875480532646179, + "validation_loss": 2.8670912644611146, + "perplexity": 17.585791317044865, + "bits_per_byte": 4.1363383490141175, + "greedy_identical_sequences": 3, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 57, + 64, + 62, + 64, + 62, + 62, + 29 + ] + }, + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s43/best_checkpoint.pt", + "checkpoint_sha256": "e2285dfc79e7aa1d430cac7720bc5997139210fef876055c23da375aab24b04c", + "seed": 43, + "device": "cpu", + "dtype": "float32", + "backend": "auto", + "max_abs_logit_error": 1.8232674148244143e-05, + "mean_abs_logit_error": 7.85169952489562e-07, + "top1_agreement": 1.0, + "validation_loss": 2.943878722872867, + "perplexity": 18.989358111792892, + "bits_per_byte": 4.24711923446722, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s43/best_checkpoint.pt", + "checkpoint_sha256": "e2285dfc79e7aa1d430cac7720bc5997139210fef876055c23da375aab24b04c", + "seed": 43, + "device": "cpu", + "dtype": "float32", + "backend": "sdpa", + "max_abs_logit_error": 1.704058125273633e-05, + "mean_abs_logit_error": 7.863243691076104e-07, + "top1_agreement": 1.0, + "validation_loss": 2.9438790016144716, + "perplexity": 18.989363404917775, + "bits_per_byte": 4.247119636606351, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s43/best_checkpoint.pt", + "checkpoint_sha256": "e2285dfc79e7aa1d430cac7720bc5997139210fef876055c23da375aab24b04c", + "seed": 43, + "device": "cpu", + "dtype": "float32", + "backend": "reference", + "max_abs_logit_error": 2.003540696282613e-05, + "mean_abs_logit_error": 7.867003130218524e-07, + "top1_agreement": 1.0, + "validation_loss": 2.9438790016144716, + "perplexity": 18.989363404917775, + "bits_per_byte": 4.247119636606351, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s43/best_checkpoint.pt", + "checkpoint_sha256": "e2285dfc79e7aa1d430cac7720bc5997139210fef876055c23da375aab24b04c", + "seed": 43, + "device": "mps", + "dtype": "float32", + "backend": "auto", + "max_abs_logit_error": 1.5982291118099567e-05, + "mean_abs_logit_error": 7.797895600914613e-07, + "top1_agreement": 1.0, + "validation_loss": 2.9438790016144716, + "perplexity": 18.989363404917775, + "bits_per_byte": 4.247119636606351, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s43/best_checkpoint.pt", + "checkpoint_sha256": "e2285dfc79e7aa1d430cac7720bc5997139210fef876055c23da375aab24b04c", + "seed": 43, + "device": "mps", + "dtype": "float32", + "backend": "sdpa", + "max_abs_logit_error": 1.5982291118099567e-05, + "mean_abs_logit_error": 7.797895600914613e-07, + "top1_agreement": 1.0, + "validation_loss": 2.9438790016144716, + "perplexity": 18.989363404917775, + "bits_per_byte": 4.247119636606351, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s43/best_checkpoint.pt", + "checkpoint_sha256": "e2285dfc79e7aa1d430cac7720bc5997139210fef876055c23da375aab24b04c", + "seed": 43, + "device": "mps", + "dtype": "float32", + "backend": "reference", + "max_abs_logit_error": 1.8390807615453753e-05, + "mean_abs_logit_error": 7.795538227143339e-07, + "top1_agreement": 1.0, + "validation_loss": 2.9438790016144716, + "perplexity": 18.989363404917775, + "bits_per_byte": 4.247119636606351, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s43/best_checkpoint.pt", + "checkpoint_sha256": "e2285dfc79e7aa1d430cac7720bc5997139210fef876055c23da375aab24b04c", + "seed": 43, + "device": "mps", + "dtype": "float16", + "backend": "auto", + "max_abs_logit_error": 0.04851746877870289, + "mean_abs_logit_error": 0.0017833628632929295, + "top1_agreement": 0.9985104203224182, + "validation_loss": 2.9438541936116884, + "perplexity": 18.988892322580906, + "bits_per_byte": 4.247083846223761, + "greedy_identical_sequences": 6, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 25, + 64, + 64, + 0, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s43/best_checkpoint.pt", + "checkpoint_sha256": "e2285dfc79e7aa1d430cac7720bc5997139210fef876055c23da375aab24b04c", + "seed": 43, + "device": "mps", + "dtype": "float16", + "backend": "sdpa", + "max_abs_logit_error": 0.04851746877870289, + "mean_abs_logit_error": 0.0017833628632929295, + "top1_agreement": 0.9985104203224182, + "validation_loss": 2.9438541936116884, + "perplexity": 18.988892322580906, + "bits_per_byte": 4.247083846223761, + "greedy_identical_sequences": 6, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 25, + 64, + 64, + 0, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s43/best_checkpoint.pt", + "checkpoint_sha256": "e2285dfc79e7aa1d430cac7720bc5997139210fef876055c23da375aab24b04c", + "seed": 43, + "device": "mps", + "dtype": "float16", + "backend": "reference", + "max_abs_logit_error": 0.04851746877870289, + "mean_abs_logit_error": 0.0017833146459032415, + "top1_agreement": 0.9985460638999939, + "validation_loss": 2.943849455004415, + "perplexity": 18.98880234189083, + "bits_per_byte": 4.247077009858548, + "greedy_identical_sequences": 6, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 25, + 64, + 64, + 0, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s43/best_checkpoint.pt", + "checkpoint_sha256": "e2285dfc79e7aa1d430cac7720bc5997139210fef876055c23da375aab24b04c", + "seed": 43, + "device": "mps", + "dtype": "bfloat16", + "backend": "auto", + "max_abs_logit_error": 0.4578789000391552, + "mean_abs_logit_error": 0.014313540970125369, + "top1_agreement": 0.9870842099189758, + "validation_loss": 2.943952310656403, + "perplexity": 18.990755547983596, + "bits_per_byte": 4.247225399197598, + "greedy_identical_sequences": 3, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 26, + 9, + 64, + 19, + 64, + 64, + 0, + 9 + ] + }, + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s43/best_checkpoint.pt", + "checkpoint_sha256": "e2285dfc79e7aa1d430cac7720bc5997139210fef876055c23da375aab24b04c", + "seed": 43, + "device": "mps", + "dtype": "bfloat16", + "backend": "sdpa", + "max_abs_logit_error": 0.4578789000391552, + "mean_abs_logit_error": 0.014313540970125369, + "top1_agreement": 0.9870842099189758, + "validation_loss": 2.943952310656403, + "perplexity": 18.990755547983596, + "bits_per_byte": 4.247225399197598, + "greedy_identical_sequences": 3, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 26, + 9, + 64, + 19, + 64, + 64, + 0, + 9 + ] + }, + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s43/best_checkpoint.pt", + "checkpoint_sha256": "e2285dfc79e7aa1d430cac7720bc5997139210fef876055c23da375aab24b04c", + "seed": 43, + "device": "mps", + "dtype": "bfloat16", + "backend": "reference", + "max_abs_logit_error": 0.4289135995637001, + "mean_abs_logit_error": 0.01428321546580854, + "top1_agreement": 0.9870128631591797, + "validation_loss": 2.943881231547306, + "perplexity": 18.98940574996995, + "bits_per_byte": 4.247122853719393, + "greedy_identical_sequences": 4, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 26, + 9, + 64, + 19, + 64, + 64, + 0, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s44/best_checkpoint.pt", + "checkpoint_sha256": "d4fe3800f75cd1f3b7ef68bfa463dc48f503585e5cd038da8f22b7f67014f794", + "seed": 44, + "device": "cpu", + "dtype": "float32", + "backend": "auto", + "max_abs_logit_error": 1.2158394370409553e-05, + "mean_abs_logit_error": 7.656799396838815e-07, + "top1_agreement": 1.0, + "validation_loss": 3.0477406320521627, + "perplexity": 21.067690943787593, + "bits_per_byte": 4.3969602957774505, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s44/best_checkpoint.pt", + "checkpoint_sha256": "d4fe3800f75cd1f3b7ef68bfa463dc48f503585e5cd038da8f22b7f67014f794", + "seed": 44, + "device": "cpu", + "dtype": "float32", + "backend": "sdpa", + "max_abs_logit_error": 9.531274606011664e-06, + "mean_abs_logit_error": 7.667224876131937e-07, + "top1_agreement": 1.0, + "validation_loss": 3.0477406320521627, + "perplexity": 21.067690943787593, + "bits_per_byte": 4.3969602957774505, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s44/best_checkpoint.pt", + "checkpoint_sha256": "d4fe3800f75cd1f3b7ef68bfa463dc48f503585e5cd038da8f22b7f67014f794", + "seed": 44, + "device": "cpu", + "dtype": "float32", + "backend": "reference", + "max_abs_logit_error": 9.774208579393928e-06, + "mean_abs_logit_error": 7.666424625083606e-07, + "top1_agreement": 1.0, + "validation_loss": 3.0477406320521627, + "perplexity": 21.067690943787593, + "bits_per_byte": 4.3969602957774505, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s44/best_checkpoint.pt", + "checkpoint_sha256": "d4fe3800f75cd1f3b7ef68bfa463dc48f503585e5cd038da8f22b7f67014f794", + "seed": 44, + "device": "mps", + "dtype": "float32", + "backend": "auto", + "max_abs_logit_error": 1.1915460397027289e-05, + "mean_abs_logit_error": 7.665002081065685e-07, + "top1_agreement": 1.0, + "validation_loss": 3.0477406320521627, + "perplexity": 21.067690943787593, + "bits_per_byte": 4.3969602957774505, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s44/best_checkpoint.pt", + "checkpoint_sha256": "d4fe3800f75cd1f3b7ef68bfa463dc48f503585e5cd038da8f22b7f67014f794", + "seed": 44, + "device": "mps", + "dtype": "float32", + "backend": "sdpa", + "max_abs_logit_error": 1.1915460397027289e-05, + "mean_abs_logit_error": 7.665002081065685e-07, + "top1_agreement": 1.0, + "validation_loss": 3.0477406320521627, + "perplexity": 21.067690943787593, + "bits_per_byte": 4.3969602957774505, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s44/best_checkpoint.pt", + "checkpoint_sha256": "d4fe3800f75cd1f3b7ef68bfa463dc48f503585e5cd038da8f22b7f67014f794", + "seed": 44, + "device": "mps", + "dtype": "float32", + "backend": "reference", + "max_abs_logit_error": 1.2392297555230414e-05, + "mean_abs_logit_error": 7.670999444943002e-07, + "top1_agreement": 1.0, + "validation_loss": 3.0477406320521627, + "perplexity": 21.067690943787593, + "bits_per_byte": 4.3969602957774505, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s44/best_checkpoint.pt", + "checkpoint_sha256": "d4fe3800f75cd1f3b7ef68bfa463dc48f503585e5cd038da8f22b7f67014f794", + "seed": 44, + "device": "mps", + "dtype": "float16", + "backend": "auto", + "max_abs_logit_error": 0.03329057317672257, + "mean_abs_logit_error": 0.001564765724955821, + "top1_agreement": 0.9980376362800598, + "validation_loss": 3.0477300398711993, + "perplexity": 21.067467792174465, + "bits_per_byte": 4.396945014490502, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s44/best_checkpoint.pt", + "checkpoint_sha256": "d4fe3800f75cd1f3b7ef68bfa463dc48f503585e5cd038da8f22b7f67014f794", + "seed": 44, + "device": "mps", + "dtype": "float16", + "backend": "sdpa", + "max_abs_logit_error": 0.03329057317672257, + "mean_abs_logit_error": 0.001564765724955821, + "top1_agreement": 0.9980376362800598, + "validation_loss": 3.0477300398711993, + "perplexity": 21.067467792174465, + "bits_per_byte": 4.396945014490502, + "greedy_identical_sequences": 8, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 64, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s44/best_checkpoint.pt", + "checkpoint_sha256": "d4fe3800f75cd1f3b7ef68bfa463dc48f503585e5cd038da8f22b7f67014f794", + "seed": 44, + "device": "mps", + "dtype": "float16", + "backend": "reference", + "max_abs_logit_error": 0.03329057317672257, + "mean_abs_logit_error": 0.0015677305375769587, + "top1_agreement": 0.998019814491272, + "validation_loss": 3.047726137488739, + "perplexity": 21.067385579018087, + "bits_per_byte": 4.396939384542679, + "greedy_identical_sequences": 7, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 30, + 64, + 64, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s44/best_checkpoint.pt", + "checkpoint_sha256": "d4fe3800f75cd1f3b7ef68bfa463dc48f503585e5cd038da8f22b7f67014f794", + "seed": 44, + "device": "mps", + "dtype": "bfloat16", + "backend": "auto", + "max_abs_logit_error": 0.28835649717008227, + "mean_abs_logit_error": 0.01237538890283738, + "top1_agreement": 0.9842209815979004, + "validation_loss": 3.0479639040772093, + "perplexity": 21.07239529496313, + "bits_per_byte": 4.397282409220754, + "greedy_identical_sequences": 5, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 34, + 64, + 0, + 64, + 30, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s44/best_checkpoint.pt", + "checkpoint_sha256": "d4fe3800f75cd1f3b7ef68bfa463dc48f503585e5cd038da8f22b7f67014f794", + "seed": 44, + "device": "mps", + "dtype": "bfloat16", + "backend": "sdpa", + "max_abs_logit_error": 0.28835649717008227, + "mean_abs_logit_error": 0.01237538890283738, + "top1_agreement": 0.9842209815979004, + "validation_loss": 3.0479639040772093, + "perplexity": 21.07239529496313, + "bits_per_byte": 4.397282409220754, + "greedy_identical_sequences": 5, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 34, + 64, + 0, + 64, + 30, + 64, + 64 + ] + }, + { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s44/best_checkpoint.pt", + "checkpoint_sha256": "d4fe3800f75cd1f3b7ef68bfa463dc48f503585e5cd038da8f22b7f67014f794", + "seed": 44, + "device": "mps", + "dtype": "bfloat16", + "backend": "reference", + "max_abs_logit_error": 0.2749939564287338, + "mean_abs_logit_error": 0.012380472953576582, + "top1_agreement": 0.9841585755348206, + "validation_loss": 3.047937981108009, + "perplexity": 21.071849042989193, + "bits_per_byte": 4.397245010281644, + "greedy_identical_sequences": 6, + "greedy_prompts": 8, + "greedy_first_divergence": [ + 64, + 64, + 64, + 0, + 64, + 30, + 64, + 64 + ] + } + ] +} diff --git a/experiments/records/evaluation/20261022T041615Z_sdpa_v1_eval_cpu_48e95f8.json b/experiments/records/evaluation/20261022T041615Z_sdpa_v1_eval_cpu_48e95f8.json new file mode 100644 index 0000000..9de6c95 --- /dev/null +++ b/experiments/records/evaluation/20261022T041615Z_sdpa_v1_eval_cpu_48e95f8.json @@ -0,0 +1,349 @@ +{ + "record_version": 1, + "created_utc": "20261022T041615Z", + "experiment": "evaluation", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "cpu", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "48e95f8ea09889e6875fb6129026662f8a699cdb", + "branch": "experiments/sdpa-training-v2", + "dirty": false + }, + "command": "python scripts/evaluate_runs.py artifacts/runs/sdpa-v1/auto-s42 artifacts/runs/sdpa-v1/auto-s43 artifacts/runs/sdpa-v1/auto-s44 artifacts/runs/sdpa-v1/sdpa-s42 artifacts/runs/sdpa-v1/sdpa-s43 artifacts/runs/sdpa-v1/sdpa-s44 --protocol science-v1 --device cpu --split validation=data/science/validation.txt --split darwin_validation=data/science/darwin-1859-validation.txt --split einstein_validation=data/science/einstein-1924-validation.txt --name sdpa_v1_eval_cpu" + }, + "settings": { + "protocol": "science-v1", + "splits": { + "validation": { + "path": "data/science/validation.txt", + "sha256": "e25d4e45706bbf4251129dbb079f3487660c453cf8b34407901350538b23e2e7", + "targets": 112111 + }, + "darwin_validation": { + "path": "data/science/darwin-1859-validation.txt", + "sha256": "62a5e5b81b8fa7c80b3e59724d5e33e0a538af3d11f48d7d8fe104ed8c104237", + "targets": 93405 + }, + "einstein_validation": { + "path": "data/science/einstein-1924-validation.txt", + "sha256": "7e8fafbb6cf18f814f5821b1ce760d1aba2e3da4f6a74304496ffec1f7ef1fd0", + "targets": 18703 + } + }, + "metric": "minilm_lab.metrics.evaluate_tokens (non-overlapping context-length chunks, every target once)", + "dtype": "float32" + }, + "results": [ + { + "run": "artifacts/runs/sdpa-v1/auto-s42", + "seed": 42, + "attention_backend": "auto", + "selected_step": 1000, + "selected_on": "validation loss (during training)", + "checkpoint_sha256": "23f0aa813704e0cfc02d0780386fe76d1ee26f70aaa674ce83a849a85fb93546", + "corpus_sha256": "f6642de41672b3cac70017cd5f02312975c0bedfa32be8e79056cf36a48e943f", + "training": { + "attention_backend": "auto", + "device": "cpu", + "steps": 1000, + "train_seconds": 111.17878607107559, + "train_tokens": 4096000, + "train_tokens_per_second": 36841.56073966723, + "eval_seconds": [ + 1.5413287499977741, + 1.5386290419992292, + 1.69361316600407, + 2.0377439169969875 + ], + "process_peak_rss_bytes": 465944576 + }, + "splits": { + "validation": { + "validation_loss": 2.866917291863034, + "perplexity": 17.582732137354572, + "bits_per_token": 4.136087359609617, + "validation_tokens": 112111, + "bits_per_byte": 4.136087359609617, + "eval_seconds": 1.7118395840079756 + }, + "darwin_validation": { + "validation_loss": 2.876895572386068, + "perplexity": 17.759055811538698, + "bits_per_token": 4.1504829754367965, + "validation_tokens": 93405, + "bits_per_byte": 4.1504829754367965, + "eval_seconds": 1.6364306659961585 + }, + "einstein_validation": { + "validation_loss": 2.813936498976771, + "perplexity": 16.675432007740543, + "bits_per_token": 4.059652232450239, + "validation_tokens": 18703, + "bits_per_byte": 4.059652232450239, + "eval_seconds": 0.3203227909980342 + } + } + }, + { + "run": "artifacts/runs/sdpa-v1/auto-s43", + "seed": 43, + "attention_backend": "auto", + "selected_step": 1000, + "selected_on": "validation loss (during training)", + "checkpoint_sha256": "e2285dfc79e7aa1d430cac7720bc5997139210fef876055c23da375aab24b04c", + "corpus_sha256": "f6642de41672b3cac70017cd5f02312975c0bedfa32be8e79056cf36a48e943f", + "training": { + "attention_backend": "auto", + "device": "cpu", + "steps": 1000, + "train_seconds": 139.8997420352389, + "train_tokens": 4096000, + "train_tokens_per_second": 29278.10974067609, + "eval_seconds": [ + 1.9438253749976866, + 2.0082519999996293, + 2.0738366250006948, + 2.048635250001098 + ], + "process_peak_rss_bytes": 470466560 + }, + "splits": { + "validation": { + "validation_loss": 2.9438789420007883, + "perplexity": 18.989362272891913, + "bits_per_token": 4.247119550601986, + "validation_tokens": 112111, + "bits_per_byte": 4.247119550601986, + "eval_seconds": 1.9297752080019563 + }, + "darwin_validation": { + "validation_loss": 2.953197417211484, + "perplexity": 19.167141202336474, + "bits_per_token": 4.260563268577103, + "validation_tokens": 93405, + "bits_per_byte": 4.260563268577103, + "eval_seconds": 1.5810221249994356 + }, + "einstein_validation": { + "validation_loss": 2.8975106876546453, + "perplexity": 18.128960507958386, + "bits_per_token": 4.180224300002127, + "validation_tokens": 18703, + "bits_per_byte": 4.180224300002127, + "eval_seconds": 0.3115091660001781 + } + } + }, + { + "run": "artifacts/runs/sdpa-v1/auto-s44", + "seed": 44, + "attention_backend": "auto", + "selected_step": 1000, + "selected_on": "validation loss (during training)", + "checkpoint_sha256": "d4fe3800f75cd1f3b7ef68bfa463dc48f503585e5cd038da8f22b7f67014f794", + "corpus_sha256": "f6642de41672b3cac70017cd5f02312975c0bedfa32be8e79056cf36a48e943f", + "training": { + "attention_backend": "auto", + "device": "cpu", + "steps": 1000, + "train_seconds": 142.06854248027958, + "train_tokens": 4096000, + "train_tokens_per_second": 28831.15381132711, + "eval_seconds": [ + 2.0271226249897154, + 1.987371292008902, + 2.073784791995422, + 2.016462874991703 + ], + "process_peak_rss_bytes": 472842240 + }, + "splits": { + "validation": { + "validation_loss": 3.0477406168084813, + "perplexity": 21.067690622638423, + "bits_per_token": 4.396960273785466, + "validation_tokens": 112111, + "bits_per_byte": 4.396960273785466, + "eval_seconds": 1.8996217090025311 + }, + "darwin_validation": { + "validation_loss": 3.0596263484649553, + "perplexity": 21.319589576197593, + "bits_per_token": 4.414107759903598, + "validation_tokens": 93405, + "bits_per_byte": 4.414107759903598, + "eval_seconds": 1.5792228749924107 + }, + "einstein_validation": { + "validation_loss": 2.9902191771955113, + "perplexity": 19.890041457359555, + "bits_per_token": 4.313974378111041, + "validation_tokens": 18703, + "bits_per_byte": 4.313974378111041, + "eval_seconds": 0.31484191700292286 + } + } + }, + { + "run": "artifacts/runs/sdpa-v1/sdpa-s42", + "seed": 42, + "attention_backend": "sdpa", + "selected_step": 1000, + "selected_on": "validation loss (during training)", + "checkpoint_sha256": "de734545979c6a4b9a5d6a21bb0299c8c91ac69c32effbfcaca23a86b7cd1fa0", + "corpus_sha256": "f6642de41672b3cac70017cd5f02312975c0bedfa32be8e79056cf36a48e943f", + "training": { + "attention_backend": "sdpa", + "device": "cpu", + "steps": 1000, + "train_seconds": 135.48510687265662, + "train_tokens": 4096000, + "train_tokens_per_second": 30232.10517042186, + "eval_seconds": [ + 1.7904888749908423, + 1.7007023339974694, + 1.6509697080036858, + 1.7257641250034794 + ], + "process_peak_rss_bytes": 470220800 + }, + "splits": { + "validation": { + "validation_loss": 2.872618086572334, + "perplexity": 17.683253938535884, + "bits_per_token": 4.1443118678658495, + "validation_tokens": 112111, + "bits_per_byte": 4.1443118678658495, + "eval_seconds": 1.5277573750063311 + }, + "darwin_validation": { + "validation_loss": 2.8827126912977254, + "perplexity": 17.862663407643005, + "bits_per_token": 4.158875304042906, + "validation_tokens": 93405, + "bits_per_byte": 4.158875304042906, + "eval_seconds": 1.2973567909939447 + }, + "einstein_validation": { + "validation_loss": 2.8195453107991217, + "perplexity": 16.769224153293777, + "bits_per_token": 4.067744037451624, + "validation_tokens": 18703, + "bits_per_byte": 4.067744037451624, + "eval_seconds": 0.27346070800558664 + } + } + }, + { + "run": "artifacts/runs/sdpa-v1/sdpa-s43", + "seed": 43, + "attention_backend": "sdpa", + "selected_step": 1000, + "selected_on": "validation loss (during training)", + "checkpoint_sha256": "0a1357240562d1851b8db80c85e3ce4b4bfd0b00a4c6d9c41ddead928d90c916", + "corpus_sha256": "f6642de41672b3cac70017cd5f02312975c0bedfa32be8e79056cf36a48e943f", + "training": { + "attention_backend": "sdpa", + "device": "cpu", + "steps": 1000, + "train_seconds": 131.15144002575835, + "train_tokens": 4096000, + "train_tokens_per_second": 31231.071494110467, + "eval_seconds": [ + 1.6984680419991491, + 1.7036259589949623, + 1.6872671670134878, + 1.6169645839981968 + ], + "process_peak_rss_bytes": 466665472 + }, + "splits": { + "validation": { + "validation_loss": 2.9449265964036897, + "perplexity": 19.00926698669534, + "bits_per_token": 4.248630996413617, + "validation_tokens": 112111, + "bits_per_byte": 4.248630996413617, + "eval_seconds": 1.5662997500039637 + }, + "darwin_validation": { + "validation_loss": 2.954296349302412, + "perplexity": 19.188216166751495, + "bits_per_token": 4.262148692454959, + "validation_tokens": 93405, + "bits_per_byte": 4.262148692454959, + "eval_seconds": 1.2868834999972023 + }, + "einstein_validation": { + "validation_loss": 2.8981708134392816, + "perplexity": 18.140931853100422, + "bits_per_token": 4.181176660197985, + "validation_tokens": 18703, + "bits_per_byte": 4.181176660197985, + "eval_seconds": 0.2537429579970194 + } + } + }, + { + "run": "artifacts/runs/sdpa-v1/sdpa-s44", + "seed": 44, + "attention_backend": "sdpa", + "selected_step": 1000, + "selected_on": "validation loss (during training)", + "checkpoint_sha256": "e7544665b394924f64facd47c731cd2950caa35d0ced5aaa66c77cdeb4034e47", + "corpus_sha256": "f6642de41672b3cac70017cd5f02312975c0bedfa32be8e79056cf36a48e943f", + "training": { + "attention_backend": "sdpa", + "device": "cpu", + "steps": 1000, + "train_seconds": 129.70830175618175, + "train_tokens": 4096000, + "train_tokens_per_second": 31578.549287457532, + "eval_seconds": [ + 1.67641724999703, + 1.6444599999958882, + 1.631016291998094, + 1.6582876669999678 + ], + "process_peak_rss_bytes": 463060992 + }, + "splits": { + "validation": { + "validation_loss": 3.016298744861834, + "perplexity": 20.415588370621776, + "bits_per_token": 4.3515992410517725, + "validation_tokens": 112111, + "bits_per_byte": 4.3515992410517725, + "eval_seconds": 1.555849374999525 + }, + "darwin_validation": { + "validation_loss": 3.028123759432106, + "perplexity": 20.658436009202205, + "bits_per_token": 4.368659130930744, + "validation_tokens": 93405, + "bits_per_byte": 4.368659130930744, + "eval_seconds": 1.318490791003569 + }, + "einstein_validation": { + "validation_loss": 2.9598152863496843, + "perplexity": 19.29440748589087, + "bits_per_token": 4.2701108355640365, + "validation_tokens": 18703, + "bits_per_byte": 4.2701108355640365, + "eval_seconds": 0.2530294589960249 + } + } + } + ] +} diff --git a/experiments/records/evaluation/20261022T041642Z_sdpa_v1_eval_mps_48e95f8.json b/experiments/records/evaluation/20261022T041642Z_sdpa_v1_eval_mps_48e95f8.json new file mode 100644 index 0000000..a9f4806 --- /dev/null +++ b/experiments/records/evaluation/20261022T041642Z_sdpa_v1_eval_mps_48e95f8.json @@ -0,0 +1,349 @@ +{ + "record_version": 1, + "created_utc": "20261022T041642Z", + "experiment": "evaluation", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "mps", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "48e95f8ea09889e6875fb6129026662f8a699cdb", + "branch": "experiments/sdpa-training-v2", + "dirty": false + }, + "command": "python scripts/evaluate_runs.py artifacts/runs/sdpa-v1/auto-s42 artifacts/runs/sdpa-v1/auto-s43 artifacts/runs/sdpa-v1/auto-s44 artifacts/runs/sdpa-v1/sdpa-s42 artifacts/runs/sdpa-v1/sdpa-s43 artifacts/runs/sdpa-v1/sdpa-s44 --protocol science-v1 --device mps --split validation=data/science/validation.txt --split darwin_validation=data/science/darwin-1859-validation.txt --split einstein_validation=data/science/einstein-1924-validation.txt --name sdpa_v1_eval_mps" + }, + "settings": { + "protocol": "science-v1", + "splits": { + "validation": { + "path": "data/science/validation.txt", + "sha256": "e25d4e45706bbf4251129dbb079f3487660c453cf8b34407901350538b23e2e7", + "targets": 112111 + }, + "darwin_validation": { + "path": "data/science/darwin-1859-validation.txt", + "sha256": "62a5e5b81b8fa7c80b3e59724d5e33e0a538af3d11f48d7d8fe104ed8c104237", + "targets": 93405 + }, + "einstein_validation": { + "path": "data/science/einstein-1924-validation.txt", + "sha256": "7e8fafbb6cf18f814f5821b1ce760d1aba2e3da4f6a74304496ffec1f7ef1fd0", + "targets": 18703 + } + }, + "metric": "minilm_lab.metrics.evaluate_tokens (non-overlapping context-length chunks, every target once)", + "dtype": "float32" + }, + "results": [ + { + "run": "artifacts/runs/sdpa-v1/auto-s42", + "seed": 42, + "attention_backend": "auto", + "selected_step": 1000, + "selected_on": "validation loss (during training)", + "checkpoint_sha256": "23f0aa813704e0cfc02d0780386fe76d1ee26f70aaa674ce83a849a85fb93546", + "corpus_sha256": "f6642de41672b3cac70017cd5f02312975c0bedfa32be8e79056cf36a48e943f", + "training": { + "attention_backend": "auto", + "device": "cpu", + "steps": 1000, + "train_seconds": 111.17878607107559, + "train_tokens": 4096000, + "train_tokens_per_second": 36841.56073966723, + "eval_seconds": [ + 1.5413287499977741, + 1.5386290419992292, + 1.69361316600407, + 2.0377439169969875 + ], + "process_peak_rss_bytes": 465944576 + }, + "splits": { + "validation": { + "validation_loss": 2.8669172491262844, + "perplexity": 17.582731385925765, + "bits_per_token": 4.1360872979535195, + "validation_tokens": 112111, + "bits_per_byte": 4.1360872979535195, + "eval_seconds": 2.33983470799285 + }, + "darwin_validation": { + "validation_loss": 2.8768955276249932, + "perplexity": 17.75905501662429, + "bits_per_token": 4.150482910860216, + "validation_tokens": 93405, + "bits_per_byte": 4.150482910860216, + "eval_seconds": 1.7832639580010436 + }, + "einstein_validation": { + "validation_loss": 2.813936483067752, + "perplexity": 16.675431742450783, + "bits_per_token": 4.059652209498377, + "validation_tokens": 18703, + "bits_per_byte": 4.059652209498377, + "eval_seconds": 0.36053433299821336 + } + } + }, + { + "run": "artifacts/runs/sdpa-v1/auto-s43", + "seed": 43, + "attention_backend": "auto", + "selected_step": 1000, + "selected_on": "validation loss (during training)", + "checkpoint_sha256": "e2285dfc79e7aa1d430cac7720bc5997139210fef876055c23da375aab24b04c", + "corpus_sha256": "f6642de41672b3cac70017cd5f02312975c0bedfa32be8e79056cf36a48e943f", + "training": { + "attention_backend": "auto", + "device": "cpu", + "steps": 1000, + "train_seconds": 139.8997420352389, + "train_tokens": 4096000, + "train_tokens_per_second": 29278.10974067609, + "eval_seconds": [ + 1.9438253749976866, + 2.0082519999996293, + 2.0738366250006948, + 2.048635250001098 + ], + "process_peak_rss_bytes": 470466560 + }, + "splits": { + "validation": { + "validation_loss": 2.9438788897367374, + "perplexity": 18.989361280430945, + "bits_per_token": 4.247119475200899, + "validation_tokens": 112111, + "bits_per_byte": 4.247119475200899, + "eval_seconds": 2.1181079170055455 + }, + "darwin_validation": { + "validation_loss": 2.953197369836624, + "perplexity": 19.167140294295862, + "bits_per_token": 4.260563200229627, + "validation_tokens": 93405, + "bits_per_byte": 4.260563200229627, + "eval_seconds": 1.8521452919958392 + }, + "einstein_validation": { + "validation_loss": 2.897510658080187, + "perplexity": 18.128959971804203, + "bits_per_token": 4.180224257335203, + "validation_tokens": 18703, + "bits_per_byte": 4.180224257335203, + "eval_seconds": 0.3544879580003908 + } + } + }, + { + "run": "artifacts/runs/sdpa-v1/auto-s44", + "seed": 44, + "attention_backend": "auto", + "selected_step": 1000, + "selected_on": "validation loss (during training)", + "checkpoint_sha256": "d4fe3800f75cd1f3b7ef68bfa463dc48f503585e5cd038da8f22b7f67014f794", + "corpus_sha256": "f6642de41672b3cac70017cd5f02312975c0bedfa32be8e79056cf36a48e943f", + "training": { + "attention_backend": "auto", + "device": "cpu", + "steps": 1000, + "train_seconds": 142.06854248027958, + "train_tokens": 4096000, + "train_tokens_per_second": 28831.15381132711, + "eval_seconds": [ + 2.0271226249897154, + 1.987371292008902, + 2.073784791995422, + 2.016462874991703 + ], + "process_peak_rss_bytes": 472842240 + }, + "splits": { + "validation": { + "validation_loss": 3.047740564000013, + "perplexity": 21.067689510085984, + "bits_per_token": 4.396960197598951, + "validation_tokens": 112111, + "bits_per_byte": 4.396960197598951, + "eval_seconds": 2.170749665994663 + }, + "darwin_validation": { + "validation_loss": 3.0596263004366486, + "perplexity": 21.319588552253833, + "bits_per_token": 4.414107690613399, + "validation_tokens": 93405, + "bits_per_byte": 4.414107690613399, + "eval_seconds": 1.9101607920019887 + }, + "einstein_validation": { + "validation_loss": 2.990219145989358, + "perplexity": 19.890040836667886, + "bits_per_token": 4.313974333090078, + "validation_tokens": 18703, + "bits_per_byte": 4.313974333090078, + "eval_seconds": 0.3954607089981437 + } + } + }, + { + "run": "artifacts/runs/sdpa-v1/sdpa-s42", + "seed": 42, + "attention_backend": "sdpa", + "selected_step": 1000, + "selected_on": "validation loss (during training)", + "checkpoint_sha256": "de734545979c6a4b9a5d6a21bb0299c8c91ac69c32effbfcaca23a86b7cd1fa0", + "corpus_sha256": "f6642de41672b3cac70017cd5f02312975c0bedfa32be8e79056cf36a48e943f", + "training": { + "attention_backend": "sdpa", + "device": "cpu", + "steps": 1000, + "train_seconds": 135.48510687265662, + "train_tokens": 4096000, + "train_tokens_per_second": 30232.10517042186, + "eval_seconds": [ + 1.7904888749908423, + 1.7007023339974694, + 1.6509697080036858, + 1.7257641250034794 + ], + "process_peak_rss_bytes": 470220800 + }, + "splits": { + "validation": { + "validation_loss": 2.8726180441077926, + "perplexity": 17.683253187624633, + "bits_per_token": 4.144311806602466, + "validation_tokens": 112111, + "bits_per_byte": 4.144311806602466, + "eval_seconds": 2.1493374580022646 + }, + "darwin_validation": { + "validation_loss": 2.8827126468633737, + "perplexity": 17.862662613927153, + "bits_per_token": 4.158875239937687, + "validation_tokens": 93405, + "bits_per_byte": 4.158875239937687, + "eval_seconds": 1.7363292500085663 + }, + "einstein_validation": { + "validation_loss": 2.8195452687829943, + "perplexity": 16.769223448715934, + "bits_per_token": 4.067743976835166, + "validation_tokens": 18703, + "bits_per_byte": 4.067743976835166, + "eval_seconds": 0.34694137499900535 + } + } + }, + { + "run": "artifacts/runs/sdpa-v1/sdpa-s43", + "seed": 43, + "attention_backend": "sdpa", + "selected_step": 1000, + "selected_on": "validation loss (during training)", + "checkpoint_sha256": "0a1357240562d1851b8db80c85e3ce4b4bfd0b00a4c6d9c41ddead928d90c916", + "corpus_sha256": "f6642de41672b3cac70017cd5f02312975c0bedfa32be8e79056cf36a48e943f", + "training": { + "attention_backend": "sdpa", + "device": "cpu", + "steps": 1000, + "train_seconds": 131.15144002575835, + "train_tokens": 4096000, + "train_tokens_per_second": 31231.071494110467, + "eval_seconds": [ + 1.6984680419991491, + 1.7036259589949623, + 1.6872671670134878, + 1.6169645839981968 + ], + "process_peak_rss_bytes": 466665472 + }, + "splits": { + "validation": { + "validation_loss": 2.9449265523058967, + "perplexity": 19.00926614842864, + "bits_per_token": 4.24863093279395, + "validation_tokens": 112111, + "bits_per_byte": 4.24863093279395, + "eval_seconds": 2.109006833998137 + }, + "darwin_validation": { + "validation_loss": 2.9542963068283994, + "perplexity": 19.188215351750973, + "bits_per_token": 4.2621486311779115, + "validation_tokens": 93405, + "bits_per_byte": 4.2621486311779115, + "eval_seconds": 1.7026253329968313 + }, + "einstein_validation": { + "validation_loss": 2.8981707565339434, + "perplexity": 18.140930820784586, + "bits_per_token": 4.181176578100936, + "validation_tokens": 18703, + "bits_per_byte": 4.181176578100936, + "eval_seconds": 0.35245508298976347 + } + } + }, + { + "run": "artifacts/runs/sdpa-v1/sdpa-s44", + "seed": 44, + "attention_backend": "sdpa", + "selected_step": 1000, + "selected_on": "validation loss (during training)", + "checkpoint_sha256": "e7544665b394924f64facd47c731cd2950caa35d0ced5aaa66c77cdeb4034e47", + "corpus_sha256": "f6642de41672b3cac70017cd5f02312975c0bedfa32be8e79056cf36a48e943f", + "training": { + "attention_backend": "sdpa", + "device": "cpu", + "steps": 1000, + "train_seconds": 129.70830175618175, + "train_tokens": 4096000, + "train_tokens_per_second": 31578.549287457532, + "eval_seconds": [ + 1.67641724999703, + 1.6444599999958882, + 1.631016291998094, + 1.6582876669999678 + ], + "process_peak_rss_bytes": 463060992 + }, + "splits": { + "validation": { + "validation_loss": 3.0162987013084583, + "perplexity": 20.415587481454008, + "bits_per_token": 4.351599178217533, + "validation_tokens": 112111, + "bits_per_byte": 4.351599178217533, + "eval_seconds": 2.2110285419912543 + }, + "darwin_validation": { + "validation_loss": 3.0281237212054943, + "perplexity": 20.65843521950021, + "bits_per_token": 4.368659075781401, + "validation_tokens": 93405, + "bits_per_byte": 4.368659075781401, + "eval_seconds": 1.8024100000038743 + }, + "einstein_validation": { + "validation_loss": 2.959815230464155, + "perplexity": 19.29440640761273, + "bits_per_token": 4.270110754938261, + "validation_tokens": 18703, + "bits_per_byte": 4.270110754938261, + "eval_seconds": 0.352532583012362 + } + } + } + ] +} diff --git a/experiments/records/evaluation/20261022T043652Z_v2_final_cpu_897db37.json b/experiments/records/evaluation/20261022T043652Z_v2_final_cpu_897db37.json new file mode 100644 index 0000000..4bf536a --- /dev/null +++ b/experiments/records/evaluation/20261022T043652Z_v2_final_cpu_897db37.json @@ -0,0 +1,402 @@ +{ + "record_version": 1, + "created_utc": "20261022T043652Z", + "experiment": "evaluation", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "cpu", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "897db37958b0c7595f097a26eaa60549b0f8831e", + "branch": "experiments/sdpa-training-v2", + "dirty": false + }, + "command": "python scripts/evaluate_runs.py artifacts/runs/v2/auto-s42 artifacts/runs/v2/sdpa-s42 artifacts/runs/v2/auto-s43 artifacts/runs/v2/sdpa-s43 artifacts/runs/v2/auto-s44 artifacts/runs/v2/sdpa-s44 --protocol science-v2 --device cpu --split validation=data/science-v2/validation.txt --split darwin_validation=data/science-v2/darwin-1859-validation.txt --split einstein_validation=data/science-v2/einstein-1924-validation.txt --split test=data/science-v2/test.txt --name v2_final_cpu" + }, + "settings": { + "protocol": "science-v2", + "splits": { + "validation": { + "path": "data/science-v2/validation.txt", + "sha256": "6457fe8eaf9011ef75600da24ed0925cfb5e9464fa1c9e453785eecfc9592b17", + "targets": 105665 + }, + "darwin_validation": { + "path": "data/science-v2/darwin-1859-validation.txt", + "sha256": "dea22adfd1f580ec7f3c6a87011bf250436e49371e38efa44209942a4120d59a", + "targets": 87710 + }, + "einstein_validation": { + "path": "data/science-v2/einstein-1924-validation.txt", + "sha256": "c6728bc6890bc714df5ebbe2030766126ad4f08b57c15b551fb948e4e405f9c8", + "targets": 17952 + }, + "test": { + "path": "data/science-v2/test.txt", + "sha256": "9e98ba26c35173180e6a43cc3b49096bea57f62cb1e8013bf5ac1e0b03d53d6b", + "targets": 211848 + } + }, + "metric": "minilm_lab.metrics.evaluate_tokens (non-overlapping context-length chunks, every target once)", + "dtype": "float32" + }, + "results": [ + { + "run": "artifacts/runs/v2/auto-s42", + "seed": 42, + "attention_backend": "auto", + "selected_step": 1000, + "selected_on": "validation loss (during training)", + "checkpoint_sha256": "1cb3fbca57f10b72d33c945325dfc9ed1335cca02fc2fe07150fae5ba40a4b8e", + "corpus_sha256": "84e98dc220b5b274380238ca9c7b45094b7b62f42ca486e70e65d8ac51449988", + "training": { + "attention_backend": "auto", + "device": "cpu", + "steps": 1000, + "train_seconds": 114.02553255201201, + "train_tokens": 4096000, + "train_tokens_per_second": 35921.77916934206, + "eval_seconds": [ + 1.472483249992365, + 1.485997416995815, + 1.610307750001084, + 2.091685250008595 + ], + "process_peak_rss_bytes": 469630976 + }, + "splits": { + "validation": { + "validation_loss": 2.656100203500967, + "perplexity": 14.240645051253535, + "bits_per_token": 3.831942591695012, + "validation_tokens": 105665, + "bits_per_byte": 3.831942591695012, + "eval_seconds": 1.5225239580031484 + }, + "darwin_validation": { + "validation_loss": 2.632080696322548, + "perplexity": 13.902667056821528, + "bits_per_token": 3.7972897678041093, + "validation_tokens": 87710, + "bits_per_byte": 3.7972897678041093, + "eval_seconds": 1.466614082994056 + }, + "einstein_validation": { + "validation_loss": 2.7735504101091952, + "perplexity": 16.015394407031422, + "bits_per_token": 4.001387422320087, + "validation_tokens": 17952, + "bits_per_byte": 4.001387422320087, + "eval_seconds": 0.30494541600637604 + }, + "test": { + "validation_loss": 2.672228437352183, + "perplexity": 14.472183640124108, + "bits_per_token": 3.8552107146904584, + "validation_tokens": 211848, + "bits_per_byte": 3.8552107146904584, + "eval_seconds": 3.5180815420026192 + } + } + }, + { + "run": "artifacts/runs/v2/sdpa-s42", + "seed": 42, + "attention_backend": "sdpa", + "selected_step": 1000, + "selected_on": "validation loss (during training)", + "checkpoint_sha256": "6e8b0471016093200fa1a7045e41b4dcb1be07bb0bb742444bd2c19eaa9f2cc2", + "corpus_sha256": "84e98dc220b5b274380238ca9c7b45094b7b62f42ca486e70e65d8ac51449988", + "training": { + "attention_backend": "sdpa", + "device": "cpu", + "steps": 1000, + "train_seconds": 138.21999243508617, + "train_tokens": 4096000, + "train_tokens_per_second": 29633.91856589524, + "eval_seconds": [ + 1.7645393749990035, + 1.6953102500119712, + 1.576022000008379, + 1.5587145000026794 + ], + "process_peak_rss_bytes": 467435520 + }, + "splits": { + "validation": { + "validation_loss": 2.6489919601914806, + "perplexity": 14.13977799982672, + "bits_per_token": 3.8216875643229837, + "validation_tokens": 105665, + "bits_per_byte": 3.8216875643229837, + "eval_seconds": 1.4187537499965401 + }, + "darwin_validation": { + "validation_loss": 2.624849106128968, + "perplexity": 13.802491317580207, + "bits_per_token": 3.7868567884940907, + "validation_tokens": 87710, + "bits_per_byte": 3.7868567884940907, + "eval_seconds": 1.2133437089942163 + }, + "einstein_validation": { + "validation_loss": 2.7667907563547827, + "perplexity": 15.907500958108317, + "bits_per_token": 3.9916353033704692, + "validation_tokens": 17952, + "bits_per_byte": 3.9916353033704692, + "eval_seconds": 0.25264866699581034 + }, + "test": { + "validation_loss": 2.6652473421022256, + "perplexity": 14.371503784380405, + "bits_per_token": 3.8451391231933716, + "validation_tokens": 211848, + "bits_per_byte": 3.8451391231933716, + "eval_seconds": 3.144351958995685 + } + } + }, + { + "run": "artifacts/runs/v2/auto-s43", + "seed": 43, + "attention_backend": "auto", + "selected_step": 1000, + "selected_on": "validation loss (during training)", + "checkpoint_sha256": "ad66af927334d8991901682c91a0c1ca539707681a0ba890d7145101ba3c041c", + "corpus_sha256": "84e98dc220b5b274380238ca9c7b45094b7b62f42ca486e70e65d8ac51449988", + "training": { + "attention_backend": "auto", + "device": "cpu", + "steps": 1000, + "train_seconds": 139.30257590580732, + "train_tokens": 4096000, + "train_tokens_per_second": 29403.61995006902, + "eval_seconds": [ + 1.8437756249913946, + 1.823977208987344, + 1.9238174159982009, + 1.9362271669961046 + ], + "process_peak_rss_bytes": 468566016 + }, + "splits": { + "validation": { + "validation_loss": 2.9176224386664407, + "perplexity": 18.49725677517611, + "bits_per_token": 4.209239423450438, + "validation_tokens": 105665, + "bits_per_byte": 4.209239423450438, + "eval_seconds": 1.9948942920018453 + }, + "darwin_validation": { + "validation_loss": 2.890859764237426, + "perplexity": 18.008786257843106, + "bits_per_token": 4.170629045770773, + "validation_tokens": 87710, + "bits_per_byte": 4.170629045770773, + "eval_seconds": 2.511365583006409 + }, + "einstein_validation": { + "validation_loss": 3.0476298455461035, + "perplexity": 21.06535705720115, + "bits_per_token": 4.396800464634561, + "validation_tokens": 17952, + "bits_per_byte": 4.396800464634561, + "eval_seconds": 0.6398053329903632 + }, + "test": { + "validation_loss": 2.924396236643034, + "perplexity": 18.62297878271021, + "bits_per_token": 4.2190119481992525, + "validation_tokens": 211848, + "bits_per_byte": 4.2190119481992525, + "eval_seconds": 4.462759666013881 + } + } + }, + { + "run": "artifacts/runs/v2/sdpa-s43", + "seed": 43, + "attention_backend": "sdpa", + "selected_step": 1000, + "selected_on": "validation loss (during training)", + "checkpoint_sha256": "3451e8577c0a5cd8f5b5cadd7aec3f7fd2d9b72fde24db01e6520f9b083de4a4", + "corpus_sha256": "84e98dc220b5b274380238ca9c7b45094b7b62f42ca486e70e65d8ac51449988", + "training": { + "attention_backend": "sdpa", + "device": "cpu", + "steps": 1000, + "train_seconds": 132.18597715096257, + "train_tokens": 4096000, + "train_tokens_per_second": 30986.645393725663, + "eval_seconds": [ + 1.6073278330004541, + 1.5991290420060977, + 1.5932842919864925, + 1.5686826249875594 + ], + "process_peak_rss_bytes": 467763200 + }, + "splits": { + "validation": { + "validation_loss": 2.917622557513584, + "perplexity": 18.497258973522364, + "bits_per_token": 4.209239594910622, + "validation_tokens": 105665, + "bits_per_byte": 4.209239594910622, + "eval_seconds": 4.521736707989476 + }, + "darwin_validation": { + "validation_loss": 2.890860384870535, + "perplexity": 18.008797434695577, + "bits_per_token": 4.170629941155081, + "validation_tokens": 87710, + "bits_per_byte": 4.170629941155081, + "eval_seconds": 2.0680420419957954 + }, + "einstein_validation": { + "validation_loss": 3.047631661522197, + "perplexity": 21.0653953114207, + "bits_per_token": 4.396803084534266, + "validation_tokens": 17952, + "bits_per_byte": 4.396803084534266, + "eval_seconds": 0.5341702500008978 + }, + "test": { + "validation_loss": 2.9243987009859413, + "perplexity": 18.623024676172438, + "bits_per_token": 4.2190155034945445, + "validation_tokens": 211848, + "bits_per_byte": 4.2190155034945445, + "eval_seconds": 6.440284333002637 + } + } + }, + { + "run": "artifacts/runs/v2/auto-s44", + "seed": 44, + "attention_backend": "auto", + "selected_step": 1000, + "selected_on": "validation loss (during training)", + "checkpoint_sha256": "cb77533a909fb6f40f08c3a50854a28765ab214aa87a25a8bb8bc5566211cf40", + "corpus_sha256": "84e98dc220b5b274380238ca9c7b45094b7b62f42ca486e70e65d8ac51449988", + "training": { + "attention_backend": "auto", + "device": "cpu", + "steps": 1000, + "train_seconds": 140.91734968296078, + "train_tokens": 4096000, + "train_tokens_per_second": 29066.68347946707, + "eval_seconds": [ + 1.8970119580044411, + 1.9106220840039896, + 1.9197136669972679, + 1.8356620419945102 + ], + "process_peak_rss_bytes": 469811200 + }, + "splits": { + "validation": { + "validation_loss": 2.602737171301438, + "perplexity": 13.50064107429109, + "bits_per_token": 3.7549560097739527, + "validation_tokens": 105665, + "bits_per_byte": 3.7549560097739527, + "eval_seconds": 2.7495797089941334 + }, + "darwin_validation": { + "validation_loss": 2.576512505154343, + "perplexity": 13.151193371023309, + "bits_per_token": 3.7171218139745705, + "validation_tokens": 87710, + "bits_per_byte": 3.7171218139745705, + "eval_seconds": 1.6367762080044486 + }, + "einstein_validation": { + "validation_loss": 2.728912345865831, + "perplexity": 15.316219208001666, + "bits_per_token": 3.936988308401302, + "validation_tokens": 17952, + "bits_per_byte": 3.936988308401302, + "eval_seconds": 0.33518950000870973 + }, + "test": { + "validation_loss": 2.6308529541475445, + "perplexity": 13.885608639942511, + "bits_per_token": 3.795518510256742, + "validation_tokens": 211848, + "bits_per_byte": 3.795518510256742, + "eval_seconds": 4.451261374997557 + } + } + }, + { + "run": "artifacts/runs/v2/sdpa-s44", + "seed": 44, + "attention_backend": "sdpa", + "selected_step": 1000, + "selected_on": "validation loss (during training)", + "checkpoint_sha256": "2488847e2e18787df5cbd5dd3000f2405527961878cb1c4a0ee510b7f561c038", + "corpus_sha256": "84e98dc220b5b274380238ca9c7b45094b7b62f42ca486e70e65d8ac51449988", + "training": { + "attention_backend": "sdpa", + "device": "cpu", + "steps": 1000, + "train_seconds": 122.27788896088896, + "train_tokens": 4096000, + "train_tokens_per_second": 33497.4706777128, + "eval_seconds": [ + 1.456463333001011, + 1.479247834009584, + 1.483557874991675, + 1.5077983329974813 + ], + "process_peak_rss_bytes": 463126528 + }, + "splits": { + "validation": { + "validation_loss": 2.6013543401671297, + "perplexity": 13.481984869643659, + "bits_per_token": 3.7529610061541, + "validation_tokens": 105665, + "bits_per_byte": 3.7529610061541, + "eval_seconds": 1.5696423750050599 + }, + "darwin_validation": { + "validation_loss": 2.5751997119928354, + "perplexity": 13.133939901896229, + "bits_per_token": 3.7152278537907506, + "validation_tokens": 87710, + "bits_per_byte": 3.7152278537907506, + "eval_seconds": 1.4025792909960728 + }, + "einstein_validation": { + "validation_loss": 2.7269350581415623, + "perplexity": 15.28596455671191, + "bits_per_token": 3.9341356852070892, + "validation_tokens": 17952, + "bits_per_byte": 3.9341356852070892, + "eval_seconds": 0.2722519579983782 + }, + "test": { + "validation_loss": 2.6248352468196083, + "perplexity": 13.802300025908691, + "bits_per_token": 3.786836793737207, + "validation_tokens": 211848, + "bits_per_byte": 3.786836793737207, + "eval_seconds": 3.1318334999959916 + } + } + } + ] +} diff --git a/experiments/records/identity/20261022T053900Z_model_identity_attestation_935a229.json b/experiments/records/identity/20261022T053900Z_model_identity_attestation_935a229.json new file mode 100644 index 0000000..2eba179 --- /dev/null +++ b/experiments/records/identity/20261022T053900Z_model_identity_attestation_935a229.json @@ -0,0 +1,212 @@ +{ + "record_version": 2, + "created_utc": "20261022T053900Z", + "experiment": "model_identity", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "cpu", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "935a22902e6513f554c972172843e60d8f2c0810", + "branch": "experiments/sdpa-training-v2", + "dirty": true + }, + "command": "python scripts/attest_model_identity.py --regenerated training/20261022T041654Z_training_science-v1_auto_seed42_48e95f8.json=/private/tmp/claude-501/-Users-anvit-Downloads-Projects-MiniLM-Lab/ad12857c-f862-4809-a0b1-016af22e423c/scratchpad/release-clone/artifacts/runs/science-v1-s42/best_checkpoint.pt" + }, + "settings": { + "scheme": "minilm-model-identity/v1", + "verification": "checkpoint file SHA-256 equals the training record before hashing tensors" + }, + "results": { + "attested": [ + { + "training_record": "training/20261022T040614Z_training_science-v1_sdpa_seed42_48e95f8.json", + "kind": "best", + "checkpoint_path": "artifacts/runs/sdpa-v1/sdpa-s42/best_checkpoint.pt", + "checkpoint_sha256": "de734545979c6a4b9a5d6a21bb0299c8c91ac69c32effbfcaca23a86b7cd1fa0", + "model_identity_sha256": "292224766ca70f7501a2ad3b1dbd7b593a7f32ff025ec8ead35c81914bdde2e9" + }, + { + "training_record": "training/20261022T040614Z_training_science-v1_sdpa_seed42_48e95f8.json", + "kind": "final", + "checkpoint_path": "artifacts/runs/sdpa-v1/sdpa-s42/tiny_checkpoint.pt", + "checkpoint_sha256": "4c52fc4c8b32fae5ca82295eb33f4c9331ad699b3b87d3c6d533ff043412b242", + "model_identity_sha256": "292224766ca70f7501a2ad3b1dbd7b593a7f32ff025ec8ead35c81914bdde2e9" + }, + { + "training_record": "training/20261022T040834Z_training_science-v1_sdpa_seed43_48e95f8.json", + "kind": "best", + "checkpoint_path": "artifacts/runs/sdpa-v1/sdpa-s43/best_checkpoint.pt", + "checkpoint_sha256": "0a1357240562d1851b8db80c85e3ce4b4bfd0b00a4c6d9c41ddead928d90c916", + "model_identity_sha256": "cc7a5dd2af26be2c8c5bd1a550820a4fe44b01684895202a272a75581a885bd6" + }, + { + "training_record": "training/20261022T040834Z_training_science-v1_sdpa_seed43_48e95f8.json", + "kind": "final", + "checkpoint_path": "artifacts/runs/sdpa-v1/sdpa-s43/tiny_checkpoint.pt", + "checkpoint_sha256": "02c866b578364f3fcbddb8b7e0f53d5088c0f540ab5ff006b4f821579544c599", + "model_identity_sha256": "cc7a5dd2af26be2c8c5bd1a550820a4fe44b01684895202a272a75581a885bd6" + }, + { + "training_record": "training/20261022T041103Z_training_science-v1_auto_seed43_48e95f8.json", + "kind": "best", + "checkpoint_path": "artifacts/runs/sdpa-v1/auto-s43/best_checkpoint.pt", + "checkpoint_sha256": "e2285dfc79e7aa1d430cac7720bc5997139210fef876055c23da375aab24b04c", + "model_identity_sha256": "c609dd197aa7bfa285b3a5fe16eb834cbeab4369e5ff512d5ea27ea6c93ff7d6" + }, + { + "training_record": "training/20261022T041103Z_training_science-v1_auto_seed43_48e95f8.json", + "kind": "final", + "checkpoint_path": "artifacts/runs/sdpa-v1/auto-s43/tiny_checkpoint.pt", + "checkpoint_sha256": "c9a217ad9a778171738c115cd23a3b271957fb570f7990e616d5444e7baed976", + "model_identity_sha256": "c609dd197aa7bfa285b3a5fe16eb834cbeab4369e5ff512d5ea27ea6c93ff7d6" + }, + { + "training_record": "training/20261022T041335Z_training_science-v1_auto_seed44_48e95f8.json", + "kind": "best", + "checkpoint_path": "artifacts/runs/sdpa-v1/auto-s44/best_checkpoint.pt", + "checkpoint_sha256": "d4fe3800f75cd1f3b7ef68bfa463dc48f503585e5cd038da8f22b7f67014f794", + "model_identity_sha256": "f2546d9a7967994e554618d99394bc4bdd5d718a3d4ec7cee8c99686f59314e8" + }, + { + "training_record": "training/20261022T041335Z_training_science-v1_auto_seed44_48e95f8.json", + "kind": "final", + "checkpoint_path": "artifacts/runs/sdpa-v1/auto-s44/tiny_checkpoint.pt", + "checkpoint_sha256": "becaa1b609e62d9591a2cce4cae60e0d158c501db10a7e2edcc5209a1a4ed5ef", + "model_identity_sha256": "f2546d9a7967994e554618d99394bc4bdd5d718a3d4ec7cee8c99686f59314e8" + }, + { + "training_record": "training/20261022T041553Z_training_science-v1_sdpa_seed44_48e95f8.json", + "kind": "best", + "checkpoint_path": "artifacts/runs/sdpa-v1/sdpa-s44/best_checkpoint.pt", + "checkpoint_sha256": "e7544665b394924f64facd47c731cd2950caa35d0ced5aaa66c77cdeb4034e47", + "model_identity_sha256": "22694540faf7514d97d05b0408f1d7a659d9380bae3822137aa814935a82afbd" + }, + { + "training_record": "training/20261022T041553Z_training_science-v1_sdpa_seed44_48e95f8.json", + "kind": "final", + "checkpoint_path": "artifacts/runs/sdpa-v1/sdpa-s44/tiny_checkpoint.pt", + "checkpoint_sha256": "5de158ff5c205dcf1e06ecce60ad189878d8f0899a7c81e9b4b19145e7628244", + "model_identity_sha256": "22694540faf7514d97d05b0408f1d7a659d9380bae3822137aa814935a82afbd" + }, + { + "training_record": "training/20261022T041654Z_training_science-v1_auto_seed42_48e95f8.json", + "kind": "best", + "checkpoint_path": "artifacts/runs/sdpa-v1/auto-s42/best_checkpoint.pt", + "checkpoint_sha256": "23f0aa813704e0cfc02d0780386fe76d1ee26f70aaa674ce83a849a85fb93546", + "model_identity_sha256": "c0c6f2995c5b97d331dc976f01a9af2f6cbea0bed3de442bf0d43986242b1425" + }, + { + "training_record": "training/20261022T041654Z_training_science-v1_auto_seed42_48e95f8.json", + "kind": "final", + "checkpoint_path": "artifacts/runs/sdpa-v1/auto-s42/tiny_checkpoint.pt", + "checkpoint_sha256": "b1af615e4dc2908a234f45c3c8c48c481f7c2b15294e76abd588f17da9b8763d", + "model_identity_sha256": "c0c6f2995c5b97d331dc976f01a9af2f6cbea0bed3de442bf0d43986242b1425" + }, + { + "training_record": "training/20261022T042403Z_training_science-v2_auto_seed42_897db37.json", + "kind": "best", + "checkpoint_path": "artifacts/runs/v2/auto-s42/best_checkpoint.pt", + "checkpoint_sha256": "1cb3fbca57f10b72d33c945325dfc9ed1335cca02fc2fe07150fae5ba40a4b8e", + "model_identity_sha256": "7b25516b2cf08abfd68d2a998e1b91963b1b5c0290a09727a14744e6fd7840af" + }, + { + "training_record": "training/20261022T042403Z_training_science-v2_auto_seed42_897db37.json", + "kind": "final", + "checkpoint_path": "artifacts/runs/v2/auto-s42/tiny_checkpoint.pt", + "checkpoint_sha256": "f4e18218a9da9670cd647c4baf264ddada9ca50fdb86234721e59292ecaae880", + "model_identity_sha256": "7b25516b2cf08abfd68d2a998e1b91963b1b5c0290a09727a14744e6fd7840af" + }, + { + "training_record": "training/20261022T042630Z_training_science-v2_sdpa_seed42_897db37.json", + "kind": "best", + "checkpoint_path": "artifacts/runs/v2/sdpa-s42/best_checkpoint.pt", + "checkpoint_sha256": "6e8b0471016093200fa1a7045e41b4dcb1be07bb0bb742444bd2c19eaa9f2cc2", + "model_identity_sha256": "cecb5f2ebbcae98441896f1da50d5306ee8e49ef063a2a3217b22e089a935625" + }, + { + "training_record": "training/20261022T042630Z_training_science-v2_sdpa_seed42_897db37.json", + "kind": "final", + "checkpoint_path": "artifacts/runs/v2/sdpa-s42/tiny_checkpoint.pt", + "checkpoint_sha256": "264b42c669549c77acc92d05c122beedd9ec87b942a077b073fd7758fb3abef9", + "model_identity_sha256": "cecb5f2ebbcae98441896f1da50d5306ee8e49ef063a2a3217b22e089a935625" + }, + { + "training_record": "training/20261022T042859Z_training_science-v2_auto_seed43_897db37.json", + "kind": "best", + "checkpoint_path": "artifacts/runs/v2/auto-s43/best_checkpoint.pt", + "checkpoint_sha256": "ad66af927334d8991901682c91a0c1ca539707681a0ba890d7145101ba3c041c", + "model_identity_sha256": "242d813f6a59b1b50abeddc211983de28e0fc76ea0dc323ca611a9f9acec4578" + }, + { + "training_record": "training/20261022T042859Z_training_science-v2_auto_seed43_897db37.json", + "kind": "final", + "checkpoint_path": "artifacts/runs/v2/auto-s43/tiny_checkpoint.pt", + "checkpoint_sha256": "46117ebc02af5b2fe2a5cf6b6301435955f8a865aeef7f30a992e5c33694543c", + "model_identity_sha256": "242d813f6a59b1b50abeddc211983de28e0fc76ea0dc323ca611a9f9acec4578" + }, + { + "training_record": "training/20261022T043119Z_training_science-v2_sdpa_seed43_897db37.json", + "kind": "best", + "checkpoint_path": "artifacts/runs/v2/sdpa-s43/best_checkpoint.pt", + "checkpoint_sha256": "3451e8577c0a5cd8f5b5cadd7aec3f7fd2d9b72fde24db01e6520f9b083de4a4", + "model_identity_sha256": "d25ea987563add6fa1a8f53f032a82f4c3d5893c233b6d9cc2802fd1cefae0cc" + }, + { + "training_record": "training/20261022T043119Z_training_science-v2_sdpa_seed43_897db37.json", + "kind": "final", + "checkpoint_path": "artifacts/runs/v2/sdpa-s43/tiny_checkpoint.pt", + "checkpoint_sha256": "59dd70bd83b42f3f4075d7f44fbbe162a908c10d8f07c0a831cc13d080c53340", + "model_identity_sha256": "d25ea987563add6fa1a8f53f032a82f4c3d5893c233b6d9cc2802fd1cefae0cc" + }, + { + "training_record": "training/20261022T043349Z_training_science-v2_auto_seed44_897db37.json", + "kind": "best", + "checkpoint_path": "artifacts/runs/v2/auto-s44/best_checkpoint.pt", + "checkpoint_sha256": "cb77533a909fb6f40f08c3a50854a28765ab214aa87a25a8bb8bc5566211cf40", + "model_identity_sha256": "6e32612238ffcfd87920db587df899a4a021e5206f50e55dc5f74041a8961062" + }, + { + "training_record": "training/20261022T043349Z_training_science-v2_auto_seed44_897db37.json", + "kind": "final", + "checkpoint_path": "artifacts/runs/v2/auto-s44/tiny_checkpoint.pt", + "checkpoint_sha256": "eeef0fd313c7d9d023674a38ac94c05423bd02c5490937cf05d3bdabadb1b219", + "model_identity_sha256": "6e32612238ffcfd87920db587df899a4a021e5206f50e55dc5f74041a8961062" + }, + { + "training_record": "training/20261022T043559Z_training_science-v2_sdpa_seed44_897db37.json", + "kind": "best", + "checkpoint_path": "artifacts/runs/v2/sdpa-s44/best_checkpoint.pt", + "checkpoint_sha256": "2488847e2e18787df5cbd5dd3000f2405527961878cb1c4a0ee510b7f561c038", + "model_identity_sha256": "95142ee6b38e36487d76a0d3bf7b021d9a0800efa956b70db9da79b85f9ceab0" + }, + { + "training_record": "training/20261022T043559Z_training_science-v2_sdpa_seed44_897db37.json", + "kind": "final", + "checkpoint_path": "artifacts/runs/v2/sdpa-s44/tiny_checkpoint.pt", + "checkpoint_sha256": "5f25f43bcdd4a01e7bab23978fa15fdf85e650008b93d8a39b7960437885c687", + "model_identity_sha256": "95142ee6b38e36487d76a0d3bf7b021d9a0800efa956b70db9da79b85f9ceab0" + } + ], + "unavailable": [], + "regeneration_checks": [ + { + "training_record": "training/20261022T041654Z_training_science-v1_auto_seed42_48e95f8.json", + "regenerated_checkpoint_sha256": "d2df75b6dc4e44c0317f82f9c9bc666339094139aa89d075fa96384644ecbfa0", + "original_checkpoint_sha256": "23f0aa813704e0cfc02d0780386fe76d1ee26f70aaa674ce83a849a85fb93546", + "file_hash_equal": false, + "model_identity_sha256": "c0c6f2995c5b97d331dc976f01a9af2f6cbea0bed3de442bf0d43986242b1425", + "model_identity_equal": true + } + ] + } +} diff --git a/experiments/records/identity/20261022T053914Z_model_identity_attestation_935a229.json b/experiments/records/identity/20261022T053914Z_model_identity_attestation_935a229.json new file mode 100644 index 0000000..7c3d83c --- /dev/null +++ b/experiments/records/identity/20261022T053914Z_model_identity_attestation_935a229.json @@ -0,0 +1,212 @@ +{ + "record_version": 2, + "created_utc": "20261022T053914Z", + "experiment": "model_identity", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "cpu", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "935a22902e6513f554c972172843e60d8f2c0810", + "branch": "experiments/sdpa-training-v2", + "dirty": false + }, + "command": "python scripts/attest_model_identity.py --regenerated training/20261022T041654Z_training_science-v1_auto_seed42_48e95f8.json=/private/tmp/claude-501/-Users-anvit-Downloads-Projects-MiniLM-Lab/ad12857c-f862-4809-a0b1-016af22e423c/scratchpad/release-clone/artifacts/runs/science-v1-s42/best_checkpoint.pt" + }, + "settings": { + "scheme": "minilm-model-identity/v1", + "verification": "checkpoint file SHA-256 equals the training record before hashing tensors" + }, + "results": { + "attested": [ + { + "training_record": "training/20261022T040614Z_training_science-v1_sdpa_seed42_48e95f8.json", + "kind": "best", + "checkpoint_path": "artifacts/runs/sdpa-v1/sdpa-s42/best_checkpoint.pt", + "checkpoint_sha256": "de734545979c6a4b9a5d6a21bb0299c8c91ac69c32effbfcaca23a86b7cd1fa0", + "model_identity_sha256": "292224766ca70f7501a2ad3b1dbd7b593a7f32ff025ec8ead35c81914bdde2e9" + }, + { + "training_record": "training/20261022T040614Z_training_science-v1_sdpa_seed42_48e95f8.json", + "kind": "final", + "checkpoint_path": "artifacts/runs/sdpa-v1/sdpa-s42/tiny_checkpoint.pt", + "checkpoint_sha256": "4c52fc4c8b32fae5ca82295eb33f4c9331ad699b3b87d3c6d533ff043412b242", + "model_identity_sha256": "292224766ca70f7501a2ad3b1dbd7b593a7f32ff025ec8ead35c81914bdde2e9" + }, + { + "training_record": "training/20261022T040834Z_training_science-v1_sdpa_seed43_48e95f8.json", + "kind": "best", + "checkpoint_path": "artifacts/runs/sdpa-v1/sdpa-s43/best_checkpoint.pt", + "checkpoint_sha256": "0a1357240562d1851b8db80c85e3ce4b4bfd0b00a4c6d9c41ddead928d90c916", + "model_identity_sha256": "cc7a5dd2af26be2c8c5bd1a550820a4fe44b01684895202a272a75581a885bd6" + }, + { + "training_record": "training/20261022T040834Z_training_science-v1_sdpa_seed43_48e95f8.json", + "kind": "final", + "checkpoint_path": "artifacts/runs/sdpa-v1/sdpa-s43/tiny_checkpoint.pt", + "checkpoint_sha256": "02c866b578364f3fcbddb8b7e0f53d5088c0f540ab5ff006b4f821579544c599", + "model_identity_sha256": "cc7a5dd2af26be2c8c5bd1a550820a4fe44b01684895202a272a75581a885bd6" + }, + { + "training_record": "training/20261022T041103Z_training_science-v1_auto_seed43_48e95f8.json", + "kind": "best", + "checkpoint_path": "artifacts/runs/sdpa-v1/auto-s43/best_checkpoint.pt", + "checkpoint_sha256": "e2285dfc79e7aa1d430cac7720bc5997139210fef876055c23da375aab24b04c", + "model_identity_sha256": "c609dd197aa7bfa285b3a5fe16eb834cbeab4369e5ff512d5ea27ea6c93ff7d6" + }, + { + "training_record": "training/20261022T041103Z_training_science-v1_auto_seed43_48e95f8.json", + "kind": "final", + "checkpoint_path": "artifacts/runs/sdpa-v1/auto-s43/tiny_checkpoint.pt", + "checkpoint_sha256": "c9a217ad9a778171738c115cd23a3b271957fb570f7990e616d5444e7baed976", + "model_identity_sha256": "c609dd197aa7bfa285b3a5fe16eb834cbeab4369e5ff512d5ea27ea6c93ff7d6" + }, + { + "training_record": "training/20261022T041335Z_training_science-v1_auto_seed44_48e95f8.json", + "kind": "best", + "checkpoint_path": "artifacts/runs/sdpa-v1/auto-s44/best_checkpoint.pt", + "checkpoint_sha256": "d4fe3800f75cd1f3b7ef68bfa463dc48f503585e5cd038da8f22b7f67014f794", + "model_identity_sha256": "f2546d9a7967994e554618d99394bc4bdd5d718a3d4ec7cee8c99686f59314e8" + }, + { + "training_record": "training/20261022T041335Z_training_science-v1_auto_seed44_48e95f8.json", + "kind": "final", + "checkpoint_path": "artifacts/runs/sdpa-v1/auto-s44/tiny_checkpoint.pt", + "checkpoint_sha256": "becaa1b609e62d9591a2cce4cae60e0d158c501db10a7e2edcc5209a1a4ed5ef", + "model_identity_sha256": "f2546d9a7967994e554618d99394bc4bdd5d718a3d4ec7cee8c99686f59314e8" + }, + { + "training_record": "training/20261022T041553Z_training_science-v1_sdpa_seed44_48e95f8.json", + "kind": "best", + "checkpoint_path": "artifacts/runs/sdpa-v1/sdpa-s44/best_checkpoint.pt", + "checkpoint_sha256": "e7544665b394924f64facd47c731cd2950caa35d0ced5aaa66c77cdeb4034e47", + "model_identity_sha256": "22694540faf7514d97d05b0408f1d7a659d9380bae3822137aa814935a82afbd" + }, + { + "training_record": "training/20261022T041553Z_training_science-v1_sdpa_seed44_48e95f8.json", + "kind": "final", + "checkpoint_path": "artifacts/runs/sdpa-v1/sdpa-s44/tiny_checkpoint.pt", + "checkpoint_sha256": "5de158ff5c205dcf1e06ecce60ad189878d8f0899a7c81e9b4b19145e7628244", + "model_identity_sha256": "22694540faf7514d97d05b0408f1d7a659d9380bae3822137aa814935a82afbd" + }, + { + "training_record": "training/20261022T041654Z_training_science-v1_auto_seed42_48e95f8.json", + "kind": "best", + "checkpoint_path": "artifacts/runs/sdpa-v1/auto-s42/best_checkpoint.pt", + "checkpoint_sha256": "23f0aa813704e0cfc02d0780386fe76d1ee26f70aaa674ce83a849a85fb93546", + "model_identity_sha256": "c0c6f2995c5b97d331dc976f01a9af2f6cbea0bed3de442bf0d43986242b1425" + }, + { + "training_record": "training/20261022T041654Z_training_science-v1_auto_seed42_48e95f8.json", + "kind": "final", + "checkpoint_path": "artifacts/runs/sdpa-v1/auto-s42/tiny_checkpoint.pt", + "checkpoint_sha256": "b1af615e4dc2908a234f45c3c8c48c481f7c2b15294e76abd588f17da9b8763d", + "model_identity_sha256": "c0c6f2995c5b97d331dc976f01a9af2f6cbea0bed3de442bf0d43986242b1425" + }, + { + "training_record": "training/20261022T042403Z_training_science-v2_auto_seed42_897db37.json", + "kind": "best", + "checkpoint_path": "artifacts/runs/v2/auto-s42/best_checkpoint.pt", + "checkpoint_sha256": "1cb3fbca57f10b72d33c945325dfc9ed1335cca02fc2fe07150fae5ba40a4b8e", + "model_identity_sha256": "7b25516b2cf08abfd68d2a998e1b91963b1b5c0290a09727a14744e6fd7840af" + }, + { + "training_record": "training/20261022T042403Z_training_science-v2_auto_seed42_897db37.json", + "kind": "final", + "checkpoint_path": "artifacts/runs/v2/auto-s42/tiny_checkpoint.pt", + "checkpoint_sha256": "f4e18218a9da9670cd647c4baf264ddada9ca50fdb86234721e59292ecaae880", + "model_identity_sha256": "7b25516b2cf08abfd68d2a998e1b91963b1b5c0290a09727a14744e6fd7840af" + }, + { + "training_record": "training/20261022T042630Z_training_science-v2_sdpa_seed42_897db37.json", + "kind": "best", + "checkpoint_path": "artifacts/runs/v2/sdpa-s42/best_checkpoint.pt", + "checkpoint_sha256": "6e8b0471016093200fa1a7045e41b4dcb1be07bb0bb742444bd2c19eaa9f2cc2", + "model_identity_sha256": "cecb5f2ebbcae98441896f1da50d5306ee8e49ef063a2a3217b22e089a935625" + }, + { + "training_record": "training/20261022T042630Z_training_science-v2_sdpa_seed42_897db37.json", + "kind": "final", + "checkpoint_path": "artifacts/runs/v2/sdpa-s42/tiny_checkpoint.pt", + "checkpoint_sha256": "264b42c669549c77acc92d05c122beedd9ec87b942a077b073fd7758fb3abef9", + "model_identity_sha256": "cecb5f2ebbcae98441896f1da50d5306ee8e49ef063a2a3217b22e089a935625" + }, + { + "training_record": "training/20261022T042859Z_training_science-v2_auto_seed43_897db37.json", + "kind": "best", + "checkpoint_path": "artifacts/runs/v2/auto-s43/best_checkpoint.pt", + "checkpoint_sha256": "ad66af927334d8991901682c91a0c1ca539707681a0ba890d7145101ba3c041c", + "model_identity_sha256": "242d813f6a59b1b50abeddc211983de28e0fc76ea0dc323ca611a9f9acec4578" + }, + { + "training_record": "training/20261022T042859Z_training_science-v2_auto_seed43_897db37.json", + "kind": "final", + "checkpoint_path": "artifacts/runs/v2/auto-s43/tiny_checkpoint.pt", + "checkpoint_sha256": "46117ebc02af5b2fe2a5cf6b6301435955f8a865aeef7f30a992e5c33694543c", + "model_identity_sha256": "242d813f6a59b1b50abeddc211983de28e0fc76ea0dc323ca611a9f9acec4578" + }, + { + "training_record": "training/20261022T043119Z_training_science-v2_sdpa_seed43_897db37.json", + "kind": "best", + "checkpoint_path": "artifacts/runs/v2/sdpa-s43/best_checkpoint.pt", + "checkpoint_sha256": "3451e8577c0a5cd8f5b5cadd7aec3f7fd2d9b72fde24db01e6520f9b083de4a4", + "model_identity_sha256": "d25ea987563add6fa1a8f53f032a82f4c3d5893c233b6d9cc2802fd1cefae0cc" + }, + { + "training_record": "training/20261022T043119Z_training_science-v2_sdpa_seed43_897db37.json", + "kind": "final", + "checkpoint_path": "artifacts/runs/v2/sdpa-s43/tiny_checkpoint.pt", + "checkpoint_sha256": "59dd70bd83b42f3f4075d7f44fbbe162a908c10d8f07c0a831cc13d080c53340", + "model_identity_sha256": "d25ea987563add6fa1a8f53f032a82f4c3d5893c233b6d9cc2802fd1cefae0cc" + }, + { + "training_record": "training/20261022T043349Z_training_science-v2_auto_seed44_897db37.json", + "kind": "best", + "checkpoint_path": "artifacts/runs/v2/auto-s44/best_checkpoint.pt", + "checkpoint_sha256": "cb77533a909fb6f40f08c3a50854a28765ab214aa87a25a8bb8bc5566211cf40", + "model_identity_sha256": "6e32612238ffcfd87920db587df899a4a021e5206f50e55dc5f74041a8961062" + }, + { + "training_record": "training/20261022T043349Z_training_science-v2_auto_seed44_897db37.json", + "kind": "final", + "checkpoint_path": "artifacts/runs/v2/auto-s44/tiny_checkpoint.pt", + "checkpoint_sha256": "eeef0fd313c7d9d023674a38ac94c05423bd02c5490937cf05d3bdabadb1b219", + "model_identity_sha256": "6e32612238ffcfd87920db587df899a4a021e5206f50e55dc5f74041a8961062" + }, + { + "training_record": "training/20261022T043559Z_training_science-v2_sdpa_seed44_897db37.json", + "kind": "best", + "checkpoint_path": "artifacts/runs/v2/sdpa-s44/best_checkpoint.pt", + "checkpoint_sha256": "2488847e2e18787df5cbd5dd3000f2405527961878cb1c4a0ee510b7f561c038", + "model_identity_sha256": "95142ee6b38e36487d76a0d3bf7b021d9a0800efa956b70db9da79b85f9ceab0" + }, + { + "training_record": "training/20261022T043559Z_training_science-v2_sdpa_seed44_897db37.json", + "kind": "final", + "checkpoint_path": "artifacts/runs/v2/sdpa-s44/tiny_checkpoint.pt", + "checkpoint_sha256": "5f25f43bcdd4a01e7bab23978fa15fdf85e650008b93d8a39b7960437885c687", + "model_identity_sha256": "95142ee6b38e36487d76a0d3bf7b021d9a0800efa956b70db9da79b85f9ceab0" + } + ], + "unavailable": [], + "regeneration_checks": [ + { + "training_record": "training/20261022T041654Z_training_science-v1_auto_seed42_48e95f8.json", + "regenerated_checkpoint_sha256": "d2df75b6dc4e44c0317f82f9c9bc666339094139aa89d075fa96384644ecbfa0", + "original_checkpoint_sha256": "23f0aa813704e0cfc02d0780386fe76d1ee26f70aaa674ce83a849a85fb93546", + "file_hash_equal": false, + "model_identity_sha256": "c0c6f2995c5b97d331dc976f01a9af2f6cbea0bed3de442bf0d43986242b1425", + "model_identity_equal": true + } + ] + } +} diff --git a/experiments/records/kv_cache/20261022T035538Z_kv_cpu_float32_93507c4.json b/experiments/records/kv_cache/20261022T035538Z_kv_cpu_float32_93507c4.json new file mode 100644 index 0000000..bd5d4fd --- /dev/null +++ b/experiments/records/kv_cache/20261022T035538Z_kv_cpu_float32_93507c4.json @@ -0,0 +1,8693 @@ +{ + "record_version": 1, + "created_utc": "20261022T035538Z", + "experiment": "kv_cache_scaling", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "cpu", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "93507c45fb853c19bfc7bbafba89d31852661b53", + "branch": "experiments/attention-kv-audit", + "dirty": false + }, + "command": "python scripts/benchmark_kv_scaling.py --device cpu" + }, + "settings": { + "dtype": "float32", + "position": "learned", + "prompt_lens": [ + 1, + 16, + 128, + 512, + 1024 + ], + "batch_sizes": [ + 1, + 8 + ], + "new_tokens": 64, + "warmup": 1, + "repeats": 5, + "seed": 0, + "modes": { + "cached": { + "attention_backend": "sdpa", + "use_cache": true + }, + "uncached_sdpa": { + "attention_backend": "sdpa", + "use_cache": false + }, + "uncached_default": { + "attention_backend": "auto", + "use_cache": false + } + }, + "weights": "seeded random initialization (timing does not depend on trained values)", + "grid_model_config": { + "vocab_size": 256, + "context_length": 1088, + "embedding_dim": 192, + "num_layers": 4, + "num_heads": 6, + "dropout": 0.0, + "positional_encoding": "learned", + "norm": "rmsnorm", + "activation": "swiglu", + "rope_theta": 10000.0 + }, + "window_model_config": { + "vocab_size": 256, + "context_length": 128, + "embedding_dim": 192, + "num_layers": 4, + "num_heads": 6, + "dropout": 0.0, + "positional_encoding": "learned", + "norm": "rmsnorm", + "activation": "swiglu", + "rope_theta": 10000.0 + }, + "decoding": "greedy argmax; step 0 is the prompt prefill" + }, + "results": { + "grid": [ + { + "batch": 1, + "prompt_len": 1, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_default": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 0.3355840017320588, + "p95": 0.3419749962631613, + "min": 0.31529099214822054, + "max": 0.3431249933782965, + "iqr": 0.002291009877808392, + "mean": 0.3332917985972017, + "cv": 0.031677026003521215 + }, + "decode_ms_per_step": { + "count": 315, + "median": 0.3241660015191883, + "p95": 0.3919625029084273, + "min": 0.28979200578760356, + "max": 1.3880000042263418, + "iqr": 0.018645499949343503, + "mean": 0.3462668827971414, + "cv": 0.33909942112194424 + }, + "total_ms": { + "count": 5, + "median": 21.1358830856625, + "p95": 26.05287044425495, + "min": 20.309743034886196, + "max": 27.2244120715186, + "iqr": 0.6629189883824438, + "mean": 22.14810541481711, + "cv": 0.12942892825731334 + }, + "tokens_per_second": { + "count": 5, + "median": 3028.0258336314473, + "p95": 3139.20199127977, + "min": 2350.8312991983753, + "max": 3151.1969348931066, + "iqr": 95.90762857288064, + "mean": 2923.3181745605793, + "cv": 0.11137338059888574 + }, + "median_step_ms_by_index": [ + 0.3355840017320588, + 0.3509579983074218, + 0.3309169987915084, + 0.32954099879134446, + 0.3257079952163622, + 0.32441699295304716, + 0.32420799834653735, + 0.3199580096406862, + 0.3176249883836135, + 0.3275420021964237, + 0.3234589967178181, + 0.3144999936921522, + 0.32233400270342827, + 0.324332999298349, + 0.31820801086723804, + 0.31612500606570393, + 0.30937499832361937, + 0.3220409998903051, + 0.3251669986639172, + 0.3240829973947257, + 0.3149580006720498, + 0.3186249959981069, + 0.3223329986212775, + 0.31874999694991857, + 0.32208299671765417, + 0.31720900733489543, + 0.32537498918827623, + 0.3215420001652092, + 0.31470900285057724, + 0.313292010105215, + 0.3251250018365681, + 0.3304999991087243, + 0.3010829968843609, + 0.3167910035699606, + 0.3183329972671345, + 0.33108299248851836, + 0.32058299984782934, + 0.3272500034654513, + 0.31799999123904854, + 0.31766599568072706, + 0.3106669901171699, + 0.32824999652802944, + 0.3327919985167682, + 0.3272909962106496, + 0.3318750095786527, + 0.32741700124461204, + 0.32824999652802944, + 0.3273329930379987, + 0.33054100640583783, + 0.32416700560133904, + 0.33162500767502934, + 0.3270410088589415, + 0.3186249959981069, + 0.32045900297816843, + 0.34070800757035613, + 0.32695799018256366, + 0.33525000617373735, + 0.337166988174431, + 0.3374170046299696, + 0.3292500041425228, + 0.33258400799240917, + 0.33149999217130244, + 0.3233749885112047, + 0.33483300649095327 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 0.3030420048162341, + "p95": 0.3347417979966849, + "min": 0.29499999072868377, + "max": 0.33829199674073607, + "iqr": 0.01841600169427693, + "mean": 0.3117999993264675, + "cv": 0.056257963444603945 + }, + "decode_ms_per_step": { + "count": 315, + "median": 0.7032080029603094, + "p95": 1.006974909978453, + "min": 0.4522499948507175, + "max": 6.182457989780232, + "iqr": 0.37143750523682684, + "mean": 0.7655229967360991, + "cv": 0.5949859159377524 + }, + "total_ms": { + "count": 5, + "median": 46.22525205195416, + "p95": 56.038206128869206, + "min": 45.537042984506115, + "max": 58.33345690916758, + "iqr": 1.1114139924757183, + "mean": 48.53974879370071, + "cv": 0.11327502556153317 + }, + "tokens_per_second": { + "count": 5, + "median": 1384.5246301322097, + "p95": 1404.166524044629, + "min": 1097.1405329133147, + "max": 1405.4491861005527, + "iqr": 33.18397062898089, + "mean": 1330.400426031793, + "cv": 0.09867834791658903 + }, + "median_step_ms_by_index": [ + 0.3030420048162341, + 0.47458299377467483, + 0.46262500109151006, + 0.4608750023180619, + 0.4750420048367232, + 0.4783750046044588, + 0.5043749988544732, + 0.4936250043101609, + 0.5010000022593886, + 0.5116670072311535, + 0.5210829986026511, + 0.5259580066194758, + 0.5301659984979779, + 0.5432920006569475, + 0.5503339925780892, + 0.5358339985832572, + 0.5487919988809153, + 0.5557089898502454, + 0.5557920085266232, + 0.5621659947792068, + 0.5718750035157427, + 0.5803329986520112, + 0.5798750062240288, + 0.5895829963264987, + 0.603374995989725, + 0.594000011915341, + 0.6036249978933483, + 0.6003330054227263, + 0.611750001553446, + 0.6004159949952736, + 0.6215000030351803, + 0.6158750038594007, + 0.7127910066628829, + 0.7342089957091957, + 0.7563749968539923, + 0.7621249969815835, + 0.738000002456829, + 0.7746670016786084, + 0.7551250018877909, + 0.7430839905282483, + 0.7418339955620468, + 0.7485409878427163, + 0.90562499826774, + 0.9203329973388463, + 0.9357919916510582, + 0.9147090022452176, + 0.9197909967042506, + 0.9224999957950786, + 0.9263340034522116, + 0.9210419957526028, + 0.910000002477318, + 0.9316249925177544, + 0.946375002968125, + 0.9347920131403953, + 0.9420420101378113, + 0.9340830001747236, + 0.9578329918440431, + 0.9811250056372955, + 0.9737919899635017, + 0.9857910044956952, + 1.0072080040117726, + 1.0051249992102385, + 1.0068750125356019, + 1.004625009954907 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_default": { + "first_token_ms": { + "count": 5, + "median": 0.3845000028377399, + "p95": 0.4020166059490293, + "min": 0.3113749990006909, + "max": 0.4057500045746565, + "iqr": 0.008708011591807008, + "mean": 0.3734166035428643, + "cv": 0.09682281678709329 + }, + "decode_ms_per_step": { + "count": 315, + "median": 0.7896669994806871, + "p95": 1.1308455985272303, + "min": 0.48241698823403567, + "max": 1.705665999907069, + "iqr": 0.39960400317795575, + "mean": 0.8254872567097967, + "cv": 0.2655708646969402 + }, + "total_ms": { + "count": 5, + "median": 52.048578989342786, + "p95": 54.18667260091752, + "min": 51.246499962871894, + "max": 54.69108100805897, + "iqr": 0.4286690236767754, + "mean": 52.379113776260056, + "cv": 0.025595152711881983 + }, + "tokens_per_second": { + "count": 5, + "median": 1229.6205053572035, + "p95": 1246.4816529696718, + "min": 1170.2090874848373, + "max": 1248.865777104154, + "iqr": 10.163884230842768, + "mean": 1222.4843597157678, + "cv": 0.0249028399660993 + }, + "median_step_ms_by_index": [ + 0.3845000028377399, + 0.49170799320563674, + 0.5050409963587299, + 0.5227079964242876, + 0.5381660012062639, + 0.5431250028777868, + 0.5545000021811575, + 0.5410000012489036, + 0.556125000002794, + 0.5707500095013529, + 0.6025829934515059, + 0.5949170008534566, + 0.6022919988026842, + 0.6216249894350767, + 0.6357090023811907, + 0.6164579972391948, + 0.6310839962679893, + 0.6350409967126325, + 0.643749997834675, + 0.6589999975403771, + 0.6582080095540732, + 0.6682909879600629, + 0.683374993968755, + 0.6815829983679578, + 0.6802499992772937, + 0.6951660034246743, + 0.7171659963205457, + 0.7111250015441328, + 0.7143330003600568, + 0.7178749947343022, + 0.7170000026235357, + 0.7120000082068145, + 0.79645799996797, + 0.7937089976621792, + 0.8119999984046444, + 0.7971250015543774, + 0.8145419997163117, + 0.8382090018130839, + 0.8206659986171871, + 0.844458001665771, + 0.8495830115862191, + 0.8436249918304384, + 0.9911250090226531, + 0.9903750033117831, + 1.0065000096801668, + 1.001249998807907, + 1.0275829990860075, + 1.0441670019645244, + 1.066915996489115, + 1.0540410003159195, + 1.0469169938005507, + 1.0637090017553419, + 1.1541669955477118, + 1.0609579912852496, + 1.069958001608029, + 1.0781250020954758, + 1.088082994101569, + 1.0992499883286655, + 1.119124994147569, + 1.1308749963063747, + 1.1127920006401837, + 1.117916006478481, + 1.120124987210147, + 1.10599999607075 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "speedup_cached_vs_uncached_sdpa": 2.1870508965537856, + "speedup_cached_vs_uncached_default": 2.4625694028677647 + }, + { + "batch": 1, + "prompt_len": 16, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_default": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 0.5748330004280433, + "p95": 0.6438502023229375, + "min": 0.5687910015694797, + "max": 0.6502920005004853, + "iqr": 0.048124013119377196, + "mean": 0.5963916017208248, + "cv": 0.06104051285892477 + }, + "decode_ms_per_step": { + "count": 315, + "median": 0.32879099308047444, + "p95": 0.3781625971896574, + "min": 0.29304200143087655, + "max": 0.5578330019488931, + "iqr": 0.019270497432444245, + "mean": 0.33478159434346866, + "cv": 0.08611416836483402 + }, + "total_ms": { + "count": 5, + "median": 21.657457065884955, + "p95": 22.631950076902285, + "min": 21.039950966951437, + "max": 22.82837410166394, + "iqr": 0.7801298634149134, + "mean": 21.68763204535935, + "cv": 0.03367178977756853 + }, + "tokens_per_second": { + "count": 5, + "median": 2955.1022451667905, + "p95": 3041.076332309045, + "min": 2803.5286137760945, + "max": 3041.8321839498667, + "iqr": 108.48888859443468, + "mean": 2953.6160011579664, + "cv": 0.03300761463307249 + }, + "median_step_ms_by_index": [ + 0.5748330004280433, + 0.37762499414384365, + 0.3419159911572933, + 0.3581670025596395, + 0.335000004270114, + 0.32304100750479847, + 0.33399999665562063, + 0.34124999365303665, + 0.3291249886387959, + 0.33312498999293894, + 0.32829098927322775, + 0.33312500454485416, + 0.32054200710263103, + 0.33900000562425703, + 0.32766698859632015, + 0.3317920054541901, + 0.32866701076272875, + 0.3207920090062544, + 0.3282079997006804, + 0.31962500361260027, + 0.32958301017060876, + 0.32116600777953863, + 0.32108399318531156, + 0.3184999950462952, + 0.31929099350236356, + 0.3158330073347315, + 0.3264170081820339, + 0.32533299236092716, + 0.32895899494178593, + 0.32870798895601183, + 0.32145899604074657, + 0.3248329885536805, + 0.3233330062357709, + 0.3259579971199855, + 0.3213339950889349, + 0.3315840003779158, + 0.3258330107200891, + 0.3145420050714165, + 0.3231250011594966, + 0.32387500687036663, + 0.3400410059839487, + 0.32566698791924864, + 0.32387500687036663, + 0.3202500083716586, + 0.3311670006951317, + 0.316707999445498, + 0.32579200342297554, + 0.33075000101234764, + 0.31533300352748483, + 0.3332920023240149, + 0.3336670051794499, + 0.32733399712014943, + 0.32750000536907464, + 0.33054100640583783, + 0.32662499870639294, + 0.3353339998284355, + 0.3305830032331869, + 0.3341249976074323, + 0.33808300213422626, + 0.3412910009501502, + 0.33024999720510095, + 0.3345000004628673, + 0.3260000084992498, + 0.34149999555666 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 0.5842920072609559, + "p95": 0.7629330066265538, + "min": 0.5688750097760931, + "max": 0.8070830081123859, + "iqr": 0.010207993909716606, + "mean": 0.6245416065212339, + "cv": 0.16376700600678704 + }, + "decode_ms_per_step": { + "count": 315, + "median": 0.9132089908234775, + "p95": 1.2226206934428774, + "min": 0.5557500116992742, + "max": 1.8201250059064478, + "iqr": 0.4067920017405413, + "mean": 0.898699587329276, + "cv": 0.2600620817980446 + }, + "total_ms": { + "count": 5, + "median": 55.862120978417806, + "p95": 60.89795140433125, + "min": 55.582088069058955, + "max": 61.795751011231914, + "iqr": 1.6403879708377644, + "mean": 57.24261560826562, + "cv": 0.04612315717856981 + }, + "tokens_per_second": { + "count": 5, + "median": 1145.6779456105191, + "p95": 1151.1014333708265, + "min": 1035.669911809429, + "max": 1151.450084431554, + "iqr": 32.910000209876216, + "mean": 1119.860319979492, + "cv": 0.043866224611039256 + }, + "median_step_ms_by_index": [ + 0.5842920072609559, + 0.5717079911846668, + 0.5686249933205545, + 0.5672499974025413, + 0.5737080064136535, + 0.581457992666401, + 0.5802499945275486, + 0.5848749860888347, + 0.595374993281439, + 0.5977920081932098, + 0.6016660045133904, + 0.6272500031627715, + 0.6196250033099204, + 0.6327500013867393, + 0.6391669885488227, + 0.6431249930756167, + 0.6264999974519014, + 0.6995420117164031, + 0.6995419971644878, + 0.7027910032775253, + 0.7000829937169328, + 0.7169589953264222, + 0.737666996428743, + 0.7840000034775585, + 0.7446249946951866, + 0.7624159916304052, + 0.7653330103494227, + 0.913042007596232, + 0.909624999621883, + 0.8972089999588206, + 0.9159159963019192, + 0.9154580038739368, + 0.9034160029841587, + 0.94541598809883, + 0.9420840069651604, + 0.9138750028796494, + 0.9108340018428862, + 0.9132089908234775, + 0.9319999953731894, + 0.9327909938292578, + 0.9307080035796389, + 0.9881250007310882, + 1.0200000106124207, + 1.0116670018760487, + 1.0150419984711334, + 1.0071660071844235, + 1.009458996122703, + 1.0121249943040311, + 1.0074169986182824, + 1.1145830067107454, + 1.125375012634322, + 1.1333330039633438, + 1.120874992921017, + 1.1325000086799264, + 1.131166995037347, + 1.1719589965650812, + 1.145750007708557, + 1.2142500054324046, + 1.2072090030414984, + 1.1636250128503889, + 1.1689170059980825, + 1.146666007116437, + 1.1780409986386076, + 1.1772500001825392 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_default": { + "first_token_ms": { + "count": 5, + "median": 0.7083749951561913, + "p95": 0.7238080084789544, + "min": 0.6794999935664237, + "max": 0.724833007552661, + "iqr": 0.015291021554730833, + "mean": 0.7073665998177603, + "cv": 0.02492726649409213 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.0501250071683899, + "p95": 1.3923250924563035, + "min": 0.6334999925456941, + "max": 2.119208002113737, + "iqr": 0.4392085029394366, + "mean": 1.0245662506407363, + "cv": 0.25336263584758595 + }, + "total_ms": { + "count": 5, + "median": 65.31845300924033, + "p95": 69.20098234841134, + "min": 62.44771096680779, + "max": 70.00099893775769, + "iqr": 3.493792944937013, + "mean": 65.25504039018415, + "cv": 0.047553231955464736 + }, + "tokens_per_second": { + "count": 5, + "median": 979.8149994603544, + "p95": 1024.66259854971, + "min": 914.2726671215999, + "max": 1024.8574208591453, + "iqr": 54.19979751492099, + "mean": 982.5023817100233, + "cv": 0.04645596770040982 + }, + "median_step_ms_by_index": [ + 0.7083749951561913, + 0.6655000033788383, + 0.641040998743847, + 0.6714589981129393, + 0.6636659963987768, + 0.6754589994670823, + 0.6566249940078706, + 0.6836249958723783, + 0.6931250100024045, + 0.6855420069769025, + 0.720458003343083, + 0.7085830002324656, + 0.7071669970173389, + 0.709084008121863, + 0.7129589939722791, + 0.7192499906523153, + 0.735500012524426, + 0.8039169915718958, + 0.7901250064605847, + 0.8182499877875671, + 0.8190830121748149, + 0.8965000015450642, + 0.9057080023922026, + 0.8479579992126673, + 0.8388340065721422, + 0.858375002280809, + 0.8437920041615143, + 1.0358330036979169, + 1.0018329921877012, + 1.057834000675939, + 1.0142079991055652, + 1.0584999981801957, + 1.0496670001884922, + 1.06545900052879, + 1.0374590056017041, + 1.0497499897610396, + 1.0421250044601038, + 1.0801669995998964, + 1.0852090053958818, + 1.0763749887701124, + 1.0669999901438132, + 1.1332080030115321, + 1.1109160113846883, + 1.1637499992502853, + 1.152625001850538, + 1.113207996240817, + 1.1032500042347237, + 1.1848750000353903, + 1.1327909887768328, + 1.2522079923655838, + 1.2275839981157333, + 1.2377910024952143, + 1.2315420026425272, + 1.2999160098843277, + 1.2839160044677556, + 1.269624990527518, + 1.2509579973993823, + 1.319916991633363, + 1.3091250002617016, + 1.3469169934978709, + 1.336042012553662, + 1.3502079964382574, + 1.363292001769878, + 1.5463750023627654 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "speedup_cached_vs_uncached_sdpa": 2.579348111298458, + "speedup_cached_vs_uncached_default": 3.015979799037931 + }, + { + "batch": 1, + "prompt_len": 128, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_default": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 1.5283749962691218, + "p95": 1.966316407197155, + "min": 1.4662089961348101, + "max": 2.0455830090213567, + "iqr": 0.1694169914117083, + "mean": 1.6338500019628555, + "cv": 0.14762714744400066 + }, + "decode_ms_per_step": { + "count": 315, + "median": 0.3803330037044361, + "p95": 0.551570497918874, + "min": 0.34620799124240875, + "max": 0.7986249984242022, + "iqr": 0.08672999683767557, + "mean": 0.4125064190861488, + "cv": 0.18266582420146543 + }, + "total_ms": { + "count": 5, + "median": 26.647161052096635, + "p95": 29.99856640235521, + "min": 26.16178698372096, + "max": 30.4215420037508, + "iqr": 1.7350460111629218, + "mean": 27.62175440439023, + "cv": 0.06397869047364226 + }, + "tokens_per_second": { + "count": 5, + "median": 2401.7567903341205, + "p95": 2438.769854693961, + "min": 2103.7723857689125, + "max": 2446.3160731269495, + "iqr": 147.63328395891085, + "mean": 2324.276385439017, + "cv": 0.061087036890529856 + }, + "median_step_ms_by_index": [ + 1.5283749962691218, + 0.5006249994039536, + 0.4083340027136728, + 0.37733299541287124, + 0.3714159975061193, + 0.36454200744628906, + 0.368500011973083, + 0.3623329976107925, + 0.3657500055851415, + 0.3632500011008233, + 0.3564999933587387, + 0.360707999789156, + 0.36266598908696324, + 0.36337500205263495, + 0.3575000009732321, + 0.3632909938460216, + 0.3648329875431955, + 0.3624579985626042, + 0.3686250129248947, + 0.3598749899538234, + 0.36945799365639687, + 0.35812500573229045, + 0.36995799746364355, + 0.3633339947555214, + 0.36691699642688036, + 0.3545420040609315, + 0.3615829918999225, + 0.36341599479783326, + 0.3616660105763003, + 0.36720800562761724, + 0.3598329931264743, + 0.3590830019675195, + 0.3607499966165051, + 0.35945800482295454, + 0.3530410031089559, + 0.35604200093075633, + 0.35091600148007274, + 0.35266700433567166, + 0.360332996933721, + 0.35854100133292377, + 0.3625410026870668, + 0.3604579978855327, + 0.36612499388866127, + 0.3967910015489906, + 0.4890830023214221, + 0.46387499605771154, + 0.45595900155603886, + 0.4455419984878972, + 0.45237499580252916, + 0.44683300075121224, + 0.4420000041136518, + 0.44662499567493796, + 0.4440000047907233, + 0.4464579978957772, + 0.4452499997569248, + 0.4468749975785613, + 0.45145899639464915, + 0.4493340093176812, + 0.4489580023800954, + 0.4414169961819425, + 0.4489999992074445, + 0.45008301094640046, + 0.4604999994626269, + 0.44758300646208227 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 1.44749999162741, + "p95": 1.4796333998674527, + "min": 1.4259169984143227, + "max": 1.4851250016363338, + "iqr": 0.02012499317061156, + "mean": 1.4507501968182623, + "cv": 0.015532854916520358 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.7002090025926009, + "p95": 2.3347001057118177, + "min": 1.4079169923206791, + "max": 2.5439999881200492, + "iqr": 0.6152705100248568, + "mean": 1.8345386061968931, + "cv": 0.16982921837258874 + }, + "total_ms": { + "count": 5, + "median": 117.43307998403907, + "p95": 117.9172104166355, + "min": 115.96357992675621, + "max": 117.98908602213487, + "iqr": 1.5117499860934913, + "mean": 117.02668238722254, + "cv": 0.007890874029137828 + }, + "tokens_per_second": { + "count": 5, + "median": 544.9912410429716, + "p95": 551.7506662208735, + "min": 542.4230507895751, + "max": 551.8974150368854, + "iqr": 7.083428889768243, + "mean": 546.9111239786632, + "cv": 0.007906326015067374 + }, + "median_step_ms_by_index": [ + 1.44749999162741, + 1.5176249871728942, + 1.5293339965865016, + 1.5219580091070384, + 1.522541992017068, + 1.5516249986831099, + 1.5443750016856939, + 1.5250830037984997, + 1.5271250013029203, + 1.5472079976461828, + 1.5483329916605726, + 1.5469170029973611, + 1.569333006045781, + 1.5497910062549636, + 1.5498749999096617, + 1.6329170030076057, + 1.5503330068895593, + 1.5684579993830994, + 1.607583006261848, + 1.5934169932734221, + 1.5841669955989346, + 1.5879580023465678, + 1.5805830043973401, + 1.600916002644226, + 1.6173750045709312, + 1.6161250096047297, + 1.6771669907029718, + 1.637666005990468, + 1.6422920016339049, + 1.633542007766664, + 1.6392500110669062, + 1.6425420035375282, + 1.6530419961782172, + 1.7407499981345609, + 1.7216249980265275, + 1.7519160028314218, + 1.727625000057742, + 1.7002090025926009, + 1.7645420011831447, + 1.7529169999761507, + 1.7644589970586821, + 1.7504589923191816, + 1.77254099980928, + 2.2518750047311187, + 2.1724580001318827, + 2.257874992210418, + 2.2127910051494837, + 2.213458006735891, + 2.187333011534065, + 2.2224580025067553, + 2.2240419930312783, + 2.18695800867863, + 2.345250002690591, + 2.1778340014861897, + 2.387583997915499, + 2.2540409991052, + 2.26470798952505, + 2.2440000029746443, + 2.224083000328392, + 2.2222920088097453, + 2.2812499955762178, + 2.260124994791113, + 2.2403330076485872, + 2.2979589994065464 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_default": { + "first_token_ms": { + "count": 5, + "median": 1.8591659900266677, + "p95": 1.9750083971302956, + "min": 1.8102500034729019, + "max": 1.9889999966835603, + "iqr": 0.06558299355674535, + "mean": 1.8861833988921717, + "cv": 0.03674821043918749 + }, + "decode_ms_per_step": { + "count": 315, + "median": 2.2296660026768222, + "p95": 3.2847918948391452, + "min": 1.722916989820078, + "max": 12.269458005903289, + "iqr": 0.9219174971804023, + "mean": 2.449276694789573, + "cv": 0.3340176650101426 + }, + "total_ms": { + "count": 5, + "median": 153.0652439541882, + "p95": 172.43504938087426, + "min": 147.869327920489, + "max": 176.90987198147923, + "iqr": 5.962885959888808, + "mean": 156.19061517063528, + "cv": 0.07636548574853264 + }, + "tokens_per_second": { + "count": 5, + "median": 418.1223532309852, + "p95": 432.40466682535066, + "min": 361.7661314383868, + "max": 432.81457283970013, + "iqr": 16.62141398541968, + "mean": 411.5223458119115, + "cv": 0.07032065962347508 + }, + "median_step_ms_by_index": [ + 1.8591659900266677, + 1.890250001451932, + 1.854124988312833, + 1.872708002338186, + 1.8505839980207384, + 1.8763340049190447, + 1.931207996676676, + 1.8372920021647587, + 1.8882500007748604, + 1.9301660067867488, + 1.9190830062143505, + 1.934291998622939, + 1.9417920120758936, + 1.8914579995907843, + 1.9598749931901693, + 1.905541997984983, + 1.931792008690536, + 1.9842909969156608, + 1.9979159987997264, + 2.033500000834465, + 1.9529999990481883, + 2.0158749975962564, + 2.0289169915486127, + 2.068540998152457, + 2.0078750094398856, + 2.0171659998595715, + 2.048791997367516, + 2.0190829964121804, + 2.0539580000331625, + 2.0917910005664453, + 2.081333994283341, + 2.1055830002296716, + 2.0478329970501363, + 2.2254170035012066, + 2.304500012542121, + 2.2872920089866966, + 2.3204589961096644, + 2.3355830053333193, + 2.353499992750585, + 2.308082999661565, + 2.2839580051368102, + 2.317250007763505, + 2.4272919981740415, + 2.8574579919222742, + 2.9272090032463893, + 2.89816599979531, + 2.860084001440555, + 2.8481670015025884, + 3.014749992871657, + 2.981125013320707, + 2.9359579930314794, + 2.9344579961616546, + 2.9020829970249906, + 2.9611249919980764, + 3.0218749889172614, + 3.0345830018632114, + 3.058542002690956, + 3.030582985957153, + 3.055041001061909, + 3.12008400214836, + 3.0464999872492626, + 3.071166996960528, + 3.1139169877860695, + 3.095583990216255 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "speedup_cached_vs_uncached_sdpa": 4.406964019711183, + "speedup_cached_vs_uncached_default": 5.744148266111253 + }, + { + "batch": 1, + "prompt_len": 512, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_default": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 5.961374990874901, + "p95": 7.071917192661203, + "min": 5.52179099759087, + "max": 7.225458990433253, + "iqr": 0.515209001605399, + "mean": 6.221783196087927, + "cv": 0.10474401469894878 + }, + "decode_ms_per_step": { + "count": 315, + "median": 0.520166999194771, + "p95": 0.7026876992313191, + "min": 0.48874999629333615, + "max": 0.981790988589637, + "iqr": 0.04018799518235028, + "mean": 0.5480874313636579, + "cv": 0.12905272334405576 + }, + "total_ms": { + "count": 5, + "median": 39.214409014675766, + "p95": 44.11618734593503, + "min": 37.85487396817189, + "max": 44.1681279335171, + "iqr": 5.297804047586396, + "mean": 40.75129137199838, + "cv": 0.07460852441793603 + }, + "tokens_per_second": { + "count": 5, + "median": 1632.0531561765567, + "p95": 1684.0488913962772, + "min": 1449.008662905847, + "max": 1690.6673643613435, + "iqr": 199.99595846579632, + "mean": 1577.376644809995, + "cv": 0.07303384147461324 + }, + "median_step_ms_by_index": [ + 5.961374990874901, + 0.7354160043178126, + 0.5853330076206475, + 0.556125000002794, + 0.5278339958749712, + 0.5203749897191301, + 0.5158330022823066, + 0.5117499968037009, + 0.5200000014156103, + 0.5237080040387809, + 0.5158749991096556, + 0.5364170065149665, + 0.5177499988349155, + 0.5215839919401333, + 0.5119580018799752, + 0.532958991243504, + 0.5378330097300932, + 0.5049999890616164, + 0.5345000099623576, + 0.5112500075483695, + 0.5015000060666353, + 0.5160420114407316, + 0.5142919981153682, + 0.516542000696063, + 0.5129159981152043, + 0.5147079937160015, + 0.5304160004016012, + 0.5525829910766333, + 0.5213340045884252, + 0.5241659964667633, + 0.510541009134613, + 0.5146670009708032, + 0.5117079999763519, + 0.5133329977979884, + 0.5082080024294555, + 0.5089579935884103, + 0.5092079954920337, + 0.513250008225441, + 0.5139999993843958, + 0.5138749984325841, + 0.5103750008856878, + 0.5130000063218176, + 0.5190410010982305, + 0.5388749996200204, + 0.5364579992601648, + 0.5281249905237928, + 0.5247079971013591, + 0.5195000121602789, + 0.5250829999567941, + 0.5613749963231385, + 0.5317909963196144, + 0.5377080087782815, + 0.5174999969312921, + 0.5232499970588833, + 0.5274160066619515, + 0.5183750035939738, + 0.5271249974612147, + 0.5212910036789253, + 0.5132079968461767, + 0.5217500001890585, + 0.5187500064494088, + 0.5387499986682087, + 0.5320420023053885, + 0.5157920095371082 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 5.90645799820777, + "p95": 6.2284500017995015, + "min": 5.598250005277805, + "max": 6.270375000895001, + "iqr": 0.18108400399796665, + "mean": 5.943099802243523, + "cv": 0.041663071728914876 + }, + "decode_ms_per_step": { + "count": 315, + "median": 5.998541004373692, + "p95": 6.724233699787874, + "min": 5.553959010285325, + "max": 20.4685420030728, + "iqr": 0.3272284957347438, + "mean": 6.166304930553047, + "cv": 0.18432427533248746 + }, + "total_ms": { + "count": 5, + "median": 392.83892305684276, + "p95": 416.18493778805714, + "min": 377.54191509156954, + "max": 419.0040879911976, + "iqr": 27.100047955173068, + "mean": 394.4203104270855, + "cv": 0.04531326610197369 + }, + "tokens_per_second": { + "count": 5, + "median": 162.91664660413338, + "p95": 169.49369803850067, + "min": 152.7431398267038, + "max": 169.51760173298203, + "iqr": 11.337618321634636, + "mean": 162.52718727266696, + "cv": 0.044783248008024065 + }, + "median_step_ms_by_index": [ + 5.90645799820777, + 6.0908750019734725, + 6.1453329981304705, + 6.148250002297573, + 6.007875010254793, + 5.83283300511539, + 6.0388749989215285, + 5.785707995528355, + 5.743250003433786, + 5.930374987656251, + 5.703207993065007, + 5.7697919983183965, + 5.91229199199006, + 5.689833997166716, + 5.812083996715955, + 5.976125001325272, + 5.788208000012673, + 5.948499994701706, + 5.887332998099737, + 6.033834011759609, + 5.82683298853226, + 5.914666995522566, + 5.8960839960491285, + 5.838750003022142, + 5.973708000965416, + 5.8878330019069836, + 6.074083998100832, + 5.819417012389749, + 5.997375003062189, + 5.920166993746534, + 5.9809579979628325, + 6.038207997335121, + 5.888708008569665, + 5.986166986986063, + 5.947750003542751, + 6.130750000011176, + 5.991875004838221, + 5.917499991483055, + 6.057000005966984, + 6.050291995052248, + 5.9956670011160895, + 6.03374998900108, + 6.000624998705462, + 6.11341699550394, + 6.06216600863263, + 5.962083989288658, + 6.008083000779152, + 5.982500006211922, + 5.951624989393167, + 6.026457995176315, + 6.092707990319468, + 6.055792007828131, + 6.014209007844329, + 6.058917002519593, + 6.019749998813495, + 5.948083009570837, + 5.9989169967593625, + 6.070791001548059, + 6.1192919965833426, + 6.052041993825696, + 6.088583002565429, + 6.089916991186328, + 6.056000012904406, + 6.186999991768971 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_default": { + "first_token_ms": { + "count": 5, + "median": 10.786208003992215, + "p95": 11.390174992266111, + "min": 10.017000007792376, + "max": 11.513249992276542, + "iqr": 0.5283749924274161, + "mean": 10.716766599216498, + "cv": 0.052831982647015246 + }, + "decode_ms_per_step": { + "count": 315, + "median": 11.577167009818368, + "p95": 13.631379495200235, + "min": 10.049165997770615, + "max": 25.41758300503716, + "iqr": 1.083146002201829, + "mean": 11.697615746137066, + "cv": 0.10132095376405886 + }, + "total_ms": { + "count": 5, + "median": 744.5167069963645, + "p95": 775.1115699968068, + "min": 721.0059560311493, + "max": 779.650545984623, + "iqr": 20.751748073962517, + "mean": 747.6665586058516, + "cv": 0.029617008595035035 + }, + "tokens_per_second": { + "count": 5, + "median": 85.96180501871869, + "p95": 88.39838325334004, + "min": 82.08805897669733, + "max": 88.76487006056722, + "iqr": 2.3832307396009753, + "mean": 85.65927507304897, + "cv": 0.02938494054438712 + }, + "median_step_ms_by_index": [ + 10.786208003992215, + 10.708750007324852, + 10.846666991710663, + 10.661707987310365, + 10.602124995784834, + 10.693040996557102, + 10.76137500058394, + 10.578666013316251, + 10.25645800109487, + 10.382833002950065, + 10.583875002339482, + 11.04529100120999, + 10.71145900641568, + 11.586292006541044, + 11.03891700040549, + 11.08379200741183, + 11.257542006205767, + 11.014334013452753, + 11.183707989403047, + 11.164084004121833, + 10.883333001402207, + 11.21220899221953, + 11.067958999774419, + 11.091209002188407, + 10.860666006919928, + 11.138874993775971, + 11.085457997978665, + 11.199750006198883, + 11.321875004796311, + 11.211207995074801, + 11.228125003981404, + 11.384583005565219, + 11.01833299617283, + 11.643750010989606, + 11.612917005550116, + 11.576457996852696, + 11.530250005307607, + 11.714332998963073, + 11.581708007724956, + 11.765290997573175, + 11.666749996948056, + 11.711125000147149, + 11.621541998465545, + 11.772208992624655, + 11.838375008665025, + 11.817333012004383, + 11.878958001034334, + 12.041291003697552, + 11.978084003203548, + 11.993416002951562, + 11.921416997211054, + 12.163292005425319, + 11.99241699941922, + 12.008916994091123, + 12.063500005751848, + 13.17150000249967, + 12.260709001566283, + 12.368791998596862, + 12.247582999407314, + 12.98641599714756, + 12.848042009864002, + 12.56216700130608, + 12.442624996765517, + 12.573125000926666 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "speedup_cached_vs_uncached_sdpa": 10.017718816311246, + "speedup_cached_vs_uncached_default": 18.985794398118646 + }, + { + "batch": 1, + "prompt_len": 1024, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_default": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 13.714082990190946, + "p95": 14.24025020096451, + "min": 12.721541992505081, + "max": 14.297166999313049, + "iqr": 0.4461250064196065, + "mean": 13.662366598146036, + "cv": 0.04365084508225801 + }, + "decode_ms_per_step": { + "count": 315, + "median": 0.7765420014038682, + "p95": 0.8937543039792217, + "min": 0.6295420025708154, + "max": 1.452583004720509, + "iqr": 0.164229997608345, + "mean": 0.764912013140034, + "cv": 0.13208343265954506 + }, + "total_ms": { + "count": 5, + "median": 64.13800103473477, + "p95": 66.31342838518322, + "min": 55.58479101455305, + "max": 66.47974497172982, + "iqr": 8.239743969170377, + "mean": 61.85182342596818, + "cv": 0.08086907290911066 + }, + "tokens_per_second": { + "count": 5, + "median": 997.8483733121019, + "p95": 1144.0790905372305, + "min": 962.6992406065289, + "max": 1151.394092374004, + "iqr": 139.92507226592807, + "mean": 1040.3309600813961, + "cv": 0.08324094802235907 + }, + "median_step_ms_by_index": [ + 13.714082990190946, + 1.1248750088270754, + 0.8929589967010543, + 0.861624997924082, + 0.8119580015772954, + 0.8055840007727966, + 0.8017910004127771, + 0.8152920054271817, + 0.7988750003278255, + 0.8170410001184791, + 0.7944999961182475, + 0.7875410083215684, + 0.7354580011451617, + 0.7875000010244548, + 0.8641660097055137, + 0.8239580056397244, + 0.7925420068204403, + 0.7800419989507645, + 0.7679590053157881, + 0.8317500032717362, + 0.8311249985126778, + 0.7719580025877804, + 0.7132089958759025, + 0.7726659969193861, + 0.7210420008050278, + 0.7806670037098229, + 0.7767080096527934, + 0.7949590071802959, + 0.772458006395027, + 0.7885829982114956, + 0.7977500063134357, + 0.7180829998105764, + 0.7867499953135848, + 0.8154160022968426, + 0.8025410061236471, + 0.8142500009853393, + 0.728458006051369, + 0.7119579968275502, + 0.7600840035593137, + 0.7709160126978531, + 0.7375830027740449, + 0.7279999990714714, + 0.7274999952642247, + 0.7361659954767674, + 0.7838329911464825, + 0.7490000134566799, + 0.7374170090770349, + 0.764124997658655, + 0.753250002162531, + 0.7439579931087792, + 0.7320419972529635, + 0.7455000013578683, + 0.7407920056721196, + 0.760458002332598, + 0.7502909866161644, + 0.737459005904384, + 0.7685409946134314, + 0.7539579964941368, + 0.7894589944044128, + 0.7717920088907704, + 0.7535000040661544, + 0.7397919980576262, + 0.8013750048121437, + 0.757250003516674 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 13.729707992752083, + "p95": 15.497091598808765, + "min": 12.275125001906417, + "max": 15.926624997518957, + "iqr": 1.265375001821667, + "mean": 13.644799799658358, + "cv": 0.1061259402160135 + }, + "decode_ms_per_step": { + "count": 315, + "median": 15.537625004071742, + "p95": 16.963878602837212, + "min": 11.885375002748333, + "max": 22.142125002574176, + "iqr": 3.362541494425386, + "mean": 14.745774733351677, + "cv": 0.12326360623062953 + }, + "total_ms": { + "count": 5, + "median": 1025.550540027325, + "p95": 1033.761608353234, + "min": 803.8472890475532, + "max": 1034.7981679515215, + "iqr": 210.28369694249704, + "mean": 942.628608000814, + "cv": 0.1270827769903264 + }, + "tokens_per_second": { + "count": 5, + "median": 62.40550563046339, + "p95": 79.31617889540706, + "min": 61.84780953632138, + "max": 79.61711244411991, + "iqr": 15.953310457560491, + "mean": 68.82840131089111, + "cv": 0.1333675617806749 + }, + "median_step_ms_by_index": [ + 13.729707992752083, + 15.343916995334439, + 15.385833990876563, + 15.36966698768083, + 15.718958005891182, + 15.402083008666523, + 15.534040998318233, + 15.330875001382083, + 15.220083005260676, + 15.495459010708146, + 15.36458299960941, + 15.406250007799827, + 15.24345799407456, + 15.33249999920372, + 15.53416698880028, + 15.502499998547137, + 15.46408300055191, + 15.513624995946884, + 15.550458003417589, + 15.534332997049205, + 15.531707991613075, + 15.715000001364388, + 15.401166005176492, + 15.473334002308547, + 15.408792009111494, + 15.622082995832898, + 15.514207989326678, + 15.575708006508648, + 15.442250005435199, + 15.600375001667999, + 15.519624997978099, + 15.42983298713807, + 15.594708995195106, + 15.743333991849795, + 15.757790999487042, + 15.69904200732708, + 15.626041989889927, + 15.76891599688679, + 15.964124992024153, + 15.880374994594604, + 15.816375002032146, + 15.734417000203393, + 15.61287499498576, + 15.805082992301323, + 15.921458005323075, + 15.75862499885261, + 15.738708010758273, + 16.059790999861434, + 15.723082993645221, + 15.944708007737063, + 15.827374998480082, + 15.679457996156998, + 15.808708994882181, + 15.757000001030974, + 16.365458999644034, + 16.086457995697856, + 15.991042004316114, + 16.029833001084626, + 15.795207989867777, + 16.113250007038005, + 16.05016599933151, + 15.95233399712015, + 16.11829201283399, + 16.187957997317426 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_default": { + "first_token_ms": { + "count": 5, + "median": 37.00416699575726, + "p95": 37.78725000738632, + "min": 36.59275000973139, + "max": 37.82550001051277, + "iqr": 0.9867920016404241, + "mean": 37.14082500082441, + "cv": 0.015199383669218032 + }, + "decode_ms_per_step": { + "count": 315, + "median": 37.595000001601875, + "p95": 40.74941690050764, + "min": 33.46162500383798, + "max": 53.562250002869405, + "iqr": 2.0885205012746155, + "mean": 37.98954578422357, + "cv": 0.06137876954009882 + }, + "total_ms": { + "count": 5, + "median": 2449.8742480645888, + "p95": 2488.3990258473204, + "min": 2359.1010449599707, + "max": 2489.283001064905, + "iqr": 115.57349700888153, + "mean": 2430.4822094069095, + "cv": 0.0257197277697974 + }, + "tokens_per_second": { + "count": 5, + "median": 26.123789843727806, + "p95": 27.105645471168017, + "min": 25.710214536724457, + "max": 27.12897785227591, + "iqr": 1.256370133587147, + "mean": 26.346248798522787, + "cv": 0.025874204732297245 + }, + "median_step_ms_by_index": [ + 37.00416699575726, + 38.03245798917487, + 37.351250008214265, + 37.12249999807682, + 36.705624996102415, + 36.712499990244396, + 36.94091700890567, + 36.867332994006574, + 36.435874993912876, + 36.72458400251344, + 36.786208002013154, + 36.71716699318495, + 36.878541999612935, + 37.19120800087694, + 36.97337499761488, + 37.04687500430737, + 36.28637500514742, + 37.20800000883173, + 36.993166999309324, + 37.19624999212101, + 37.13416600658093, + 37.515416988753714, + 37.464499997440726, + 37.26604100666009, + 37.24158299155533, + 37.276292001479305, + 37.27408299164381, + 37.2533750050934, + 37.528250002651475, + 37.719250001828186, + 38.70283399010077, + 37.37212500709575, + 36.99862500070594, + 38.3165840030415, + 38.17579099268187, + 38.53975000674836, + 38.30554099113215, + 38.73891700641252, + 38.877416009199806, + 38.503540999954566, + 38.3457910065772, + 38.956541000516154, + 38.72879200207535, + 39.24783300317358, + 38.96520799025893, + 39.3294999958016, + 38.72349999437574, + 38.730875006876886, + 39.19754199159797, + 38.91674999613315, + 38.79470800166018, + 39.01562499231659, + 38.98891700373497, + 39.256832998944446, + 39.28050000104122, + 39.56758399726823, + 39.103415998397395, + 40.19925001193769, + 39.59454200230539, + 39.5185420056805, + 39.47720800351817, + 40.69691699987743, + 39.8921669984702, + 39.7979589906754 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "speedup_cached_vs_uncached_sdpa": 15.989749033056468, + "speedup_cached_vs_uncached_default": 38.19692239453842 + }, + { + "batch": 8, + "prompt_len": 1, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_default": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 0.5971659993520007, + "p95": 0.603458602563478, + "min": 0.5729999975301325, + "max": 0.6047920032870024, + "iqr": 0.007624999852851033, + "mean": 0.5927165999310091, + "cv": 0.020464630566885928 + }, + "decode_ms_per_step": { + "count": 315, + "median": 0.6064579938538373, + "p95": 0.699395700939931, + "min": 0.47533299948554486, + "max": 0.8455000061076134, + "iqr": 0.11431249731685966, + "mean": 0.6084561968843142, + "cv": 0.12403018063502266 + }, + "total_ms": { + "count": 5, + "median": 38.72037504334003, + "p95": 39.78721502935514, + "min": 38.49446098320186, + "max": 40.01666602562182, + "iqr": 0.3430391225265339, + "mean": 38.925457003642805, + "cv": 0.016150917062585018 + }, + "tokens_per_second": { + "count": 5, + "median": 13223.01241728455, + "p95": 13298.411517271192, + "min": 12794.669092926866, + "max": 13300.61486569264, + "iqr": 117.28636880683916, + "mean": 13156.041250853601, + "cv": 0.015856873420501943 + }, + "median_step_ms_by_index": [ + 0.5971659993520007, + 0.5841250094817951, + 0.558999992790632, + 0.5601669981842861, + 0.5617499991785735, + 0.5585000035353005, + 0.5558749980991706, + 0.5582500016316772, + 0.5464999994728714, + 0.49737500376068056, + 0.47866599925328046, + 0.4913339944323525, + 0.4869580006925389, + 0.4824170027859509, + 0.4859159962506965, + 0.4811249964404851, + 0.4898750048596412, + 0.4884999943897128, + 0.48879200767260045, + 0.4943750100210309, + 0.5024999991292134, + 0.5035000067437068, + 0.5698339955415577, + 0.5732499994337559, + 0.5775830068159848, + 0.5737500032410026, + 0.5708750104531646, + 0.5759170016972348, + 0.5908750026719645, + 0.5895420035813004, + 0.589667004533112, + 0.5988749908283353, + 0.5908750026719645, + 0.5969579942757264, + 0.6105419888626784, + 0.6094589916756377, + 0.6170419947011396, + 0.6472920067608356, + 0.6615419988520443, + 0.6636670004809275, + 0.6621250067837536, + 0.6777079979656264, + 0.6699579971609637, + 0.6617499893764034, + 0.6738749943906441, + 0.6701670063193887, + 0.6691250018775463, + 0.6617909966735169, + 0.6680000078631565, + 0.6824169977335259, + 0.6809999904362485, + 0.6769579922547564, + 0.6761250115232542, + 0.6853340019006282, + 0.6790830084355548, + 0.677084011840634, + 0.6798750109737739, + 0.6790409970562905, + 0.6995419971644878, + 0.6893329991726205, + 0.6847090116934851, + 0.6923339969944209, + 0.6893750105518848, + 0.7062500080792233 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 0.5507079913513735, + "p95": 0.6738166004652157, + "min": 0.5223330081207678, + "max": 0.6937500002095476, + "iqr": 0.07149900193326175, + "mean": 0.5766916001448408, + "cv": 0.12432748639845266 + }, + "decode_ms_per_step": { + "count": 315, + "median": 2.353416013647802, + "p95": 3.6694206937681884, + "min": 0.534083999809809, + "max": 4.010000004200265, + "iqr": 1.7233125036000274, + "mean": 2.305482901653112, + "cv": 0.4119506441157358 + }, + "total_ms": { + "count": 5, + "median": 145.19262297835667, + "p95": 147.70903963362798, + "min": 143.99899706768338, + "max": 147.7952480490785, + "iqr": 2.6047080173157156, + "mean": 145.82211440429091, + "cv": 0.011434264623410679 + }, + "tokens_per_second": { + "count": 5, + "median": 3526.3499584019637, + "p95": 3551.8444375269733, + "min": 3464.252110663123, + "max": 3555.580319488936, + "iqr": 62.515819870490304, + "mean": 3511.493677608355, + "cv": 0.011407777050188824 + }, + "median_step_ms_by_index": [ + 0.5507079913513735, + 0.5609159998130053, + 0.5960830021649599, + 0.6232079904293641, + 0.7535420008935034, + 0.891707997652702, + 0.9120419999817386, + 0.9485839982517064, + 1.0863339994102716, + 1.1085419973824173, + 1.141750006354414, + 1.150915995822288, + 1.2702499952865764, + 1.3105830003041774, + 1.3481249916367233, + 1.3329159992281348, + 1.4025840064277872, + 1.416541010257788, + 1.459833001717925, + 1.4488329907180741, + 1.6050420090323314, + 2.0465419947868213, + 2.074874995741993, + 2.077083001495339, + 2.188915997976437, + 2.1242080110823736, + 2.1365410066209733, + 2.197458001319319, + 2.2884580102982, + 2.2453749988926575, + 2.2522500075865537, + 2.2147080017020926, + 2.3393329902319238, + 2.379792000283487, + 2.4489999923389405, + 2.464457997120917, + 2.6420829963171855, + 2.5899579923134297, + 2.696249997825362, + 2.6845829997910187, + 2.780666996841319, + 2.877916005672887, + 2.8423339972505346, + 2.8738750115735456, + 3.0640419863630086, + 3.0347920110216364, + 3.0692500004079193, + 2.9962920089019462, + 3.1714170036138967, + 3.1049169920152053, + 3.194707998773083, + 3.1548330007353798, + 3.261125006247312, + 3.354542001034133, + 3.3413339988328516, + 3.3279170020250604, + 3.490583010716364, + 3.5256249975645915, + 3.5097920044790953, + 3.4928329987451434, + 3.6549999931594357, + 3.6677079915534705, + 3.716708000865765, + 3.6893750075250864 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_default": { + "first_token_ms": { + "count": 5, + "median": 0.6758749950677156, + "p95": 0.7174585975008085, + "min": 0.5717910098610446, + "max": 0.726416998077184, + "iqr": 0.009249997674487531, + "mean": 0.6656165991444141, + "cv": 0.08534151551701089 + }, + "decode_ms_per_step": { + "count": 315, + "median": 2.8572919982252643, + "p95": 4.228258805233054, + "min": 0.5884589918423444, + "max": 4.452625013072975, + "iqr": 1.814124989323318, + "mean": 2.6947598762547833, + "cv": 0.38517133066732356 + }, + "total_ms": { + "count": 5, + "median": 168.65579305158462, + "p95": 174.93963178130798, + "min": 166.97282501263544, + "max": 175.09178999171127, + "iqr": 7.204961919342168, + "mean": 170.43548880319577, + "cv": 0.023278924833497826 + }, + "tokens_per_second": { + "count": 5, + "median": 3035.76883269821, + "p95": 3065.80502092658, + "min": 2924.18051140055, + "max": 3066.3672364724926, + "iqr": 126.61434624799267, + "mean": 3005.362910361823, + "cv": 0.02311546876366952 + }, + "median_step_ms_by_index": [ + 0.6758749950677156, + 0.6082500040065497, + 0.6434160022763535, + 0.813459002529271, + 0.9582499915268272, + 1.1383339879103005, + 1.1937079980270937, + 1.1787919938797131, + 1.3292910007294267, + 1.4151669893180951, + 1.4887500001350418, + 1.4590000064345077, + 1.600624993443489, + 1.6792910028016195, + 1.8125829956261441, + 1.6245839942712337, + 1.622416006284766, + 1.675040999543853, + 1.7528750031488016, + 1.7125830054283142, + 1.7992909997701645, + 2.4242079962277785, + 2.38470800104551, + 2.3199169954750687, + 2.424457998131402, + 2.451749998726882, + 2.533375009079464, + 2.4896660033846274, + 2.8085420053685084, + 2.6512920012464747, + 2.6228749920846894, + 2.656041004229337, + 2.813957995385863, + 2.8354169917292893, + 2.8572919982252643, + 2.911625007982366, + 2.9369160038186237, + 3.012417000718415, + 3.037999995285645, + 3.010165994055569, + 3.1270839972421527, + 3.186290996382013, + 3.1652500038035214, + 3.1703749991720542, + 3.288458989118226, + 3.3304589887848124, + 3.4231250028824434, + 3.345000004628673, + 3.5324999917065725, + 3.628375008702278, + 3.6055829987162724, + 3.666832999442704, + 3.7653330073226243, + 3.7663330003852025, + 3.873041001497768, + 3.861832999973558, + 3.951167003833689, + 4.079291000380181, + 4.051083000376821, + 4.043332999572158, + 4.236250009853393, + 4.278958003851585, + 4.261666996171698, + 4.393416995299049 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "speedup_cached_vs_uncached_sdpa": 3.749773157306493, + "speedup_cached_vs_uncached_default": 4.355737589390775 + }, + { + "batch": 8, + "prompt_len": 16, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_default": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 1.4437079953495413, + "p95": 1.5172831976087764, + "min": 1.321207993896678, + "max": 1.532207999844104, + "iqr": 0.048250993131659925, + "mean": 1.432808194658719, + "cv": 0.05363021577187168 + }, + "decode_ms_per_step": { + "count": 315, + "median": 0.6479590083472431, + "p95": 0.7285544066689909, + "min": 0.5122500006109476, + "max": 1.0019159963121638, + "iqr": 0.08366599649889395, + "mean": 0.652496660514439, + "cv": 0.08842521842206326 + }, + "total_ms": { + "count": 5, + "median": 42.86066499480512, + "p95": 43.051505167386495, + "min": 41.88312002224848, + "max": 43.098495953017846, + "iqr": 0.8688759844517335, + "mean": 42.54009780706838, + "cv": 0.013132022884763414 + }, + "tokens_per_second": { + "count": 5, + "median": 11945.684931907996, + "p95": 12218.0010794191, + "min": 11879.764912402905, + "max": 12224.495207807431, + "iqr": 247.14143644457, + "mean": 12037.370549481062, + "cv": 0.013183317804300461 + }, + "median_step_ms_by_index": [ + 1.4437079953495413, + 0.6401660066330805, + 0.5903330020373687, + 0.5768749979324639, + 0.5760420026490465, + 0.5758749903179705, + 0.575708007090725, + 0.608083006227389, + 0.5850000015925616, + 0.573874989640899, + 0.5717499880120158, + 0.5882080004084855, + 0.5837920034537092, + 0.5908330058446154, + 0.5986250034766272, + 0.5935829976806417, + 0.5947919999016449, + 0.5986670003039762, + 0.6018340063747019, + 0.6025000038789585, + 0.6155000010039657, + 0.6108339875936508, + 0.6129579996922985, + 0.6104999920353293, + 0.626042005023919, + 0.6374170043272898, + 0.6382080027833581, + 0.6377499958034605, + 0.6450829969253391, + 0.634916010312736, + 0.6185419915709645, + 0.6244579999474809, + 0.6345419969875365, + 0.6322499975794926, + 0.6296670035226271, + 0.6293340120464563, + 0.6406249885912985, + 0.635749995126389, + 0.6369159964378923, + 0.6637499900534749, + 0.6908750074217096, + 0.6763749988749623, + 0.6844160088803619, + 0.7025840022834018, + 0.6938750011613593, + 0.6835839885752648, + 0.6927079957677051, + 0.701457989634946, + 0.7002080092206597, + 0.6907080096425489, + 0.6884159956825897, + 0.6925420020706952, + 0.6928329967195168, + 0.6925409979885444, + 0.6936249992577359, + 0.7082919910317287, + 0.6976250006118789, + 0.7014580041868612, + 0.7049589912639931, + 0.7095420005498454, + 0.7042499928502366, + 0.7127499993657693, + 0.7254579977598041, + 0.7402089977404103 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 1.3431250117719173, + "p95": 1.4394668047316372, + "min": 1.3127499987604097, + "max": 1.4398750063264742, + "iqr": 0.1010839914670214, + "mean": 1.3740668044192716, + "cv": 0.043827421249312215 + }, + "decode_ms_per_step": { + "count": 315, + "median": 3.107625001575798, + "p95": 4.370028502307832, + "min": 1.3109999999869615, + "max": 4.753707995405421, + "iqr": 1.475270495575387, + "mean": 3.0594336727286673, + "cv": 0.29048071571295536 + }, + "total_ms": { + "count": 5, + "median": 192.9090000485303, + "p95": 197.41262379684485, + "min": 191.6892010049196, + "max": 197.6724990090588, + "iqr": 4.425005026860163, + "mean": 194.11838818632532, + "cv": 0.014058292343790765 + }, + "tokens_per_second": { + "count": 5, + "median": 2654.1011558361492, + "p95": 2670.269743784365, + "min": 2590.1427996644916, + "max": 2670.9903182645107, + "iqr": 60.105999636503384, + "mean": 2637.9806331712425, + "cv": 0.013986340808981621 + }, + "median_step_ms_by_index": [ + 1.3431250117719173, + 1.3719999988097697, + 1.4076249935897067, + 1.3923750084359199, + 1.421791996108368, + 1.5185829979600385, + 1.948208999237977, + 1.9375830015633255, + 1.938291999977082, + 2.042416002950631, + 2.0377909968374297, + 2.0537919917842373, + 2.058666999801062, + 2.189040998928249, + 2.2457500017480925, + 2.2655840002698824, + 2.215000000433065, + 2.3827080003684387, + 2.3788749967934564, + 2.46829200477805, + 2.5171669985866174, + 2.639166996232234, + 2.684874998521991, + 2.755458001047373, + 2.670917005161755, + 2.8065830119885504, + 2.8769579948857427, + 2.864958005375229, + 2.8701249975711107, + 3.036082998733036, + 3.0395840003620833, + 3.1276249937945977, + 2.984582999488339, + 3.122541995253414, + 3.1587920093443245, + 3.1793749949429184, + 3.176832993631251, + 3.2565419969614595, + 3.3487919863546267, + 3.365333002875559, + 3.324459001305513, + 3.5136669903295115, + 3.525332998833619, + 3.559834003681317, + 3.5252500092610717, + 3.678875000332482, + 3.6547920026350766, + 3.721541987033561, + 3.725292001035996, + 3.8629170012427494, + 3.933041007257998, + 4.022624998469837, + 3.9480829873355106, + 4.083374995389022, + 4.163332996540703, + 4.238834007992409, + 4.121624995605089, + 4.275500003132038, + 4.326333000790328, + 4.281625006115064, + 4.256915999576449, + 4.41595799929928, + 4.382540995720774, + 4.42904100054875 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_default": { + "first_token_ms": { + "count": 5, + "median": 1.7623330058995634, + "p95": 1.8657420005183667, + "min": 1.7319999897154048, + "max": 1.8657920008990914, + "iqr": 0.13020799087826163, + "mean": 1.7922002007253468, + "cv": 0.037992057618951976 + }, + "decode_ms_per_step": { + "count": 315, + "median": 3.4982499928446487, + "p95": 5.190170602872968, + "min": 1.6879159957170486, + "max": 5.598333998932503, + "iqr": 1.5830210031708702, + "mean": 3.575833707711556, + "cv": 0.2815922363375796 + }, + "total_ms": { + "count": 5, + "median": 225.90891804429702, + "p95": 232.0411907567177, + "min": 223.71124304481782, + "max": 233.06612395390403, + "iqr": 3.220582046196796, + "mean": 227.06972378655337, + "cv": 0.016309527677596417 + }, + "tokens_per_second": { + "count": 5, + "median": 2266.4001245829754, + "p95": 2286.608071913843, + "min": 2196.801454085467, + "max": 2288.664588473218, + "iqr": 32.19123123661893, + "mean": 2255.287789451546, + "cv": 0.01610082651893686 + }, + "median_step_ms_by_index": [ + 1.7623330058995634, + 1.780749997124076, + 1.7670410015853122, + 1.786833003279753, + 1.8494170071789995, + 1.8803339917212725, + 2.33762500283774, + 2.358750003622845, + 2.4386669974774122, + 2.483707998180762, + 2.5135839969152585, + 2.5615000049583614, + 2.456332993460819, + 2.569874995970167, + 2.6489589945413172, + 2.62904200644698, + 2.784500000416301, + 2.7636669983621687, + 2.821957998094149, + 2.8436249995138496, + 2.902332998928614, + 2.935458003776148, + 2.9803749930579215, + 3.006207989528775, + 3.002375000505708, + 3.144666989101097, + 3.151290991809219, + 3.1860000017331913, + 3.2051250018412247, + 3.3026250021066517, + 3.326874997583218, + 3.4338749974267557, + 3.4534590085968375, + 3.5518749937182292, + 3.6389160086400807, + 3.6734169989358634, + 3.709875003551133, + 3.736915998160839, + 3.8572919875150546, + 3.8809160032542422, + 3.853125002933666, + 4.08587499987334, + 4.070125010912307, + 3.999708002083935, + 4.09074999333825, + 4.247042001225054, + 4.232666004099883, + 4.329666000558063, + 4.360667007858865, + 4.484624994802289, + 4.5443749986588955, + 4.566167001030408, + 4.598917003022507, + 4.633999997167848, + 4.858375003095716, + 4.747917002532631, + 4.83449999592267, + 5.006458988646045, + 4.966708002029918, + 5.029957988881506, + 5.10533299529925, + 5.2711669995915145, + 5.2545409998856485, + 5.426916992291808 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "speedup_cached_vs_uncached_sdpa": 4.500840107635092, + "speedup_cached_vs_uncached_default": 5.2707749184871036 + }, + { + "batch": 8, + "prompt_len": 128, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_default": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 7.879374999902211, + "p95": 10.192724806256592, + "min": 7.649499995750375, + "max": 10.442708007758483, + "iqr": 1.4193340030033141, + "mean": 8.587566600181162, + "cv": 0.1408499399268735 + }, + "decode_ms_per_step": { + "count": 315, + "median": 0.9888750064419582, + "p95": 1.4986122943810192, + "min": 0.8331659919349477, + "max": 3.616582995164208, + "iqr": 0.1655624946579337, + "mean": 1.0721866760702272, + "cv": 0.2743284191499413 + }, + "total_ms": { + "count": 5, + "median": 73.07558505272027, + "p95": 93.96435717935674, + "min": 66.04095493094064, + "max": 98.72929997800384, + "iqr": 6.978375968174078, + "mean": 76.13532719260547, + "cv": 0.1725889119641835 + }, + "tokens_per_second": { + "count": 5, + "median": 7006.444076097624, + "p95": 7709.730183636607, + "min": 5185.897196820699, + "max": 7752.764940110284, + "iqr": 702.2286326207523, + "mean": 6863.611979178329, + "cv": 0.14715945546843595 + }, + "median_step_ms_by_index": [ + 7.879374999902211, + 1.1925840080948547, + 1.0166249994654208, + 1.000499993097037, + 0.941624995903112, + 0.9537500009173527, + 0.9542089974274859, + 0.9308330045314506, + 0.9000420104712248, + 0.9440409921808168, + 0.980041004368104, + 0.9239579958375543, + 0.9682499949121848, + 1.0018750035669655, + 0.9583339997334406, + 0.9425000025657937, + 0.9466670016990975, + 0.9507500071777031, + 1.0123749962076545, + 0.9621250064810738, + 0.9748340089572594, + 0.9729589946800843, + 0.9870409994618967, + 0.9478749998379499, + 0.9282499959226698, + 0.9489170042797923, + 0.9477920102654025, + 0.9459589928155765, + 0.971500005107373, + 1.0402909974800423, + 0.9429999918211251, + 0.9370420011691749, + 0.9311659960076213, + 0.9406249882886186, + 0.9776250080903992, + 0.9963749907910824, + 1.051541999913752, + 1.0100420040544122, + 1.0793339897645637, + 0.9902500023599714, + 0.9470829972997308, + 0.9672919986769557, + 1.0013749997597188, + 0.9829999908106402, + 1.0198749951086938, + 1.1237919970881194, + 0.967875006608665, + 1.014209003187716, + 0.9802499989746138, + 1.076250002370216, + 1.0534590110182762, + 1.0007500095525756, + 1.007541999570094, + 1.0533340100664645, + 1.0080000065499917, + 1.0532919986872002, + 1.0084590030601248, + 1.080457994248718, + 1.027125006658025, + 1.054749998729676, + 1.0352079989388585, + 1.0386249923612922, + 1.0416669974802062, + 1.0342500027036294 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 7.975000000442378, + "p95": 8.583658206043765, + "min": 7.131999998819083, + "max": 8.595125007559545, + "iqr": 0.8987070032162592, + "mean": 7.975800000713207, + "cv": 0.07741025570651701 + }, + "decode_ms_per_step": { + "count": 315, + "median": 10.358040992286988, + "p95": 13.362892002623994, + "min": 7.334624999202788, + "max": 28.604833991266787, + "iqr": 2.9846044999430887, + "mean": 10.576215104961266, + "cv": 0.24054737033320833 + }, + "total_ms": { + "count": 5, + "median": 664.4339180202223, + "p95": 702.2733180550858, + "min": 653.2350869238144, + "max": 705.0412510579918, + "iqr": 33.72667002258822, + "mean": 674.2773516132729, + "cv": 0.03361401176797972 + }, + "tokens_per_second": { + "count": 5, + "median": 770.580769755973, + "p95": 782.7804536740343, + "min": 726.1986433158171, + "max": 783.7913337005328, + "iqr": 37.99789254418761, + "mean": 760.0093442728431, + "cv": 0.03317320438194293 + }, + "median_step_ms_by_index": [ + 7.975000000442378, + 7.900084005086683, + 7.964207994518802, + 8.163124992279336, + 8.152042006258853, + 8.06262499827426, + 8.131041991873644, + 8.052207995206118, + 7.9132499959087, + 8.329042000696063, + 8.289165998576209, + 8.542915995349176, + 8.302125002956018, + 8.660624996991828, + 8.26654100092128, + 8.966915993369184, + 8.612458012066782, + 8.729124994715676, + 8.48633398709353, + 8.62675000098534, + 8.61758399696555, + 8.66225000936538, + 9.318500000517815, + 10.338583000702783, + 10.05329200415872, + 10.225249992799945, + 9.82229199144058, + 9.78995900368318, + 10.494292000657879, + 9.75812500109896, + 9.449207995203324, + 10.00791700789705, + 10.12787499348633, + 10.43704200128559, + 10.18866700178478, + 10.628957999870181, + 10.543875003349967, + 10.53124999452848, + 10.358040992286988, + 11.052125002606772, + 10.761166995507665, + 10.686665991670452, + 11.318874996504746, + 11.388167011318728, + 11.892792012076825, + 11.553165997611359, + 11.728792000212707, + 11.746791991754435, + 11.825416993815452, + 11.869542009662837, + 12.170334011898376, + 12.09654200647492, + 11.568374990019947, + 12.103583998396061, + 12.655708007514477, + 12.7983749989653, + 12.520666001364589, + 12.643458001548424, + 12.5207910023164, + 12.743207989842631, + 12.737042008666322, + 13.126374993589707, + 13.432749998173676, + 12.964709007064812 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_default": { + "first_token_ms": { + "count": 5, + "median": 9.087250000447966, + "p95": 9.786400003940798, + "min": 8.899417007341981, + "max": 9.950000006938353, + "iqr": 0.06525000208057463, + "mean": 9.227083399309777, + "cv": 0.04482505248680488 + }, + "decode_ms_per_step": { + "count": 315, + "median": 13.019042002269998, + "p95": 17.888899899844546, + "min": 9.156916014035232, + "max": 33.7974169960944, + "iqr": 4.620687504939269, + "mean": 13.57750529811407, + "cv": 0.2538334414750088 + }, + "total_ms": { + "count": 5, + "median": 852.5214599794708, + "p95": 907.5371853716206, + "min": 838.3506639511324, + "max": 918.2185019599274, + "iqr": 15.66487802483607, + "mean": 864.6099171804963, + "cv": 0.03633932953045135 + }, + "tokens_per_second": { + "count": 5, + "median": 600.5713920824107, + "p95": 609.1699685848212, + "min": 557.6014847306404, + "max": 610.7229611854217, + "iqr": 10.92175453172274, + "mean": 592.7780159663178, + "cv": 0.03502578714398366 + }, + "median_step_ms_by_index": [ + 9.087250000447966, + 9.20149999728892, + 9.288542001741007, + 9.246207991964184, + 9.206292001181282, + 9.61624999763444, + 9.680958988610655, + 9.706416007247753, + 9.6257919940399, + 9.954542008927092, + 9.800957996048965, + 10.09199999680277, + 10.204332997091115, + 10.432291004690342, + 10.404583998024464, + 11.418333000619896, + 10.660874992026947, + 12.354667007457465, + 11.542416003067046, + 11.249999995925464, + 11.08491700142622, + 11.930708002182655, + 11.728208002750762, + 11.784040994825773, + 11.864166997838765, + 12.108166003599763, + 12.094166988390498, + 12.119916995288804, + 12.844791999668814, + 12.697291997028515, + 12.435750002623536, + 12.725833003059961, + 12.828417005948722, + 12.874209001893178, + 13.384749996475875, + 13.223916990682483, + 13.484249997418374, + 13.434417007374577, + 14.078709005843848, + 14.383417001226917, + 13.901124999392778, + 14.189499997883104, + 14.105666006798856, + 14.23225000326056, + 14.421208994463086, + 15.48012500279583, + 15.08125000691507, + 15.076875002705492, + 15.159082991885953, + 15.740832997835241, + 15.486499993130565, + 15.51287500478793, + 15.944915998261422, + 16.004541001166217, + 16.064458002801985, + 16.538959011086263, + 16.216082993196324, + 16.54833300563041, + 16.831208995427005, + 16.9417919969419, + 16.98170900635887, + 17.540959001053125, + 17.676666990155354, + 17.91404199320823 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "speedup_cached_vs_uncached_sdpa": 9.092420095451407, + "speedup_cached_vs_uncached_default": 11.666296744178243 + }, + { + "batch": 8, + "prompt_len": 512, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_default": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 46.791207991191186, + "p95": 51.53039160359185, + "min": 44.48199999751523, + "max": 52.301250005257316, + "iqr": 1.9772079976974055, + "mean": 47.698233198025264, + "cv": 0.06150362859898716 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.6927910037338734, + "p95": 2.478008600883186, + "min": 1.5373339992947876, + "max": 5.064292010501958, + "iqr": 0.2341465005883947, + "mean": 1.8198236828163592, + "cv": 0.20105001169589165 + }, + "total_ms": { + "count": 5, + "median": 159.21708100358956, + "p95": 185.0051132205408, + "min": 147.66267001687083, + "max": 190.00957801472396, + "iqr": 15.128211045521311, + "mean": 162.3471252154559, + "cv": 0.10460351574421989 + }, + "tokens_per_second": { + "count": 5, + "median": 3215.735376962833, + "p95": 3457.1987557655475, + "min": 2694.6010056415407, + "max": 3467.3624683984294, + "iqr": 313.2738801201326, + "mean": 3179.502556270142, + "cv": 0.09710631008551725 + }, + "median_step_ms_by_index": [ + 46.791207991191186, + 2.3207500053104013, + 1.979583001229912, + 1.6948750126175582, + 1.8152919947169721, + 1.7721250042086467, + 1.6911660059122369, + 1.65033400116954, + 1.6281249991152436, + 1.635084001463838, + 1.7389159911544994, + 1.5770419995533302, + 1.6258749965345487, + 1.5777920052642003, + 1.6805420018499717, + 1.6230420005740598, + 1.6905420052353293, + 1.6057920001912862, + 1.7090840119635686, + 1.6054580046329647, + 1.6985830006888136, + 1.658499997574836, + 1.705290997051634, + 1.69070900301449, + 1.699583008303307, + 1.7058329976862296, + 1.6927910037338734, + 1.6113750025397167, + 1.6875000001164153, + 1.6019169997889549, + 1.6878330061445013, + 1.6705839952919632, + 1.7503749986644834, + 1.692207995802164, + 1.7029170121531934, + 1.7117499955929816, + 1.69162500242237, + 1.7239579901797697, + 1.6935829917201772, + 1.6671250050421804, + 1.6991250013234094, + 1.6973750025499612, + 1.6999159997794777, + 1.6622919938527048, + 1.7870000010589138, + 1.6460840124636889, + 1.7091249901568517, + 1.6838750016177073, + 1.6675000078976154, + 1.7895829951157793, + 1.785166998161003, + 1.6762080049375072, + 1.8981659959536046, + 1.6602090036030859, + 1.7967909952858463, + 1.676584011875093, + 1.647625002078712, + 1.6648750024614856, + 1.8103329930454493, + 1.8012079963227734, + 1.8250419961987063, + 1.720582993584685, + 1.811666996218264, + 1.7300409963354468 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 53.90308400092181, + "p95": 56.86023359885439, + "min": 42.30404099507723, + "max": 57.40954200155102, + "iqr": 10.867832985240966, + "mean": 50.414966797688976, + "cv": 0.13625393329348168 + }, + "decode_ms_per_step": { + "count": 315, + "median": 49.28004200337455, + "p95": 58.592237696575474, + "min": 44.25858399190474, + "max": 72.92691701150034, + "iqr": 6.279000001086388, + "mean": 50.80759088898684, + "cv": 0.0986659744517202 + }, + "total_ms": { + "count": 5, + "median": 3143.185751978308, + "p95": 3577.891319221817, + "min": 3058.6009160469985, + "max": 3651.018337041023, + "iqr": 167.10553693701513, + "mean": 3251.2931928038597, + "cv": 0.0733431703458986 + }, + "tokens_per_second": { + "count": 5, + "median": 162.89205933112586, + "p95": 166.75607874245327, + "min": 140.23484757815586, + "max": 167.39679809607844, + "iqr": 8.35141321380911, + "mean": 158.11173888949125, + "cv": 0.06861462390390524 + }, + "median_step_ms_by_index": [ + 53.90308400092181, + 57.2782079980243, + 51.035041993600316, + 48.52483300783206, + 46.26704200927634, + 46.471833004034124, + 46.305832991492935, + 45.95337499631569, + 46.099458006210625, + 46.908624994102865, + 46.65358300553635, + 46.20287498983089, + 45.49245900125243, + 47.04154199862387, + 46.41349999292288, + 47.32691700337455, + 47.00474999845028, + 47.188833996187896, + 48.57658299442846, + 47.11025000142399, + 46.875542000634596, + 47.55754199868534, + 48.39512499165721, + 49.772458005463704, + 47.44441600632854, + 47.67637499026023, + 47.92625000118278, + 47.81837499467656, + 47.57141700247303, + 48.28874999657273, + 50.18429100164212, + 50.78041699016467, + 47.629250009777024, + 49.20762500842102, + 48.59645900432952, + 49.85779199341778, + 49.13183300232049, + 48.48166600277182, + 48.28558399458416, + 48.92308299895376, + 49.167708013555966, + 49.19854199397378, + 50.00079199089669, + 49.404791003325954, + 49.43779199675191, + 49.5898329972988, + 49.73037498712074, + 51.88937499769963, + 48.86691700085066, + 50.121583000873215, + 53.19154199969489, + 50.64275000768248, + 49.74620799475815, + 52.08887500339188, + 50.70175000582822, + 49.710541003150865, + 50.44562500552274, + 50.32583400316071, + 51.525584000046365, + 50.79716700129211, + 51.09354099840857, + 51.838292012689635, + 51.157875001081266, + 50.524374993983656 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_default": { + "first_token_ms": { + "count": 5, + "median": 85.37299999443348, + "p95": 95.60804219509009, + "min": 77.88404199527577, + "max": 95.74520899332128, + "iqr": 16.30200000363402, + "mean": 86.56380019674543, + "cv": 0.0990678052204921 + }, + "decode_ms_per_step": { + "count": 315, + "median": 93.68737500335556, + "p95": 114.27888399630321, + "min": 78.53287500620354, + "max": 130.3154170018388, + "iqr": 12.098937499104068, + "mean": 94.89229286670692, + "cv": 0.1019409888308987 + }, + "total_ms": { + "count": 5, + "median": 6026.178748055827, + "p95": 6671.104116347851, + "min": 5575.724833077402, + "max": 6808.216248944518, + "iqr": 331.53974800370634, + "mean": 6064.7782507992815, + "cv": 0.07696880022641296 + }, + "tokens_per_second": { + "count": 5, + "median": 84.96263078243108, + "p95": 91.14355820215575, + "min": 75.20325167100496, + "max": 91.82662619263665, + "iqr": 4.787441519327032, + "mean": 84.80552792144196, + "cv": 0.07362065070333071 + }, + "median_step_ms_by_index": [ + 85.37299999443348, + 88.02683299290948, + 85.68654100236017, + 82.14408300409559, + 89.92374999797903, + 85.84758400684223, + 84.68820800771937, + 85.49808400857728, + 85.31429199501872, + 86.94433400523849, + 85.17933401162736, + 85.80349999829195, + 85.6271250086138, + 87.08883399958722, + 86.20641700690612, + 88.4107499878155, + 86.97420799580868, + 88.22054200572893, + 88.90158300346229, + 89.5094169973163, + 88.9679999963846, + 90.02291699289344, + 90.15954101050738, + 90.49720800248906, + 89.44866601086687, + 90.39466599642765, + 91.60091599915177, + 90.86879099777434, + 91.53883399267215, + 92.27012499468401, + 91.61079200566746, + 91.6020829899935, + 92.06100000301376, + 92.78766599891242, + 93.5239999962505, + 93.03958399686962, + 92.9069999983767, + 94.00562499649823, + 95.13262499240227, + 94.16766600043047, + 94.24166599637829, + 97.89779200218618, + 97.3020829987945, + 96.65887500159442, + 96.80862500681542, + 95.32687500177417, + 95.54058400681242, + 97.17695800645743, + 96.44487500190735, + 98.08412499842234, + 98.4138749918202, + 97.9067500011297, + 97.0724580110982, + 98.23037499154452, + 107.67512500751764, + 100.60574999079108, + 101.35666700080037, + 106.36145899479743, + 101.97424999205396, + 101.56850000203121, + 102.39883301255759, + 102.89599999669008, + 103.2176669978071, + 101.97670900379308 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "speedup_cached_vs_uncached_sdpa": 19.741510974613615, + "speedup_cached_vs_uncached_default": 37.84882067973578 + }, + { + "batch": 8, + "prompt_len": 1024, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_default": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 137.40829199377913, + "p95": 145.68345060106367, + "min": 117.0921669981908, + "max": 146.01345900155138, + "iqr": 14.623250986915082, + "mean": 134.92350020096637, + "cv": 0.08793746377122724 + }, + "decode_ms_per_step": { + "count": 315, + "median": 4.38383300206624, + "p95": 6.770133200916463, + "min": 2.7592080004978925, + "max": 9.94225000613369, + "iqr": 1.3731665021623485, + "mean": 4.317311778071425, + "cv": 0.29037138278067615 + }, + "total_ms": { + "count": 5, + "median": 399.1735860327026, + "p95": 457.00975003128406, + "min": 343.10408696183003, + "max": 460.7523740414763, + "iqr": 52.53784391970839, + "mean": 406.91414221946616, + "cv": 0.1137328560829146 + }, + "tokens_per_second": { + "count": 5, + "median": 1282.649999687239, + "p95": 1456.7070265326242, + "min": 1111.2259618089574, + "max": 1492.2585286982007, + "iqr": 156.23284286569356, + "mean": 1271.7807366138682, + "cv": 0.11743038710546976 + }, + "median_step_ms_by_index": [ + 137.40829199377913, + 4.46979099069722, + 3.449666008236818, + 3.114209001068957, + 3.09016699611675, + 2.9687079950235784, + 3.14083400007803, + 2.916083001764491, + 3.0052500078454614, + 3.1109580013435334, + 3.1692909979028627, + 3.0691249994561076, + 3.3989999938057736, + 3.22154100285843, + 3.226083004847169, + 3.2685000041965395, + 3.2488330034539104, + 3.2186250027734786, + 3.473125005257316, + 3.4025839995592833, + 3.7983750080456957, + 3.5932920000050217, + 3.648582991445437, + 3.73316700279247, + 3.6486669996520504, + 3.6631669936468825, + 3.970250007114373, + 3.6653750139521435, + 4.326832990045659, + 4.309500000090338, + 4.538666995358653, + 4.401124999276362, + 4.544708004686981, + 4.666875000111759, + 4.463499994017184, + 4.644291999284178, + 4.665875007049181, + 4.476582995266654, + 4.908999995677732, + 4.655292010284029, + 4.434417001903057, + 4.78629200370051, + 4.5295419986359775, + 4.791917002876289, + 5.0033750012516975, + 4.527708995738067, + 5.061667005065829, + 4.644667002139613, + 4.588583004078828, + 4.650875009247102, + 4.733583002234809, + 4.648707996238954, + 4.569459008052945, + 4.543375005596317, + 4.857209001784213, + 4.655500000808388, + 4.639250008040108, + 4.724124999484047, + 4.874749996815808, + 4.978750002919696, + 4.854917002376169, + 4.626584006473422, + 4.617957994923927, + 5.188625000300817 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 142.84587500151247, + "p95": 164.76435840304475, + "min": 122.78070799948182, + "max": 166.9183750054799, + "iqr": 30.06741698482074, + "mean": 142.95482500165235, + "cv": 0.13272750486416685 + }, + "decode_ms_per_step": { + "count": 315, + "median": 142.77887500065845, + "p95": 158.11977479606867, + "min": 126.55545800225809, + "max": 170.14737499994226, + "iqr": 5.422145994089078, + "mean": 143.41527725752843, + "cv": 0.04562809658709105 + }, + "total_ms": { + "count": 5, + "median": 9256.204168035765, + "p95": 9322.710033180192, + "min": 8966.412669062265, + "max": 9334.282499970868, + "iqr": 219.15320797415916, + "mean": 9178.117292225943, + "cv": 0.01719038230006837 + }, + "tokens_per_second": { + "count": 5, + "median": 55.31425092891509, + "p95": 56.9874301926256, + "min": 54.8515646490877, + "max": 57.10198926786029, + "iqr": 1.3354886867823978, + "mean": 55.798140788490876, + "cv": 0.017306220258138872 + }, + "median_step_ms_by_index": [ + 142.84587500151247, + 130.13733299158048, + 130.46516699250787, + 129.21379100589547, + 134.6111250022659, + 139.32991700130515, + 141.05508300417569, + 140.3424579912098, + 138.5367920011049, + 140.54870899417438, + 138.95795799908228, + 141.33058300649282, + 140.76491599553265, + 140.2480840042699, + 139.40549999824725, + 140.97912500437815, + 141.95650001056492, + 141.73841699084733, + 139.09566699294373, + 141.731499999878, + 140.4237499955343, + 140.04558400483802, + 139.71091600251384, + 142.9732919932576, + 142.17079200898297, + 140.63850000093225, + 143.12074999907054, + 144.78795800823718, + 143.09483299439307, + 142.37624999077525, + 141.6845409985399, + 141.79858400893863, + 143.91849999083206, + 143.58083299885038, + 143.6133749957662, + 142.423166995286, + 143.6559169960674, + 142.58858301036526, + 144.5156669942662, + 143.34824999968987, + 162.95216701109894, + 144.33866599574685, + 142.6437499903841, + 143.0977910058573, + 142.91575000970624, + 142.96495899907313, + 143.9914160000626, + 143.25875000213273, + 143.80612500826828, + 143.70291600062046, + 143.55049999721814, + 143.51120899664238, + 144.94075000402518, + 144.70112501294352, + 144.0750420006225, + 144.88716699997894, + 145.4048339946894, + 145.69562500400934, + 146.74454099440482, + 146.1261249933159, + 145.30550000199582, + 145.9112919983454, + 145.98008300527, + 146.62929199403152 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_default": { + "first_token_ms": { + "count": 5, + "median": 401.41741700062994, + "p95": 406.3203420024365, + "min": 376.8332920008106, + "max": 406.37541700561997, + "iqr": 20.2182509819977, + "mean": 395.32159180089366, + "cv": 0.033617052188523434 + }, + "decode_ms_per_step": { + "count": 315, + "median": 391.65316699654795, + "p95": 438.6158374021761, + "min": 339.3349170073634, + "max": 554.5839999977034, + "iqr": 33.81510399776744, + "mean": 392.25644666381874, + "cv": 0.06910006375093881 + }, + "total_ms": { + "count": 5, + "median": 24852.538954044576, + "p95": 26208.363458429812, + "min": 24066.607376080356, + "max": 26402.85100103938, + "iqr": 645.4352490400197, + "mean": 25107.477731621475, + "cv": 0.034691581603600585 + }, + "tokens_per_second": { + "count": 5, + "median": 20.601516848912357, + "p95": 21.15096733959492, + "min": 19.3918452208, + "max": 21.274290638441773, + "iqr": 0.5243009975834809, + "mean": 20.41173999979714, + "cv": 0.03427794053833768 + }, + "median_step_ms_by_index": [ + 401.41741700062994, + 383.40979200438596, + 366.71941600798164, + 355.17924999294337, + 350.13341599551495, + 363.39733299973886, + 355.0732500007143, + 359.67929199978244, + 357.33741699368693, + 362.915666992194, + 369.0999999962514, + 366.7204590019537, + 365.2984579966869, + 378.0704160017194, + 379.40662499750033, + 373.3627500041621, + 382.0695419999538, + 385.1247919956222, + 392.7531670051394, + 390.43066700105555, + 385.9469160088338, + 386.0355420038104, + 386.91512499644887, + 416.01758399337996, + 395.7933750061784, + 397.85049999773037, + 389.60333399882074, + 387.9010410018964, + 386.89566700486466, + 403.6758330039447, + 384.3671250069747, + 389.2955410119612, + 377.8068750107195, + 385.4280829982599, + 387.48566698632203, + 392.5101250060834, + 385.74370900460053, + 387.7962499973364, + 402.93408300203737, + 399.1650420066435, + 405.38662500330247, + 393.81774999492336, + 402.2490409988677, + 402.6504590001423, + 405.34729200589936, + 401.5271670068614, + 398.6293749912875, + 400.03916699788533, + 379.2206670041196, + 390.6008330086479, + 398.792041989509, + 397.69766699464526, + 406.120292012929, + 407.6630000054138, + 404.0420419914881, + 408.902458002558, + 420.48087499279063, + 420.1353330136044, + 412.9839169909246, + 415.63204099657014, + 422.9162499977974, + 415.0547080062097, + 420.15641600301024, + 427.45554199791513 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "speedup_cached_vs_uncached_sdpa": 23.188418502414244, + "speedup_cached_vs_uncached_default": 62.259978675063216 + } + ], + "window": { + "batch": 1, + "prompt_len": 32, + "context_length": 128, + "new_tokens": 224, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_default": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 0.790375008364208, + "p95": 0.834108202252537, + "min": 0.7602919940836728, + "max": 0.8445830026175827, + "iqr": 0.0034170079743489623, + "mean": 0.7952501997351646, + "cv": 0.03840177302769063 + }, + "decode_ms_per_step": { + "count": 1115, + "median": 1.5052499948069453, + "p95": 1.65196260495577, + "min": 0.35099999513477087, + "max": 2.1829589968547225, + "iqr": 1.1916665025637485, + "mean": 1.0613093131499558, + "cv": 0.5595022068913268 + }, + "total_ms": { + "count": 5, + "median": 237.90190191357397, + "p95": 240.25711464928463, + "min": 232.29739505040925, + "max": 240.31765601830557, + "iqr": 3.2107161678140983, + "mean": 237.4672270321753, + "cv": 0.013640917001253213 + }, + "tokens_per_second": { + "count": 5, + "median": 941.5645616880175, + "p95": 960.6107299419516, + "min": 932.0996372523598, + "max": 964.2811532664467, + "iqr": 12.65383536325453, + "mean": 943.4299180263024, + "cv": 0.013779433426095188 + }, + "median_step_ms_by_index": [ + 0.790375008364208, + 0.464917000499554, + 0.4061660001752898, + 0.39216699951794, + 0.3829160123132169, + 0.3785829903790727, + 0.375540999812074, + 0.37316700036171824, + 0.3770420007640496, + 0.37599999632220715, + 0.3735000063898042, + 0.3761669941013679, + 0.370459005353041, + 0.3741659893421456, + 0.3717920044437051, + 0.3697909996844828, + 0.368291002814658, + 0.3709169977810234, + 0.3692089958349243, + 0.3716670034918934, + 0.367915999959223, + 0.3668749995995313, + 0.36945799365639687, + 0.370084002497606, + 0.3740839892998338, + 0.37020900344941765, + 0.3688329888973385, + 0.3750830073840916, + 0.3697500069392845, + 0.37645800330210477, + 0.37408299976959825, + 0.3724160051206127, + 0.37962499482091516, + 0.3800830018008128, + 0.3764589928323403, + 0.37258399243000895, + 0.37141700158827007, + 0.37370801146607846, + 0.37970799894537777, + 0.3715000057127327, + 0.37829100620001554, + 0.37595900357700884, + 0.3778329992201179, + 0.3735409991350025, + 0.3732920013135299, + 0.3757499944185838, + 0.3717090003192425, + 0.3723330009961501, + 0.37812499795109034, + 0.3814170049736276, + 0.3807079920079559, + 0.3828330081887543, + 0.3863340098178014, + 0.3812909999396652, + 0.38279099680949, + 0.3856250114040449, + 0.3884170000674203, + 0.38141600089147687, + 0.3853329981211573, + 0.38329199014697224, + 0.3807500033872202, + 0.3827920008916408, + 0.3771250048885122, + 0.37820800207555294, + 0.3897920105373487, + 0.3863750025629997, + 0.38516600034199655, + 0.3823340084636584, + 0.38587499875575304, + 0.38620799023192376, + 0.38841599598526955, + 0.379708013497293, + 0.36516600812319666, + 0.3819580015260726, + 0.36958299460820854, + 0.3771659976337105, + 0.3757090016733855, + 0.3904170007444918, + 0.37133300793357193, + 0.37645900738425553, + 0.37295800575520843, + 0.3760419931495562, + 0.3744160057976842, + 0.3766669979086146, + 0.37316700036171824, + 0.38300000596791506, + 0.3840829886030406, + 0.3749589959625155, + 0.3770000039367005, + 0.3750830073840916, + 0.3751250042114407, + 0.3759579994948581, + 0.38279099680949, + 0.37954199069645256, + 0.39237499004229903, + 0.37295899528544396, + 0.3832500078715384, + 1.5641670033801347, + 1.5254590107360855, + 1.5098750009201467, + 1.5699579962529242, + 1.5318750083679333, + 1.5892500086920336, + 1.5607079985784367, + 1.553917012643069, + 1.582124998094514, + 1.5290000010281801, + 1.5096669958438724, + 1.5676660113967955, + 1.5374170034192502, + 1.5215419989544898, + 1.5230830031214282, + 1.5135419962462038, + 1.4959580003051087, + 1.5040830039652064, + 1.5612500137649477, + 1.5552080003544688, + 1.5185829979600385, + 1.4985420129960403, + 1.509708003140986, + 1.4976250095060095, + 1.516792006441392, + 1.547291991300881, + 1.5787500014994293, + 1.5576250007143244, + 1.5627910033799708, + 1.5630830021109432, + 1.5755000058561563, + 1.553209003759548, + 1.557374998810701, + 1.5486660122405738, + 1.555291994009167, + 1.5608749963575974, + 1.5577079902868718, + 1.5596659941365942, + 1.5702079981565475, + 1.5515840059379116, + 1.5653750015189871, + 1.568415988003835, + 1.5663749945815653, + 1.5586670051561669, + 1.569167012348771, + 1.555625000037253, + 1.556792005430907, + 1.570916996570304, + 1.562334000482224, + 1.576166003360413, + 1.5730829909443855, + 1.5674169990234077, + 1.5755839995108545, + 1.5614580042893067, + 1.5690829895902425, + 1.5567079972242936, + 1.5894579992163926, + 1.6656250081723556, + 1.6032920102588832, + 1.5723329997854307, + 1.5832909994060174, + 1.6095409955596551, + 1.5885830071056262, + 1.6659160028211772, + 1.6254580114036798, + 1.5720419905846938, + 1.5822080022189766, + 1.5835829981369898, + 1.6242919955402613, + 1.6073750011855736, + 1.5926670021144673, + 1.5928330103633925, + 1.603249998879619, + 1.5847080067032948, + 1.6004579956643283, + 1.5888750058365986, + 1.5774579951539636, + 1.5942080062814057, + 1.5888750058365986, + 1.564792008139193, + 1.5859580016694963, + 1.563499987241812, + 1.5804580034455284, + 1.5705420082667843, + 1.5695829933974892, + 1.570290987729095, + 1.5944170008879155, + 1.577167000505142, + 1.5840420091990381, + 1.5693339955760166, + 1.5690829895902425, + 1.5724999975645915, + 1.5713340108050033, + 1.5643329970771447, + 1.575457994476892, + 1.5711250016465783, + 1.5691660082666203, + 1.5572499978588894, + 1.5663329977542162, + 1.5614169969921932, + 1.602665986865759, + 1.5690409927628934, + 1.567459010402672, + 1.5690000000176951, + 1.5836670063436031, + 1.5649589913664386, + 1.569917003507726, + 1.5687499981140718, + 1.5771250036777928, + 1.5655410097679123, + 1.5652919973945245, + 1.5660409990232438, + 1.5896249969955534, + 1.573208995978348, + 1.5726249985164031, + 1.5616669988958165, + 1.5799999964656308, + 1.5603749925503507, + 1.5845830057514831, + 1.564417005283758, + 1.5727080026408657, + 1.5677920018788427, + 1.5720840019639581, + 1.565291007864289, + 1.5792079939274117, + 1.5694169997004792, + 1.5690419968450442 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 0.7703750015934929, + "p95": 0.7923749944893643, + "min": 0.7630000036442652, + "max": 0.7923749944893643, + "iqr": 0.028999987989664078, + "mean": 0.7763000001432374, + "cv": 0.019277954117947006 + }, + "decode_ms_per_step": { + "count": 1115, + "median": 1.5558339946437627, + "p95": 1.6877747955732048, + "min": 0.8194160036509857, + "max": 15.06070900359191, + "iqr": 0.2494584914529696, + "mean": 1.4896706983759256, + "cv": 0.39912936957693956 + }, + "total_ms": { + "count": 5, + "median": 328.4660790086491, + "p95": 348.4693632548442, + "min": 318.0780780967325, + "max": 350.1180308230687, + "iqr": 15.54724520246964, + "mean": 332.97286573797464, + "cv": 0.03856357828254156 + }, + "tokens_per_second": { + "count": 5, + "median": 681.9577859487332, + "p95": 700.6691035430079, + "min": 639.7842449685143, + "max": 704.2296072094541, + "iqr": 31.216262810523972, + "mean": 673.5219106141245, + "cv": 0.038245891022533965 + }, + "median_step_ms_by_index": [ + 0.7703750015934929, + 0.8346669928869233, + 0.8264169882750139, + 0.8335419988725334, + 0.8333330042660236, + 0.8379579958273098, + 0.851875010994263, + 0.870000003487803, + 0.8756659954087809, + 0.8809169958112761, + 0.8920839900383726, + 0.9987079974962398, + 0.99608299206011, + 1.0064999951282516, + 1.0093750024680048, + 1.0142920073121786, + 0.9925420017680153, + 1.0048749973066151, + 1.001583004835993, + 1.0130000009667128, + 1.011875006952323, + 1.0205830039922148, + 1.0259159898851067, + 1.0240830015391111, + 1.0407079971628264, + 1.0827920050360262, + 1.0849999962374568, + 1.0912079887930304, + 1.0919590131379664, + 1.1005829874193296, + 1.1043750128010288, + 1.1087500024586916, + 1.110375000280328, + 1.2538750015664846, + 1.2307500001043081, + 1.2395829980960116, + 1.2380409898469225, + 1.2497080024331808, + 1.2658329942496493, + 1.2867500045103952, + 1.266374994884245, + 1.2721669918391854, + 1.2936249986523762, + 1.2680410000029951, + 1.262749996385537, + 1.2812089989893138, + 1.2899169960292056, + 1.2884589959867299, + 1.2981249892618507, + 1.3002079940633848, + 1.3144169934093952, + 1.320457988185808, + 1.328583006397821, + 1.3265409943414852, + 1.3113749882904813, + 1.2977920123375952, + 1.3028749963268638, + 1.3165829877834767, + 1.3182499969843775, + 1.3221660046838224, + 1.3186659925850108, + 1.3368749932851642, + 1.3336659903870896, + 1.3433330022962764, + 1.3318340061232448, + 1.4514590002363548, + 1.4494579954771325, + 1.4542499993694946, + 1.4560830022674054, + 1.4687910006614402, + 1.4742499915882945, + 1.4898749941494316, + 1.4686670037917793, + 1.4776669995626435, + 1.4826249971520156, + 1.4775419986108318, + 1.48695899406448, + 1.486875000409782, + 1.4864160038996488, + 1.4958749961806461, + 1.5147079975577071, + 1.5125840000109747, + 1.5072910027811304, + 1.5162080089794472, + 1.5043750026961789, + 1.5167090023169294, + 1.5316249919123948, + 1.5292080061044544, + 1.52058299863711, + 1.532583002699539, + 1.5480830043088645, + 1.555666996864602, + 1.6099579952424392, + 1.5874170057941228, + 1.5685420075897127, + 1.5842499997233972, + 1.5826250019017607, + 1.563249999890104, + 1.569708008901216, + 1.572207998833619, + 1.5736250061308965, + 1.5671250002924353, + 1.5623329964000732, + 1.572541004861705, + 1.5630830021109432, + 1.6278330003842711, + 1.5960840100888163, + 1.5675829927204177, + 1.5575419965898618, + 1.6902079951250926, + 1.582707991474308, + 1.569959000335075, + 1.5949160006130114, + 1.5841249987715855, + 1.5747500001452863, + 1.5704590041423216, + 1.5631250134902075, + 1.5657500043744221, + 1.5623750077793375, + 1.5658749907743186, + 1.5836660022614524, + 1.5655000024707988, + 1.5698750066803768, + 1.5713750035502017, + 1.5684579993830994, + 1.5727910067653283, + 1.5738750080345199, + 1.5666249964851886, + 1.5756249922560528, + 1.5691250009695068, + 1.570083011756651, + 1.5635000017937273, + 1.560000004246831, + 1.563625002745539, + 1.5625410014763474, + 1.5721249947091565, + 1.6229589964495972, + 1.5627079992555082, + 1.5620000049239025, + 1.5922919992590323, + 1.6034589934861287, + 1.642916991841048, + 1.6198749945033342, + 1.5682090015616268, + 1.5730830054963008, + 1.5642090002074838, + 1.5847499889787287, + 1.5818329993635416, + 1.5654169983463362, + 1.5598750032950193, + 1.5637920005246997, + 1.5560829924652353, + 1.56079200678505, + 1.5535830025328323, + 1.5642920043319464, + 1.5714170003775507, + 1.5590420080116019, + 1.5607499954057857, + 1.5691669977968559, + 1.5638750046491623, + 1.5584170032525435, + 1.5702499949838966, + 1.5694579924456775, + 1.5717079950263724, + 1.5623749932274222, + 1.5622919891029596, + 1.5675829927204177, + 1.5650840068701655, + 1.6019579925341532, + 1.5511669917032123, + 1.6179999947780743, + 1.5848749899305403, + 1.5767080039950088, + 1.566083010402508, + 1.5694589965278283, + 1.6273329965770245, + 1.5793339989613742, + 1.606958991033025, + 1.5649590059183538, + 1.570874999742955, + 1.5734159969724715, + 1.5979160089045763, + 1.562290999572724, + 1.56879099085927, + 1.567250001244247, + 1.572958004544489, + 1.6070419951574877, + 1.5666669933125377, + 1.5870000061113387, + 1.5844160079723224, + 1.6646250005578622, + 1.6471660055685788, + 1.6612079925835133, + 1.592957996763289, + 1.567415994941257, + 1.5585830115014687, + 1.5537090075667948, + 1.5601250051986426, + 1.5713330067228526, + 1.5739579976070672, + 1.560209013405256, + 1.5675409958930686, + 1.6458339960081503, + 1.6088749980553985, + 1.5646660031052306, + 1.5599170001223683, + 1.6003339987946674, + 1.5791249898029491, + 1.5736250061308965, + 1.5804580034455284, + 1.56020799477119, + 1.5755000058561563, + 1.600042000063695, + 1.6856669972185045, + 1.6393749974668026, + 1.5949580119922757, + 1.5809159958735108, + 1.5775830106576905, + 1.5525420021731406, + 1.5671669971197844, + 1.5545829955954105, + 1.6547500126762316, + 1.6510830027982593, + 1.6075839957920834 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_default": { + "first_token_ms": { + "count": 5, + "median": 0.9177079919027165, + "p95": 0.9772668010555208, + "min": 0.8824590040603653, + "max": 0.9865419997368008, + "iqr": 0.03349900362081826, + "mean": 0.9267084009479731, + "cv": 0.04249004702956992 + }, + "decode_ms_per_step": { + "count": 1115, + "median": 1.9434160058153793, + "p95": 2.149341892800294, + "min": 0.9013329981826246, + "max": 5.413957987911999, + "iqr": 0.38072950701462105, + "mean": 1.7989233865599972, + "cv": 0.19963419584126224 + }, + "total_ms": { + "count": 5, + "median": 395.49699604685884, + "p95": 427.65984660654794, + "min": 388.434691936709, + "max": 435.60962300398387, + "iqr": 0.8296750020235777, + "mean": 402.0866236038273, + "cv": 0.04722293215958467 + }, + "tokens_per_second": { + "count": 5, + "median": 566.3759832286065, + "p95": 574.7476158844108, + "min": 514.2218816363279, + "max": 576.6735171957768, + "iqr": 1.188453897868044, + "mean": 558.0341898881476, + "cv": 0.04461047668468971 + }, + "median_step_ms_by_index": [ + 0.9177079919027165, + 0.938750003115274, + 0.924082996789366, + 0.9257909987354651, + 0.927542001591064, + 0.9630830027163029, + 0.9494580008322373, + 0.9603330108802766, + 0.9530000097583979, + 0.9620000055292621, + 0.9629999985918403, + 1.0817919974215329, + 1.083709008526057, + 1.0893340077018365, + 1.1036670039175078, + 1.1116669920738786, + 1.119874999858439, + 1.1334999871905893, + 1.1321250058244914, + 1.141750006354414, + 1.1789580021286383, + 1.1551659990800545, + 1.155874997493811, + 1.1592910013860092, + 1.1772500001825392, + 1.2189999979455024, + 1.219708996359259, + 1.2316249922150746, + 1.2401249987306073, + 1.2418329861247912, + 1.2422090076142922, + 1.24879099894315, + 1.292834000196308, + 1.4338339969981462, + 1.3836659927619621, + 1.3930840068496764, + 1.3892499991925433, + 1.4032499893801287, + 1.4112089993432164, + 1.4222500030882657, + 1.4387080009328201, + 1.5034159878268838, + 1.4932079939171672, + 1.4959590043872595, + 1.4907919976394624, + 1.4982499997131526, + 1.507082997704856, + 1.5120000025490299, + 1.5076660056365654, + 1.5400830016005784, + 1.5332500042859465, + 1.5353749913629144, + 1.5530420059803873, + 1.5660420031053945, + 1.5580000035697594, + 1.538875003461726, + 1.5381250123027712, + 1.5646669926354662, + 1.5570830000797287, + 1.5678750060033053, + 1.566541992360726, + 1.584792000357993, + 1.5888749912846833, + 1.5935410046949983, + 1.5865830064285547, + 1.7263749905396253, + 1.7258750012842938, + 1.7324159998679534, + 1.7303750064456835, + 1.743250002618879, + 1.777541998308152, + 1.770832997863181, + 1.7633329989621416, + 1.7759999900590628, + 1.789166999515146, + 1.8071249942295253, + 1.813541995943524, + 1.8109579978045076, + 1.8216250027762726, + 1.8306660058442503, + 1.8223750084871426, + 1.8653330043889582, + 1.8612499989103526, + 1.8564159981906414, + 1.8580830073915422, + 1.9070419948548079, + 1.9046660017920658, + 1.912542007630691, + 1.8863330042222515, + 1.9101250072708353, + 1.9133750029141083, + 1.9494160078465939, + 1.9285830057924613, + 1.9484160002321005, + 1.9657910015666857, + 1.9731249922188, + 1.9640420068753883, + 1.963792004971765, + 1.968374999705702, + 1.953125, + 1.9496250024531037, + 1.946499993209727, + 1.9480829942040145, + 1.9503750081639737, + 1.975916005903855, + 1.9615409983089194, + 1.9537500047590584, + 1.9636250071926042, + 1.956374995643273, + 1.9753340020542964, + 1.95479100511875, + 1.9462499913061038, + 1.9560829969123006, + 1.949332989170216, + 1.9477909954730421, + 1.9453749991953373, + 1.9989589927718043, + 1.9590410083765164, + 1.9485829980112612, + 1.9686250016093254, + 1.9834999984595925, + 2.009374991757795, + 1.978708998649381, + 1.9716660026460886, + 1.9635000062407926, + 1.9594159966800362, + 1.9598329963628203, + 1.9764589960686862, + 1.9832089892588556, + 1.9714579975698143, + 1.9788750068983063, + 1.9507919932948425, + 1.9549160060705617, + 1.9616669887909666, + 1.9568330026231706, + 1.9482909992802888, + 1.9632080075098202, + 1.9650839967653155, + 2.0187500049360096, + 1.9539579952834174, + 1.9557500054361299, + 1.9497909961501136, + 1.948791992617771, + 1.9434160058153793, + 1.946250005858019, + 1.9583329994929954, + 1.9560410000849515, + 1.959750006790273, + 1.9540410139597952, + 1.946708987816237, + 1.9476669986033812, + 1.9477080059004948, + 2.024708010139875, + 1.956167005118914, + 1.9516659958753735, + 1.9630409951787442, + 1.9487909885356203, + 1.9446250080363825, + 1.9606249989010394, + 1.961125002708286, + 1.9989169959444553, + 1.972375001059845, + 1.948749995790422, + 1.9460830080788583, + 1.9482500065350905, + 1.954457999090664, + 1.9677910022437572, + 1.949458004673943, + 1.9735409878194332, + 1.980165994609706, + 1.948749995790422, + 1.949833007529378, + 1.980374989216216, + 1.94266700418666, + 1.9494579901220277, + 1.9450830004643649, + 1.9588750001275912, + 1.9937920005759224, + 1.950833000591956, + 1.9559580105124041, + 1.9558329950086772, + 1.9682499987538904, + 1.9470830011414364, + 1.9848329975502566, + 1.961958987521939, + 1.9692080095410347, + 1.9431249966146424, + 1.9696249946719036, + 1.9579579966375604, + 1.9521250069374219, + 1.9548329873941839, + 1.9434580026427284, + 1.9521669892128557, + 1.9454159919405356, + 1.9502499926602468, + 1.952584003447555, + 1.9557089981390163, + 1.9435409922152758, + 1.9947919936385006, + 1.953125, + 1.981833003810607, + 1.9612920004874468, + 1.9486250093905255, + 1.9524160015862435, + 1.9503330113366246, + 1.9450409890851006, + 1.9429170060902834, + 1.9446660007815808, + 1.9357919954927638, + 1.9659999961731955, + 1.9519580091582611, + 1.9642499973997474, + 1.9597079954110086, + 1.9699580006999895, + 1.9953750015702099, + 2.0369580015540123, + 1.9631659961305559, + 1.963750008144416, + 1.9486670062178746, + 1.9652920018415898, + 1.9542499940143898, + 2.0040829986101016, + 1.9614169868873432, + 1.9568749994505197 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "speedup_cached_vs_uncached_sdpa": 1.3806786594248233, + "speedup_cached_vs_uncached_default": 1.6624373023740546 + }, + "memory": [ + { + "batch": 1, + "prompt_len": 128, + "steps": [ + { + "step": 0, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 1, + "cached_positions": 129, + "cache_bytes": 792576 + }, + { + "step": 2, + "cached_positions": 130, + "cache_bytes": 798720 + }, + { + "step": 3, + "cached_positions": 131, + "cache_bytes": 804864 + }, + { + "step": 4, + "cached_positions": 132, + "cache_bytes": 811008 + }, + { + "step": 5, + "cached_positions": 133, + "cache_bytes": 817152 + }, + { + "step": 6, + "cached_positions": 134, + "cache_bytes": 823296 + }, + { + "step": 7, + "cached_positions": 135, + "cache_bytes": 829440 + }, + { + "step": 8, + "cached_positions": 136, + "cache_bytes": 835584 + }, + { + "step": 9, + "cached_positions": 137, + "cache_bytes": 841728 + }, + { + "step": 10, + "cached_positions": 138, + "cache_bytes": 847872 + }, + { + "step": 11, + "cached_positions": 139, + "cache_bytes": 854016 + }, + { + "step": 12, + "cached_positions": 140, + "cache_bytes": 860160 + }, + { + "step": 13, + "cached_positions": 141, + "cache_bytes": 866304 + }, + { + "step": 14, + "cached_positions": 142, + "cache_bytes": 872448 + }, + { + "step": 15, + "cached_positions": 143, + "cache_bytes": 878592 + }, + { + "step": 16, + "cached_positions": 144, + "cache_bytes": 884736 + }, + { + "step": 17, + "cached_positions": 145, + "cache_bytes": 890880 + }, + { + "step": 18, + "cached_positions": 146, + "cache_bytes": 897024 + }, + { + "step": 19, + "cached_positions": 147, + "cache_bytes": 903168 + }, + { + "step": 20, + "cached_positions": 148, + "cache_bytes": 909312 + }, + { + "step": 21, + "cached_positions": 149, + "cache_bytes": 915456 + }, + { + "step": 22, + "cached_positions": 150, + "cache_bytes": 921600 + }, + { + "step": 23, + "cached_positions": 151, + "cache_bytes": 927744 + }, + { + "step": 24, + "cached_positions": 152, + "cache_bytes": 933888 + }, + { + "step": 25, + "cached_positions": 153, + "cache_bytes": 940032 + }, + { + "step": 26, + "cached_positions": 154, + "cache_bytes": 946176 + }, + { + "step": 27, + "cached_positions": 155, + "cache_bytes": 952320 + }, + { + "step": 28, + "cached_positions": 156, + "cache_bytes": 958464 + }, + { + "step": 29, + "cached_positions": 157, + "cache_bytes": 964608 + }, + { + "step": 30, + "cached_positions": 158, + "cache_bytes": 970752 + }, + { + "step": 31, + "cached_positions": 159, + "cache_bytes": 976896 + }, + { + "step": 32, + "cached_positions": 160, + "cache_bytes": 983040 + }, + { + "step": 33, + "cached_positions": 161, + "cache_bytes": 989184 + }, + { + "step": 34, + "cached_positions": 162, + "cache_bytes": 995328 + }, + { + "step": 35, + "cached_positions": 163, + "cache_bytes": 1001472 + }, + { + "step": 36, + "cached_positions": 164, + "cache_bytes": 1007616 + }, + { + "step": 37, + "cached_positions": 165, + "cache_bytes": 1013760 + }, + { + "step": 38, + "cached_positions": 166, + "cache_bytes": 1019904 + }, + { + "step": 39, + "cached_positions": 167, + "cache_bytes": 1026048 + }, + { + "step": 40, + "cached_positions": 168, + "cache_bytes": 1032192 + }, + { + "step": 41, + "cached_positions": 169, + "cache_bytes": 1038336 + }, + { + "step": 42, + "cached_positions": 170, + "cache_bytes": 1044480 + }, + { + "step": 43, + "cached_positions": 171, + "cache_bytes": 1050624 + }, + { + "step": 44, + "cached_positions": 172, + "cache_bytes": 1056768 + }, + { + "step": 45, + "cached_positions": 173, + "cache_bytes": 1062912 + }, + { + "step": 46, + "cached_positions": 174, + "cache_bytes": 1069056 + }, + { + "step": 47, + "cached_positions": 175, + "cache_bytes": 1075200 + }, + { + "step": 48, + "cached_positions": 176, + "cache_bytes": 1081344 + }, + { + "step": 49, + "cached_positions": 177, + "cache_bytes": 1087488 + }, + { + "step": 50, + "cached_positions": 178, + "cache_bytes": 1093632 + }, + { + "step": 51, + "cached_positions": 179, + "cache_bytes": 1099776 + }, + { + "step": 52, + "cached_positions": 180, + "cache_bytes": 1105920 + }, + { + "step": 53, + "cached_positions": 181, + "cache_bytes": 1112064 + }, + { + "step": 54, + "cached_positions": 182, + "cache_bytes": 1118208 + }, + { + "step": 55, + "cached_positions": 183, + "cache_bytes": 1124352 + }, + { + "step": 56, + "cached_positions": 184, + "cache_bytes": 1130496 + }, + { + "step": 57, + "cached_positions": 185, + "cache_bytes": 1136640 + }, + { + "step": 58, + "cached_positions": 186, + "cache_bytes": 1142784 + }, + { + "step": 59, + "cached_positions": 187, + "cache_bytes": 1148928 + }, + { + "step": 60, + "cached_positions": 188, + "cache_bytes": 1155072 + }, + { + "step": 61, + "cached_positions": 189, + "cache_bytes": 1161216 + }, + { + "step": 62, + "cached_positions": 190, + "cache_bytes": 1167360 + }, + { + "step": 63, + "cached_positions": 191, + "cache_bytes": 1173504 + } + ] + }, + { + "batch": 8, + "prompt_len": 128, + "steps": [ + { + "step": 0, + "cached_positions": 128, + "cache_bytes": 6291456 + }, + { + "step": 1, + "cached_positions": 129, + "cache_bytes": 6340608 + }, + { + "step": 2, + "cached_positions": 130, + "cache_bytes": 6389760 + }, + { + "step": 3, + "cached_positions": 131, + "cache_bytes": 6438912 + }, + { + "step": 4, + "cached_positions": 132, + "cache_bytes": 6488064 + }, + { + "step": 5, + "cached_positions": 133, + "cache_bytes": 6537216 + }, + { + "step": 6, + "cached_positions": 134, + "cache_bytes": 6586368 + }, + { + "step": 7, + "cached_positions": 135, + "cache_bytes": 6635520 + }, + { + "step": 8, + "cached_positions": 136, + "cache_bytes": 6684672 + }, + { + "step": 9, + "cached_positions": 137, + "cache_bytes": 6733824 + }, + { + "step": 10, + "cached_positions": 138, + "cache_bytes": 6782976 + }, + { + "step": 11, + "cached_positions": 139, + "cache_bytes": 6832128 + }, + { + "step": 12, + "cached_positions": 140, + "cache_bytes": 6881280 + }, + { + "step": 13, + "cached_positions": 141, + "cache_bytes": 6930432 + }, + { + "step": 14, + "cached_positions": 142, + "cache_bytes": 6979584 + }, + { + "step": 15, + "cached_positions": 143, + "cache_bytes": 7028736 + }, + { + "step": 16, + "cached_positions": 144, + "cache_bytes": 7077888 + }, + { + "step": 17, + "cached_positions": 145, + "cache_bytes": 7127040 + }, + { + "step": 18, + "cached_positions": 146, + "cache_bytes": 7176192 + }, + { + "step": 19, + "cached_positions": 147, + "cache_bytes": 7225344 + }, + { + "step": 20, + "cached_positions": 148, + "cache_bytes": 7274496 + }, + { + "step": 21, + "cached_positions": 149, + "cache_bytes": 7323648 + }, + { + "step": 22, + "cached_positions": 150, + "cache_bytes": 7372800 + }, + { + "step": 23, + "cached_positions": 151, + "cache_bytes": 7421952 + }, + { + "step": 24, + "cached_positions": 152, + "cache_bytes": 7471104 + }, + { + "step": 25, + "cached_positions": 153, + "cache_bytes": 7520256 + }, + { + "step": 26, + "cached_positions": 154, + "cache_bytes": 7569408 + }, + { + "step": 27, + "cached_positions": 155, + "cache_bytes": 7618560 + }, + { + "step": 28, + "cached_positions": 156, + "cache_bytes": 7667712 + }, + { + "step": 29, + "cached_positions": 157, + "cache_bytes": 7716864 + }, + { + "step": 30, + "cached_positions": 158, + "cache_bytes": 7766016 + }, + { + "step": 31, + "cached_positions": 159, + "cache_bytes": 7815168 + }, + { + "step": 32, + "cached_positions": 160, + "cache_bytes": 7864320 + }, + { + "step": 33, + "cached_positions": 161, + "cache_bytes": 7913472 + }, + { + "step": 34, + "cached_positions": 162, + "cache_bytes": 7962624 + }, + { + "step": 35, + "cached_positions": 163, + "cache_bytes": 8011776 + }, + { + "step": 36, + "cached_positions": 164, + "cache_bytes": 8060928 + }, + { + "step": 37, + "cached_positions": 165, + "cache_bytes": 8110080 + }, + { + "step": 38, + "cached_positions": 166, + "cache_bytes": 8159232 + }, + { + "step": 39, + "cached_positions": 167, + "cache_bytes": 8208384 + }, + { + "step": 40, + "cached_positions": 168, + "cache_bytes": 8257536 + }, + { + "step": 41, + "cached_positions": 169, + "cache_bytes": 8306688 + }, + { + "step": 42, + "cached_positions": 170, + "cache_bytes": 8355840 + }, + { + "step": 43, + "cached_positions": 171, + "cache_bytes": 8404992 + }, + { + "step": 44, + "cached_positions": 172, + "cache_bytes": 8454144 + }, + { + "step": 45, + "cached_positions": 173, + "cache_bytes": 8503296 + }, + { + "step": 46, + "cached_positions": 174, + "cache_bytes": 8552448 + }, + { + "step": 47, + "cached_positions": 175, + "cache_bytes": 8601600 + }, + { + "step": 48, + "cached_positions": 176, + "cache_bytes": 8650752 + }, + { + "step": 49, + "cached_positions": 177, + "cache_bytes": 8699904 + }, + { + "step": 50, + "cached_positions": 178, + "cache_bytes": 8749056 + }, + { + "step": 51, + "cached_positions": 179, + "cache_bytes": 8798208 + }, + { + "step": 52, + "cached_positions": 180, + "cache_bytes": 8847360 + }, + { + "step": 53, + "cached_positions": 181, + "cache_bytes": 8896512 + }, + { + "step": 54, + "cached_positions": 182, + "cache_bytes": 8945664 + }, + { + "step": 55, + "cached_positions": 183, + "cache_bytes": 8994816 + }, + { + "step": 56, + "cached_positions": 184, + "cache_bytes": 9043968 + }, + { + "step": 57, + "cached_positions": 185, + "cache_bytes": 9093120 + }, + { + "step": 58, + "cached_positions": 186, + "cache_bytes": 9142272 + }, + { + "step": 59, + "cached_positions": 187, + "cache_bytes": 9191424 + }, + { + "step": 60, + "cached_positions": 188, + "cache_bytes": 9240576 + }, + { + "step": 61, + "cached_positions": 189, + "cache_bytes": 9289728 + }, + { + "step": 62, + "cached_positions": 190, + "cache_bytes": 9338880 + }, + { + "step": 63, + "cached_positions": 191, + "cache_bytes": 9388032 + } + ] + }, + { + "batch": 1, + "prompt_len": 32, + "steps": [ + { + "step": 0, + "cached_positions": 32, + "cache_bytes": 196608 + }, + { + "step": 1, + "cached_positions": 33, + "cache_bytes": 202752 + }, + { + "step": 2, + "cached_positions": 34, + "cache_bytes": 208896 + }, + { + "step": 3, + "cached_positions": 35, + "cache_bytes": 215040 + }, + { + "step": 4, + "cached_positions": 36, + "cache_bytes": 221184 + }, + { + "step": 5, + "cached_positions": 37, + "cache_bytes": 227328 + }, + { + "step": 6, + "cached_positions": 38, + "cache_bytes": 233472 + }, + { + "step": 7, + "cached_positions": 39, + "cache_bytes": 239616 + }, + { + "step": 8, + "cached_positions": 40, + "cache_bytes": 245760 + }, + { + "step": 9, + "cached_positions": 41, + "cache_bytes": 251904 + }, + { + "step": 10, + "cached_positions": 42, + "cache_bytes": 258048 + }, + { + "step": 11, + "cached_positions": 43, + "cache_bytes": 264192 + }, + { + "step": 12, + "cached_positions": 44, + "cache_bytes": 270336 + }, + { + "step": 13, + "cached_positions": 45, + "cache_bytes": 276480 + }, + { + "step": 14, + "cached_positions": 46, + "cache_bytes": 282624 + }, + { + "step": 15, + "cached_positions": 47, + "cache_bytes": 288768 + }, + { + "step": 16, + "cached_positions": 48, + "cache_bytes": 294912 + }, + { + "step": 17, + "cached_positions": 49, + "cache_bytes": 301056 + }, + { + "step": 18, + "cached_positions": 50, + "cache_bytes": 307200 + }, + { + "step": 19, + "cached_positions": 51, + "cache_bytes": 313344 + }, + { + "step": 20, + "cached_positions": 52, + "cache_bytes": 319488 + }, + { + "step": 21, + "cached_positions": 53, + "cache_bytes": 325632 + }, + { + "step": 22, + "cached_positions": 54, + "cache_bytes": 331776 + }, + { + "step": 23, + "cached_positions": 55, + "cache_bytes": 337920 + }, + { + "step": 24, + "cached_positions": 56, + "cache_bytes": 344064 + }, + { + "step": 25, + "cached_positions": 57, + "cache_bytes": 350208 + }, + { + "step": 26, + "cached_positions": 58, + "cache_bytes": 356352 + }, + { + "step": 27, + "cached_positions": 59, + "cache_bytes": 362496 + }, + { + "step": 28, + "cached_positions": 60, + "cache_bytes": 368640 + }, + { + "step": 29, + "cached_positions": 61, + "cache_bytes": 374784 + }, + { + "step": 30, + "cached_positions": 62, + "cache_bytes": 380928 + }, + { + "step": 31, + "cached_positions": 63, + "cache_bytes": 387072 + }, + { + "step": 32, + "cached_positions": 64, + "cache_bytes": 393216 + }, + { + "step": 33, + "cached_positions": 65, + "cache_bytes": 399360 + }, + { + "step": 34, + "cached_positions": 66, + "cache_bytes": 405504 + }, + { + "step": 35, + "cached_positions": 67, + "cache_bytes": 411648 + }, + { + "step": 36, + "cached_positions": 68, + "cache_bytes": 417792 + }, + { + "step": 37, + "cached_positions": 69, + "cache_bytes": 423936 + }, + { + "step": 38, + "cached_positions": 70, + "cache_bytes": 430080 + }, + { + "step": 39, + "cached_positions": 71, + "cache_bytes": 436224 + }, + { + "step": 40, + "cached_positions": 72, + "cache_bytes": 442368 + }, + { + "step": 41, + "cached_positions": 73, + "cache_bytes": 448512 + }, + { + "step": 42, + "cached_positions": 74, + "cache_bytes": 454656 + }, + { + "step": 43, + "cached_positions": 75, + "cache_bytes": 460800 + }, + { + "step": 44, + "cached_positions": 76, + "cache_bytes": 466944 + }, + { + "step": 45, + "cached_positions": 77, + "cache_bytes": 473088 + }, + { + "step": 46, + "cached_positions": 78, + "cache_bytes": 479232 + }, + { + "step": 47, + "cached_positions": 79, + "cache_bytes": 485376 + }, + { + "step": 48, + "cached_positions": 80, + "cache_bytes": 491520 + }, + { + "step": 49, + "cached_positions": 81, + "cache_bytes": 497664 + }, + { + "step": 50, + "cached_positions": 82, + "cache_bytes": 503808 + }, + { + "step": 51, + "cached_positions": 83, + "cache_bytes": 509952 + }, + { + "step": 52, + "cached_positions": 84, + "cache_bytes": 516096 + }, + { + "step": 53, + "cached_positions": 85, + "cache_bytes": 522240 + }, + { + "step": 54, + "cached_positions": 86, + "cache_bytes": 528384 + }, + { + "step": 55, + "cached_positions": 87, + "cache_bytes": 534528 + }, + { + "step": 56, + "cached_positions": 88, + "cache_bytes": 540672 + }, + { + "step": 57, + "cached_positions": 89, + "cache_bytes": 546816 + }, + { + "step": 58, + "cached_positions": 90, + "cache_bytes": 552960 + }, + { + "step": 59, + "cached_positions": 91, + "cache_bytes": 559104 + }, + { + "step": 60, + "cached_positions": 92, + "cache_bytes": 565248 + }, + { + "step": 61, + "cached_positions": 93, + "cache_bytes": 571392 + }, + { + "step": 62, + "cached_positions": 94, + "cache_bytes": 577536 + }, + { + "step": 63, + "cached_positions": 95, + "cache_bytes": 583680 + }, + { + "step": 64, + "cached_positions": 96, + "cache_bytes": 589824 + }, + { + "step": 65, + "cached_positions": 97, + "cache_bytes": 595968 + }, + { + "step": 66, + "cached_positions": 98, + "cache_bytes": 602112 + }, + { + "step": 67, + "cached_positions": 99, + "cache_bytes": 608256 + }, + { + "step": 68, + "cached_positions": 100, + "cache_bytes": 614400 + }, + { + "step": 69, + "cached_positions": 101, + "cache_bytes": 620544 + }, + { + "step": 70, + "cached_positions": 102, + "cache_bytes": 626688 + }, + { + "step": 71, + "cached_positions": 103, + "cache_bytes": 632832 + }, + { + "step": 72, + "cached_positions": 104, + "cache_bytes": 638976 + }, + { + "step": 73, + "cached_positions": 105, + "cache_bytes": 645120 + }, + { + "step": 74, + "cached_positions": 106, + "cache_bytes": 651264 + }, + { + "step": 75, + "cached_positions": 107, + "cache_bytes": 657408 + }, + { + "step": 76, + "cached_positions": 108, + "cache_bytes": 663552 + }, + { + "step": 77, + "cached_positions": 109, + "cache_bytes": 669696 + }, + { + "step": 78, + "cached_positions": 110, + "cache_bytes": 675840 + }, + { + "step": 79, + "cached_positions": 111, + "cache_bytes": 681984 + }, + { + "step": 80, + "cached_positions": 112, + "cache_bytes": 688128 + }, + { + "step": 81, + "cached_positions": 113, + "cache_bytes": 694272 + }, + { + "step": 82, + "cached_positions": 114, + "cache_bytes": 700416 + }, + { + "step": 83, + "cached_positions": 115, + "cache_bytes": 706560 + }, + { + "step": 84, + "cached_positions": 116, + "cache_bytes": 712704 + }, + { + "step": 85, + "cached_positions": 117, + "cache_bytes": 718848 + }, + { + "step": 86, + "cached_positions": 118, + "cache_bytes": 724992 + }, + { + "step": 87, + "cached_positions": 119, + "cache_bytes": 731136 + }, + { + "step": 88, + "cached_positions": 120, + "cache_bytes": 737280 + }, + { + "step": 89, + "cached_positions": 121, + "cache_bytes": 743424 + }, + { + "step": 90, + "cached_positions": 122, + "cache_bytes": 749568 + }, + { + "step": 91, + "cached_positions": 123, + "cache_bytes": 755712 + }, + { + "step": 92, + "cached_positions": 124, + "cache_bytes": 761856 + }, + { + "step": 93, + "cached_positions": 125, + "cache_bytes": 768000 + }, + { + "step": 94, + "cached_positions": 126, + "cache_bytes": 774144 + }, + { + "step": 95, + "cached_positions": 127, + "cache_bytes": 780288 + }, + { + "step": 96, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 97, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 98, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 99, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 100, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 101, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 102, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 103, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 104, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 105, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 106, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 107, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 108, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 109, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 110, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 111, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 112, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 113, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 114, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 115, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 116, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 117, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 118, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 119, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 120, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 121, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 122, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 123, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 124, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 125, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 126, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 127, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 128, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 129, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 130, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 131, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 132, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 133, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 134, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 135, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 136, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 137, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 138, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 139, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 140, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 141, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 142, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 143, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 144, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 145, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 146, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 147, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 148, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 149, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 150, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 151, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 152, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 153, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 154, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 155, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 156, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 157, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 158, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 159, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 160, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 161, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 162, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 163, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 164, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 165, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 166, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 167, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 168, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 169, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 170, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 171, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 172, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 173, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 174, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 175, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 176, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 177, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 178, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 179, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 180, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 181, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 182, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 183, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 184, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 185, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 186, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 187, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 188, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 189, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 190, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 191, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 192, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 193, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 194, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 195, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 196, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 197, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 198, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 199, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 200, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 201, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 202, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 203, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 204, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 205, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 206, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 207, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 208, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 209, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 210, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 211, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 212, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 213, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 214, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 215, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 216, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 217, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 218, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 219, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 220, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 221, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 222, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 223, + "cached_positions": 128, + "cache_bytes": 786432 + } + ] + } + ] + } +} diff --git a/experiments/records/kv_cache/20261022T035808Z_kv_mps_float32_93507c4.json b/experiments/records/kv_cache/20261022T035808Z_kv_mps_float32_93507c4.json new file mode 100644 index 0000000..42faba8 --- /dev/null +++ b/experiments/records/kv_cache/20261022T035808Z_kv_mps_float32_93507c4.json @@ -0,0 +1,9045 @@ +{ + "record_version": 1, + "created_utc": "20261022T035808Z", + "experiment": "kv_cache_scaling", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "mps", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "93507c45fb853c19bfc7bbafba89d31852661b53", + "branch": "experiments/attention-kv-audit", + "dirty": false + }, + "command": "python scripts/benchmark_kv_scaling.py --device mps" + }, + "settings": { + "dtype": "float32", + "position": "learned", + "prompt_lens": [ + 1, + 16, + 128, + 512, + 1024 + ], + "batch_sizes": [ + 1, + 8 + ], + "new_tokens": 64, + "warmup": 1, + "repeats": 5, + "seed": 0, + "modes": { + "cached": { + "attention_backend": "sdpa", + "use_cache": true + }, + "uncached_sdpa": { + "attention_backend": "sdpa", + "use_cache": false + }, + "uncached_default": { + "attention_backend": "auto", + "use_cache": false + } + }, + "weights": "seeded random initialization (timing does not depend on trained values)", + "grid_model_config": { + "vocab_size": 256, + "context_length": 1088, + "embedding_dim": 192, + "num_layers": 4, + "num_heads": 6, + "dropout": 0.0, + "positional_encoding": "learned", + "norm": "rmsnorm", + "activation": "swiglu", + "rope_theta": 10000.0 + }, + "window_model_config": { + "vocab_size": 256, + "context_length": 128, + "embedding_dim": 192, + "num_layers": 4, + "num_heads": 6, + "dropout": 0.0, + "positional_encoding": "learned", + "norm": "rmsnorm", + "activation": "swiglu", + "rope_theta": 10000.0 + }, + "decoding": "greedy argmax; step 0 is the prompt prefill" + }, + "results": { + "grid": [ + { + "batch": 1, + "prompt_len": 1, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_default": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 1.484042004449293, + "p95": 1.5200995985651389, + "min": 1.4645000046584755, + "max": 1.5203329967334867, + "iqr": 0.04695799725595862, + "mean": 1.4920498040737584, + "cv": 0.017579806791159294 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.6825829952722415, + "p95": 1.8291000029421414, + "min": 1.5283339889720082, + "max": 2.2975840111030266, + "iqr": 0.09460450382903218, + "mean": 1.677481596550298, + "cv": 0.05274812185556959 + }, + "total_ms": { + "count": 5, + "median": 106.68095800792798, + "p95": 109.08215848030522, + "min": 106.20666190516204, + "max": 109.68025008332916, + "iqr": 0.08050219912547618, + "mean": 107.17339038674254, + "cv": 0.013206675332811571 + }, + "tokens_per_second": { + "count": 5, + "median": 599.9196219745594, + "p95": 602.1435707889684, + "min": 583.5143515024469, + "max": 602.5987339395832, + "iqr": 0.4529703747903113, + "mean": 597.2451146829636, + "cv": 0.012988682611421061 + }, + "median_step_ms_by_index": [ + 1.484042004449293, + 1.6481249913340434, + 1.6837919974932447, + 1.6461250052088872, + 1.6865829966263846, + 1.6627500008326024, + 1.699625005130656, + 1.6671250050421804, + 1.685708004515618, + 1.5952910034684464, + 1.6216670046560466, + 1.6942080110311508, + 1.6806250059744343, + 1.6691249911673367, + 1.5901660080999136, + 1.6314999957103282, + 1.6587080026511103, + 1.6857500013429672, + 1.6881670017028227, + 1.7217920103576034, + 1.6673749923938885, + 1.7276670114370063, + 1.6735829995013773, + 1.640624992433004, + 1.586875005159527, + 1.69162500242237, + 1.6276249953079969, + 1.593374996446073, + 1.5799999964656308, + 1.5656670002499595, + 1.586875005159527, + 1.6772080125520006, + 1.6911660059122369, + 1.5955839917296544, + 1.6838750016177073, + 1.6903750074561685, + 1.6402920009568334, + 1.7112500063376501, + 1.6489999979967251, + 1.7032920004567131, + 1.7159580020233989, + 1.6810410015750676, + 1.7216669948538765, + 1.7128749896073714, + 1.632792002055794, + 1.7099999968195334, + 1.6921249916777015, + 1.632957995752804, + 1.7022079991875216, + 1.6506250103702769, + 1.729916999465786, + 1.7012500029522926, + 1.694124992354773, + 1.6884579963516444, + 1.6867499944055453, + 1.7010829906212166, + 1.6844579949975014, + 1.691417011898011, + 1.6442920023109764, + 1.7042079998645931, + 1.691832992946729, + 1.7059160018106923, + 1.6896249871933833, + 1.6957089974312112 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 1.4757500030100346, + "p95": 1.6238409967627376, + "min": 1.42937499913387, + "max": 1.6421659965999424, + "iqr": 0.08412498573306948, + "mean": 1.5128496015677229, + "cv": 0.055935801021473365 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.4672079996671528, + "p95": 1.6233581001870334, + "min": 1.2783749989466742, + "max": 1.95387500571087, + "iqr": 0.11170900688739493, + "mean": 1.4695223744234278, + "cv": 0.061370437699535145 + }, + "total_ms": { + "count": 5, + "median": 94.11578901926987, + "p95": 94.57229561230633, + "min": 93.24195900990162, + "max": 94.57496205868665, + "iqr": 0.5921737902099267, + "mean": 94.09275919024367, + "cv": 0.005801544085639437 + }, + "tokens_per_second": { + "count": 5, + "median": 680.0134246007992, + "p95": 685.3234858417256, + "min": 676.7118760278862, + "max": 686.3862651491875, + "iqr": 4.26508306452547, + "mean": 680.1982439874208, + "cv": 0.005822612585826578 + }, + "median_step_ms_by_index": [ + 1.4757500030100346, + 1.4963749999878928, + 1.5028329944470897, + 1.5425839956151322, + 1.5275000041583553, + 1.5571249969070777, + 1.6801669989945367, + 1.7106250015785918, + 1.3995000044815242, + 1.4265419886214659, + 1.4346669922815636, + 1.4094579964876175, + 1.4707090012961999, + 1.4775839954381809, + 1.3962090015411377, + 1.3553750031860545, + 1.4016250061104074, + 1.4135420060483739, + 1.3520000065909699, + 1.3889169931644574, + 1.405375005560927, + 1.3538329949369654, + 1.4010839950060472, + 1.392459002090618, + 1.3854579883627594, + 1.354541993350722, + 1.3809169904561713, + 1.4095830119913444, + 1.3976250047562644, + 1.375124993501231, + 1.3795830018352717, + 1.3998750073369592, + 1.4134579978417605, + 1.4507919986499473, + 1.4823750098003075, + 1.430916992831044, + 1.431832992238924, + 1.473416996304877, + 1.497707999078557, + 1.434125006198883, + 1.5012919902801514, + 1.475625002058223, + 1.444000008632429, + 1.4280000032158569, + 1.5112499968381599, + 1.509749999968335, + 1.5182089991867542, + 1.4957500097807497, + 1.5178750036284328, + 1.5420419949805364, + 1.5631670103175566, + 1.489959002356045, + 1.4888330042595044, + 1.516083997557871, + 1.5464169991901144, + 1.545332997920923, + 1.484583001001738, + 1.5209590055746958, + 1.5412920038215816, + 1.5666249964851886, + 1.5307919966289774, + 1.5220829955069348, + 1.5508749929722399, + 1.4925420109648257 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_default": { + "first_token_ms": { + "count": 5, + "median": 1.6102500085253268, + "p95": 1.7107919993577525, + "min": 1.550125001813285, + "max": 1.7281670006923378, + "iqr": 0.03154200385324657, + "mean": 1.6279167990433052, + "cv": 0.039955150177459334 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.5801249974174425, + "p95": 1.7333417883492077, + "min": 1.3682919961865991, + "max": 2.3685409978497773, + "iqr": 0.128771505842451, + "mean": 1.5847202379345184, + "cv": 0.06780531378906747 + }, + "total_ms": { + "count": 5, + "median": 101.18824997334741, + "p95": 102.15727396425791, + "min": 100.98491598910186, + "max": 102.26641595363617, + "iqr": 0.5545349849853665, + "mean": 101.46529178891797, + "cv": 0.005177754912323994 + }, + "tokens_per_second": { + "count": 5, + "median": 632.4845030609517, + "p95": 633.530920628792, + "min": 625.816397330432, + "max": 633.7580159685114, + "iqr": 3.4487701087346068, + "mean": 630.7710449581981, + "cv": 0.005160697341838709 + }, + "median_step_ms_by_index": [ + 1.6102500085253268, + 1.631832987186499, + 1.6782089951448143, + 1.6519170021638274, + 1.667791002546437, + 1.7651249945629388, + 1.8189580005127937, + 1.795083997421898, + 1.5123330085771158, + 1.5353750059148297, + 1.4668749936390668, + 1.591250009369105, + 1.5883330052020028, + 1.5633750008419156, + 1.5029999922262505, + 1.5593749994877726, + 1.455457997508347, + 1.4742920029675588, + 1.4890830061631277, + 1.505708001786843, + 1.5120000025490299, + 1.4692910044686869, + 1.504041996668093, + 1.5079589938977733, + 1.5117909933906049, + 1.4559999981429428, + 1.5087079955264926, + 1.5228340052999556, + 1.5245419926941395, + 1.4588749909307808, + 1.49049999890849, + 1.501917009591125, + 1.5818749961908907, + 1.5736670029582456, + 1.602874996024184, + 1.5632910071872175, + 1.583791003213264, + 1.5870420029386878, + 1.5969169908203185, + 1.5347500011557713, + 1.5437080000992864, + 1.5683749952586368, + 1.6250830085482448, + 1.5805419971002266, + 1.5440830029547215, + 1.618082998902537, + 1.6330840007867664, + 1.6018750029616058, + 1.6182499966816977, + 1.6026250086724758, + 1.600000003236346, + 1.6395000129705295, + 1.6610000020591542, + 1.643249997869134, + 1.6201249964069575, + 1.6170000017154962, + 1.6315000102622434, + 1.674333994742483, + 1.6068330005509779, + 1.6795420087873936, + 1.6651659971103072, + 1.6175829950952902, + 1.6593750042375177, + 1.6182499966816977 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "speedup_cached_vs_uncached_sdpa": 0.8822173214105901, + "speedup_cached_vs_uncached_default": 0.9485127605043406 + }, + { + "batch": 1, + "prompt_len": 16, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_default": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 1.3935419992776588, + "p95": 1.408558600815013, + "min": 1.317957998253405, + "max": 1.4091670018387958, + "iqr": 0.07079099304974079, + "mean": 1.3724251999519765, + "cv": 0.03107188660680898 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.7040420061675832, + "p95": 1.825208000082057, + "min": 1.5379579999716952, + "max": 2.5080420018639416, + "iqr": 0.09770800534170121, + "mean": 1.6989255452894472, + "cv": 0.05795170885906323 + }, + "total_ms": { + "count": 5, + "median": 108.74112094461452, + "p95": 109.27337918546982, + "min": 107.33262891881168, + "max": 109.37029597698711, + "iqr": 1.1917971132788807, + "mean": 108.40473455318715, + "cv": 0.00789705479758772 + }, + "tokens_per_second": { + "count": 5, + "median": 588.5538004762461, + "p95": 595.8771332157155, + "min": 585.1680241723622, + "max": 596.2772052141828, + "iqr": 6.504594730460894, + "mean": 590.4096251152044, + "cv": 0.00791078254432462 + }, + "median_step_ms_by_index": [ + 1.3935419992776588, + 1.9254169892519712, + 1.6957499901764095, + 1.7302500054938719, + 1.690541990683414, + 1.719042003969662, + 1.7187909979838878, + 1.742833002936095, + 1.614082997548394, + 1.6498330078320578, + 1.712750003207475, + 1.703791000181809, + 1.688042000751011, + 1.6607090074103326, + 1.6585410048719496, + 1.6744159947847947, + 1.6822500037960708, + 1.7259170126635581, + 1.7158330010715872, + 1.7246250063180923, + 1.7016249912558123, + 1.7187500052386895, + 1.7546250019222498, + 1.6518330085091293, + 1.7067080043489113, + 1.7241669993381947, + 1.709959004074335, + 1.7302909982390702, + 1.72266700246837, + 1.6089999990072101, + 1.7181669973069802, + 1.6697500104783103, + 1.7044580017682165, + 1.724792004097253, + 1.6986669943435118, + 1.7072499904315919, + 1.6508750122739002, + 1.6088340053102002, + 1.7088329914258793, + 1.7566669994266704, + 1.708959011011757, + 1.7093330097850412, + 1.6905000084079802, + 1.680624991422519, + 1.7233329999726266, + 1.720750005915761, + 1.7283329943893477, + 1.7151669890154153, + 1.6777919954620302, + 1.6386249917559326, + 1.7033330077538267, + 1.7275419959332794, + 1.591250009369105, + 1.7119159892899916, + 1.703040994470939, + 1.7265829956158996, + 1.639459005673416, + 1.6490419948240742, + 1.6962089866865426, + 1.7242080066353083, + 1.625791992410086, + 1.6937920008786023, + 1.6960420034592971, + 1.709125004708767 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 1.4066250005271286, + "p95": 1.4260338037274778, + "min": 1.3321669975994155, + "max": 1.4289590035332367, + "iqr": 0.07395799912046641, + "mean": 1.3844918023096398, + "cv": 0.03238609148183435 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.4833750028628856, + "p95": 1.6292669999529608, + "min": 1.2975000136066228, + "max": 1.7593329976079985, + "iqr": 0.11525049922056496, + "mean": 1.4811889395036453, + "cv": 0.05840049356550385 + }, + "total_ms": { + "count": 5, + "median": 94.9674590228824, + "p95": 95.35981378576253, + "min": 93.09971702168696, + "max": 95.41071398416534, + "iqr": 0.2933410578407347, + "mean": 94.6993949910393, + "cv": 0.00969572096403206 + }, + "tokens_per_second": { + "count": 5, + "median": 673.9150510974418, + "p95": 684.8795754646403, + "min": 670.7842057509562, + "max": 687.4349573489211, + "iqr": 2.079789634715098, + "mean": 675.8741040835278, + "cv": 0.009807066169571471 + }, + "median_step_ms_by_index": [ + 1.4066250005271286, + 1.4132919895928353, + 1.3826250069541857, + 1.424458998371847, + 1.426875009201467, + 1.357750006718561, + 1.344417003565468, + 1.4022499963175505, + 1.4180419966578484, + 1.3439579925034195, + 1.4099999971222132, + 1.4380420034285635, + 1.3385830097831786, + 1.3995420013088733, + 1.4204159961082041, + 1.3411660038400441, + 1.4027500001247972, + 1.4321670023491606, + 1.4618749992223457, + 1.462875006836839, + 1.4140830026008189, + 1.4635830011684448, + 1.4407499984372407, + 1.4635419938713312, + 1.4995829988038167, + 1.4900839887559414, + 1.505708001786843, + 1.426875009201467, + 1.4923330018064007, + 1.4957079984014854, + 1.4991249918239191, + 1.4521670091198757, + 1.5187079989118502, + 1.3891249982407317, + 1.492208000854589, + 1.492083989433013, + 1.4594159874832258, + 1.5480830043088645, + 1.5476249973289669, + 1.4681250031571835, + 1.4795419992879033, + 1.5541669999947771, + 1.5528750082012266, + 1.5204999945126474, + 1.4999170089140534, + 1.485749991843477, + 1.476417004596442, + 1.5288750000763685, + 1.4686250069644302, + 1.5492499951506034, + 1.6175830096472055, + 1.5350419998867437, + 1.5800410037627444, + 1.537584001198411, + 1.5502500027650967, + 1.5290000010281801, + 1.616583002032712, + 1.550125001813285, + 1.5332500042859465, + 1.550332992337644, + 1.6107080009533092, + 1.5307080029742792, + 1.541875011753291, + 1.5309580048779026 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_default": { + "first_token_ms": { + "count": 5, + "median": 1.4990830095484853, + "p95": 1.5518334053922445, + "min": 1.4134999946691096, + "max": 1.564000005600974, + "iqr": 0.02620801387820393, + "mean": 1.4913418010110036, + "cv": 0.03633480291322903 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.602333999471739, + "p95": 1.7384208054863848, + "min": 1.3574580079875886, + "max": 1.7917500081239268, + "iqr": 0.13462499191518873, + "mean": 1.5988362633848623, + "cv": 0.057125027023657154 + }, + "total_ms": { + "count": 5, + "median": 102.087000006577, + "p95": 102.93947478348855, + "min": 101.61974999937229, + "max": 103.07241496047936, + "iqr": 0.5044611461926252, + "mean": 102.21802639425732, + "cv": 0.005447270530345362 + }, + "tokens_per_second": { + "count": 5, + "median": 626.9162576613749, + "p95": 629.4484033412069, + "min": 620.9226787258187, + "max": 629.7988333999575, + "iqr": 3.093762540081457, + "mean": 626.1274746918958, + "cv": 0.005431019817418244 + }, + "median_step_ms_by_index": [ + 1.4990830095484853, + 1.530334004200995, + 1.432207995094359, + 1.5119579911697656, + 1.4780420024180785, + 1.444834007997997, + 1.5205839881673455, + 1.5431659994646907, + 1.4430000010179356, + 1.426167000317946, + 1.5243340021697804, + 1.5490829973714426, + 1.516999996965751, + 1.5379999967990443, + 1.503666993812658, + 1.5372920024674386, + 1.4928330056136474, + 1.5543339977739379, + 1.5842080028960481, + 1.5961249882820994, + 1.6233330097747967, + 1.6075409948825836, + 1.5299999940907583, + 1.6092919977381825, + 1.5195410087471828, + 1.6305830067722127, + 1.6073750011855736, + 1.5780409885337576, + 1.5985420031938702, + 1.6060000052675605, + 1.5822499990463257, + 1.5652919973945245, + 1.6345830081263557, + 1.6338749992428347, + 1.6632089973427355, + 1.6806670028017834, + 1.6841670003486797, + 1.589542007423006, + 1.5943339967634529, + 1.6417910082964227, + 1.7026669956976548, + 1.6749170026741922, + 1.6515840106876567, + 1.659459012444131, + 1.5848329931031913, + 1.655208005104214, + 1.6451249975943938, + 1.6483749932376668, + 1.6327499906765297, + 1.7392089939676225, + 1.6555000038351864, + 1.6832090041134506, + 1.6945840034168214, + 1.6979159991024062, + 1.633332998608239, + 1.6830419917823747, + 1.670665995334275, + 1.6925000090850517, + 1.7285420035477728, + 1.7222919996129349, + 1.7127499886555597, + 1.6936669999267906, + 1.683624999714084, + 1.671624995651655 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "speedup_cached_vs_uncached_sdpa": 0.8733352957732751, + "speedup_cached_vs_uncached_default": 0.9388076848920227 + }, + { + "batch": 1, + "prompt_len": 128, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_default": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 1.7709579988149926, + "p95": 1.7892254050821066, + "min": 1.74079199496191, + "max": 1.7922920087585226, + "iqr": 0.032292999094352126, + "mean": 1.7651333968387917, + "cv": 0.012421018288744446 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.7231249948963523, + "p95": 1.8347955992794596, + "min": 1.5762499970151111, + "max": 2.4730839941184968, + "iqr": 0.0835210012155585, + "mean": 1.718812034648311, + "cv": 0.05227251055406031 + }, + "total_ms": { + "count": 5, + "median": 110.12311992817558, + "p95": 111.10519381472841, + "min": 109.22241397202015, + "max": 111.29891901509836, + "iqr": 1.0535810433793813, + "mean": 110.05029157968238, + "cv": 0.007772471792047785 + }, + "tokens_per_second": { + "count": 5, + "median": 581.1676970443812, + "p95": 585.902083100325, + "min": 575.0280466903548, + "max": 585.9603141200952, + "iqr": 5.592751607780315, + "mean": 581.5803248579077, + "cv": 0.0077502922409912 + }, + "median_step_ms_by_index": [ + 1.7709579988149926, + 1.9006670045200735, + 1.733208991936408, + 1.7229579971171916, + 1.671624995651655, + 1.749584000208415, + 1.7277089937124401, + 1.7490829923190176, + 1.7237919964827597, + 1.7003749962896109, + 1.6385000053560361, + 1.7214999970747158, + 1.7279999883612618, + 1.679333989159204, + 1.7104579892475158, + 1.7220419977093115, + 1.6687089955667034, + 1.693583995802328, + 1.7267500079469755, + 1.7050840106094256, + 1.7152920045191422, + 1.6750420036260039, + 1.7373750015394762, + 1.700584005448036, + 1.7269999952986836, + 1.7194160027429461, + 1.6122919914778322, + 1.7367910040775314, + 1.7224159964825958, + 1.6875839937711135, + 1.677332998951897, + 1.7445830017095432, + 1.7138749972218648, + 1.7242500034626573, + 1.6967499977909029, + 1.6785830084700137, + 1.6754170064814389, + 1.707625007838942, + 1.7399999924236909, + 1.7087090091081336, + 1.7377920012222603, + 1.7392919980920851, + 1.7340839985990897, + 1.731790995108895, + 1.7187920020660385, + 1.7284159985138103, + 1.7482499970356002, + 1.752209005644545, + 1.7144169978564605, + 1.7368330009048805, + 1.7270419921260327, + 1.7462910036556423, + 1.73366600938607, + 1.6793329996289685, + 1.7718750023050234, + 1.7442079988541082, + 1.7437500064261258, + 1.748915994539857, + 1.7185420001624152, + 1.7307079979218543, + 1.7311249976046383, + 1.7214579856954515, + 1.6717920079827309, + 1.7270410025957972 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 1.8090829980792478, + "p95": 1.951558396103792, + "min": 1.7171669896924868, + "max": 1.985124996281229, + "iqr": 0.06408299668692052, + "mean": 1.816375195630826, + "cv": 0.05665738242528713 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.935583000886254, + "p95": 2.2224079977604565, + "min": 1.737541999318637, + "max": 2.5349170027766377, + "iqr": 0.10783400648506358, + "mean": 1.9496965556061783, + "cv": 0.06306046085363025 + }, + "total_ms": { + "count": 5, + "median": 124.16100199334323, + "p95": 127.12399658048525, + "min": 122.67975497525185, + "max": 127.51287096762098, + "iqr": 2.254335006000474, + "mean": 124.64725819882005, + "cv": 0.015507304987729181 + }, + "tokens_per_second": { + "count": 5, + "median": 515.4597576735995, + "p95": 521.1466912861417, + "min": 501.9101171069339, + "max": 521.6834677645933, + "iqr": 9.317615026257215, + "mean": 513.546979652708, + "cv": 0.015392661977237147 + }, + "median_step_ms_by_index": [ + 1.8090829980792478, + 1.9062919891439378, + 1.8964170012623072, + 1.8746669957181439, + 1.8950829980894923, + 1.9046659872401506, + 1.9111670117126778, + 1.9068749970756471, + 1.8780419923132285, + 1.8916669941972941, + 1.8599159957375377, + 1.905917000840418, + 1.9213750056223944, + 1.8879590061260387, + 1.8908339989138767, + 1.919957998325117, + 1.860333009972237, + 1.8977080035256222, + 1.8551250104792416, + 1.9126249972032383, + 1.8682919908314943, + 1.902625008369796, + 1.8705840047914535, + 1.867042010417208, + 1.8781670078169554, + 1.9035829900531098, + 1.8974580016219988, + 1.8208330002380535, + 1.8589589890325442, + 1.8819999968400225, + 1.9305000023450702, + 1.8545409984653816, + 1.8890420033130795, + 1.9573339959606528, + 1.9619579979917035, + 1.9079579942626879, + 1.9634999916888773, + 1.9673339993460104, + 1.9557499908842146, + 1.9084160012425855, + 1.9192920008208603, + 2.0014589972561225, + 1.96508300723508, + 1.9170840096194297, + 2.0346660021459684, + 1.992707999306731, + 1.9867500086547807, + 1.9905000081053004, + 1.97387499792967, + 2.0120410044910386, + 1.970625002286397, + 1.9767920020967722, + 1.9902919884771109, + 1.999083993723616, + 1.9916659948648885, + 1.9573330064304173, + 1.9970419962191954, + 2.0030829909956083, + 1.9949169945903122, + 1.9979170028818771, + 1.9841250032186508, + 1.9983749953098595, + 2.0244159968569875, + 1.9918330071959645 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_default": { + "first_token_ms": { + "count": 5, + "median": 1.9668750028358772, + "p95": 2.057784007047303, + "min": 1.959666988113895, + "max": 2.0717090083053336, + "iqr": 0.036667988752014935, + "mean": 1.9931502029066905, + "cv": 0.023575418022929708 + }, + "decode_ms_per_step": { + "count": 315, + "median": 2.0705829956568778, + "p95": 2.2493875949294306, + "min": 1.8834999937098473, + "max": 2.4474579986417666, + "iqr": 0.14635499974247068, + "mean": 2.0868456540887967, + "cv": 0.04680775921872352 + }, + "total_ms": { + "count": 5, + "median": 133.38932901388034, + "p95": 133.64526880905032, + "min": 133.3165470714448, + "max": 133.64704401465133, + "iqr": 0.3071240207646042, + "mean": 133.4644264105009, + "cv": 0.0012359104750341214 + }, + "tokens_per_second": { + "count": 5, + "median": 479.79850017343017, + "p95": 480.04999889429104, + "min": 478.8732924985896, + "max": 480.06043815177856, + "iqr": 1.103143386822353, + "mean": 479.5291142331315, + "cv": 0.0012354595279435627 + }, + "median_step_ms_by_index": [ + 1.9668750028358772, + 2.008916999329813, + 2.0216249977238476, + 1.992332996451296, + 2.0366670069051906, + 1.9974169990746304, + 1.9775830005528405, + 1.9726659957086667, + 2.049292001174763, + 1.9682499987538904, + 1.9452080014161766, + 1.9949999987147748, + 2.043040993157774, + 1.9873330020345747, + 1.949749988853, + 1.9872499979101121, + 2.0052499894518405, + 2.028458009590395, + 2.013082994380966, + 1.9953330047428608, + 2.021499996772036, + 2.0387499971548095, + 2.0182910084258765, + 2.041874991846271, + 2.0442500099306926, + 1.9933339935960248, + 2.0073749910807237, + 2.028916002018377, + 2.029665993177332, + 2.030292002018541, + 2.0532080088742077, + 2.044540990027599, + 2.0210000075167045, + 2.1159169991733506, + 2.147417006199248, + 2.1383749990491197, + 2.1522500028368086, + 2.119208002113737, + 2.114041999448091, + 2.107374995830469, + 2.1029579947935417, + 2.106625004671514, + 2.100292011164129, + 2.1237909968476743, + 2.1395420044427738, + 2.129625005181879, + 2.140500000678003, + 2.17500000144355, + 2.224875002866611, + 2.209041005698964, + 2.170291991205886, + 2.224500000011176, + 2.1918330021435395, + 2.177874994231388, + 2.1974170085741207, + 2.189874998293817, + 2.228708006441593, + 2.187042002333328, + 2.2489999973913655, + 2.181958989240229, + 2.2005829960107803, + 2.1465420140884817, + 2.1912500087637454, + 2.1935829863650724 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "speedup_cached_vs_uncached_sdpa": 1.1274744311124079, + "speedup_cached_vs_uncached_default": 1.211274518020189 + }, + { + "batch": 1, + "prompt_len": 512, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_default": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 3.9490000053774565, + "p95": 4.004691797308624, + "min": 3.5432500008028, + "max": 4.011124998214655, + "iqr": 0.036999990697950125, + "mean": 3.8848586002131924, + "cv": 0.049658896800037225 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.811375012039207, + "p95": 2.339175001543481, + "min": 1.6522919904673472, + "max": 2.6307090010959655, + "iqr": 0.42020799446618184, + "mean": 1.9274194634312556, + "cv": 0.12284748997731353 + }, + "total_ms": { + "count": 5, + "median": 126.12604101013858, + "p95": 128.1566950114211, + "min": 118.74679299944546, + "max": 128.35700501454994, + "iqr": 1.3793250400340185, + "mean": 125.3122847963823, + "cv": 0.0302962558874362 + }, + "tokens_per_second": { + "count": 5, + "median": 507.42891386605396, + "p95": 532.7760894565488, + "min": 498.60932788783333, + "max": 538.9619237994821, + "iqr": 5.502255840662656, + "mean": 511.1126827764676, + "cv": 0.03137485354337352 + }, + "median_step_ms_by_index": [ + 3.9490000053774565, + 2.5329159980174154, + 2.3050000017974526, + 2.2876250004628673, + 2.320374987903051, + 2.0686670031864196, + 2.2873340058140457, + 2.2588330029975623, + 1.9640410027932376, + 2.2284169972408563, + 2.2054169967304915, + 2.287041992531158, + 2.232124999864027, + 1.8977080035256222, + 1.8023329903371632, + 2.2743750014342368, + 2.1050420036772266, + 2.265874994918704, + 2.1973330003675073, + 2.032500007771887, + 1.788749999832362, + 1.7962080019060522, + 1.8061250011669472, + 1.7912920011440292, + 1.754124998115003, + 1.80245800584089, + 1.7954580107470974, + 2.1249999990686774, + 1.9425000064074993, + 1.8094590050168335, + 1.864208999904804, + 1.8035419925581664, + 2.2156670020194724, + 1.8299579969607294, + 1.7088330059777945, + 1.8089170043822378, + 1.795666990801692, + 1.754333992721513, + 1.822499994887039, + 1.8158750026486814, + 1.7913329938892275, + 1.726292000967078, + 1.7432909953640774, + 1.78974999289494, + 1.7932500049937516, + 1.7947080050362274, + 1.7970420012716204, + 1.7957499949261546, + 1.7913329938892275, + 1.7919169913511723, + 1.7729590035742149, + 1.769916998455301, + 1.7995420057559386, + 1.8034579989034683, + 1.7859999934444204, + 1.8237500044051558, + 1.8009170016739517, + 1.7978329997276887, + 1.8160420004278421, + 1.7905830027302727, + 1.741125000989996, + 1.772165996953845, + 1.8004159937845543, + 1.810874993680045 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 3.534457995556295, + "p95": 3.97159120766446, + "min": 3.4797499974956736, + "max": 4.000291010015644, + "iqr": 0.3455419937381521, + "mean": 3.6765082011697814, + "cv": 0.06430014762640826 + }, + "decode_ms_per_step": { + "count": 315, + "median": 3.9492079959018156, + "p95": 4.348437506996561, + "min": 3.5507499997038394, + "max": 4.754708003019914, + "iqr": 0.38947900611674413, + "mean": 3.9702656069060875, + "cv": 0.05839076949990394 + }, + "total_ms": { + "count": 5, + "median": 252.61091203719843, + "p95": 259.8154108040035, + "min": 246.4102100202581, + "max": 259.99275299545843, + "iqr": 8.209751948015764, + "mean": 253.8032414362533, + "cv": 0.02254446734867868 + }, + "tokens_per_second": { + "count": 5, + "median": 253.35405934711017, + "p95": 258.80069344232595, + "min": 246.1607074144792, + "max": 259.7294973886771, + "iqr": 8.082360722394213, + "mean": 252.26657174834298, + "cv": 0.02258207245421512 + }, + "median_step_ms_by_index": [ + 3.534457995556295, + 3.694457991514355, + 3.676916006952524, + 3.64754100155551, + 3.667208002298139, + 3.777291000005789, + 3.7537080061156303, + 3.7152909935684875, + 3.712000005180016, + 3.772916999878362, + 3.672749997349456, + 3.835166004137136, + 3.736541999387555, + 3.80587499239482, + 3.7098749889992177, + 3.8184589939191937, + 3.9867500017862767, + 3.7726659938925877, + 3.7526250089285895, + 3.9038329996401444, + 3.931915998691693, + 3.8745839992770925, + 3.7823750026291236, + 4.129207998630591, + 3.9770000003045425, + 3.788125002756715, + 3.721499990206212, + 3.777833000640385, + 4.189833009149879, + 4.108166991500184, + 3.7613750027958304, + 3.8573750061914325, + 3.938375011784956, + 3.960083005949855, + 4.08570900617633, + 3.8346660003298894, + 3.8484160031657666, + 3.9059999980963767, + 3.9457500097341835, + 3.944833006244153, + 3.9035000081639737, + 4.034708996186964, + 4.139416996622458, + 3.894458001013845, + 3.978166001616046, + 4.180916002951562, + 4.042415996082127, + 4.253291001077741, + 4.198624999844469, + 3.9600420132046565, + 4.0069999959087, + 3.9891659980639815, + 4.1667079931357875, + 4.083624997292645, + 4.060124992975034, + 4.053459007991478, + 3.922207994037308, + 3.9104169991333038, + 4.245417003403418, + 4.033082994283177, + 3.9543330058222637, + 4.0800000133458525, + 4.299334003007971, + 4.0682079998077825 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_default": { + "first_token_ms": { + "count": 5, + "median": 4.0567079995526, + "p95": 4.472283602808602, + "min": 4.000791988801211, + "max": 4.494167005759664, + "iqr": 0.3287499857833609, + "mean": 4.198483398067765, + "cv": 0.05347257969169888 + }, + "decode_ms_per_step": { + "count": 315, + "median": 4.768707993207499, + "p95": 5.19983300328022, + "min": 4.06362499052193, + "max": 6.690750000416301, + "iqr": 0.5561459984164685, + "mean": 4.7472914189769195, + "cv": 0.08681107150720188 + }, + "total_ms": { + "count": 5, + "median": 301.0087039292557, + "p95": 315.64143403666094, + "min": 291.4243769919267, + "max": 317.43379203544464, + "iqr": 10.421663071610965, + "mean": 303.2778427936137, + "cv": 0.03298751336307654 + }, + "tokens_per_second": { + "count": 5, + "median": 212.61843649226017, + "p95": 218.63456876038705, + "min": 201.61684611338973, + "max": 219.61100392700843, + "iqr": 7.254569242414618, + "mean": 211.20987469560936, + "cv": 0.032710677919541034 + }, + "median_step_ms_by_index": [ + 4.0567079995526, + 4.276291001588106, + 4.226749995723367, + 4.261583992047235, + 4.220041009830311, + 4.295791004551575, + 4.274917009752244, + 4.292415993404575, + 4.42591700993944, + 4.6019169967621565, + 4.332291995524429, + 4.328082999563776, + 4.41783299902454, + 4.288084004656412, + 4.315708996728063, + 4.301082997699268, + 4.253207996953279, + 4.309124997234903, + 4.273667000234127, + 4.743749988847412, + 4.610916003002785, + 4.725124992546625, + 4.725666993181221, + 4.7403749922523275, + 4.678792000049725, + 4.691999987699091, + 4.717124989838339, + 4.33149999298621, + 4.786833000252955, + 4.817000008188188, + 4.8658330051694065, + 4.768707993207499, + 4.423167003551498, + 4.880834007053636, + 4.863499998464249, + 4.891041011433117, + 4.872833000263199, + 4.853207996347919, + 4.9001669976860285, + 4.856375002418645, + 4.733416993985884, + 4.963166997185908, + 4.925332992570475, + 4.9090410029748455, + 4.971750007825904, + 4.871042008744553, + 4.741915996419266, + 4.8607499920763075, + 4.901584004983306, + 4.943416002788581, + 5.030708009144291, + 4.974374998710118, + 5.003250000299886, + 4.994457995053381, + 5.056207999587059, + 5.069833001471125, + 4.946792003465816, + 5.024208992836066, + 4.998582997359335, + 4.9822080036392435, + 4.854500002693385, + 4.985167004633695, + 5.057542002759874, + 4.941541003063321 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "speedup_cached_vs_uncached_sdpa": 2.0028450113398266, + "speedup_cached_vs_uncached_default": 2.3865706202976695 + }, + { + "batch": 1, + "prompt_len": 1024, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_default": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 10.173041009693407, + "p95": 17.954358196584508, + "min": 7.108584002708085, + "max": 19.12774999800604, + "iqr": 3.7007490027463064, + "mean": 11.846041597891599, + "cv": 0.39026723113449396 + }, + "decode_ms_per_step": { + "count": 315, + "median": 2.4368749873246998, + "p95": 3.7676670006476343, + "min": 1.737791986670345, + "max": 6.174042006023228, + "iqr": 0.40372902003582567, + "mean": 2.5936149557339148, + "cv": 0.21559671358558252 + }, + "total_ms": { + "count": 5, + "median": 160.11728896410204, + "p95": 222.65196423395537, + "min": 149.13895801873878, + "max": 233.47528804151807, + "iqr": 25.22995398612693, + "mean": 175.24378380912822, + "cv": 0.19692978308018658 + }, + "tokens_per_second": { + "count": 5, + "median": 399.70699238074576, + "p95": 426.3514654035008, + "min": 274.1189465354428, + "max": 429.12999292886724, + "iqr": 58.41044330773326, + "mean": 375.00403982827856, + "cv": 0.16688756995554063 + }, + "median_step_ms_by_index": [ + 10.173041009693407, + 3.438125000684522, + 2.5973750016419217, + 3.002041994477622, + 2.7344159898348153, + 2.5607499992474914, + 2.5392079987796023, + 2.474417007761076, + 2.4368749873246998, + 2.6573330105748028, + 2.2695410007145256, + 2.539374996558763, + 2.311417003511451, + 2.4154170096153393, + 2.61987499834504, + 2.6221669977530837, + 2.419708005618304, + 2.73800001014024, + 2.5330830103484914, + 2.522584007238038, + 2.4021249992074445, + 2.6146669988520443, + 2.4141670000972226, + 2.460083007463254, + 2.3887089919298887, + 2.609041999676265, + 2.5171669985866174, + 2.420167002128437, + 2.390000008745119, + 2.431957997032441, + 2.537833002861589, + 2.4612909910501912, + 2.3729160020593554, + 2.3472500033676624, + 2.281833003507927, + 2.3415000032400712, + 2.442708006128669, + 2.3895000049378723, + 2.3415000032400712, + 2.3805829987395555, + 2.381250000325963, + 2.388499997323379, + 2.436875001876615, + 2.350583003135398, + 2.4036669929046184, + 2.5491670094197616, + 2.409417007584125, + 2.35499998962041, + 2.28266698832158, + 2.427958999760449, + 2.420083008473739, + 2.4470839998684824, + 2.406750005320646, + 2.3352499993052334, + 2.4428750039078295, + 2.275916005601175, + 2.459750001435168, + 2.3844169918447733, + 2.2831249953014776, + 2.42733299091924, + 2.358624988119118, + 2.402207988779992, + 2.2605830017710105, + 2.3421670048264787 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 7.533583004260436, + "p95": 8.16888339468278, + "min": 6.297750005614944, + "max": 8.180624994565733, + "iqr": 1.2150840047979727, + "mean": 7.408141597989015, + "cv": 0.1089598187186531 + }, + "decode_ms_per_step": { + "count": 315, + "median": 8.145917003275827, + "p95": 9.91427520348225, + "min": 6.3815000030444935, + "max": 12.47729200986214, + "iqr": 1.3313954914337955, + "mean": 8.29158743241957, + "cv": 0.12277128524106326 + }, + "total_ms": { + "count": 5, + "median": 547.4534130044049, + "p95": 571.9360072514974, + "min": 451.71579204907175, + "max": 576.3081230397802, + "iqr": 35.48166708787903, + "mean": 529.778149840422, + "cv": 0.09099963559223766 + }, + "tokens_per_second": { + "count": 5, + "median": 116.90492465609132, + "p95": 138.01004030912975, + "min": 111.05170557448892, + "max": 141.68200697541124, + "iqr": 7.891957239896897, + "mean": 121.67820545082043, + "cv": 0.09875755815235426 + }, + "median_step_ms_by_index": [ + 7.533583004260436, + 7.551999995484948, + 7.938916998682544, + 8.062749999226071, + 7.850541005609557, + 8.028249998460524, + 8.10816598823294, + 7.714374994975515, + 8.190667009330355, + 7.963500000187196, + 8.563583003706299, + 7.711917001870461, + 8.46258299134206, + 7.824208005331457, + 8.577250002417713, + 8.24037499842234, + 8.529041006113403, + 8.34558400674723, + 8.230583000113256, + 7.930749998195097, + 8.272417006082833, + 8.634666999569163, + 8.188457999494858, + 8.46000001183711, + 7.934542009024881, + 8.131458002026193, + 7.897500006947666, + 8.414292009547353, + 8.2668749964796, + 8.145917003275827, + 8.103833009954542, + 8.477417009999044, + 7.958375004818663, + 8.506166996085085, + 8.018791995709762, + 8.037708001211286, + 8.22741599404253, + 8.646125002996996, + 8.151333997375332, + 8.129082998493686, + 7.892791996710002, + 8.168499989551492, + 8.724041006644256, + 8.332667013746686, + 8.22633299685549, + 7.95458399807103, + 7.8349589894060045, + 8.269290992757306, + 7.98820800264366, + 8.178541000233963, + 7.8025419934419915, + 8.12354200752452, + 7.870000001275912, + 8.046917006140575, + 7.811417002812959, + 7.980084003065713, + 7.856542011722922, + 7.897874995251186, + 7.8168339969124645, + 7.85024999640882, + 7.789166003931314, + 7.778083992889151, + 7.8209999919636175, + 7.817707999492995 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_default": { + "first_token_ms": { + "count": 5, + "median": 13.754166997387074, + "p95": 18.442983605200425, + "min": 8.754125010455027, + "max": 19.597417005570605, + "iqr": 1.7210829973919317, + "mean": 13.607025204692036, + "cv": 0.28878326695470896 + }, + "decode_ms_per_step": { + "count": 315, + "median": 14.038666005944833, + "p95": 20.62667479622178, + "min": 8.450166002148762, + "max": 27.975499993772246, + "iqr": 5.464979003590997, + "mean": 14.432636910766762, + "cv": 0.2624491172440428 + }, + "total_ms": { + "count": 5, + "median": 985.7982890534913, + "p95": 1103.3204601641046, + "min": 603.5409599571722, + "max": 1124.578209972242, + "iqr": 136.1806279310258, + "mean": 922.8631505829981, + "cv": 0.21493758869996152 + }, + "tokens_per_second": { + "count": 5, + "median": 64.92200352817537, + "p95": 99.34336589965795, + "min": 56.91022592513127, + "max": 106.04085595870991, + "iqr": 9.702907396040693, + "mean": 72.65539786857522, + "cv": 0.26816526514757155 + }, + "median_step_ms_by_index": [ + 13.754166997387074, + 13.706707992241718, + 13.410792002105154, + 13.463375013088807, + 13.992375010275282, + 14.160250007989816, + 14.071915997192264, + 13.01525000599213, + 12.490124994656071, + 11.868540997966193, + 12.289082995266654, + 12.068957992596552, + 12.352124991593882, + 12.183540995465592, + 12.432583011104725, + 12.91487499838695, + 13.786500014248304, + 12.835875008022413, + 13.23054201202467, + 13.823375003994443, + 13.430457998765633, + 13.3228750055423, + 13.71933399059344, + 13.849041992216371, + 14.231875000405125, + 14.396041995496489, + 13.67295799718704, + 13.27070799015928, + 13.783874994260259, + 14.172749986755662, + 14.164457999868318, + 13.840334009728394, + 13.95695899554994, + 13.702416006708518, + 14.181375008774921, + 15.033666000817902, + 14.262708995374851, + 14.420040999539196, + 15.239041997119784, + 15.339541991124861, + 15.729667007690296, + 15.672166002332233, + 15.642416008631699, + 14.721375002409332, + 15.264916990417987, + 15.61941699765157, + 14.25283300341107, + 17.014999990351498, + 15.462249997653998, + 14.85004099959042, + 15.123499993933365, + 15.238915992085822, + 15.457417001016438, + 16.38312499562744, + 15.106459002709016, + 14.156958000967279, + 14.21262499934528, + 14.290040999185294, + 14.619207999203354, + 14.42570801009424, + 14.554666995536536, + 14.120084000751376, + 14.40833299420774, + 13.381542012211867 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "speedup_cached_vs_uncached_sdpa": 3.4190774559463266, + "speedup_cached_vs_uncached_default": 6.156726081432126 + }, + { + "batch": 8, + "prompt_len": 1, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_default": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 1.8252910085720941, + "p95": 1.833074796013534, + "min": 1.6782500024419278, + "max": 1.8334579945076257, + "iqr": 0.0484589982079342, + "mean": 1.7903248022776097, + "cv": 0.03683430776632052 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.9739999988814816, + "p95": 2.132783296110574, + "min": 1.7797079926822335, + "max": 2.566750001278706, + "iqr": 0.09025000326801091, + "mean": 1.987286199795245, + "cv": 0.05653874041247432 + }, + "total_ms": { + "count": 5, + "median": 127.22041302185971, + "p95": 128.4260541840922, + "min": 125.37341393181123, + "max": 128.6202439951012, + "iqr": 1.565883881994523, + "mean": 126.98935538937803, + "cv": 0.010100409379532844 + }, + "tokens_per_second": { + "count": 5, + "median": 4024.511380198281, + "p95": 4079.201101332217, + "min": 3980.7108437727793, + "max": 4083.8004162387197, + "iqr": 49.81419824257728, + "mean": 4032.1632250759235, + "cv": 0.010105213112341669 + }, + "median_step_ms_by_index": [ + 1.8252910085720941, + 1.9495840097079054, + 1.95237499428913, + 1.975582999875769, + 1.8732499884208664, + 1.9698340038303286, + 1.997290994040668, + 1.9672500056913123, + 2.0320409967098385, + 1.9540830107871443, + 1.9501249917084351, + 1.9853749981848523, + 1.889166989712976, + 1.9447090016910806, + 1.9297080143587664, + 1.9334160024300218, + 1.9666670123115182, + 1.9520420028129593, + 1.879582996480167, + 1.968207987374626, + 1.9515000021783635, + 1.8722079985309392, + 1.941499998793006, + 1.944959003594704, + 1.920750000863336, + 1.912708001327701, + 1.9174169865436852, + 1.984749993425794, + 1.9773749954765663, + 1.9768750062212348, + 1.9942499930039048, + 2.0185419998597354, + 2.012041994021274, + 1.9856250000884756, + 1.937875000294298, + 1.9416669965721667, + 1.9881250045727938, + 1.9444580102572218, + 1.9233330094721168, + 1.962250011274591, + 1.9987499981652945, + 1.9806670024991035, + 2.0441249944269657, + 1.9739999988814816, + 1.9966659892816097, + 1.9939589983550832, + 1.9469160033622757, + 1.9927500106859952, + 1.9651660113595426, + 1.9930840062443167, + 2.015999998548068, + 2.0038750080857426, + 1.9709579937625676, + 2.0602919976226985, + 2.0427499985089526, + 2.0269579981686547, + 2.0354579901322722, + 2.0520840043900535, + 2.0685419876826927, + 2.0552080095512792, + 2.0737909944728017, + 2.028125003562309, + 2.087000000756234, + 2.0760829938808456 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 1.759582999511622, + "p95": 2.1662754006683826, + "min": 1.6465840017190203, + "max": 2.2402920003514737, + "iqr": 0.1282920129597187, + "mean": 1.8517169984988868, + "cv": 0.12489883446263685 + }, + "decode_ms_per_step": { + "count": 315, + "median": 2.4264999956358224, + "p95": 3.330100000312086, + "min": 1.514875009888783, + "max": 3.587583007174544, + "iqr": 0.9910414883051999, + "mean": 2.478418765415896, + "cv": 0.22336992471128497 + }, + "total_ms": { + "count": 5, + "median": 156.27508307807148, + "p95": 163.65204053581692, + "min": 152.73578904452734, + "max": 164.30258193577174, + "iqr": 5.452707831864245, + "mean": 157.99209921970032, + "cv": 0.029260325242693223 + }, + "tokens_per_second": { + "count": 5, + "median": 3276.2740541575427, + "p95": 3339.864891038553, + "min": 3116.2017904268127, + "max": 3352.19402867481, + "iqr": 111.40895771864052, + "mean": 3242.871519305515, + "cv": 0.029028953750545303 + }, + "median_step_ms_by_index": [ + 1.759582999511622, + 1.7216249980265275, + 1.7968329921131954, + 1.7168750055134296, + 1.9289579940959811, + 1.8525829946156591, + 1.9300830026622862, + 1.8806250009220093, + 1.6457090096082538, + 1.673209000728093, + 1.6756659897509962, + 1.6930000128922984, + 1.793707997421734, + 1.7753329884726554, + 1.7261250031879172, + 1.7522080015623942, + 1.9287909963168204, + 1.9361669983481988, + 1.936000000569038, + 1.9134589965688065, + 2.0157910039415583, + 2.0357089961180463, + 2.0767920068465173, + 2.046791007160209, + 2.1132080000825226, + 2.1082500024931505, + 2.14508299541194, + 2.25312499969732, + 2.2914580040378496, + 2.2552500013262033, + 2.3603330046171322, + 2.219165995484218, + 2.4469170020893216, + 2.4660829949425533, + 2.4916250113165006, + 2.581458000349812, + 2.6082080003106967, + 2.664625004399568, + 2.5427499931538478, + 2.5858750013867393, + 2.6468340074643493, + 2.6940420066239312, + 2.9188340122345835, + 2.8322910075075924, + 2.8156670014141127, + 2.800791000481695, + 2.853375000995584, + 2.956333992187865, + 3.016832997673191, + 2.9658749990630895, + 3.0470000056084245, + 3.0054160015424713, + 3.1220410019159317, + 3.252208000048995, + 3.241584010538645, + 3.106749994913116, + 3.198916994733736, + 3.243000013753772, + 3.338125010486692, + 3.418082997086458, + 3.255791001720354, + 3.25204199180007, + 3.2976670045172796, + 3.506875000311993 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_default": { + "first_token_ms": { + "count": 5, + "median": 1.8891249928856269, + "p95": 1.9816997955786064, + "min": 1.7685839993646368, + "max": 1.9942079961765558, + "iqr": 0.06720799137838185, + "mean": 1.8896085966844112, + "cv": 0.04426869753066074 + }, + "decode_ms_per_step": { + "count": 315, + "median": 2.788042009342462, + "p95": 3.7487917987164114, + "min": 1.638874993659556, + "max": 6.158207994303666, + "iqr": 1.1669165032799356, + "mean": 2.795885701567703, + "cv": 0.24079141337232468 + }, + "total_ms": { + "count": 5, + "median": 174.3656670441851, + "p95": 190.49932359193917, + "min": 172.3801260086475, + "max": 194.27454099059105, + "iqr": 1.6652030608383939, + "mean": 178.03040779544972, + "cv": 0.0513750385900434 + }, + "tokens_per_second": { + "count": 5, + "median": 2936.357877553135, + "p95": 2965.553323181221, + "min": 2635.4456810931124, + "max": 2970.1799845204628, + "iqr": 27.978759427498517, + "mean": 2881.6196278775437, + "cv": 0.04818677230307029 + }, + "median_step_ms_by_index": [ + 1.8891249928856269, + 1.964582988875918, + 1.957665997906588, + 1.969874996575527, + 2.0952080085407943, + 2.0906660065520555, + 2.1914579992881045, + 2.1062090090708807, + 1.7867920105345547, + 1.799834004486911, + 1.762499989126809, + 1.815958006773144, + 1.9773329986492172, + 1.988791991607286, + 2.0481249957811087, + 2.011791992117651, + 2.1112920076120645, + 2.1603750064969063, + 2.1303329995134845, + 2.1113340044394135, + 2.1891659998800606, + 2.2235419892240316, + 2.2836659918539226, + 2.310459007276222, + 2.3447499988833442, + 2.4619159958092496, + 2.480082999682054, + 2.5354579993290827, + 2.4987079959828407, + 2.6267909997841343, + 2.5161660014418885, + 2.4881659919628873, + 2.7841669943882152, + 2.8512080025393516, + 2.789375008433126, + 2.78920799610205, + 2.834582992363721, + 2.870374999474734, + 2.9308329976629466, + 3.0264999950304627, + 3.1108330003917217, + 3.1062500056577846, + 3.1357499974546954, + 3.141167006106116, + 3.2415000023320317, + 3.2409170089522377, + 3.3618750021560118, + 3.21295800677035, + 3.276083996752277, + 3.34399999701418, + 3.506499997456558, + 3.4412920067552477, + 3.425333008635789, + 3.451625001616776, + 3.6190419923514128, + 3.6814170016441494, + 3.7331659987103194, + 3.5754580021603033, + 3.6339159996714443, + 3.763957996852696, + 3.8345000066328794, + 3.71558299229946, + 3.6991669912822545, + 3.7844580074306577 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "speedup_cached_vs_uncached_sdpa": 1.228380566970958, + "speedup_cached_vs_uncached_default": 1.3705793190140376 + }, + { + "batch": 8, + "prompt_len": 16, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_default": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 2.1914999961154535, + "p95": 2.423000003909692, + "min": 1.9124579994240776, + "max": 2.4770000018179417, + "iqr": 0.2635420096339658, + "mean": 2.146283202455379, + "cv": 0.10698890627301266 + }, + "decode_ms_per_step": { + "count": 315, + "median": 2.119208002113737, + "p95": 2.633975088247098, + "min": 1.8825000006472692, + "max": 2.995083006680943, + "iqr": 0.22164499387145042, + "mean": 2.184719567663879, + "cv": 0.0952553539069284 + }, + "total_ms": { + "count": 5, + "median": 136.73149210808333, + "p95": 148.11485694663133, + "min": 133.22708387568127, + "max": 148.74524595506955, + "iqr": 10.972343938192353, + "mean": 139.78361596527975, + "cv": 0.04969523463530852 + }, + "tokens_per_second": { + "count": 5, + "median": 3744.565294403976, + "p95": 3835.104167045788, + "min": 3442.1268169784485, + "max": 3843.0624247376354, + "iqr": 286.6258181921262, + "mean": 3669.9341980969466, + "cv": 0.04887956473751463 + }, + "median_step_ms_by_index": [ + 2.1914999961154535, + 2.613624994410202, + 2.1688339911634102, + 2.4605409998912364, + 2.077083001495339, + 2.2587910061702132, + 2.440292009850964, + 2.2979589994065464, + 2.3577079991810024, + 2.066624991130084, + 2.216708002379164, + 2.3416249896399677, + 2.0470409945119172, + 2.3658330028411, + 2.0322920026956126, + 2.0626249897759408, + 2.1606249938486144, + 2.0356669992906973, + 2.1156250004423782, + 2.0326250087236986, + 1.969709002878517, + 1.9666249863803387, + 2.1458750125020742, + 1.9757080008275807, + 2.0787079993169755, + 2.1137919975444674, + 2.0903749973513186, + 2.146000013453886, + 2.06433299172204, + 2.0533750066533685, + 2.064375003101304, + 2.123333004419692, + 2.1096660057082772, + 2.0875000045634806, + 2.1419580007204786, + 2.0924160053255036, + 2.1319170045899227, + 2.115041992510669, + 2.1115419949637726, + 2.1072920062579215, + 2.119833996403031, + 2.088749999529682, + 2.090457986923866, + 2.108458007569425, + 2.0877909992123023, + 2.09704099688679, + 2.1372089977376163, + 2.1743749966844916, + 2.1415829978650436, + 2.0937499939464033, + 2.187207996030338, + 2.089292000164278, + 2.099374993122183, + 2.1052079973742366, + 2.15408400981687, + 2.099750010529533, + 2.0985419978387654, + 2.1777919901069254, + 2.1450839994940907, + 2.161875003366731, + 2.1861249988432974, + 2.171583997551352, + 2.1971670066704974, + 2.1975410054437816 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 1.9717499963007867, + "p95": 2.353941200999543, + "min": 1.927999997860752, + "max": 2.4264160019811243, + "iqr": 0.10300098801963031, + "mean": 2.070249800453894, + "cv": 0.09921509132080715 + }, + "decode_ms_per_step": { + "count": 315, + "median": 3.218833007849753, + "p95": 4.434095697070006, + "min": 1.9324169988976792, + "max": 5.2062079921597615, + "iqr": 1.1494169957586564, + "mean": 3.2675815051853925, + "cv": 0.22383603625921772 + }, + "total_ms": { + "count": 5, + "median": 205.74691497313324, + "p95": 218.87804806174245, + "min": 196.91296396194957, + "max": 220.50266306905542, + "iqr": 8.28229493345134, + "mean": 207.9278846271336, + "cv": 0.04292884014537767 + }, + "tokens_per_second": { + "count": 5, + "median": 2488.494177746761, + "p95": 2581.8283234438486, + "min": 2321.9674214984675, + "max": 2600.1335295472786, + "iqr": 97.82968020475073, + "mean": 2465.996089329603, + "cv": 0.042583035548729616 + }, + "median_step_ms_by_index": [ + 1.9717499963007867, + 2.132499997969717, + 2.128749998519197, + 2.1062079904368147, + 2.1964589977869764, + 2.300915992236696, + 2.413250011159107, + 2.4169999960577115, + 2.366209009778686, + 2.424666003207676, + 2.5028329982887954, + 2.461000010953285, + 2.51537500298582, + 2.3702920007053763, + 2.678083998034708, + 2.471125000738539, + 2.4774170015007257, + 2.7279159985482693, + 2.6575000083539635, + 2.650333000929095, + 2.8878329903818667, + 2.8759999986505136, + 2.842540998244658, + 2.7722499944502488, + 2.791749997413717, + 2.8630000015255064, + 3.0924580059945583, + 3.039041010197252, + 3.0990840023150668, + 3.0705410026712343, + 3.0670419946545735, + 3.2738750014686957, + 3.119583008810878, + 3.284000005805865, + 3.2511250028619543, + 3.236459000618197, + 3.2507909927517176, + 3.3580419985810295, + 3.405916999327019, + 3.57708299998194, + 3.448749994277023, + 3.4292920026928186, + 3.5254579997854307, + 3.5606250021373853, + 3.6407089937711135, + 3.6661250051110983, + 3.699624998262152, + 3.646666998974979, + 3.7230830057524145, + 4.007708994322456, + 3.943458999856375, + 4.109457993763499, + 3.9539999997941777, + 4.045249996124767, + 3.9863749989308417, + 4.042582993861288, + 4.247165998094715, + 4.315083002438769, + 4.178333008894697, + 4.219291993649676, + 4.443084006197751, + 4.540125009953044, + 4.289083008188754, + 4.339457998867147 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_default": { + "first_token_ms": { + "count": 5, + "median": 2.418165997369215, + "p95": 2.947500004665926, + "min": 2.0044999982928857, + "max": 3.0335000046761706, + "iqr": 0.2537920081522316, + "mean": 2.481874800287187, + "cv": 0.15189870927020166 + }, + "decode_ms_per_step": { + "count": 315, + "median": 3.5786669905064628, + "p95": 5.281945609021932, + "min": 1.9471669947961345, + "max": 5.947790996287949, + "iqr": 1.3248539980850182, + "mean": 3.6466317682338523, + "cv": 0.24722284323299723 + }, + "total_ms": { + "count": 5, + "median": 234.10149902338162, + "p95": 250.0154017820023, + "min": 212.7677900280105, + "max": 250.7877099851612, + "iqr": 30.410955980187282, + "mean": 232.2196761990199, + "cv": 0.07426282845888696 + }, + "tokens_per_second": { + "count": 5, + "median": 2187.0855254492085, + "p95": 2398.0494007148573, + "min": 2041.5673480582218, + "max": 2406.3792735385186, + "iqr": 291.235625128626, + "mean": 2214.651268151549, + "cv": 0.07485412280208131 + }, + "median_step_ms_by_index": [ + 2.418165997369215, + 2.4079169961623847, + 2.4512919917469844, + 2.4566250067437068, + 2.2539589990628883, + 2.3778750037308782, + 2.6217080012429506, + 2.6018750068033114, + 2.5289580080425367, + 2.5404170010006055, + 2.6181669963989407, + 2.5095839955611154, + 2.9633329977514222, + 2.9247920028865337, + 2.729250001721084, + 2.7382919943192974, + 2.7201659977436066, + 2.9057919891783968, + 3.27220800681971, + 3.2163750001927838, + 2.9516669892473146, + 3.022959004738368, + 3.051583989872597, + 3.257167001720518, + 3.345125005580485, + 3.178500002832152, + 3.228790999855846, + 3.3689589909045026, + 3.6015829973621294, + 3.738291998161003, + 3.4135410096496344, + 3.5510000016074628, + 3.4003749897237867, + 3.6660839978139848, + 3.549707995261997, + 3.6505419993773103, + 3.6276659957366064, + 3.826667001703754, + 3.891207990818657, + 3.780583996558562, + 3.860374999931082, + 4.058124992297962, + 3.898166003637016, + 4.064458000357263, + 4.2104580061277375, + 3.9554160030093044, + 4.009166994364932, + 4.101084006833844, + 4.201999996439554, + 4.259416004060768, + 4.749208004795946, + 4.6245420089690015, + 4.324916997575201, + 4.516791988862678, + 4.507749996264465, + 4.565667011775076, + 4.494749999139458, + 4.681958991568536, + 4.9485000054119155, + 5.174707999685779, + 5.118125001899898, + 5.184749999898486, + 5.047375001595356, + 5.74816700827796 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "speedup_cached_vs_uncached_sdpa": 1.504751478982579, + "speedup_cached_vs_uncached_default": 1.7121256808806662 + }, + { + "batch": 8, + "prompt_len": 128, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_default": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 9.00129199726507, + "p95": 11.672125005861744, + "min": 6.836583997937851, + "max": 11.945375008508563, + "iqr": 3.172666984028183, + "mean": 9.153766802046448, + "cv": 0.23347701346880687 + }, + "decode_ms_per_step": { + "count": 315, + "median": 3.750041010789573, + "p95": 4.165737390576396, + "min": 2.632625008118339, + "max": 5.917792004765943, + "iqr": 0.7494590026908554, + "mean": 3.6097799969692934, + "cv": 0.1374292985167658 + }, + "total_ms": { + "count": 5, + "median": 255.42049600335304, + "p95": 257.1864952275064, + "min": 198.55362096859608, + "max": 257.27908802218735, + "iqr": 42.03592003614176, + "mean": 236.56990661111195, + "cv": 0.11794526301309922 + }, + "tokens_per_second": { + "count": 5, + "median": 2004.5376467880585, + "p95": 2539.68528399938, + "min": 1990.0568053780023, + "max": 2578.6485157124366, + "iqr": 390.1880643817001, + "mean": 2190.143923558221, + "cv": 0.125364722293049 + }, + "median_step_ms_by_index": [ + 9.00129199726507, + 4.332666998379864, + 3.705499999341555, + 3.8187080062925816, + 3.528207991621457, + 3.763792003155686, + 3.840542005491443, + 3.8297080027405173, + 3.6677090101875365, + 3.9001660043140873, + 3.844708000542596, + 3.692541999043897, + 3.68512500426732, + 3.65691700426396, + 3.8314580015139654, + 3.7053749983897433, + 3.8419999909820035, + 3.537124997819774, + 3.405959010706283, + 3.899666990037076, + 3.902583004673943, + 3.8115419883979484, + 3.743249995750375, + 3.845541999908164, + 3.797334007686004, + 3.891416999977082, + 3.856582989101298, + 3.8956670032348484, + 3.972666003392078, + 3.8331669929903, + 3.767666989006102, + 3.655459004221484, + 3.696915999171324, + 3.931375002139248, + 3.8112500042188913, + 3.752000004169531, + 3.555832998245023, + 3.9558749995194376, + 3.687250005896203, + 3.781666004215367, + 3.724791997228749, + 3.5142499982612208, + 3.825040999799967, + 3.8749590021325275, + 3.9093749946914613, + 3.6385420098667964, + 3.903958000591956, + 3.9398329972755164, + 3.7859999865759164, + 3.609749997849576, + 3.770415991311893, + 3.531375012244098, + 3.6780420050490648, + 3.582250006729737, + 3.7211660091998056, + 3.6022910062456504, + 3.5630410129670054, + 3.444417001446709, + 3.3566670026630163, + 3.8044160028221086, + 3.9291660068556666, + 3.622292002546601, + 3.634082997450605, + 3.6951249931007624 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 6.6650829976424575, + "p95": 10.061716605559923, + "min": 6.093000003602356, + "max": 10.372250006184913, + "iqr": 2.5004170020110905, + "mean": 7.6538164023077115, + "cv": 0.24394083346637144 + }, + "decode_ms_per_step": { + "count": 315, + "median": 8.755375005421229, + "p95": 17.40387870377163, + "min": 6.380208011250943, + "max": 18.48529100243468, + "iqr": 2.5017084990395233, + "mean": 9.700808733783012, + "cv": 0.3120446020138121 + }, + "total_ms": { + "count": 5, + "median": 542.7477470657323, + "p95": 880.3305164881749, + "min": 480.0325819669524, + "max": 944.4166251050774, + "iqr": 121.14528502570465, + "mean": 618.8047666306375, + "cv": 0.3071922954997203 + }, + "tokens_per_second": { + "count": 5, + "median": 943.3479968696979, + "p95": 1056.918397018373, + "min": 542.1336160225197, + "max": 1066.5942672100712, + "iqr": 197.68379423991132, + "mean": 878.164383673108, + "cv": 0.23859716399742373 + }, + "median_step_ms_by_index": [ + 6.6650829976424575, + 6.865957999252714, + 7.031958011793904, + 6.721666999510489, + 7.077667003613897, + 7.719542001723312, + 7.496916994568892, + 7.954625005368143, + 7.050500003970228, + 7.611749999341555, + 7.252584007801488, + 7.644750003237277, + 7.404042000416666, + 7.885125000029802, + 7.469749994925223, + 7.787334005115554, + 7.694624990108423, + 7.73062500229571, + 7.548416993813589, + 7.537291996413842, + 7.627917002537288, + 7.59437499800697, + 7.7562079968629405, + 7.666332996450365, + 7.767499992041849, + 8.146874999511056, + 7.919167008367367, + 8.49195801129099, + 8.482250006636605, + 8.769583000685088, + 8.744291990296915, + 9.024375001899898, + 9.442000009585172, + 9.466500006965362, + 9.22020799771417, + 9.259124999516644, + 9.846957997069694, + 9.5917919970816, + 9.747000003699213, + 9.963499993318692, + 10.058667001430877, + 10.231499996734783, + 9.694417007267475, + 9.82208299683407, + 9.87258400709834, + 9.530041003017686, + 10.206124992691912, + 9.474250007770024, + 9.419041991350241, + 9.293124996474944, + 8.822166011668742, + 9.019416000228375, + 8.795916000963189, + 9.515374986222014, + 8.77474999288097, + 9.222000007866882, + 8.755375005421229, + 9.241042003850453, + 8.716374999494292, + 9.5050420059124, + 9.19212499866262, + 9.461916008149274, + 9.275667005567811, + 9.383625001646578 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_default": { + "first_token_ms": { + "count": 5, + "median": 7.423750008456409, + "p95": 10.832658398430794, + "min": 6.880667002405971, + "max": 11.49312499910593, + "iqr": 1.3005840010009706, + "mean": 8.175708400085568, + "cv": 0.23607281800530266 + }, + "decode_ms_per_step": { + "count": 315, + "median": 9.94895800249651, + "p95": 18.095387490757275, + "min": 7.3926670011132956, + "max": 19.64991699787788, + "iqr": 4.413875001773704, + "mean": 11.029432682802767, + "cv": 0.28776335792292657 + }, + "total_ms": { + "count": 5, + "median": 677.8125830605859, + "p95": 952.0915303844959, + "min": 535.0863300263882, + "max": 1012.8533379756846, + "iqr": 128.6910140188411, + "mean": 703.0299674166599, + "cv": 0.26603071784337684 + }, + "tokens_per_second": { + "count": 5, + "median": 755.3710462088533, + "p95": 941.9282041844508, + "min": 505.5026041808745, + "max": 956.854943341106, + "iqr": 160.1225014206069, + "mean": 764.4097174851773, + "cv": 0.22641625994895387 + }, + "median_step_ms_by_index": [ + 7.423750008456409, + 7.74670799728483, + 7.828458998119459, + 8.122917002765462, + 7.9558340075891465, + 8.06033301341813, + 8.211874999688007, + 8.383833992411382, + 8.223500000895001, + 8.322749999933876, + 8.187957995687611, + 8.381916995858774, + 8.011625002836809, + 8.197542003472336, + 8.251542007201351, + 8.193209010642022, + 8.948750008130446, + 8.557709006709047, + 9.09608299843967, + 8.89045899384655, + 9.562540988554247, + 9.139666988630779, + 9.929249994456768, + 9.238958009518683, + 9.94895800249651, + 9.305624989792705, + 9.937916998751462, + 10.353000005125068, + 9.534915996482596, + 10.161291997064836, + 9.74283400864806, + 10.320000001229346, + 10.200749995419756, + 10.337375002563931, + 10.672624994185753, + 10.997542005497962, + 10.902167006861418, + 10.954583995044231, + 11.392083004466258, + 10.848542005987838, + 11.30191600532271, + 11.809624993475154, + 11.31741599238012, + 11.811916992883198, + 12.103166998713277, + 11.724790994776413, + 11.638333002338186, + 12.141582992626354, + 12.36545800929889, + 11.920749995624647, + 12.44737500383053, + 12.586084005306475, + 12.37595800193958, + 12.584667012561113, + 12.277208006707951, + 12.796625000191852, + 13.40324999182485, + 13.768667005933821, + 13.028166009462439, + 14.186957996571437, + 14.814291993388906, + 14.867584002786316, + 14.912458005710505, + 14.876249988446943 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "speedup_cached_vs_uncached_sdpa": 2.124918538481764, + "speedup_cached_vs_uncached_default": 2.653712578538286 + }, + { + "batch": 8, + "prompt_len": 512, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_default": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 32.145749995834194, + "p95": 32.70317520073149, + "min": 30.694957997184247, + "max": 32.718292000936344, + "iqr": 1.1450409947428852, + "mean": 31.939874999807216, + "cv": 0.02660171718316953 + }, + "decode_ms_per_step": { + "count": 315, + "median": 4.502249998040497, + "p95": 6.884616804018152, + "min": 3.2501660025445744, + "max": 9.498290994088165, + "iqr": 0.49393699737265706, + "mean": 4.790965834861651, + "cv": 0.18525988765030849 + }, + "total_ms": { + "count": 5, + "median": 338.63070797815453, + "p95": 355.9827783261426, + "min": 310.83621106517967, + "max": 360.0074539135676, + "iqr": 20.38891192933079, + "mean": 333.7707225960912, + "cv": 0.057613823940512676 + }, + "tokens_per_second": { + "count": 5, + "median": 1511.971560573974, + "p95": 1638.2414231687703, + "min": 1422.1927752721574, + "max": 1647.1697369024937, + "iqr": 96.13220505410482, + "mean": 1538.0516408324547, + "cv": 0.05739368099267047 + }, + "median_step_ms_by_index": [ + 32.145749995834194, + 4.919458006042987, + 4.496583002037369, + 4.2118749988731, + 4.177499999059364, + 4.491792002227157, + 4.341250009019859, + 4.825583004276268, + 4.213083011563867, + 4.577584011713043, + 4.401208003400825, + 4.516500004683621, + 4.306792005081661, + 4.390000001876615, + 4.4707499910146, + 4.521041992120445, + 4.512291008722968, + 4.449125000974163, + 4.878667008597404, + 4.484082994167693, + 4.656207995139994, + 4.578332998789847, + 4.462166005396284, + 4.489208004088141, + 4.299833002733067, + 4.40458400407806, + 4.348041009507142, + 4.716374998679385, + 4.391541006043553, + 5.0029580015689135, + 4.659291007556021, + 4.6925839997129515, + 4.7079579962883145, + 4.770917003042996, + 4.5521659922087565, + 4.566375006106682, + 4.804750002222136, + 4.584916008752771, + 5.057750007836148, + 4.691374997491948, + 4.338082988397218, + 4.57562500378117, + 4.419292003149167, + 4.545750009128824, + 4.494790991884656, + 4.539540997939184, + 4.4570830068551, + 4.714833004982211, + 4.5293329894775525, + 4.700707999290898, + 4.453042012755759, + 4.330958006903529, + 4.457832998014055, + 4.507292003836483, + 4.34812500316184, + 4.41595799929928, + 4.376291995868087, + 4.3912499968428165, + 4.2821669921977445, + 4.472749991691671, + 4.339000006439164, + 4.359083002782427, + 4.549791992758401, + 4.524542004219256 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 28.857083001639694, + "p95": 30.697341609629802, + "min": 25.629541007219814, + "max": 31.012500010547228, + "iqr": 1.688708012807183, + "mean": 28.53676640370395, + "cv": 0.07033833343798207 + }, + "decode_ms_per_step": { + "count": 315, + "median": 33.22533301252406, + "p95": 51.52277860033793, + "min": 25.547084005665965, + "max": 74.8668330052169, + "iqr": 3.949125501094386, + "mean": 34.53571242927056, + "cv": 0.22686265601270075 + }, + "total_ms": { + "count": 5, + "median": 2115.2752870402765, + "p95": 2501.8981434230227, + "min": 2042.695290161646, + "max": 2574.275918028434, + "iqr": 135.58733799436595, + "mean": 2204.2866494477494, + "cv": 0.098153885514053 + }, + "tokens_per_second": { + "count": 5, + "median": 242.04887332485114, + "p95": 249.82601590589542, + "min": 198.8908789513622, + "max": 250.64922921493766, + "iqr": 15.108918364552864, + "mean": 233.90927769321019, + "cv": 0.0891221304241894 + }, + "median_step_ms_by_index": [ + 28.857083001639694, + 29.167625005356967, + 28.634542002691887, + 29.012875005719252, + 30.07429200806655, + 31.17258301062975, + 30.594249998102896, + 30.57612499105744, + 30.506541996146552, + 31.158249999862164, + 30.94649998820387, + 29.630416000145487, + 31.144082997343503, + 30.99816699977964, + 30.800500011537224, + 31.87133400933817, + 32.87862500292249, + 29.69658399524633, + 30.59574999497272, + 31.289625010685995, + 31.519208001554944, + 31.23670800414402, + 31.266916004824452, + 31.54158299730625, + 31.605208991095424, + 31.15000000980217, + 32.03154199582059, + 31.329875011579134, + 31.958583000232466, + 31.976083002518862, + 32.070417000795715, + 32.92729100212455, + 33.145042005344294, + 33.4144579974236, + 33.36250000575092, + 33.18720799870789, + 33.59845899103675, + 34.116416005417705, + 34.00341600354295, + 34.79545899608638, + 34.75312500086147, + 34.60104200348724, + 34.81195800122805, + 34.51875000610016, + 35.171750001609325, + 35.39795799588319, + 34.57770800741855, + 35.89549999742303, + 35.6076669995673, + 35.59637500438839, + 35.055917003774084, + 35.22599999269005, + 34.51541699178051, + 34.63454199663829, + 34.552290991996415, + 35.781666010734625, + 36.57312500581611, + 35.87274999881629, + 35.563333993195556, + 36.11116700631101, + 36.724916004459374, + 35.86145800363738, + 36.434791996725835, + 36.001541011501104 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_default": { + "first_token_ms": { + "count": 5, + "median": 36.254165999707766, + "p95": 55.44660000014119, + "min": 32.18133299378678, + "max": 58.18662499950733, + "iqr": 12.139500002376735, + "mean": 40.69112479919568, + "cv": 0.26983306142055835 + }, + "decode_ms_per_step": { + "count": 315, + "median": 41.17045900784433, + "p95": 49.05705829878571, + "min": 32.41408400936052, + "max": 60.215416000573896, + "iqr": 3.7070204998599365, + "mean": 41.23868875257257, + "cv": 0.10455824783278786 + }, + "total_ms": { + "count": 5, + "median": 2672.5652069872012, + "p95": 2740.3279367863433, + "min": 2488.216834011837, + "max": 2741.029170982074, + "iqr": 183.21463093161583, + "mean": 2638.7285162112676, + "cv": 0.04285517830636678 + }, + "tokens_per_second": { + "count": 5, + "median": 191.5762424285919, + "p95": 204.70500838336008, + "min": 186.79115327202348, + "max": 205.7698481102569, + "iqr": 13.415257402588338, + "mean": 194.32265707196592, + "cv": 0.04351197985287354 + }, + "median_step_ms_by_index": [ + 36.254165999707766, + 35.28904200356919, + 37.01841599831823, + 36.329708993434906, + 37.74200000043493, + 37.19354099303018, + 35.5719159997534, + 36.90150000329595, + 36.982874997192994, + 36.80283400171902, + 37.4710410105763, + 37.289375002728775, + 38.624749999144115, + 37.84479100431781, + 36.55191700090654, + 37.23862499464303, + 36.80887499649543, + 36.62149999581743, + 38.73970900895074, + 39.275375005672686, + 40.15833299490623, + 39.44404200592544, + 39.755000005243346, + 41.55908399843611, + 39.99645799922291, + 40.85495900653768, + 39.907416998175904, + 40.31587499775924, + 41.71266699268017, + 41.26616701250896, + 39.71608300344087, + 39.67674999148585, + 39.71866698702797, + 41.12945900124032, + 40.46354100864846, + 40.413166003418155, + 43.05941698839888, + 41.212666998035274, + 41.40266698959749, + 43.46983300638385, + 41.10037500504404, + 41.68783299974166, + 42.42737499589566, + 42.019208005513065, + 41.24974999285769, + 41.78095899987966, + 41.82479198789224, + 42.025374990771525, + 41.17045900784433, + 41.63520901056472, + 42.64329100260511, + 41.67683300329372, + 41.71541699906811, + 42.15445800218731, + 41.43024999939371, + 40.747999999439344, + 41.589999993448146, + 42.34908400394488, + 43.01925000618212, + 44.29658300068695, + 42.51420799118932, + 43.93454099772498, + 44.32575000100769, + 41.80212500796188 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "speedup_cached_vs_uncached_sdpa": 6.246554837480171, + "speedup_cached_vs_uncached_default": 7.892270677234658 + }, + { + "batch": 8, + "prompt_len": 1024, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_default": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 72.37187500868458, + "p95": 98.1412250053836, + "min": 65.91137500072364, + "max": 104.09337500459515, + "iqr": 3.045958001166582, + "mean": 77.59918340598233, + "cv": 0.1950531828339179 + }, + "decode_ms_per_step": { + "count": 315, + "median": 6.380250008078292, + "p95": 9.275171195622534, + "min": 4.5867089938838035, + "max": 11.451500002294779, + "iqr": 1.6718330007279292, + "mean": 6.66661812999611, + "cv": 0.20159652828510186 + }, + "total_ms": { + "count": 5, + "median": 475.1056660315953, + "p95": 570.8882170030847, + "min": 466.6039999719942, + "max": 593.8375830155564, + "iqr": 5.748126946855336, + "mean": 497.59612559573725, + "cv": 0.10850049199992205 + }, + "tokens_per_second": { + "count": 5, + "median": 1077.6550073093658, + "p95": 1094.165947102424, + "min": 862.1886095521634, + "max": 1097.2902076080156, + "iqr": 12.977854701932756, + "mean": 1037.4987559855458, + "cv": 0.09498408127576376 + }, + "median_step_ms_by_index": [ + 72.37187500868458, + 10.224625002592802, + 8.359749990631826, + 8.371082993107848, + 8.007917000213638, + 7.972666004206985, + 7.758624997222796, + 7.499832994653843, + 8.071417003520764, + 8.229625003878027, + 8.338708998053335, + 7.317541996599175, + 6.323582987533882, + 6.918707993463613, + 6.314250000286847, + 7.122167007764801, + 6.73358300991822, + 6.446665996918455, + 6.200417003128678, + 6.098166995798238, + 6.787583988625556, + 6.60741698811762, + 6.192249988089316, + 6.130041001597419, + 6.296166990068741, + 6.401000006007962, + 6.800165996537544, + 6.644541994319297, + 6.4971669926308095, + 6.506166988401674, + 6.708999993861653, + 6.50441600009799, + 6.228708996786736, + 6.599250002182089, + 6.380250008078292, + 6.700708996504545, + 6.186917002196424, + 6.017666994011961, + 6.066667003324255, + 5.985833995509893, + 5.9853749989997596, + 6.117707991506904, + 6.041915999958292, + 6.151875000796281, + 5.876582989003509, + 6.419375000405125, + 6.135917006758973, + 5.972957995254546, + 6.323541994788684, + 5.900208998355083, + 6.337916012853384, + 5.956624998361804, + 5.927624995820224, + 5.931457999395207, + 6.3887080032145604, + 5.7793339947238564, + 5.898250004975125, + 5.683041992597282, + 5.544583997107111, + 6.05116700171493, + 5.881958000827581, + 5.917457994655706, + 6.236083005205728, + 5.499999999301508 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 61.56120900413953, + "p95": 67.33006679278333, + "min": 51.60516699834261, + "max": 67.9437499929918, + "iqr": 12.429458991391584, + "mean": 59.68626699759625, + "cv": 0.12321947637871572 + }, + "decode_ms_per_step": { + "count": 315, + "median": 70.44308401236776, + "p95": 80.93112880305853, + "min": 53.13062499044463, + "max": 91.07916698849294, + "iqr": 4.254770508850925, + "mean": 70.92825937803671, + "cv": 0.07579972981561092 + }, + "total_ms": { + "count": 5, + "median": 4506.922206986928, + "p95": 4697.587945047417, + "min": 4413.03241206333, + "max": 4743.241213058354, + "iqr": 52.31253904639743, + "mean": 4528.16660781391, + "cv": 0.02802778154484169 + }, + "tokens_per_second": { + "count": 5, + "median": 113.60302585348906, + "p95": 115.76194029191545, + "min": 107.94306614439957, + "max": 116.01999536654489, + "iqr": 1.329310378408394, + "mean": 113.13924339456409, + "cv": 0.027282683086201298 + }, + "median_step_ms_by_index": [ + 61.56120900413953, + 63.7118340091547, + 64.27933300437871, + 64.95349999750033, + 66.69962500745896, + 67.4548329989193, + 68.60933400457725, + 68.53279100323562, + 68.73520799854305, + 68.42229100584518, + 68.54525000380818, + 71.55350000539329, + 68.66387500485871, + 69.45429199549835, + 71.05204099207185, + 69.41275000281166, + 70.60879201162606, + 69.89966599212494, + 69.08737499907147, + 71.53299999481533, + 70.71233401075006, + 70.82366599934176, + 69.07349999528378, + 71.37079199310392, + 70.2387089986587, + 69.88420800189488, + 69.26758300687652, + 69.40516599570401, + 68.53770800807979, + 71.4951249974547, + 70.88108400057536, + 69.85841599816922, + 69.45083299069665, + 69.81199998699594, + 71.86958300007973, + 71.124541995232, + 70.45066700084135, + 71.74270799441729, + 71.18675000674557, + 70.5104999942705, + 70.73370899888687, + 70.50262500706594, + 70.69929198769387, + 71.40966699807905, + 70.48129200120457, + 71.06804099748842, + 71.13591600500513, + 70.70324999222066, + 71.29362499108538, + 70.06904200534336, + 70.44308401236776, + 70.66654101072345, + 71.5155420039082, + 74.68854098988231, + 74.63620800990611, + 72.62408400129061, + 69.86704100563657, + 70.79199999861885, + 70.40625000081491, + 71.28933300555218, + 69.80175001081079, + 70.80691699229646, + 71.32016599643975, + 72.42691601277329 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_default": { + "first_token_ms": { + "count": 5, + "median": 83.40404200134799, + "p95": 88.28755840077065, + "min": 71.94779100245796, + "max": 89.0030000009574, + "iqr": 4.85562501125969, + "mean": 82.07015839871019, + "cv": 0.07845092979793131 + }, + "decode_ms_per_step": { + "count": 315, + "median": 88.84416600631084, + "p95": 100.22059180628275, + "min": 77.19724999333266, + "max": 143.77216598950326, + "iqr": 5.502875006641261, + "mean": 90.48571865733099, + "cv": 0.08287047597859783 + }, + "total_ms": { + "count": 5, + "median": 5765.051169015351, + "p95": 5911.177876594593, + "min": 5673.952582059428, + "max": 5936.199502000818, + "iqr": 84.03383396216668, + "mean": 5782.670433810563, + "cv": 0.017206741433421317 + }, + "tokens_per_second": { + "count": 5, + "median": 88.81100704739238, + "p95": 90.06957213421795, + "min": 86.25047049504123, + "max": 90.23691907805186, + "iqr": 1.2928105520695823, + "mean": 88.56119095723611, + "cv": 0.017049761321109087 + }, + "median_step_ms_by_index": [ + 83.40404200134799, + 84.66187500744127, + 86.42300000065006, + 86.60029199381825, + 85.64829199167434, + 86.02079200500157, + 88.17229099804536, + 92.52454199304339, + 89.33516699471511, + 92.28045899362769, + 92.24499999254476, + 87.87633299652953, + 89.76837500813417, + 94.74979199876543, + 88.56529201148078, + 88.81070800998714, + 91.03420899191406, + 90.31050000339746, + 92.5533749978058, + 91.76054199633654, + 88.78504099266138, + 89.43358300894033, + 90.44587500102352, + 87.13949999946635, + 88.1586660107132, + 87.8817080083536, + 88.51691699237563, + 88.49062499939464, + 85.38270799908787, + 85.31095800572075, + 84.88020900404081, + 87.17458300816361, + 86.47708401258569, + 87.06475001235958, + 87.51116700295825, + 87.88512500177603, + 86.65250000194646, + 89.35183299763594, + 90.78645799309015, + 88.87404199049342, + 88.70166599808726, + 88.70375000697095, + 88.97279200027697, + 88.58158299699426, + 88.6101670039352, + 90.30079199874308, + 89.59912499994971, + 89.32249998906627, + 89.89762498822529, + 88.62283299094997, + 87.30183400621172, + 87.63283300504554, + 88.78904100856744, + 90.61625000322238, + 89.56849999958649, + 89.96416701120324, + 86.87045800616033, + 90.83850000752136, + 89.75787500094157, + 92.49279199866578, + 103.30666700610891, + 96.14870800578501, + 91.6750000033062, + 93.32083300978411 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "speedup_cached_vs_uncached_sdpa": 9.48614703889305, + "speedup_cached_vs_uncached_default": 12.13425050719552 + } + ], + "window": { + "batch": 1, + "prompt_len": 32, + "context_length": 128, + "new_tokens": 224, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_default": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 1.6048329998739064, + "p95": 2.1824580064276233, + "min": 1.4668329968117177, + "max": 2.313083008630201, + "iqr": 0.08949899347499013, + "mean": 1.7230332014150918, + "cv": 0.19574670148617462 + }, + "decode_ms_per_step": { + "count": 1115, + "median": 1.8058749992633238, + "p95": 2.050528705876786, + "min": 1.5551250107819214, + "max": 5.059500006609596, + "iqr": 0.13058350305072963, + "mean": 1.8336505363925022, + "cv": 0.1231572816174616 + }, + "total_ms": { + "count": 5, + "median": 399.18033515277784, + "p95": 431.69391514966264, + "min": 396.357579054893, + "max": 432.853798934957, + "iqr": 29.36495907488279, + "mean": 410.6271028169431, + "cv": 0.0433235763672045 + }, + "tokens_per_second": { + "count": 5, + "median": 561.1498870911784, + "p95": 564.7677220803469, + "min": 517.4957469500215, + "max": 565.146251357483, + "iqr": 38.73024100220812, + "mean": 546.3137708680158, + "cv": 0.042604984984523614 + }, + "median_step_ms_by_index": [ + 1.6048329998739064, + 2.6793750002980232, + 2.459542010910809, + 1.8929999932879582, + 1.8509170040488243, + 1.8952090031234547, + 1.8406669987598434, + 1.6936669999267906, + 1.7608330090297386, + 1.7467909929109737, + 1.7642500024521723, + 1.8162909982493147, + 1.668416996835731, + 1.7800420027924702, + 1.7585410096216947, + 1.597374997800216, + 1.7779999907361344, + 1.8122080073226243, + 1.7124160076491535, + 1.8051660008495674, + 1.7150420026155189, + 1.7687910003587604, + 1.8217089964309707, + 1.7947919986909255, + 1.8552090041339397, + 1.7513750062789768, + 1.727249997202307, + 1.7674580012680963, + 1.8379579996690154, + 1.684582995949313, + 1.81108400283847, + 1.8390829936834052, + 1.7828339914558455, + 1.7631660011829808, + 1.785500004189089, + 1.7267500079469755, + 1.808000000892207, + 1.8255840113852173, + 1.6596659988863394, + 1.8009580089710653, + 1.8414999940432608, + 1.7919170059030876, + 1.7127080063801259, + 1.6607500001555309, + 1.8479169957572594, + 1.8398329993942752, + 1.807417007512413, + 1.7241660098079592, + 1.771291994373314, + 1.7625829932512715, + 1.8502080056350678, + 1.7282500048168004, + 1.8111670069629326, + 1.779166006599553, + 1.7966250015888363, + 1.7762909992597997, + 1.6409170057158917, + 1.7945420113392174, + 1.8288749997736886, + 1.8629169935593382, + 1.712041994323954, + 1.7498749948572367, + 1.7135829984908924, + 1.7350419948343188, + 1.8862920114770532, + 1.8248340056743473, + 1.799582998501137, + 1.775249998900108, + 1.6332920058630407, + 1.860791991930455, + 1.7818330088630319, + 1.7826249968493357, + 1.8310000014025718, + 1.7980420088861138, + 1.8173339922213927, + 1.7920420068548992, + 1.786875000107102, + 1.7512920021545142, + 1.85087499266956, + 1.828749998821877, + 1.778416000888683, + 1.7907500005094334, + 1.7787920078262687, + 1.760750004905276, + 1.8128329975297675, + 1.7878329963423312, + 1.7678750009508803, + 1.820208999561146, + 1.731208001729101, + 1.7823329981183633, + 1.8030000064754859, + 1.7806249961722642, + 1.7470830061938614, + 1.7317919991910458, + 1.7741659976309165, + 1.7755409935489297, + 1.738874998409301, + 1.8810000037774444, + 1.8489999929443002, + 1.8877919937949628, + 1.8912500090664253, + 1.7737909947754815, + 1.7490830068709329, + 1.8233750015497208, + 1.8758749938569963, + 1.807624998036772, + 1.8011659994954243, + 1.8014590023085475, + 1.8261670047650114, + 1.8127090006601065, + 1.7960420082090423, + 1.7316659941570833, + 1.7950420005945489, + 1.8230840069008991, + 1.7534999933559448, + 1.715999998850748, + 1.8501669983379543, + 1.8462500011082739, + 1.7547090101288632, + 1.9007920054718852, + 1.81387500197161, + 1.7884580011013895, + 1.8240839999634773, + 1.8191249982919544, + 1.8094169936375692, + 1.8879170092986897, + 1.780375008820556, + 1.8523750040913, + 1.8052079976769164, + 1.7837089981185272, + 1.787999994121492, + 1.7903330008266494, + 1.7772909923223779, + 1.7799170018406585, + 1.8358329980401322, + 1.7986250022659078, + 1.8644999945536256, + 1.793917006580159, + 1.7946250009117648, + 1.7804590024752542, + 1.8129580130334944, + 1.792957991710864, + 1.7589159979252145, + 1.7823750094976276, + 1.838333992054686, + 1.8433340010233223, + 1.8081250018440187, + 1.846499988459982, + 1.8283750105183572, + 1.8468750058673322, + 1.782457999070175, + 1.6821669996716082, + 1.8542079924372956, + 1.8582499906187877, + 1.8120829918188974, + 1.836083989473991, + 1.869707994046621, + 1.8255829927511513, + 1.8239589990116656, + 1.8142909975722432, + 1.7951670015463606, + 1.755792007315904, + 1.8112910038325936, + 1.823625003453344, + 1.7777920002117753, + 1.8310419982299209, + 1.8758329970296472, + 1.7909169982885942, + 1.8662090005818754, + 1.7914170020958409, + 1.7631670052651316, + 1.7586250032763928, + 1.8048749916488305, + 1.761291001457721, + 1.8764999986160547, + 1.8823330028681085, + 1.8242910009576008, + 1.8098750006174669, + 1.8031249928753823, + 1.8097499996656552, + 1.8209999980172142, + 1.826874999096617, + 1.8699590000323951, + 1.8664170056581497, + 1.7940000107046217, + 1.8103329930454493, + 1.808000000892207, + 1.7173750093206763, + 1.866833001258783, + 1.8260419892612845, + 1.887207996333018, + 1.8513339891796932, + 1.860959004261531, + 1.8687080009840429, + 1.8082919996231794, + 1.8122500041499734, + 1.8163339991588145, + 1.7900409875437617, + 1.8056249973597005, + 1.7798750050133094, + 1.8244999955641106, + 1.8673330050660297, + 1.861500000813976, + 1.8303750111954287, + 1.7269590025534853, + 1.7359170014970005, + 1.8242920050397515, + 1.9010830001207069, + 1.807624998036772, + 1.784749998478219, + 1.836374998674728, + 1.8696250044740736, + 1.878666997072287, + 1.8125420028809458, + 1.8392909987596795, + 1.8478330021025613, + 1.8523750040913, + 1.8460830033291131, + 1.7825830000219867, + 1.8084999901475385, + 1.8330829916521907, + 1.8273339956067502, + 1.7302499909419566, + 1.7952500056708232 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 1.5097090072231367, + "p95": 1.6195003991015255, + "min": 1.4709580136695877, + "max": 1.6342919989256188, + "iqr": 0.08437600627075881, + "mean": 1.5302502026315778, + "cv": 0.04457975347561168 + }, + "decode_ms_per_step": { + "count": 1115, + "median": 1.8014579982263967, + "p95": 2.0161125052254647, + "min": 1.3249170006019995, + "max": 3.620749994297512, + "iqr": 0.18856200040318072, + "mean": 1.7981575588061947, + "cv": 0.09873406986493691 + }, + "total_ms": { + "count": 5, + "median": 403.56813291145954, + "p95": 415.8798636548454, + "min": 390.6500468874583, + "max": 418.2807890611002, + "iqr": 12.454363837605342, + "mean": 402.519385816413, + "cv": 0.027219374439954005 + }, + "tokens_per_second": { + "count": 5, + "median": 555.048780447549, + "p95": 572.4796214881569, + "min": 535.5254313802093, + "max": 573.4032333663888, + "iqr": 17.436064835138268, + "mean": 556.8223456618937, + "cv": 0.02700875502804847 + }, + "median_step_ms_by_index": [ + 1.5097090072231367, + 1.567457991768606, + 1.5474579995498061, + 1.5415829984704033, + 1.6097920015454292, + 1.5604999935021624, + 1.5517499996349216, + 1.5026670007500798, + 1.5326670109061524, + 1.547708990983665, + 1.5748339937999845, + 1.5750000020489097, + 1.5269580035237595, + 1.5546670038020238, + 1.4845419937046245, + 1.565666010719724, + 1.5819580003153533, + 1.5898749988991767, + 1.606292003998533, + 1.5672920126235113, + 1.6685000009601936, + 1.6168340080184862, + 1.6762500017648563, + 1.6140419902512804, + 1.5334589988924563, + 1.5300000086426735, + 1.6082500078482553, + 1.6392910038121045, + 1.5500419976888224, + 1.5901250008028, + 1.5560830070171505, + 1.5793750062584877, + 1.6447919915663078, + 1.74079199496191, + 1.614499997231178, + 1.6769590001786128, + 1.6752500087022781, + 1.7072910122806206, + 1.6473340074298903, + 1.7128750041592866, + 1.6804999904707074, + 1.6206670115934685, + 1.6340419970219955, + 1.664000010350719, + 1.6385829949285835, + 1.6368330107070506, + 1.6239169926848263, + 1.6304169985232875, + 1.6960829962044954, + 1.6790830122772604, + 1.690499993856065, + 1.6990839940262958, + 1.7395000031683594, + 1.703666988760233, + 1.669167002546601, + 1.653541999985464, + 1.7291249969275668, + 1.785917003871873, + 1.7223330069100484, + 1.739250001264736, + 1.708416995825246, + 1.7364579980494455, + 1.7375410097884014, + 1.755208009853959, + 1.7450829909648746, + 1.8122499895980582, + 1.8361249967711046, + 1.8412920035189018, + 1.8147500086342916, + 1.7974999936996028, + 1.8055420077871531, + 1.8739579973043874, + 1.8537920113885775, + 1.785332991858013, + 1.8509999936213717, + 1.8828750035027042, + 1.7568750045029446, + 1.837666000938043, + 1.8173749995185062, + 1.8482919986126944, + 1.901499999803491, + 1.8212090071756393, + 1.803249993827194, + 1.7801670037442818, + 1.8284159887116402, + 1.8256250041304156, + 1.8610419938340783, + 1.8570000102045015, + 1.804582992917858, + 1.847958003054373, + 1.8910840008175, + 1.8330409948248416, + 1.83183400076814, + 1.8279579962836578, + 1.7773750005289912, + 1.8216250027762726, + 1.7869169969344512, + 1.8124170019291341, + 1.8362919945502654, + 1.7941669939318672, + 1.815166004234925, + 1.8227920081699267, + 1.8255409959238023, + 1.8226660031359643, + 1.81200000224635, + 1.8301669915672392, + 1.7765829979907721, + 1.7975829978240654, + 1.8194580043200403, + 1.7887089925352484, + 1.8776249926304445, + 1.8215830059489235, + 1.7919169913511723, + 1.8366659933235496, + 1.828374995966442, + 1.788749999832362, + 1.7830830038292333, + 1.8102089961757883, + 1.8350419995840639, + 1.8264170066686347, + 1.83275000017602, + 1.8171660049119964, + 1.8456250109011307, + 1.8357500084675848, + 1.84320799598936, + 1.771832990925759, + 1.8144580099033192, + 1.8137910083169118, + 1.806667001801543, + 1.8073329993057996, + 1.8189580005127937, + 1.8248329870402813, + 1.8198329926235601, + 1.8417909886920825, + 1.7934579955181107, + 1.7662500031292439, + 1.81275000795722, + 1.7920000100275502, + 1.7865410045487806, + 1.8142080079996958, + 1.886291996925138, + 1.8926670018117875, + 1.840041994000785, + 1.8357499939156696, + 1.8397500098217279, + 1.785166998161003, + 1.814457995351404, + 1.9030840048799291, + 1.833332993555814, + 1.7709590028971434, + 1.7765840020729229, + 1.833374990383163, + 1.808540997444652, + 1.7556670063640922, + 1.8185829976573586, + 1.7884999979287386, + 1.8500830046832561, + 1.8514169933041558, + 1.8898330017691478, + 1.835458999266848, + 1.8308750004507601, + 1.7407499981345609, + 1.9353749958099797, + 1.9517499895300716, + 1.9599170045694336, + 1.8207079992862418, + 1.832791997003369, + 1.8492919916752726, + 1.8730000010691583, + 1.8628750112839043, + 1.8614579894347116, + 1.8800830002874136, + 1.8452079966664314, + 1.8997910083271563, + 1.8998750019818544, + 1.7924170097103342, + 1.9252500060247257, + 1.8677500047488138, + 1.9410000095376745, + 1.927582998177968, + 1.7949999892152846, + 1.871583008323796, + 1.9068750116275623, + 1.9560839864425361, + 1.9322919979458675, + 1.955582993105054, + 1.965458010090515, + 1.913125001010485, + 1.8934169929707423, + 1.9482079951558262, + 1.948333010659553, + 1.9644999993033707, + 2.0441249944269657, + 1.9692499918164685, + 1.9600829982664436, + 1.9380000012461096, + 1.9218340021325275, + 1.9029579998459667, + 1.9380830053705722, + 1.8529170047258958, + 1.8752500036498532, + 1.9068749970756471, + 1.879624993307516, + 1.8633330037118867, + 1.8480830040061846, + 1.8477079865988344, + 1.8364580027991906, + 1.8862089928006753, + 1.7692499968688935, + 1.8441669963067397, + 1.9644160056486726, + 1.8890829960582778, + 1.8634580046636984, + 1.826874999096617, + 1.8652500002644956, + 1.8615830049384385, + 1.9141249940730631, + 1.9035000004805624, + 1.9165419944329187, + 1.899833994684741, + 1.8642500072019175, + 1.826499996241182, + 1.7924999992828816, + 1.8029170023510233 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_default": { + "first_token_ms": { + "count": 5, + "median": 1.6924999945331365, + "p95": 1.7808750009862706, + "min": 1.5571249969070777, + "max": 1.786875000107102, + "iqr": 0.1412919955328107, + "mean": 1.681791601004079, + "cv": 0.05697042297585549 + }, + "decode_ms_per_step": { + "count": 1115, + "median": 1.9454580033198, + "p95": 4.041337402304634, + "min": 1.5034590032882988, + "max": 6.131750007625669, + "iqr": 0.20356199820525944, + "mean": 2.1513823641416088, + "cv": 0.3451898266668087 + }, + "total_ms": { + "count": 5, + "median": 439.9458250409225, + "p95": 609.7621788590914, + "min": 420.1328380731866, + "max": 641.912503837375, + "iqr": 57.1126308204839, + "mean": 481.4400588045828, + "cv": 0.19297396228675093 + }, + "tokens_per_second": { + "count": 5, + "median": 509.1535985803801, + "p95": 532.1801180619426, + "min": 348.9572156032486, + "max": 533.16470340026, + "iqr": 62.701019341407175, + "mean": 477.0116103319654, + "cv": 0.16013800637758824 + }, + "median_step_ms_by_index": [ + 1.6924999945331365, + 1.7115000082412735, + 1.7255419952562079, + 1.6627500008326024, + 1.6713329969206825, + 1.6323749878210947, + 1.6841249889694154, + 1.692207995802164, + 1.6865409997990355, + 1.661374990362674, + 1.6431670082965866, + 1.6254579968517646, + 1.6218340024352074, + 1.7481249960837886, + 1.7029580048983917, + 1.7777500033844262, + 1.7241669993381947, + 1.6747500048950315, + 1.6830829990794882, + 1.7394999886164442, + 1.74154200067278, + 1.7042079998645931, + 1.7766670061973855, + 1.7659590084804222, + 1.720165993901901, + 1.7583330045454204, + 1.7568750045029446, + 1.7955000075744465, + 1.7510000034235418, + 1.7165839963126928, + 1.7610419890843332, + 1.7258750012842938, + 1.7422910023014992, + 1.8119590095011517, + 1.855042006354779, + 1.84208300197497, + 1.7941660044016317, + 1.8452499934937805, + 1.8295409972779453, + 1.8157500016968697, + 1.8341660033911467, + 1.822499994887039, + 1.7947090091183782, + 1.7948750028153881, + 1.7998749972321093, + 1.8200829945271835, + 1.800582991563715, + 1.8305840058019385, + 1.8092919926857576, + 1.8624170043040067, + 1.8339579983148724, + 1.8488329951651394, + 1.8222499929834157, + 1.7940419929800555, + 1.8730829906417057, + 1.9351670052856207, + 1.9036670128116384, + 1.8837090028682724, + 1.8652500002644956, + 1.9337499979883432, + 1.9377080025151372, + 1.919957998325117, + 1.8879590061260387, + 1.8162079941248521, + 1.9047500099986792, + 1.8431249918648973, + 1.8837079987861216, + 1.9145420083077624, + 1.9408329972065985, + 1.942583010531962, + 1.9324579916428775, + 1.9235000072512776, + 1.8606669909786433, + 1.90079199091997, + 1.9286249880678952, + 1.9331250077812, + 1.929458012455143, + 1.9952919974457473, + 1.972167010535486, + 1.9525000097928569, + 1.9813339895335957, + 1.9561249937396497, + 1.949500001501292, + 1.887792008346878, + 1.8558750016381964, + 1.9432499975664541, + 1.9965829997090623, + 1.9624999986262992, + 1.9316669931868091, + 1.9817080028587952, + 1.967582997167483, + 1.9697079987963662, + 2.0099999965168536, + 2.031583004281856, + 2.053292002528906, + 2.049457994871773, + 1.9951669964939356, + 1.9982909871032462, + 1.9442920020082965, + 1.9870830001309514, + 1.988917007111013, + 1.9772499945247546, + 1.9834580016322434, + 2.0194999960949644, + 2.0027919963467866, + 2.0296250004321337, + 1.9806250056717545, + 1.9449160026852041, + 1.9654579955385998, + 1.9952909933635965, + 1.987791998544708, + 2.054249998764135, + 2.089665998937562, + 2.068833011435345, + 1.9848329975502566, + 2.0057920046383515, + 2.045332992565818, + 2.0364590018289164, + 2.0419999927980825, + 2.0452499884413555, + 2.0143329893471673, + 2.018040991970338, + 1.989500000490807, + 1.946082993526943, + 2.031000010902062, + 2.0187079935567454, + 1.9749999919440597, + 1.9580000080168247, + 2.01500000548549, + 2.0103749993722886, + 1.9968330016126856, + 1.996167004108429, + 2.0219580037519336, + 1.9945829990319908, + 1.949292010976933, + 1.966416006325744, + 1.9469999970169738, + 2.0151669887127355, + 1.9734169909497723, + 1.9867500086547807, + 2.062667001155205, + 1.9311250071041286, + 1.9832909893011674, + 2.023374996497296, + 1.977499996428378, + 1.9849579985020682, + 2.017500009969808, + 2.011834003496915, + 1.9678749958984554, + 1.9413330010138452, + 1.9842079927911982, + 1.9766670011449605, + 1.966041003470309, + 2.0202080049784854, + 1.990667005884461, + 2.0367920078570023, + 2.005917005590163, + 2.0002499950351194, + 1.9968330016126856, + 2.0070409955224022, + 1.9954579911427572, + 2.0475420024013147, + 2.0496249926509336, + 2.0337080059107393, + 1.9846669893013313, + 1.9653749914141372, + 1.9184170087100938, + 1.9126249972032383, + 1.9130000000586733, + 1.9186670106137171, + 1.9410000095376745, + 1.9416249997448176, + 1.886082987766713, + 1.952791993971914, + 1.9780000002356246, + 1.964416995178908, + 2.0341670024208724, + 1.9869579991791397, + 1.9633339979918674, + 1.9387079955777153, + 2.0005419937660918, + 2.007041999604553, + 2.11666600080207, + 1.9685830047819763, + 2.00379200396128, + 2.104375002090819, + 2.3901249951450154, + 2.0772080024471506, + 2.011333010159433, + 2.00133299222216, + 1.9667500018840656, + 2.02266700216569, + 2.03200000396464, + 2.031790994806215, + 2.092499998980202, + 2.0218340068822727, + 2.073707990348339, + 2.1017920080339536, + 2.090624999254942, + 2.048125010333024, + 2.072165996651165, + 2.0743329951073974, + 2.1135420101927593, + 2.1114580013090745, + 2.039375001913868, + 2.005459013162181, + 2.015541002037935, + 2.0217499986756593, + 2.025875000981614, + 2.0117079984629527, + 2.0583750010700896, + 2.0149579941062257, + 2.0367920078570023, + 2.037083002505824, + 2.064333006273955, + 1.9849579985020682, + 1.9923340005334467, + 2.001499990001321, + 2.0107080054003745, + 2.0820419886149466, + 2.1267499978421256, + 2.308291004737839, + 2.2080409980844706, + 2.2725829912815243 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "speedup_cached_vs_uncached_sdpa": 1.010992018825282, + "speedup_cached_vs_uncached_default": 1.1021229912854864 + }, + "memory": [ + { + "batch": 1, + "prompt_len": 128, + "steps": [ + { + "step": 0, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 1, + "cached_positions": 129, + "cache_bytes": 792576, + "step_peak_minus_start_bytes": 805120 + }, + { + "step": 2, + "cached_positions": 130, + "cache_bytes": 798720, + "step_peak_minus_start_bytes": 811264 + }, + { + "step": 3, + "cached_positions": 131, + "cache_bytes": 804864, + "step_peak_minus_start_bytes": 817408 + }, + { + "step": 4, + "cached_positions": 132, + "cache_bytes": 811008, + "step_peak_minus_start_bytes": 823552 + }, + { + "step": 5, + "cached_positions": 133, + "cache_bytes": 817152, + "step_peak_minus_start_bytes": 829696 + }, + { + "step": 6, + "cached_positions": 134, + "cache_bytes": 823296, + "step_peak_minus_start_bytes": 835840 + }, + { + "step": 7, + "cached_positions": 135, + "cache_bytes": 829440, + "step_peak_minus_start_bytes": 841984 + }, + { + "step": 8, + "cached_positions": 136, + "cache_bytes": 835584, + "step_peak_minus_start_bytes": 848128 + }, + { + "step": 9, + "cached_positions": 137, + "cache_bytes": 841728, + "step_peak_minus_start_bytes": 854272 + }, + { + "step": 10, + "cached_positions": 138, + "cache_bytes": 847872, + "step_peak_minus_start_bytes": 860416 + }, + { + "step": 11, + "cached_positions": 139, + "cache_bytes": 854016, + "step_peak_minus_start_bytes": 866560 + }, + { + "step": 12, + "cached_positions": 140, + "cache_bytes": 860160, + "step_peak_minus_start_bytes": 872704 + }, + { + "step": 13, + "cached_positions": 141, + "cache_bytes": 866304, + "step_peak_minus_start_bytes": 878848 + }, + { + "step": 14, + "cached_positions": 142, + "cache_bytes": 872448, + "step_peak_minus_start_bytes": 884992 + }, + { + "step": 15, + "cached_positions": 143, + "cache_bytes": 878592, + "step_peak_minus_start_bytes": 891136 + }, + { + "step": 16, + "cached_positions": 144, + "cache_bytes": 884736, + "step_peak_minus_start_bytes": 897280 + }, + { + "step": 17, + "cached_positions": 145, + "cache_bytes": 890880, + "step_peak_minus_start_bytes": 903424 + }, + { + "step": 18, + "cached_positions": 146, + "cache_bytes": 897024, + "step_peak_minus_start_bytes": 909568 + }, + { + "step": 19, + "cached_positions": 147, + "cache_bytes": 903168, + "step_peak_minus_start_bytes": 915712 + }, + { + "step": 20, + "cached_positions": 148, + "cache_bytes": 909312, + "step_peak_minus_start_bytes": 921856 + }, + { + "step": 21, + "cached_positions": 149, + "cache_bytes": 915456, + "step_peak_minus_start_bytes": 928000 + }, + { + "step": 22, + "cached_positions": 150, + "cache_bytes": 921600, + "step_peak_minus_start_bytes": 934144 + }, + { + "step": 23, + "cached_positions": 151, + "cache_bytes": 927744, + "step_peak_minus_start_bytes": 940288 + }, + { + "step": 24, + "cached_positions": 152, + "cache_bytes": 933888, + "step_peak_minus_start_bytes": 946432 + }, + { + "step": 25, + "cached_positions": 153, + "cache_bytes": 940032, + "step_peak_minus_start_bytes": 952576 + }, + { + "step": 26, + "cached_positions": 154, + "cache_bytes": 946176, + "step_peak_minus_start_bytes": 958720 + }, + { + "step": 27, + "cached_positions": 155, + "cache_bytes": 952320, + "step_peak_minus_start_bytes": 964864 + }, + { + "step": 28, + "cached_positions": 156, + "cache_bytes": 958464, + "step_peak_minus_start_bytes": 971008 + }, + { + "step": 29, + "cached_positions": 157, + "cache_bytes": 964608, + "step_peak_minus_start_bytes": 977152 + }, + { + "step": 30, + "cached_positions": 158, + "cache_bytes": 970752, + "step_peak_minus_start_bytes": 983296 + }, + { + "step": 31, + "cached_positions": 159, + "cache_bytes": 976896, + "step_peak_minus_start_bytes": 989440 + }, + { + "step": 32, + "cached_positions": 160, + "cache_bytes": 983040, + "step_peak_minus_start_bytes": 995584 + }, + { + "step": 33, + "cached_positions": 161, + "cache_bytes": 989184, + "step_peak_minus_start_bytes": 1001728 + }, + { + "step": 34, + "cached_positions": 162, + "cache_bytes": 995328, + "step_peak_minus_start_bytes": 1007872 + }, + { + "step": 35, + "cached_positions": 163, + "cache_bytes": 1001472, + "step_peak_minus_start_bytes": 1014016 + }, + { + "step": 36, + "cached_positions": 164, + "cache_bytes": 1007616, + "step_peak_minus_start_bytes": 1020160 + }, + { + "step": 37, + "cached_positions": 165, + "cache_bytes": 1013760, + "step_peak_minus_start_bytes": 1026304 + }, + { + "step": 38, + "cached_positions": 166, + "cache_bytes": 1019904, + "step_peak_minus_start_bytes": 1032448 + }, + { + "step": 39, + "cached_positions": 167, + "cache_bytes": 1026048, + "step_peak_minus_start_bytes": 1038592 + }, + { + "step": 40, + "cached_positions": 168, + "cache_bytes": 1032192, + "step_peak_minus_start_bytes": 1044736 + }, + { + "step": 41, + "cached_positions": 169, + "cache_bytes": 1038336, + "step_peak_minus_start_bytes": 1050880 + }, + { + "step": 42, + "cached_positions": 170, + "cache_bytes": 1044480, + "step_peak_minus_start_bytes": 1057024 + }, + { + "step": 43, + "cached_positions": 171, + "cache_bytes": 1050624, + "step_peak_minus_start_bytes": 1063168 + }, + { + "step": 44, + "cached_positions": 172, + "cache_bytes": 1056768, + "step_peak_minus_start_bytes": 1069312 + }, + { + "step": 45, + "cached_positions": 173, + "cache_bytes": 1062912, + "step_peak_minus_start_bytes": 1075456 + }, + { + "step": 46, + "cached_positions": 174, + "cache_bytes": 1069056, + "step_peak_minus_start_bytes": 1081600 + }, + { + "step": 47, + "cached_positions": 175, + "cache_bytes": 1075200, + "step_peak_minus_start_bytes": 1087744 + }, + { + "step": 48, + "cached_positions": 176, + "cache_bytes": 1081344, + "step_peak_minus_start_bytes": 1093888 + }, + { + "step": 49, + "cached_positions": 177, + "cache_bytes": 1087488, + "step_peak_minus_start_bytes": 1100032 + }, + { + "step": 50, + "cached_positions": 178, + "cache_bytes": 1093632, + "step_peak_minus_start_bytes": 1106176 + }, + { + "step": 51, + "cached_positions": 179, + "cache_bytes": 1099776, + "step_peak_minus_start_bytes": 1112320 + }, + { + "step": 52, + "cached_positions": 180, + "cache_bytes": 1105920, + "step_peak_minus_start_bytes": 1118464 + }, + { + "step": 53, + "cached_positions": 181, + "cache_bytes": 1112064, + "step_peak_minus_start_bytes": 1124608 + }, + { + "step": 54, + "cached_positions": 182, + "cache_bytes": 1118208, + "step_peak_minus_start_bytes": 1130752 + }, + { + "step": 55, + "cached_positions": 183, + "cache_bytes": 1124352, + "step_peak_minus_start_bytes": 1136896 + }, + { + "step": 56, + "cached_positions": 184, + "cache_bytes": 1130496, + "step_peak_minus_start_bytes": 1143040 + }, + { + "step": 57, + "cached_positions": 185, + "cache_bytes": 1136640, + "step_peak_minus_start_bytes": 1149184 + }, + { + "step": 58, + "cached_positions": 186, + "cache_bytes": 1142784, + "step_peak_minus_start_bytes": 1155328 + }, + { + "step": 59, + "cached_positions": 187, + "cache_bytes": 1148928, + "step_peak_minus_start_bytes": 1161472 + }, + { + "step": 60, + "cached_positions": 188, + "cache_bytes": 1155072, + "step_peak_minus_start_bytes": 1167616 + }, + { + "step": 61, + "cached_positions": 189, + "cache_bytes": 1161216, + "step_peak_minus_start_bytes": 1173760 + }, + { + "step": 62, + "cached_positions": 190, + "cache_bytes": 1167360, + "step_peak_minus_start_bytes": 1179904 + }, + { + "step": 63, + "cached_positions": 191, + "cache_bytes": 1173504, + "step_peak_minus_start_bytes": 1186048 + } + ] + }, + { + "batch": 8, + "prompt_len": 128, + "steps": [ + { + "step": 0, + "cached_positions": 128, + "cache_bytes": 6291456, + "step_peak_minus_start_bytes": 25953280 + }, + { + "step": 1, + "cached_positions": 129, + "cache_bytes": 6340608, + "step_peak_minus_start_bytes": 6439168 + }, + { + "step": 2, + "cached_positions": 130, + "cache_bytes": 6389760, + "step_peak_minus_start_bytes": 6488320 + }, + { + "step": 3, + "cached_positions": 131, + "cache_bytes": 6438912, + "step_peak_minus_start_bytes": 6537472 + }, + { + "step": 4, + "cached_positions": 132, + "cache_bytes": 6488064, + "step_peak_minus_start_bytes": 6586624 + }, + { + "step": 5, + "cached_positions": 133, + "cache_bytes": 6537216, + "step_peak_minus_start_bytes": 6635776 + }, + { + "step": 6, + "cached_positions": 134, + "cache_bytes": 6586368, + "step_peak_minus_start_bytes": 6684928 + }, + { + "step": 7, + "cached_positions": 135, + "cache_bytes": 6635520, + "step_peak_minus_start_bytes": 6734080 + }, + { + "step": 8, + "cached_positions": 136, + "cache_bytes": 6684672, + "step_peak_minus_start_bytes": 6783232 + }, + { + "step": 9, + "cached_positions": 137, + "cache_bytes": 6733824, + "step_peak_minus_start_bytes": 6832384 + }, + { + "step": 10, + "cached_positions": 138, + "cache_bytes": 6782976, + "step_peak_minus_start_bytes": 6881536 + }, + { + "step": 11, + "cached_positions": 139, + "cache_bytes": 6832128, + "step_peak_minus_start_bytes": 6930688 + }, + { + "step": 12, + "cached_positions": 140, + "cache_bytes": 6881280, + "step_peak_minus_start_bytes": 6979840 + }, + { + "step": 13, + "cached_positions": 141, + "cache_bytes": 6930432, + "step_peak_minus_start_bytes": 7028992 + }, + { + "step": 14, + "cached_positions": 142, + "cache_bytes": 6979584, + "step_peak_minus_start_bytes": 7078144 + }, + { + "step": 15, + "cached_positions": 143, + "cache_bytes": 7028736, + "step_peak_minus_start_bytes": 7127296 + }, + { + "step": 16, + "cached_positions": 144, + "cache_bytes": 7077888, + "step_peak_minus_start_bytes": 7176448 + }, + { + "step": 17, + "cached_positions": 145, + "cache_bytes": 7127040, + "step_peak_minus_start_bytes": 7225600 + }, + { + "step": 18, + "cached_positions": 146, + "cache_bytes": 7176192, + "step_peak_minus_start_bytes": 7274752 + }, + { + "step": 19, + "cached_positions": 147, + "cache_bytes": 7225344, + "step_peak_minus_start_bytes": 7323904 + }, + { + "step": 20, + "cached_positions": 148, + "cache_bytes": 7274496, + "step_peak_minus_start_bytes": 7373056 + }, + { + "step": 21, + "cached_positions": 149, + "cache_bytes": 7323648, + "step_peak_minus_start_bytes": 7422208 + }, + { + "step": 22, + "cached_positions": 150, + "cache_bytes": 7372800, + "step_peak_minus_start_bytes": 7471360 + }, + { + "step": 23, + "cached_positions": 151, + "cache_bytes": 7421952, + "step_peak_minus_start_bytes": 7520512 + }, + { + "step": 24, + "cached_positions": 152, + "cache_bytes": 7471104, + "step_peak_minus_start_bytes": 7569664 + }, + { + "step": 25, + "cached_positions": 153, + "cache_bytes": 7520256, + "step_peak_minus_start_bytes": 7618816 + }, + { + "step": 26, + "cached_positions": 154, + "cache_bytes": 7569408, + "step_peak_minus_start_bytes": 7667968 + }, + { + "step": 27, + "cached_positions": 155, + "cache_bytes": 7618560, + "step_peak_minus_start_bytes": 7717120 + }, + { + "step": 28, + "cached_positions": 156, + "cache_bytes": 7667712, + "step_peak_minus_start_bytes": 7766272 + }, + { + "step": 29, + "cached_positions": 157, + "cache_bytes": 7716864, + "step_peak_minus_start_bytes": 7815424 + }, + { + "step": 30, + "cached_positions": 158, + "cache_bytes": 7766016, + "step_peak_minus_start_bytes": 7864576 + }, + { + "step": 31, + "cached_positions": 159, + "cache_bytes": 7815168, + "step_peak_minus_start_bytes": 7913728 + }, + { + "step": 32, + "cached_positions": 160, + "cache_bytes": 7864320, + "step_peak_minus_start_bytes": 7962880 + }, + { + "step": 33, + "cached_positions": 161, + "cache_bytes": 7913472, + "step_peak_minus_start_bytes": 8012032 + }, + { + "step": 34, + "cached_positions": 162, + "cache_bytes": 7962624, + "step_peak_minus_start_bytes": 8061184 + }, + { + "step": 35, + "cached_positions": 163, + "cache_bytes": 8011776, + "step_peak_minus_start_bytes": 8110336 + }, + { + "step": 36, + "cached_positions": 164, + "cache_bytes": 8060928, + "step_peak_minus_start_bytes": 8159488 + }, + { + "step": 37, + "cached_positions": 165, + "cache_bytes": 8110080, + "step_peak_minus_start_bytes": 8208640 + }, + { + "step": 38, + "cached_positions": 166, + "cache_bytes": 8159232, + "step_peak_minus_start_bytes": 8257792 + }, + { + "step": 39, + "cached_positions": 167, + "cache_bytes": 8208384, + "step_peak_minus_start_bytes": 8306944 + }, + { + "step": 40, + "cached_positions": 168, + "cache_bytes": 8257536, + "step_peak_minus_start_bytes": 8356096 + }, + { + "step": 41, + "cached_positions": 169, + "cache_bytes": 8306688, + "step_peak_minus_start_bytes": 8405248 + }, + { + "step": 42, + "cached_positions": 170, + "cache_bytes": 8355840, + "step_peak_minus_start_bytes": 8454400 + }, + { + "step": 43, + "cached_positions": 171, + "cache_bytes": 8404992, + "step_peak_minus_start_bytes": 11993344 + }, + { + "step": 44, + "cached_positions": 172, + "cache_bytes": 8454144, + "step_peak_minus_start_bytes": 13777152 + }, + { + "step": 45, + "cached_positions": 173, + "cache_bytes": 8503296, + "step_peak_minus_start_bytes": 11993344 + }, + { + "step": 46, + "cached_positions": 174, + "cache_bytes": 8552448, + "step_peak_minus_start_bytes": 13777152 + }, + { + "step": 47, + "cached_positions": 175, + "cache_bytes": 8601600, + "step_peak_minus_start_bytes": 11993344 + }, + { + "step": 48, + "cached_positions": 176, + "cache_bytes": 8650752, + "step_peak_minus_start_bytes": 13777152 + }, + { + "step": 49, + "cached_positions": 177, + "cache_bytes": 8699904, + "step_peak_minus_start_bytes": 11993344 + }, + { + "step": 50, + "cached_positions": 178, + "cache_bytes": 8749056, + "step_peak_minus_start_bytes": 13973760 + }, + { + "step": 51, + "cached_positions": 179, + "cache_bytes": 8798208, + "step_peak_minus_start_bytes": 11993344 + }, + { + "step": 52, + "cached_positions": 180, + "cache_bytes": 8847360, + "step_peak_minus_start_bytes": 13973760 + }, + { + "step": 53, + "cached_positions": 181, + "cache_bytes": 8896512, + "step_peak_minus_start_bytes": 11993344 + }, + { + "step": 54, + "cached_positions": 182, + "cache_bytes": 8945664, + "step_peak_minus_start_bytes": 13973760 + }, + { + "step": 55, + "cached_positions": 183, + "cache_bytes": 8994816, + "step_peak_minus_start_bytes": 11993344 + }, + { + "step": 56, + "cached_positions": 184, + "cache_bytes": 9043968, + "step_peak_minus_start_bytes": 13973760 + }, + { + "step": 57, + "cached_positions": 185, + "cache_bytes": 9093120, + "step_peak_minus_start_bytes": 11993344 + }, + { + "step": 58, + "cached_positions": 186, + "cache_bytes": 9142272, + "step_peak_minus_start_bytes": 13973760 + }, + { + "step": 59, + "cached_positions": 187, + "cache_bytes": 9191424, + "step_peak_minus_start_bytes": 11993344 + }, + { + "step": 60, + "cached_positions": 188, + "cache_bytes": 9240576, + "step_peak_minus_start_bytes": 13973760 + }, + { + "step": 61, + "cached_positions": 189, + "cache_bytes": 9289728, + "step_peak_minus_start_bytes": 11993344 + }, + { + "step": 62, + "cached_positions": 190, + "cache_bytes": 9338880, + "step_peak_minus_start_bytes": 13973760 + }, + { + "step": 63, + "cached_positions": 191, + "cache_bytes": 9388032, + "step_peak_minus_start_bytes": 11993344 + } + ] + }, + { + "batch": 1, + "prompt_len": 32, + "steps": [ + { + "step": 0, + "cached_positions": 32, + "cache_bytes": 196608, + "step_peak_minus_start_bytes": 688384 + }, + { + "step": 1, + "cached_positions": 33, + "cache_bytes": 202752, + "step_peak_minus_start_bytes": 215296 + }, + { + "step": 2, + "cached_positions": 34, + "cache_bytes": 208896, + "step_peak_minus_start_bytes": 221440 + }, + { + "step": 3, + "cached_positions": 35, + "cache_bytes": 215040, + "step_peak_minus_start_bytes": 227584 + }, + { + "step": 4, + "cached_positions": 36, + "cache_bytes": 221184, + "step_peak_minus_start_bytes": 233728 + }, + { + "step": 5, + "cached_positions": 37, + "cache_bytes": 227328, + "step_peak_minus_start_bytes": 239872 + }, + { + "step": 6, + "cached_positions": 38, + "cache_bytes": 233472, + "step_peak_minus_start_bytes": 246016 + }, + { + "step": 7, + "cached_positions": 39, + "cache_bytes": 239616, + "step_peak_minus_start_bytes": 252160 + }, + { + "step": 8, + "cached_positions": 40, + "cache_bytes": 245760, + "step_peak_minus_start_bytes": 258304 + }, + { + "step": 9, + "cached_positions": 41, + "cache_bytes": 251904, + "step_peak_minus_start_bytes": 264448 + }, + { + "step": 10, + "cached_positions": 42, + "cache_bytes": 258048, + "step_peak_minus_start_bytes": 270592 + }, + { + "step": 11, + "cached_positions": 43, + "cache_bytes": 264192, + "step_peak_minus_start_bytes": 276736 + }, + { + "step": 12, + "cached_positions": 44, + "cache_bytes": 270336, + "step_peak_minus_start_bytes": 282880 + }, + { + "step": 13, + "cached_positions": 45, + "cache_bytes": 276480, + "step_peak_minus_start_bytes": 289024 + }, + { + "step": 14, + "cached_positions": 46, + "cache_bytes": 282624, + "step_peak_minus_start_bytes": 295168 + }, + { + "step": 15, + "cached_positions": 47, + "cache_bytes": 288768, + "step_peak_minus_start_bytes": 301312 + }, + { + "step": 16, + "cached_positions": 48, + "cache_bytes": 294912, + "step_peak_minus_start_bytes": 307456 + }, + { + "step": 17, + "cached_positions": 49, + "cache_bytes": 301056, + "step_peak_minus_start_bytes": 313600 + }, + { + "step": 18, + "cached_positions": 50, + "cache_bytes": 307200, + "step_peak_minus_start_bytes": 319744 + }, + { + "step": 19, + "cached_positions": 51, + "cache_bytes": 313344, + "step_peak_minus_start_bytes": 325888 + }, + { + "step": 20, + "cached_positions": 52, + "cache_bytes": 319488, + "step_peak_minus_start_bytes": 332032 + }, + { + "step": 21, + "cached_positions": 53, + "cache_bytes": 325632, + "step_peak_minus_start_bytes": 338176 + }, + { + "step": 22, + "cached_positions": 54, + "cache_bytes": 331776, + "step_peak_minus_start_bytes": 344320 + }, + { + "step": 23, + "cached_positions": 55, + "cache_bytes": 337920, + "step_peak_minus_start_bytes": 350464 + }, + { + "step": 24, + "cached_positions": 56, + "cache_bytes": 344064, + "step_peak_minus_start_bytes": 356608 + }, + { + "step": 25, + "cached_positions": 57, + "cache_bytes": 350208, + "step_peak_minus_start_bytes": 362752 + }, + { + "step": 26, + "cached_positions": 58, + "cache_bytes": 356352, + "step_peak_minus_start_bytes": 368896 + }, + { + "step": 27, + "cached_positions": 59, + "cache_bytes": 362496, + "step_peak_minus_start_bytes": 375040 + }, + { + "step": 28, + "cached_positions": 60, + "cache_bytes": 368640, + "step_peak_minus_start_bytes": 381184 + }, + { + "step": 29, + "cached_positions": 61, + "cache_bytes": 374784, + "step_peak_minus_start_bytes": 387328 + }, + { + "step": 30, + "cached_positions": 62, + "cache_bytes": 380928, + "step_peak_minus_start_bytes": 393472 + }, + { + "step": 31, + "cached_positions": 63, + "cache_bytes": 387072, + "step_peak_minus_start_bytes": 399616 + }, + { + "step": 32, + "cached_positions": 64, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 405760 + }, + { + "step": 33, + "cached_positions": 65, + "cache_bytes": 399360, + "step_peak_minus_start_bytes": 411904 + }, + { + "step": 34, + "cached_positions": 66, + "cache_bytes": 405504, + "step_peak_minus_start_bytes": 418048 + }, + { + "step": 35, + "cached_positions": 67, + "cache_bytes": 411648, + "step_peak_minus_start_bytes": 424192 + }, + { + "step": 36, + "cached_positions": 68, + "cache_bytes": 417792, + "step_peak_minus_start_bytes": 430336 + }, + { + "step": 37, + "cached_positions": 69, + "cache_bytes": 423936, + "step_peak_minus_start_bytes": 436480 + }, + { + "step": 38, + "cached_positions": 70, + "cache_bytes": 430080, + "step_peak_minus_start_bytes": 442624 + }, + { + "step": 39, + "cached_positions": 71, + "cache_bytes": 436224, + "step_peak_minus_start_bytes": 448768 + }, + { + "step": 40, + "cached_positions": 72, + "cache_bytes": 442368, + "step_peak_minus_start_bytes": 454912 + }, + { + "step": 41, + "cached_positions": 73, + "cache_bytes": 448512, + "step_peak_minus_start_bytes": 461056 + }, + { + "step": 42, + "cached_positions": 74, + "cache_bytes": 454656, + "step_peak_minus_start_bytes": 467200 + }, + { + "step": 43, + "cached_positions": 75, + "cache_bytes": 460800, + "step_peak_minus_start_bytes": 473344 + }, + { + "step": 44, + "cached_positions": 76, + "cache_bytes": 466944, + "step_peak_minus_start_bytes": 479488 + }, + { + "step": 45, + "cached_positions": 77, + "cache_bytes": 473088, + "step_peak_minus_start_bytes": 485632 + }, + { + "step": 46, + "cached_positions": 78, + "cache_bytes": 479232, + "step_peak_minus_start_bytes": 491776 + }, + { + "step": 47, + "cached_positions": 79, + "cache_bytes": 485376, + "step_peak_minus_start_bytes": 497920 + }, + { + "step": 48, + "cached_positions": 80, + "cache_bytes": 491520, + "step_peak_minus_start_bytes": 504064 + }, + { + "step": 49, + "cached_positions": 81, + "cache_bytes": 497664, + "step_peak_minus_start_bytes": 510208 + }, + { + "step": 50, + "cached_positions": 82, + "cache_bytes": 503808, + "step_peak_minus_start_bytes": 516352 + }, + { + "step": 51, + "cached_positions": 83, + "cache_bytes": 509952, + "step_peak_minus_start_bytes": 522496 + }, + { + "step": 52, + "cached_positions": 84, + "cache_bytes": 516096, + "step_peak_minus_start_bytes": 528640 + }, + { + "step": 53, + "cached_positions": 85, + "cache_bytes": 522240, + "step_peak_minus_start_bytes": 534784 + }, + { + "step": 54, + "cached_positions": 86, + "cache_bytes": 528384, + "step_peak_minus_start_bytes": 540928 + }, + { + "step": 55, + "cached_positions": 87, + "cache_bytes": 534528, + "step_peak_minus_start_bytes": 547072 + }, + { + "step": 56, + "cached_positions": 88, + "cache_bytes": 540672, + "step_peak_minus_start_bytes": 553216 + }, + { + "step": 57, + "cached_positions": 89, + "cache_bytes": 546816, + "step_peak_minus_start_bytes": 559360 + }, + { + "step": 58, + "cached_positions": 90, + "cache_bytes": 552960, + "step_peak_minus_start_bytes": 565504 + }, + { + "step": 59, + "cached_positions": 91, + "cache_bytes": 559104, + "step_peak_minus_start_bytes": 571648 + }, + { + "step": 60, + "cached_positions": 92, + "cache_bytes": 565248, + "step_peak_minus_start_bytes": 577792 + }, + { + "step": 61, + "cached_positions": 93, + "cache_bytes": 571392, + "step_peak_minus_start_bytes": 583936 + }, + { + "step": 62, + "cached_positions": 94, + "cache_bytes": 577536, + "step_peak_minus_start_bytes": 590080 + }, + { + "step": 63, + "cached_positions": 95, + "cache_bytes": 583680, + "step_peak_minus_start_bytes": 596224 + }, + { + "step": 64, + "cached_positions": 96, + "cache_bytes": 589824, + "step_peak_minus_start_bytes": 602368 + }, + { + "step": 65, + "cached_positions": 97, + "cache_bytes": 595968, + "step_peak_minus_start_bytes": 608512 + }, + { + "step": 66, + "cached_positions": 98, + "cache_bytes": 602112, + "step_peak_minus_start_bytes": 614656 + }, + { + "step": 67, + "cached_positions": 99, + "cache_bytes": 608256, + "step_peak_minus_start_bytes": 620800 + }, + { + "step": 68, + "cached_positions": 100, + "cache_bytes": 614400, + "step_peak_minus_start_bytes": 626944 + }, + { + "step": 69, + "cached_positions": 101, + "cache_bytes": 620544, + "step_peak_minus_start_bytes": 633088 + }, + { + "step": 70, + "cached_positions": 102, + "cache_bytes": 626688, + "step_peak_minus_start_bytes": 639232 + }, + { + "step": 71, + "cached_positions": 103, + "cache_bytes": 632832, + "step_peak_minus_start_bytes": 645376 + }, + { + "step": 72, + "cached_positions": 104, + "cache_bytes": 638976, + "step_peak_minus_start_bytes": 651520 + }, + { + "step": 73, + "cached_positions": 105, + "cache_bytes": 645120, + "step_peak_minus_start_bytes": 657664 + }, + { + "step": 74, + "cached_positions": 106, + "cache_bytes": 651264, + "step_peak_minus_start_bytes": 663808 + }, + { + "step": 75, + "cached_positions": 107, + "cache_bytes": 657408, + "step_peak_minus_start_bytes": 669952 + }, + { + "step": 76, + "cached_positions": 108, + "cache_bytes": 663552, + "step_peak_minus_start_bytes": 676096 + }, + { + "step": 77, + "cached_positions": 109, + "cache_bytes": 669696, + "step_peak_minus_start_bytes": 682240 + }, + { + "step": 78, + "cached_positions": 110, + "cache_bytes": 675840, + "step_peak_minus_start_bytes": 688384 + }, + { + "step": 79, + "cached_positions": 111, + "cache_bytes": 681984, + "step_peak_minus_start_bytes": 694528 + }, + { + "step": 80, + "cached_positions": 112, + "cache_bytes": 688128, + "step_peak_minus_start_bytes": 700672 + }, + { + "step": 81, + "cached_positions": 113, + "cache_bytes": 694272, + "step_peak_minus_start_bytes": 706816 + }, + { + "step": 82, + "cached_positions": 114, + "cache_bytes": 700416, + "step_peak_minus_start_bytes": 712960 + }, + { + "step": 83, + "cached_positions": 115, + "cache_bytes": 706560, + "step_peak_minus_start_bytes": 719104 + }, + { + "step": 84, + "cached_positions": 116, + "cache_bytes": 712704, + "step_peak_minus_start_bytes": 725248 + }, + { + "step": 85, + "cached_positions": 117, + "cache_bytes": 718848, + "step_peak_minus_start_bytes": 731392 + }, + { + "step": 86, + "cached_positions": 118, + "cache_bytes": 724992, + "step_peak_minus_start_bytes": 737536 + }, + { + "step": 87, + "cached_positions": 119, + "cache_bytes": 731136, + "step_peak_minus_start_bytes": 743680 + }, + { + "step": 88, + "cached_positions": 120, + "cache_bytes": 737280, + "step_peak_minus_start_bytes": 749824 + }, + { + "step": 89, + "cached_positions": 121, + "cache_bytes": 743424, + "step_peak_minus_start_bytes": 755968 + }, + { + "step": 90, + "cached_positions": 122, + "cache_bytes": 749568, + "step_peak_minus_start_bytes": 762112 + }, + { + "step": 91, + "cached_positions": 123, + "cache_bytes": 755712, + "step_peak_minus_start_bytes": 768256 + }, + { + "step": 92, + "cached_positions": 124, + "cache_bytes": 761856, + "step_peak_minus_start_bytes": 774400 + }, + { + "step": 93, + "cached_positions": 125, + "cache_bytes": 768000, + "step_peak_minus_start_bytes": 780544 + }, + { + "step": 94, + "cached_positions": 126, + "cache_bytes": 774144, + "step_peak_minus_start_bytes": 786688 + }, + { + "step": 95, + "cached_positions": 127, + "cache_bytes": 780288, + "step_peak_minus_start_bytes": 792832 + }, + { + "step": 96, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 798976 + }, + { + "step": 97, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 98, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 99, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 100, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 101, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 102, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 103, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 104, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 105, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 106, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 107, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 108, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 109, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 110, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 111, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 112, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 113, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 114, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 115, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 116, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 117, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 118, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 119, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 120, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 121, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 122, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 123, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 124, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 125, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 126, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 127, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 128, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 129, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 130, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 131, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 132, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 133, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 134, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 135, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 136, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 137, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 138, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 139, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 140, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 141, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 142, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 143, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 144, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 145, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 146, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 147, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 148, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 149, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 150, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 151, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 152, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 153, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 154, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 155, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 156, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 157, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 158, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 159, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 160, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 161, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 162, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 163, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 164, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 165, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 166, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 167, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 168, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 169, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 170, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 171, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 172, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 173, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 174, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 175, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 176, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 177, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 178, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 179, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 180, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 181, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 182, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 183, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 184, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 185, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 186, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 187, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 188, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 189, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 190, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 191, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 192, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 193, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 194, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 195, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 196, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 197, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 198, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 199, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 200, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 201, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 202, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 203, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 204, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 205, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 206, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 207, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 208, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 209, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 210, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 211, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 212, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 213, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 214, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 215, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 216, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 217, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 218, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 219, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 220, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 221, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 222, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 223, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + } + ] + } + ] + } +} diff --git a/experiments/records/kv_cache/20261022T040019Z_kv_mps_float16_93507c4.json b/experiments/records/kv_cache/20261022T040019Z_kv_mps_float16_93507c4.json new file mode 100644 index 0000000..5f417a7 --- /dev/null +++ b/experiments/records/kv_cache/20261022T040019Z_kv_mps_float16_93507c4.json @@ -0,0 +1,9045 @@ +{ + "record_version": 1, + "created_utc": "20261022T040019Z", + "experiment": "kv_cache_scaling", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "mps", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "93507c45fb853c19bfc7bbafba89d31852661b53", + "branch": "experiments/attention-kv-audit", + "dirty": false + }, + "command": "python scripts/benchmark_kv_scaling.py --device mps --dtype float16" + }, + "settings": { + "dtype": "float16", + "position": "learned", + "prompt_lens": [ + 1, + 16, + 128, + 512, + 1024 + ], + "batch_sizes": [ + 1, + 8 + ], + "new_tokens": 64, + "warmup": 1, + "repeats": 5, + "seed": 0, + "modes": { + "cached": { + "attention_backend": "sdpa", + "use_cache": true + }, + "uncached_sdpa": { + "attention_backend": "sdpa", + "use_cache": false + }, + "uncached_default": { + "attention_backend": "auto", + "use_cache": false + } + }, + "weights": "seeded random initialization (timing does not depend on trained values)", + "grid_model_config": { + "vocab_size": 256, + "context_length": 1088, + "embedding_dim": 192, + "num_layers": 4, + "num_heads": 6, + "dropout": 0.0, + "positional_encoding": "learned", + "norm": "rmsnorm", + "activation": "swiglu", + "rope_theta": 10000.0 + }, + "window_model_config": { + "vocab_size": 256, + "context_length": 128, + "embedding_dim": 192, + "num_layers": 4, + "num_heads": 6, + "dropout": 0.0, + "positional_encoding": "learned", + "norm": "rmsnorm", + "activation": "swiglu", + "rope_theta": 10000.0 + }, + "decoding": "greedy argmax; step 0 is the prompt prefill" + }, + "results": { + "grid": [ + { + "batch": 1, + "prompt_len": 1, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_default": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 1.4848339924355969, + "p95": 1.676000002771616, + "min": 1.4121249987510964, + "max": 1.7126250022556633, + "iqr": 0.0957910087890923, + "mean": 1.5145585988648236, + "cv": 0.07907017806052204 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.5585829969495535, + "p95": 1.7502418966614641, + "min": 1.4874170010443777, + "max": 5.71374999708496, + "iqr": 0.05618700379272923, + "mean": 1.6284285783065513, + "cv": 0.23388933928297267 + }, + "total_ms": { + "count": 5, + "median": 101.00554805831052, + "p95": 114.99854887661058, + "min": 99.5852530177217, + "max": 118.40962308633607, + "iqr": 1.1811330768978223, + "mean": 104.10555903217755, + "cv": 0.07709730286989286 + }, + "tokens_per_second": { + "count": 5, + "median": 633.6285603148531, + "p95": 641.911139085334, + "min": 540.4966111017484, + "max": 642.6654354998815, + "iqr": 7.4453588858024204, + "mean": 617.4266309769936, + "cv": 0.07001574899879853 + }, + "median_step_ms_by_index": [ + 1.4848339924355969, + 1.6140419902512804, + 1.5722920070402324, + 1.5539580053882673, + 1.5622919891029596, + 1.6747920017223805, + 1.5843750006752089, + 1.568042003782466, + 1.6124579997267574, + 1.5615410084137693, + 1.5364999999292195, + 1.5394169895444065, + 1.5786669973749667, + 1.582707991474308, + 1.674041006481275, + 1.539208009489812, + 1.5763749979669228, + 1.554125003167428, + 1.537208998342976, + 1.5635829913662747, + 1.546542000141926, + 1.567084007547237, + 1.5642080106772482, + 1.5637080068700016, + 1.554541988298297, + 1.5363330021500587, + 1.5809169999556616, + 1.5589999966323376, + 1.5539169980911538, + 1.5458749985555187, + 1.5480000001844019, + 1.5346250002039596, + 1.5522920002695173, + 1.5422080032294616, + 1.5358749951701611, + 1.5728330035926774, + 1.5422500000568107, + 1.5704160032328218, + 1.549167005578056, + 1.5530000091530383, + 1.5473749954253435, + 1.5712910098955035, + 1.552417001221329, + 1.546875006170012, + 1.5279999934136868, + 1.5472499944735318, + 1.5457090048585087, + 1.5447079931618646, + 1.5674169990234077, + 1.5552080003544688, + 1.6219999961322173, + 1.5734580083517358, + 1.5353330090874806, + 1.588750004884787, + 1.5682920056860894, + 1.5459580026799813, + 1.5483340102946386, + 1.55125001037959, + 1.549332999275066, + 1.6082910005934536, + 1.5405000012833625, + 1.5713750035502017, + 1.5932920068735257, + 1.5558330051135272 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 1.4144999877316877, + "p95": 1.5386756014777347, + "min": 1.3755419931840152, + "max": 1.5432090003741905, + "iqr": 0.14291700790636241, + "mean": 1.4462835970334709, + "cv": 0.05535459891155955 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.4168329944368452, + "p95": 1.5416170019307174, + "min": 1.1565419990802184, + "max": 1.9112910085823387, + "iqr": 0.13752099766861647, + "mean": 1.4038140036439197, + "cv": 0.07216461231738028 + }, + "total_ms": { + "count": 5, + "median": 89.81670507637318, + "p95": 90.74499824200757, + "min": 89.17699902667664, + "max": 90.84991404961329, + "iqr": 1.0614590428303927, + "mean": 89.88656582660042, + "cv": 0.007899366245327596 + }, + "tokens_per_second": { + "count": 5, + "median": 712.5623228505137, + "p95": 717.5341508791731, + "min": 704.458564100011, + "max": 717.6738475002379, + "iqr": 8.425542888116524, + "mean": 712.0439840704947, + "cv": 0.007883166471316561 + }, + "median_step_ms_by_index": [ + 1.4144999877316877, + 1.4600830036215484, + 1.429750001989305, + 1.4348749973578379, + 1.462500003981404, + 1.4766250096727163, + 1.4953330100979656, + 1.486875000409782, + 1.1979999981122091, + 1.196625002194196, + 1.216625008964911, + 1.2185410014353693, + 1.236583004356362, + 1.23666699801106, + 1.228291992447339, + 1.2722090032184497, + 1.360415990347974, + 1.3811250100843608, + 1.3574589975178242, + 1.3942500081611797, + 1.3626250001834705, + 1.3484579976648092, + 1.317916001426056, + 1.3262080028653145, + 1.3498749904101714, + 1.3338749995455146, + 1.3465840020217001, + 1.365375006571412, + 1.3495830062311143, + 1.4075829967623577, + 1.3614579947898164, + 1.3829160016030073, + 1.3738749985350296, + 1.3719999988097697, + 1.364417010336183, + 1.3793329999316484, + 1.340541013632901, + 1.3934999878983945, + 1.4119999977992848, + 1.442709006369114, + 1.4028330042492598, + 1.35029100056272, + 1.4414170000236481, + 1.446208989364095, + 1.4987500035203993, + 1.4439159858739004, + 1.5095410053618252, + 1.5119590098038316, + 1.5080000011948869, + 1.4866249985061586, + 1.4896249922458082, + 1.4169579953886569, + 1.4788330008741468, + 1.4553329965565354, + 1.4914580096956342, + 1.476790988817811, + 1.4299580070655793, + 1.4114170044194907, + 1.508999994257465, + 1.4422920066863298, + 1.4588329941034317, + 1.5048750065034255, + 1.5428329934366047, + 1.527874992461875 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_default": { + "first_token_ms": { + "count": 5, + "median": 1.498833007644862, + "p95": 1.613050006562844, + "min": 1.4862079988233745, + "max": 1.625125005375594, + "iqr": 0.07791700772941113, + "mean": 1.5323498053476214, + "cv": 0.03993795809264771 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.5327089931815863, + "p95": 1.6784787003416568, + "min": 1.2425830063875765, + "max": 1.9720830023288727, + "iqr": 0.12197949399705976, + "mean": 1.5216044855714288, + "cv": 0.06859743814825343 + }, + "total_ms": { + "count": 5, + "median": 97.23058102827054, + "p95": 97.99728198850062, + "min": 96.7825399566209, + "max": 98.04024797631428, + "iqr": 0.7370430539594963, + "mean": 97.39343239634763, + "cv": 0.005378887755394542 + }, + "tokens_per_second": { + "count": 5, + "median": 658.2291221873035, + "p95": 660.8596876310331, + "min": 652.7931265072062, + "max": 661.2763007530654, + "iqr": 4.966539422240771, + "mean": 657.1436960622284, + "cv": 0.0053744220292577536 + }, + "median_step_ms_by_index": [ + 1.498833007644862, + 1.5624169900547713, + 1.540166005725041, + 1.5582920023007318, + 1.5981249889591709, + 1.6027499950723723, + 1.6306670004269108, + 1.621875009732321, + 1.3145840057404712, + 1.3065410021226853, + 1.3340000004973263, + 1.3150419981684536, + 1.3330000074347481, + 1.4092500059632584, + 1.3472080026986077, + 1.4477920049102977, + 1.5259579959092662, + 1.4216670097084716, + 1.5193750004982576, + 1.4659580047009513, + 1.5067499916767702, + 1.4250410022214055, + 1.4896669890731573, + 1.4602500014007092, + 1.5009169874247164, + 1.451709002139978, + 1.4821660006418824, + 1.4804999955231324, + 1.5129169914871454, + 1.4877919893478975, + 1.504999992903322, + 1.479874990764074, + 1.546707993838936, + 1.4725420041941106, + 1.5463750023627654, + 1.540624987683259, + 1.5332500042859465, + 1.4968330069677904, + 1.5262500091921538, + 1.560582997626625, + 1.5235419996315613, + 1.5251250006258488, + 1.5606669912813231, + 1.5592909912811592, + 1.5774159983266145, + 1.5390000044135377, + 1.6069580015027896, + 1.6120829968713224, + 1.552249988890253, + 1.5102500037755817, + 1.5967090002959594, + 1.6132919990923256, + 1.549125008750707, + 1.5629999979864806, + 1.5880410064710304, + 1.564582998980768, + 1.6237499949056655, + 1.5523749898420647, + 1.5521250024903566, + 1.6035839944379404, + 1.640916001633741, + 1.6282500000670552, + 1.5692919987486675, + 1.636000000871718 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "speedup_cached_vs_uncached_sdpa": 0.8892254614025953, + "speedup_cached_vs_uncached_default": 0.9626261418049957 + }, + { + "batch": 1, + "prompt_len": 16, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_default": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 1.2949160009156913, + "p95": 1.407949993154034, + "min": 1.2646660034079105, + "max": 1.4091249904595315, + "iqr": 0.11325000377837569, + "mean": 1.332391399773769, + "cv": 0.051312822614196105 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.5755000058561563, + "p95": 1.8176581099396572, + "min": 1.4972920034779236, + "max": 2.3599999985890463, + "iqr": 0.052833507652394474, + "mean": 1.607841965077179, + "cv": 0.06502062869792379 + }, + "total_ms": { + "count": 5, + "median": 102.02392099017743, + "p95": 105.3931070084218, + "min": 101.04829094780143, + "max": 106.15049699845258, + "iqr": 0.8176270348485559, + "mean": 102.62643519963603, + "cv": 0.019795831190920137 + }, + "tokens_per_second": { + "count": 5, + "median": 627.3038653960549, + "p95": 632.7397780433218, + "min": 602.9175727828477, + "max": 633.3605388047633, + "iqr": 5.034164605357319, + "mean": 623.812256474684, + "cv": 0.019360877643731497 + }, + "median_step_ms_by_index": [ + 1.2949160009156913, + 1.745915986248292, + 1.5740000089863315, + 1.6329589998349547, + 1.6027919918997213, + 1.5739579976070672, + 1.5646669926354662, + 1.5638750046491623, + 1.5402499993797392, + 1.5603749925503507, + 1.5934999973978847, + 1.5445830067619681, + 1.5474579995498061, + 1.5407080063596368, + 1.6179589874809608, + 1.6194580093724653, + 1.6550410073250532, + 1.5798340027686208, + 1.5838330000406131, + 1.6417500009993091, + 1.5839590050745755, + 1.5729589940747246, + 1.5691250009695068, + 1.5913340030238032, + 1.5883330052020028, + 1.5862080035731196, + 1.6021670016925782, + 1.5729589940747246, + 1.5542500041192397, + 1.5686249971622601, + 1.5528749936493114, + 1.5770839963806793, + 1.5831250057090074, + 1.6014580032788217, + 1.5818329993635416, + 1.5661659999750555, + 1.5603330102749169, + 1.676167012192309, + 1.5887909976299852, + 1.5542499895673245, + 1.5545839996775612, + 1.5499169967370108, + 1.5690000000176951, + 1.5826660091988742, + 1.575832997332327, + 1.567667000927031, + 1.655625004786998, + 1.560375007102266, + 1.570916996570304, + 1.5900419966783375, + 1.5888749912846833, + 1.5842910070205107, + 1.5657079929951578, + 1.5555409918306395, + 1.5806669980520383, + 1.6195829957723618, + 1.596417001564987, + 1.5602920029778033, + 1.5863330045249313, + 1.5683749952586368, + 1.5750830061733723, + 1.5558750019408762, + 1.5772920014569536, + 1.5616669988958165 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 1.3114579924149439, + "p95": 1.3830672047333792, + "min": 1.266540988581255, + "max": 1.3854590069968253, + "iqr": 0.10624999413266778, + "mean": 1.3208415970439091, + "cv": 0.04292287619897833 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.46975000097882, + "p95": 1.6422499902546406, + "min": 1.2608750112121925, + "max": 1.8732919998001307, + "iqr": 0.134374997287523, + "mean": 1.4792979907869022, + "cv": 0.06637139693933544 + }, + "total_ms": { + "count": 5, + "median": 94.70683403196745, + "p95": 94.84798863413744, + "min": 93.99753801699262, + "max": 94.87974604417104, + "iqr": 0.44296099804341793, + "mean": 94.51661501661874, + "cv": 0.0038736316559239547 + }, + "tokens_per_second": { + "count": 5, + "median": 675.7696068522085, + "p95": 680.4638043321881, + "min": 674.5380617925026, + "max": 680.8688966771689, + "iqr": 3.1746000954310603, + "mean": 677.137767026196, + "cv": 0.0038805632200569003 + }, + "median_step_ms_by_index": [ + 1.3114579924149439, + 1.3809590018354356, + 1.4266250072978437, + 1.3885000080335885, + 1.382542002829723, + 1.4077499945415184, + 1.3941669894848019, + 1.3444580108625814, + 1.4004160038894042, + 1.4156249962979928, + 1.3591669994639233, + 1.37779199576471, + 1.4092079945839942, + 1.4205000043148175, + 1.400666995323263, + 1.4405419933609664, + 1.4087919989833608, + 1.4383330126293004, + 1.394083010382019, + 1.4725420041941106, + 1.4699169987579808, + 1.4557920076185837, + 1.4007080026203766, + 1.4485419960692525, + 1.4452500035986304, + 1.4326660020742565, + 1.458458005799912, + 1.3964999961899593, + 1.5067909989738837, + 1.4597080007661134, + 1.49200001033023, + 1.5039580030133948, + 1.4024159900145605, + 1.3889169931644574, + 1.5070830122567713, + 1.5319999947678298, + 1.5075830015121028, + 1.4797920011915267, + 1.5074159891810268, + 1.4761249913135543, + 1.4459170051850379, + 1.463082997361198, + 1.5280840016203001, + 1.527625005110167, + 1.520042002084665, + 1.4502090052701533, + 1.528624998172745, + 1.5647499967599288, + 1.5303749969461933, + 1.602499993168749, + 1.592540997080505, + 1.6214579954976216, + 1.5552920085610822, + 1.626458004466258, + 1.538124997750856, + 1.5659170021535829, + 1.61070900503546, + 1.636624991078861, + 1.6186660068342462, + 1.592375003383495, + 1.5747909928904846, + 1.5927499916870147, + 1.6190000023925677, + 1.6117499908432364 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_default": { + "first_token_ms": { + "count": 5, + "median": 1.4993329969001934, + "p95": 1.545425402582623, + "min": 1.4218330034054816, + "max": 1.5506670024478808, + "iqr": 0.09020899597089738, + "mean": 1.4861084026051685, + "cv": 0.03781738732682617 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.564999998663552, + "p95": 1.7522081005154178, + "min": 1.377374996081926, + "max": 1.8767079891404137, + "iqr": 0.1387080119457096, + "mean": 1.5764157681833095, + "cv": 0.0620979009332524 + }, + "total_ms": { + "count": 5, + "median": 100.66487302538007, + "p95": 101.67963199492078, + "min": 100.11350296554156, + "max": 101.84495600697119, + "iqr": 0.6584949005628005, + "mean": 100.80030179815367, + "cv": 0.00669710764762354 + }, + "tokens_per_second": { + "count": 5, + "median": 635.7729173697367, + "p95": 638.9605786079192, + "min": 628.4061823898206, + "max": 639.274404592839, + "iqr": 4.1569252502092695, + "mean": 634.9414256877333, + "cv": 0.006670562581908183 + }, + "median_step_ms_by_index": [ + 1.4993329969001934, + 1.4937090018065646, + 1.4837920025456697, + 1.5297500067390501, + 1.549374996102415, + 1.4749999972991645, + 1.4137909893179312, + 1.473542011808604, + 1.5051669906824827, + 1.4385829999810085, + 1.47329100582283, + 1.517167009296827, + 1.520375008112751, + 1.4978749968577176, + 1.420541011611931, + 1.5105420025065541, + 1.5362919948529452, + 1.5206250100163743, + 1.5005419991211966, + 1.5207090036710724, + 1.5649169945390895, + 1.5550000098301098, + 1.497707999078557, + 1.5422500000568107, + 1.5589590038871393, + 1.5399590047309175, + 1.504041996668093, + 1.5251669974531978, + 1.5748340083518997, + 1.602874996024184, + 1.5252920129569247, + 1.5310000017052516, + 1.5709159924881533, + 1.5706249978393316, + 1.5881250001257285, + 1.5919590077828616, + 1.5771250036777928, + 1.566709004691802, + 1.5300420054700226, + 1.5148329985095188, + 1.56170800619293, + 1.5875840035732836, + 1.5559169987682253, + 1.5928749926388264, + 1.6191659960895777, + 1.6477079916512594, + 1.5686660044593737, + 1.6187909932341427, + 1.6278749972116202, + 1.709917007246986, + 1.696375009487383, + 1.7187920020660385, + 1.6564999968977645, + 1.7437500064261258, + 1.7089169996324927, + 1.731291995383799, + 1.680999994277954, + 1.7525000002933666, + 1.6732920048525557, + 1.7014170007314533, + 1.737374986987561, + 1.739042010740377, + 1.7243749898625538, + 1.6755840042605996 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "speedup_cached_vs_uncached_sdpa": 0.9282806729324347, + "speedup_cached_vs_uncached_default": 0.9866791243503746 + }, + { + "batch": 1, + "prompt_len": 128, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": false, + "uncached_default": false + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 1.7167909973068163, + "p95": 1.7712750006467104, + "min": 1.69070900301449, + "max": 1.7813750018831342, + "iqr": 0.020749997929669917, + "mean": 1.7259749991353601, + "cv": 0.01980190021616581 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.574082998558879, + "p95": 1.7422082004486583, + "min": 1.5047910128487274, + "max": 2.4849580076988786, + "iqr": 0.03781249688472599, + "mean": 1.599556253899613, + "cv": 0.05395003992871762 + }, + "total_ms": { + "count": 5, + "median": 102.66146196227055, + "p95": 103.29218594997656, + "min": 101.23987702536397, + "max": 103.32383595232386, + "iqr": 1.0662518470780924, + "mean": 102.49801899481099, + "cv": 0.008301739238859124 + }, + "tokens_per_second": { + "count": 5, + "median": 623.4082271643555, + "p95": 631.097679823758, + "min": 619.4117689313351, + "max": 632.1619689834853, + "iqr": 6.478612606864431, + "mean": 624.4368797684017, + "cv": 0.008335573460259456 + }, + "median_step_ms_by_index": [ + 1.7167909973068163, + 1.8342499970458448, + 1.7162500007543713, + 1.5977919974830002, + 1.6058750043157488, + 1.5707500133430585, + 1.5807500021765009, + 1.5790000034030527, + 1.5642920043319464, + 1.5452920051757246, + 1.5731659950688481, + 1.587208011187613, + 1.5824579895706847, + 1.5603749925503507, + 1.5690839936723933, + 1.5703749959357083, + 1.5851659991312772, + 1.6147499991348013, + 1.5719579969299957, + 1.5675000031478703, + 1.67275000421796, + 1.568417006637901, + 1.6792089882073924, + 1.5667910047341138, + 1.5718749928055331, + 1.5536249993601814, + 1.5821670094737783, + 1.58862498938106, + 1.5717089991085231, + 1.588833998539485, + 1.5645840030629188, + 1.5626249951310456, + 1.577749993884936, + 1.5735000051790848, + 1.5720840019639581, + 1.5693750028731301, + 1.5639579942217097, + 1.5816669911146164, + 1.572417007992044, + 1.5669580025132746, + 1.6778340068412945, + 1.5819580003153533, + 1.5716249909019098, + 1.6142499953275546, + 1.5600840124534443, + 1.5687499981140718, + 1.5652080037398264, + 1.6007499943953007, + 1.546332990983501, + 1.5714170003775507, + 1.5643340011592954, + 1.574458001414314, + 1.5723339893156663, + 1.5629580011591315, + 1.5666669933125377, + 1.5549579984508455, + 1.672041995334439, + 1.5854999946895987, + 1.569165993714705, + 1.563249999890104, + 1.5662079968024045, + 1.5865830064285547, + 1.562499994179234, + 1.5962089964887127 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 1.756167010171339, + "p95": 1.7829994088970125, + "min": 1.691583005595021, + "max": 1.7851660086307675, + "iqr": 0.08149900531861931, + "mean": 1.7400166078004986, + "cv": 0.02578016655293914 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.9215829961467534, + "p95": 2.036325201333966, + "min": 1.6784170002210885, + "max": 2.11312499595806, + "iqr": 0.1055624961736612, + "mean": 1.9185896798421347, + "cv": 0.03997469849762095 + }, + "total_ms": { + "count": 5, + "median": 122.66258097952232, + "p95": 123.62418581324164, + "min": 121.51712106424384, + "max": 123.802712012548, + "iqr": 0.7467438990715891, + "mean": 122.61116643785499, + "cv": 0.006955547074875146 + }, + "tokens_per_second": { + "count": 5, + "median": 521.7565086999463, + "p95": 526.1175575419136, + "min": 516.9515187479358, + "max": 526.6747552895398, + "iqr": 3.182901979890744, + "mean": 521.9954827720696, + "cv": 0.006949748447096107 + }, + "median_step_ms_by_index": [ + 1.756167010171339, + 1.8498330027796328, + 1.7361250065732747, + 1.7655830015428364, + 1.8782080005621538, + 1.8526250059949234, + 1.9040829938603565, + 1.8895409884862602, + 1.8886670004576445, + 1.8816670053638518, + 1.8891249928856269, + 1.898457994684577, + 1.832209003623575, + 1.8494999967515469, + 1.8388750031590462, + 1.8469170026946813, + 1.8194169970229268, + 1.8783329869620502, + 1.855834008892998, + 1.8749170121736825, + 1.827040992793627, + 1.8589579995023087, + 1.8522500031394884, + 1.861124997958541, + 1.9088749977527186, + 1.9029999966733158, + 1.908167003421113, + 1.9072919967584312, + 1.8743749969871715, + 1.916708002681844, + 1.8850000051315874, + 1.8955419945996255, + 1.9008340022992343, + 1.968374999705702, + 1.962333990377374, + 1.9192500039935112, + 1.9483750074869022, + 1.9507090037222952, + 1.9215829961467534, + 1.9955000025220215, + 1.9481249910313636, + 1.9736249960260466, + 1.9376669952180237, + 1.9510419951984659, + 1.9662090053316206, + 1.9529160053934902, + 1.9386249914532527, + 1.9900839979527518, + 2.0004590041935444, + 1.985499999136664, + 1.9712080102181062, + 1.993458005017601, + 1.9582499953685328, + 1.9482499919831753, + 1.9751669897232205, + 1.9805420015472919, + 1.9762909942073748, + 1.9722909928532317, + 1.9608329894253984, + 1.9900000042980537, + 1.9732919899979606, + 1.981875000637956, + 1.9737080001505092, + 1.9896250014426187 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_default": { + "first_token_ms": { + "count": 5, + "median": 1.9287499890197068, + "p95": 2.0206665998557582, + "min": 1.8481669976608828, + "max": 2.036749996477738, + "iqr": 0.05725001392420381, + "mean": 1.9338165991939604, + "cv": 0.036262633391052304 + }, + "decode_ms_per_step": { + "count": 315, + "median": 2.0425840048119426, + "p95": 2.20644990040455, + "min": 1.8508339999243617, + "max": 2.3973750066943467, + "iqr": 0.15622900536982343, + "mean": 2.0524719278679213, + "cv": 0.047375227509146925 + }, + "total_ms": { + "count": 5, + "median": 131.0646170459222, + "p95": 132.1740162675269, + "min": 130.2301660325611, + "max": 132.28954207443167, + "iqr": 0.8104109583655372, + "mean": 131.239548054873, + "cv": 0.006011014102079833 + }, + "tokens_per_second": { + "count": 5, + "median": 488.3087551964981, + "p95": 490.93352481418685, + "min": 483.7872971394134, + "max": 491.43759813681146, + "iqr": 3.008261538502495, + "mean": 487.6719703963196, + "cv": 0.006007732734495126 + }, + "median_step_ms_by_index": [ + 1.9287499890197068, + 1.933083010953851, + 1.9370829977560788, + 1.9704580045072362, + 1.9648329907795414, + 1.9934589945478365, + 1.9640829996205866, + 1.9381669990252703, + 1.9719589909072965, + 1.9923750078305602, + 1.9596670026658103, + 1.972125013708137, + 2.009333999012597, + 1.9505420059431344, + 1.9834169943351299, + 1.9487910030875355, + 1.9869169918820262, + 2.028375005465932, + 2.000125008635223, + 1.9844580092467368, + 1.9725000020116568, + 1.990791002754122, + 1.9894579891115427, + 1.9382089958526194, + 1.9720829877769575, + 1.9965839892392978, + 1.9849579985020682, + 2.0008749997941777, + 1.9660420075524598, + 1.9703750003827736, + 1.9623749976744875, + 1.9961669895565137, + 1.9394160044612363, + 2.0185419998597354, + 2.0680839952547103, + 2.0484580018091947, + 2.0653329993365332, + 2.05199999618344, + 2.073333991575055, + 2.057458012131974, + 2.0748330134665594, + 2.1147500083316118, + 2.132957990397699, + 2.0968330063624308, + 2.1827499876962975, + 2.1177500020712614, + 2.121166995493695, + 2.08733300678432, + 2.152291010133922, + 2.136708004400134, + 2.157499999157153, + 2.1326249989215285, + 2.1413750073406845, + 2.1348750015022233, + 2.202667004894465, + 2.18604200927075, + 2.12266699236352, + 2.1597909944830462, + 2.1634580043610185, + 2.064707994577475, + 2.1434160007629544, + 2.1545409981627017, + 2.1362080005928874, + 2.1459580020746216 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "speedup_cached_vs_uncached_sdpa": 1.1948259710601281, + "speedup_cached_vs_uncached_default": 1.2766681336964614 + }, + { + "batch": 1, + "prompt_len": 512, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_default": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 3.324416989926249, + "p95": 3.610741996089928, + "min": 3.275583003414795, + "max": 3.67316699703224, + "iqr": 0.0545009970664978, + "mean": 3.388149995589629, + "cv": 0.047901708321497476 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.7656669951975346, + "p95": 2.2255502015468664, + "min": 1.5785830037202686, + "max": 2.7164170023752376, + "iqr": 0.18170799739891663, + "mean": 1.8447496093964826, + "cv": 0.11038471269703577 + }, + "total_ms": { + "count": 5, + "median": 120.46887294854969, + "p95": 121.01006383891217, + "min": 117.1163310209522, + "max": 121.04796405765228, + "iqr": 2.3132170172175393, + "mean": 119.60737538756803, + "cv": 0.014308821119027216 + }, + "tokens_per_second": { + "count": 5, + "median": 531.2575641620999, + "p95": 545.1478152137233, + "min": 528.7160382930382, + "max": 546.4652063643485, + "iqr": 10.33320733949779, + "mean": 535.1724205404869, + "cv": 0.014423944279559998 + }, + "median_step_ms_by_index": [ + 3.324416989926249, + 2.4519160069758072, + 2.0497080113273114, + 1.813916998798959, + 2.169250001315959, + 1.7138340044766665, + 2.141666001989506, + 1.8530829984229058, + 2.144124999176711, + 1.884792000055313, + 2.054291995591484, + 2.0064999989699572, + 1.777666009729728, + 1.8840000120690092, + 1.8431250064168125, + 1.8705830007093027, + 1.7936670046765357, + 1.92320799396839, + 1.8156670121243224, + 1.776916004018858, + 1.8463750020600855, + 1.770082992152311, + 2.100707992212847, + 1.757707999786362, + 1.7597910045878962, + 1.8162499909522012, + 1.8221670034108683, + 1.7992500070249662, + 1.7765829979907721, + 1.8052079976769164, + 1.7401250079274178, + 2.147249993868172, + 1.7269999952986836, + 1.7524580034660175, + 1.762541985954158, + 1.777917001163587, + 1.7635000112932175, + 1.7500419926363975, + 1.7379999917466193, + 1.7691249959170818, + 1.6735830140532926, + 1.7712500120978802, + 1.7618749989196658, + 1.70974999491591, + 1.7804579983931035, + 1.717666003969498, + 1.6925000090850517, + 1.7155419918708503, + 1.7627920024096966, + 1.7044160049408674, + 1.8318329966859892, + 1.772042000084184, + 1.7190829967148602, + 1.6757080011302605, + 1.6497909964527935, + 1.7281670006923378, + 1.7427499988116324, + 1.7019999941112474, + 1.7368749977322295, + 1.6752920055296272, + 1.791999995475635, + 1.742166990879923, + 1.7256669962080196, + 1.744582987157628 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 3.3001249976223335, + "p95": 3.952491591917351, + "min": 3.201041996362619, + "max": 4.031374992337078, + "iqr": 0.35120800021104515, + "mean": 3.4910499933175743, + "cv": 0.09880808263736486 + }, + "decode_ms_per_step": { + "count": 315, + "median": 3.6137919960310683, + "p95": 4.07847510650754, + "min": 3.2652919908287004, + "max": 4.579207990900613, + "iqr": 0.3777289966819808, + "mean": 3.669419193350237, + "cv": 0.06400143505531597 + }, + "total_ms": { + "count": 5, + "median": 231.12716995819937, + "p95": 246.81093634862918, + "min": 222.6344169175718, + "max": 248.01146292884368, + "iqr": 12.46841398824472, + "mean": 234.6644591743825, + "cv": 0.043435623056600994 + }, + "tokens_per_second": { + "count": 5, + "median": 276.90383615035284, + "p95": 285.73702802306065, + "min": 258.05258855459465, + "max": 287.4667847231157, + "iqr": 14.364840597850787, + "mean": 273.1388742551787, + "cv": 0.043115828973762 + }, + "median_step_ms_by_index": [ + 3.3001249976223335, + 3.4667919971980155, + 3.4725000004982576, + 3.5887089907191694, + 3.3837079972727224, + 3.4284579887753353, + 3.5463749954942614, + 3.5065829870291054, + 3.5595420049503446, + 3.49520800227765, + 3.4701250115176663, + 3.6376249918248504, + 3.5012919979635626, + 3.4524579969001934, + 3.4398749994579703, + 3.536208998411894, + 3.6343329993542284, + 3.564458995242603, + 3.530125002725981, + 3.5317080037202686, + 3.4337090037297457, + 3.597500006435439, + 3.5090000019408762, + 3.6741670046467334, + 3.4509580000303686, + 3.6844160058535635, + 3.7572500004898757, + 3.6887910100631416, + 3.636374996858649, + 3.4763749863486737, + 3.627957994467579, + 3.7107499956618994, + 3.507207991788164, + 3.5597079986473545, + 3.5781670012511313, + 3.856791998259723, + 3.6792920000152662, + 3.5397499887039885, + 3.587250001146458, + 3.8368749956134707, + 3.6326669942354783, + 3.6766669945791364, + 3.6505419993773103, + 3.4877919970313087, + 3.850417007924989, + 3.799624988459982, + 3.7149580020923167, + 3.8655829994240776, + 3.7929579993942752, + 3.7086670054122806, + 3.624582997872494, + 3.7645410047844052, + 3.6966250045225024, + 3.650832994026132, + 3.890291991410777, + 3.620249990490265, + 3.5952909965999424, + 3.78866599930916, + 3.91670799581334, + 3.6154160043224692, + 3.795624987105839, + 3.891957996529527, + 3.5768749949056655, + 3.6071669892407954 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_default": { + "first_token_ms": { + "count": 5, + "median": 3.833499999018386, + "p95": 4.156133794458583, + "min": 3.799041995080188, + "max": 4.16233399300836, + "iqr": 0.3137500025331974, + "mean": 3.948758397018537, + "cv": 0.04597872555972467 + }, + "decode_ms_per_step": { + "count": 315, + "median": 4.136207993724383, + "p95": 4.574554400460329, + "min": 3.7469579983735457, + "max": 5.072625004686415, + "iqr": 0.2630204980960116, + "mean": 4.172735335998281, + "cv": 0.04965634310762 + }, + "total_ms": { + "count": 5, + "median": 266.23612696130294, + "p95": 270.27036558138207, + "min": 262.7188358892454, + "max": 270.43949796643574, + "iqr": 4.42671007476747, + "mean": 266.83108456491027, + "cv": 0.011952538817231703 + }, + "tokens_per_second": { + "count": 5, + "median": 240.38811235149282, + "p95": 243.15659471876566, + "min": 236.65182224211586, + "max": 243.60643873658353, + "iqr": 3.963066986596715, + "mean": 239.87954872771675, + "cv": 0.011960646431876467 + }, + "median_step_ms_by_index": [ + 3.833499999018386, + 3.937415996915661, + 3.9121660083765164, + 3.94754100125283, + 3.945292002754286, + 3.9643750060349703, + 3.9952500083018094, + 4.158333002123982, + 4.001292007160373, + 4.1161660046782345, + 3.976499996497296, + 4.0566669922554865, + 4.028875002404675, + 4.070833005243912, + 3.9877090021036565, + 3.977166998083703, + 3.986832991358824, + 4.030041993246414, + 4.23262499680277, + 4.242832990712486, + 3.9999169966904446, + 4.2505000019446015, + 4.363583997474052, + 4.0585839888080955, + 4.012667006463744, + 4.072374998941086, + 3.9847919979365543, + 4.049916999065317, + 4.0103329956764355, + 4.127708001760766, + 4.248416997143067, + 4.100332997040823, + 4.0193339955294505, + 4.128791988478042, + 4.335667006671429, + 4.0868750074878335, + 4.090041009476408, + 4.118582990486175, + 4.108124994672835, + 4.100874997675419, + 4.1588750027585775, + 4.141874989727512, + 4.167582999798469, + 4.099541998584755, + 4.098749996046536, + 4.207666992442682, + 4.207334000966512, + 4.101250000530854, + 4.167042003246024, + 4.174750007223338, + 4.182459000730887, + 4.242625000188127, + 4.147166997427121, + 4.216500004986301, + 4.46441701205913, + 4.1653749940451235, + 4.220000002533197, + 4.141916011576541, + 4.19879199762363, + 4.345416004071012, + 4.523124996921979, + 4.214582993881777, + 4.203707998385653, + 4.312082994147204 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "speedup_cached_vs_uncached_sdpa": 1.9185633956823855, + "speedup_cached_vs_uncached_default": 2.209999317209584 + }, + { + "batch": 1, + "prompt_len": 1024, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_default": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 7.314207992749289, + "p95": 9.95528319326695, + "min": 6.569249992026016, + "max": 9.959374991012737, + "iqr": 3.0882910068612546, + "mean": 8.12647479469888, + "cv": 0.20734775889246718 + }, + "decode_ms_per_step": { + "count": 315, + "median": 2.2726249881088734, + "p95": 2.963821101002395, + "min": 1.7257500003324822, + "max": 5.819333004183136, + "iqr": 0.4624584980774671, + "mean": 2.298699440674797, + "cv": 0.19835731101493356 + }, + "total_ms": { + "count": 5, + "median": 153.5857879644027, + "p95": 171.17506198119372, + "min": 139.05770698329434, + "max": 174.43799298780505, + "iqr": 18.60546605894342, + "mean": 152.9445395572111, + "cv": 0.0959997407113049 + }, + "tokens_per_second": { + "count": 5, + "median": 416.70522284805145, + "p95": 459.93698428585583, + "min": 366.89254963208754, + "max": 460.2405820461905, + "iqr": 53.97525595825607, + "mean": 421.46165701142155, + "cv": 0.09320164049998357 + }, + "median_step_ms_by_index": [ + 7.314207992749289, + 3.047708989470266, + 2.656083001056686, + 2.582792003522627, + 2.452500004437752, + 2.402750003966503, + 2.710957996896468, + 2.4735420010983944, + 2.457458002027124, + 2.3706670035608113, + 2.387915999861434, + 2.039415994659066, + 2.373666997300461, + 2.4295000039273873, + 2.4579579912824556, + 2.623708001920022, + 2.376374992309138, + 2.548625008785166, + 2.165541998692788, + 2.188124999520369, + 2.4155830033123493, + 2.391707996139303, + 2.4127500073518604, + 2.172709006117657, + 2.3789170081727207, + 2.3984580038813874, + 2.168374994653277, + 2.2028749954188243, + 2.3322079941863194, + 2.3211659863591194, + 2.0663750037783757, + 2.372167000430636, + 1.9801250018645078, + 2.2726249881088734, + 2.026832997216843, + 2.0106659940211102, + 2.059999998891726, + 2.1317499922588468, + 1.9113329908577725, + 2.0007090060971677, + 2.034874996752478, + 2.362625004025176, + 2.0532910129986703, + 2.0750830008182675, + 2.0780830091098323, + 2.2120420035207644, + 1.9187500001862645, + 1.9800419977400452, + 2.392082998994738, + 2.3367910034721717, + 2.0088340097572654, + 2.1010000054957345, + 1.8937089917017147, + 2.1775420027552173, + 2.086958003928885, + 2.042499996605329, + 2.3442499950760975, + 2.1470409992616624, + 2.340125007322058, + 1.899583003250882, + 2.046041001449339, + 2.017167003941722, + 2.0378339977469295, + 2.0671250094892457 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 7.0384589926106855, + "p95": 9.426891795010306, + "min": 6.372333999024704, + "max": 9.428916993783787, + "iqr": 2.952998998807743, + "mean": 7.7448585972888395, + "cv": 0.20062092604728193 + }, + "decode_ms_per_step": { + "count": 315, + "median": 8.061208005528897, + "p95": 12.382720501045696, + "min": 6.36800000211224, + "max": 16.052292005042545, + "iqr": 2.392250498814974, + "mean": 8.695082574489614, + "cv": 0.22243497165151915 + }, + "total_ms": { + "count": 5, + "median": 505.14433592616115, + "p95": 711.2592429795768, + "min": 447.0658790669404, + "max": 745.0515429663938, + "iqr": 71.76654007344041, + "mean": 555.5350607901346, + "cv": 0.20772332774714106 + }, + "tokens_per_second": { + "count": 5, + "median": 126.69646168091474, + "p95": 139.9050427186633, + "min": 85.90009725392486, + "max": 143.15563543693548, + "iqr": 15.808927431716299, + "mean": 118.74972212624161, + "cv": 0.18173630678039687 + }, + "median_step_ms_by_index": [ + 7.0384589926106855, + 7.796083009452559, + 7.151416997658089, + 7.912291999673471, + 7.197832994279452, + 7.689542006119154, + 7.303165999474004, + 7.460249995347112, + 7.653291992028244, + 7.280999998329207, + 7.823167004971765, + 7.539708007243462, + 7.988333003595471, + 7.451166995451786, + 8.273667001049034, + 7.576374991913326, + 8.411875009187497, + 8.046999995713122, + 7.992833998287097, + 7.576916992547922, + 9.031749999849126, + 8.222583011956885, + 8.829041995340958, + 9.141208007349633, + 8.844500000122935, + 9.10541600023862, + 8.848958008456975, + 9.30766599776689, + 9.124042000621557, + 9.549042006256059, + 8.76858300762251, + 8.672832991578616, + 8.30783300625626, + 8.296667001559399, + 7.867457999964245, + 8.231500003603287, + 7.89249999797903, + 8.184666992747225, + 7.792457996401936, + 8.060541993472725, + 7.914083995274268, + 8.095875004073605, + 8.007999989786185, + 8.09491699328646, + 8.078333004959859, + 8.05529199715238, + 8.121208011289127, + 8.02220799960196, + 8.061208005528897, + 7.835917000193149, + 8.274332998553291, + 7.958957998198457, + 8.196166003472172, + 7.63941700279247, + 8.433584007434547, + 8.49758301046677, + 7.956417000968941, + 7.909624997409992, + 8.127291992423125, + 7.985374992131256, + 8.060667008976452, + 7.879624987253919, + 8.23362500523217, + 7.923375000245869 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_default": { + "first_token_ms": { + "count": 5, + "median": 7.970000006025657, + "p95": 9.310016399831511, + "min": 7.737958003417589, + "max": 9.531957999570295, + "iqr": 0.49116699665319175, + "mean": 8.31864980282262, + "cv": 0.08693187631909625 + }, + "decode_ms_per_step": { + "count": 315, + "median": 9.215957994456403, + "p95": 11.392716795671731, + "min": 7.480792002752423, + "max": 13.379999989410862, + "iqr": 1.0862080016522668, + "mean": 9.366209790182655, + "cv": 0.10320819543918683 + }, + "total_ms": { + "count": 5, + "median": 591.2270820554113, + "p95": 651.3188181328587, + "min": 552.7427079359768, + "max": 659.6503759210464, + "iqr": 47.65600695100147, + "mean": 598.3898665843299, + "cv": 0.07020363235236746 + }, + "tokens_per_second": { + "count": 5, + "median": 108.24943907762628, + "p95": 115.07188025377161, + "min": 97.0210922879246, + "max": 115.78624029068693, + "iqr": 8.6533273219856, + "mean": 107.36646490929459, + "cv": 0.0685509271744509 + }, + "median_step_ms_by_index": [ + 7.970000006025657, + 8.496209004078992, + 8.325667004100978, + 8.526666002580896, + 8.583791990531608, + 8.51791599416174, + 8.658958991873078, + 8.74820799799636, + 8.749041997361928, + 8.732000002055429, + 8.75383299717214, + 8.775958005571738, + 8.629042000393383, + 8.806750003714114, + 8.790084000793286, + 8.675832999870181, + 8.786000005784445, + 8.901542008970864, + 9.449250006582588, + 9.00033299694769, + 9.3886249960633, + 9.77999999304302, + 9.530333001748659, + 9.25645801180508, + 9.111333987675607, + 8.875375002389774, + 9.06145799672231, + 9.027749998494983, + 8.939749997807667, + 9.56433299870696, + 9.354792011436075, + 9.260125007131137, + 9.277000004658476, + 9.7065419977298, + 9.657125003286637, + 9.455291001359, + 9.617208008421585, + 9.153416991466656, + 9.105375007493421, + 9.476124992943369, + 9.609500004444271, + 9.371124993776903, + 9.524041000986472, + 9.198667001328431, + 9.615208007744513, + 9.512832999462262, + 9.069249994354323, + 9.736208012327552, + 9.269707996281795, + 9.502874992904253, + 9.185959002934396, + 9.79854199977126, + 9.746083000209183, + 9.421250011655502, + 9.915083996020257, + 9.312750000390224, + 9.399207992828451, + 9.125208991463296, + 9.301042009610683, + 9.334540998679586, + 9.79162500880193, + 9.494582991464995, + 9.385958008351736, + 10.093542005051859 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "speedup_cached_vs_uncached_sdpa": 3.2890044230084676, + "speedup_cached_vs_uncached_default": 3.849490827839115 + }, + { + "batch": 8, + "prompt_len": 1, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": false, + "uncached_default": false + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 1.6744169988669455, + "p95": 1.8481581937521696, + "min": 1.624791999347508, + "max": 1.8854999943869188, + "iqr": 0.04195798828732222, + "mean": 1.708066597348079, + "cv": 0.06017726293567603 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.8580420000944287, + "p95": 2.4654669003211893, + "min": 1.6593330074101686, + "max": 3.8860000058775768, + "iqr": 0.1433334982721135, + "mean": 1.9279320353812108, + "cv": 0.14345569216074322 + }, + "total_ms": { + "count": 5, + "median": 118.40496097283904, + "p95": 136.44733660912607, + "min": 118.10295804752968, + "max": 140.3785870061256, + "iqr": 2.492251936928369, + "mean": 123.16778482636437, + "cv": 0.07860248346629253 + }, + "tokens_per_second": { + "count": 5, + "median": 4324.143142257763, + "p95": 4334.2682052476475, + "min": 3647.2799087061494, + "max": 4335.200476468585, + "iqr": 89.40180380691072, + "mean": 4175.659992870676, + "cv": 0.07134031814710333 + }, + "median_step_ms_by_index": [ + 1.6744169988669455, + 1.8395410006633028, + 1.784541003871709, + 1.8317500071134418, + 1.8299999937880784, + 1.7869169969344512, + 1.8300000083399937, + 1.8155420111725107, + 1.8167920061387122, + 1.8649589910637587, + 1.8002500000875443, + 1.829125001677312, + 1.7674580012680963, + 1.7911249888129532, + 1.8234169983770698, + 1.7912090115714818, + 1.780541002517566, + 1.727625000057742, + 1.8240839999634773, + 1.837167001212947, + 1.8090829980792478, + 1.8097909924108535, + 1.8354590138187632, + 1.8290000007255003, + 1.8488329951651394, + 1.8549579981481656, + 1.8395410006633028, + 1.8838329997379333, + 1.8419159896438941, + 1.836416995502077, + 1.8305419944226742, + 1.8285410042153671, + 1.835458999266848, + 1.8280829972354695, + 1.8189170077675954, + 1.8582080083433539, + 1.860000003944151, + 1.8991669930983335, + 1.8689580028876662, + 1.8704590038396418, + 1.8950830126414075, + 1.8675000028451905, + 1.8703340028878301, + 1.8529579974710941, + 1.9073750008828938, + 1.8065420008497313, + 1.8631669954629615, + 1.8193329888163134, + 1.8580420000944287, + 1.9054169970331714, + 1.9194580090697855, + 1.8400840053800493, + 1.9520829955581576, + 1.9598749931901693, + 1.8572499975562096, + 1.9755420071305707, + 1.9574579928303137, + 1.9140419899486005, + 1.979958004085347, + 1.9584169931476936, + 1.9765000033657998, + 1.9507499964674935, + 1.9844170019496232, + 1.960333000170067 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 1.607834012247622, + "p95": 2.0871496060863137, + "min": 1.5035830001579598, + "max": 2.2042080090614036, + "iqr": 0.07437399472109973, + "mean": 1.6958166030235589, + "cv": 0.16987529422528522 + }, + "decode_ms_per_step": { + "count": 315, + "median": 2.4100840091705322, + "p95": 3.290387302695308, + "min": 1.575625006807968, + "max": 5.1799590000882745, + "iqr": 0.8357919941772707, + "mean": 2.435090069960256, + "cv": 0.2319335871431328 + }, + "total_ms": { + "count": 5, + "median": 151.04358100506943, + "p95": 163.05574502330273, + "min": 149.74853601597715, + "max": 163.8117130205501, + "iqr": 9.135121057624929, + "mean": 155.10649101051968, + "cv": 0.04115317291439044 + }, + "tokens_per_second": { + "count": 5, + "median": 3389.7501409398915, + "p95": 3413.8618186804974, + "min": 3125.53962448198, + "max": 3419.065144953224, + "iqr": 193.68584731486135, + "mean": 3305.353218047884, + "cv": 0.0403951546254738 + }, + "median_step_ms_by_index": [ + 1.607834012247622, + 1.6980410000542179, + 1.7383749946020544, + 1.7114170041168109, + 1.804416999220848, + 1.865541998995468, + 1.9687079911818728, + 1.980333006940782, + 1.7046659922925755, + 1.6974169993773103, + 1.6785839980002493, + 1.6685410082573071, + 1.7019170045387, + 1.703166009974666, + 1.7592909862287343, + 1.7901660030474886, + 1.9533749873517081, + 1.9406249921303242, + 1.9772920059040189, + 1.9473749998724088, + 1.9896250014426187, + 2.0137919927947223, + 2.012166994973086, + 2.0644579926738515, + 2.127041996573098, + 2.2052910062484443, + 2.213832995039411, + 2.214792009908706, + 2.186583005823195, + 2.1966659987811, + 2.1993340051267296, + 2.1717499912483618, + 2.495542008546181, + 2.490375001798384, + 2.4100840091705322, + 2.4599580065114424, + 2.4943339958554134, + 2.483959004166536, + 2.4766249989625067, + 2.612291995319538, + 2.6372909924248233, + 2.6339589967392385, + 2.642624996951781, + 2.666333006345667, + 2.6579590048640966, + 2.7363330009393394, + 2.833459002431482, + 2.6890000008279458, + 2.817041997332126, + 2.8617919888347387, + 2.8157920023659244, + 2.8579170029843226, + 3.0340000084834173, + 2.92041600914672, + 2.9394589946605265, + 2.9550000035669655, + 3.110499994363636, + 3.195875004166737, + 3.074499996728264, + 3.11912500183098, + 3.057083988096565, + 3.0610829999204725, + 3.189999988535419, + 3.1862500036368147 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_default": { + "first_token_ms": { + "count": 5, + "median": 1.7701670003589243, + "p95": 2.2073496045777574, + "min": 1.7268340016016737, + "max": 2.3024160036584362, + "iqr": 0.09366701124235988, + "mean": 1.8719836021773517, + "cv": 0.1302827354409208 + }, + "decode_ms_per_step": { + "count": 315, + "median": 2.600292005809024, + "p95": 3.4630874943104573, + "min": 1.7254580016015097, + "max": 3.897125003277324, + "iqr": 1.009770501696039, + "mean": 2.617394173654565, + "cv": 0.21078020966630237 + }, + "total_ms": { + "count": 5, + "median": 166.5133268397767, + "p95": 167.82720751361921, + "min": 166.115421030554, + "max": 168.08620688971132, + "iqr": 0.45829206646885723, + "mean": 166.76781654241495, + "cv": 0.004662360509546071 + }, + "tokens_per_second": { + "count": 5, + "median": 3074.8289624449053, + "p95": 3081.388221463147, + "min": 3046.056005868141, + "max": 3082.1942768686517, + "iqr": 8.457868615132611, + "mean": 3070.1898752497646, + "cv": 0.00464038124042387 + }, + "median_step_ms_by_index": [ + 1.7701670003589243, + 1.9198749942006543, + 1.9071670103585348, + 1.8686249968595803, + 1.9625830027507618, + 2.025082998443395, + 2.1857500105397776, + 2.109499997459352, + 1.7634579999139532, + 1.8415419908706099, + 1.8341660033911467, + 1.9041670020669699, + 1.9684170110849664, + 1.976333005586639, + 1.9631250033853576, + 2.0261250028852373, + 2.142625002306886, + 2.078958001220599, + 2.1105839987285435, + 2.0831250003539026, + 2.095207993988879, + 2.1555839921347797, + 2.2147499985294417, + 2.336208999622613, + 2.2901670017745346, + 2.3563750000903383, + 2.3508750018663704, + 2.380124991759658, + 2.4595830036560073, + 2.47445899003651, + 2.537958003813401, + 2.4795419885776937, + 2.6818750047823414, + 2.705834005610086, + 2.6217919948976487, + 2.742750002653338, + 2.7362079999875277, + 2.782416995614767, + 2.7758329961216077, + 2.7755829942179844, + 2.8387909987941384, + 2.8963750082766637, + 3.036541005712934, + 2.9280000017024577, + 2.874792000511661, + 2.9847919940948486, + 3.0352499888977036, + 3.0641250050393865, + 3.233208990423009, + 3.109708006377332, + 3.100832997006364, + 3.1420419982168823, + 3.2077500072773546, + 3.4178749920101836, + 3.260915997088887, + 3.2323749910574406, + 3.34287500299979, + 3.4146669931942597, + 3.5292499960632995, + 3.3892500068759546, + 3.4272909979335964, + 3.3926250034710392, + 3.5449169954517856, + 3.5982080007670447 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "speedup_cached_vs_uncached_sdpa": 1.275652470674074, + "speedup_cached_vs_uncached_default": 1.4063036334936445 + }, + { + "batch": 8, + "prompt_len": 16, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_default": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 1.936000000569038, + "p95": 2.3718993994407356, + "min": 1.8540000019129366, + "max": 2.4779160012258217, + "iqr": 0.05545798921957612, + "mean": 2.0216247998178005, + "cv": 0.12750649158836722 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.9697089883266017, + "p95": 2.4713000093470328, + "min": 1.7051249888027087, + "max": 2.699959004530683, + "iqr": 0.20041700190631673, + "mean": 2.0208831708822843, + "cv": 0.09821653527690868 + }, + "total_ms": { + "count": 5, + "median": 125.66112294734921, + "p95": 138.7193363829283, + "min": 122.05820897361264, + "max": 139.1842859738972, + "iqr": 13.936371105955914, + "mean": 129.3372645654017, + "cv": 0.062477368702034033 + }, + "tokens_per_second": { + "count": 5, + "median": 4074.450299274526, + "p95": 4188.81662396235, + "min": 3678.5761870849497, + "max": 4194.719915238865, + "iqr": 424.1415832219991, + "mean": 3970.8023472177847, + "cv": 0.061281439885843746 + }, + "median_step_ms_by_index": [ + 1.936000000569038, + 2.4781250103842467, + 1.9092500006081536, + 2.0038749935338274, + 2.1813749917782843, + 2.015083999140188, + 1.920333001180552, + 1.9678339886013418, + 1.918082998599857, + 1.8807499873219058, + 1.886084006400779, + 2.2036250011296943, + 2.11291600135155, + 2.154957997845486, + 1.9095409952569753, + 1.982832996873185, + 1.9103329977951944, + 1.8100409943144768, + 1.8765839922707528, + 1.89479099935852, + 1.847999999881722, + 1.965458010090515, + 1.986832998227328, + 1.9213750056223944, + 1.8971660028910264, + 1.8781249964376912, + 1.9074999872827902, + 1.8882919976022094, + 1.8612920102896169, + 1.8860830023186281, + 1.905709010316059, + 1.9377920107217506, + 1.939958005095832, + 1.8869999912567437, + 1.8989579984918237, + 1.9757090049097314, + 1.9517909968271852, + 1.9509160047164187, + 1.9693330104928464, + 1.9745000026887283, + 1.9954579911427572, + 2.00191700423602, + 2.0012090099044144, + 2.0061250106664374, + 1.9614170014392585, + 1.9767909980146214, + 2.0326249941717833, + 2.053290998446755, + 1.9748750055441633, + 2.0399170025484636, + 2.026084010140039, + 1.9902909989468753, + 1.9507499964674935, + 2.0090420002816245, + 1.9876660080626607, + 2.00995898921974, + 1.9697089883266017, + 2.029124996624887, + 1.967042000615038, + 1.9754579989239573, + 2.0845830003963783, + 2.0518330129561946, + 1.99966698710341, + 2.0284589991206303 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 1.8608750106068328, + "p95": 2.3499002039898187, + "min": 1.81124999653548, + "max": 2.4284170067403466, + "iqr": 0.21020800340920687, + "mean": 1.9923999992897734, + "cv": 0.1303812896248139 + }, + "decode_ms_per_step": { + "count": 315, + "median": 2.956166004878469, + "p95": 3.9007330968161114, + "min": 1.7490830068709329, + "max": 4.436292001628317, + "iqr": 0.894458506081719, + "mean": 2.9449764697346836, + "cv": 0.203857692134249 + }, + "total_ms": { + "count": 5, + "median": 184.09342202357948, + "p95": 196.24409157549962, + "min": 181.03016704844777, + "max": 197.23487399460282, + "iqr": 9.29079890192952, + "mean": 187.52591759257484, + "cv": 0.036889865387914715 + }, + "tokens_per_second": { + "count": 5, + "median": 2781.196603181296, + "p95": 2822.199241540328, + "min": 2595.889812133378, + "max": 2828.257899485765, + "iqr": 135.1944897052549, + "mean": 2733.215808922468, + "cv": 0.03628780111689907 + }, + "median_step_ms_by_index": [ + 1.8608750106068328, + 1.9777909910771996, + 1.9320829887874424, + 1.9896250014426187, + 2.048374997684732, + 2.079375000903383, + 2.259540997329168, + 2.2833339899079874, + 2.0856669871136546, + 2.1320829982869327, + 2.099584002280608, + 2.170792009565048, + 2.293666999321431, + 2.334459000849165, + 2.2754590027034283, + 2.5595829938538373, + 2.2387499921023846, + 2.544292001402937, + 2.386666994425468, + 2.592290999018587, + 2.7425000007497147, + 2.5705000007292256, + 2.7796670037787408, + 2.6270840025972575, + 2.5588749995222315, + 2.781834002234973, + 2.689834000193514, + 2.7713339950423688, + 2.889625000534579, + 2.911874995334074, + 2.7685420063789934, + 2.7821249968837947, + 2.8467920055845752, + 2.9835830064257607, + 2.9842910007573664, + 2.973916009068489, + 2.988624997669831, + 3.038708004169166, + 3.0965420010033995, + 3.1441669998457655, + 3.1614590116078034, + 3.2552910124650225, + 3.228000001399778, + 3.1961670028977096, + 3.2193749939324334, + 3.2618749974062666, + 3.3037500106729567, + 3.279790995293297, + 3.2332919945474714, + 3.544332997989841, + 3.489584007184021, + 3.4905409993370995, + 3.48324999504257, + 3.661540991743095, + 3.598333991249092, + 3.6990410008002073, + 3.646458004368469, + 3.7053749983897433, + 3.89783299760893, + 3.8192909996723756, + 3.776375000597909, + 3.8564999995287508, + 3.980541994678788, + 3.8446670077973977 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_default": { + "first_token_ms": { + "count": 5, + "median": 2.1522500028368086, + "p95": 2.6772995915962383, + "min": 1.9540840003173798, + "max": 2.6866659900406376, + "iqr": 0.5485839938046411, + "mean": 2.3048167990054935, + "cv": 0.14551705642652768 + }, + "decode_ms_per_step": { + "count": 315, + "median": 3.235291995224543, + "p95": 4.426049999892712, + "min": 2.0735410071210936, + "max": 6.152500005555339, + "iqr": 1.1174380051670596, + "mean": 3.3081211682800795, + "cv": 0.22009344627179434 + }, + "total_ms": { + "count": 5, + "median": 213.8272059673909, + "p95": 219.5431015745271, + "min": 200.16000504256226, + "max": 219.65271097724326, + "iqr": 18.266997911268845, + "mean": 210.7164504006505, + "cv": 0.045576436871352925 + }, + "tokens_per_second": { + "count": 5, + "median": 2394.4567656095223, + "p95": 2556.2273771244895, + "min": 2330.952337087453, + "max": 2557.95357264868, + "iqr": 212.53984135292148, + "mean": 2433.8936048096375, + "cv": 0.04607244304480863 + }, + "median_step_ms_by_index": [ + 2.1522500028368086, + 2.2821250022388995, + 2.2341250005410984, + 2.1522919996641576, + 2.2410830133594573, + 2.3287500080186874, + 2.3254589905263856, + 2.405249993898906, + 2.5748750049388036, + 2.488707992597483, + 2.379750003456138, + 2.6382500072941184, + 2.5021250039571896, + 2.583875000709668, + 2.642790990648791, + 2.5527499965392053, + 2.5998339988291264, + 2.80866700632032, + 2.7874590014107525, + 2.845374998287298, + 2.8478329913923517, + 2.838957996573299, + 2.9108330054441467, + 2.9704579937970266, + 2.966875006677583, + 3.042333002667874, + 3.015874986886047, + 3.174083001795225, + 3.0801670072833076, + 3.1759999983478338, + 3.1317920074798167, + 3.2167080062208697, + 3.1765409949002787, + 3.370292004547082, + 3.2814170117489994, + 3.310749991214834, + 3.367583005456254, + 3.4608749992912635, + 3.549749992089346, + 3.5357500019017607, + 3.4880829916801304, + 3.726499999174848, + 3.524083993397653, + 3.6521660076687112, + 3.65616699855309, + 3.7697919906349853, + 3.8660419959342107, + 3.7046669895062223, + 3.733082994585857, + 4.016582999611273, + 4.251999998814426, + 4.054665987496264, + 3.929832993890159, + 3.9798750076442957, + 4.279999993741512, + 4.209958002320491, + 4.0941250044852495, + 4.251708000083454, + 4.291457997169346, + 4.348250004113652, + 4.22216700098943, + 4.331000003730878, + 4.400749996420927, + 4.418249998707324 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "speedup_cached_vs_uncached_sdpa": 1.4649990204266503, + "speedup_cached_vs_uncached_default": 1.7016178190368585 + }, + { + "batch": 8, + "prompt_len": 128, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_default": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 6.7485420004231855, + "p95": 8.129391999682412, + "min": 5.438583000795916, + "max": 8.130166999762878, + "iqr": 2.4853759969118983, + "mean": 6.816900000558235, + "cv": 0.19023006534741208 + }, + "decode_ms_per_step": { + "count": 315, + "median": 2.968291999422945, + "p95": 3.6323124077171087, + "min": 2.190208004321903, + "max": 5.70870800584089, + "iqr": 0.7290005014510825, + "mean": 3.018058612453012, + "cv": 0.15246520771043173 + }, + "total_ms": { + "count": 5, + "median": 194.35195898404345, + "p95": 225.9459771827096, + "min": 166.63212391722482, + "max": 226.00921097910032, + "iqr": 53.606414949172176, + "mean": 196.954592585098, + "cv": 0.14393779642414697 + }, + "tokens_per_second": { + "count": 5, + "median": 2634.395879909993, + "p95": 3053.1586024648614, + "min": 2265.3943960157712, + "max": 3072.6368239436115, + "iqr": 706.6777736955455, + "mean": 2643.2481518547106, + "cv": 0.14379164423000446 + }, + "median_step_ms_by_index": [ + 6.7485420004231855, + 3.3550419902894646, + 3.1352499936474487, + 3.184250002959743, + 3.001791992573999, + 3.079667003476061, + 3.1342919974122196, + 2.892999997129664, + 2.9639170097652823, + 2.81837499642279, + 2.81558399728965, + 2.973833994474262, + 2.7417079982114956, + 3.1018750014482066, + 2.6577080134302378, + 2.9115409997757524, + 3.052291998756118, + 2.801416994770989, + 2.778209003736265, + 2.7879999979631975, + 2.6482090033823624, + 2.766125006019138, + 2.8899169992655516, + 2.9550420003943145, + 2.913000003900379, + 2.966000000014901, + 2.775874992948957, + 2.8460830071708187, + 2.9066249990137294, + 2.9466249980032444, + 2.777415997115895, + 2.9607920005219057, + 2.8855000127805397, + 2.968291999422945, + 2.743874996667728, + 2.854209000361152, + 2.78658299066592, + 2.996208000695333, + 2.9468340071616694, + 2.8956250025657937, + 2.9989580070832744, + 2.9687500064028427, + 2.944084000773728, + 3.0331670132, + 3.152250006678514, + 2.878792001865804, + 3.0359999946085736, + 2.9937079962110147, + 2.987083003972657, + 2.762417003395967, + 2.9835830064257607, + 2.9860829963581637, + 3.06024999008514, + 3.001624994794838, + 3.007291001267731, + 3.071541999815963, + 2.9211250075604767, + 3.078875000937842, + 3.038582988665439, + 2.845249997335486, + 2.963457998703234, + 3.2293340045725927, + 3.228957997635007, + 3.5496250056894496 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 6.926250003743917, + "p95": 7.9707581957336515, + "min": 5.7031669857678935, + "max": 8.154249997460283, + "iqr": 1.3527069968404248, + "mean": 6.7809083935571834, + "cv": 0.14895564113944382 + }, + "decode_ms_per_step": { + "count": 315, + "median": 7.535833996371366, + "p95": 10.91698750096839, + "min": 5.471291995490901, + "max": 11.768084004870616, + "iqr": 2.274499995110091, + "mean": 7.932526191016511, + "cv": 0.18904448815723193 + }, + "total_ms": { + "count": 5, + "median": 461.3755031459732, + "p95": 631.2001669604797, + "min": 424.34162605786696, + "max": 649.5557499438291, + "iqr": 118.1782570638461, + "mean": 506.5300584275974, + "cv": 0.18821502509984817 + }, + "tokens_per_second": { + "count": 5, + "median": 1109.7251512246194, + "p95": 1198.199263906639, + "min": 788.2310333551441, + "max": 1206.5750059839268, + "iqr": 246.76810295939129, + "mean": 1037.4311357598551, + "cv": 0.17135964413342897 + }, + "median_step_ms_by_index": [ + 6.926250003743917, + 6.9915420026518404, + 6.457666007918306, + 6.96054199943319, + 6.626290996791795, + 7.01100000878796, + 6.697415999951772, + 6.625583991990425, + 7.006875006482005, + 6.64425001014024, + 7.222875006846152, + 6.646375011769123, + 6.945000000996515, + 6.737415998941287, + 6.563290997291915, + 6.841583002824336, + 6.4910000073723495, + 7.175334001658484, + 6.652791998931207, + 6.628792005358264, + 6.886666000355035, + 6.792708998546004, + 6.9565000012516975, + 7.018125004833564, + 6.909375006216578, + 7.066459002089687, + 7.189084004494362, + 7.114665990229696, + 7.130875004804693, + 7.006042011198588, + 7.1804999897722155, + 6.960166996577755, + 7.174084006692283, + 7.107832992915064, + 7.696667002164759, + 7.226000001537614, + 7.395583001198247, + 7.277458003954962, + 7.267499997396953, + 7.26612500147894, + 7.316083007026464, + 7.502666994696483, + 7.5010419968748465, + 7.644999990588985, + 7.461583008989692, + 7.593875008751638, + 7.508833004976623, + 7.880292003392242, + 7.6485830068122596, + 7.628708000993356, + 7.769667005049996, + 7.580792007502168, + 7.515167002566159, + 7.69233399478253, + 7.673583007999696, + 7.696875007241033, + 7.536750010331161, + 7.78895900293719, + 7.714291001320817, + 7.900582990259863, + 7.777666003676131, + 7.884999999077991, + 7.802583000739105, + 7.865166000556201 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_default": { + "first_token_ms": { + "count": 5, + "median": 6.559790999745019, + "p95": 8.462574996519834, + "min": 5.630125000607222, + "max": 8.587874996010214, + "iqr": 1.5904579922789708, + "mean": 7.022016600240022, + "cv": 0.1730909703780754 + }, + "decode_ms_per_step": { + "count": 315, + "median": 8.136749995173886, + "p95": 11.886874999618158, + "min": 5.775583005743101, + "max": 13.655874994583428, + "iqr": 1.8102709946106188, + "mean": 8.519505431953197, + "cv": 0.1881637778828045 + }, + "total_ms": { + "count": 5, + "median": 516.8952120293397, + "p95": 625.6836900283815, + "min": 465.1905420323601, + "max": 633.7273740355158, + "iqr": 84.07674203044735, + "mean": 543.7508588132914, + "cv": 0.12555109171906595 + }, + "tokens_per_second": { + "count": 5, + "median": 990.5295852710242, + "p95": 1081.5075127592856, + "min": 807.9183904265213, + "max": 1100.6242684194206, + "iqr": 142.37448221866316, + "mean": 953.3557484271593, + "cv": 0.12312925692025384 + }, + "median_step_ms_by_index": [ + 6.559790999745019, + 6.797667010687292, + 6.6405000106897205, + 6.261416987399571, + 6.725833998643793, + 6.627249997109175, + 6.6867920104414225, + 6.80612500582356, + 6.845125011750497, + 7.782332992064767, + 7.8059590014163405, + 8.095875004073605, + 7.779541003401391, + 7.987416000105441, + 7.713665996561758, + 7.304958999156952, + 7.126166994567029, + 6.986958993365988, + 7.240666993311606, + 7.36354201217182, + 7.681041010073386, + 7.229541995911859, + 7.465208997018635, + 7.417874992825091, + 7.403083000099286, + 7.65625000349246, + 7.463000001735054, + 7.765208996715955, + 7.675333006773144, + 7.6724590035155416, + 7.629250001627952, + 7.838749996153638, + 7.694458996411413, + 8.27420799760148, + 9.005291998619214, + 9.000457997899503, + 8.336458005942404, + 7.936334004625678, + 9.142708993749693, + 9.132332997978665, + 8.847666991641745, + 8.674291995703243, + 8.863457987899892, + 8.586791998823173, + 9.97608300531283, + 10.231708001811057, + 10.62058299430646, + 10.203666999586858, + 10.411916999146342, + 8.802958996966481, + 8.637916995212436, + 8.465208011330105, + 8.263833995442837, + 8.56108299922198, + 8.74183299310971, + 9.303250000812113, + 9.100875002332032, + 9.88020800286904, + 9.055333997821435, + 10.096250000060536, + 9.153041988611221, + 9.841541992500424, + 9.629583000787534, + 9.733250000863336 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "speedup_cached_vs_uncached_sdpa": 2.373917430818656, + "speedup_cached_vs_uncached_default": 2.6595832361626854 + }, + { + "batch": 8, + "prompt_len": 512, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_default": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 26.80370800953824, + "p95": 33.01501679525245, + "min": 26.12424999824725, + "max": 34.3776249937946, + "iqr": 0.8776670001680031, + "mean": 28.31141680071596, + "cv": 0.12114343067157976 + }, + "decode_ms_per_step": { + "count": 315, + "median": 4.17241700051818, + "p95": 6.579883005178999, + "min": 3.1424579938175157, + "max": 8.50654199894052, + "iqr": 0.996083501377143, + "mean": 4.494775117731594, + "cv": 0.22351573918346998 + }, + "total_ms": { + "count": 5, + "median": 314.0922090387903, + "p95": 323.3219862013357, + "min": 295.48920498928055, + "max": 324.54871300433297, + "iqr": 13.549038922064938, + "mean": 311.4822492178064, + "cv": 0.03678600261140848 + }, + "tokens_per_second": { + "count": 5, + "median": 1630.094555884919, + "p95": 1722.0610804080611, + "min": 1577.5751974501418, + "max": 1732.7198129575454, + "iqr": 71.4621001874475, + "mean": 1645.5559533050812, + "cv": 0.0372364264894457 + }, + "median_step_ms_by_index": [ + 26.80370800953824, + 4.201916992315091, + 3.4729170001810417, + 3.438291998463683, + 3.4565419919090346, + 3.5499999939929694, + 3.408999997191131, + 3.384041992831044, + 3.425916002015583, + 3.5555420035962015, + 3.641332994448021, + 3.4096250019501895, + 3.3795420022215694, + 3.589125000871718, + 3.523207997204736, + 3.649291000328958, + 3.56287500471808, + 3.597083006752655, + 3.5629170015454292, + 3.767375004827045, + 3.715084007126279, + 3.6582910106517375, + 3.8327910006046295, + 4.079042002558708, + 4.10458400438074, + 3.8555840001208708, + 4.197542002657428, + 4.049417009809986, + 4.0954169962788, + 3.9697500033071265, + 4.207124991808087, + 4.176457994617522, + 4.111250003916211, + 4.778834001626819, + 4.886374998022802, + 4.702791004092433, + 5.691750004189089, + 5.749249990913086, + 5.809291003970429, + 6.51283300248906, + 6.413291004719213, + 6.154082991997711, + 6.309333009994589, + 5.977250009891577, + 6.012042009388097, + 5.094499996630475, + 5.538042008993216, + 5.454458994790912, + 4.61037500645034, + 4.445375001523644, + 4.5666670048376545, + 4.382291997899301, + 4.5123330055503175, + 4.550917001324706, + 4.5447500015143305, + 4.614958001184277, + 4.436041999724694, + 4.570124991005287, + 4.44216700270772, + 4.6062500041443855, + 4.599791005603038, + 4.434833012055606, + 4.572958001517691, + 4.575375001877546 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 24.271624992252328, + "p95": 24.90179139713291, + "min": 21.961833001114428, + "max": 24.994082996272482, + "iqr": 0.9404579905094579, + "mean": 23.870466600055806, + "cv": 0.04948948110870243 + }, + "decode_ms_per_step": { + "count": 315, + "median": 27.070874988567084, + "p95": 31.389812493580393, + "min": 21.47891699860338, + "max": 34.16149999247864, + "iqr": 3.828811997664161, + "mean": 26.997853565355022, + "cv": 0.09573691413369338 + }, + "total_ms": { + "count": 5, + "median": 1698.2649949059123, + "p95": 1827.210040637874, + "min": 1651.411668091896, + "max": 1853.1005810509669, + "iqr": 26.396795932669193, + "mean": 1724.7352412174223, + "cv": 0.0442619225939337 + }, + "tokens_per_second": { + "count": 5, + "median": 301.4841626811992, + "p95": 308.36307333748533, + "min": 276.2936913600364, + "max": 310.03777549397137, + "iqr": 4.619835717522562, + "mean": 297.30486464815334, + "cv": 0.04255041755157222 + }, + "median_step_ms_by_index": [ + 24.271624992252328, + 24.900249991333112, + 24.57037501153536, + 25.228875005268492, + 24.571750007453375, + 24.67662500566803, + 24.66795899090357, + 24.52329100924544, + 23.28887500334531, + 23.133166003390215, + 23.756292008329183, + 23.52220899774693, + 23.817166991648264, + 23.34354199410882, + 24.114208994433284, + 23.827875003917143, + 23.639625011128373, + 24.185458009014837, + 24.370625003939494, + 24.686958000529557, + 24.794957993435673, + 24.796625002636574, + 25.521124989609234, + 25.625458001741208, + 26.726790994871408, + 25.56650000042282, + 26.29870800592471, + 26.635207992512733, + 28.343958998448215, + 26.308166998205706, + 25.6882089888677, + 26.66316699469462, + 26.187333001871593, + 26.909415988484398, + 27.13870799925644, + 26.581833997624926, + 27.29179200832732, + 27.259915994363837, + 27.991875002044253, + 27.962667008978315, + 27.900333996512927, + 27.733541995985433, + 27.889500008313917, + 27.83800000906922, + 28.250082992599346, + 28.695708999293856, + 28.141250004409812, + 28.464833012549207, + 29.22250000119675, + 28.864624997368082, + 28.78879100899212, + 29.16712500154972, + 29.08041600312572, + 28.872249997220933, + 29.129083006409928, + 29.33912500157021, + 29.305042000487447, + 29.060166998533532, + 28.90758399735205, + 29.310874990187585, + 29.48037500027567, + 29.20937500311993, + 29.0222920011729, + 29.689500006497838 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_default": { + "first_token_ms": { + "count": 5, + "median": 30.16758400190156, + "p95": 32.38535901182331, + "min": 24.250749993370846, + "max": 32.689834013581276, + "iqr": 3.56570900476072, + "mean": 29.175475402735174, + "cv": 0.11372052760272518 + }, + "decode_ms_per_step": { + "count": 315, + "median": 32.28745800151955, + "p95": 35.73159170045983, + "min": 25.198917006491683, + "max": 39.16041599586606, + "iqr": 2.74100000387989, + "mean": 32.06062156167473, + "cv": 0.0776389622910279 + }, + "total_ms": { + "count": 5, + "median": 2062.443454939057, + "p95": 2074.271099237376, + "min": 2016.942418034887, + "max": 2074.6880400402006, + "iqr": 54.30741612508427, + "mean": 2048.994633788243, + "cv": 0.014162006400574675 + }, + "tokens_per_second": { + "count": 5, + "median": 248.2492301904728, + "p95": 253.81553996155205, + "min": 246.78409000231144, + "max": 253.84958708877923, + "iqr": 6.647036537196271, + "mean": 249.91891472993075, + "cv": 0.014222700157349673 + }, + "median_step_ms_by_index": [ + 30.16758400190156, + 30.70929199748207, + 31.09620801114943, + 30.275666998932138, + 31.332624988863245, + 29.780166994896717, + 31.12837500520982, + 29.33600000687875, + 30.736916000023484, + 31.433666998054832, + 30.666417005704716, + 31.012167004519142, + 30.889540998032317, + 31.05054100160487, + 31.917958011035807, + 31.036458996823058, + 31.852750005782582, + 31.379708991153166, + 30.127082995022647, + 30.871416995069012, + 30.361625002115034, + 31.110291994991712, + 31.962958004442044, + 31.631665988243185, + 31.490541994571686, + 32.03608399780933, + 31.941332999849692, + 32.230166005319916, + 31.99266700539738, + 31.499459000770003, + 30.88983400084544, + 31.640667002648115, + 31.51895799965132, + 31.886000011581928, + 32.301625004038215, + 32.15687499323394, + 32.29341600672342, + 33.08154200203717, + 33.16933299356606, + 32.81079200678505, + 33.454707998316735, + 33.74137499486096, + 33.71629200410098, + 32.982749995426275, + 33.956541010411456, + 34.397542010992765, + 33.46845800115261, + 32.663792007952, + 32.906167005421594, + 33.773624993045814, + 33.3067920000758, + 33.05379199446179, + 33.1422909948742, + 33.164791006129235, + 33.91262498917058, + 32.95287500077393, + 33.131875010440126, + 33.716834004735574, + 33.51883300638292, + 33.68283300369512, + 34.33624999888707, + 34.11954200419132, + 33.915083011379465, + 34.34329200536013 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "speedup_cached_vs_uncached_sdpa": 5.40689945829308, + "speedup_cached_vs_uncached_default": 6.566362983821563 + }, + { + "batch": 8, + "prompt_len": 1024, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_default": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 54.32358400139492, + "p95": 62.528166591073386, + "min": 53.42999999993481, + "max": 63.93374998879153, + "iqr": 2.6999580004485324, + "mean": 56.559808398014866, + "cv": 0.07647265826079962 + }, + "decode_ms_per_step": { + "count": 315, + "median": 5.178792009246536, + "p95": 7.609445706475526, + "min": 4.007499999715947, + "max": 10.088166993227787, + "iqr": 0.8313324942719191, + "mean": 5.424421783564391, + "cv": 0.1819327932092771 + }, + "total_ms": { + "count": 5, + "median": 405.1970370201161, + "p95": 415.04264092072845, + "min": 373.2584149984177, + "max": 416.94574990833644, + "iqr": 18.76970805460587, + "mean": 398.29838076257147, + "cv": 0.04346079305540237 + }, + "tokens_per_second": { + "count": 5, + "median": 1263.5827837373392, + "p95": 1360.8320888968194, + "min": 1227.9775009400162, + "max": 1371.7038368771148, + "iqr": 60.68814382380356, + "mean": 1287.4532343363885, + "cv": 0.0443606800237574 + }, + "median_step_ms_by_index": [ + 54.32358400139492, + 6.735167000442743, + 6.697375007206574, + 5.539875011891127, + 5.510124989086762, + 5.465209003887139, + 5.769541006884538, + 5.295791997923516, + 5.492249998496845, + 5.484708002768457, + 5.4507080058101565, + 5.611000000499189, + 5.46254200162366, + 5.335750000085682, + 5.433917001937516, + 5.454832993564196, + 5.147916992427781, + 5.311083004926331, + 5.237457997282036, + 4.977959004463628, + 5.148416996235028, + 5.273291011690162, + 5.295790993841365, + 5.517457990208641, + 5.757833001553081, + 6.103833002271131, + 5.64908298838418, + 5.233541989582591, + 6.061499996576458, + 6.561917005456053, + 6.093165997299366, + 5.649165992508642, + 5.900207994272932, + 4.905959009192884, + 4.958207995514385, + 4.8342919908463955, + 4.82962500245776, + 5.1473749917931855, + 4.908250004518777, + 4.849916993407533, + 5.170291988179088, + 5.200832994887605, + 5.062500000349246, + 5.044874997111037, + 5.4195839911699295, + 4.932791998726316, + 5.3007090027676895, + 5.079291004221886, + 5.01770900154952, + 4.991583002265543, + 5.182374996365979, + 5.04683300096076, + 5.107417004182935, + 5.184166002436541, + 5.314207999617793, + 5.06570799916517, + 5.1920830010203645, + 5.003583995858207, + 5.191040996578522, + 5.211832991335541, + 5.131707992404699, + 5.024375001084991, + 5.4183330066734925, + 5.110917001729831 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 47.1817910001846, + "p95": 53.76165860216133, + "min": 44.308582990197465, + "max": 54.15116700169165, + "iqr": 6.775084009859711, + "mean": 48.65474139805883, + "cv": 0.08856422231258516 + }, + "decode_ms_per_step": { + "count": 315, + "median": 57.806041004369035, + "p95": 67.42135370586766, + "min": 45.25787501188461, + "max": 79.48783300525974, + "iqr": 6.075041492294986, + "mean": 58.718185212799455, + "cv": 0.09612357417718952 + }, + "total_ms": { + "count": 5, + "median": 3712.864042070578, + "p95": 3924.842756174621, + "min": 3585.3117110964376, + "max": 3947.6989549875725, + "iqr": 173.20858097809833, + "mean": 3747.9004098044243, + "cv": 0.038338286955465414 + }, + "tokens_per_second": { + "count": 5, + "median": 137.89893575377715, + "p95": 142.2204436711122, + "min": 129.69580655412764, + "max": 142.80487758299358, + "iqr": 6.320439254765347, + "mean": 136.76891933666127, + "cv": 0.03794238228454239 + }, + "median_step_ms_by_index": [ + 47.1817910001846, + 48.12254098942503, + 50.62995800108183, + 50.16795800474938, + 50.82733400922734, + 50.98458299471531, + 51.2866669887444, + 51.555209007347, + 53.29612499917857, + 52.98550000588875, + 53.24475000088569, + 54.362957991543226, + 53.965792001690716, + 55.780750000849366, + 56.86241699731909, + 56.40483299794141, + 56.064375006826594, + 57.17400000139605, + 58.07329200615641, + 57.258583008660935, + 57.25808299030177, + 57.3274579946883, + 60.162916997796856, + 57.10979200375732, + 57.774041008087806, + 57.30179099191446, + 59.55974999233149, + 61.43533300200943, + 60.70483299845364, + 64.23150000046007, + 66.64270799956284, + 67.31558300089091, + 62.80658299510833, + 60.32604200299829, + 60.021957993740216, + 60.5387499963399, + 59.07308300083969, + 59.86362500698306, + 59.428209002362564, + 60.277167009189725, + 57.63283399573993, + 61.3952919957228, + 61.34250000468455, + 60.400874994229525, + 62.09354099701159, + 60.83237500570249, + 63.5192080080742, + 65.13550000090618, + 60.650667001027614, + 59.608333001961, + 61.47754201083444, + 61.77933300205041, + 62.74891700013541, + 63.39154200395569, + 58.821958999033086, + 56.74512499535922, + 58.0100829974981, + 56.75491700822022, + 58.24154200672638, + 57.626832989626564, + 57.61083299876191, + 58.09845899057109, + 59.25083300098777, + 59.29045799712185 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_default": { + "first_token_ms": { + "count": 5, + "median": 68.47866700263694, + "p95": 73.58466660080012, + "min": 58.544041006825864, + "max": 73.68375000078231, + "iqr": 14.571125007933006, + "mean": 66.50239980081096, + "cv": 0.11295089147217234 + }, + "decode_ms_per_step": { + "count": 315, + "median": 72.98037500004284, + "p95": 81.72996260400397, + "min": 61.223832992254756, + "max": 86.36404099524952, + "iqr": 3.9680835106992163, + "mean": 73.30274709827444, + "cv": 0.05629775736539282 + }, + "total_ms": { + "count": 5, + "median": 4647.7621720550815, + "p95": 4892.949358379701, + "min": 4554.25178699079, + "max": 4948.724707981455, + "iqr": 67.5572520121932, + "mean": 4684.575466992101, + "cv": 0.03291505237707016 + }, + "tokens_per_second": { + "count": 5, + "median": 110.1605420084589, + "p95": 112.18772435343996, + "min": 103.46099858297445, + "max": 112.42241842283005, + "iqr": 1.6094042644800624, + "mean": 109.38649018030851, + "cv": 0.0318199251842066 + }, + "median_step_ms_by_index": [ + 68.47866700263694, + 68.56420799158514, + 68.55529099993873, + 68.58700000157114, + 68.65575000119861, + 69.74812500993721, + 68.43499999376945, + 71.15412499115337, + 69.5127910003066, + 70.13116701273248, + 70.3393750009127, + 72.65387500228826, + 71.07258299947716, + 74.61774999683257, + 72.83616700442508, + 72.1515420009382, + 71.97491699480452, + 72.33791600447148, + 71.22362499649171, + 71.36712501232978, + 71.43162500869948, + 71.50945799367037, + 72.69087500753812, + 71.77187499473803, + 72.06154099549167, + 71.57437498972286, + 70.94179099658504, + 71.60224999825004, + 72.36945899785496, + 71.2209160119528, + 71.32012498914264, + 71.45187500282191, + 71.58804200298619, + 73.3516670006793, + 72.51345900294837, + 73.84187501156703, + 73.26450000982732, + 74.39145899843425, + 72.38733299891464, + 71.64733301033266, + 72.77395800338127, + 74.26270899304654, + 72.01562500267755, + 71.93299999926239, + 72.84916698699817, + 74.4710830040276, + 74.3019999936223, + 74.21433400304522, + 74.30649999878369, + 73.94458299677353, + 74.62795799074229, + 74.1847909957869, + 76.34825000423007, + 75.754457997391, + 75.33279099152423, + 74.6122499986086, + 74.35583400365431, + 74.09358299628366, + 73.97075000335462, + 73.41804199677426, + 74.33595901238732, + 73.74829200853128, + 74.81479199486785, + 73.63537501078099 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "speedup_cached_vs_uncached_sdpa": 9.163107581870722, + "speedup_cached_vs_uncached_default": 11.470375514676684 + } + ], + "window": { + "batch": 1, + "prompt_len": 32, + "context_length": 128, + "new_tokens": 224, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_default": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 1.461209001718089, + "p95": 1.4907500008121133, + "min": 1.3040000048931688, + "max": 1.4924999995855615, + "iqr": 0.11187500786036253, + "mean": 1.4226668019546196, + "cv": 0.05748689911342628 + }, + "decode_ms_per_step": { + "count": 1115, + "median": 1.6982499946607277, + "p95": 1.8692831043154001, + "min": 1.5137089940253645, + "max": 2.5843750045169145, + "iqr": 0.16308299382217228, + "mean": 1.7047319244233172, + "cv": 0.07222467424572206 + }, + "total_ms": { + "count": 5, + "median": 380.10850503633264, + "p95": 386.87931176973507, + "min": 379.0547127719037, + "max": 388.41575395781547, + "iqr": 1.156628059106879, + "mean": 381.5778859483544, + "cv": 0.010149807521840814 + }, + "tokens_per_second": { + "count": 5, + "median": 589.3054141963727, + "p95": 590.7811166663639, + "min": 576.7016340545442, + "max": 590.9437145945526, + "iqr": 1.7927544539757037, + "mean": 587.0838916597424, + "cv": 0.010022227792351807 + }, + "median_step_ms_by_index": [ + 1.461209001718089, + 2.2882090124767274, + 1.6120830114232376, + 1.7218749999301508, + 1.5889580099610612, + 1.5898330020718277, + 1.5771250036777928, + 1.5990420070011169, + 1.5853750082897022, + 1.600624993443489, + 1.6547089908272028, + 1.6077500040410087, + 1.5737500070827082, + 1.5720419905846938, + 1.59891699149739, + 1.5679579955758527, + 1.6485000087413937, + 1.578499999595806, + 1.5777920052642003, + 1.6477079916512594, + 1.5927089989418164, + 1.5892080118646845, + 1.5882500010775402, + 1.5907909983070567, + 1.6202499973587692, + 1.6222499980358407, + 1.5914579998934641, + 1.6031249979278073, + 1.6894580039661378, + 1.5847910108277574, + 1.5660000062780455, + 1.633500010939315, + 1.6237920062849298, + 1.6263330035144463, + 1.644083997234702, + 1.6179999947780743, + 1.582500000949949, + 1.5950829983921722, + 1.566874998388812, + 1.5710830048192292, + 1.5728330035926774, + 1.5933330141706392, + 1.5732909960206598, + 1.5644169907318428, + 1.6254579968517646, + 1.589125007740222, + 1.675666993833147, + 1.5909999929135665, + 1.596249989233911, + 1.6309999919030815, + 1.592417000210844, + 1.6564999968977645, + 1.5763330011395738, + 1.5833329962333664, + 1.5858749975450337, + 1.615625005797483, + 1.5775409992784262, + 1.592375003383495, + 1.6202080005314201, + 1.597374997800216, + 1.5842080028960481, + 1.5983330085873604, + 1.5927909989841282, + 1.5955419949023053, + 1.5828339965082705, + 1.56170800619293, + 1.586124999448657, + 1.6051249986048788, + 1.5943750040605664, + 1.6531250003026798, + 1.625167002202943, + 1.637708002817817, + 1.5959589945850894, + 1.5874579985393211, + 1.5927499916870147, + 1.6405420028604567, + 1.5974589914549142, + 1.6119999927468598, + 1.609333005035296, + 1.6682499990565702, + 1.608916005352512, + 1.5897919947747141, + 1.6382920002797619, + 1.5854160010349005, + 1.6059999907156453, + 1.5988329978426918, + 1.6229169996222481, + 1.6016250010579824, + 1.6090839926619083, + 1.6143750108312815, + 1.597167007275857, + 1.7553749930812046, + 1.7254169943043962, + 1.6323330055456609, + 1.6191250033443794, + 1.6230000037467107, + 1.5940000012051314, + 1.7922080005519092, + 1.7895829951157793, + 1.7225830088136718, + 1.7724579956848174, + 1.709332995233126, + 1.7612919909879565, + 1.720708009088412, + 1.7794589948607609, + 1.7113329959101975, + 1.756959012709558, + 1.737666010740213, + 1.7347079992759973, + 1.7698749870760366, + 1.732708013150841, + 1.687832991592586, + 1.807792010367848, + 1.7717089940560982, + 1.77087499469053, + 1.774082993506454, + 1.767207999364473, + 1.770292001310736, + 1.7019999941112474, + 1.766499990480952, + 1.7353750008624047, + 1.7586250032763928, + 1.7737500020302832, + 1.7871659947559237, + 1.7499579989816993, + 1.7780830094125122, + 1.702166991890408, + 1.7587079928489402, + 1.7100840050261468, + 1.748041991959326, + 1.7790410056477413, + 1.7620419966988266, + 1.780791993951425, + 1.7361670034006238, + 1.777583995135501, + 1.6977080085780472, + 1.7639169964240864, + 1.7222500027855858, + 1.720165993901901, + 1.8383749993517995, + 1.7569170013302937, + 1.7635840049479157, + 1.784209001925774, + 1.767666995874606, + 1.7481249960837886, + 1.7854999896371737, + 1.7351669957861304, + 1.6924169904086739, + 1.7076249932870269, + 1.6850829997565597, + 1.763291991665028, + 1.749500006553717, + 1.741791988024488, + 1.6812920075608417, + 1.7844999965745956, + 1.6941670037340373, + 1.6820000018924475, + 1.7514999926788732, + 1.7071670008590445, + 1.6926660027820617, + 1.7582919972483069, + 1.7669170047156513, + 1.8078749999403954, + 1.7820000066421926, + 1.7849580035544932, + 1.7455419874750078, + 1.7647919885348529, + 1.746041001752019, + 1.7410000000381842, + 1.849666005000472, + 1.850291999289766, + 1.7949579923879355, + 1.7441250092815608, + 1.7648750072112307, + 1.7689170053927228, + 1.7620839935261756, + 1.7385839892085642, + 1.772708012140356, + 1.7254589911317453, + 1.7358750046696514, + 1.8047499906970188, + 1.7582909931661561, + 1.720375003060326, + 1.7761670023901388, + 1.6916659951675683, + 1.767624999047257, + 1.7027080029947683, + 1.7277500010095537, + 1.7574579978827387, + 1.780749997124076, + 1.7567090108059347, + 1.775042008375749, + 1.778166988515295, + 1.7218749999301508, + 1.7583750013727695, + 1.6848330124048516, + 1.734874997055158, + 1.7505419964436442, + 1.7477079964010045, + 1.8042499868897721, + 1.7645420011831447, + 1.787083994713612, + 1.709374992060475, + 1.7335830052616075, + 1.7137910035671666, + 1.7294579884037375, + 1.7781670030672103, + 1.7734160064719617, + 1.7636660049902275, + 1.740916006383486, + 1.6879579925443977, + 1.751415999024175, + 1.7532920028315857, + 1.6767499910201877, + 1.7481670074630529, + 1.7189589998451993, + 1.770832997863181, + 1.762458006851375, + 1.756333003868349, + 1.7644580075284466, + 1.7774580046534538, + 1.7704589990898967, + 1.699040993116796, + 1.6770410002209246 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 1.44041700696107, + "p95": 1.4666580012999475, + "min": 1.3469580007949844, + "max": 1.4673330006189644, + "iqr": 0.08954199438448995, + "mean": 1.4186164044076577, + "cv": 0.03859842503958307 + }, + "decode_ms_per_step": { + "count": 1115, + "median": 1.7039590020431206, + "p95": 1.8306454978301188, + "min": 1.3259170082164928, + "max": 4.357625002739951, + "iqr": 0.12256249465281144, + "mean": 1.6977690591872565, + "cv": 0.1021291412932686 + }, + "total_ms": { + "count": 5, + "median": 377.62533890781924, + "p95": 388.02404880407266, + "min": 376.71854098152835, + "max": 390.4310819780221, + "iqr": 1.4612110680900514, + "mean": 380.02111660316586, + "cv": 0.01541025441086353 + }, + "tokens_per_second": { + "count": 5, + "median": 593.1805335093783, + "p95": 594.540177307541, + "min": 573.7248142877345, + "max": 594.6083763660133, + "iqr": 2.2948188330373114, + "mean": 589.5507334954784, + "cv": 0.015107836990863094 + }, + "median_step_ms_by_index": [ + 1.44041700696107, + 1.444624998839572, + 1.473374999477528, + 1.4667909999843687, + 1.4882920077070594, + 1.4721659972565249, + 1.4363340014824644, + 1.4652909885626286, + 1.4786670071771368, + 1.4666670031147078, + 1.4383750094566494, + 1.519041004939936, + 1.5242500085150823, + 1.444042005459778, + 1.5083329926710576, + 1.5012919902801514, + 1.5655000024707988, + 1.5033340023364872, + 1.5162080089794472, + 1.5103339974302799, + 1.4690420066472143, + 1.5899999998509884, + 1.5087079955264926, + 1.5406250022351742, + 1.534499999252148, + 1.5304169937735423, + 1.5140420000534505, + 1.5117080038180575, + 1.5300840022973716, + 1.4525419974233955, + 1.5378329990198836, + 1.5019999991636723, + 1.5217089967336506, + 1.6311250074068084, + 1.6265829908661544, + 1.6115839971462265, + 1.5972499968484044, + 1.6506660031154752, + 1.5786250005476177, + 1.6292079963022843, + 1.6359169967472553, + 1.614875000086613, + 1.589915991644375, + 1.5770829922985286, + 1.6005830111680552, + 1.6382500034524128, + 1.6339589928975329, + 1.6433749988209456, + 1.5873329975875095, + 1.6029999969759956, + 1.6119999927468598, + 1.6026249941205606, + 1.5917499986244366, + 1.6627909935778007, + 1.6405420028604567, + 1.6601249953964725, + 1.606124991667457, + 1.67199999850709, + 1.6652079939376563, + 1.6900420014280826, + 1.6406250069849193, + 1.6909580008359626, + 1.7013339966069907, + 1.7004170076688752, + 1.6343750030500814, + 1.7646250053076074, + 1.70808398979716, + 1.7083330021705478, + 1.665332994889468, + 1.7579169943928719, + 1.7504169954918325, + 1.7651669913902879, + 1.7526250012451783, + 1.7710829997668043, + 1.7122080025728792, + 1.6814169939607382, + 1.7679169977782294, + 1.7785829986678436, + 1.7666249914327636, + 1.763832988217473, + 1.7672909889370203, + 1.7439579969504848, + 1.7519170069135725, + 1.739958010148257, + 1.738833001581952, + 1.7543750000186265, + 1.7225000046892092, + 1.737875005346723, + 1.7484580021118745, + 1.7335840093437582, + 1.7760420014383271, + 1.770333998138085, + 1.7238330037798733, + 1.7227079952135682, + 1.7389580025337636, + 1.7771660059224814, + 1.7326660017715767, + 1.786124994396232, + 1.6677079984219745, + 1.701959001366049, + 1.7108339961851016, + 1.7187920020660385, + 1.7385000101057813, + 1.7364590021315962, + 1.7903749976539984, + 1.8090840021613985, + 1.7584589950274676, + 1.7082499980460852, + 1.6837919974932447, + 1.7022910033119842, + 1.6673330101184547, + 1.7377909971401095, + 1.7027909925673157, + 1.7347919929306954, + 1.7509589961264282, + 1.7519160028314218, + 1.7608750058570877, + 1.7218329885508865, + 1.7308340029558167, + 1.7587079928489402, + 1.7077500087907538, + 1.754124998115003, + 1.7454580083722249, + 1.7217500135302544, + 1.7569589981576428, + 1.7064169951481745, + 1.6847499937284738, + 1.706833005300723, + 1.756584009854123, + 1.7696249997243285, + 1.764249987900257, + 1.7215419939020649, + 1.7383329977747053, + 1.7703330086078495, + 1.7628329951548949, + 1.7735830042511225, + 1.7168750055134296, + 1.7381250072503462, + 1.7722920019878075, + 1.7703749908832833, + 1.7566669994266704, + 1.7502499977126718, + 1.804792002076283, + 1.7296669975621626, + 1.6724160086596385, + 1.7322089988738298, + 1.7105419974541292, + 1.726499991491437, + 1.7178330017486587, + 1.734667006530799, + 1.7731670086504892, + 1.763249994837679, + 1.7645420011831447, + 1.7202909948537126, + 1.7594160017324612, + 1.739166007610038, + 1.6719159903004766, + 1.7357080068904907, + 1.6690000047674403, + 1.684792005107738, + 1.7281249893130735, + 1.767666995874606, + 1.7446669953642413, + 1.7355000018142164, + 1.7540840053698048, + 1.709666001261212, + 1.7134590016212314, + 1.6975829930743203, + 1.6975409962469712, + 1.7003329994622618, + 1.7282499902648851, + 1.6853749984875321, + 1.7329590045846999, + 1.7649169894866645, + 1.7687079962342978, + 1.7385840037604794, + 1.7543750000186265, + 1.7154580127680674, + 1.761708001140505, + 1.702875000773929, + 1.7246250063180923, + 1.7586250032763928, + 1.680999994277954, + 1.7780839989427477, + 1.760333005222492, + 1.7479589878348634, + 1.7165000026579946, + 1.7278330051340163, + 1.739457991789095, + 1.7063330014934763, + 1.7056249926099554, + 1.7387080006301403, + 1.763583000865765, + 1.669500008574687, + 1.728041999740526, + 1.746041001752019, + 1.748915994539857, + 1.7058749945135787, + 1.7178330017486587, + 1.6896250017452985, + 1.7290419928031042, + 1.7291249969275668, + 1.738874998409301, + 1.7054999916581437, + 1.7329170077573508, + 1.7686659994069487, + 1.7654999974183738, + 1.7155829991679639, + 1.787624991266057, + 1.7595410026842728, + 1.7614999960642308, + 1.7072079936042428, + 1.726709000649862, + 1.7506250005681068, + 1.6767910128692165, + 1.706875002128072, + 1.7636250122450292, + 1.7377910116920248, + 1.7639169964240864, + 1.7628330097068101, + 1.6900839982554317, + 1.6876659938134253, + 1.7540839908178896, + 1.6971250006463379 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "uncached_default": { + "first_token_ms": { + "count": 5, + "median": 1.6057089960668236, + "p95": 1.6878499940503389, + "min": 1.5602920029778033, + "max": 1.7018749931594357, + "iqr": 0.03487500362098217, + "mean": 1.6193001967621967, + "cv": 0.032594704744553886 + }, + "decode_ms_per_step": { + "count": 1115, + "median": 1.8249580025440082, + "p95": 1.9527705007931218, + "min": 1.4377089974004775, + "max": 2.145457998267375, + "iqr": 0.1314784967689775, + "mean": 1.8029370350134664, + "cv": 0.06582583200393874 + }, + "total_ms": { + "count": 5, + "median": 404.73266900517046, + "p95": 405.4802531551104, + "min": 401.0690820723539, + "max": 405.5978269461775, + "iqr": 3.0481989815598354, + "mean": 403.6742590047652, + "cv": 0.005004408760186724 + }, + "tokens_per_second": { + "count": 5, + "median": 553.4517402575634, + "p95": 558.2592080492232, + "min": 552.2712034394716, + "max": 558.5072747133119, + "iqr": 4.194120390612966, + "mean": 554.9139961610942, + "cv": 0.00501358201524361 + }, + "median_step_ms_by_index": [ + 1.6057089960668236, + 1.5496669948333874, + 1.5520420129178092, + 1.5287090063793585, + 1.5145829966058955, + 1.5537919971393421, + 1.487250003265217, + 1.5610420086886734, + 1.5213340084301308, + 1.5812500059837475, + 1.559708995046094, + 1.526166990515776, + 1.595291992998682, + 1.5744170086691156, + 1.597708003828302, + 1.5817910025361925, + 1.6312090010615066, + 1.658124994719401, + 1.5529159863945097, + 1.589584004250355, + 1.6385409981012344, + 1.6576249909121543, + 1.667459000600502, + 1.6529169952264056, + 1.6421250038547441, + 1.5818340034456924, + 1.595249996171333, + 1.658541994402185, + 1.6347499913536012, + 1.6509999986737967, + 1.5648340049665421, + 1.5598750032950193, + 1.6113750025397167, + 1.7637089913478121, + 1.7357080068904907, + 1.7407089908374473, + 1.7221250018337741, + 1.7430000007152557, + 1.676582993241027, + 1.669917008257471, + 1.6534169990336522, + 1.7624169995542616, + 1.7332499992335215, + 1.7136250098701566, + 1.6797499993117526, + 1.6619579982943833, + 1.7132079956354573, + 1.6817499999888241, + 1.7024169937940314, + 1.752416996168904, + 1.7394589958712459, + 1.7898749938467517, + 1.7011659947456792, + 1.7548329924466088, + 1.7919169913511723, + 1.790124995750375, + 1.756792000378482, + 1.7800420027924702, + 1.8034169916063547, + 1.810124987969175, + 1.8529999942984432, + 1.798666999093257, + 1.7642080056248233, + 1.7626250046305358, + 1.8070419901050627, + 1.8477080011507496, + 1.9509999983711168, + 1.8439590057823807, + 1.8249169952468947, + 1.9151669985149056, + 1.868959006969817, + 1.9128329877275974, + 1.8957499996758997, + 1.863208002760075, + 1.8796250078594312, + 1.8549590022303164, + 1.8631670100148767, + 1.7874580080388114, + 1.8499160069040954, + 1.843999998527579, + 1.8513749964768067, + 1.8324580014450476, + 1.8687909905565903, + 1.9106249965261668, + 1.843750011175871, + 1.8541250028647482, + 1.9125000108033419, + 1.9285419984953478, + 1.910457998747006, + 1.891750012873672, + 1.8774579948512837, + 1.9031249976251274, + 1.9055829907301813, + 1.9400000019231811, + 1.8978339940076694, + 1.8736250058282167, + 1.820416990085505, + 1.858250005170703, + 1.8994580022990704, + 1.8899579881690443, + 1.88079199870117, + 1.8857090035453439, + 1.8782080005621538, + 1.8881660071201622, + 1.8791670008795336, + 1.88433300354518, + 1.8414169899187982, + 1.8775419885059819, + 1.805541993235238, + 1.8414590012980625, + 1.7914170020958409, + 1.8013749941019341, + 1.7871670133899897, + 1.8484169995645061, + 1.8810829933499917, + 1.880374999018386, + 1.8578330054879189, + 1.8904169992310926, + 1.890832994831726, + 1.8767910078167915, + 1.8935409898404032, + 1.8968340009450912, + 1.7960830009542406, + 1.8747919966699556, + 1.8612499989103526, + 1.8706669943640009, + 1.8466250039637089, + 1.8264589889440686, + 1.8038749985862523, + 1.8098749860655516, + 1.862416000221856, + 1.8691659934120253, + 1.8893750093411654, + 1.865875005023554, + 1.7979580006795004, + 1.8540420132922009, + 1.8831250054063275, + 1.8468330090399832, + 1.86191700049676, + 1.885541991214268, + 1.8082089954987168, + 1.8734589975792915, + 1.8247090047225356, + 1.8163329950766638, + 1.8856250098906457, + 1.8580000032670796, + 1.7992500070249662, + 1.8608749960549176, + 1.8598330061649904, + 1.8718750070547685, + 1.8434580124448985, + 1.812375005101785, + 1.843666992499493, + 1.926333992742002, + 1.8597500020405278, + 1.826040999731049, + 1.8619580077938735, + 1.7790830024750903, + 1.8852500070352107, + 1.8498340068617836, + 1.886916987132281, + 1.8872500077122822, + 1.9105420069536194, + 1.8332920008106157, + 1.8940000009024516, + 1.8802919948939234, + 1.7939169920282438, + 1.872083987109363, + 1.892375003080815, + 1.8854579975595698, + 1.8157919985242188, + 1.8278750067111105, + 1.8349580059293658, + 1.8244170059915632, + 1.8382080015726388, + 1.8317080102860928, + 1.8734159966697916, + 1.894333996460773, + 1.8267090053996071, + 1.8913329986389726, + 1.8821250123437494, + 1.8737909995252267, + 1.8910830112872645, + 1.8825409933924675, + 1.7851660086307675, + 1.8657920008990914, + 1.883249991806224, + 1.812375005101785, + 1.8526250059949234, + 1.8653330043889582, + 1.7823330126702785, + 1.845916995080188, + 1.8249160057166591, + 1.8031249928753823, + 1.844041995354928, + 1.8790410103974864, + 1.8786250002449378, + 1.844792001065798, + 1.8868339975597337, + 1.867624989245087, + 1.8176250014221296, + 1.8855829985113814, + 1.8039579881587997, + 1.8845830054488033, + 1.8312500033061951, + 1.8539580050855875, + 1.8594580033095554, + 1.84058299055323, + 1.8504170002415776, + 1.795459000277333, + 1.7903330008266494, + 1.8589579995023087, + 1.8490000074962154, + 1.87329099571798, + 1.877167000202462, + 1.8861249991459772, + 1.8968340009450912, + 1.8160839972551912, + 1.816584001062438, + 1.8217909964732826, + 1.817500000470318, + 1.8615830049384385, + 1.8204999942099676, + 1.8275420006830245 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + }, + "speedup_cached_vs_uncached_sdpa": 0.9934672176612411, + "speedup_cached_vs_uncached_default": 1.0647819336914972 + }, + "memory": [ + { + "batch": 1, + "prompt_len": 128, + "steps": [ + { + "step": 0, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 1, + "cached_positions": 129, + "cache_bytes": 396288, + "step_peak_minus_start_bytes": 404224 + }, + { + "step": 2, + "cached_positions": 130, + "cache_bytes": 399360, + "step_peak_minus_start_bytes": 406272 + }, + { + "step": 3, + "cached_positions": 131, + "cache_bytes": 402432, + "step_peak_minus_start_bytes": 410368 + }, + { + "step": 4, + "cached_positions": 132, + "cache_bytes": 405504, + "step_peak_minus_start_bytes": 412416 + }, + { + "step": 5, + "cached_positions": 133, + "cache_bytes": 408576, + "step_peak_minus_start_bytes": 416512 + }, + { + "step": 6, + "cached_positions": 134, + "cache_bytes": 411648, + "step_peak_minus_start_bytes": 418560 + }, + { + "step": 7, + "cached_positions": 135, + "cache_bytes": 414720, + "step_peak_minus_start_bytes": 422656 + }, + { + "step": 8, + "cached_positions": 136, + "cache_bytes": 417792, + "step_peak_minus_start_bytes": 424704 + }, + { + "step": 9, + "cached_positions": 137, + "cache_bytes": 420864, + "step_peak_minus_start_bytes": 428800 + }, + { + "step": 10, + "cached_positions": 138, + "cache_bytes": 423936, + "step_peak_minus_start_bytes": 430848 + }, + { + "step": 11, + "cached_positions": 139, + "cache_bytes": 427008, + "step_peak_minus_start_bytes": 434944 + }, + { + "step": 12, + "cached_positions": 140, + "cache_bytes": 430080, + "step_peak_minus_start_bytes": 436992 + }, + { + "step": 13, + "cached_positions": 141, + "cache_bytes": 433152, + "step_peak_minus_start_bytes": 441088 + }, + { + "step": 14, + "cached_positions": 142, + "cache_bytes": 436224, + "step_peak_minus_start_bytes": 443136 + }, + { + "step": 15, + "cached_positions": 143, + "cache_bytes": 439296, + "step_peak_minus_start_bytes": 447232 + }, + { + "step": 16, + "cached_positions": 144, + "cache_bytes": 442368, + "step_peak_minus_start_bytes": 449280 + }, + { + "step": 17, + "cached_positions": 145, + "cache_bytes": 445440, + "step_peak_minus_start_bytes": 453376 + }, + { + "step": 18, + "cached_positions": 146, + "cache_bytes": 448512, + "step_peak_minus_start_bytes": 455424 + }, + { + "step": 19, + "cached_positions": 147, + "cache_bytes": 451584, + "step_peak_minus_start_bytes": 459520 + }, + { + "step": 20, + "cached_positions": 148, + "cache_bytes": 454656, + "step_peak_minus_start_bytes": 461568 + }, + { + "step": 21, + "cached_positions": 149, + "cache_bytes": 457728, + "step_peak_minus_start_bytes": 465664 + }, + { + "step": 22, + "cached_positions": 150, + "cache_bytes": 460800, + "step_peak_minus_start_bytes": 467712 + }, + { + "step": 23, + "cached_positions": 151, + "cache_bytes": 463872, + "step_peak_minus_start_bytes": 471808 + }, + { + "step": 24, + "cached_positions": 152, + "cache_bytes": 466944, + "step_peak_minus_start_bytes": 473856 + }, + { + "step": 25, + "cached_positions": 153, + "cache_bytes": 470016, + "step_peak_minus_start_bytes": 477952 + }, + { + "step": 26, + "cached_positions": 154, + "cache_bytes": 473088, + "step_peak_minus_start_bytes": 480000 + }, + { + "step": 27, + "cached_positions": 155, + "cache_bytes": 476160, + "step_peak_minus_start_bytes": 484096 + }, + { + "step": 28, + "cached_positions": 156, + "cache_bytes": 479232, + "step_peak_minus_start_bytes": 486144 + }, + { + "step": 29, + "cached_positions": 157, + "cache_bytes": 482304, + "step_peak_minus_start_bytes": 490240 + }, + { + "step": 30, + "cached_positions": 158, + "cache_bytes": 485376, + "step_peak_minus_start_bytes": 492288 + }, + { + "step": 31, + "cached_positions": 159, + "cache_bytes": 488448, + "step_peak_minus_start_bytes": 496384 + }, + { + "step": 32, + "cached_positions": 160, + "cache_bytes": 491520, + "step_peak_minus_start_bytes": 498432 + }, + { + "step": 33, + "cached_positions": 161, + "cache_bytes": 494592, + "step_peak_minus_start_bytes": 502528 + }, + { + "step": 34, + "cached_positions": 162, + "cache_bytes": 497664, + "step_peak_minus_start_bytes": 504576 + }, + { + "step": 35, + "cached_positions": 163, + "cache_bytes": 500736, + "step_peak_minus_start_bytes": 508672 + }, + { + "step": 36, + "cached_positions": 164, + "cache_bytes": 503808, + "step_peak_minus_start_bytes": 510720 + }, + { + "step": 37, + "cached_positions": 165, + "cache_bytes": 506880, + "step_peak_minus_start_bytes": 514816 + }, + { + "step": 38, + "cached_positions": 166, + "cache_bytes": 509952, + "step_peak_minus_start_bytes": 516864 + }, + { + "step": 39, + "cached_positions": 167, + "cache_bytes": 513024, + "step_peak_minus_start_bytes": 520960 + }, + { + "step": 40, + "cached_positions": 168, + "cache_bytes": 516096, + "step_peak_minus_start_bytes": 523008 + }, + { + "step": 41, + "cached_positions": 169, + "cache_bytes": 519168, + "step_peak_minus_start_bytes": 527104 + }, + { + "step": 42, + "cached_positions": 170, + "cache_bytes": 522240, + "step_peak_minus_start_bytes": 529152 + }, + { + "step": 43, + "cached_positions": 171, + "cache_bytes": 525312, + "step_peak_minus_start_bytes": 533248 + }, + { + "step": 44, + "cached_positions": 172, + "cache_bytes": 528384, + "step_peak_minus_start_bytes": 535296 + }, + { + "step": 45, + "cached_positions": 173, + "cache_bytes": 531456, + "step_peak_minus_start_bytes": 539392 + }, + { + "step": 46, + "cached_positions": 174, + "cache_bytes": 534528, + "step_peak_minus_start_bytes": 541440 + }, + { + "step": 47, + "cached_positions": 175, + "cache_bytes": 537600, + "step_peak_minus_start_bytes": 545536 + }, + { + "step": 48, + "cached_positions": 176, + "cache_bytes": 540672, + "step_peak_minus_start_bytes": 547584 + }, + { + "step": 49, + "cached_positions": 177, + "cache_bytes": 543744, + "step_peak_minus_start_bytes": 551680 + }, + { + "step": 50, + "cached_positions": 178, + "cache_bytes": 546816, + "step_peak_minus_start_bytes": 553728 + }, + { + "step": 51, + "cached_positions": 179, + "cache_bytes": 549888, + "step_peak_minus_start_bytes": 557824 + }, + { + "step": 52, + "cached_positions": 180, + "cache_bytes": 552960, + "step_peak_minus_start_bytes": 559872 + }, + { + "step": 53, + "cached_positions": 181, + "cache_bytes": 556032, + "step_peak_minus_start_bytes": 563968 + }, + { + "step": 54, + "cached_positions": 182, + "cache_bytes": 559104, + "step_peak_minus_start_bytes": 566016 + }, + { + "step": 55, + "cached_positions": 183, + "cache_bytes": 562176, + "step_peak_minus_start_bytes": 570112 + }, + { + "step": 56, + "cached_positions": 184, + "cache_bytes": 565248, + "step_peak_minus_start_bytes": 572160 + }, + { + "step": 57, + "cached_positions": 185, + "cache_bytes": 568320, + "step_peak_minus_start_bytes": 576256 + }, + { + "step": 58, + "cached_positions": 186, + "cache_bytes": 571392, + "step_peak_minus_start_bytes": 578304 + }, + { + "step": 59, + "cached_positions": 187, + "cache_bytes": 574464, + "step_peak_minus_start_bytes": 582400 + }, + { + "step": 60, + "cached_positions": 188, + "cache_bytes": 577536, + "step_peak_minus_start_bytes": 584448 + }, + { + "step": 61, + "cached_positions": 189, + "cache_bytes": 580608, + "step_peak_minus_start_bytes": 588544 + }, + { + "step": 62, + "cached_positions": 190, + "cache_bytes": 583680, + "step_peak_minus_start_bytes": 590592 + }, + { + "step": 63, + "cached_positions": 191, + "cache_bytes": 586752, + "step_peak_minus_start_bytes": 594688 + } + ] + }, + { + "batch": 8, + "prompt_len": 128, + "steps": [ + { + "step": 0, + "cached_positions": 128, + "cache_bytes": 3145728, + "step_peak_minus_start_bytes": 14189568 + }, + { + "step": 1, + "cached_positions": 129, + "cache_bytes": 3170304, + "step_peak_minus_start_bytes": 3219712 + }, + { + "step": 2, + "cached_positions": 130, + "cache_bytes": 3194880, + "step_peak_minus_start_bytes": 3244288 + }, + { + "step": 3, + "cached_positions": 131, + "cache_bytes": 3219456, + "step_peak_minus_start_bytes": 3268864 + }, + { + "step": 4, + "cached_positions": 132, + "cache_bytes": 3244032, + "step_peak_minus_start_bytes": 3293440 + }, + { + "step": 5, + "cached_positions": 133, + "cache_bytes": 3268608, + "step_peak_minus_start_bytes": 3318016 + }, + { + "step": 6, + "cached_positions": 134, + "cache_bytes": 3293184, + "step_peak_minus_start_bytes": 3342592 + }, + { + "step": 7, + "cached_positions": 135, + "cache_bytes": 3317760, + "step_peak_minus_start_bytes": 3367168 + }, + { + "step": 8, + "cached_positions": 136, + "cache_bytes": 3342336, + "step_peak_minus_start_bytes": 3391744 + }, + { + "step": 9, + "cached_positions": 137, + "cache_bytes": 3366912, + "step_peak_minus_start_bytes": 3416320 + }, + { + "step": 10, + "cached_positions": 138, + "cache_bytes": 3391488, + "step_peak_minus_start_bytes": 3440896 + }, + { + "step": 11, + "cached_positions": 139, + "cache_bytes": 3416064, + "step_peak_minus_start_bytes": 3465472 + }, + { + "step": 12, + "cached_positions": 140, + "cache_bytes": 3440640, + "step_peak_minus_start_bytes": 3490048 + }, + { + "step": 13, + "cached_positions": 141, + "cache_bytes": 3465216, + "step_peak_minus_start_bytes": 3514624 + }, + { + "step": 14, + "cached_positions": 142, + "cache_bytes": 3489792, + "step_peak_minus_start_bytes": 3539200 + }, + { + "step": 15, + "cached_positions": 143, + "cache_bytes": 3514368, + "step_peak_minus_start_bytes": 3563776 + }, + { + "step": 16, + "cached_positions": 144, + "cache_bytes": 3538944, + "step_peak_minus_start_bytes": 3588352 + }, + { + "step": 17, + "cached_positions": 145, + "cache_bytes": 3563520, + "step_peak_minus_start_bytes": 3612928 + }, + { + "step": 18, + "cached_positions": 146, + "cache_bytes": 3588096, + "step_peak_minus_start_bytes": 3637504 + }, + { + "step": 19, + "cached_positions": 147, + "cache_bytes": 3612672, + "step_peak_minus_start_bytes": 3662080 + }, + { + "step": 20, + "cached_positions": 148, + "cache_bytes": 3637248, + "step_peak_minus_start_bytes": 3686656 + }, + { + "step": 21, + "cached_positions": 149, + "cache_bytes": 3661824, + "step_peak_minus_start_bytes": 3711232 + }, + { + "step": 22, + "cached_positions": 150, + "cache_bytes": 3686400, + "step_peak_minus_start_bytes": 3735808 + }, + { + "step": 23, + "cached_positions": 151, + "cache_bytes": 3710976, + "step_peak_minus_start_bytes": 3760384 + }, + { + "step": 24, + "cached_positions": 152, + "cache_bytes": 3735552, + "step_peak_minus_start_bytes": 3784960 + }, + { + "step": 25, + "cached_positions": 153, + "cache_bytes": 3760128, + "step_peak_minus_start_bytes": 3809536 + }, + { + "step": 26, + "cached_positions": 154, + "cache_bytes": 3784704, + "step_peak_minus_start_bytes": 3834112 + }, + { + "step": 27, + "cached_positions": 155, + "cache_bytes": 3809280, + "step_peak_minus_start_bytes": 3858688 + }, + { + "step": 28, + "cached_positions": 156, + "cache_bytes": 3833856, + "step_peak_minus_start_bytes": 3883264 + }, + { + "step": 29, + "cached_positions": 157, + "cache_bytes": 3858432, + "step_peak_minus_start_bytes": 3907840 + }, + { + "step": 30, + "cached_positions": 158, + "cache_bytes": 3883008, + "step_peak_minus_start_bytes": 3932416 + }, + { + "step": 31, + "cached_positions": 159, + "cache_bytes": 3907584, + "step_peak_minus_start_bytes": 3956992 + }, + { + "step": 32, + "cached_positions": 160, + "cache_bytes": 3932160, + "step_peak_minus_start_bytes": 3981568 + }, + { + "step": 33, + "cached_positions": 161, + "cache_bytes": 3956736, + "step_peak_minus_start_bytes": 4006144 + }, + { + "step": 34, + "cached_positions": 162, + "cache_bytes": 3981312, + "step_peak_minus_start_bytes": 4030720 + }, + { + "step": 35, + "cached_positions": 163, + "cache_bytes": 4005888, + "step_peak_minus_start_bytes": 4055296 + }, + { + "step": 36, + "cached_positions": 164, + "cache_bytes": 4030464, + "step_peak_minus_start_bytes": 4079872 + }, + { + "step": 37, + "cached_positions": 165, + "cache_bytes": 4055040, + "step_peak_minus_start_bytes": 4104448 + }, + { + "step": 38, + "cached_positions": 166, + "cache_bytes": 4079616, + "step_peak_minus_start_bytes": 4129024 + }, + { + "step": 39, + "cached_positions": 167, + "cache_bytes": 4104192, + "step_peak_minus_start_bytes": 4153600 + }, + { + "step": 40, + "cached_positions": 168, + "cache_bytes": 4128768, + "step_peak_minus_start_bytes": 4178176 + }, + { + "step": 41, + "cached_positions": 169, + "cache_bytes": 4153344, + "step_peak_minus_start_bytes": 4202752 + }, + { + "step": 42, + "cached_positions": 170, + "cache_bytes": 4177920, + "step_peak_minus_start_bytes": 4227328 + }, + { + "step": 43, + "cached_positions": 171, + "cache_bytes": 4202496, + "step_peak_minus_start_bytes": 4251904 + }, + { + "step": 44, + "cached_positions": 172, + "cache_bytes": 4227072, + "step_peak_minus_start_bytes": 4276480 + }, + { + "step": 45, + "cached_positions": 173, + "cache_bytes": 4251648, + "step_peak_minus_start_bytes": 4301056 + }, + { + "step": 46, + "cached_positions": 174, + "cache_bytes": 4276224, + "step_peak_minus_start_bytes": 4325632 + }, + { + "step": 47, + "cached_positions": 175, + "cache_bytes": 4300800, + "step_peak_minus_start_bytes": 4350208 + }, + { + "step": 48, + "cached_positions": 176, + "cache_bytes": 4325376, + "step_peak_minus_start_bytes": 4374784 + }, + { + "step": 49, + "cached_positions": 177, + "cache_bytes": 4349952, + "step_peak_minus_start_bytes": 4399360 + }, + { + "step": 50, + "cached_positions": 178, + "cache_bytes": 4374528, + "step_peak_minus_start_bytes": 4423936 + }, + { + "step": 51, + "cached_positions": 179, + "cache_bytes": 4399104, + "step_peak_minus_start_bytes": 4448512 + }, + { + "step": 52, + "cached_positions": 180, + "cache_bytes": 4423680, + "step_peak_minus_start_bytes": 4473088 + }, + { + "step": 53, + "cached_positions": 181, + "cache_bytes": 4448256, + "step_peak_minus_start_bytes": 4497664 + }, + { + "step": 54, + "cached_positions": 182, + "cache_bytes": 4472832, + "step_peak_minus_start_bytes": 4522240 + }, + { + "step": 55, + "cached_positions": 183, + "cache_bytes": 4497408, + "step_peak_minus_start_bytes": 4546816 + }, + { + "step": 56, + "cached_positions": 184, + "cache_bytes": 4521984, + "step_peak_minus_start_bytes": 4571392 + }, + { + "step": 57, + "cached_positions": 185, + "cache_bytes": 4546560, + "step_peak_minus_start_bytes": 4595968 + }, + { + "step": 58, + "cached_positions": 186, + "cache_bytes": 4571136, + "step_peak_minus_start_bytes": 4620544 + }, + { + "step": 59, + "cached_positions": 187, + "cache_bytes": 4595712, + "step_peak_minus_start_bytes": 4645120 + }, + { + "step": 60, + "cached_positions": 188, + "cache_bytes": 4620288, + "step_peak_minus_start_bytes": 4669696 + }, + { + "step": 61, + "cached_positions": 189, + "cache_bytes": 4644864, + "step_peak_minus_start_bytes": 4694272 + }, + { + "step": 62, + "cached_positions": 190, + "cache_bytes": 4669440, + "step_peak_minus_start_bytes": 4718848 + }, + { + "step": 63, + "cached_positions": 191, + "cache_bytes": 4694016, + "step_peak_minus_start_bytes": 4743424 + } + ] + }, + { + "batch": 1, + "prompt_len": 32, + "steps": [ + { + "step": 0, + "cached_positions": 32, + "cache_bytes": 98304, + "step_peak_minus_start_bytes": 344320 + }, + { + "step": 1, + "cached_positions": 33, + "cache_bytes": 101376, + "step_peak_minus_start_bytes": 109312 + }, + { + "step": 2, + "cached_positions": 34, + "cache_bytes": 104448, + "step_peak_minus_start_bytes": 111360 + }, + { + "step": 3, + "cached_positions": 35, + "cache_bytes": 107520, + "step_peak_minus_start_bytes": 115456 + }, + { + "step": 4, + "cached_positions": 36, + "cache_bytes": 110592, + "step_peak_minus_start_bytes": 117504 + }, + { + "step": 5, + "cached_positions": 37, + "cache_bytes": 113664, + "step_peak_minus_start_bytes": 121600 + }, + { + "step": 6, + "cached_positions": 38, + "cache_bytes": 116736, + "step_peak_minus_start_bytes": 123648 + }, + { + "step": 7, + "cached_positions": 39, + "cache_bytes": 119808, + "step_peak_minus_start_bytes": 127744 + }, + { + "step": 8, + "cached_positions": 40, + "cache_bytes": 122880, + "step_peak_minus_start_bytes": 129792 + }, + { + "step": 9, + "cached_positions": 41, + "cache_bytes": 125952, + "step_peak_minus_start_bytes": 133888 + }, + { + "step": 10, + "cached_positions": 42, + "cache_bytes": 129024, + "step_peak_minus_start_bytes": 135936 + }, + { + "step": 11, + "cached_positions": 43, + "cache_bytes": 132096, + "step_peak_minus_start_bytes": 140032 + }, + { + "step": 12, + "cached_positions": 44, + "cache_bytes": 135168, + "step_peak_minus_start_bytes": 142080 + }, + { + "step": 13, + "cached_positions": 45, + "cache_bytes": 138240, + "step_peak_minus_start_bytes": 146176 + }, + { + "step": 14, + "cached_positions": 46, + "cache_bytes": 141312, + "step_peak_minus_start_bytes": 148224 + }, + { + "step": 15, + "cached_positions": 47, + "cache_bytes": 144384, + "step_peak_minus_start_bytes": 152320 + }, + { + "step": 16, + "cached_positions": 48, + "cache_bytes": 147456, + "step_peak_minus_start_bytes": 154368 + }, + { + "step": 17, + "cached_positions": 49, + "cache_bytes": 150528, + "step_peak_minus_start_bytes": 158464 + }, + { + "step": 18, + "cached_positions": 50, + "cache_bytes": 153600, + "step_peak_minus_start_bytes": 160512 + }, + { + "step": 19, + "cached_positions": 51, + "cache_bytes": 156672, + "step_peak_minus_start_bytes": 164608 + }, + { + "step": 20, + "cached_positions": 52, + "cache_bytes": 159744, + "step_peak_minus_start_bytes": 166656 + }, + { + "step": 21, + "cached_positions": 53, + "cache_bytes": 162816, + "step_peak_minus_start_bytes": 170752 + }, + { + "step": 22, + "cached_positions": 54, + "cache_bytes": 165888, + "step_peak_minus_start_bytes": 172800 + }, + { + "step": 23, + "cached_positions": 55, + "cache_bytes": 168960, + "step_peak_minus_start_bytes": 176896 + }, + { + "step": 24, + "cached_positions": 56, + "cache_bytes": 172032, + "step_peak_minus_start_bytes": 178944 + }, + { + "step": 25, + "cached_positions": 57, + "cache_bytes": 175104, + "step_peak_minus_start_bytes": 183040 + }, + { + "step": 26, + "cached_positions": 58, + "cache_bytes": 178176, + "step_peak_minus_start_bytes": 185088 + }, + { + "step": 27, + "cached_positions": 59, + "cache_bytes": 181248, + "step_peak_minus_start_bytes": 189184 + }, + { + "step": 28, + "cached_positions": 60, + "cache_bytes": 184320, + "step_peak_minus_start_bytes": 191232 + }, + { + "step": 29, + "cached_positions": 61, + "cache_bytes": 187392, + "step_peak_minus_start_bytes": 195328 + }, + { + "step": 30, + "cached_positions": 62, + "cache_bytes": 190464, + "step_peak_minus_start_bytes": 197376 + }, + { + "step": 31, + "cached_positions": 63, + "cache_bytes": 193536, + "step_peak_minus_start_bytes": 201472 + }, + { + "step": 32, + "cached_positions": 64, + "cache_bytes": 196608, + "step_peak_minus_start_bytes": 203520 + }, + { + "step": 33, + "cached_positions": 65, + "cache_bytes": 199680, + "step_peak_minus_start_bytes": 207616 + }, + { + "step": 34, + "cached_positions": 66, + "cache_bytes": 202752, + "step_peak_minus_start_bytes": 209664 + }, + { + "step": 35, + "cached_positions": 67, + "cache_bytes": 205824, + "step_peak_minus_start_bytes": 213760 + }, + { + "step": 36, + "cached_positions": 68, + "cache_bytes": 208896, + "step_peak_minus_start_bytes": 215808 + }, + { + "step": 37, + "cached_positions": 69, + "cache_bytes": 211968, + "step_peak_minus_start_bytes": 219904 + }, + { + "step": 38, + "cached_positions": 70, + "cache_bytes": 215040, + "step_peak_minus_start_bytes": 221952 + }, + { + "step": 39, + "cached_positions": 71, + "cache_bytes": 218112, + "step_peak_minus_start_bytes": 226048 + }, + { + "step": 40, + "cached_positions": 72, + "cache_bytes": 221184, + "step_peak_minus_start_bytes": 228096 + }, + { + "step": 41, + "cached_positions": 73, + "cache_bytes": 224256, + "step_peak_minus_start_bytes": 232192 + }, + { + "step": 42, + "cached_positions": 74, + "cache_bytes": 227328, + "step_peak_minus_start_bytes": 234240 + }, + { + "step": 43, + "cached_positions": 75, + "cache_bytes": 230400, + "step_peak_minus_start_bytes": 238336 + }, + { + "step": 44, + "cached_positions": 76, + "cache_bytes": 233472, + "step_peak_minus_start_bytes": 240384 + }, + { + "step": 45, + "cached_positions": 77, + "cache_bytes": 236544, + "step_peak_minus_start_bytes": 244480 + }, + { + "step": 46, + "cached_positions": 78, + "cache_bytes": 239616, + "step_peak_minus_start_bytes": 246528 + }, + { + "step": 47, + "cached_positions": 79, + "cache_bytes": 242688, + "step_peak_minus_start_bytes": 250624 + }, + { + "step": 48, + "cached_positions": 80, + "cache_bytes": 245760, + "step_peak_minus_start_bytes": 252672 + }, + { + "step": 49, + "cached_positions": 81, + "cache_bytes": 248832, + "step_peak_minus_start_bytes": 256768 + }, + { + "step": 50, + "cached_positions": 82, + "cache_bytes": 251904, + "step_peak_minus_start_bytes": 258816 + }, + { + "step": 51, + "cached_positions": 83, + "cache_bytes": 254976, + "step_peak_minus_start_bytes": 262912 + }, + { + "step": 52, + "cached_positions": 84, + "cache_bytes": 258048, + "step_peak_minus_start_bytes": 264960 + }, + { + "step": 53, + "cached_positions": 85, + "cache_bytes": 261120, + "step_peak_minus_start_bytes": 269056 + }, + { + "step": 54, + "cached_positions": 86, + "cache_bytes": 264192, + "step_peak_minus_start_bytes": 271104 + }, + { + "step": 55, + "cached_positions": 87, + "cache_bytes": 267264, + "step_peak_minus_start_bytes": 275200 + }, + { + "step": 56, + "cached_positions": 88, + "cache_bytes": 270336, + "step_peak_minus_start_bytes": 277248 + }, + { + "step": 57, + "cached_positions": 89, + "cache_bytes": 273408, + "step_peak_minus_start_bytes": 281344 + }, + { + "step": 58, + "cached_positions": 90, + "cache_bytes": 276480, + "step_peak_minus_start_bytes": 283392 + }, + { + "step": 59, + "cached_positions": 91, + "cache_bytes": 279552, + "step_peak_minus_start_bytes": 287488 + }, + { + "step": 60, + "cached_positions": 92, + "cache_bytes": 282624, + "step_peak_minus_start_bytes": 289536 + }, + { + "step": 61, + "cached_positions": 93, + "cache_bytes": 285696, + "step_peak_minus_start_bytes": 293632 + }, + { + "step": 62, + "cached_positions": 94, + "cache_bytes": 288768, + "step_peak_minus_start_bytes": 295680 + }, + { + "step": 63, + "cached_positions": 95, + "cache_bytes": 291840, + "step_peak_minus_start_bytes": 299776 + }, + { + "step": 64, + "cached_positions": 96, + "cache_bytes": 294912, + "step_peak_minus_start_bytes": 301824 + }, + { + "step": 65, + "cached_positions": 97, + "cache_bytes": 297984, + "step_peak_minus_start_bytes": 305920 + }, + { + "step": 66, + "cached_positions": 98, + "cache_bytes": 301056, + "step_peak_minus_start_bytes": 307968 + }, + { + "step": 67, + "cached_positions": 99, + "cache_bytes": 304128, + "step_peak_minus_start_bytes": 312064 + }, + { + "step": 68, + "cached_positions": 100, + "cache_bytes": 307200, + "step_peak_minus_start_bytes": 314112 + }, + { + "step": 69, + "cached_positions": 101, + "cache_bytes": 310272, + "step_peak_minus_start_bytes": 318208 + }, + { + "step": 70, + "cached_positions": 102, + "cache_bytes": 313344, + "step_peak_minus_start_bytes": 320256 + }, + { + "step": 71, + "cached_positions": 103, + "cache_bytes": 316416, + "step_peak_minus_start_bytes": 324352 + }, + { + "step": 72, + "cached_positions": 104, + "cache_bytes": 319488, + "step_peak_minus_start_bytes": 326400 + }, + { + "step": 73, + "cached_positions": 105, + "cache_bytes": 322560, + "step_peak_minus_start_bytes": 330496 + }, + { + "step": 74, + "cached_positions": 106, + "cache_bytes": 325632, + "step_peak_minus_start_bytes": 332544 + }, + { + "step": 75, + "cached_positions": 107, + "cache_bytes": 328704, + "step_peak_minus_start_bytes": 336640 + }, + { + "step": 76, + "cached_positions": 108, + "cache_bytes": 331776, + "step_peak_minus_start_bytes": 338688 + }, + { + "step": 77, + "cached_positions": 109, + "cache_bytes": 334848, + "step_peak_minus_start_bytes": 342784 + }, + { + "step": 78, + "cached_positions": 110, + "cache_bytes": 337920, + "step_peak_minus_start_bytes": 344832 + }, + { + "step": 79, + "cached_positions": 111, + "cache_bytes": 340992, + "step_peak_minus_start_bytes": 348928 + }, + { + "step": 80, + "cached_positions": 112, + "cache_bytes": 344064, + "step_peak_minus_start_bytes": 350976 + }, + { + "step": 81, + "cached_positions": 113, + "cache_bytes": 347136, + "step_peak_minus_start_bytes": 355072 + }, + { + "step": 82, + "cached_positions": 114, + "cache_bytes": 350208, + "step_peak_minus_start_bytes": 357120 + }, + { + "step": 83, + "cached_positions": 115, + "cache_bytes": 353280, + "step_peak_minus_start_bytes": 361216 + }, + { + "step": 84, + "cached_positions": 116, + "cache_bytes": 356352, + "step_peak_minus_start_bytes": 363264 + }, + { + "step": 85, + "cached_positions": 117, + "cache_bytes": 359424, + "step_peak_minus_start_bytes": 367360 + }, + { + "step": 86, + "cached_positions": 118, + "cache_bytes": 362496, + "step_peak_minus_start_bytes": 369408 + }, + { + "step": 87, + "cached_positions": 119, + "cache_bytes": 365568, + "step_peak_minus_start_bytes": 373504 + }, + { + "step": 88, + "cached_positions": 120, + "cache_bytes": 368640, + "step_peak_minus_start_bytes": 375552 + }, + { + "step": 89, + "cached_positions": 121, + "cache_bytes": 371712, + "step_peak_minus_start_bytes": 379648 + }, + { + "step": 90, + "cached_positions": 122, + "cache_bytes": 374784, + "step_peak_minus_start_bytes": 381696 + }, + { + "step": 91, + "cached_positions": 123, + "cache_bytes": 377856, + "step_peak_minus_start_bytes": 385792 + }, + { + "step": 92, + "cached_positions": 124, + "cache_bytes": 380928, + "step_peak_minus_start_bytes": 387840 + }, + { + "step": 93, + "cached_positions": 125, + "cache_bytes": 384000, + "step_peak_minus_start_bytes": 391936 + }, + { + "step": 94, + "cached_positions": 126, + "cache_bytes": 387072, + "step_peak_minus_start_bytes": 393984 + }, + { + "step": 95, + "cached_positions": 127, + "cache_bytes": 390144, + "step_peak_minus_start_bytes": 398080 + }, + { + "step": 96, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 400128 + }, + { + "step": 97, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 98, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 99, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 100, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 101, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 102, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 103, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 104, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 105, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 106, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 107, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 108, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 109, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 110, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 111, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 112, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 113, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 114, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 115, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 116, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 117, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 118, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 119, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 120, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 121, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 122, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 123, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 124, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 125, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 126, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 127, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 128, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 129, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 130, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 131, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 132, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 133, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 134, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 135, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 136, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 137, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 138, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 139, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 140, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 141, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 142, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 143, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 144, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 145, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 146, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 147, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 148, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 149, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 150, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 151, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 152, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 153, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 154, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 155, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 156, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 157, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 158, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 159, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 160, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 161, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 162, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 163, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 164, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 165, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 166, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 167, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 168, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 169, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 170, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 171, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 172, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 173, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 174, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 175, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 176, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 177, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 178, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 179, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 180, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 181, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 182, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 183, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 184, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 185, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 186, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 187, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 188, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 189, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 190, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 191, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 192, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 193, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 194, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 195, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 196, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 197, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 198, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 199, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 200, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 201, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 202, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 203, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 204, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 205, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 206, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 207, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 208, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 209, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 210, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 211, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 212, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 213, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 214, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 215, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 216, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 217, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 218, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 219, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 220, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 221, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 222, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 223, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + } + ] + } + ] + } +} diff --git a/experiments/records/kv_cache/20261022T053124Z_kv_cpu_float32_57be26d.json b/experiments/records/kv_cache/20261022T053124Z_kv_cpu_float32_57be26d.json new file mode 100644 index 0000000..49503b3 --- /dev/null +++ b/experiments/records/kv_cache/20261022T053124Z_kv_cpu_float32_57be26d.json @@ -0,0 +1,35510 @@ +{ + "record_version": 2, + "created_utc": "20261022T053124Z", + "experiment": "kv_cache_scaling", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "cpu", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "57be26dc620ddc37c1928c7d7a674e4ff0d3d39f", + "branch": "experiments/sdpa-training-v2", + "dirty": false + }, + "command": "python scripts/benchmark_kv_scaling.py --device cpu --dtype float32" + }, + "settings": { + "dtype": "float32", + "position": "learned", + "prompt_lens": [ + 1, + 16, + 128, + 512, + 1024 + ], + "batch_sizes": [ + 1, + 8 + ], + "new_tokens": 64, + "warmup": 1, + "repeats": 5, + "seed": 0, + "modes": { + "cached": { + "attention_backend": "sdpa", + "use_cache": true + }, + "uncached_sdpa": { + "attention_backend": "sdpa", + "use_cache": false + }, + "uncached_mha": { + "attention_backend": "mha", + "use_cache": false + } + }, + "weights": "seeded random initialization (timing does not depend on trained values)", + "grid_model_config": { + "vocab_size": 256, + "context_length": 1088, + "embedding_dim": 192, + "num_layers": 4, + "num_heads": 6, + "dropout": 0.0, + "positional_encoding": "learned", + "norm": "rmsnorm", + "activation": "swiglu", + "rope_theta": 10000.0 + }, + "window_model_config": { + "vocab_size": 256, + "context_length": 128, + "embedding_dim": 192, + "num_layers": 4, + "num_heads": 6, + "dropout": 0.0, + "positional_encoding": "learned", + "norm": "rmsnorm", + "activation": "swiglu", + "rope_theta": 10000.0 + }, + "decoding": "greedy argmax; step 0 is the prompt prefill", + "timer": "time.perf_counter", + "synchronization": "device synchronize before and after every step (no-op on CPU); backend/mode order rotated each repeat", + "units": { + "raw": "seconds per step, one list per measured repeat", + "summaries": "milliseconds (ms fields) or tokens/second" + } + }, + "results": { + "grid": [ + { + "batch": 1, + "prompt_len": 1, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_mha": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 0.3512499970383942, + "p95": 0.38252499361988157, + "min": 0.3261669917264953, + "max": 0.38899999344721437, + "iqr": 0.017833997844718397, + "mean": 0.35236659459769726, + "cv": 0.0670397465695303 + }, + "decode_ms_per_step": { + "count": 315, + "median": 0.3343330026837066, + "p95": 0.37503760249819607, + "min": 0.3113749990006909, + "max": 0.5972910003038123, + "iqr": 0.01393751153955236, + "mean": 0.3401987458808377, + "cv": 0.08324837828115896 + }, + "total_ms": { + "count": 5, + "median": 21.631366995279677, + "p95": 22.29469615558628, + "min": 21.2437059963122, + "max": 22.326703954604454, + "iqr": 0.6106689397711307, + "mean": 21.784887585090473, + "cv": 0.020633000202727115 + }, + "tokens_per_second": { + "count": 5, + "median": 2958.666459404339, + "p95": 3003.9279152667523, + "min": 2866.5225341871937, + "max": 3012.6570199714715, + "iqr": 81.79322897763996, + "mean": 2938.8151554962233, + "cv": 0.02059072891710592 + }, + "median_step_ms_by_index": [ + 0.3512499970383942, + 0.34366699401289225, + 0.3417080006329343, + 0.3404999879421666, + 0.3316249931231141, + 0.3320840041851625, + 0.33799999800976366, + 0.3304999991087243, + 0.3306659928057343, + 0.33150000672321767, + 0.3277079958934337, + 0.33000000985339284, + 0.32862499938346446, + 0.33495899697300047, + 0.3400839923415333, + 0.3349170001456514, + 0.3309580060886219, + 0.33395799982827157, + 0.32662499870639294, + 0.32137500238604844, + 0.32916599593590945, + 0.33291599538642913, + 0.33187499502673745, + 0.33579100272618234, + 0.33599999733269215, + 0.3345000004628673, + 0.3396669926587492, + 0.32508300500921905, + 0.334249998559244, + 0.334249998559244, + 0.3365419979672879, + 0.32645900500938296, + 0.32175000524148345, + 0.32870798895601183, + 0.3302080003777519, + 0.3295420028734952, + 0.3327919985167682, + 0.3299999953014776, + 0.33487500331830233, + 0.31799999123904854, + 0.33575000998098403, + 0.33583400363568217, + 0.33887499012053013, + 0.34141600190196186, + 0.33162500767502934, + 0.3369999903952703, + 0.33941600122489035, + 0.338790996465832, + 0.3394590021343902, + 0.342790997819975, + 0.32912500319071114, + 0.3428750060265884, + 0.3292920009698719, + 0.33820800308603793, + 0.3401669964659959, + 0.33616600558161736, + 0.3314159985166043, + 0.33362500835210085, + 0.32895899494178593, + 0.3417080006329343, + 0.33237499883398414, + 0.33762499515432864, + 0.3335410001454875, + 0.33933299710042775 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.00038899999344721437, + 0.00038558300002478063, + 0.0003562920028343797, + 0.0003482920001260936, + 0.0003754170029424131, + 0.00033120799344033003, + 0.00033799999800976366, + 0.0003322909906273708, + 0.00033683299261610955, + 0.0003149999974993989, + 0.00032258300052490085, + 0.0003260000084992498, + 0.00032862499938346446, + 0.0003218329948140308, + 0.0004750420048367232, + 0.000490624996018596, + 0.00032783299684524536, + 0.00033395799982827157, + 0.00032654199458193034, + 0.00032137500238604844, + 0.00031437500729225576, + 0.00034066599619109184, + 0.00032249999640043825, + 0.0003297500079497695, + 0.00033599999733269215, + 0.0003345000004628673, + 0.00032720800663810223, + 0.00032500000088475645, + 0.00033070798963308334, + 0.0003214999887859449, + 0.000324332999298349, + 0.00032645900500938296, + 0.00032087499857880175, + 0.0003271669993409887, + 0.00031541600765194744, + 0.00031620801019016653, + 0.0003259170043747872, + 0.00033841699769254774, + 0.00033912499202415347, + 0.0003177920007146895, + 0.000319041995680891, + 0.0003224160027457401, + 0.0003362080024089664, + 0.00034141600190196186, + 0.0003187080001225695, + 0.0003369999903952703, + 0.00033941600122489035, + 0.0003243749961256981, + 0.00033395799982827157, + 0.00034258300729561597, + 0.00032912500319071114, + 0.0003428750060265884, + 0.0003343330026837066, + 0.00034266599686816335, + 0.0003344160068081692, + 0.00033616600558161736, + 0.0003406250034458935, + 0.0003273329930379987, + 0.0003277080104453489, + 0.0003439169959165156, + 0.000338249999913387, + 0.00033762499515432864, + 0.0003335410001454875, + 0.00033141700259875506 + ], + [ + 0.0003512499970383942, + 0.00036366700078360736, + 0.00034170800063293427, + 0.0003404999879421666, + 0.0003331659972900525, + 0.0003320840041851625, + 0.0003537500015227124, + 0.00033866601006593555, + 0.0003452500095590949, + 0.0003350830083945766, + 0.0003228330024285242, + 0.0003477910067886114, + 0.0003363339928910136, + 0.0003284580016043037, + 0.0003309169987915084, + 0.0003349170001456514, + 0.0003391250065760687, + 0.000338249999913387, + 0.00032662499870639294, + 0.00031725000008009374, + 0.00032916599593590945, + 0.00032004200329538435, + 0.00037900000461377203, + 0.0003727919975062832, + 0.0003897499991580844, + 0.000344959000358358, + 0.0003396669926587492, + 0.0003415829996811226, + 0.000334249998559244, + 0.0003394590021343902, + 0.0003365419979672879, + 0.0003227499983040616, + 0.00033845799043774605, + 0.00033349999284837395, + 0.0003302080003777519, + 0.0003295420028734952, + 0.0003235830081393942, + 0.0003299999953014776, + 0.0003342500131111592, + 0.0003160410124110058, + 0.00032379099866375327, + 0.0003380419948371127, + 0.00033887499012053013, + 0.0003453329991316423, + 0.00033162500767502934, + 0.0003153750003548339, + 0.00033812499896157533, + 0.000338790996465832, + 0.0003334159991936758, + 0.000342790997819975, + 0.0003362079878570512, + 0.0003345000004628673, + 0.0003292920009698719, + 0.00033799999800976366, + 0.00034212500031571835, + 0.0003325419966131449, + 0.0003311249893158674, + 0.00033362500835210085, + 0.00033858300594147295, + 0.00033441701089031994, + 0.00033237499883398414, + 0.0003257080097682774, + 0.00031883300107438117, + 0.00033933299710042775 + ], + [ + 0.0003261669917264953, + 0.00034366699401289225, + 0.00033687500399537385, + 0.00032145899604074657, + 0.0003256250056438148, + 0.00033704101224429905, + 0.0003302080003777519, + 0.00032449999707750976, + 0.0003282079997006804, + 0.00032858400663826615, + 0.0003292499895906076, + 0.0003486250061541796, + 0.0003656249900814146, + 0.0003600830095820129, + 0.00033608300145715475, + 0.00032862499938346446, + 0.0003309580060886219, + 0.0003421669971430674, + 0.00032324998755939305, + 0.00031966700043994933, + 0.00037408299976959825, + 0.0003333750064484775, + 0.00033187499502673745, + 0.0003307919978396967, + 0.00032337500306311995, + 0.00031791700166650116, + 0.0003434169921092689, + 0.0003378340043127537, + 0.00033429200993850827, + 0.0003354579966980964, + 0.00042158400174230337, + 0.0003747080045286566, + 0.00046424999891314656, + 0.0003913749969797209, + 0.0003624579985626042, + 0.00037066699587740004, + 0.0003462089953245595, + 0.00032541599648538977, + 0.00033487500331830233, + 0.0003510409878799692, + 0.0003378330002306029, + 0.00033583400363568217, + 0.00034095799492206424, + 0.0003337920061312616, + 0.0003324169956613332, + 0.00034016599238384515, + 0.0003704999980982393, + 0.00039616599678993225, + 0.00037166599940974265, + 0.0003679999936139211, + 0.00036187500518281013, + 0.00034587499976623803, + 0.00034583298838697374, + 0.00036629200621973723, + 0.00034195900661870837, + 0.00034191600570920855, + 0.0003314159985166043, + 0.00034466599754523486, + 0.0003486659988993779, + 0.00035645799653138965, + 0.0003451669908827171, + 0.0003259590012021363, + 0.0003443750028964132, + 0.0003472079988569021 + ], + [ + 0.0003566249943105504, + 0.0003420830034883693, + 0.0003429999924264848, + 0.0003412500082049519, + 0.0003316249931231141, + 0.0003333329950692132, + 0.0003296250069979578, + 0.0003149580006720498, + 0.0003306659928057343, + 0.00033150000672321767, + 0.0003277079958934337, + 0.00032824999652802944, + 0.00032824999652802944, + 0.00041537500510457903, + 0.0003400839923415333, + 0.00034683400008361787, + 0.0003481659950921312, + 0.0003316670045023784, + 0.00033962499583140016, + 0.0003404999879421666, + 0.000334084004862234, + 0.00033291599538642913, + 0.000337499994202517, + 0.00033579100272618234, + 0.0003157500032102689, + 0.00033758400240913033, + 0.0003367909957887605, + 0.00032508300500921905, + 0.000340958999004215, + 0.0003268750006100163, + 0.00037237499782349914, + 0.00033120799344033003, + 0.0003169999981764704, + 0.00032870798895601183, + 0.00033325000549666584, + 0.0003152919962303713, + 0.0003327919985167682, + 0.0003232500021113083, + 0.00031445799686480314, + 0.00033204100327566266, + 0.00033837500086519867, + 0.00032958301017060876, + 0.00035604200093075633, + 0.0004552909958874807, + 0.0005972910003038123, + 0.0005097090033814311, + 0.000428917002864182, + 0.0003617080074036494, + 0.00035424999077804387, + 0.00034583298838697374, + 0.00032904199906624854, + 0.00034425000194460154, + 0.000328291003825143, + 0.00033820800308603793, + 0.0003401669964659959, + 0.000341124992701225, + 0.0003416660038055852, + 0.00034833299287129194, + 0.00032895899494178593, + 0.00034170800063293427, + 0.0003301660035504028, + 0.0003836250107269734, + 0.00037487500230781734, + 0.0003544589999364689 + ], + [ + 0.000338790996465832, + 0.0003427500050747767, + 0.0003113749990006909, + 0.0003191249998053536, + 0.0003207910049241036, + 0.0003308750019641593, + 0.00034241699904669076, + 0.0003304999991087243, + 0.00032220799766946584, + 0.0003446249902481213, + 0.0003398749977350235, + 0.00033000000985339284, + 0.00032491699676029384, + 0.00033495899697300047, + 0.00034316600067541003, + 0.0003330000035930425, + 0.00032379099866375327, + 0.0003257080097682774, + 0.000373333998140879, + 0.0003383330040378496, + 0.0003240410005673766, + 0.0003252080059610307, + 0.0003233749885112047, + 0.00033891599741764367, + 0.00034258299274370074, + 0.0003176660102326423, + 0.0003415829996811226, + 0.00031895899155642837, + 0.0003220830112695694, + 0.000334249998559244, + 0.00032045799889601767, + 0.00032470800215378404, + 0.00032175000524148345, + 0.00031795899849385023, + 0.0003183750086463988, + 0.00034179200883954763, + 0.0003689580044010654, + 0.00033066699688788503, + 0.0003389170014997944, + 0.00031799999123904854, + 0.00033575000998098403, + 0.0003399580018594861, + 0.0003387090109754354, + 0.0003281659883214161, + 0.0003309579915367067, + 0.00032520899549126625, + 0.0003390420024516061, + 0.00033570799860171974, + 0.0003394590021343902, + 0.0003326250007376075, + 0.0003267919964855537, + 0.00032862499938346446, + 0.00031629200384486467, + 0.0003341659903526306, + 0.0003284159902250394, + 0.00033483299193903804, + 0.000327041998389177, + 0.0003320840041851625, + 0.0003279999946244061, + 0.0003337499947519973, + 0.00032750000536907464, + 0.00035391699930187315, + 0.0003325830039102584, + 0.00033333399915136397 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + ] + } + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 0.3192919975845143, + "p95": 0.33880820556078106, + "min": 0.3012910019606352, + "max": 0.3391250065760687, + "iqr": 0.024831999326124787, + "mean": 0.3219916019588709, + "cv": 0.05049239377339302 + }, + "decode_ms_per_step": { + "count": 315, + "median": 0.7549159927293658, + "p95": 1.0400037033832632, + "min": 0.4583330010063946, + "max": 1.1500410037115216, + "iqr": 0.38318699807859957, + "mean": 0.7421545938418675, + "cv": 0.26598369660722043 + }, + "total_ms": { + "count": 5, + "median": 46.86532796768006, + "p95": 47.795719967689365, + "min": 46.56225100916345, + "max": 47.93770096148364, + "iqr": 0.4322168533690274, + "mean": 47.07773101399653, + "cv": 0.011401900394782423 + }, + "tokens_per_second": { + "count": 5, + "median": 1365.6151098342168, + "p95": 1373.133307449126, + "min": 1335.0661111475054, + "max": 1374.5039943924705, + "iqr": 12.516392285279608, + "mean": 1359.593988488082, + "cv": 0.011305023101199758 + }, + "median_step_ms_by_index": [ + 0.3192919975845143, + 0.4946670087520033, + 0.4844169889111072, + 0.46887500502634794, + 0.4791669925907627, + 0.49283298722002655, + 0.4920840001432225, + 0.49412499356549233, + 0.5007500003557652, + 0.5240000027697533, + 0.5257919983705506, + 0.5296250019455329, + 0.5355829925974831, + 0.5483339919010177, + 0.5656659923261032, + 0.5448329902719706, + 0.5752499855589122, + 0.5614169931504875, + 0.5586249899351969, + 0.5616660055238754, + 0.5777079932158813, + 0.5841660022269934, + 0.5844169936608523, + 0.5890839966014028, + 0.5922089912928641, + 0.5956250097369775, + 0.6066250061849132, + 0.6167090032249689, + 0.6269579898798838, + 0.6237500056158751, + 0.6326660077320412, + 0.6210410065250471, + 0.7687919860472903, + 0.7593750051455572, + 0.7660830015083775, + 0.7698749977862462, + 0.7712499937042594, + 0.785832991823554, + 0.7910830026958138, + 0.7899999909568578, + 0.799749992438592, + 0.811707999673672, + 0.9405000018887222, + 0.9388330072397366, + 0.9380830015288666, + 0.9517080034129322, + 0.9525000059511513, + 0.919541998882778, + 0.9294159972341731, + 0.9388750040670857, + 0.9440830035600811, + 0.9525409986963496, + 0.942915998166427, + 0.9693340107332915, + 0.9577919990988448, + 0.948874992900528, + 1.0032090067397803, + 1.0194159985985607, + 1.0162499966099858, + 1.0134579933946952, + 1.0243330034427345, + 1.0456669988343492, + 1.051541999913752, + 1.042749994667247 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0003391250065760687, + 0.0005045409925514832, + 0.000485416006995365, + 0.00046887500502634794, + 0.0004730420041596517, + 0.00048504199367016554, + 0.0004872919962508604, + 0.0004899589985143393, + 0.0005145409959368408, + 0.0005055829969933257, + 0.0005309169937390834, + 0.0005281249905237928, + 0.0005296250019455329, + 0.000542792011401616, + 0.0005539999983739108, + 0.0005327920080162585, + 0.0005526250024558976, + 0.0005614169931504875, + 0.0005567920015892014, + 0.0005605839978670701, + 0.0005777079932158813, + 0.0005776250036433339, + 0.000583708009799011, + 0.0005898750096093863, + 0.0005922089912928641, + 0.0005952919891569763, + 0.0005990000063320622, + 0.0006043750036042184, + 0.0006081659957999364, + 0.0006082920008338988, + 0.0006135830044513568, + 0.0006204579985933378, + 0.0007770000083837658, + 0.0007525000110035762, + 0.0007660830015083775, + 0.0007645409932592884, + 0.0007722909940639511, + 0.00078524999844376, + 0.0007891249988460913, + 0.0007899999909568578, + 0.0007989160076249391, + 0.0008037920051719993, + 0.0009199999913107604, + 0.0009143749921349809, + 0.0009380830015288666, + 0.0009867920016404241, + 0.0009636670001782477, + 0.0009077499998966232, + 0.0009165420051431283, + 0.000935290998313576, + 0.0009440830035600811, + 0.0009547500085318461, + 0.0009459579887334257, + 0.0009612090070731938, + 0.0009435840038349852, + 0.0009573330025887117, + 0.0010032090067397803, + 0.0009901669982355088, + 0.001007749990094453, + 0.0010113330063177273, + 0.0010186669969698414, + 0.001068666999344714, + 0.001057417000993155, + 0.001042749994667247 + ], + [ + 0.0003012910019606352, + 0.0004794999986188486, + 0.0004720410070149228, + 0.00046550000843126327, + 0.00048212498950306326, + 0.00049533300625626, + 0.0005089590122224763, + 0.0004906250105705112, + 0.0005001250101486221, + 0.0005240000027697533, + 0.0005257919983705506, + 0.0005246670043561608, + 0.0005357079935492948, + 0.0005629160004900768, + 0.0005918339884374291, + 0.0005390410078689456, + 0.0005573750095209107, + 0.000548875003005378, + 0.0005611250089714304, + 0.0005616660055238754, + 0.0005759580089943483, + 0.0005841660022269934, + 0.0005809160065837204, + 0.000586916008614935, + 0.0005910830077482387, + 0.0005956250097369775, + 0.0006012499943608418, + 0.0006167090032249689, + 0.0006269579898798838, + 0.0006120409962022677, + 0.0006326660077320412, + 0.0006210410065250471, + 0.0007785000052535906, + 0.000771167004131712, + 0.0007788749935571104, + 0.0007698749977862462, + 0.0007712499937042594, + 0.000789249999797903, + 0.000800499998149462, + 0.0007946660043671727, + 0.0008039999956963584, + 0.0008090829942375422, + 0.0009507089998805895, + 0.0009179170010611415, + 0.000985999999102205, + 0.0009517080034129322, + 0.0009728340082801878, + 0.000919541998882778, + 0.0009445410105399787, + 0.0009388750040670857, + 0.0009374169894726947, + 0.0009470000077271834, + 0.0009304580016760156, + 0.0009694170003058389, + 0.0009737090003909543, + 0.0009695420012576506, + 0.0010076670005219057, + 0.000994375004665926, + 0.000993000008747913, + 0.0010252089996356517, + 0.0010243330034427345, + 0.0010413749987492338, + 0.0010356670100009069, + 0.001043290991219692 + ], + [ + 0.0003127090021735057, + 0.0005142919981153682, + 0.0004844169889111072, + 0.00046983400534372777, + 0.00047770798846613616, + 0.0004929169954266399, + 0.0004920840001432225, + 0.0004941249935654923, + 0.0004967499990016222, + 0.0005523750005522743, + 0.0005252499977359548, + 0.0005301250057527795, + 0.000528125005075708, + 0.0005481249972945079, + 0.0005656659923261032, + 0.0005448329902719706, + 0.0005752499855589122, + 0.0005370839935494587, + 0.0005450419994303957, + 0.000556166996830143, + 0.0005701249901903793, + 0.0005690840043826029, + 0.000585457994020544, + 0.0005890839966014028, + 0.0005933750071562827, + 0.000586958005442284, + 0.0006175410089781508, + 0.0006175409944262356, + 0.0006404999876394868, + 0.0006451670051319525, + 0.0006510410021292046, + 0.0006640830106334761, + 0.0007687919860472903, + 0.0007593750051455572, + 0.0007496670004911721, + 0.0007581249956274405, + 0.0007635000074515119, + 0.0007842909981263801, + 0.0007884160004323348, + 0.0007873750000726432, + 0.0007933330052765086, + 0.0008262920018751174, + 0.0009279999940190464, + 0.0009388330072397366, + 0.000933166011236608, + 0.0009426250035176054, + 0.0009283330000471324, + 0.0009259170037694275, + 0.000921958009712398, + 0.0009521250030957162, + 0.0009439159912290052, + 0.0009612909925635904, + 0.0009439999994356185, + 0.0009497080027358606, + 0.0009657909977249801, + 0.000948874992900528, + 0.0010929159907391295, + 0.0010527080012252554, + 0.0010220000112894922, + 0.0010134579933946952, + 0.0010135830088984221, + 0.0010324580071028322, + 0.0010516250040382147, + 0.0010500409989617765 + ], + [ + 0.0003375410014996305, + 0.0004946670087520033, + 0.0005146659968886524, + 0.0005055000074207783, + 0.0004889580013696104, + 0.0004928329872200266, + 0.0005149999924469739, + 0.0004966660053469241, + 0.0005007500003557652, + 0.0005164589965716004, + 0.000519584005814977, + 0.0005361669900594279, + 0.0005575410032179207, + 0.0005491670017363504, + 0.0006977919983910397, + 0.000758999987738207, + 0.0006917499995324761, + 0.0005748749972553924, + 0.0005692080012522638, + 0.0005745420057792217, + 0.0005946669989498332, + 0.0005889589956495911, + 0.0005913330096518621, + 0.000653874987619929, + 0.000661666999803856, + 0.0006217500049388036, + 0.0006244999967748299, + 0.0006398749974323437, + 0.0006300839886534959, + 0.0006443330057663843, + 0.0006743330013705418, + 0.0006584579969057813, + 0.0007524159882450476, + 0.0007549159927293658, + 0.000763917007134296, + 0.0007748329953756183, + 0.0007814999989932403, + 0.0007914160087238997, + 0.0007970419974299148, + 0.000793625004007481, + 0.0008095829980447888, + 0.000811707999673672, + 0.0009405000018887222, + 0.000996874994598329, + 0.0009575419971952215, + 0.0009433329978492111, + 0.0009266660053981468, + 0.0009135000000242144, + 0.0009294159972341731, + 0.000939666002523154, + 0.0010394160053692758, + 0.0009525409986963496, + 0.000942915998166427, + 0.0009696250053821132, + 0.0009552920091664419, + 0.0009486659982940182, + 0.000993541005300358, + 0.0010194159985985607, + 0.001049624988809228, + 0.001044292002916336, + 0.0010309579956810921, + 0.001053416999639012, + 0.001051541999913752, + 0.001031499996315688 + ], + [ + 0.0003192919975845143, + 0.00047633399663027376, + 0.00046558299800381064, + 0.0004583330010063946, + 0.0004791669925907627, + 0.00048662499466445297, + 0.0004849580000154674, + 0.0004969590081600472, + 0.0005082080024294555, + 0.0005290830013109371, + 0.0005259169993223622, + 0.0005296250019455329, + 0.0005355829925974831, + 0.0005483339919010177, + 0.0005576249968726188, + 0.0005533329967875034, + 0.0005818339996039867, + 0.0006012499943608418, + 0.0005586249899351969, + 0.0005764169909525663, + 0.0005795839970232919, + 0.000586207999731414, + 0.0005844169936608523, + 0.0005877500079805031, + 0.0005890830070711672, + 0.0006026670016581193, + 0.0006066250061849132, + 0.0006107090011937544, + 0.0006018339918227866, + 0.0006237500056158751, + 0.000631584000075236, + 0.0006168749969219789, + 0.0007549580041086301, + 0.000811707999673672, + 0.0008027089934330434, + 0.0007790410018060356, + 0.0007659999973839149, + 0.000785832991823554, + 0.0007910830026958138, + 0.000786375007010065, + 0.000799749992438592, + 0.0008533329964848235, + 0.0011500410037115216, + 0.0010285000025760382, + 0.000934209005208686, + 0.0009559590107528493, + 0.0009525000059511513, + 0.0009811659983824939, + 0.0009770830074558035, + 0.0009204160014633089, + 0.000982041994575411, + 0.0009518749866401777, + 0.0009348750027129427, + 0.0009693340107332915, + 0.0009577919990988448, + 0.0009471670055063441, + 0.0009950840030796826, + 0.001025207995553501, + 0.0010162499966099858, + 0.0010075839963974431, + 0.001058500012732111, + 0.0010456669988343492, + 0.00102954100293573, + 0.0009820420091273263 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_mha": { + "first_token_ms": { + "count": 5, + "median": 0.3918750007869676, + "p95": 0.44569159217644483, + "min": 0.3564169892342761, + "max": 0.457499991171062, + "iqr": 0.018457983969710767, + "mean": 0.39684999792370945, + "cv": 0.0944900104454673 + }, + "decode_ms_per_step": { + "count": 315, + "median": 0.8230829989770427, + "p95": 1.1861832972499542, + "min": 0.45841699466109276, + "max": 1.3775410043308511, + "iqr": 0.41620800038799644, + "mean": 0.8504276155560676, + "cv": 0.26858171686554255 + }, + "total_ms": { + "count": 5, + "median": 54.160872954525985, + "p95": 54.71894278016407, + "min": 52.702079934533685, + "max": 54.826376974233426, + "iqr": 0.3987929812865332, + "mean": 53.97378977795597, + "cv": 0.014605009431057083 + }, + "tokens_per_second": { + "count": 5, + "median": 1181.664853403213, + "p95": 1209.0177147939703, + "min": 1167.3213429747123, + "max": 1214.3733241553377, + "iqr": 8.723735270353927, + "mean": 1185.9652679919818, + "cv": 0.014762446280799013 + }, + "median_step_ms_by_index": [ + 0.3918750007869676, + 0.49325000145472586, + 0.4956250049872324, + 0.5316250026226044, + 0.542207999387756, + 0.5587499908870086, + 0.5637909926008433, + 0.551959004951641, + 0.5660000024363399, + 0.5914590001339093, + 0.6036669947206974, + 0.6098340090829879, + 0.6230420112842694, + 0.6375840021064505, + 0.644458006718196, + 0.6172920111566782, + 0.6575829902430996, + 0.654249990475364, + 0.6637500046053901, + 0.6750409957021475, + 0.6782080017728731, + 0.687542007653974, + 0.6905830086907372, + 0.6961670005694032, + 0.703291007084772, + 0.710000007529743, + 0.7147920114221051, + 0.7115420012269169, + 0.7215829973574728, + 0.7312079978873953, + 0.7362920005107298, + 0.7462499925168231, + 0.846041992190294, + 0.8487909944960847, + 0.8555419917684048, + 0.869084004079923, + 0.8870830060914159, + 0.8988750050775707, + 0.8886249997885898, + 0.8925829897634685, + 0.8967499888967723, + 0.9087920043384656, + 1.033624997944571, + 1.0452919959789142, + 1.0936250037048012, + 1.0743339953478426, + 1.0457079915795475, + 1.0527080012252554, + 1.0787500068545341, + 1.0856250009965152, + 1.0837089939741418, + 1.0910420096479356, + 1.0997499921359122, + 1.1130409984616563, + 1.1061669938499108, + 1.1308329994790256, + 1.1546250025276095, + 1.1616249976214021, + 1.1554589873412624, + 1.1551250063348562, + 1.1742920032702386, + 1.2001249997410923, + 1.1910839966731146, + 1.1774579907068983 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0003800000122282654, + 0.0004862079949816689, + 0.0004956250049872324, + 0.0005316250026226044, + 0.000542207999387756, + 0.000546583003597334, + 0.0005637909926008433, + 0.0007314579997910187, + 0.0006890829972689971, + 0.0007379169983323663, + 0.0006853749946458265, + 0.0006145000079413876, + 0.0006230420112842694, + 0.0006375840021064505, + 0.0006565829971805215, + 0.0006449169886764139, + 0.0006654589960817248, + 0.0006673749885521829, + 0.00066654100373853, + 0.0006745419959770516, + 0.0006786669982830063, + 0.0006850000063423067, + 0.0006905830086907372, + 0.0007032499997876585, + 0.0006959580059628934, + 0.0007069169951137155, + 0.0007074169989209622, + 0.0007171670004026964, + 0.0007175409991759807, + 0.0007287499902304262, + 0.0007362920005107298, + 0.0007372910040430725, + 0.0009902919991873205, + 0.0009067079954547808, + 0.0008779170020716265, + 0.000880082996445708, + 0.0008870830060914159, + 0.0008906659932108596, + 0.0008886249997885898, + 0.0008925829897634685, + 0.0008989999914774671, + 0.0009008750057546422, + 0.0010292500082869083, + 0.0010975420009344816, + 0.0010936250037048012, + 0.0010733340022852644, + 0.0010419159953016788, + 0.0010527080012252554, + 0.0010800000018207356, + 0.0010813330009113997, + 0.001079166991985403, + 0.0010701669962145388, + 0.0010997499921359122, + 0.001094959006877616, + 0.0011061669938499108, + 0.001094374994863756, + 0.0011367080005584285, + 0.0011616249976214021, + 0.0011342079960741103, + 0.0011551250063348562, + 0.0011581669969018549, + 0.0012001249997410923, + 0.0011840829974971712, + 0.0011774579907068983 + ], + [ + 0.0003564169892342761, + 0.0004932500014547259, + 0.0004908330010948703, + 0.0005172920064069331, + 0.0005305420054355636, + 0.0005587499908870086, + 0.0005695419968105853, + 0.000551959004951641, + 0.0005660000024363399, + 0.0005914590001339093, + 0.000600125000346452, + 0.0006015840044710785, + 0.0006207910046214238, + 0.0006209999992279336, + 0.0006406669999705628, + 0.0006172920111566782, + 0.0006575829902430996, + 0.0006457500130636618, + 0.0006596659950446337, + 0.0006802080024499446, + 0.0006669580034213141, + 0.000665832994855009, + 0.0006904159963596612, + 0.0006939580052858219, + 0.0007062080112518743, + 0.0007064169913064688, + 0.0007162910042097792, + 0.0007099159993231297, + 0.0007215829973574728, + 0.0007282910082722083, + 0.000734957997337915, + 0.0007699589914409444, + 0.000846041992190294, + 0.0008097090030787513, + 0.0008078749960986897, + 0.0008230829989770427, + 0.0008453339978586882, + 0.0009199159976560622, + 0.0008997080003609881, + 0.0008460840035695583, + 0.0008967499888967723, + 0.0008766660030232742, + 0.0010247499885736033, + 0.0010452919959789142, + 0.0013775410043308511, + 0.001145375004853122, + 0.0010568340076133609, + 0.0010513750021345913, + 0.0010437909950269386, + 0.001153916004113853, + 0.0011630000080913305, + 0.0011245000059716403, + 0.0010912500001722947, + 0.0011130409984616563, + 0.0011053329944843426, + 0.0011377079936210066, + 0.0012352079938864335, + 0.0011968750040978193, + 0.0011742500064428896, + 0.0011811250005848706, + 0.001198333004140295, + 0.0011963750002905726, + 0.0012407920003170148, + 0.0011810840078396723 + ], + [ + 0.00039845799619797617, + 0.0004827499942621216, + 0.00045841699466109276, + 0.00046595800085924566, + 0.0004774579865625128, + 0.0004995829949621111, + 0.0005008330044802278, + 0.0005406249983934686, + 0.0005338750052032992, + 0.0005601659941021353, + 0.0006036669947206974, + 0.0006098340090829879, + 0.0006318329978967085, + 0.0006373749929480255, + 0.000644458006718196, + 0.0006110829999670386, + 0.0006443750025937334, + 0.000654249990475364, + 0.0006637500046053901, + 0.0006750409957021475, + 0.0006782080017728731, + 0.0007051250140648335, + 0.0006892079982208088, + 0.0006812089995946735, + 0.0006927909998921677, + 0.0007150000019464642, + 0.0007147920114221051, + 0.0007095409964676946, + 0.0007151250028982759, + 0.0007312079978873953, + 0.0007317090057767928, + 0.0007395409920718521, + 0.0008380420040339231, + 0.0008487909944960847, + 0.0008489999891025946, + 0.000855500009492971, + 0.0008740829944144934, + 0.0008841249946272001, + 0.00088429098832421, + 0.0008874999912222847, + 0.0008912089979276061, + 0.0009351249900646508, + 0.001033624997944571, + 0.0010432499984744936, + 0.0010273749940097332, + 0.0010388749942649156, + 0.001035875000525266, + 0.0010702500003390014, + 0.0010545830009505153, + 0.0010856250009965152, + 0.0010745830077212304, + 0.0010910420096479356, + 0.0010782500030472875, + 0.001082374990801327, + 0.0010873330029426143, + 0.0010930830030702055, + 0.0011546250025276095, + 0.0011369999992894009, + 0.0011554589873412624, + 0.001130584001657553, + 0.0011742920032702386, + 0.0011633339890977368, + 0.0011705000069923699, + 0.0011752499995054677 + ], + [ + 0.000457499991171062, + 0.0005567499902099371, + 0.0005364590033423156, + 0.0005802920059068128, + 0.0005539999983739108, + 0.000571333002881147, + 0.0005499580001924187, + 0.0005425419949460775, + 0.000567374998354353, + 0.0005802079977001995, + 0.000600125000346452, + 0.0005993750091874972, + 0.0006089999951655045, + 0.0006729159940732643, + 0.000651333000860177, + 0.0006054170080460608, + 0.0006122920021880418, + 0.000618750011199154, + 0.0006346669979393482, + 0.0006750830070814118, + 0.0006690839945804328, + 0.0006889579963171855, + 0.0007087089907145128, + 0.0006961670005694032, + 0.0007104160031303763, + 0.000710000007529743, + 0.0007123329996829852, + 0.0007115420012269169, + 0.0007251249917317182, + 0.0007328749925363809, + 0.0007424170034937561, + 0.0008070000039879233, + 0.0009682499949121848, + 0.0008682499901624396, + 0.0008658750011818483, + 0.000875749989063479, + 0.000927665998460725, + 0.0008988750050775707, + 0.000890750001417473, + 0.0009077499998966232, + 0.0009078329894691706, + 0.0009112080006161705, + 0.001061999995727092, + 0.0010762090096250176, + 0.001102374997572042, + 0.00109816700569354, + 0.0010935000027529895, + 0.0012142920022597536, + 0.0013291249924805015, + 0.0011106250021839514, + 0.0011152920051245019, + 0.001116084007662721, + 0.0011073330097133294, + 0.0011222079920116812, + 0.0011149170022690669, + 0.0011379170027794316, + 0.0011567919864319265, + 0.0012413749936968088, + 0.0011670420062728226, + 0.001143125002272427, + 0.0011644999904092401, + 0.0012182920036138967, + 0.001303041004575789, + 0.0012199579941807315 + ], + [ + 0.0003918750007869676, + 0.0005015000060666353, + 0.0005199580045882612, + 0.0005343749944586307, + 0.0005865419952897355, + 0.0006003750022500753, + 0.0005892909975955263, + 0.0005577909905696288, + 0.000563749999855645, + 0.0006453750102082267, + 0.000629082991508767, + 0.0006189579871715978, + 0.0006251250015338883, + 0.0006392500072252005, + 0.0006387909961631522, + 0.0006781250122003257, + 0.0008240000024670735, + 0.000766083991038613, + 0.0006911660020705312, + 0.000669790999381803, + 0.0006846670003142208, + 0.000687542007653974, + 0.0006981250044191256, + 0.0006964170024730265, + 0.000703291007084772, + 0.0007238329999381676, + 0.0007275419920915738, + 0.0007366249919869006, + 0.0007485829992219806, + 0.0007740839937468991, + 0.000778207991970703, + 0.0007462499925168231, + 0.0008239580056397244, + 0.0008230829989770427, + 0.0008555419917684048, + 0.000869084004079923, + 0.0008896670042304322, + 0.0008997910044854507, + 0.0008798750059213489, + 0.0009153340070042759, + 0.0008924580033635721, + 0.0009087920043384656, + 0.001039499999023974, + 0.0010389999952167273, + 0.0010353329998906702, + 0.0010743339953478426, + 0.0010457079915795475, + 0.0010199579992331564, + 0.0010787500068545341, + 0.0010833749984158203, + 0.0010837089939741418, + 0.0010827920050360262, + 0.0011106250021839514, + 0.0011792500008596107, + 0.001151874996139668, + 0.0011308329994790256, + 0.0011479580134619027, + 0.0011505840084282681, + 0.0011510829936014488, + 0.0011565410095499828, + 0.0011976249952567741, + 0.001233000002685003, + 0.0011910839966731146, + 0.0011723329953383654 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "speedup_cached_vs_uncached_sdpa": 2.166544905733736, + "speedup_cached_vs_uncached_mha": 2.503811847228369 + }, + { + "batch": 1, + "prompt_len": 16, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_mha": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 0.5945000011706725, + "p95": 0.596325597143732, + "min": 0.558999992790632, + "max": 0.5967089964542538, + "iqr": 0.005500987754203379, + "mean": 0.5868584004929289, + "cv": 0.02694752691023944 + }, + "decode_ms_per_step": { + "count": 315, + "median": 0.33437499951105565, + "p95": 0.38738289586035535, + "min": 0.2867909934138879, + "max": 0.49108399252872914, + "iqr": 0.018416496459394693, + "mean": 0.33953885047970017, + "cv": 0.07654585955896279 + }, + "total_ms": { + "count": 5, + "median": 22.00141405046452, + "p95": 22.453913959907368, + "min": 21.29658591002226, + "max": 22.4782059667632, + "iqr": 0.6006678886478767, + "mean": 21.977805980714038, + "cv": 0.02167580650421372 + }, + "tokens_per_second": { + "count": 5, + "median": 2908.903939228795, + "p95": 2992.4825534872584, + "min": 2847.202312080951, + "max": 3005.1765231478416, + "iqr": 79.0360432030966, + "mean": 2913.1320161888684, + "cv": 0.021845537626422128 + }, + "median_step_ms_by_index": [ + 0.5945000011706725, + 0.38720799784641713, + 0.34183300158474594, + 0.34133299777749926, + 0.3329580067656934, + 0.3374579973751679, + 0.33683299261610955, + 0.33433400676585734, + 0.3391250065760687, + 0.3292089968454093, + 0.3326659934828058, + 0.3284580016043037, + 0.33799999800976366, + 0.3285420098109171, + 0.334834010573104, + 0.33133300894405693, + 0.3408329939702526, + 0.32183300936594605, + 0.3276250063208863, + 0.3290830063633621, + 0.337709003360942, + 0.3293750050943345, + 0.3258749929955229, + 0.3220840007998049, + 0.32020799699239433, + 0.31954199948813766, + 0.33149999217130244, + 0.3352920030010864, + 0.33599999733269215, + 0.3371659986441955, + 0.3318329981993884, + 0.33095899561885744, + 0.3429999924264848, + 0.3320419928058982, + 0.3293750050943345, + 0.32704099430702627, + 0.3333749918965623, + 0.33245899248868227, + 0.3305830032331869, + 0.3488750080578029, + 0.34174999746028334, + 0.33379199157934636, + 0.33399999665562063, + 0.3333749918965623, + 0.32866599212866277, + 0.3235000040149316, + 0.33379199157934636, + 0.33266699756495655, + 0.33075000101234764, + 0.33754199102986604, + 0.34058299206662923, + 0.32820900378283113, + 0.3415000101085752, + 0.33887500467244536, + 0.3402090078452602, + 0.34512500860728323, + 0.3416660038055852, + 0.3362499992363155, + 0.34250000317115337, + 0.3460000007180497, + 0.3428329946473241, + 0.3421669971430674, + 0.3434160025790334, + 0.3453329991316423 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0005945000011706725, + 0.0003669159923447296, + 0.00032724998891353607, + 0.0003515410062391311, + 0.0003309580060886219, + 0.00032387500687036663, + 0.00032970900065265596, + 0.00033433400676585734, + 0.0003366250020917505, + 0.0003400409914320335, + 0.0003516249998938292, + 0.0003385419986443594, + 0.0003400410059839487, + 0.0003257080097682774, + 0.000334834010573104, + 0.00033966598857659847, + 0.0003408329939702526, + 0.00032183300936594605, + 0.00034558399056550115, + 0.00031866699282545596, + 0.00033483300649095327, + 0.0003387080068932846, + 0.0003217089979443699, + 0.00031695800134912133, + 0.0003165829984936863, + 0.0003284580016043037, + 0.00032491701131220907, + 0.00033708399860188365, + 0.00034441601019352674, + 0.0003437080013100058, + 0.00033629199606366456, + 0.00032291699608322233, + 0.0003429999924264848, + 0.0003326250007376075, + 0.000331375005771406, + 0.0003154579899273813, + 0.0003255829942645505, + 0.00031962500361260027, + 0.00032308300433214754, + 0.0003542500053299591, + 0.0003414170059841126, + 0.0003549999964889139, + 0.0003540420002536848, + 0.00033024999720510095, + 0.0003473329998087138, + 0.0003387079923413694, + 0.0003489580121822655, + 0.0003411250072531402, + 0.00034012499963864684, + 0.00034404100733809173, + 0.00035245799517724663, + 0.00033754200558178127, + 0.0003415000101085752, + 0.00033887500467244536, + 0.0003286250139353797, + 0.0003518330049701035, + 0.0003479589940980077, + 0.0003507920046104118, + 0.0003802919964073226, + 0.000385499995900318, + 0.0003664999967440963, + 0.0003604160010581836, + 0.00035587498859968036, + 0.0003575839946279302 + ], + [ + 0.0005947919999016449, + 0.0003877909912262112, + 0.0003606249956646934, + 0.00034133299777749926, + 0.0003329580067656934, + 0.000359957994078286, + 0.0003619170020101592, + 0.0003421660076128319, + 0.0003708750009536743, + 0.0003326250007376075, + 0.0003326659934828058, + 0.00032741700124461204, + 0.00032212500809691846, + 0.00034550001146271825, + 0.000375375006115064, + 0.0004395419964566827, + 0.0004257079999661073, + 0.00041791600233409554, + 0.0004623339918907732, + 0.00047112499305512756, + 0.00037458300357684493, + 0.00032824999652802944, + 0.0003235420008422807, + 0.00034262500412296504, + 0.00032170799386221915, + 0.00031550000130664557, + 0.00033149999217130244, + 0.000332666008034721, + 0.00033599999733269215, + 0.00031925000075716525, + 0.0004007499956060201, + 0.0003792909992625937, + 0.0003522500046528876, + 0.0003320419928058982, + 0.0003293750050943345, + 0.0003341670089866966, + 0.00032070899032987654, + 0.00033245899248868227, + 0.0003345419972902164, + 0.00031362500158138573, + 0.00032791600096970797, + 0.0003220840007998049, + 0.0003145829978166148, + 0.0003322080010548234, + 0.00032866599212866277, + 0.0003124170034425333, + 0.0003195000026607886, + 0.0003361249982845038, + 0.00033120799344033003, + 0.00033645798976067454, + 0.0003369579935679212, + 0.00031091600249055773, + 0.0003194579912815243, + 0.0003329580067656934, + 0.0003402079892111942, + 0.0003304169949842617, + 0.00033387499570380896, + 0.00033333399915136397, + 0.00033291599538642913, + 0.0003297500079497695, + 0.0003256249910918996, + 0.0003421669971430674, + 0.0003362499992363155, + 0.0003311670006951317 + ], + [ + 0.000558999992790632, + 0.0003715839993674308, + 0.00034041699836961925, + 0.0003272500034654513, + 0.00032199999259319156, + 0.00032500000088475645, + 0.00033683299261610955, + 0.0003285830025561154, + 0.0003391250065760687, + 0.00032708300568629056, + 0.00032358399766962975, + 0.0003284580016043037, + 0.00033799999800976366, + 0.00033525000617373735, + 0.00032979199022520334, + 0.00033133300894405693, + 0.0003650419967016205, + 0.00035862500953953713, + 0.00030262500513345003, + 0.00030345800041686743, + 0.00029883299430366606, + 0.00032737500441726297, + 0.0003333749918965623, + 0.0003212910087313503, + 0.00030258299375418574, + 0.0002867909934138879, + 0.00029562501003965735, + 0.00030029199842829257, + 0.00030266599787864834, + 0.0002918329992098734, + 0.0003215420001652092, + 0.00033095899561885744, + 0.00032700000156182796, + 0.00032333299168385565, + 0.000303833992802538, + 0.0003104579955106601, + 0.00040120900666806847, + 0.00035483299870975316, + 0.00031954199948813766, + 0.00037304199940990657, + 0.0003721659886650741, + 0.0004910839925287291, + 0.00038033399323467165, + 0.00035091699101030827, + 0.00031841700547374785, + 0.00031379199936054647, + 0.0003070840029977262, + 0.0003105000068899244, + 0.00031216700153890997, + 0.00031149999995250255, + 0.00030841698753647506, + 0.000310208008158952, + 0.0003452499950071797, + 0.00034495799627620727, + 0.0003455419937381521, + 0.00033516700204927474, + 0.0003149169933749363, + 0.00031200000375974923, + 0.00031608299468643963, + 0.0003174159937771037, + 0.00031550000130664557, + 0.000312374992063269, + 0.0003169169940520078, + 0.0003144169895676896 + ], + [ + 0.0005967089964542538, + 0.00039541600563097745, + 0.0004055419994983822, + 0.00039320799987763166, + 0.0003362499992363155, + 0.00034337499528191984, + 0.000350583010003902, + 0.0003345830045873299, + 0.000341874998412095, + 0.0003217499906895682, + 0.00033445800363551825, + 0.0003276670031482354, + 0.00033874998916871846, + 0.0003285420098109171, + 0.0003100000030826777, + 0.0003183329972671345, + 0.00033166600042022765, + 0.000315624987706542, + 0.0003267499996582046, + 0.00033941600122489035, + 0.00034891699033323675, + 0.00033149999217130244, + 0.0003258749929955229, + 0.0003220840007998049, + 0.00032020799699239433, + 0.00032287499925587326, + 0.00034174999746028334, + 0.0003352920030010864, + 0.00034004201006609946, + 0.0003393749939277768, + 0.0003318329981993884, + 0.00033095899561885744, + 0.00037900000461377203, + 0.0003680829977383837, + 0.00034949999826494604, + 0.0003317920054541901, + 0.00034949999826494604, + 0.0003510829992592335, + 0.0003344169963384047, + 0.0003292079927632585, + 0.0003452499950071797, + 0.0003232910094084218, + 0.0003235830081393942, + 0.00034570900606922805, + 0.0003214579919585958, + 0.0003235000040149316, + 0.0003477080026641488, + 0.00033266699756495655, + 0.00033075000101234764, + 0.00035033299354836345, + 0.00037779100239276886, + 0.00039083299634512514, + 0.0004570419987430796, + 0.0004106660053366795, + 0.00041237499681301415, + 0.0003646669938461855, + 0.000360332996933721, + 0.0003549170069163665, + 0.00035400000342633575, + 0.0003541670012054965, + 0.0003597080067265779, + 0.000350708985934034, + 0.000357083001290448, + 0.0003558750031515956 + ], + [ + 0.0005892910121474415, + 0.00038720799784641713, + 0.00034183300158474594, + 0.00033266699756495655, + 0.00033437499951105565, + 0.0003374579973751679, + 0.0003256659983890131, + 0.00033299998904112726, + 0.00033399999665562063, + 0.0003292089968454093, + 0.0003265830018790439, + 0.0003355830122018233, + 0.00033145799534395337, + 0.0003276250063208863, + 0.00034054099523928016, + 0.00032391599961556494, + 0.0003334579960210249, + 0.0003179580089636147, + 0.0003276250063208863, + 0.0003290830063633621, + 0.000337709003360942, + 0.0003293750050943345, + 0.00033145899942610413, + 0.00035612500505521894, + 0.00033908399927895516, + 0.00031954199948813766, + 0.0003335000074002892, + 0.00033754200558178127, + 0.0003316249931231141, + 0.0003371659986441955, + 0.0003305830032331869, + 0.00034245899587403983, + 0.0003398749977350235, + 0.0003287919971626252, + 0.000328500012983568, + 0.00032704099430702627, + 0.0003333749918965623, + 0.00033120799344033003, + 0.0003305830032331869, + 0.0003488750080578029, + 0.00034174999746028334, + 0.00033379199157934636, + 0.00033399999665562063, + 0.0003333749918965623, + 0.0003465409972704947, + 0.0003383330040378496, + 0.00033379199157934636, + 0.0003308750019641593, + 0.0003289580054115504, + 0.00033754199102986604, + 0.00034058299206662923, + 0.00032820900378283113, + 0.00033441701089031994, + 0.00033570801315363497, + 0.0003402090078452602, + 0.00034512500860728323, + 0.0003416660038055852, + 0.0003362499992363155, + 0.00034250000317115337, + 0.0003460000007180497, + 0.0003428329946473241, + 0.00034191600570920855, + 0.0003434160025790334, + 0.0003453329991316423 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + ] + } + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 0.5715000006603077, + "p95": 0.6272670056205243, + "min": 0.5170000076759607, + "max": 0.6396670069079846, + "iqr": 0.014542005374096334, + "mean": 0.5737918021623045, + "cv": 0.07645924889926016 + }, + "decode_ms_per_step": { + "count": 315, + "median": 0.9562500054016709, + "p95": 1.2151919057941996, + "min": 0.49649999709799886, + "max": 1.3765000039711595, + "iqr": 0.43233399628661573, + "mean": 0.9086044353713828, + "cv": 0.24615456427931992 + }, + "total_ms": { + "count": 5, + "median": 57.69803503062576, + "p95": 58.769446617225185, + "min": 56.93586802226491, + "max": 58.897371010971256, + "iqr": 0.9674159955466166, + "mean": 57.815871230559424, + "cv": 0.013479400648516835 + }, + "tokens_per_second": { + "count": 5, + "median": 1109.2232164583975, + "p95": 1122.6807712886641, + "min": 1086.6359380977844, + "max": 1124.0717358515135, + "iqr": 18.55060980444523, + "mean": 1107.1228213355566, + "cv": 0.013431347616927318 + }, + "median_step_ms_by_index": [ + 0.5715000006603077, + 0.5726249946746975, + 0.5617499991785735, + 0.5674580024788156, + 0.567708004382439, + 0.578582999878563, + 0.585292000323534, + 0.5964999872958288, + 0.5931250052526593, + 0.6241660012165084, + 0.6260840018512681, + 0.6302090041572228, + 0.6193329900270328, + 0.628875000984408, + 0.632499999483116, + 0.6442919984692708, + 0.6457909912569448, + 0.748874997952953, + 0.7590000022901222, + 0.7639590039616451, + 0.7621660042786971, + 0.7721250003669411, + 0.781790993642062, + 0.792499995441176, + 0.7864590006647632, + 0.8100410050246865, + 0.806582989753224, + 0.9319590026279911, + 0.929166009882465, + 0.9123750060098246, + 0.9385000012116507, + 0.9444160095881671, + 0.9710410085972399, + 0.9587920067133382, + 0.9458330023335293, + 0.9704580006655306, + 0.9586249943822622, + 0.9592080023139715, + 0.9786659938981757, + 0.9739160013850778, + 0.9711249877000228, + 1.0169159941142425, + 1.0214159992756322, + 1.0135000047739595, + 1.0357909923186526, + 1.03629200020805, + 1.0363330075051636, + 1.02954100293573, + 1.0337499988963827, + 1.180834005936049, + 1.1858329962706193, + 1.1788339907070622, + 1.1923749989364296, + 1.1822499945992604, + 1.1694580025505275, + 1.1914170027012005, + 1.1552910000318661, + 1.141209009801969, + 1.1365420068614185, + 1.2059580039931461, + 1.14545899850782, + 1.1565000022528693, + 1.153375007561408, + 1.1417909990996122 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.000577667000470683, + 0.0005726249946746975, + 0.0005887080042157322, + 0.0005840000085299835, + 0.0005913750064792112, + 0.0006052089884178713, + 0.0005964160081930459, + 0.0006003750022500753, + 0.0006134580034995452, + 0.0006128749955678359, + 0.0006189169944263995, + 0.0006309169984888285, + 0.0006291250028880313, + 0.000628875000984408, + 0.000632499999483116, + 0.0006474580004578456, + 0.000676875002682209, + 0.0007401670009130612, + 0.0007624999998370185, + 0.0007665419980185106, + 0.0007621660042786971, + 0.0007683750009164214, + 0.0007657079986529425, + 0.0007818330050213262, + 0.0007821250037522987, + 0.000797250002506189, + 0.0007977080094860867, + 0.0009313339978689328, + 0.0008966669993242249, + 0.0009065000049304217, + 0.0009071250096894801, + 0.0009192919969791546, + 0.0010067080002045259, + 0.0009625000093365088, + 0.0009400840062880889, + 0.0010042499925475568, + 0.0009549160022288561, + 0.0009526659996481612, + 0.0009866670006886125, + 0.0009860409918474033, + 0.0009660409996286035, + 0.0010169159941142425, + 0.0010214159992756322, + 0.0010135000047739595, + 0.0012092079996364191, + 0.0011843749962281436, + 0.0010334589896956459, + 0.00102954100293573, + 0.0010337499988963827, + 0.001180834005936049, + 0.0011687920050462708, + 0.0011245420027989894, + 0.0011125000019092113, + 0.0011822499945992604, + 0.0011694580025505275, + 0.0011468339944258332, + 0.0011252499971305951, + 0.0011285419896012172, + 0.0011403750104364008, + 0.001229417001013644, + 0.0011436670029070228, + 0.0011510420008562505, + 0.001153375007561408, + 0.001138749998062849 + ], + [ + 0.0006396670069079846, + 0.0006115829892223701, + 0.000599374994635582, + 0.0005734580045100302, + 0.0005734170117648318, + 0.000578582999878563, + 0.000585292000323534, + 0.0005964999872958288, + 0.0005931250052526593, + 0.0007027079991530627, + 0.0006460830045398325, + 0.0006162500067148358, + 0.000604083004873246, + 0.0006051669915905222, + 0.0006193750014062971, + 0.0006172499997774139, + 0.0006200000061653554, + 0.0007594159978907555, + 0.0007639590039616451, + 0.0007664580043638125, + 0.0007887080137152225, + 0.0008820840012049302, + 0.0008134999952744693, + 0.000792499995441176, + 0.0007864590006647632, + 0.0007917500042822212, + 0.0007970829901751131, + 0.0009177920001093298, + 0.0010492500005057082, + 0.0011977910035056993, + 0.0010645830043358728, + 0.0009760419925441965, + 0.0009226249967468902, + 0.0009587920067133382, + 0.0010133750038221478, + 0.0009704580006655306, + 0.0009422080038348213, + 0.000948874992900528, + 0.0009786659938981757, + 0.0009725829877424985, + 0.0009834579977905378, + 0.0010171249887207523, + 0.0010497080074856058, + 0.0010720830032369122, + 0.0010462500067660585, + 0.00103629200020805, + 0.0010363330075051636, + 0.0011343340011080727, + 0.0010567080025793985, + 0.0011740829941118136, + 0.0012005840108031407, + 0.0011759170010918751, + 0.0012147920060670003, + 0.0012134580028941855, + 0.0012278750073164701, + 0.001217375000123866, + 0.0011329580011079088, + 0.001141209009801969, + 0.0011353329900885, + 0.0011581669969018549, + 0.00114545899850782, + 0.0011565000022528693, + 0.001155499994638376, + 0.0011393750028219074 + ], + [ + 0.0005170000076759607, + 0.0004964999970979989, + 0.000512166996486485, + 0.0004993750044377521, + 0.0005052909982623532, + 0.0005137080006534234, + 0.000552334007807076, + 0.0005664999916916713, + 0.0005735419981647283, + 0.0006305830029305071, + 0.0006260840018512681, + 0.0006338750099530444, + 0.0006193329900270328, + 0.000643542007310316, + 0.000648665998596698, + 0.0006434160022763535, + 0.000673082991852425, + 0.000748874997952953, + 0.0007590000022901222, + 0.0007632080087205395, + 0.0007493750017601997, + 0.0007733329985057935, + 0.0007755840051686391, + 0.0007925829995656386, + 0.0007922090007923543, + 0.0008100410050246865, + 0.000806582989753224, + 0.0009319590026279911, + 0.0009183329966617748, + 0.0009051660017576069, + 0.0009274999902117997, + 0.0009265830012736842, + 0.0009146660013357177, + 0.0009358329989481717, + 0.0009361249976791441, + 0.0009743340051500127, + 0.0010342499881517142, + 0.0009592080023139715, + 0.0009620000055292621, + 0.0009562500054016709, + 0.0009711249877000228, + 0.0010203750134678558, + 0.0010175409988733009, + 0.0010035830055130646, + 0.0010131659946637228, + 0.0011516249942360446, + 0.0010377500002505258, + 0.0009794579964363948, + 0.0009810409974306822, + 0.0010865410004043952, + 0.001105000003008172, + 0.0013139999937266111, + 0.0013103749952279031, + 0.0011396670015528798, + 0.0011200419976375997, + 0.0011458330118330196, + 0.0011552910000318661, + 0.0012862080038757995, + 0.0012483750033425167, + 0.0012525420024758205, + 0.0012044589966535568, + 0.0011737910099327564, + 0.001168292001239024, + 0.0011417909990996122 + ], + [ + 0.0005715000006603077, + 0.00057458299852442, + 0.0005617499991785735, + 0.0005674580024788156, + 0.0005654999986290932, + 0.0005866249994141981, + 0.0006001660076435655, + 0.0006071660027373582, + 0.0006087080109864473, + 0.0006241660012165084, + 0.0006290830060606822, + 0.0006302090041572228, + 0.0006306669965852052, + 0.0006329579919110984, + 0.0006378749967552722, + 0.0006442919984692708, + 0.0006457909912569448, + 0.0007287079934030771, + 0.0007561669917777181, + 0.0007639590039616451, + 0.0007668329926673323, + 0.0007690410129725933, + 0.0007850830006645992, + 0.0007970000006025657, + 0.0007804159977240488, + 0.0013365829945541918, + 0.0010604170092847198, + 0.0010878749890252948, + 0.001012292006635107, + 0.001018957991618663, + 0.0011033330083591864, + 0.001116583007387817, + 0.0009881250007310882, + 0.0009407499892404303, + 0.0009458330023335293, + 0.0009395420056534931, + 0.0009713339968584478, + 0.0009645419922890142, + 0.000942333004786633, + 0.0009739160013850778, + 0.0009697499917820096, + 0.0010013329883804545, + 0.0011464169947430491, + 0.0010611670004436746, + 0.0010160000092582777, + 0.0010146249987883493, + 0.0010077909973915666, + 0.0009835420059971511, + 0.0010888750111917034, + 0.0013765000039711595, + 0.0012161250051576644, + 0.0011788339907070622, + 0.0011891670001205057, + 0.001168583010439761, + 0.0011242089967709035, + 0.0011914170027012005, + 0.0011858329962706193, + 0.0011260419996688142, + 0.0011365420068614185, + 0.0011364579986548051, + 0.0011857909994432703, + 0.0012324160052230582, + 0.0011482089903438464, + 0.0011436250060796738 + ], + [ + 0.0005631249950965866, + 0.0005579589924309403, + 0.0005579169956035912, + 0.0005644579941872507, + 0.000567708004382439, + 0.0005751249991590157, + 0.0005764579982496798, + 0.0005795000033685938, + 0.0005821659869980067, + 0.0005747500108554959, + 0.0005935829976806417, + 0.0006026250048307702, + 0.0005996249965392053, + 0.0006034170073689893, + 0.00061495800036937, + 0.0006484170007752255, + 0.000638166005956009, + 0.0007539170037489384, + 0.0007562910031992942, + 0.0007546660053776577, + 0.0007611250039190054, + 0.0007721250003669411, + 0.000781790993642062, + 0.0007815000135451555, + 0.0007920000061858445, + 0.0008177919953595847, + 0.0009155409934464842, + 0.0009605410014046356, + 0.000929166009882465, + 0.0009123750060098246, + 0.0009385000012116507, + 0.0009444160095881671, + 0.0009710410085972399, + 0.001201000006403774, + 0.0010725839965743944, + 0.0009332499903393909, + 0.0009586249943822622, + 0.000984291997156106, + 0.0010504999954719096, + 0.0010065410024253651, + 0.0009818339894991368, + 0.0010084999958053231, + 0.0010174999915761873, + 0.0010087080008815974, + 0.0010357909923186526, + 0.001022750002448447, + 0.0010363750043325126, + 0.0010330410004826263, + 0.0010207500017713755, + 0.0011871669994434342, + 0.0011858329962706193, + 0.001223708997713402, + 0.0011923749989364296, + 0.0011915829963982105, + 0.0011844160035252571, + 0.0012456250115064904, + 0.0012338340020505711, + 0.0012226670078234747, + 0.0011340829951222986, + 0.0012059580039931461, + 0.0011438749934313819, + 0.0011459170054877177, + 0.0011436250060796738, + 0.0011425830016378313 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_mha": { + "first_token_ms": { + "count": 5, + "median": 0.6907499919179827, + "p95": 0.6951998017029837, + "min": 0.6737920048180968, + "max": 0.695333001203835, + "iqr": 0.007000999175943434, + "mean": 0.6884416012326255, + "cv": 0.012721022695223653 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.0727920016506687, + "p95": 1.4490707922959702, + "min": 0.6110000103944913, + "max": 3.2397920003859326, + "iqr": 0.5395840053097345, + "mean": 1.0713557526099657, + "cv": 0.33206220170320844 + }, + "total_ms": { + "count": 5, + "median": 65.03507899469696, + "p95": 77.70392165984958, + "min": 63.510580992442556, + "max": 80.09712205966935, + "iqr": 3.985752089647576, + "mean": 68.18385401566047, + "cv": 0.10107797664999932 + }, + "tokens_per_second": { + "count": 5, + "median": 984.0842971101587, + "p95": 1005.7116360846169, + "min": 799.0299570604099, + "max": 1007.7061018795542, + "iqr": 58.368620194884784, + "mean": 945.5838563329946, + "cv": 0.0909395513034802 + }, + "median_step_ms_by_index": [ + 0.6907499919179827, + 0.6602920038858429, + 0.6425000028684735, + 0.6732909969286993, + 0.6782919954275712, + 0.6572080019395798, + 0.6594169972231612, + 0.6759579991921782, + 0.6706669955747202, + 0.667583997710608, + 0.6734170019626617, + 0.6911250093253329, + 0.6862089940113947, + 0.6989999965298921, + 0.6992909911787137, + 0.7032079884083942, + 0.7203749992186204, + 0.8251249964814633, + 0.7868329994380474, + 0.8716660086065531, + 0.8030419994611293, + 0.8634999976493418, + 0.8839579968480393, + 0.887542002601549, + 0.8893329941201955, + 0.9010420035338029, + 0.9060419979505241, + 1.0287920013070107, + 1.0231659980490804, + 1.0190839966526255, + 1.0296669934177771, + 1.0425829968880862, + 1.0483330115675926, + 1.0590829915599898, + 1.0662499989848584, + 1.0844589996850118, + 1.0922910005319864, + 1.0918339976342395, + 1.1265000066487119, + 1.1321250058244914, + 1.1161659931531176, + 1.1521670094225556, + 1.1344999948050827, + 1.178125006845221, + 1.1742079950636253, + 1.174374992842786, + 1.186333000077866, + 1.1757920001400635, + 1.19029100460466, + 1.3201660040067509, + 1.330541999777779, + 1.3261249987408519, + 1.3393330009421334, + 1.3367079955060035, + 1.3405420031631365, + 1.3504160015145317, + 1.360958005534485, + 1.3538329949369654, + 1.3931660068919882, + 1.4084580034250394, + 1.404249996994622, + 1.4187080087140203, + 1.440624997485429, + 1.4580839924747124 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0006737920048180968, + 0.0006356250087264925, + 0.0006399999983841553, + 0.0006553749990416691, + 0.0006465000042226166, + 0.0006495830020867288, + 0.0006110000103944913, + 0.0006146669911686331, + 0.0006633750017499551, + 0.0006424169987440109, + 0.0006181659991852939, + 0.0006234160100575536, + 0.0006389579939423129, + 0.0006592919962713495, + 0.0006991659902269021, + 0.0007108749996405095, + 0.0007255410018842667, + 0.0008251249964814633, + 0.0007559160003438592, + 0.0007917079929029569, + 0.0007650420011486858, + 0.0007563749968539923, + 0.0007842079940019175, + 0.0008554999949410558, + 0.0008767499966779724, + 0.0009190830023726448, + 0.0009178329928545281, + 0.0010287920013070107, + 0.0010231659980490804, + 0.00102616599178873, + 0.0010309169883839786, + 0.0010929999989457428, + 0.0010483330115675926, + 0.0010590829915599898, + 0.0010662499989848584, + 0.0010979159997077659, + 0.0010922910005319864, + 0.0010972080053761601, + 0.0010847500088857487, + 0.0010949160059681162, + 0.0011165409960085526, + 0.0011521670094225556, + 0.0011348749976605177, + 0.0011322079953970388, + 0.001160291998530738, + 0.001174374992842786, + 0.0011701250041369349, + 0.0011737499880837277, + 0.0011692089901771396, + 0.0013211670011514798, + 0.001330541999777779, + 0.0013291249924805015, + 0.0013393330009421334, + 0.0013367079955060035, + 0.0013405420031631365, + 0.001378125001792796, + 0.0013462500064633787, + 0.0013529160059988499, + 0.0013931660068919882, + 0.0013823750050505623, + 0.0014087080053286627, + 0.0014258340088417754, + 0.0014208749926183373, + 0.0014580839924747124 + ], + [ + 0.0006876660045236349, + 0.0006602920038858429, + 0.0006630829884670675, + 0.0006732909969286993, + 0.0006782919954275712, + 0.0006849170022178441, + 0.0007711660000495613, + 0.0010226669983239844, + 0.0007597079966217279, + 0.0007036250026430935, + 0.0007123749965103343, + 0.0007472500001313165, + 0.0007197079976322129, + 0.0007242079882416874, + 0.0024662500072736293, + 0.002510750011424534, + 0.002675124997040257, + 0.0032397920003859326, + 0.0025918330065906048, + 0.0019383749895496294, + 0.0010449999972479418, + 0.0010560000082477927, + 0.0026263329928042367, + 0.0019522920047165826, + 0.000991541994153522, + 0.0009314579947385937, + 0.0009060419979505241, + 0.0010527080012252554, + 0.0010564580006757751, + 0.0010350420052418485, + 0.0010284159943694249, + 0.0010154170013265684, + 0.0010522090015001595, + 0.0010938330087810755, + 0.0010727920016506687, + 0.0010844589996850118, + 0.0010989169968524948, + 0.0010918339976342395, + 0.0011375419999239966, + 0.0011321250058244914, + 0.001102291003917344, + 0.0011664589983411133, + 0.0011287080124020576, + 0.001178125006845221, + 0.001177999991341494, + 0.0011891250032931566, + 0.00121254101395607, + 0.001172791002318263, + 0.00119029100460466, + 0.0013201660040067509, + 0.0012993750133318827, + 0.0013140000082785264, + 0.0013342919992282987, + 0.001334125001449138, + 0.0013472080026986077, + 0.0013342500024009496, + 0.001360958005534485, + 0.0013538329949369654, + 0.0013917499891249463, + 0.0014084580034250394, + 0.001404249996994622, + 0.0014187080087140203, + 0.0014249590021790937, + 0.0014416670019272715 + ], + [ + 0.0006946670036995783, + 0.000669040993670933, + 0.0006425000028684735, + 0.0007118330104276538, + 0.0006825839955126867, + 0.0006533330015372485, + 0.0006589999975403771, + 0.0006623340013902634, + 0.0006676250050077215, + 0.0006367080059135333, + 0.0006605830130865797, + 0.0006911250093253329, + 0.0006862089940113947, + 0.0006925829948158935, + 0.0006849170022178441, + 0.0006864999886602163, + 0.0007052080036373809, + 0.0007876249874243513, + 0.0007836250006221235, + 0.0007914580055512488, + 0.0007941249932628125, + 0.0008027089934330434, + 0.0008085410081548616, + 0.0008132910006679595, + 0.0008325409871758893, + 0.0008388749993173406, + 0.0008340409985976294, + 0.0009776660008355975, + 0.0009898749995045364, + 0.0009962089970940724, + 0.0010020000045187771, + 0.001014374996884726, + 0.0010104579996550456, + 0.0010130000009667128, + 0.0010224580037174746, + 0.0010245420126011595, + 0.0010331670055165887, + 0.0010394589917268604, + 0.0010915830062003806, + 0.0010950829891953617, + 0.0011155829997733235, + 0.0011377499904483557, + 0.0011289579997537658, + 0.0012029579957015812, + 0.0011589999921852723, + 0.0011551670031622052, + 0.001186333000077866, + 0.0011757920001400635, + 0.0011674169945763424, + 0.0013182919938117266, + 0.001315667002927512, + 0.0013215840008342639, + 0.0013208750024205074, + 0.0013533340097637847, + 0.001283041012356989, + 0.0012545829958980903, + 0.001251499998033978, + 0.001307458005612716, + 0.0013702910073334351, + 0.0014023330004420131, + 0.001400249995640479, + 0.001415457998518832, + 0.0014634169929195195, + 0.001420083994162269 + ], + [ + 0.000695333001203835, + 0.0006317499937722459, + 0.0006394579977495596, + 0.000639749996480532, + 0.0006371250055963174, + 0.0006572080019395798, + 0.0006594169972231612, + 0.0006759579991921782, + 0.0006706669955747202, + 0.000667583997710608, + 0.0006734170019626617, + 0.0006839160050731152, + 0.0006845410098321736, + 0.0006989999965298921, + 0.0007026250095805153, + 0.0007032079884083942, + 0.0007069590064929798, + 0.000779333000537008, + 0.0007868329994380474, + 0.0008842079987516627, + 0.0008030419994611293, + 0.0008704589999979362, + 0.0008839579968480393, + 0.0008907080045901239, + 0.0008929579926189035, + 0.0009010420035338029, + 0.0009081250027520582, + 0.0010842089977813885, + 0.0010347920033382252, + 0.0010129169968422502, + 0.0010515410103835166, + 0.0010425829968880862, + 0.001036667003063485, + 0.0010813330009113997, + 0.001066125012584962, + 0.0010741669975686818, + 0.001087083001038991, + 0.0010811669926624745, + 0.0011265000066487119, + 0.0011582909937715158, + 0.0011161659931531176, + 0.0011236249993089586, + 0.0011344999948050827, + 0.001141375003498979, + 0.0011742079950636253, + 0.001130959004512988, + 0.0011822499945992604, + 0.00118787499377504, + 0.001204250002047047, + 0.001318208989687264, + 0.0013744160096393898, + 0.0013261249987408519, + 0.0013414999993983656, + 0.0013184579875087366, + 0.001333584004896693, + 0.0013504160015145317, + 0.0013796660059597343, + 0.001382665999699384, + 0.0014014159969519824, + 0.0014219580043572932, + 0.0014011250023031607, + 0.0013972500019008294, + 0.0014452079922193661, + 0.0014819170028204098 + ], + [ + 0.0006907499919179827, + 0.0006857920088805258, + 0.0006825000018579885, + 0.000704916994436644, + 0.0006952919939067215, + 0.0006751250039087608, + 0.0006888750067446381, + 0.0006929160008439794, + 0.0006954580021556467, + 0.000698916002875194, + 0.0007091250008670613, + 0.0007117089990060776, + 0.0007272910006577149, + 0.0007207090093288571, + 0.0006992909911787137, + 0.0006994170107645914, + 0.0007203749992186204, + 0.0008549160120310262, + 0.0008597080013714731, + 0.0008716660086065531, + 0.000859291001688689, + 0.0008634999976493418, + 0.0009017499978654087, + 0.000887542002601549, + 0.0008893329941201955, + 0.0008940409898059443, + 0.0008991249924292788, + 0.0010180829995078966, + 0.0010208330058958381, + 0.0010190839966526255, + 0.0010296669934177771, + 0.0010897079919232056, + 0.0010866660013562068, + 0.0010579579975456, + 0.0010781250020954758, + 0.0011158330016769469, + 0.0012563329946715385, + 0.0012277500063646585, + 0.001194666008814238, + 0.0012843750009778887, + 0.0013224159920355305, + 0.0012982500047655776, + 0.0012990420073037967, + 0.0012999579921597615, + 0.001329000006080605, + 0.0012732500035781413, + 0.001283874997170642, + 0.0012550420069601387, + 0.0012482090096455067, + 0.0014274999994086102, + 0.0013521670043701306, + 0.0013702909927815199, + 0.001381957990815863, + 0.0013592500035883859, + 0.001364583004033193, + 0.001378875007503666, + 0.0014033750048838556, + 0.0014587500045308843, + 0.0015102920006029308, + 0.0014820000069448724, + 0.0015153749991441146, + 0.0014355840103235096, + 0.001440624997485429, + 0.0014830419968347996 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "speedup_cached_vs_uncached_sdpa": 2.622469396661692, + "speedup_cached_vs_uncached_mha": 2.955949960558279 + }, + { + "batch": 1, + "prompt_len": 128, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_mha": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 1.4904159907018766, + "p95": 1.5691752079874277, + "min": 1.3509159907698631, + "max": 1.5801670087967068, + "iqr": 0.05179100844543427, + "mean": 1.484024798264727, + "cv": 0.057188119498531365 + }, + "decode_ms_per_step": { + "count": 315, + "median": 0.3730830067070201, + "p95": 0.46875810221536085, + "min": 0.31341699650511146, + "max": 0.6747080042259768, + "iqr": 0.08420898666372523, + "mean": 0.39443684107113985, + "cv": 0.13283893030801142 + }, + "total_ms": { + "count": 5, + "median": 26.655787965864874, + "p95": 27.377156211878173, + "min": 23.877576924860477, + "max": 27.408957030274905, + "iqr": 0.7744988688500598, + "mean": 26.333545785746537, + "cv": 0.054211906317252305 + }, + "tokens_per_second": { + "count": 5, + "median": 2400.9794826533634, + "p95": 2627.7378624965786, + "min": 2335.003113373048, + "max": 2680.338972476118, + "iqr": 68.70551320437698, + "mean": 2436.456580090998, + "cv": 0.05771995493887325 + }, + "median_step_ms_by_index": [ + 1.4904159907018766, + 0.4784170014318079, + 0.3985410003224388, + 0.3759579994948581, + 0.3684999974211678, + 0.36520800495054573, + 0.36995799746364355, + 0.3701250097947195, + 0.36412499321158975, + 0.36012499185744673, + 0.3730830067070201, + 0.36779200308956206, + 0.35616599780041724, + 0.36008299503009766, + 0.3566250088624656, + 0.3631250001490116, + 0.3599159972509369, + 0.3641660005087033, + 0.3701660025399178, + 0.35983300767838955, + 0.37345899909269065, + 0.36520800495054573, + 0.36520899448078126, + 0.36750000435858965, + 0.370459005353041, + 0.36633299896493554, + 0.36658400495070964, + 0.3611249994719401, + 0.37312498898245394, + 0.36662499769590795, + 0.3664159885374829, + 0.37250001332722604, + 0.36820900277234614, + 0.37079099274706095, + 0.3724999987753108, + 0.3670839942060411, + 0.36329199792817235, + 0.3647909907158464, + 0.3675420011859387, + 0.36962500598747283, + 0.3651250008260831, + 0.36450001061894, + 0.3703340044012293, + 0.3812500071944669, + 0.46387499605771154, + 0.4546669952105731, + 0.4572080069920048, + 0.47304100007750094, + 0.4596249927999452, + 0.4482080112211406, + 0.45379099901765585, + 0.45287499960977584, + 0.45274999865796417, + 0.45554198732133955, + 0.450624997029081, + 0.4553339967969805, + 0.45408299774862826, + 0.4517920024227351, + 0.4472919972613454, + 0.44800000614486635, + 0.4541249945759773, + 0.45741599751636386, + 0.45516699901781976, + 0.45812499593012035 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.001473416996304877, + 0.0004620830004569143, + 0.0003974999999627471, + 0.00036966599873267114, + 0.0003917499998351559, + 0.0003835829993477091, + 0.0003737499937415123, + 0.00037104199873283505, + 0.00036412499321158975, + 0.00036012499185744673, + 0.00039154200931079686, + 0.00038387501263059676, + 0.0003559159958967939, + 0.0003615840105339885, + 0.0003566250088624656, + 0.0003560409968486056, + 0.00035812500573229045, + 0.0003506250068312511, + 0.00036258299951441586, + 0.00034704200515989214, + 0.00037558400072157383, + 0.00036520800495054573, + 0.00036520899448078126, + 0.00036750000435858965, + 0.00037729200266767293, + 0.00036633299896493554, + 0.000371750007616356, + 0.00035345900687389076, + 0.00037312498898245394, + 0.00036604100023396313, + 0.0003652909945230931, + 0.00037250001332722604, + 0.00038283399771898985, + 0.0003664579999167472, + 0.0003678330103866756, + 0.00036304201057646424, + 0.000356167001882568, + 0.0003647909907158464, + 0.00036874999932479113, + 0.0003633330052252859, + 0.00036195800930727273, + 0.0003605419915402308, + 0.0003703340044012293, + 0.0003677079948829487, + 0.0004594160127453506, + 0.0004523329989751801, + 0.0004572080069920048, + 0.0004826250078622252, + 0.0004552080063149333, + 0.0004453750007087365, + 0.0004457080067368224, + 0.0004550829908112064, + 0.000464749988168478, + 0.0004678330005845055, + 0.00044729100773110986, + 0.00045537500409409404, + 0.000443959012045525, + 0.0004456249880604446, + 0.000444290999439545, + 0.00044787499064113945, + 0.0004512919986154884, + 0.00045741599751636386, + 0.00044737498683389276, + 0.0004653750074794516 + ], + [ + 0.0015801670087967068, + 0.00048137499834410846, + 0.0004004160000476986, + 0.0003759579994948581, + 0.0003684999974211678, + 0.00036520800495054573, + 0.0003624579985626042, + 0.00036941700091119856, + 0.00036066600296180695, + 0.0003493330004857853, + 0.0003581670025596395, + 0.0003655829932540655, + 0.00035616599780041724, + 0.0003559580072760582, + 0.0003536669973982498, + 0.00039975000254344195, + 0.00040595800965093076, + 0.00039808299334254116, + 0.0003723330009961501, + 0.0003729169984580949, + 0.0003765410074265674, + 0.00039608299266546965, + 0.0003698329965118319, + 0.00036441699194256216, + 0.000370459005353041, + 0.0003651250008260831, + 0.00036658400495070964, + 0.0003815829986706376, + 0.00038329200469888747, + 0.00040962500497698784, + 0.00036879201070405543, + 0.0003915840061381459, + 0.00043166700925212353, + 0.00038012499862816185, + 0.00037516700103878975, + 0.0003670839942060411, + 0.00036587500653695315, + 0.0003763330023502931, + 0.0003675420011859387, + 0.0003698329965118319, + 0.00037850000080652535, + 0.00037841699668206275, + 0.0003942500043194741, + 0.0003812500071944669, + 0.0005100419948576018, + 0.0006747080042259768, + 0.0005403749964898452, + 0.00047654200170654804, + 0.0004596249927999452, + 0.0004617079976014793, + 0.0004598749947035685, + 0.0004489999992074445, + 0.00045649999810848385, + 0.0004471669963095337, + 0.0004558750079013407, + 0.0004633329954231158, + 0.000457709000329487, + 0.0004591250035446137, + 0.00046204098907765, + 0.00046366600145120174, + 0.0004697499971371144, + 0.000466583005618304, + 0.00045516699901781976, + 0.00045812499593012035 + ], + [ + 0.0013509159907698631, + 0.00039799998921807855, + 0.0003358749963808805, + 0.00032300000020768493, + 0.000320708000799641, + 0.00031687501177657396, + 0.0003200409992132336, + 0.0003142919886158779, + 0.00031370899523608387, + 0.00031341699650511146, + 0.00032379099866375327, + 0.0003141659981338307, + 0.00031450000824406743, + 0.00032612499489914626, + 0.00033045800228137523, + 0.0003208330017514527, + 0.0003385829913895577, + 0.00031550000130664557, + 0.0003321659896755591, + 0.0003181249921908602, + 0.0003227499983040616, + 0.00032383301004301757, + 0.000324332999298349, + 0.00032087499857880175, + 0.0003249580040574074, + 0.0003169169940520078, + 0.0003181249921908602, + 0.00031620899972040206, + 0.0003234579926356673, + 0.00032045799889601767, + 0.00032212499354500324, + 0.0003251250018365681, + 0.0003232910094084218, + 0.00032204200397245586, + 0.00032058299984782934, + 0.00032591600029263645, + 0.0003190830029780045, + 0.0003190830029780045, + 0.00032029100111685693, + 0.00031850000959821045, + 0.00035283301258459687, + 0.00033329099824186414, + 0.000343833991792053, + 0.00034341700666118413, + 0.00041412501013837755, + 0.0004141670069657266, + 0.00041870899440255016, + 0.00040087499655783176, + 0.0004006249946542084, + 0.0003974169958382845, + 0.00040370899660047144, + 0.00042262500210199505, + 0.0004369999951450154, + 0.0004327079950599, + 0.00043237498903181404, + 0.00042958300036843866, + 0.0004385000065667555, + 0.00043083299533464015, + 0.00043149999692104757, + 0.0004355409910203889, + 0.0004516670014709234, + 0.00044770899694412947, + 0.00045520799176301807, + 0.00044029099808540195 + ], + [ + 0.0015252080047503114, + 0.0004784170014318079, + 0.0003985410003224388, + 0.00037687500298488885, + 0.0003752080083359033, + 0.00037237499782349914, + 0.00037954199069645256, + 0.0003747910086531192, + 0.00038304099871311337, + 0.0003643340023700148, + 0.00037924999196548015, + 0.00036779200308956206, + 0.0003666660049930215, + 0.0003663749957922846, + 0.0003647919947979972, + 0.0003631250001490116, + 0.0003633330052252859, + 0.0003641660005087033, + 0.00037045800127089024, + 0.00035983300767838955, + 0.00037345899909269065, + 0.00037291599437594414, + 0.00036904199805576354, + 0.0005217500001890585, + 0.0004952500021317974, + 0.0004020409978693351, + 0.0003759579994948581, + 0.00036858400562778115, + 0.00037079099274706095, + 0.0003704169939737767, + 0.0003664159885374829, + 0.0003713329933816567, + 0.00036820900277234614, + 0.00037079099274706095, + 0.0003724999987753108, + 0.0003713749902090058, + 0.00036329199792817235, + 0.00036258299951441586, + 0.00036204200296197087, + 0.00037116599560249597, + 0.0003651250008260831, + 0.00036462499701883644, + 0.0003761669941013679, + 0.0003973339917138219, + 0.00046645800466649234, + 0.00048787498963065445, + 0.0004821250040549785, + 0.00047304100007750094, + 0.0004633750068023801, + 0.0004630409966921434, + 0.00045379099901765585, + 0.0004568330041365698, + 0.00045274999865796417, + 0.00045554198732133955, + 0.000450624997029081, + 0.0004553339967969805, + 0.00045408299774862826, + 0.0004517920024227351, + 0.0004472919972613454, + 0.00044983399857301265, + 0.0004650409973692149, + 0.00045870800386182964, + 0.0004983329999959096, + 0.000566792004974559 + ], + [ + 0.0014904159907018766, + 0.0004860420012846589, + 0.0004121669917367399, + 0.0003808750043390319, + 0.0003658749919850379, + 0.0003635419998317957, + 0.00036995799746364355, + 0.0003701250097947195, + 0.00037108299147803336, + 0.00036499999987427145, + 0.0003730830067070201, + 0.0003727080038515851, + 0.0003690839948831126, + 0.00036008299503009766, + 0.00035729099181480706, + 0.0003674580075312406, + 0.0003599159972509369, + 0.0003663749957922846, + 0.0003701660025399178, + 0.0003668330027721822, + 0.0003715839993674308, + 0.0003604579978855327, + 0.0003628330014180392, + 0.0003690000012284145, + 0.00036187500518281013, + 0.00036720800562761724, + 0.0003588750114431605, + 0.0003611249994719401, + 0.0003799159894697368, + 0.00036662499769590795, + 0.0003667499986477196, + 0.0003758750099223107, + 0.00036220799665898085, + 0.0003716249921126291, + 0.0003947079967474565, + 0.00037145899841561913, + 0.0004228329926263541, + 0.0003872499946737662, + 0.0003787500027101487, + 0.00036962500598747283, + 0.00036541699955705553, + 0.00036450001061894, + 0.0003622499934863299, + 0.0003812909999396652, + 0.00046387499605771154, + 0.0004546669952105731, + 0.0004505419929046184, + 0.00045216600119601935, + 0.0004683330043917522, + 0.0004482080112211406, + 0.0004570419987430796, + 0.00045287499960977584, + 0.0004500420036492869, + 0.000457124988315627, + 0.00045316699834074825, + 0.00045404200500342995, + 0.000457874994026497, + 0.00045670899271499366, + 0.0004504999960772693, + 0.00044800000614486635, + 0.0004541249945759773, + 0.00045074999798089266, + 0.0004457080067368224, + 0.0004579170054057613 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + ] + } + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 1.4724579959874973, + "p95": 1.4910417929058895, + "min": 1.390125005855225, + "max": 1.495124990469776, + "iqr": 0.024791996111162007, + "mean": 1.4564668003004044, + "cv": 0.027735057063984676 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.698833002592437, + "p95": 2.322675102914218, + "min": 1.3948749983683228, + "max": 5.1005419954890385, + "iqr": 0.6143125065136701, + "mean": 1.8524369840706063, + "cv": 0.22546150707865836 + }, + "total_ms": { + "count": 5, + "median": 117.54262403701432, + "p95": 123.19274034816772, + "min": 113.07316098827869, + "max": 124.06183294660877, + "iqr": 3.310373896965757, + "mean": 118.1599967967486, + "cv": 0.03452995388081232 + }, + "tokens_per_second": { + "count": 5, + "median": 544.4833355076905, + "p95": 562.7641519425849, + "min": 515.8717913473278, + "max": 566.0052256488551, + "iqr": 15.202959263219213, + "mean": 542.1514214951324, + "cv": 0.03426794650959053 + }, + "median_step_ms_by_index": [ + 1.4724579959874973, + 1.52395800978411, + 1.5282500098692253, + 1.5500419976888224, + 1.5483750030398369, + 1.5319590020226315, + 1.520541001809761, + 1.5241249930113554, + 1.5262910019373521, + 1.5372920024674386, + 1.5439580020029098, + 1.5445000026375055, + 1.543791004223749, + 1.638083005673252, + 1.553957990836352, + 1.5597919991705567, + 1.564541002153419, + 1.5838750114198774, + 1.593708002474159, + 1.5802919951966032, + 1.602124990313314, + 1.640333008253947, + 1.599457988049835, + 1.6082920046756044, + 1.6068330005509779, + 1.602665986865759, + 1.618166992557235, + 1.6143749962793663, + 1.6520839999429882, + 1.6274579975288361, + 1.6490830021211877, + 1.6350000078091398, + 1.6414160054409876, + 1.7703340126900002, + 1.7627920024096966, + 1.7764170042937621, + 1.7088749882532284, + 1.7844170070020482, + 1.732333010295406, + 1.7325420049019158, + 1.7182080046040937, + 1.7554999940330163, + 1.8108750082319602, + 2.1887909970246255, + 2.1577919978881255, + 2.151791995856911, + 2.173291999497451, + 2.1935830009169877, + 2.202625008067116, + 2.161083000828512, + 2.1466660109581426, + 2.18191699241288, + 2.216332999523729, + 2.2279169934336096, + 2.2122920054243878, + 2.2562500089406967, + 2.1649999980581924, + 2.2318340052152053, + 2.2820830054115504, + 2.3181249998742715, + 2.2628750011790544, + 2.2369579965015873, + 2.265415998408571, + 2.280875007272698 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0014724579959874973, + 0.0014919160021236166, + 0.0015218749904306605, + 0.0016206669970415533, + 0.0015778749948367476, + 0.0015319590020226315, + 0.0015057090058689937, + 0.0015241249930113554, + 0.001606874997378327, + 0.002204374992288649, + 0.0016312499938067049, + 0.0015518750005867332, + 0.0015560840110993013, + 0.001841916993726045, + 0.0018496670090826228, + 0.0015834999940125272, + 0.0016245000006165355, + 0.0015838750114198774, + 0.0017059999954653904, + 0.0016010410035960376, + 0.001602124990313314, + 0.0017714999994495884, + 0.0016837080038385466, + 0.001694124992354773, + 0.0023726669896859676, + 0.001672792001045309, + 0.0016344999894499779, + 0.0017013330070767552, + 0.0017463750118622556, + 0.001644750009290874, + 0.0016490830021211877, + 0.0016909160040086135, + 0.0016414160054409876, + 0.002231375008705072, + 0.0018284590041730553, + 0.0017951669869944453, + 0.0017517090018372983, + 0.0018252499867230654, + 0.0017161249998025596, + 0.0016760409926064312, + 0.0017182080046040937, + 0.0017621669976506382, + 0.0021235829917714, + 0.004850665995036252, + 0.0022368329955497757, + 0.002151791995856911, + 0.0020705000060843304, + 0.002219624991994351, + 0.0022419160086428747, + 0.002658792000147514, + 0.00215445899812039, + 0.00218191699241288, + 0.0023811249993741512, + 0.0022279169934336096, + 0.002215582993812859, + 0.002148708008462563, + 0.002610208000987768, + 0.00234620799892582, + 0.00227537500904873, + 0.0023794579901732504, + 0.0023819589987397194, + 0.0022129580029286444, + 0.0022616249916609377, + 0.002333292010007426 + ], + [ + 0.0014747090026503429, + 0.001570916996570304, + 0.0015304170083254576, + 0.0015673340094508603, + 0.001581749995239079, + 0.0015479169960599393, + 0.001520541001809761, + 0.0015170830010902137, + 0.0015694999892730266, + 0.001600207993760705, + 0.0015439580020029098, + 0.0015445000026375055, + 0.001543791004223749, + 0.0015894579992163926, + 0.0015723329997854307, + 0.0016624590061837807, + 0.0015945840132189915, + 0.001586166996276006, + 0.001664000010350719, + 0.0015956670104060322, + 0.0016168749862117693, + 0.0015872499934630468, + 0.001599457988049835, + 0.001713167002890259, + 0.0016068330005509779, + 0.0016686669987393543, + 0.001618166992557235, + 0.0016311250074068084, + 0.0016520839999429882, + 0.0016553330060560256, + 0.001724916000966914, + 0.0016350000078091398, + 0.0016202919941861182, + 0.001887417005491443, + 0.0017777920002117753, + 0.002025290988967754, + 0.0019406249921303242, + 0.0017844170070020482, + 0.001732333010295406, + 0.0017325420049019158, + 0.0016587080026511103, + 0.0017206670017912984, + 0.0018108750082319602, + 0.0021887909970246255, + 0.0022477079910458997, + 0.002222666997113265, + 0.002173291999497451, + 0.0022082500072428957, + 0.002202625008067116, + 0.002161083000828512, + 0.0020615419925889, + 0.0022081659990362823, + 0.0024087909987429157, + 0.0022499170008813962, + 0.0022440000029746443, + 0.0022613750043092296, + 0.0021649999980581924, + 0.00214320799568668, + 0.0022820830054115504, + 0.0023181249998742715, + 0.0022628750011790544, + 0.0022369579965015873, + 0.002304124995134771, + 0.002214916006778367 + ], + [ + 0.001390125005855225, + 0.00147125001240056, + 0.0014764999941689894, + 0.00153087500075344, + 0.0015483750030398369, + 0.0015250420110533014, + 0.0014935829967726022, + 0.0014122919965302572, + 0.0013948749983683228, + 0.0014169579953886569, + 0.0014935420040274039, + 0.0014349580014823005, + 0.0014200000005075708, + 0.0014342920039780438, + 0.0014515829971060157, + 0.0014492499904008582, + 0.0014950000040698797, + 0.0016511669964529574, + 0.0014730830007465556, + 0.0014602500014007092, + 0.0014630000077886507, + 0.0018620000046212226, + 0.0016170000017154962, + 0.0014866249985061586, + 0.0015174159925663844, + 0.001517374999821186, + 0.0015873329975875095, + 0.001532415990368463, + 0.0015219999913824722, + 0.0015327919973060489, + 0.0015603329957230017, + 0.0016219160024775192, + 0.001698833002592437, + 0.0017703340126900002, + 0.0017627920024096966, + 0.0016480829945066944, + 0.001661917005549185, + 0.0017248749936698005, + 0.0017364579980494455, + 0.0017501659895060584, + 0.0017258750012842938, + 0.0017554999940330163, + 0.001697707994026132, + 0.0020444999972824007, + 0.00213612501102034, + 0.0021996659925207496, + 0.0022016660077497363, + 0.0021621250052703544, + 0.0021625420049531385, + 0.002138333991752006, + 0.0021466660109581426, + 0.0021409579931059852, + 0.00219237498822622, + 0.0022354579996317625, + 0.002205875003710389, + 0.002092750000883825, + 0.0021217500034254044, + 0.0022318340052152053, + 0.0022192909964360297, + 0.0022518329933518544, + 0.002225292002549395, + 0.0022730000055162236, + 0.002265415998408571, + 0.0022499579936265945 + ], + [ + 0.001495124990469776, + 0.0015315829950850457, + 0.0016203340055653825, + 0.001548792002722621, + 0.0015106249920791015, + 0.0015228340052999556, + 0.0015405840094899759, + 0.0015369589964393526, + 0.001512957998784259, + 0.0015195410087471828, + 0.001516582997282967, + 0.0015370409964816645, + 0.0015433330117957667, + 0.001638083005673252, + 0.0015221250068861991, + 0.0015180420014075935, + 0.0014502499980153516, + 0.0014665409980807453, + 0.0014890409947838634, + 0.0014679159939987585, + 0.001456999991205521, + 0.0014635000115958974, + 0.0014705830108141527, + 0.0014817909977864474, + 0.001507832988863811, + 0.001489208996645175, + 0.0014834589965175837, + 0.00158920900139492, + 0.0017006249981932342, + 0.0015545829955954105, + 0.001502707993495278, + 0.0014979160041548312, + 0.0015937909920467064, + 0.0017147500038845465, + 0.0017180419963551685, + 0.0017764170042937621, + 0.0017088749882532284, + 0.0017430830048397183, + 0.001718499988783151, + 0.0016350829973816872, + 0.0016457500023534521, + 0.0016925000090850517, + 0.0017670410015853122, + 0.0021594999998342246, + 0.0021577919978881255, + 0.002151000007870607, + 0.002168708000681363, + 0.002176915993914008, + 0.0022516659955726936, + 0.002191165986005217, + 0.0022263749997364357, + 0.0020660419977502897, + 0.0021367909939726815, + 0.0022214170021470636, + 0.002197499998146668, + 0.0022800000006100163, + 0.0022144579997984692, + 0.002317666992894374, + 0.0032955000060610473, + 0.0034474580024834722, + 0.002271750010550022, + 0.0024749589938437566, + 0.0051005419954890385, + 0.0028086249949410558 + ], + [ + 0.0014499170065391809, + 0.0015239580097841099, + 0.0015282500098692253, + 0.0015500419976888224, + 0.0015037080011097714, + 0.001541707999422215, + 0.0015286249981727451, + 0.0015333750052377582, + 0.0015262910019373521, + 0.0015372920024674386, + 0.001580208001541905, + 0.0015494589897571132, + 0.001546458006487228, + 0.0016655409999657422, + 0.0015539579908363521, + 0.0015597919991705567, + 0.001564541002153419, + 0.0015707499987911433, + 0.001593708002474159, + 0.0015802919951966032, + 0.0016067499964265153, + 0.001640333008253947, + 0.0015957910072756931, + 0.0016082920046756044, + 0.0016204580024350435, + 0.001602665986865759, + 0.001629374994081445, + 0.0016143749962793663, + 0.0016068750119302422, + 0.0016274579975288361, + 0.0016753340023569763, + 0.001692207995802164, + 0.0016698749968782067, + 0.0017190420039696619, + 0.0017354169976897538, + 0.0016709999908925965, + 0.0016406670038122684, + 0.0018274580070283264, + 0.0017911659961100668, + 0.0017674580012680963, + 0.0017836670012911782, + 0.0021754580084234476, + 0.0018171249976148829, + 0.002258000007714145, + 0.002154583009541966, + 0.0021112919930601493, + 0.0021903750021010637, + 0.0021935830009169877, + 0.0021968329965602607, + 0.002077833007206209, + 0.0020741250045830384, + 0.00236154199228622, + 0.002216332999523729, + 0.0022120829962659627, + 0.0022122920054243878, + 0.0022562500089406967, + 0.0021392920025391504, + 0.0021048329945188016, + 0.0022907499951543286, + 0.002256209001643583, + 0.0022308750048978254, + 0.0022343339951476082, + 0.0022475830046460032, + 0.002280875007272698 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_mha": { + "first_token_ms": { + "count": 5, + "median": 1.88079199870117, + "p95": 2.160407992778346, + "min": 1.8201670027337968, + "max": 2.2190829913597554, + "iqr": 0.06941599713172764, + "mean": 1.9404083985136822, + "cv": 0.08268335688558577 + }, + "decode_ms_per_step": { + "count": 315, + "median": 2.219000001787208, + "p95": 3.1410661948029883, + "min": 1.731958007439971, + "max": 3.487082998617552, + "iqr": 0.8791874934104271, + "mean": 2.3856523965988012, + "cv": 0.19626987640841884 + }, + "total_ms": { + "count": 5, + "median": 151.26867101935204, + "p95": 156.82207400386687, + "min": 148.31900197896175, + "max": 157.90108201326802, + "iqr": 1.318292022915557, + "mean": 152.23650938423816, + "cv": 0.023117695570193775 + }, + "tokens_per_second": { + "count": 5, + "median": 423.08826783975894, + "p95": 429.8648352617451, + "min": 405.31704522849464, + "max": 431.5023641345568, + "iqr": 3.6592151436179847, + "mean": 420.57558032003783, + "cv": 0.022771323767066795 + }, + "median_step_ms_by_index": [ + 1.88079199870117, + 1.8867920007323846, + 1.9090839923592284, + 1.9218329980503768, + 1.9279170082882047, + 1.9168340077158064, + 1.8778329977067187, + 1.95479100511875, + 1.9845829956466332, + 1.9155000045429915, + 1.9294590019853786, + 1.9393749971641228, + 1.9448340026428923, + 1.9312910008011386, + 1.9364580075489357, + 1.9619590020738542, + 1.9879169994965196, + 1.9917080062441528, + 1.9887080125045031, + 2.0052079926244915, + 2.0097500091651455, + 2.0164160087006167, + 2.0086669974261895, + 2.0168339979136363, + 2.0290409884182736, + 2.068917005090043, + 2.052374999038875, + 2.0776249875780195, + 2.093707997119054, + 2.0731659897137433, + 2.0762499916600063, + 2.09087498660665, + 2.0976249943487346, + 2.21574999159202, + 2.2232080082176253, + 2.247000011266209, + 2.2555420000571758, + 2.2509590053232387, + 2.290417003678158, + 2.31516599887982, + 2.3409999994328246, + 2.297125000040978, + 2.3125830048229545, + 2.843916998244822, + 2.8587090055225417, + 2.884041008655913, + 2.8464579954743385, + 2.8467920055845752, + 2.834917002473958, + 2.876415994251147, + 2.8864589985460043, + 2.92250000347849, + 2.9165000014472753, + 2.911542003857903, + 2.9958340019220486, + 3.072457999223843, + 3.023083001608029, + 3.1033330014906824, + 3.0538749997504056, + 3.0865830049151555, + 3.0812920012976974, + 3.1240419921232387, + 3.149750002194196, + 3.1052909907884896 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0019257079984527081, + 0.0020932499901391566, + 0.001938333996804431, + 0.0019218329980503768, + 0.0019159590010531247, + 0.0019168340077158064, + 0.0018498330027796328, + 0.0019294590019853786, + 0.002202084011514671, + 0.0018457499973010272, + 0.001795459000277333, + 0.0018632919964147732, + 0.0018400830012978986, + 0.0019067499961238354, + 0.00191183399874717, + 0.0019178749935235828, + 0.0019518750050337985, + 0.001991708006244153, + 0.0019324159948155284, + 0.0019518329936545342, + 0.002003166009671986, + 0.0020164160087006167, + 0.0019657920056488365, + 0.0019718330004252493, + 0.002024125002208166, + 0.0021080839942442253, + 0.001987208001082763, + 0.002003999994485639, + 0.002093707997119054, + 0.002068208996206522, + 0.002074208008707501, + 0.002089457993861288, + 0.002087458997266367, + 0.002409084001556039, + 0.0022465000074589625, + 0.0022234579955693334, + 0.002371791997575201, + 0.0022730420023435727, + 0.0022964590025367215, + 0.0023210419894894585, + 0.0025901670014718547, + 0.002348875001189299, + 0.0023116249940358102, + 0.0029150420014047995, + 0.0029055420018266886, + 0.002910750001319684, + 0.002927458001067862, + 0.0028331250068731606, + 0.002834917002473958, + 0.0028843750042142347, + 0.0028725000011036173, + 0.00292250000347849, + 0.0030022080027265474, + 0.002911542003857903, + 0.0031460419995710254, + 0.0030988329963292927, + 0.003013249996001832, + 0.0031258329981938004, + 0.0030538749997504056, + 0.0030652499990537763, + 0.0030812920012976974, + 0.003057207999518141, + 0.0031437500001629815, + 0.003075500004342757 + ], + [ + 0.0018201670027337968, + 0.0018752920004772022, + 0.0019090839923592284, + 0.002005166999879293, + 0.0019279170082882047, + 0.001877542003057897, + 0.002022291999310255, + 0.0022148330026539043, + 0.0019819579902105033, + 0.0019196670036762953, + 0.002031583004281856, + 0.0019393749971641228, + 0.001965916992048733, + 0.0019312910008011386, + 0.0019566659902920946, + 0.001987415991607122, + 0.0020061669929418713, + 0.001979541004402563, + 0.0020477079960983247, + 0.002061124992906116, + 0.0020186659967293963, + 0.0020265829953132197, + 0.0020663750037783757, + 0.002076500008115545, + 0.00203408399829641, + 0.002049707996775396, + 0.0020671249949373305, + 0.002084083011141047, + 0.002119209006195888, + 0.0020731659897137433, + 0.00216416698822286, + 0.002154625006369315, + 0.002140124997822568, + 0.0022040420008124784, + 0.002300499996636063, + 0.002275582999573089, + 0.0022342089941957965, + 0.002236749991425313, + 0.002290417003678158, + 0.00231516599887982, + 0.0023663330066483468, + 0.0023428749991580844, + 0.002335540993954055, + 0.002843916998244822, + 0.0028190839948365465, + 0.002884041008655913, + 0.0028464579954743385, + 0.0028489170072134584, + 0.002864459005650133, + 0.002852249992429279, + 0.0029346670053200796, + 0.0029647920018760487, + 0.0029165000014472753, + 0.0029607500036945567, + 0.0029855000029783696, + 0.0031327499891631305, + 0.003102083006524481, + 0.0034032090043183416, + 0.0031045000068843365, + 0.003203667001798749, + 0.003035625006305054, + 0.0031240419921232387, + 0.003137000006972812, + 0.0031052909907884896 + ], + [ + 0.0018562920013209805, + 0.0018867920007323846, + 0.0017400420038029552, + 0.0017481249960837886, + 0.001731958007439971, + 0.0017372919974150136, + 0.0017412919987691566, + 0.0018001669959630817, + 0.0018353329942328855, + 0.0017780000052880496, + 0.0017938339879037812, + 0.0022164580004755408, + 0.0019311250071041286, + 0.0018194579897681251, + 0.0019240000110585243, + 0.0019103750091744587, + 0.001928666009916924, + 0.001964750001206994, + 0.001935207998030819, + 0.0020052079926244915, + 0.001956542007974349, + 0.0019902079948224127, + 0.001961709000170231, + 0.001983457987080328, + 0.0019579579966375604, + 0.0019720830023288727, + 0.0019824169867206365, + 0.0020903330005239695, + 0.0020244999905116856, + 0.002009542004088871, + 0.0020282079931348562, + 0.00209087498660665, + 0.0020737919985549524, + 0.0022094169980846345, + 0.0021963750041322783, + 0.002247000011266209, + 0.0022269170003710315, + 0.0022695420047966763, + 0.002248959004646167, + 0.0022731250064680353, + 0.002246625008410774, + 0.002287167008034885, + 0.002321124993613921, + 0.0028374579997034743, + 0.002801833994453773, + 0.0028504579968284816, + 0.002845417009666562, + 0.0028467920055845752, + 0.0028169580036774278, + 0.0028440419991966337, + 0.0028584159881575033, + 0.0028589580033440143, + 0.0028639999945880845, + 0.002885624999180436, + 0.0029958340019220486, + 0.0029782920028083026, + 0.002973915994516574, + 0.003046332989470102, + 0.0030107089987723157, + 0.00321033300133422, + 0.003062791991396807, + 0.0034870829986175522, + 0.003149750002194196, + 0.003161750006256625 + ], + [ + 0.0022190829913597554, + 0.002278458996443078, + 0.0022490000119432807, + 0.0030677909962832928, + 0.0022035000001778826, + 0.002463333003106527, + 0.0021902090084040537, + 0.0025615839986130595, + 0.001984582995646633, + 0.0019155000045429915, + 0.0019294590019853786, + 0.0019085829990217462, + 0.0021274159953463823, + 0.0020326250087236986, + 0.001996333012357354, + 0.0020080829999642447, + 0.0020145419985055923, + 0.0019981670047855005, + 0.0019932499999413267, + 0.002065499997115694, + 0.0020462079992285, + 0.0020248749933671206, + 0.0020339580078143626, + 0.0020821669895667583, + 0.002318832994205877, + 0.002068917005090043, + 0.002056583994999528, + 0.0020776249875780195, + 0.00211312499595806, + 0.0021098339930176735, + 0.0020867080020252615, + 0.0023055839992593974, + 0.002112500005750917, + 0.00221574999159202, + 0.0022232080082176253, + 0.002310208001290448, + 0.0022555420000571758, + 0.0022509590053232387, + 0.0024055420071817935, + 0.0023279580054804683, + 0.0023409999994328246, + 0.002297125000040978, + 0.0023125830048229545, + 0.0031399159925058484, + 0.0028818749997299165, + 0.003147124996758066, + 0.00290024999412708, + 0.0029310420068213716, + 0.0029025830008322373, + 0.0029260419978527352, + 0.0029756250005448237, + 0.00332416599849239, + 0.0029114159988239408, + 0.002934333009761758, + 0.003024999998160638, + 0.003025375001016073, + 0.0033695830061333254, + 0.0031033330014906824, + 0.003086500000790693, + 0.0030865830049151555, + 0.0030843330023344606, + 0.003322124990518205, + 0.0031667919975006953, + 0.0030732919985894114 + ], + [ + 0.00188079199870117, + 0.001866499995230697, + 0.0018620420014485717, + 0.001890124985948205, + 0.0020503330015344545, + 0.0019301659922348335, + 0.0018778329977067187, + 0.00195479100511875, + 0.00222154198854696, + 0.00197350000962615, + 0.001990834003663622, + 0.0019534579914761707, + 0.0019448340026428923, + 0.0019344169995747507, + 0.0019364580075489357, + 0.001961959002073854, + 0.0019879169994965196, + 0.0020037089998368174, + 0.001988708012504503, + 0.0019995000038761646, + 0.0020097500091651455, + 0.0020004160032840446, + 0.0020086669974261895, + 0.0020168339979136363, + 0.0020290409884182736, + 0.0021701249934267253, + 0.002052374999038875, + 0.0020529580069705844, + 0.002057833000435494, + 0.00208116699650418, + 0.0020762499916600063, + 0.002090042005875148, + 0.0020976249943487346, + 0.002301208005519584, + 0.0022149590076878667, + 0.002219000001787208, + 0.002256540989037603, + 0.00224283299758099, + 0.002234166007838212, + 0.0022602079989155754, + 0.002268541997182183, + 0.0022760840074624866, + 0.0022924579971004277, + 0.0028333329973975196, + 0.0028587090055225417, + 0.0028338749980321154, + 0.002845582988811657, + 0.002814959007082507, + 0.0028303329891059548, + 0.002876415994251147, + 0.0028864589985460043, + 0.002870791999157518, + 0.002934917007223703, + 0.0028974579909117892, + 0.0029654999962076545, + 0.0030724579992238432, + 0.003023083001608029, + 0.003034749999642372, + 0.003041042000404559, + 0.0030222499917726964, + 0.0032241250009974465, + 0.0030978750000940636, + 0.0034102499921573326, + 0.003295082991826348 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "speedup_cached_vs_uncached_sdpa": 4.4096473226579604, + "speedup_cached_vs_uncached_mha": 5.674890241964151 + }, + { + "batch": 1, + "prompt_len": 512, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_mha": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 6.05304200144019, + "p95": 6.238074795692228, + "min": 5.484834007802419, + "max": 6.267457996727899, + "iqr": 0.1824999926611781, + "mean": 5.972783599281684, + "cv": 0.0498362589748911 + }, + "decode_ms_per_step": { + "count": 315, + "median": 0.5246250075288117, + "p95": 0.6053957011317835, + "min": 0.5016250070184469, + "max": 0.8420410013059154, + "iqr": 0.02954150113509968, + "mean": 0.5388221878092736, + "cv": 0.08154910752375169 + }, + "total_ms": { + "count": 5, + "median": 39.46974400605541, + "p95": 41.684897482628, + "min": 38.96749802515842, + "max": 42.079789098352194, + "iqr": 1.1347860126988962, + "mean": 39.91858143126592, + "cv": 0.03244679119127361 + }, + "tokens_per_second": { + "count": 5, + "median": 1621.4951885723196, + "p95": 1642.3687034705295, + "min": 1520.920170260696, + "max": 1642.3943861799892, + "iqr": 46.468147936693185, + "mean": 1604.5747084683385, + "cv": 0.031490060295037 + }, + "median_step_ms_by_index": [ + 6.05304200144019, + 0.7561670063296333, + 0.5762910004705191, + 0.5511670024134219, + 0.5295830051181838, + 0.5253329873085022, + 0.5187500064494088, + 0.5206249916227534, + 0.5293750000419095, + 0.5189170042285696, + 0.5225000058999285, + 0.5283750069793314, + 0.5285840015858412, + 0.5232499970588833, + 0.5344169912859797, + 0.513833001605235, + 0.5257920129224658, + 0.5124589952174574, + 0.5142500012880191, + 0.5075419903732836, + 0.5127500044181943, + 0.5145830073161051, + 0.5179169966140762, + 0.5236249999143183, + 0.5364579992601648, + 0.5213339900365099, + 0.52541600598488, + 0.5131670041009784, + 0.5200419982429594, + 0.5185000045457855, + 0.5150409997440875, + 0.531750003574416, + 0.5307910032570362, + 0.5386670090956613, + 0.5225840141065419, + 0.5207919894019142, + 0.5186659982427955, + 0.5238340090727434, + 0.5252080009086058, + 0.5376669869292527, + 0.516917003551498, + 0.5190829979255795, + 0.5300830089254305, + 0.5777080077677965, + 0.5502090061781928, + 0.53404200298246, + 0.5411669990280643, + 0.524125003721565, + 0.5270410038065165, + 0.5165409966139123, + 0.5177080020075664, + 0.5176249978831038, + 0.5218339938437566, + 0.5297909956425428, + 0.5212079995544627, + 0.515791994985193, + 0.5174169928068295, + 0.5231669929344207, + 0.544166992767714, + 0.5416659987531602, + 0.5227079964242876, + 0.5222500039963052, + 0.5235830030869693, + 0.5445410060929134 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.005938041998888366, + 0.000749457991332747, + 0.000556332990527153, + 0.0005390000005718321, + 0.0005228750087553635, + 0.0005217090074438602, + 0.0005104169977130368, + 0.0005236249999143183, + 0.0005246250075288117, + 0.0005126250034663826, + 0.0005156249972060323, + 0.000517250009579584, + 0.0005189579969737679, + 0.0005067499878350645, + 0.0005416250060079619, + 0.0005115000094519928, + 0.0005257920129224658, + 0.0005124589952174574, + 0.0005253749986877665, + 0.0005070829938631505, + 0.0005117079999763519, + 0.0005133340018801391, + 0.0005179169966140762, + 0.0005236249999143183, + 0.0005207920039538294, + 0.0005176669947104529, + 0.00052541600598488, + 0.0005115830135764554, + 0.0005104579904582351, + 0.0005229579983279109, + 0.0005141669971635565, + 0.0005118749977555126, + 0.0005123750015627593, + 0.0005477079976117238, + 0.0005225840141065419, + 0.0005151249933987856, + 0.0005186659982427955, + 0.0005158749991096556, + 0.000515584004460834, + 0.0005141659930814058, + 0.0005166250048205256, + 0.0005190829979255795, + 0.0005195000121602789, + 0.0005321659991750494, + 0.0005368749989429489, + 0.0005312499997671694, + 0.0005513749929377809, + 0.000524125003721565, + 0.0005270410038065165, + 0.0005165409966139123, + 0.0005177080020075664, + 0.0005176249978831038, + 0.0005187500064494088, + 0.000516665997565724, + 0.0005165409966139123, + 0.0005144579918123782, + 0.0005174169928068295, + 0.0005126670002937317, + 0.0005206670030020177, + 0.0005170420045033097, + 0.0005160000000614673, + 0.0005222500039963052, + 0.0005235830030869693, + 0.0005168339994270355 + ], + [ + 0.006120541991549544, + 0.0008420410013059154, + 0.0006142920028651133, + 0.0005941670096945018, + 0.0005660840106429532, + 0.0005451670003822073, + 0.0005562910082517192, + 0.0005541669961530715, + 0.0005500830011442304, + 0.0005357909976737574, + 0.0005531250062631443, + 0.0005576249968726188, + 0.0005461669934447855, + 0.0005430830060504377, + 0.000547791991266422, + 0.0005514579970622435, + 0.0005434580089058727, + 0.0005467500013764948, + 0.0005440000095404685, + 0.0005424999981187284, + 0.0005469169991556555, + 0.0005447500006994233, + 0.0005640410090563819, + 0.0005443749978439882, + 0.0005452080076793209, + 0.0005532089999178424, + 0.0005669579986715689, + 0.0005815420008730143, + 0.000596166995819658, + 0.0005478329985635355, + 0.0005487500020535663, + 0.0005354160093702376, + 0.0005307910032570362, + 0.0005378340138122439, + 0.0005442500114440918, + 0.0005312499997671694, + 0.0005327920080162585, + 0.0005315000016707927, + 0.0005468330055009574, + 0.0006243329989956692, + 0.000759208996896632, + 0.0006816670065745711, + 0.0005875000060768798, + 0.0005777080077677965, + 0.0005628749931929633, + 0.0005554160015890375, + 0.0007033330039121211, + 0.0006339999963529408, + 0.0005687079974450171, + 0.0006311250035651028, + 0.0005656660068780184, + 0.0005522089923033491, + 0.0005453749909065664, + 0.0005430000019259751, + 0.0006456660048570484, + 0.0005852910107932985, + 0.0005548339977394789, + 0.000552707992028445, + 0.000544166992767714, + 0.0005455420032376423, + 0.0005388330027926713, + 0.0005447089934023097, + 0.0005463749985210598, + 0.0005445410060929134 + ], + [ + 0.005484834007802419, + 0.0007561670063296333, + 0.0005796670011477545, + 0.0005511670024134219, + 0.0005295830051181838, + 0.0005386660050135106, + 0.0005187500064494088, + 0.0005206249916227534, + 0.0005293750000419095, + 0.0005189170042285696, + 0.0005225000058999285, + 0.0006015830003889278, + 0.0006196670001372695, + 0.0005271249974612147, + 0.0005344169912859797, + 0.0005674999993061647, + 0.0005366249970393255, + 0.0005080410046502948, + 0.0005110839992994443, + 0.0005075419903732836, + 0.0005016250070184469, + 0.0005040420073783025, + 0.0005138749984325841, + 0.0005142079899087548, + 0.0005112079961691052, + 0.000516332991537638, + 0.0005478329985635355, + 0.0005131670041009784, + 0.0005200419982429594, + 0.0005062919954070821, + 0.0005053329950897023, + 0.0005329160048859194, + 0.0005491250049090013, + 0.0005386670090956613, + 0.0005186670023249462, + 0.0005207919894019142, + 0.0005067910096840933, + 0.0005112079961691052, + 0.0005072089988971129, + 0.00050224999722559, + 0.0005075840017525479, + 0.0005121249996591359, + 0.0005078329995740205, + 0.0005816249904455617, + 0.0005502090061781928, + 0.00053404200298246, + 0.0005327499966369942, + 0.0005209999944781885, + 0.0005229590024100617, + 0.0005115000094519928, + 0.0005114579980727285, + 0.0005104580050101504, + 0.000526624993653968, + 0.0005297909956425428, + 0.0005212079995544627, + 0.000515791994985193, + 0.0005172079982003197, + 0.0005231669929344207, + 0.0005562090082094073, + 0.0005432500038295984, + 0.0005227079964242876, + 0.0005191670061321929, + 0.000534083999809809, + 0.0005523750005522743 + ], + [ + 0.006267457996727899, + 0.0007865840016165748, + 0.0005727079987991601, + 0.0005605829937849194, + 0.000534624996362254, + 0.0005253329873085022, + 0.0005236659926595166, + 0.0005194999976083636, + 0.0005242500046733767, + 0.0005146250041434541, + 0.0005210000090301037, + 0.0005145420000189915, + 0.0005285840015858412, + 0.0005232499970588833, + 0.0005130839999765158, + 0.000513833001605235, + 0.000512875005370006, + 0.00051254199934192, + 0.00051270799303893, + 0.00051254199934192, + 0.0005127500044181943, + 0.0005145830073161051, + 0.0005148339987499639, + 0.0005159160064067692, + 0.0005364579992601648, + 0.0005213339900365099, + 0.0005188750074012205, + 0.000512332990183495, + 0.0005126250034663826, + 0.0005147910123923793, + 0.0005150409997440875, + 0.000531750003574416, + 0.0005193329998292029, + 0.000513791004777886, + 0.0005111659993417561, + 0.0005114999949000776, + 0.0005141669971635565, + 0.0005740000051446259, + 0.0005550000059884042, + 0.0005376669869292527, + 0.0005169170035514981, + 0.0005156250117579475, + 0.000530792007339187, + 0.0005419160006567836, + 0.0005489999894052744, + 0.0005251249967841431, + 0.0005397909990279004, + 0.0005270830006338656, + 0.0005270420078886673, + 0.0005151660006958991, + 0.000516665997565724, + 0.0005137080006534234, + 0.0005182079912628978, + 0.0005165410111658275, + 0.0005129579949425533, + 0.0005146250041434541, + 0.0005170419899513945, + 0.0005150840006535873, + 0.0005157080013304949, + 0.0005139169952599332, + 0.0005154579994268715, + 0.0005167910130694509, + 0.0005188329960219562, + 0.0005235419957898557 + ], + [ + 0.00605304200144019, + 0.0007427919917972758, + 0.0005762910004705191, + 0.0005506250017788261, + 0.0005214999982854351, + 0.0005204170010983944, + 0.0005094999942230061, + 0.0005115410021971911, + 0.000534459002665244, + 0.0005372909945435822, + 0.0005372079904191196, + 0.0005283750069793314, + 0.0005279169999994338, + 0.0005080830014776438, + 0.0005132910009706393, + 0.0005106250027893111, + 0.0005112499929964542, + 0.0005116669926792383, + 0.0005142500012880191, + 0.0005044999998062849, + 0.0005211249954300001, + 0.0005758750048698857, + 0.0005585000035353005, + 0.0005363749951357022, + 0.0005365840042941272, + 0.0005253749986877665, + 0.000524290997418575, + 0.0005599169962806627, + 0.0005407089920481667, + 0.0005185000045457855, + 0.0005151660006958991, + 0.0005112500075483695, + 0.0005455829959828407, + 0.0005947079916950315, + 0.0007048330007819459, + 0.000590375013416633, + 0.0005467500013764948, + 0.0005238340090727434, + 0.0005252080009086058, + 0.0005427500000223517, + 0.0005214999982854351, + 0.0005232500116107985, + 0.0005300830089254305, + 0.000633625008049421, + 0.0005744160007452592, + 0.0005365420074667782, + 0.0005411669990280643, + 0.0005207920039538294, + 0.0005268329987302423, + 0.0005232919938862324, + 0.0005246250075288117, + 0.0005204999906709418, + 0.0005218339938437566, + 0.0005403329996624961, + 0.0005256250005913898, + 0.0005302079953253269, + 0.0005198330036364496, + 0.0005272909911582246, + 0.000571708005736582, + 0.0005416659987531602, + 0.0005234170093899593, + 0.0005289590044412762, + 0.0005233330011833459, + 0.000552166995476 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + ] + } + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 5.998708002152853, + "p95": 6.643700203858316, + "min": 5.528250010684133, + "max": 6.798917005653493, + "iqr": 0.05395800690166652, + "mean": 6.063516600988805, + "cv": 0.0756722520376409 + }, + "decode_ms_per_step": { + "count": 315, + "median": 5.996249994495884, + "p95": 7.033924898132683, + "min": 5.570082998019643, + "max": 19.392084010178223, + "iqr": 0.37302100827218965, + "mean": 6.1667148254826545, + "cv": 0.18121541602851235 + }, + "total_ms": { + "count": 5, + "median": 385.0707919918932, + "p95": 427.3205496429, + "min": 375.36141903547104, + "max": 434.0337920584716, + "iqr": 22.568410015082918, + "mean": 394.56655060639605, + "cv": 0.061159205598200504 + }, + "tokens_per_second": { + "count": 5, + "median": 166.20320556887987, + "p95": 170.27334032467897, + "min": 147.45395674486593, + "max": 170.50233922403226, + "iqr": 9.544158343743334, + "mean": 162.66600652971326, + "cv": 0.05817810784396854 + }, + "median_step_ms_by_index": [ + 5.998708002152853, + 5.918374998145737, + 5.895458001759835, + 5.864000006113201, + 5.8712919999379665, + 6.089792004786432, + 5.882999990717508, + 5.946207995293662, + 5.64516699523665, + 5.697457992937416, + 5.793042000732385, + 5.745041999034584, + 5.665332995704375, + 5.680667003616691, + 5.810000002384186, + 5.793124990304932, + 5.731958997785114, + 5.762832995969802, + 5.770749994553626, + 6.055290999938734, + 5.7701250043464825, + 5.9007920062867925, + 5.791957999463193, + 5.771292009740137, + 5.785874993307516, + 5.843582999659702, + 5.824499996379018, + 6.047792005119845, + 6.021458000759594, + 5.8939170121448115, + 5.994958992232569, + 5.859458993654698, + 5.859500000951812, + 6.013958001858555, + 5.943083000602201, + 5.938417001743801, + 6.028457995853387, + 5.88758300000336, + 5.983249997370876, + 5.965917007415555, + 5.965167001704685, + 5.97058299172204, + 6.077457990613766, + 6.00441599090118, + 6.037374987499788, + 6.068624992622063, + 6.236625005840324, + 6.0457499930635095, + 6.017791005433537, + 6.002167006954551, + 6.143917009467259, + 5.997332991682924, + 6.122040998889133, + 6.113750001532026, + 6.067791997338645, + 6.188958010170609, + 6.062499989639036, + 6.066458998247981, + 6.154083996079862, + 6.1184999940451235, + 6.074999997508712, + 6.700334008201025, + 6.245208001928404, + 6.217207992449403 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.005998708002152853, + 0.005918374998145737, + 0.00584012501349207, + 0.005823542000143789, + 0.005800208993605338, + 0.005570082998019643, + 0.005685333992005326, + 0.006109583002398722, + 0.00564516699523665, + 0.005702042006305419, + 0.005813790994579904, + 0.005594250003923662, + 0.005600042000878602, + 0.005627999998978339, + 0.005648167003528215, + 0.00567337499524001, + 0.005731958997785114, + 0.005621917007374577, + 0.005770749994553626, + 0.005785208006273024, + 0.005689750003512017, + 0.005663208998157643, + 0.005711999998311512, + 0.005694458988728002, + 0.005692291000741534, + 0.005686459000571631, + 0.005824499996379018, + 0.0061668749985983595, + 0.006724958002450876, + 0.0057330839917995036, + 0.005724000002373941, + 0.005747959003201686, + 0.00578779200441204, + 0.00589058299374301, + 0.005943083000602201, + 0.005996249994495884, + 0.005891708002309315, + 0.005859915996552445, + 0.005884999991394579, + 0.005929833991103806, + 0.005834333001985215, + 0.0058974169951397926, + 0.005875999995623715, + 0.005962125011137687, + 0.005936583009315655, + 0.006006999989040196, + 0.006408582994481549, + 0.0060457499930635095, + 0.0059400419995654374, + 0.005898082992644049, + 0.006143917009467259, + 0.005956499997409992, + 0.0061876249965280294, + 0.006023458001436666, + 0.005954583000857383, + 0.006188958010170609, + 0.006044458001269959, + 0.005997583997668698, + 0.006163666999782436, + 0.006027791008818895, + 0.006050084004527889, + 0.006211999992956407, + 0.006279000008362345, + 0.006659291990217753 + ], + [ + 0.005968874989775941, + 0.006083792002755217, + 0.0064958339935401455, + 0.006136209005489945, + 0.006248374993447214, + 0.006145791994640604, + 0.006176124996272847, + 0.005722540998249315, + 0.00575304200174287, + 0.005707541000447236, + 0.005757499995524995, + 0.005745041999034584, + 0.007700417001615278, + 0.018752167001366615, + 0.019392084010178223, + 0.008932875003665686, + 0.007644790995982476, + 0.006998709010076709, + 0.007732707992545329, + 0.007195167010650039, + 0.007341916003497317, + 0.006396666998625733, + 0.006905458008986898, + 0.006463375000748783, + 0.006522167008370161, + 0.007069333005347289, + 0.007117374989320524, + 0.0067999159946339205, + 0.007458041000063531, + 0.006540791000588797, + 0.006168834006530233, + 0.005859458993654698, + 0.005859500000951812, + 0.006156832998385653, + 0.005918000009842217, + 0.005886833008844405, + 0.006576250001671724, + 0.00588758300000336, + 0.006076292003854178, + 0.005965917007415555, + 0.006429958011722192, + 0.005997041997034103, + 0.0061840419948566705, + 0.006924333996721543, + 0.0060697080043610185, + 0.0061415000091074035, + 0.006236625005840324, + 0.006549540994456038, + 0.006029124997439794, + 0.006242083996767178, + 0.005961250004475005, + 0.005997332991682924, + 0.006122040998889133, + 0.006113750001532026, + 0.005990707999444567, + 0.00612833299965132, + 0.006062499989639036, + 0.0062398329901043326, + 0.005967874996713363, + 0.006034125006408431, + 0.006274292012676597, + 0.006700334008201025, + 0.006245208001928404, + 0.006132125010481104 + ], + [ + 0.005528250010684133, + 0.005575582996243611, + 0.005590124987065792, + 0.005603541998425499, + 0.005767375012510456, + 0.005632333006360568, + 0.005651875006151386, + 0.005624292010907084, + 0.005617332994006574, + 0.005624083001748659, + 0.005793042000732385, + 0.00662933400599286, + 0.005882124998606741, + 0.005680667003616691, + 0.005810000002384186, + 0.005864499995368533, + 0.005658582987962291, + 0.005762832995969802, + 0.005737833998864517, + 0.006114415999036282, + 0.0057577919942559674, + 0.0059007920062867925, + 0.005791957999463193, + 0.005813499999931082, + 0.005785874993307516, + 0.006065792011213489, + 0.006378167003276758, + 0.005930291998083703, + 0.006021458000759594, + 0.006287957992753945, + 0.0059949589922325686, + 0.006062832995667122, + 0.006645665998803452, + 0.006013958001858555, + 0.006453542009694502, + 0.005938417001743801, + 0.006028457995853387, + 0.006098166995798238, + 0.005983249997370876, + 0.006258875000639819, + 0.005965167001704685, + 0.00597058299172204, + 0.0060827090055681765, + 0.00600441599090118, + 0.006037374987499788, + 0.0060686249926220626, + 0.006219624992809258, + 0.006650959010585211, + 0.006231083010789007, + 0.006002167006954551, + 0.006178875002660789, + 0.0062178750085877255, + 0.005999957997119054, + 0.006090249997214414, + 0.00611770800605882, + 0.0062592080066679046, + 0.006367833004333079, + 0.006066458998247981, + 0.006164416001411155, + 0.006632874996284954, + 0.006074999997508712, + 0.006924458997673355, + 0.0061661249928874895, + 0.006217207992449403 + ], + [ + 0.006798917005653493, + 0.006071833005989902, + 0.006541874987306073, + 0.006340291001833975, + 0.006168875013827346, + 0.006555500003742054, + 0.006075166995287873, + 0.005946207995293662, + 0.006184541998663917, + 0.005697457992937416, + 0.005960082999081351, + 0.005841334001161158, + 0.005647250000038184, + 0.00657529199088458, + 0.007412083010422066, + 0.005793124990304932, + 0.006674208008917049, + 0.006085041997721419, + 0.006253667001146823, + 0.006055290999938734, + 0.0057701250043464825, + 0.005950624996330589, + 0.005813790994579904, + 0.0057712920097401366, + 0.005846667001605965, + 0.005843582999659702, + 0.005730000004405156, + 0.006047792005119845, + 0.005907207989366725, + 0.0058939170121448115, + 0.0060733330028597265, + 0.005875500006368384, + 0.006074292003177106, + 0.006321624998236075, + 0.005946374993072823, + 0.00611712499812711, + 0.006056499987607822, + 0.0060402079980121925, + 0.006234957996639423, + 0.005990791003569029, + 0.006114250005339272, + 0.00652516599802766, + 0.006077457990613766, + 0.006395542004611343, + 0.006253250001464039, + 0.0060718749882653356, + 0.006278041997575201, + 0.005913166998652741, + 0.006017791005433537, + 0.006326207992970012, + 0.006199207986355759, + 0.007119083005818538, + 0.0069233749964041635, + 0.007128792000003159, + 0.006550749996677041, + 0.006506707999506034, + 0.007098292000591755, + 0.007018749995040707, + 0.006154083996079862, + 0.006991207992541604, + 0.006400958009180613, + 0.007293208007467911, + 0.006844250005087815, + 0.006282417001784779 + ], + [ + 0.006022832996677607, + 0.005887084000278264, + 0.005895458001759835, + 0.005864000006113201, + 0.0058712919999379665, + 0.006089792004786432, + 0.005882999990717508, + 0.006041458997060545, + 0.005593624999164604, + 0.005651416999171488, + 0.005631958003505133, + 0.005650000006426126, + 0.005665332995704375, + 0.0056549170112702996, + 0.005635333000100218, + 0.005692375008948147, + 0.005634165994706564, + 0.005718958011129871, + 0.005715375009458512, + 0.005756375001510605, + 0.00582162500359118, + 0.005761125008575618, + 0.005774833000032231, + 0.005766290996689349, + 0.005776207995950244, + 0.005773541997768916, + 0.005686541990144178, + 0.00576795800589025, + 0.005816749995574355, + 0.005729541997425258, + 0.005805125008919276, + 0.00581899999815505, + 0.00578533299267292, + 0.00584162499580998, + 0.005913125001825392, + 0.005876666997210123, + 0.005808500005514361, + 0.005878875002963468, + 0.005929249993641861, + 0.005848957996931858, + 0.005905708996579051, + 0.005932499989285134, + 0.005871583998668939, + 0.005937000008998439, + 0.005889874999411404, + 0.005905250000068918, + 0.005908333012484945, + 0.005873416987014934, + 0.005902917007915676, + 0.005920250012422912, + 0.005857208001543768, + 0.005972541999653913, + 0.006096334007452242, + 0.006455041992012411, + 0.006067791997338645, + 0.0060429169971030205, + 0.00600608300010208, + 0.005961874994682148, + 0.005969000005279668, + 0.0061184999940451235, + 0.005996374995447695, + 0.0060024999984307215, + 0.006041583008482121, + 0.005991209007333964 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_mha": { + "first_token_ms": { + "count": 5, + "median": 10.43329200183507, + "p95": 11.792208201950416, + "min": 10.167250002268702, + "max": 12.021125003229827, + "iqr": 0.5775829922640696, + "mean": 10.759433201747015, + "cv": 0.07008732935428347 + }, + "decode_ms_per_step": { + "count": 315, + "median": 11.957334005273879, + "p95": 13.971987600962168, + "min": 10.15854100114666, + "max": 24.440791996312328, + "iqr": 1.4116674865363166, + "mean": 12.128669961488672, + "cv": 0.13272964287371425 + }, + "total_ms": { + "count": 5, + "median": 770.3100799844833, + "p95": 808.5838331840932, + "min": 736.4082899293862, + "max": 815.1069579762407, + "iqr": 12.479792043450288, + "mean": 774.8656407755334, + "cv": 0.036506713498238905 + }, + "tokens_per_second": { + "count": 5, + "median": 83.08342531528235, + "p95": 86.14977550653478, + "min": 78.51730300389059, + "max": 86.90831007094845, + "iqr": 1.3255940651788904, + "mean": 82.68294376454054, + "cv": 0.036459712062265816 + }, + "median_step_ms_by_index": [ + 10.43329200183507, + 10.927166003966704, + 10.864541996852495, + 10.83608299086336, + 11.027291999198496, + 10.826166995684616, + 10.817832997418009, + 11.203125002793968, + 10.440707992529497, + 10.78883299487643, + 11.41520899545867, + 11.106416990514845, + 10.96491698990576, + 11.125167002319358, + 11.403665994293988, + 11.12174999434501, + 10.610416997224092, + 10.827292004250921, + 11.161416012328118, + 11.2176249967888, + 11.477874999400228, + 11.604749990510754, + 11.677625006996095, + 11.18641700304579, + 11.10208299360238, + 11.56174999778159, + 11.775666003813967, + 12.249083010829054, + 11.94679198670201, + 11.46862500172574, + 12.050582998199388, + 11.957334005273879, + 11.773416001233272, + 11.76925000618212, + 12.36287500069011, + 11.882374994456768, + 11.939083997276612, + 11.991750012384728, + 12.270291001186706, + 12.237458999152295, + 12.137042009271681, + 12.087667011655867, + 12.311792001128197, + 12.206875006086193, + 12.052124991896562, + 12.083958004950546, + 12.287250006920658, + 12.464125000406057, + 12.029457997414283, + 12.419624996255152, + 12.295082997297868, + 12.369625008432195, + 12.266541991266422, + 12.19670799036976, + 12.755832998664118, + 13.3182499994291, + 13.145667006028816, + 12.951958997291513, + 12.966875001438893, + 12.858917005360126, + 13.015375006943941, + 12.696250007138588, + 12.829290993977338, + 12.565208991873078 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.01043329200183507, + 0.010927166003966704, + 0.010864541996852495, + 0.01083608299086336, + 0.011027291999198496, + 0.010826166995684616, + 0.010817832997418009, + 0.010633375000907108, + 0.010248167003737763, + 0.010711124996305443, + 0.010901290996116586, + 0.011249625007621944, + 0.010897375002969056, + 0.011125167002319358, + 0.01082137500634417, + 0.011080500000389293, + 0.010600291992886923, + 0.010794832996907644, + 0.011161416012328118, + 0.010915457998635247, + 0.01151441600813996, + 0.011469249991932884, + 0.011190165998414159, + 0.010992083000019193, + 0.010768499996629544, + 0.011161583999637514, + 0.010960333995171823, + 0.011083041987149045, + 0.011034875002223998, + 0.011056999996071681, + 0.011129708000225946, + 0.01137591699080076, + 0.01088583399541676, + 0.011699042006512173, + 0.011706249992130324, + 0.011769541990361176, + 0.011434375002863817, + 0.01150454199523665, + 0.011657916998956352, + 0.012131584007875063, + 0.011706292003509589, + 0.012087667011655867, + 0.012019084009807557, + 0.012182790989754722, + 0.011864790998515673, + 0.011757957996451296, + 0.011935708011151291, + 0.012091082986444235, + 0.012029457997414283, + 0.012840583003708161, + 0.012032166996505111, + 0.012262958000064828, + 0.012266541991266422, + 0.01219670799036976, + 0.012049083001329564, + 0.012306290998822078, + 0.01211837500159163, + 0.012246790996869095, + 0.01225091599917505, + 0.012822624994441867, + 0.012573749991133809, + 0.012459000005037524, + 0.012462792001315393, + 0.012446541993995197 + ], + [ + 0.010298958004568703, + 0.010410917006083764, + 0.010562999988906085, + 0.010420790989883244, + 0.010282917006406933, + 0.010453790993778966, + 0.010393207994638942, + 0.010408292000647634, + 0.01015854100114666, + 0.01053554100508336, + 0.01125816699641291, + 0.010745416991994716, + 0.010606167008518241, + 0.010619374996167608, + 0.010708707995945588, + 0.011271374998614192, + 0.010610416997224092, + 0.010767208994366229, + 0.010955083998851478, + 0.010942209002678283, + 0.010748832995886914, + 0.011091874999692664, + 0.011677625006996095, + 0.011009042005753145, + 0.010970832998282276, + 0.010953333010547794, + 0.010919249994913116, + 0.011115790999610908, + 0.01089874999888707, + 0.010972333009704016, + 0.011103249998996034, + 0.011957334005273879, + 0.010842542003956623, + 0.011612916990998201, + 0.011346458995831199, + 0.011414041990065016, + 0.011290249996818602, + 0.011529833005624823, + 0.011420249997172505, + 0.011532042000908405, + 0.012023292001686059, + 0.011619915996561758, + 0.012311792001128197, + 0.011760875000618398, + 0.01197558299463708, + 0.011801957996794954, + 0.0118908750009723, + 0.011786875009420328, + 0.011989124992396683, + 0.01184133299102541, + 0.011962749995291233, + 0.011839459009934217, + 0.011693874999764375, + 0.011877291995915584, + 0.012090958000044338, + 0.02072104199032765, + 0.019071625007200055, + 0.016818374991999008, + 0.023187750004581176, + 0.016409082993050106, + 0.015078166004968807, + 0.01313279200985562, + 0.014348833006806672, + 0.015961249999236315 + ], + [ + 0.010876540996832773, + 0.01181279199954588, + 0.011510625001392327, + 0.011154207997606136, + 0.011776667000958696, + 0.011198415988474153, + 0.011087165999924764, + 0.011203125002793968, + 0.010440707992529497, + 0.01078883299487643, + 0.011481875000754371, + 0.01213149999966845, + 0.0110424580052495, + 0.01233837500330992, + 0.011854165990371257, + 0.011362167002516799, + 0.011673375003738329, + 0.024440791996312328, + 0.015946708008414134, + 0.012076291997800581, + 0.012984957997105084, + 0.014442500003497116, + 0.013946999999461696, + 0.01249745799577795, + 0.011333041999023408, + 0.011946707993047312, + 0.012351375000434928, + 0.012249083010829054, + 0.012885750009445474, + 0.013455916996463202, + 0.012717208999674767, + 0.01258754200534895, + 0.011773416001233272, + 0.012503165999078192, + 0.01249541700235568, + 0.011882374994456768, + 0.012605957992491312, + 0.01355074999446515, + 0.012940665998030454, + 0.012932916011777706, + 0.012137042009271681, + 0.012825291996705346, + 0.013095541988150217, + 0.01300904199888464, + 0.01235745899612084, + 0.013755708991084248, + 0.013503125010174699, + 0.013202957998146303, + 0.012062416994012892, + 0.01327637498616241, + 0.01315729200723581, + 0.01316249999217689, + 0.012467542008380406, + 0.01286158298898954, + 0.013544124987674877, + 0.0133182499994291, + 0.013145667006028816, + 0.012951958997291513, + 0.012966875001438893, + 0.01317862501309719, + 0.014329375000670552, + 0.01330945800873451, + 0.013178459004848264, + 0.014030292004463263 + ], + [ + 0.012021125003229827, + 0.011962958000367507, + 0.011436417000368237, + 0.011489832992083393, + 0.012018166002235375, + 0.011233708006329834, + 0.011315750001813285, + 0.01132362500356976, + 0.010785125006805174, + 0.010974125005304813, + 0.011468374999822117, + 0.011106416990514845, + 0.011322292004479095, + 0.011916584000573494, + 0.011794541002018377, + 0.01112174999434501, + 0.011164875002577901, + 0.011617541997111402, + 0.01145495798846241, + 0.0112176249967888, + 0.011477874999400228, + 0.012081374996341765, + 0.012422209008946083, + 0.01258550000784453, + 0.013300250007887371, + 0.012035207997541875, + 0.011775666003813967, + 0.012475833005737513, + 0.01194679198670201, + 0.01146862500172574, + 0.012050582998199388, + 0.01188083399028983, + 0.012381124994135462, + 0.01176925000618212, + 0.01236287500069011, + 0.012311166996369138, + 0.011939083997276612, + 0.011991750012384728, + 0.012270291001186706, + 0.012237458999152295, + 0.012751707996358164, + 0.012377459002891555, + 0.013117042006342672, + 0.013253583005280234, + 0.013265957997646183, + 0.013323125007445924, + 0.012670957992668264, + 0.012943374997121282, + 0.013117832990246825, + 0.012419624996255152, + 0.013202458998421207, + 0.01349391700932756, + 0.012862083007348701, + 0.013142332987627015, + 0.012755832998664118, + 0.012899624998681247, + 0.013344999999389984, + 0.013200374989537522, + 0.013252792006824166, + 0.012806708007701673, + 0.013015375006943941, + 0.012696250007138588, + 0.012503167003160343, + 0.012565208991873078 + ], + [ + 0.010167250002268702, + 0.01037479100341443, + 0.010328000003937632, + 0.010336749997804873, + 0.010241625001071952, + 0.010455583003931679, + 0.010470790992258117, + 0.01133483300509397, + 0.010483916994417086, + 0.011473541002487764, + 0.01141520899545867, + 0.010708542002248578, + 0.01096491698990576, + 0.010975416997098364, + 0.011403665994293988, + 0.010915459002717398, + 0.010430082998936996, + 0.010827292004250921, + 0.011014874995453283, + 0.011538042002939619, + 0.010822917000041343, + 0.011604749990510754, + 0.011062916993978433, + 0.01118641700304579, + 0.01110208299360238, + 0.01156174999778159, + 0.017097916002967395, + 0.018749583992757834, + 0.01389408299291972, + 0.015140541989239864, + 0.012256875008461066, + 0.012455166011932306, + 0.011955082998611033, + 0.012686166999628767, + 0.012365583010250703, + 0.01228679199994076, + 0.012415457997121848, + 0.01207895799598191, + 0.012493041998823173, + 0.012864791002357379, + 0.012196124996989965, + 0.011962459000642411, + 0.012216707997140475, + 0.012206875006086193, + 0.012052124991896562, + 0.012083958004950546, + 0.012287250006920658, + 0.012464125000406057, + 0.011984541997662745, + 0.012246333004441112, + 0.012295082997297868, + 0.012369625008432195, + 0.012102915992727503, + 0.012184791004983708, + 0.012854583008447662, + 0.01366420900740195, + 0.012519374999101274, + 0.012761166988639161, + 0.012759167002514005, + 0.012858917005360126, + 0.012525707992608659, + 0.012553916007163934, + 0.012829290993977338, + 0.012389374998747371 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "speedup_cached_vs_uncached_sdpa": 9.756100569915429, + "speedup_cached_vs_uncached_mha": 19.516470131306225 + }, + { + "batch": 1, + "prompt_len": 1024, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_mha": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 13.47258299938403, + "p95": 14.67161720211152, + "min": 12.451207992853597, + "max": 14.693459001136944, + "iqr": 2.0469170121941715, + "mean": 13.54776659864001, + "cv": 0.07929287780439645 + }, + "decode_ms_per_step": { + "count": 315, + "median": 0.8073330100160092, + "p95": 0.8783296041656286, + "min": 0.6476660055341199, + "max": 1.3297920086188242, + "iqr": 0.17302099877269939, + "mean": 0.7732203619236807, + "cv": 0.12910230469656211 + }, + "total_ms": { + "count": 5, + "median": 65.78345801972318, + "p95": 67.5273689936148, + "min": 55.2004549826961, + "max": 67.61212799756322, + "iqr": 11.669459956465289, + "mean": 62.26064939983189, + "cv": 0.1017812101486334 + }, + "tokens_per_second": { + "count": 5, + "median": 972.8889591181347, + "p95": 1158.0808171065214, + "min": 946.5757386353317, + "max": 1159.4107334814962, + "iqr": 200.2148215596127, + "mean": 1036.8365825777191, + "cv": 0.10543680451576416 + }, + "median_step_ms_by_index": [ + 13.47258299938403, + 1.1468340089777485, + 0.8272919949376956, + 0.8264159987447783, + 0.8150419889716431, + 0.7972919993335381, + 0.8283330098493025, + 0.8342499932041392, + 0.7317499985219911, + 0.7364580087596551, + 0.73533300019335, + 0.7303750026039779, + 0.7541250088252127, + 0.7913750014267862, + 0.802416994702071, + 0.8225839992519468, + 0.8289999968837947, + 0.8229590021073818, + 0.8081249980023131, + 0.7874579896451905, + 0.8284159994218498, + 0.7319589931285009, + 0.7939579954836518, + 0.8187499915948138, + 0.8067089947871864, + 0.8318750042235479, + 0.7964169926708564, + 0.7731249934295192, + 0.7967499986989424, + 0.819332999526523, + 0.8204579935409129, + 0.7115420012269169, + 0.7053330045891926, + 0.7327079947572201, + 0.8200830052373931, + 0.8019579981919378, + 0.8279169996967539, + 0.8095410012174398, + 0.8355830068467185, + 0.8276250009657815, + 0.808083001174964, + 0.8242909971158952, + 0.8264170028269291, + 0.8301249908981845, + 0.8360830106539652, + 0.8462910045636818, + 0.839166998048313, + 0.8355829922948033, + 0.8317920000990853, + 0.8258330053649843, + 0.8229170052800328, + 0.7126249984139577, + 0.7315830007428303, + 0.8112089999485761, + 0.796124993939884, + 0.816041007055901, + 0.8247499936260283, + 0.82579099398572, + 0.8141669968608767, + 0.7668329926673323, + 0.7281250000232831, + 0.8228340011555701, + 0.7996250060386956, + 0.8202499884646386 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.012451207992853597, + 0.0013297920086188242, + 0.0007814589916961268, + 0.0006817500106990337, + 0.0006803750002291054, + 0.0007372910040430725, + 0.0006929579976713285, + 0.0006700829981127754, + 0.000661625002976507, + 0.0006578330066986382, + 0.0006627080001635477, + 0.0006649999995715916, + 0.0006506659992737696, + 0.0006492910033557564, + 0.000659291006741114, + 0.0006496249989140779, + 0.0006562079943250865, + 0.0006597500032512471, + 0.0006552910053869709, + 0.0006682910025119781, + 0.0006577080057468265, + 0.0006495419947896153, + 0.0006590419943677261, + 0.0006898749998072162, + 0.0006548329984070733, + 0.0007071250001899898, + 0.0006645829998888075, + 0.0006660829967586324, + 0.0006598340114578605, + 0.0006605829985346645, + 0.0006602920038858429, + 0.0006627919938182458, + 0.0006770000036340207, + 0.0006730419991072267, + 0.0006587499956367537, + 0.0006716659991070628, + 0.0006580830086022615, + 0.00066149998747278, + 0.0006526249926537275, + 0.0006569170072907582, + 0.0006541659968206659, + 0.0006633340090047568, + 0.0006556669977726415, + 0.0006708329892717302, + 0.0006727500003762543, + 0.0006702499958919361, + 0.0006708750006509945, + 0.0006697500066366047, + 0.0006516249995911494, + 0.0006700839876430109, + 0.0006552920094691217, + 0.0006476660055341199, + 0.0006608749972656369, + 0.0006598750042030588, + 0.0006505419878521934, + 0.0006634999881498516, + 0.000656916992738843, + 0.0006767089944332838, + 0.0006817079993197694, + 0.0006576250016223639, + 0.00066562500433065, + 0.0007217499951366335, + 0.000671208996209316, + 0.0006594580045202747 + ], + [ + 0.014584250006009825, + 0.0011731249978765845, + 0.0008272919949376956, + 0.0008677919977344573, + 0.0008711670088814571, + 0.0008663339976919815, + 0.0008470000029774383, + 0.0009060419979505241, + 0.0008787090046098456, + 0.0010380840103607625, + 0.00085404199489858, + 0.0008114580123219639, + 0.0007605840073665604, + 0.0007913750014267862, + 0.000802416994702071, + 0.0008225839992519468, + 0.0008289999968837947, + 0.0008747079991735518, + 0.0008298330067191273, + 0.0008451670000795275, + 0.0008418750076089054, + 0.0008330839918926358, + 0.0007939579954836518, + 0.0008439999946858734, + 0.0008094999939203262, + 0.0008318750042235479, + 0.0008701669867150486, + 0.0008429169974988326, + 0.0008431660098722205, + 0.0008315830054925755, + 0.0008204579935409129, + 0.0008287919918075204, + 0.0007368329970631748, + 0.0007327079947572201, + 0.0008234999986598268, + 0.0008019579981919378, + 0.0008495830115862191, + 0.0008095410012174398, + 0.0008519160037394613, + 0.0008339589985553175, + 0.000808083001174964, + 0.0008493750065099448, + 0.0008673749980516732, + 0.0008389169961446896, + 0.0008507909951731563, + 0.0009097080037463456, + 0.0008770840067882091, + 0.0008770419954089448, + 0.0008395409968215972, + 0.0008756250026635826, + 0.0008520839910488576, + 0.0008039579988690093, + 0.0007771670061629266, + 0.0008308749966090545, + 0.000796124993939884, + 0.000816041007055901, + 0.0008247499936260283, + 0.0008474170026602224, + 0.0008141669968608767, + 0.0008036669896682724, + 0.0008435000054305419, + 0.0008364170062122867, + 0.0007996250060386956, + 0.0008604579925304279 + ], + [ + 0.01347258299938403, + 0.0010882079950533807, + 0.000840458000311628, + 0.0008586670010117814, + 0.0008565419993828982, + 0.0007972919993335381, + 0.0008457499934593216, + 0.0008574999956181273, + 0.0007317499985219911, + 0.0007364580087596551, + 0.0007499170023947954, + 0.0007414589927066118, + 0.0008073330100160092, + 0.0008537500107195228, + 0.0008222499891417101, + 0.0008553749939892441, + 0.000829625001642853, + 0.0008629170042695478, + 0.0008293749997392297, + 0.0008428750006714836, + 0.0008284159994218498, + 0.0007319589931285009, + 0.0008247920050052926, + 0.0008273749990621582, + 0.0008067089947871864, + 0.0008336669998243451, + 0.0007964169926708564, + 0.0007731249934295192, + 0.0008359580097021535, + 0.000819332999526523, + 0.0008781670039752498, + 0.0008327920077135786, + 0.0008419590012636036, + 0.0008274159918073565, + 0.0008706660009920597, + 0.000816041007055901, + 0.0008279169996967539, + 0.0008365410030819476, + 0.0008355830068467185, + 0.0008276250009657815, + 0.0008365829999092966, + 0.0008276669977931306, + 0.0008264170028269291, + 0.0008301249908981845, + 0.0008360830106539652, + 0.0008462910045636818, + 0.000839166998048313, + 0.0008355829922948033, + 0.0008560420101275668, + 0.0008313329890370369, + 0.0008229170052800328, + 0.0008245839999290183, + 0.0008161250007105991, + 0.0008489589963573962, + 0.00080750000779517, + 0.0008427080028923228, + 0.0008417909994022921, + 0.00082579099398572, + 0.0008469580061500892, + 0.0008288750104838982, + 0.0008442079997621477, + 0.0008228340011555701, + 0.0008444169943686575, + 0.0008479579992126673 + ], + [ + 0.014693459001136944, + 0.0011468340089777485, + 0.0009672089945524931, + 0.0008264159987447783, + 0.0008150419889716431, + 0.0008741250057937577, + 0.0008283330098493025, + 0.0008342499932041392, + 0.0008207080099964514, + 0.0008467909938190132, + 0.00073533300019335, + 0.0007303750026039779, + 0.0007541250088252127, + 0.0008731669950066134, + 0.0008290000114357099, + 0.0008566250035073608, + 0.000869208000949584, + 0.0008229590021073818, + 0.0008081249980023131, + 0.0007874579896451905, + 0.0008322080102516338, + 0.0008042499975999817, + 0.00083683300181292, + 0.0008187499915948138, + 0.0008481660042889416, + 0.0008414589974563569, + 0.0008141669968608767, + 0.0008479999960400164, + 0.0007967499986989424, + 0.0008252079860540107, + 0.0008372090087505057, + 0.0007115420012269169, + 0.0007053330045891926, + 0.0008324590016854927, + 0.0008200830052373931, + 0.000846416995045729, + 0.000876457997947, + 0.0008588749915361404, + 0.0009440829890081659, + 0.0009081250027520582, + 0.0007833749987185001, + 0.0008242909971158952, + 0.0008450829918729141, + 0.000886791996890679, + 0.0008676249999552965, + 0.0008533340005669743, + 0.0008524589939042926, + 0.0008508750033797696, + 0.0008317920000990853, + 0.0008258330053649843, + 0.0008406660053879023, + 0.0007126249984139577, + 0.0007315830007428303, + 0.0008112089999485761, + 0.0008697079902049154, + 0.0008310000121127814, + 0.0008510829939041287, + 0.000912499992409721, + 0.0008462920086458325, + 0.0007668329926673323, + 0.0007281250000232831, + 0.0008277500019175932, + 0.0008913330093491822, + 0.0008202499884646386 + ], + [ + 0.012537332993815653, + 0.001021417003357783, + 0.0007176660001277924, + 0.0006817499961471185, + 0.000690041997586377, + 0.0006726249994244426, + 0.0006635419995291159, + 0.0006794590008212253, + 0.000661084006424062, + 0.0006605840026168153, + 0.0006587499956367537, + 0.0006590830016648397, + 0.0006565829971805215, + 0.0006845419993624091, + 0.0006778329989174381, + 0.0006590419943677261, + 0.0006608749972656369, + 0.0006543329945998266, + 0.0006558750028489158, + 0.0006624579982599244, + 0.0007112919993232936, + 0.0006586669915122911, + 0.0006549999961862341, + 0.000675584000418894, + 0.0006624999950872734, + 0.0006756250077160075, + 0.0006571669946424663, + 0.0006539580062963068, + 0.0006638329941779375, + 0.0007296670082723722, + 0.000670000008540228, + 0.0006782910058973357, + 0.0006685830012429506, + 0.0006671250012004748, + 0.0006655420002061874, + 0.0006644579989369959, + 0.000657666998449713, + 0.0006588329997612163, + 0.0006537500012200326, + 0.0006659579958068207, + 0.0009514580015093088, + 0.0007895000017015263, + 0.0007833340059733018, + 0.0006859159911982715, + 0.0006995420117164031, + 0.00067758301156573, + 0.0006677920027868822, + 0.0006611669959966093, + 0.0006605829985346645, + 0.0006662920059170574, + 0.000676875002682209, + 0.0006719160010106862, + 0.0006587079988094047, + 0.0006587910029338673, + 0.0006591669953195378, + 0.0006571249978151172, + 0.0006547499942826107, + 0.0006612079887418076, + 0.0006547500088345259, + 0.0006582909991266206, + 0.0006642080115852877, + 0.0006586250092368573, + 0.000668499997118488, + 0.0006644160021096468 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + ] + } + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 13.483833987265825, + "p95": 13.766974591999315, + "min": 12.037749998853542, + "max": 13.834040990332142, + "iqr": 1.4154170057736337, + "mean": 12.987525193602778, + "cv": 0.06605526011942893 + }, + "decode_ms_per_step": { + "count": 315, + "median": 15.507707998040132, + "p95": 16.49378290603636, + "min": 11.833625001600012, + "max": 33.63558401179034, + "iqr": 2.991979505168274, + "mean": 14.694696139324353, + "cv": 0.13939552901160845 + }, + "total_ms": { + "count": 5, + "median": 1017.0902500103693, + "p95": 1033.9767987752566, + "min": 809.5406179199927, + "max": 1038.1405399675714, + "iqr": 205.64816605474334, + "mean": 938.753381971037, + "cv": 0.1249478435704878 + }, + "tokens_per_second": { + "count": 5, + "median": 62.9246028062382, + "p95": 79.01563013813313, + "min": 61.64868583400007, + "max": 79.05718204040153, + "iqr": 15.93914393218563, + "mean": 69.07803436131465, + "cv": 0.1307219212584541 + }, + "median_step_ms_by_index": [ + 13.483833987265825, + 15.26016699790489, + 15.14162500097882, + 15.657249998184852, + 15.40870800090488, + 15.34949999768287, + 15.418875002069399, + 15.42570799938403, + 15.34216699656099, + 15.365625004051253, + 15.38162499491591, + 15.444999997271225, + 15.372374997241423, + 15.45437501044944, + 15.330084002926014, + 15.29300000402145, + 15.298958998755552, + 15.249625008436851, + 15.31658299791161, + 16.036500004702248, + 15.635582996765152, + 15.528499992797151, + 15.367125000921078, + 15.430125000420958, + 15.707958998973481, + 15.52462499239482, + 15.442624993738718, + 15.507707998040132, + 15.388290994451381, + 15.537083992967382, + 15.537375002168119, + 15.804166992893443, + 16.011375002563, + 15.869082999415696, + 15.579082988551818, + 15.672624998842366, + 15.650750006898306, + 15.581083003780805, + 15.703833007137291, + 15.762916998937726, + 15.474582993192598, + 15.68779200897552, + 15.714582987129688, + 15.779500012286007, + 15.751792001537979, + 15.737624999019317, + 16.34108398866374, + 15.842624998185784, + 15.86633300757967, + 15.731499996036291, + 15.757207991555333, + 15.627749991836026, + 15.910290996544063, + 16.437583006336354, + 15.96425000752788, + 15.725833000033163, + 15.68425000004936, + 15.715417001047172, + 15.808208001544699, + 15.817833002074622, + 15.839624989894219, + 15.734084008727223, + 15.865833993302658, + 15.741791998152621 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.012083291992894374, + 0.012441208004020154, + 0.012285707998671569, + 0.013433458007057197, + 0.012734207994071767, + 0.01210904199979268, + 0.011912082991329953, + 0.012157250006566755, + 0.011987165999016725, + 0.01192004200129304, + 0.011909917011507787, + 0.01206754099985119, + 0.01291558300727047, + 0.012585958000272512, + 0.01264679100131616, + 0.012225875005242415, + 0.012182209000457078, + 0.011998082991340198, + 0.012052499994751997, + 0.012072083001839928, + 0.012235999995027669, + 0.012270582999917679, + 0.012143125000875443, + 0.01198754200595431, + 0.01225404099386651, + 0.01281524999649264, + 0.012555458000861108, + 0.012150082999141887, + 0.012202332989545539, + 0.012302125003770925, + 0.013112916989484802, + 0.012693208002019674, + 0.012896290994831361, + 0.013433874992188066, + 0.013481458008754998, + 0.012968790993909352, + 0.012507832987466827, + 0.012461374994018115, + 0.012415958000929095, + 0.012624458991922438, + 0.012540666008135304, + 0.012566292003612034, + 0.012434708987711929, + 0.012637749998248182, + 0.012454041003366001, + 0.012608166987774894, + 0.01294999998935964, + 0.013468250006553717, + 0.013038249991950579, + 0.012762625003233552, + 0.012770083005307242, + 0.012803541991161183, + 0.01263650000328198, + 0.012948167001013644, + 0.013322500002686866, + 0.013442999988910742, + 0.014327125012641773, + 0.013556125006289221, + 0.01337749999947846, + 0.013508207994163968, + 0.0133059170038905, + 0.013100290991133079, + 0.012868207995779812, + 0.012879999994765967 + ], + [ + 0.013483833987265825, + 0.01526016699790489, + 0.015888041991274804, + 0.015657249998184852, + 0.016721583000617102, + 0.015418583003338426, + 0.015418875002069399, + 0.015678916999604553, + 0.0156420419953065, + 0.015843417000724003, + 0.015956749994074926, + 0.015795333005371504, + 0.015617540993844159, + 0.01545437501044944, + 0.015330084002926014, + 0.015530582997598685, + 0.01538645799155347, + 0.015690540996729396, + 0.015988999992259778, + 0.01613479199295398, + 0.015876375007792376, + 0.015637624994269572, + 0.01644158299313858, + 0.015869916998781264, + 0.016005166005925275, + 0.01572637500066776, + 0.015578457998344675, + 0.015689250009017996, + 0.016044500007410534, + 0.016404582987888716, + 0.015882834006333724, + 0.01655566600675229, + 0.01611879200208932, + 0.016025791002903134, + 0.015806583003723063, + 0.016064750001532957, + 0.016682708010193892, + 0.015776957996422425, + 0.01626925000164192, + 0.015785167008289136, + 0.015746874996693805, + 0.01568779200897552, + 0.016067415999714285, + 0.015834334000828676, + 0.015820417000213638, + 0.01671866599645, + 0.01634108398866374, + 0.015842624998185784, + 0.01586633300757967, + 0.01573149999603629, + 0.015763834002427757, + 0.015627749991836026, + 0.016380583998397924, + 0.016556042013689876, + 0.01596425000752788, + 0.015725833000033163, + 0.01690204199985601, + 0.015816958009963855, + 0.016359665998606943, + 0.016425875001004897, + 0.016133249999256805, + 0.01582708400383126, + 0.01594870899862144, + 0.015860832994803786 + ], + [ + 0.013498708998668008, + 0.015531333003309555, + 0.01514162500097882, + 0.01579054100147914, + 0.01540870800090488, + 0.01602270800503902, + 0.01613450000877492, + 0.01542570799938403, + 0.015379625008790754, + 0.015365625004051253, + 0.01538162499491591, + 0.015444999997271225, + 0.015513082995312288, + 0.016203207997023128, + 0.015522959001827985, + 0.015486040996620432, + 0.015298958998755552, + 0.015249625008436851, + 0.01531658299791161, + 0.016036500004702248, + 0.017118999996455386, + 0.016147583999554627, + 0.01653025000996422, + 0.01619116599613335, + 0.01570795899897348, + 0.01552462499239482, + 0.015442624993738718, + 0.015507707998040132, + 0.015388290994451381, + 0.015537083992967382, + 0.015537375002168119, + 0.015804166992893443, + 0.01605208299588412, + 0.024079832990537398, + 0.018487166002159938, + 0.016085249997559004, + 0.015756875000079162, + 0.015741792012704536, + 0.015714417007984594, + 0.0163884590001544, + 0.01601249999657739, + 0.016290083003696054, + 0.015714582987129688, + 0.015779500012286007, + 0.01576708299398888, + 0.015737624999019317, + 0.016404541995143518, + 0.03363558401179034, + 0.01642720800009556, + 0.015956624993123114, + 0.015819124993868172, + 0.015688083003624342, + 0.015910290996544063, + 0.016437583006336354, + 0.016348250006558374, + 0.015926958993077278, + 0.01582462499209214, + 0.015749790996778756, + 0.01581495799473487, + 0.01581783300207462, + 0.01583962498989422, + 0.015734084008727223, + 0.015865833993302658, + 0.01574179199815262 + ], + [ + 0.013834040990332142, + 0.015547417002380826, + 0.015856999991228804, + 0.016046999997342937, + 0.01597629200841766, + 0.01534949999768287, + 0.015469000005396083, + 0.015463292002095841, + 0.01534216699656099, + 0.015409625004394911, + 0.016250332992058247, + 0.015707166996435262, + 0.015372374997241423, + 0.015455707994988188, + 0.015518583997618407, + 0.015293000004021451, + 0.015467250006622635, + 0.01597849999961909, + 0.015788374992553145, + 0.01619191700592637, + 0.01563558299676515, + 0.015528499992797151, + 0.015367125000921078, + 0.015430125000420958, + 0.02001987500989344, + 0.016212499991524965, + 0.016200792000745423, + 0.015628499997546896, + 0.015571666997857392, + 0.015571625001030043, + 0.015562208005576394, + 0.01606691699998919, + 0.016011375002563, + 0.015869082999415696, + 0.015579082988551818, + 0.015672624998842366, + 0.015650750006898306, + 0.015581083003780805, + 0.01570383300713729, + 0.015762916998937726, + 0.015474582993192598, + 0.016962292007519864, + 0.016512916001374833, + 0.01615999999921769, + 0.01575179200153798, + 0.016334291998646222, + 0.01745191701047588, + 0.016453832999104634, + 0.016217249998589978, + 0.015952417001244612, + 0.015757207991555333, + 0.015652000001864508, + 0.016567291007959284, + 0.016485583008034155, + 0.01598216700949706, + 0.01586245901125949, + 0.01568425000004936, + 0.01571541700104717, + 0.0158082080015447, + 0.016262499993899837, + 0.016468957997858524, + 0.016149624992976896, + 0.015906749991700053, + 0.015803417001734488 + ], + [ + 0.012037749998853542, + 0.01211258300463669, + 0.011964333010837436, + 0.012974417011719197, + 0.013058166994596832, + 0.012586458993609995, + 0.012296582994167693, + 0.012168375003966503, + 0.012094541991245933, + 0.011978791997535154, + 0.011833625001600012, + 0.011886457999935374, + 0.012144458989496343, + 0.012069874996086583, + 0.012049541008309461, + 0.011942625002120622, + 0.012060332999681123, + 0.012088624993339181, + 0.012162875005742535, + 0.012410791998263448, + 0.01255112499347888, + 0.013078083997243084, + 0.012670040989178233, + 0.012407292000716552, + 0.012373417004710063, + 0.012358791995211504, + 0.012321333008003421, + 0.012148542009526864, + 0.01256358298996929, + 0.012364374997559935, + 0.013572166993981227, + 0.012991708994377404, + 0.01334925000264775, + 0.013281833002110943, + 0.012885459000244737, + 0.012717417004751042, + 0.012884582989499904, + 0.012648792006075382, + 0.0126215409982251, + 0.013145874996553175, + 0.013483333008480258, + 0.013221750006778166, + 0.01266458300233353, + 0.012775208000675775, + 0.012544458993943408, + 0.012597875000210479, + 0.013270041992655024, + 0.012744708001264371, + 0.013065915991319343, + 0.013833000004524365, + 0.013241250009741634, + 0.012893292005173862, + 0.012747291999403387, + 0.012993957992875949, + 0.012818333998438902, + 0.012943915993673727, + 0.013753541992628016, + 0.013463832990964875, + 0.01300716699915938, + 0.013006666005821899, + 0.012882374998298474, + 0.012983709006221034, + 0.012982457992620766, + 0.012898582994239405 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_mha": { + "first_token_ms": { + "count": 5, + "median": 36.79004199511837, + "p95": 37.51055000757333, + "min": 36.50704200845212, + "max": 37.62075000850018, + "iqr": 0.2844160044332966, + "mean": 36.95458360307384, + "cv": 0.011425174686059001 + }, + "decode_ms_per_step": { + "count": 315, + "median": 37.69974999886472, + "p95": 40.57857500738464, + "min": 33.73125000507571, + "max": 55.563916990649886, + "iqr": 2.226979995612055, + "mean": 37.980256889002504, + "cv": 0.062256326185626355 + }, + "total_ms": { + "count": 5, + "median": 2459.354874008568, + "p95": 2467.9039015667513, + "min": 2306.7690430325456, + "max": 2469.92321095604, + "iqr": 7.146617965190671, + "mean": 2429.7107676102314, + "cv": 0.028399214493257538 + }, + "tokens_per_second": { + "count": 5, + "median": 26.023084621246504, + "p95": 27.414327681477026, + "min": 25.911736735826434, + "max": 27.744433363759615, + "iqr": 0.07581150844606555, + "mean": 26.358250623415966, + "cv": 0.029502126049320267 + }, + "median_step_ms_by_index": [ + 36.79004199511837, + 37.9747090046294, + 36.64741599641275, + 36.55225000693463, + 36.57345900137443, + 36.55187500407919, + 36.924707994330674, + 37.18637500423938, + 36.37333300139289, + 36.53154199128039, + 36.60612499515992, + 36.61433300294448, + 36.45933300140314, + 36.8089170078747, + 36.667082997155376, + 36.822041001869366, + 36.61675000330433, + 37.11804200429469, + 36.8845000048168, + 37.55754200392403, + 36.95449999941047, + 37.091958001838066, + 37.54408299573697, + 37.82579100516159, + 37.16358300880529, + 37.46779200446326, + 37.54612499324139, + 38.00779199809767, + 38.127499996335246, + 37.61566599132493, + 37.63662501296494, + 37.750666990177706, + 37.379124987637624, + 38.19550000480376, + 38.63120899768546, + 38.22550000040792, + 38.189042010344565, + 38.97520899772644, + 38.443041004939005, + 38.615208002738655, + 39.13158299110364, + 38.86341699399054, + 38.97500000311993, + 38.916499994229525, + 38.63450000062585, + 39.15570900426246, + 39.26358401076868, + 38.84166698844638, + 38.691791996825486, + 39.63645899784751, + 38.963415994658135, + 39.351500003249384, + 39.13829200610053, + 39.42037500382867, + 39.41208300238941, + 39.51083400170319, + 39.26295798737556, + 39.71608399297111, + 40.14054100844078, + 39.90775000420399, + 39.64020899729803, + 39.71708299650345, + 41.06150000006892, + 39.986292002140544 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.03706975000386592, + 0.036847541996394284, + 0.03613133299222682, + 0.03565316699678078, + 0.034726291996776126, + 0.03465433399833273, + 0.03480579199094791, + 0.03407512498961296, + 0.03373125000507571, + 0.03403562499443069, + 0.03382850000343751, + 0.03401891699468251, + 0.03393745800713077, + 0.0339302909997059, + 0.034107958010281436, + 0.035030999992159195, + 0.03485483300755732, + 0.03462400000717025, + 0.03451804199721664, + 0.03443112500826828, + 0.03432208301092032, + 0.03442754098796286, + 0.035269916988909245, + 0.03493991600407753, + 0.034522999994806014, + 0.034842417007894255, + 0.035748249996686354, + 0.03516104200389236, + 0.03484754200326279, + 0.03587537500425242, + 0.035274207999464124, + 0.03507458299282007, + 0.035663417002069764, + 0.03546762499900069, + 0.03547820799576584, + 0.036228583005140536, + 0.0360543749993667, + 0.03613245900487527, + 0.03596016700612381, + 0.0359612920001382, + 0.035906958000850864, + 0.036165625002468005, + 0.050483333005104214, + 0.04093291600293014, + 0.03709433300537057, + 0.036587249996955507, + 0.03627879198757, + 0.03644695800903719, + 0.03596366700367071, + 0.03655733300547581, + 0.03767341701313853, + 0.03675870900042355, + 0.03646733399364166, + 0.037473082993528806, + 0.03726745799940545, + 0.03693291699164547, + 0.037655042004189454, + 0.037109041993971914, + 0.03709974999947008, + 0.037977833999320865, + 0.03706383300595917, + 0.037074209001730196, + 0.037908041005721316, + 0.03762687501148321 + ], + [ + 0.03678533399943262, + 0.0379747090046294, + 0.03889887499099132, + 0.03716774999338668, + 0.03657345900137443, + 0.036409707987331785, + 0.03716004200396128, + 0.03698854098911397, + 0.03660929099714849, + 0.03653154199128039, + 0.03644641699793283, + 0.03661433300294448, + 0.0385167909989832, + 0.036756208006409, + 0.03667658299673349, + 0.03757237498939503, + 0.03638604099978693, + 0.03723579100915231, + 0.037292000008164905, + 0.03702950000297278, + 0.03693141700932756, + 0.037692332989536226, + 0.037305417004972696, + 0.0403703750052955, + 0.03849004198855255, + 0.03746779200446326, + 0.037658165994798765, + 0.03800779199809767, + 0.038127499996335246, + 0.03774945800250862, + 0.037706917006289586, + 0.03772237501107156, + 0.037379124987637624, + 0.03875399999378715, + 0.03933716600295156, + 0.038622458989266306, + 0.038329417002387345, + 0.03897520899772644, + 0.03876908299571369, + 0.03883025000686757, + 0.038387874999898486, + 0.03919487500388641, + 0.03885679200175218, + 0.03884254199510906, + 0.040010958997299895, + 0.03942037500382867, + 0.03912908400525339, + 0.053213459003018215, + 0.03870437499426771, + 0.03969258400320541, + 0.039567750005517155, + 0.03940191600122489, + 0.03967154100246262, + 0.04012966700247489, + 0.039702666996163316, + 0.039604416000656784, + 0.03926295798737556, + 0.03971608399297111, + 0.040369833004660904, + 0.03998049999063369, + 0.03964020899729803, + 0.04052337500615977, + 0.04106150000006892, + 0.039986292002140544 + ], + [ + 0.03650704200845212, + 0.03810199999134056, + 0.03664741599641275, + 0.03655225000693463, + 0.036913125004502945, + 0.03655187500407919, + 0.036924707994330674, + 0.037603749995469116, + 0.036373333001392893, + 0.03652574999432545, + 0.03660612499515992, + 0.037137207997147925, + 0.036434624998946674, + 0.0368089170078747, + 0.03659783300827257, + 0.036822041001869366, + 0.03716387499298435, + 0.03680649999296293, + 0.036740041003213264, + 0.03755754200392403, + 0.03723170899320394, + 0.03692195800249465, + 0.03754408299573697, + 0.03775795800902415, + 0.03716358300880529, + 0.03800391699769534, + 0.03754612499324139, + 0.03745291699306108, + 0.03724112499912735, + 0.03746966600010637, + 0.03763662501296494, + 0.03821849999076221, + 0.039123249996919185, + 0.03819550000480376, + 0.03862579198903404, + 0.03822550000040792, + 0.038189042010344565, + 0.03852895798627287, + 0.038443041004939005, + 0.038615208002738655, + 0.03913158299110364, + 0.03886341699399054, + 0.03897500000311993, + 0.03921841700503137, + 0.03873679200478364, + 0.03915570900426246, + 0.03926358401076868, + 0.038885250003659166, + 0.03888312500203028, + 0.03963645899784751, + 0.03897979200701229, + 0.03962787499767728, + 0.03924041699792724, + 0.039498916012234986, + 0.03941208300238941, + 0.039970375000848435, + 0.03947362500184681, + 0.04109749999770429, + 0.04014054100844078, + 0.03990775000420399, + 0.03995679199579172, + 0.03971708299650345, + 0.055563916990649886, + 0.04050845898746047 + ], + [ + 0.03679004199511837, + 0.0376041669951519, + 0.0365028749947669, + 0.03654395800549537, + 0.03634016599971801, + 0.03660891699837521, + 0.03651279199402779, + 0.03718637500423938, + 0.03630416700616479, + 0.03673612499551382, + 0.03732208399742376, + 0.036548417003359646, + 0.03645933300140314, + 0.037280959004419856, + 0.036667082997155376, + 0.03648362500825897, + 0.03661675000330433, + 0.037118042004294693, + 0.0368845000048168, + 0.037696040992159396, + 0.03695449999941047, + 0.037091958001838066, + 0.037718499996117316, + 0.03782579100516159, + 0.036926250002579764, + 0.03736291700624861, + 0.03748466599790845, + 0.03856074999202974, + 0.038180083996849135, + 0.03761566599132493, + 0.03760695899836719, + 0.037750666990177706, + 0.03711687499890104, + 0.03803166700527072, + 0.03863120899768546, + 0.03798662500048522, + 0.03809100000944454, + 0.039079124995623715, + 0.03814891699585132, + 0.03844154201215133, + 0.0391351250000298, + 0.03863433400692884, + 0.03952820799895562, + 0.038916499994229525, + 0.03862195801048074, + 0.03928595900651999, + 0.039349667000351474, + 0.03884166698844638, + 0.038691791996825486, + 0.03985529099009, + 0.038963415994658135, + 0.039351500003249384, + 0.03913829200610053, + 0.03942037500382867, + 0.03993133299809415, + 0.03951083400170319, + 0.040707375010242686, + 0.040848667005775496, + 0.04111587500665337, + 0.04266529199958313, + 0.04161033300624695, + 0.0423328750039218, + 0.041368375008460134, + 0.042042917004437186 + ], + [ + 0.03762075000850018, + 0.03844133298844099, + 0.03759937500581145, + 0.0378358330053743, + 0.03727658301068004, + 0.03882879199227318, + 0.037783834006404504, + 0.038321708008879796, + 0.03740791601012461, + 0.03816166700562462, + 0.03769974999886472, + 0.03755454201018438, + 0.0375905840046471, + 0.03843749999941792, + 0.03812912499415688, + 0.03775270799815189, + 0.03805649999412708, + 0.0381268750061281, + 0.03825420899374876, + 0.03817095799604431, + 0.038620207997155376, + 0.038363582993042655, + 0.03801324999949429, + 0.03917008299322333, + 0.038432207991718315, + 0.03865816700272262, + 0.03919491700071376, + 0.03911741699266713, + 0.03836399999272544, + 0.03899512501084246, + 0.03874470799928531, + 0.03890799998771399, + 0.038851499994052574, + 0.03923508299340028, + 0.039223833999130875, + 0.05236304098798428, + 0.04211141700216103, + 0.03947054099990055, + 0.04022333299508318, + 0.03980954100552481, + 0.04021525000280235, + 0.0399003329948755, + 0.038870374992256984, + 0.03890458299429156, + 0.03863450000062585, + 0.03907737499685027, + 0.03936262498609722, + 0.03654875000938773, + 0.03667308299918659, + 0.0365431670070393, + 0.03730408300179988, + 0.03718545800074935, + 0.03712729200196918, + 0.036864166002487764, + 0.03724708400841337, + 0.037048041995149106, + 0.03665950000868179, + 0.03736816700256895, + 0.03745129200979136, + 0.03692008300276939, + 0.03753625000535976, + 0.03739425000094343, + 0.036875375008094124, + 0.03719508300127927 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "speedup_cached_vs_uncached_sdpa": 15.461185541590494, + "speedup_cached_vs_uncached_mha": 37.3856125543173 + }, + { + "batch": 8, + "prompt_len": 1, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_mha": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 0.592665994190611, + "p95": 0.6208083941601217, + "min": 0.5545840103877708, + "max": 0.6219999922905117, + "iqr": 0.05112499638926238, + "mean": 0.5900418007513508, + "cv": 0.050792513133797595 + }, + "decode_ms_per_step": { + "count": 315, + "median": 0.6356669910019264, + "p95": 0.7265375024871901, + "min": 0.4758750001201406, + "max": 0.8314580045407638, + "iqr": 0.11987549805780873, + "mean": 0.6157765491581004, + "cv": 0.13498238504793358 + }, + "total_ms": { + "count": 5, + "median": 39.01274902455043, + "p95": 40.21967178559862, + "min": 38.65816594043281, + "max": 40.2419549645856, + "iqr": 1.2541260803118348, + "mean": 39.38396439771168, + "cv": 0.018897634653242252 + }, + "tokens_per_second": { + "count": 5, + "median": 13123.91494579893, + "p95": 13229.4206794477, + "min": 12723.039933089205, + "max": 13244.291019623777, + "iqr": 411.5759408836693, + "mean": 13003.909719023004, + "cv": 0.018794570037523016 + }, + "median_step_ms_by_index": [ + 0.592665994190611, + 0.5783329979749396, + 0.4937499907100573, + 0.49454200780019164, + 0.49783299618866295, + 0.48741699720267206, + 0.49320799007546157, + 0.48070801130961627, + 0.4847920063184574, + 0.4834589926758781, + 0.48474999493919313, + 0.4837080050492659, + 0.4870420088991523, + 0.4967499990016222, + 0.4930830036755651, + 0.5148339987499639, + 0.4985409905202687, + 0.49691701133269817, + 0.5013749905629084, + 0.4979579971404746, + 0.5039999959990382, + 0.5021660035708919, + 0.6012919911881909, + 0.5744160007452592, + 0.593415999901481, + 0.5962499999441206, + 0.592500000493601, + 0.6105420034145936, + 0.6265840056585148, + 0.6152920104796067, + 0.6193750014062971, + 0.6615840102313086, + 0.6546669901581481, + 0.6537500012200326, + 0.6580000044777989, + 0.6664159882348031, + 0.6727080035489053, + 0.6647080008406192, + 0.6694170006085187, + 0.6661249935859814, + 0.6683750107185915, + 0.6812499923398718, + 0.679959004628472, + 0.6713750044582412, + 0.6756250077160075, + 0.6748750020051375, + 0.673250004183501, + 0.6731250032316893, + 0.6749169988324866, + 0.6762499979231507, + 0.6825409946031868, + 0.6777079979656264, + 0.6778340029995888, + 0.6790830084355548, + 0.6784580036764964, + 0.6927080103196204, + 0.6953749980311841, + 0.6857080006739125, + 0.7037910108920187, + 0.7196250080596656, + 0.7191660115495324, + 0.7255829987116158, + 0.7275840034708381, + 0.7243750005727634 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0005545840103877708, + 0.00053570800810121, + 0.0004932910087518394, + 0.0004914589953841642, + 0.000497832996188663, + 0.00048741699720267206, + 0.0004833330021938309, + 0.00048579199938103557, + 0.0004847920063184574, + 0.0004834589926758781, + 0.00048474999493919313, + 0.0004853749996982515, + 0.0004845410003326833, + 0.0004939999926136807, + 0.0004930830036755651, + 0.0004911669966531917, + 0.000491209008032456, + 0.0004911669966531917, + 0.0004947499983245507, + 0.0004979579971404746, + 0.0004993339971406385, + 0.0005021660035708919, + 0.0006142079946585, + 0.0005744160007452592, + 0.0005823749961564317, + 0.000581124986638315, + 0.0005825829866807908, + 0.0005849580047652125, + 0.0006000830035191029, + 0.0006186250102473423, + 0.0006919580046087503, + 0.0006615840102313086, + 0.0006575829902430996, + 0.0006568749959114939, + 0.0006575840088771656, + 0.0006611670105485246, + 0.0006727080035489053, + 0.0006626669928664342, + 0.0006694170006085187, + 0.0006661249935859814, + 0.0006683750107185915, + 0.0006812499923398718, + 0.0006761660042684525, + 0.0006739579985151067, + 0.0006997079908614978, + 0.0006766250007785857, + 0.0006776660011382774, + 0.0006787909951526672, + 0.0006749169988324866, + 0.0006714999908581376, + 0.0006825409946031868, + 0.0006886670016683638, + 0.0006762499979231507, + 0.0006814999942434952, + 0.0006784580036764964, + 0.0006982920021982864, + 0.0006923329929122701, + 0.0006857080006739125, + 0.0006941250030649826, + 0.0007203330023912713, + 0.0007191660115495324, + 0.0007255829987116158, + 0.0007275840034708381, + 0.000723707998986356 + ], + [ + 0.0006160420016385615, + 0.0005904590070713311, + 0.0004937499907100573, + 0.0004791670071426779, + 0.0004844579962082207, + 0.0004784579941770062, + 0.0004932079900754616, + 0.00048033300845418125, + 0.00048116600373759866, + 0.00048054200306069106, + 0.0004863340000156313, + 0.0004809579986613244, + 0.0004904999950667843, + 0.0004970419977325946, + 0.0004906670073978603, + 0.0005184169858694077, + 0.0004985409905202687, + 0.0004969170113326982, + 0.000494624997372739, + 0.000497667002491653, + 0.0004944589891238138, + 0.0004965420084772632, + 0.0005670000100508332, + 0.0005637909926008433, + 0.0005759579944424331, + 0.0005962499999441206, + 0.000592500000493601, + 0.0006040000007487833, + 0.0006265840056585148, + 0.000600125000346452, + 0.0006110840040491894, + 0.0006158329924801365, + 0.0006546669901581481, + 0.0006537500012200326, + 0.0006580000044777989, + 0.0006664159882348031, + 0.0006605410017073154, + 0.0006647080008406192, + 0.0006653339951299131, + 0.0006601659988518804, + 0.0006644999957643449, + 0.0006796250090701506, + 0.000679959004628472, + 0.0006713750044582412, + 0.0006756249931640923, + 0.0006703330000163987, + 0.0006679159996565431, + 0.0006674999895039946, + 0.000669250002829358, + 0.0006762499979231507, + 0.0006780419935239479, + 0.0006705419946229085, + 0.0006778340029995888, + 0.0006780409894417971, + 0.000677040996379219, + 0.0006810410122852772, + 0.0006953749980311841, + 0.0006789159961044788, + 0.000715374990249984, + 0.0006916660058777779, + 0.0006785839941585436, + 0.0007059170020511374, + 0.0007261249993462116, + 0.0007243750005727634 + ], + [ + 0.0005926659941906109, + 0.0005783329979749396, + 0.0005605830083368346, + 0.0005652499967254698, + 0.0005512920033652335, + 0.0005696250009350479, + 0.0005622500029858202, + 0.0005664159980369732, + 0.0005578329873969778, + 0.000558916013687849, + 0.0005636249989038333, + 0.0006030420045135543, + 0.0005652919935528189, + 0.0005631670064758509, + 0.0005673329869750887, + 0.0005732499994337559, + 0.0005707499949494377, + 0.0005665419885190204, + 0.0005780409992439672, + 0.000567374998354353, + 0.000574207995668985, + 0.0005691249971278012, + 0.000651374997687526, + 0.0006500000017695129, + 0.000654957999358885, + 0.0006641249929089099, + 0.0006682499952148646, + 0.0006634169985773042, + 0.0006747089937562123, + 0.0006800419942010194, + 0.0006849579949630424, + 0.0006794579967390746, + 0.0006250000005820766, + 0.0006165419908938929, + 0.0006190830026753247, + 0.0006275000050663948, + 0.0006273749895626679, + 0.0006229590071598068, + 0.0006330830074148253, + 0.0006227499980013818, + 0.0006277500069700181, + 0.0006395000091288239, + 0.000632875002338551, + 0.000629415997536853, + 0.0006346660084091127, + 0.0006314579950412735, + 0.0006312500045169145, + 0.0006301249959506094, + 0.0006395420059561729, + 0.0006356669910019264, + 0.0006350000039674342, + 0.0006422909937100485, + 0.0006494999979622662, + 0.0006452090019593015, + 0.0006504999910248443, + 0.0006769589963369071, + 0.0006577080057468265, + 0.0006567499949596822, + 0.0006860419962322339, + 0.0008314580045407638, + 0.0008096660021692514, + 0.000764250013162382, + 0.000709958010702394, + 0.0007039170013740659 + ], + [ + 0.0006219999922905117, + 0.0006064579938538373, + 0.0005589170032180846, + 0.0005562919977819547, + 0.0006567919917870313, + 0.000610458999290131, + 0.0005085420125396922, + 0.00048070801130961627, + 0.00048750000132713467, + 0.0004861659981543198, + 0.00048179199802689254, + 0.00048270799743477255, + 0.00048474999493919313, + 0.0004967499990016222, + 0.0005535829986911267, + 0.0005148339987499639, + 0.0005015409988118336, + 0.0005083330033812672, + 0.000502208000398241, + 0.0005065000004833564, + 0.0005920419935137033, + 0.0005596249975496903, + 0.0006012919911881909, + 0.0005817500059492886, + 0.0005958330002613366, + 0.0005963330040685833, + 0.0006017500127200037, + 0.0006105420034145936, + 0.000612458010436967, + 0.0006152920104796067, + 0.0006145420047687367, + 0.0007275000098161399, + 0.0006779159884899855, + 0.000673250004183501, + 0.0006741249962942675, + 0.0006772909982828423, + 0.0006807500030845404, + 0.0006793329957872629, + 0.0006890830118209124, + 0.0006862090085633099, + 0.0006852079968666658, + 0.0007005420047789812, + 0.0006967080116737634, + 0.0006668749992968515, + 0.0006756250077160075, + 0.0006748750020051375, + 0.000673250004183501, + 0.0006731250032316893, + 0.000675208997563459, + 0.0006978750025155023, + 0.0006900000007590279, + 0.0006777079979656264, + 0.0006787089951103553, + 0.0006790830084355548, + 0.0006838330009486526, + 0.0007512919983128086, + 0.0007347499922616407, + 0.000697666997439228, + 0.0007037910108920187, + 0.0007196250080596656, + 0.0007280409918166697, + 0.0007296660041902214, + 0.0007427499949699268, + 0.0007405829965136945 + ], + [ + 0.0005649170052492991, + 0.0005010830063838512, + 0.0004902920045424253, + 0.0004945420078001916, + 0.0004917919868603349, + 0.00048516700917389244, + 0.0004800829919986427, + 0.0004758750001201406, + 0.0004794999986188486, + 0.0004784170014318079, + 0.0004776659916387871, + 0.0004837080050492659, + 0.0004870420088991523, + 0.00048154201067518443, + 0.00048591700033284724, + 0.0004957909986842424, + 0.0004973329923814163, + 0.0004886669921688735, + 0.0005013749905629084, + 0.000494459003675729, + 0.0005039999959990382, + 0.00050149999151472, + 0.0005682499904651195, + 0.0005675410066032782, + 0.000593415999901481, + 0.000588875001994893, + 0.0005897499941056594, + 0.0006554169958690181, + 0.0006301249959506094, + 0.0006071249954402447, + 0.0006193750014062971, + 0.0006168749969219789, + 0.000625291999313049, + 0.000643749997834675, + 0.0006675420008832589, + 0.0006688329885946587, + 0.0006758340023225173, + 0.0006723330006934702, + 0.0006731659959768876, + 0.0006769159954274073, + 0.0006717910000588745, + 0.0006816670065745711, + 0.0006932079995749518, + 0.0006771249900339171, + 0.0006779999966965988, + 0.0006857499975012615, + 0.0006747080042259768, + 0.0006808340112911537, + 0.000687332998495549, + 0.0006854170060250908, + 0.0006889170035719872, + 0.0006860840076114982, + 0.0006873750098748133, + 0.0006930420058779418, + 0.0006937920115888119, + 0.0006927080103196204, + 0.0007876659947214648, + 0.0007597910007461905, + 0.0007330410007853061, + 0.0007137080101529136, + 0.0006972499977564439, + 0.0006908340001245961, + 0.0007426250085700303, + 0.0007489999989047647 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + ] + } + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 0.5464169953484088, + "p95": 0.605925201671198, + "min": 0.5336250032996759, + "max": 0.6066670030122623, + "iqr": 0.05708300159312785, + "mean": 0.5671083985362202, + "cv": 0.06140221344143905 + }, + "decode_ms_per_step": { + "count": 315, + "median": 2.386042004218325, + "p95": 3.6474036998697557, + "min": 0.5422499962151051, + "max": 3.995584003860131, + "iqr": 1.7523545029689558, + "mean": 2.284490301584204, + "cv": 0.4129788905695759 + }, + "total_ms": { + "count": 5, + "median": 144.64195699838456, + "p95": 144.69287501415238, + "min": 144.07070001470856, + "max": 144.70079202146735, + "iqr": 0.2858760126400739, + "mean": 144.48999739834107, + "cv": 0.0018499561508287272 + }, + "tokens_per_second": { + "count": 5, + "median": 3539.7751152227447, + "p95": 3552.311029588204, + "min": 3538.335850463357, + "max": 3553.8107328397, + "iqr": 7.008137269027429, + "mean": 3543.5075988842423, + "cv": 0.0018525311302050494 + }, + "median_step_ms_by_index": [ + 0.5464169953484088, + 0.5701250047422945, + 0.6160409975564107, + 0.6298340013017878, + 0.7574169867439196, + 0.8820000075502321, + 0.9205410024151206, + 0.9471249941270798, + 1.0754170070867985, + 1.0861659975489601, + 1.09570799395442, + 1.102750000427477, + 1.2749159941449761, + 1.2522499891929328, + 1.3264999870443717, + 1.3053329894319177, + 1.4015000051585957, + 1.4142080035526305, + 1.4310419937828556, + 1.4399589999811724, + 1.531250003608875, + 1.9166660058544949, + 1.9435839931247756, + 1.949083001818508, + 2.0768340036738664, + 2.1347500005504116, + 2.087999993818812, + 2.1431660134112462, + 2.3267919896170497, + 2.3502910044044256, + 2.2991250007180497, + 2.2844999912194908, + 2.4050419888226315, + 2.3722080077277496, + 2.4659159971633926, + 2.4178750027203932, + 2.5299169938080013, + 2.5858750013867393, + 2.6193330122623593, + 2.5956250028684735, + 2.702375000808388, + 2.823916991474107, + 2.8716249944409356, + 2.8731250058626756, + 2.882957996916957, + 3.0272500007413328, + 3.103124996414408, + 2.899749990319833, + 3.1388329953188077, + 3.12100000155624, + 3.1725409935461357, + 3.1621660018572584, + 3.258041004301049, + 3.295457994681783, + 3.3029999904101714, + 3.4280840045539662, + 3.5162919957656413, + 3.516791999572888, + 3.484167013084516, + 3.4842500026570633, + 3.6526250041788444, + 3.6664590006694198, + 3.661125010694377, + 3.701833004015498 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0006029579963069409, + 0.0005701250047422945, + 0.0006160409975564107, + 0.0006318340019788593, + 0.0007574169867439196, + 0.0008820000075502321, + 0.0009205410024151206, + 0.0009519160084892064, + 0.000993249996099621, + 0.0010230829939246178, + 0.00103645799390506, + 0.001050584003678523, + 0.0011655420094029978, + 0.0011988329933956265, + 0.0012417079997248948, + 0.0012159170000813901, + 0.0012953749974258244, + 0.0013100409996695817, + 0.0013464579969877377, + 0.0014056250074645504, + 0.0014751249982509762, + 0.0018465409957570955, + 0.0018641250062501058, + 0.0018820830009644851, + 0.00201162499433849, + 0.0021347500005504116, + 0.0020702089968835935, + 0.0020526250009424984, + 0.002487707999534905, + 0.0024657079920871183, + 0.0022991250007180497, + 0.0023719169985270128, + 0.0024080000002868474, + 0.002386042004218325, + 0.002510958001948893, + 0.0024181249900721014, + 0.0025941670028259978, + 0.0026251670060446486, + 0.0026182080036960542, + 0.002707499996176921, + 0.002702375000808388, + 0.002959040997666307, + 0.002898500009905547, + 0.0028715840016957372, + 0.0028797499981010333, + 0.002984457998536527, + 0.0030374999914783984, + 0.0029936670034658164, + 0.0032462500093970448, + 0.003022332995897159, + 0.0031227089930325747, + 0.0031621660018572584, + 0.0032903749961405993, + 0.003296625000075437, + 0.003995584003860131, + 0.003428249998250976, + 0.003524999992805533, + 0.003612374988733791, + 0.0035351669939700514, + 0.0035619999980553985, + 0.0037802080041728914, + 0.0036640830076066777, + 0.003687042000819929, + 0.0036728749983012676 + ], + [ + 0.0005464169953484088, + 0.000576916994759813, + 0.0006228340062079951, + 0.0006324580026557669, + 0.0007677910034544766, + 0.0009024579921970144, + 0.0009222499938914552, + 0.0009455410036025569, + 0.0010821660107467324, + 0.0011237500002607703, + 0.0011428340076236054, + 0.001156250000349246, + 0.0012753330083796754, + 0.0013203749986132607, + 0.0013541660009650514, + 0.0013053329894319177, + 0.0014015000051585957, + 0.0014852080057607964, + 0.0014523749996442348, + 0.0014399589999811724, + 0.001531250003608875, + 0.0020278750016586855, + 0.0020383750088512897, + 0.0021100000012665987, + 0.0021730000007664785, + 0.0021918749989708886, + 0.002220667010988109, + 0.002285624999785796, + 0.0023267919896170497, + 0.0023723330086795613, + 0.0024954170075943694, + 0.002340749997529201, + 0.0024247499968623742, + 0.002572875004261732, + 0.00249820901080966, + 0.002412500005448237, + 0.0025299169938080013, + 0.002567707997513935, + 0.0026230410003336146, + 0.0025747079926077276, + 0.002691332992981188, + 0.0027310409932397306, + 0.0028716249944409356, + 0.0028073749999748543, + 0.002925624998169951, + 0.003044166005565785, + 0.00310791700030677, + 0.002899749990319833, + 0.0030136659916024655, + 0.00312100000155624, + 0.0032157920068129897, + 0.0031175420008366928, + 0.0032114170026034117, + 0.003275290990131907, + 0.0033029999904101714, + 0.003305209000245668, + 0.0033456249948358163, + 0.0034230419987579808, + 0.0034782500006258488, + 0.0034770420024869964, + 0.0036035420052940026, + 0.0037159169878577814, + 0.003635000000940636, + 0.003548207998392172 + ], + [ + 0.0005336250032996759, + 0.0005422499962151051, + 0.0005834579933434725, + 0.0006029170035617426, + 0.0007251250062836334, + 0.0008599169959779829, + 0.0008919579995563254, + 0.0009297500073444098, + 0.0010412910050945356, + 0.0010861659975489601, + 0.00109570799395442, + 0.001102750000427477, + 0.0012275409972062334, + 0.0012522499891929328, + 0.001305792000493966, + 0.0014011250023031607, + 0.0014822079974692315, + 0.0014564169978257269, + 0.0014594999956898391, + 0.0014534579968312755, + 0.0015477080014534295, + 0.0019166660058544949, + 0.0019435839931247756, + 0.001949083001818508, + 0.0020492079929681495, + 0.0020821660000365227, + 0.002087999993818812, + 0.0021431660134112462, + 0.0022179579973453656, + 0.0021758330112788826, + 0.0022099170018918812, + 0.0022075000015320256, + 0.002330250004888512, + 0.0023687080101808533, + 0.0024659159971633926, + 0.0024178750027203932, + 0.002524874988012016, + 0.0025599579967092723, + 0.0026193330122623593, + 0.0026033750036731362, + 0.002798166999127716, + 0.0029032080055912957, + 0.0027868750039488077, + 0.0032180409907596186, + 0.002984042002935894, + 0.0030699589988216758, + 0.003103124996414408, + 0.002875875012250617, + 0.0030879170080879703, + 0.003297167000710033, + 0.0031725409935461357, + 0.003253000002587214, + 0.003258041004301049, + 0.003295457994681783, + 0.003354333995957859, + 0.0033945830073207617, + 0.0035162919957656413, + 0.003527958004269749, + 0.003404999995836988, + 0.003586791004636325, + 0.0036526250041788444, + 0.0036675000010291114, + 0.00370341599045787, + 0.0037044999917270616 + ], + [ + 0.0006066670030122623, + 0.0005830000009154901, + 0.000617457990301773, + 0.0006298340013017878, + 0.0007708749908488244, + 0.000910000002477318, + 0.0011116249952465296, + 0.001251875000889413, + 0.0010754170070867985, + 0.0010459170007379726, + 0.0010492909932509065, + 0.0010522499942453578, + 0.0012749159941449761, + 0.0012192919966764748, + 0.0013264999870443717, + 0.0012169999972684309, + 0.0013016669981880113, + 0.0013203329872339964, + 0.0013417090085567906, + 0.0013627500011352822, + 0.001464582994231023, + 0.001895624998724088, + 0.0019299999985378236, + 0.0018932920065708458, + 0.0020768340036738664, + 0.0020638330024667084, + 0.002075708005577326, + 0.0021126249921508133, + 0.00221633400360588, + 0.0021859160042367876, + 0.00219499999366235, + 0.0022034580033505335, + 0.0023838749912101775, + 0.0023617500119144097, + 0.0024327080027433112, + 0.0025897500017890707, + 0.0026834999880520627, + 0.0025858750013867393, + 0.0026059160009026527, + 0.0025956250028684735, + 0.0027035830134991556, + 0.002823916991474107, + 0.0028910420078318566, + 0.0028731250058626756, + 0.002872208002372645, + 0.0029837500042049214, + 0.0031348339980468154, + 0.003001165998284705, + 0.0031388329953188077, + 0.003006749990163371, + 0.0038808749959571287, + 0.003427457995712757, + 0.0034720830008154735, + 0.0034770409984048456, + 0.00323295799898915, + 0.0034280840045539662, + 0.003479958002571948, + 0.003516791999572888, + 0.003498250007396564, + 0.0034695829963311553, + 0.0036989170039305463, + 0.0036664590006694198, + 0.003661125010694377, + 0.003701833004015498 + ], + [ + 0.0005458749947138131, + 0.0005578340060310438, + 0.0005986670003039762, + 0.0006292080070124939, + 0.0007372079999186099, + 0.0008690419927006587, + 0.0009180420020129532, + 0.0009471249941270798, + 0.0010946249967673793, + 0.0011185830080648884, + 0.0011468750017229468, + 0.0011565840104594827, + 0.0013138330104993656, + 0.0013254999939817935, + 0.0013442909985315055, + 0.0013165420095901936, + 0.0014035839994903654, + 0.0014142080035526305, + 0.0014310419937828556, + 0.0014418339997064322, + 0.0015391670021926984, + 0.00203291600337252, + 0.002059249993180856, + 0.0020932499901391566, + 0.002238291999674402, + 0.0022036250011296943, + 0.0022265000006882474, + 0.002221292001195252, + 0.0023384579981211573, + 0.0023502910044044256, + 0.0023544580035377294, + 0.0022844999912194908, + 0.0024050419888226315, + 0.0023722080077277496, + 0.0024219159968197346, + 0.0024069170030998066, + 0.002525292002246715, + 0.002616708996356465, + 0.002622792002512142, + 0.0025693749921629205, + 0.0026893339963862672, + 0.00271837500622496, + 0.002826666008331813, + 0.002959707999252714, + 0.002882957996916957, + 0.0030272500007413328, + 0.003026332997251302, + 0.0028977079928154126, + 0.0031864589982433245, + 0.0031719580001663417, + 0.0031414590048370883, + 0.0031022080074762926, + 0.00316325000312645, + 0.003269749999162741, + 0.00330174999544397, + 0.0036451659980230033, + 0.0035822909994749352, + 0.003429875010624528, + 0.003484167013084516, + 0.0034842500026570633, + 0.0035968749871244654, + 0.003615207999246195, + 0.0035336250002728775, + 0.003771417003008537 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_mha": { + "first_token_ms": { + "count": 5, + "median": 0.6203339871717617, + "p95": 0.6475834001321346, + "min": 0.6039169966243207, + "max": 0.6540000031236559, + "iqr": 0.012916993000544608, + "mean": 0.6218335940502584, + "cv": 0.03137576700505535 + }, + "decode_ms_per_step": { + "count": 315, + "median": 2.9164169973228127, + "p95": 4.2174087968305685, + "min": 0.5330839921953157, + "max": 4.462833006982692, + "iqr": 1.749229006236419, + "mean": 2.7204523652067616, + "cv": 0.3841227073995998 + }, + "total_ms": { + "count": 5, + "median": 169.6057450171793, + "p95": 176.85749061929528, + "min": 168.12325302453246, + "max": 176.92383204121143, + "iqr": 7.785416935803369, + "mean": 172.01033260207623, + "cv": 0.025389204752510292 + }, + "tokens_per_second": { + "count": 5, + "median": 3018.7656670954143, + "p95": 3042.918808178328, + "min": 2893.9006921393056, + "max": 3045.384804238169, + "iqr": 133.7182867166739, + "mean": 2978.088504926828, + "cv": 0.025180681857539848 + }, + "median_step_ms_by_index": [ + 0.6203339871717617, + 0.5779589992016554, + 0.5810000002384186, + 0.7751660014037043, + 0.9005419997265562, + 1.1167079937877133, + 1.1971670028287917, + 1.1130829952890053, + 1.2796249939128757, + 1.4315830048872158, + 1.51087500853464, + 1.4658750005764887, + 1.636999993934296, + 1.6998749924823642, + 1.7765829979907721, + 1.6282919968944043, + 1.8603749922476709, + 1.758458005497232, + 1.830416003940627, + 1.860749995103106, + 1.9582079985411838, + 2.413499998510815, + 2.4689999991096556, + 2.629833994433284, + 2.4574589915573597, + 2.4972089886432514, + 2.5278749963035807, + 2.527791992179118, + 2.6347910024924204, + 2.8152500017313287, + 2.783458010526374, + 2.715292008360848, + 2.8953330038348213, + 2.9119160026311874, + 2.9834159940946847, + 2.9767500091111287, + 3.0764999974053353, + 3.3218750031664968, + 3.1206659914460033, + 3.0588340014219284, + 3.2279169972753152, + 3.257125004893169, + 3.264291997766122, + 3.2131250045495108, + 3.3371660101693124, + 3.3194999996339902, + 3.3812500041676685, + 3.4041670005535707, + 3.5114160127704963, + 3.541875004884787, + 3.5692079982254654, + 3.6217500019120052, + 3.804374995524995, + 3.7537080061156303, + 3.788791989791207, + 3.8619160040980205, + 4.049082999699749, + 4.019082989543676, + 4.104499996174127, + 4.109750007046387, + 4.230125006870367, + 4.211958992527798, + 4.272457998013124, + 4.276917010429315 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0006219169881660491, + 0.0005330839921953157, + 0.0005810000002384186, + 0.0007650420011486858, + 0.0009005419997265562, + 0.0011167079937877133, + 0.0011971670028287917, + 0.001108666998334229, + 0.0012674169993260875, + 0.0014315830048872158, + 0.0016600829985691234, + 0.0015307089925045148, + 0.001649333004024811, + 0.0017205420008394867, + 0.0017765829979907721, + 0.0016282919968944043, + 0.002029875002335757, + 0.001758458005497232, + 0.0017457909998483956, + 0.0017753329884726554, + 0.0019072499999310821, + 0.002413499998510815, + 0.002326957997865975, + 0.0024100410082610324, + 0.002452291999361478, + 0.0024370000028284267, + 0.0025057080056285486, + 0.002520583992009051, + 0.0026347910024924204, + 0.0026797919999808073, + 0.002751916996203363, + 0.0026958749949699268, + 0.0028953330038348213, + 0.0029119160026311874, + 0.002846375005901791, + 0.002836000014212914, + 0.0029635410028276965, + 0.0029853329906472936, + 0.00307091700960882, + 0.00301858400052879, + 0.0031222499965224415, + 0.00319070799741894, + 0.0031688750023022294, + 0.003177916005370207, + 0.0033371660101693124, + 0.0033052079961635172, + 0.0033902910072356462, + 0.0033711249998304993, + 0.003503166008158587, + 0.003511791001074016, + 0.0035568750026868656, + 0.003740000000107102, + 0.0038191670028027147, + 0.0037537080061156303, + 0.004249124991474673, + 0.00390533400059212, + 0.00403895799536258, + 0.003954875006456859, + 0.004154624999500811, + 0.004109750007046387, + 0.004292333993362263, + 0.004209457998513244, + 0.004303041001548991, + 0.004378165991511196 + ], + [ + 0.0006039169966243207, + 0.0005982080037938431, + 0.000645333988359198, + 0.000869958006660454, + 0.0009505000052740797, + 0.001142332999734208, + 0.0012111250107409433, + 0.0012003750016447157, + 0.0013496659958036616, + 0.0014440829982049763, + 0.00151087500853464, + 0.0014658750005764887, + 0.0017049580055754632, + 0.0017981670098379254, + 0.0018329169979551807, + 0.001756416997523047, + 0.0018704170070122927, + 0.0019315420067869127, + 0.00190320800174959, + 0.0020349579863250256, + 0.0021960000012768432, + 0.0025765829923329875, + 0.0025820830051088706, + 0.002629833994433284, + 0.0027420410042395815, + 0.0027037919935537502, + 0.002843166992533952, + 0.0028190000011818483, + 0.0029708749934798107, + 0.00294079098966904, + 0.0029565420118160546, + 0.003033957997104153, + 0.00311912500183098, + 0.0031151659932220355, + 0.0031975419988157228, + 0.003222875006031245, + 0.003296166003565304, + 0.0033218750031664968, + 0.00335670800996013, + 0.00335595800424926, + 0.0034626669948920608, + 0.003290665990789421, + 0.0032642919977661222, + 0.0032291249954141676, + 0.0033422499982407317, + 0.003300416996353306, + 0.003366292003192939, + 0.003455000012763776, + 0.0035610409977380186, + 0.003541875004884787, + 0.003559417003998533, + 0.0036200830072630197, + 0.0036963340098736808, + 0.003875375012285076, + 0.003777874997467734, + 0.004267708995030262, + 0.004150708002271131, + 0.003981542002293281, + 0.004114249997655861, + 0.00411224999697879, + 0.004305041991756298, + 0.004299749998608604, + 0.004272457998013124, + 0.004272500009392388 + ], + [ + 0.0006540000031236559, + 0.0005779589992016554, + 0.0005650419916491956, + 0.0007433749997289851, + 0.0008932920027291402, + 0.001047332989401184, + 0.0011370840074960142, + 0.0010944590030703694, + 0.0012796249939128757, + 0.001342167000984773, + 0.0014119170082267374, + 0.0013752090017078444, + 0.0015433329972438514, + 0.0016195839998545125, + 0.001721332999295555, + 0.0015172499988693744, + 0.0016202910046558827, + 0.001693958998657763, + 0.001830416003940627, + 0.001860749995103106, + 0.0019582079985411838, + 0.0023009579890640453, + 0.0024689999991096556, + 0.002647083005285822, + 0.0024574589915573597, + 0.0024972089886432514, + 0.0025278749963035807, + 0.002527791992179118, + 0.002623749998747371, + 0.0028152500017313287, + 0.002783458010526374, + 0.002715292008360848, + 0.002881249994970858, + 0.0029024579998804256, + 0.0028327500040177256, + 0.0028994159947615117, + 0.0029614579980261624, + 0.0030640419863630086, + 0.003103667011600919, + 0.0030115830013528466, + 0.0032279169972753152, + 0.003257125004893169, + 0.0032182499999180436, + 0.0031835420086281374, + 0.003296625000075437, + 0.003326291000121273, + 0.0033812500041676685, + 0.003353041989612393, + 0.0035616250097518787, + 0.0035204580053687096, + 0.003602458004024811, + 0.0036831670004175976, + 0.004070790993864648, + 0.0037422910099849105, + 0.0037861659948248416, + 0.0037970829871483147, + 0.004057415993884206, + 0.0041617919923737645, + 0.003997833002358675, + 0.004122250000364147, + 0.004230125006870367, + 0.004191708008875139, + 0.00425225000071805, + 0.004276917010429315 + ], + [ + 0.0006203339871717617, + 0.0005381669907364994, + 0.0005708749959012493, + 0.0007751660014037043, + 0.0008973340009106323, + 0.0010849999962374568, + 0.0011400830117054284, + 0.0011130829952890053, + 0.0012685830006375909, + 0.0013522500084945932, + 0.001456417012377642, + 0.0013912090071244165, + 0.0015431670035468414, + 0.0016458749887533486, + 0.001726292000967078, + 0.0015310000017052516, + 0.0016782500024419278, + 0.001723624998703599, + 0.0017187909979838878, + 0.0016865830111782998, + 0.0018045000033453107, + 0.00232470799528528, + 0.0022847920045023784, + 0.0023109169997042045, + 0.0024190840049413964, + 0.00246754199906718, + 0.00241312499565538, + 0.0024852920032572, + 0.002630375005537644, + 0.002599542000098154, + 0.002632916992297396, + 0.0026956670044455677, + 0.0028066249942639843, + 0.002897958009270951, + 0.0029834159940946847, + 0.0029767500091111287, + 0.0030764999974053353, + 0.003369707992533222, + 0.0031206659914460033, + 0.0030588340014219284, + 0.0032075829949462786, + 0.003152916004182771, + 0.003348458994878456, + 0.0032131250045495108, + 0.003328834005515091, + 0.0033194999996339902, + 0.003372166000190191, + 0.0034442089963704348, + 0.0035032090090680867, + 0.0036118749994784594, + 0.003601125004934147, + 0.0036217500019120052, + 0.003804374995524995, + 0.003730249998625368, + 0.003788791989791207, + 0.0038619160040980205, + 0.003958125002100132, + 0.004039541992824525, + 0.004029999996419065, + 0.004063542000949383, + 0.004141625002375804, + 0.004401250000228174, + 0.00429525000799913, + 0.004462833006982692 + ], + [ + 0.0006089999951655045, + 0.0006193340086610988, + 0.0006511249957839027, + 0.0008370410068891943, + 0.0009653750021243468, + 0.001145750007708557, + 0.0012054999970132485, + 0.0011905830033356324, + 0.0013350419903872535, + 0.0014355419989442453, + 0.0015207079995889217, + 0.0015397499955724925, + 0.001636999993934296, + 0.0016998749924823642, + 0.001817875003325753, + 0.0017568749899510294, + 0.0018603749922476709, + 0.00189200000022538, + 0.0018873749941121787, + 0.001931750011863187, + 0.002022415996179916, + 0.00254920800216496, + 0.0025790840008994564, + 0.002637792000314221, + 0.0026829589914996177, + 0.0026886249979725108, + 0.002749084000242874, + 0.002784291995340027, + 0.0029164169973228127, + 0.0029592089995276183, + 0.002991707995533943, + 0.0029684170003747568, + 0.0031168749992502853, + 0.003129167002043687, + 0.0031827909988351166, + 0.0032043749961303547, + 0.003457290993537754, + 0.003357749999850057, + 0.003753458004212007, + 0.0033830419997684658, + 0.0035735830024350435, + 0.003525709005771205, + 0.0034529170079622418, + 0.0033459999976912513, + 0.0034115000016754493, + 0.003378041001269594, + 0.0034405000042170286, + 0.0034041670005535707, + 0.0035114160127704963, + 0.0035467080015223473, + 0.0035692079982254654, + 0.003592666005715728, + 0.0037321249983506277, + 0.0037748339964309707, + 0.0038602909917244688, + 0.0037615829933201894, + 0.004049082999699749, + 0.004019082989543676, + 0.004104499996174127, + 0.004040165993501432, + 0.004153667003265582, + 0.004211958992527798, + 0.004209208986139856, + 0.004271375000826083 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "speedup_cached_vs_uncached_sdpa": 3.7075561352357522, + "speedup_cached_vs_uncached_mha": 4.34744408578969 + }, + { + "batch": 8, + "prompt_len": 16, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_mha": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 1.4035420026630163, + "p95": 1.594458002364263, + "min": 1.3405000063357875, + "max": 1.637708002817817, + "iqr": 0.07541599916294217, + "mean": 1.4298500027507544, + "cv": 0.084920462095834 + }, + "decode_ms_per_step": { + "count": 315, + "median": 0.6912910030223429, + "p95": 0.882020803692285, + "min": 0.5038329982198775, + "max": 1.7164160090032965, + "iqr": 0.10810399544425309, + "mean": 0.6953462413678478, + "cv": 0.1538305338183095 + }, + "total_ms": { + "count": 5, + "median": 43.04786901047919, + "p95": 51.867951775784604, + "min": 42.05299902241677, + "max": 53.74645095434971, + "iqr": 1.3719130656681955, + "mean": 45.236663208925165, + "cv": 0.10670880306230611 + }, + "tokens_per_second": { + "count": 5, + "median": 11893.73624685959, + "p95": 12122.480414736412, + "min": 9526.210399173598, + "max": 12175.112641242858, + "iqr": 368.44880889962405, + "mean": 11410.102699159534, + "cv": 0.09437094705963711 + }, + "median_step_ms_by_index": [ + 1.4035420026630163, + 0.6522500043502077, + 0.6068750080885366, + 0.5870000022696331, + 0.5842919927090406, + 0.5752910074079409, + 0.5776249890914187, + 0.6570409896085039, + 0.6510410021292046, + 0.651708003715612, + 0.6156670133350417, + 0.5927500023972243, + 0.5999169952701777, + 0.6232080049812794, + 0.6138750031823292, + 0.6102080078562722, + 0.6141659978311509, + 0.6135420117061585, + 0.6089999951655045, + 0.61479200667236, + 0.6252909952308983, + 0.6199589988682419, + 0.6431249930756167, + 0.6506659992737696, + 0.6390000053215772, + 0.6542920018546283, + 0.6912500102771446, + 0.6769169995095581, + 0.658958000713028, + 0.6420840072678402, + 0.6332919874694198, + 0.6649580027442425, + 0.6508329970529303, + 0.6358329992508516, + 0.644500003545545, + 0.6724580016452819, + 0.6738329975632951, + 0.6732909969286993, + 0.6764580029994249, + 0.6819999980507419, + 0.6975830037845299, + 0.6943330081412569, + 0.702916004229337, + 0.7040000054985285, + 0.7296249968931079, + 0.698042000294663, + 0.6971660041017458, + 0.7032080029603094, + 0.7038749899948016, + 0.7015419978415594, + 0.7066249963827431, + 0.7010420085862279, + 0.7064170058583841, + 0.6999589968472719, + 0.6977080047363415, + 0.7287089974852279, + 0.7426660013152286, + 0.7113749888958409, + 0.7243329891934991, + 0.7680409908061847, + 0.7719159912085161, + 0.7429999968735501, + 0.7266250031534582, + 0.7395420107059181 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0014214580005500466, + 0.0006522500043502077, + 0.0006074999982956797, + 0.0005947500030742958, + 0.0005842919927090406, + 0.0005757920007454231, + 0.0005776249890914187, + 0.0006570409896085039, + 0.0006510410021292046, + 0.000651708003715612, + 0.0006156670133350417, + 0.0005898340023122728, + 0.0005883749981876463, + 0.0006103750056354329, + 0.0006212500011315569, + 0.0006103330088080838, + 0.0006141659978311509, + 0.0006135420117061585, + 0.0006100000027799979, + 0.0006145420047687367, + 0.0006275420018937439, + 0.0006199589988682419, + 0.0006217499903868884, + 0.0006506659992737696, + 0.0006390000053215772, + 0.0006542920018546283, + 0.0007255419914145023, + 0.0006832079961895943, + 0.000658958000713028, + 0.0006420840072678402, + 0.0006332919874694198, + 0.0006501660100184381, + 0.0006508329970529303, + 0.0006358329992508516, + 0.000644500003545545, + 0.0006482500029960647, + 0.0006494169938378036, + 0.0006474580004578456, + 0.0006592089921468869, + 0.000665291998302564, + 0.0006756249931640923, + 0.0006806669989600778, + 0.000702916004229337, + 0.0007040000054985285, + 0.0007030830020084977, + 0.000698042000294663, + 0.0006971660041017458, + 0.0007032080029603094, + 0.0007038749899948016, + 0.0007015419978415594, + 0.0007066249963827431, + 0.0007010420085862279, + 0.0007064170058583841, + 0.0006999589968472719, + 0.0006977080047363415, + 0.0007199589890660718, + 0.0007116670021787286, + 0.0007002499914960936, + 0.0007136249914765358, + 0.0007104160031303763, + 0.0007190420001279563, + 0.000719584000762552, + 0.000713790999725461, + 0.0007230409973999485 + ], + [ + 0.0013405000063357875, + 0.000619708007434383, + 0.0005649590020766482, + 0.0005153749953024089, + 0.0005142919981153682, + 0.0005038329982198775, + 0.000509833000251092, + 0.000586207999731414, + 0.0005736249877372757, + 0.000574958001379855, + 0.0005848329892614856, + 0.0005927500023972243, + 0.0005977079999865964, + 0.0006232080049812794, + 0.0006138750031823292, + 0.0006102080078562722, + 0.0006181670032674447, + 0.0006071249954402447, + 0.0006080000021029264, + 0.0006117910088505596, + 0.0006223329983185977, + 0.000618042002315633, + 0.0007299579883692786, + 0.0007645000005140901, + 0.0009746670111780986, + 0.0007313340029213578, + 0.0006912500102771446, + 0.0006769169995095581, + 0.00070691600558348, + 0.0007201670086942613, + 0.0007067080005072057, + 0.0006914590048836544, + 0.0006849159981356934, + 0.0006824580050306395, + 0.0006929590017534792, + 0.0006896249979035929, + 0.0006979589961702004, + 0.0006850829959148541, + 0.0006892500096000731, + 0.0006973750132601708, + 0.0007077089976519346, + 0.0006943330081412569, + 0.0007107910059858114, + 0.0007744580070720986, + 0.0007653339998796582, + 0.0007061250071274117, + 0.000713250003173016, + 0.0007111249869922176, + 0.0007197500090114772, + 0.0007044580124784261, + 0.0007179579988587648, + 0.0007974589971126989, + 0.000741291994927451, + 0.0007149159937398508, + 0.0007194170029833913, + 0.0007287089974852279, + 0.0007677909889025614, + 0.0007402500050375238, + 0.0007737080013612285, + 0.0007680409908061847, + 0.0007719159912085161, + 0.0008315409941133112, + 0.0007694579981034622, + 0.0007793340046191588 + ], + [ + 0.0014035420026630163, + 0.0006600830092793331, + 0.0006068750080885366, + 0.0005870000022696331, + 0.0005956669920124114, + 0.0005752910074079409, + 0.0005791660078102723, + 0.000661666999803856, + 0.0006928329967195168, + 0.0006640840001637116, + 0.00065608300792519, + 0.0006615829915972427, + 0.0006647919944953173, + 0.0006590419943677261, + 0.0006036669947206974, + 0.0006071249954402447, + 0.0006083750049583614, + 0.0006150830013211817, + 0.0006084170017857105, + 0.00061479200667236, + 0.0006252909952308983, + 0.0006356249941745773, + 0.0006431249930756167, + 0.0006200409989105538, + 0.0006212500011315569, + 0.0006216250039869919, + 0.000634166004601866, + 0.0006333750061457977, + 0.0006261249945964664, + 0.0006264170078793541, + 0.000622417006525211, + 0.0006151249981485307, + 0.0006200839998200536, + 0.000623333005933091, + 0.0006257089989958331, + 0.0006383330037351698, + 0.0006379999977070838, + 0.0006304999988060445, + 0.0006272909959079698, + 0.0006819999980507419, + 0.0006975830037845299, + 0.0006997909949859604, + 0.0006994580035097897, + 0.0006918750004842877, + 0.0007296249968931079, + 0.0006960000027902424, + 0.0006936249992577359, + 0.0006961659964872524, + 0.0006969590031076223, + 0.0006923750042915344, + 0.0007003749924479052, + 0.0006966660002944991, + 0.0006949999951757491, + 0.0006967079971218482, + 0.0006941250030649826, + 0.0007078749913489446, + 0.0007426660013152286, + 0.0007113749888958409, + 0.0007066660036798567, + 0.0007890830020187423, + 0.0007925410027382895, + 0.0007425840012729168, + 0.0007159579981816933, + 0.0007277910044649616 + ], + [ + 0.0013460420013871044, + 0.00061495800036937, + 0.0005304580117808655, + 0.000574958001379855, + 0.0005436669889604673, + 0.0005224999913480133, + 0.0005055419896962121, + 0.0005755419988417998, + 0.0005735000013373792, + 0.0005782910011475906, + 0.0005846669955644757, + 0.0005863330006832257, + 0.0005999169952701777, + 0.0006043329922249541, + 0.0006025420007063076, + 0.0006025840120855719, + 0.000611542011029087, + 0.0006038749997969717, + 0.0006089999951655045, + 0.0006162920035421848, + 0.0006153750000521541, + 0.0006135419971542433, + 0.0006115419964771718, + 0.0006108329980634153, + 0.0006145000079413876, + 0.0006222080119187012, + 0.0006234159955056384, + 0.0006235420005396008, + 0.0006226669938769192, + 0.0006281250098254532, + 0.0006272080063354224, + 0.0006649580027442425, + 0.0006300840032054111, + 0.0006232079904293641, + 0.0006269589939620346, + 0.0006724580016452819, + 0.0006738329975632951, + 0.0006732909969286993, + 0.0006764580029994249, + 0.0006799999973736703, + 0.0006974999996600673, + 0.0006831249920651317, + 0.0006838749977760017, + 0.0006982500053709373, + 0.0007053330045891926, + 0.0006790409970562905, + 0.0006865000032121316, + 0.0006925830093678087, + 0.0006912910030223429, + 0.0006926659989403561, + 0.0006855829997221008, + 0.0006878340063849464, + 0.0006859580025775358, + 0.0006881669978611171, + 0.0006882079906063154, + 0.0007453340076608583, + 0.0007102500094333664, + 0.0006992499984335154, + 0.0007243329891934991, + 0.0007452090067090467, + 0.0007527919951826334, + 0.0007429999968735501, + 0.0007266250031534582, + 0.0007395420107059181 + ], + [ + 0.001637708002817817, + 0.0007373340049525723, + 0.0006834579980932176, + 0.0007057919865474105, + 0.0006531249964609742, + 0.0006243340030778199, + 0.0006967909866943955, + 0.0007900839991634712, + 0.0008250000100815669, + 0.0007116249907994643, + 0.0007502499938709661, + 0.0007269580091815442, + 0.0007576659991173074, + 0.0007371660030912608, + 0.0007329159998334944, + 0.000777291992562823, + 0.0007773749966872856, + 0.0007759160071145743, + 0.0007553750037914142, + 0.0007419159956043586, + 0.0008299579931190237, + 0.0017164160090032965, + 0.0010490830027265474, + 0.0009020419965963811, + 0.0008882080001058057, + 0.0008986250031739473, + 0.0008617500134278089, + 0.0008347499970113859, + 0.0008044999995036051, + 0.0008215000125346705, + 0.0008164579921867698, + 0.0008246670040534809, + 0.0009231659932993352, + 0.0008810830040602013, + 0.0009527090005576611, + 0.0009864169987849891, + 0.0009045000042533502, + 0.0008698340097907931, + 0.0008709159883437678, + 0.000846582988742739, + 0.000742000003810972, + 0.0007006249943515286, + 0.0007449999975506216, + 0.0007424999930663034, + 0.0010296669934177771, + 0.0008579580025980249, + 0.0007746250048512593, + 0.0007699160050833598, + 0.0007697499968344346, + 0.0007659999973839149, + 0.000775291002355516, + 0.000915541997528635, + 0.0008453750051558018, + 0.0008227079961216077, + 0.0008145000028889626, + 0.0008997909899335355, + 0.0008987909968709573, + 0.0008845830016070977, + 0.0008842090028338134, + 0.0007972909952513874, + 0.0007906249957159162, + 0.0007891249988460913, + 0.0007840829930501059, + 0.00086524999642279 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + ] + } + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 1.4442919928114861, + "p95": 1.663324996479787, + "min": 1.3345830084290355, + "max": 1.6983749956125394, + "iqr": 0.13949999993201345, + "mean": 1.4767999993637204, + "cv": 0.09648083756614094 + }, + "decode_ms_per_step": { + "count": 315, + "median": 3.1062079942785203, + "p95": 4.371000202081632, + "min": 1.3813340046908706, + "max": 18.90841701242607, + "iqr": 1.42385399522027, + "mean": 3.165171673430854, + "cv": 0.4055924178180211 + }, + "total_ms": { + "count": 5, + "median": 196.55703796888702, + "p95": 214.74877586006187, + "min": 192.29162503324915, + "max": 217.43158406752627, + "iqr": 9.90225600253325, + "mean": 200.88261542550754, + "cv": 0.05113147287552291 + }, + "tokens_per_second": { + "count": 5, + "median": 2604.841858071978, + "p95": 2657.619534300252, + "min": 2354.763693580927, + "max": 2662.622461646315, + "iqr": 128.0197159944055, + "mean": 2553.884789427362, + "cv": 0.04915659878536504 + }, + "median_step_ms_by_index": [ + 1.4442919928114861, + 1.4532920031342655, + 1.4643330068793148, + 1.4716250007040799, + 1.483874992118217, + 1.5938750002533197, + 2.054000011412427, + 2.0947500015608966, + 2.0843340025749058, + 2.1854590013390407, + 2.189874998293817, + 2.192458006902598, + 2.209415994002484, + 2.277124993270263, + 2.245499999844469, + 2.2632919863099232, + 2.2326670004986227, + 2.3962919949553907, + 2.389041008427739, + 2.4492910015396774, + 2.4828749883454293, + 2.6767499948618934, + 2.757583002676256, + 2.779583999654278, + 2.6192920049652457, + 2.777291985694319, + 2.8694580105366185, + 2.777000001515262, + 2.864250011043623, + 2.9645000031450763, + 3.0310000001918525, + 3.1346250034403056, + 2.9390419949777424, + 3.058707996387966, + 3.114541992545128, + 3.150542004732415, + 3.1502500060014427, + 3.2079999946290627, + 3.2601249986328185, + 3.334833003464155, + 3.2877080084290355, + 3.426665993174538, + 3.4608330024639145, + 3.4881250030593947, + 3.4806659969035536, + 3.637917005107738, + 3.6350830050650984, + 3.75633301155176, + 3.693958991789259, + 3.8080410013208166, + 3.985082992585376, + 4.034291006973945, + 3.893084009177983, + 4.0750829939497635, + 4.126166008063592, + 4.229166006552987, + 4.1377500019734725, + 4.2613330006133765, + 4.27408299583476, + 4.241916001774371, + 4.235749991494231, + 4.344667002442293, + 4.444040998350829, + 4.398624994792044 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0014442919928114861, + 0.0014532920031342655, + 0.001420833999873139, + 0.0014418749924516305, + 0.0014435000048251823, + 0.0016341669979738072, + 0.0018603750067995861, + 0.001880750001873821, + 0.0018782920087687671, + 0.0020774169970536605, + 0.0020608339982572943, + 0.0020246250060154125, + 0.0020652499952120706, + 0.0021575830032816157, + 0.0021941250015515834, + 0.0021840419940417632, + 0.0022051660052966326, + 0.0023962919949553907, + 0.0023552500060759485, + 0.0024439580010948703, + 0.0024828749883454293, + 0.0025691670016385615, + 0.002592541990452446, + 0.0026209579955320805, + 0.002613625008962117, + 0.0027917910047108307, + 0.0028740830020979047, + 0.0029233750101411715, + 0.002759542010608129, + 0.0029645000031450763, + 0.0030310000001918525, + 0.0029888749995734543, + 0.003021958007593639, + 0.003091292004683055, + 0.0031288330064853653, + 0.0030903750011930242, + 0.0031502500060014427, + 0.0032498340005986392, + 0.003217332996428013, + 0.003231124996091239, + 0.003536749994964339, + 0.0035062910028500482, + 0.003988833006587811, + 0.0035499579971656203, + 0.0034922920021926984, + 0.003666874996270053, + 0.003707291994942352, + 0.003923541997210123, + 0.003693958991789259, + 0.0038739160081604496, + 0.0038276660052360967, + 0.004004916991107166, + 0.003877875002217479, + 0.00402475000009872, + 0.004077707999385893, + 0.004188833001535386, + 0.0041377500019734725, + 0.00424120799289085, + 0.004286540992325172, + 0.004130417000851594, + 0.004247582997777499, + 0.004375083997729234, + 0.004478458009543829, + 0.004437875002622604 + ], + [ + 0.0013836250000167638, + 0.0014304159994935617, + 0.0014315409935079515, + 0.0014716250007040799, + 0.0015018749982118607, + 0.0015938750002533197, + 0.0020627500052796677, + 0.0020745829970110208, + 0.0020843340025749058, + 0.0021845410083187744, + 0.0021958330034976825, + 0.0023112920025596395, + 0.0022541250073118135, + 0.0022876669972902164, + 0.002245499999844469, + 0.0022632919863099232, + 0.0022767920017940924, + 0.0023234580003190786, + 0.0023642080050194636, + 0.0025847080105450004, + 0.002409624998108484, + 0.002948042005300522, + 0.002757583002676256, + 0.002779583999654278, + 0.0026830839924514294, + 0.002725625003222376, + 0.0028694580105366185, + 0.0027605829964159057, + 0.002771500003291294, + 0.0030154169944580644, + 0.0030363749974640086, + 0.0032633329974487424, + 0.007392208994133398, + 0.003061333001824096, + 0.003114541992545128, + 0.003140792003250681, + 0.003128166004898958, + 0.003341707997606136, + 0.0033742079976946115, + 0.0033073750091716647, + 0.0032009590067900717, + 0.003418791005969979, + 0.003442291999817826, + 0.0035169589973520488, + 0.0034578749909996986, + 0.003577916999347508, + 0.003515999997034669, + 0.003661166992969811, + 0.0036550410004565492, + 0.0037932080012978986, + 0.0038664580060867593, + 0.004049167007906362, + 0.003893084009177983, + 0.004028458002721891, + 0.004168750005192123, + 0.01890841701242607, + 0.008477874987875111, + 0.004369250003946945, + 0.00419745800900273, + 0.004457541988813318, + 0.004246042008162476, + 0.00441300000238698, + 0.004356667006504722, + 0.004552625003270805 + ], + [ + 0.0013345830084290355, + 0.0013813340046908706, + 0.0017581669962964952, + 0.0016350000078091398, + 0.001483874992118217, + 0.0015729169972473755, + 0.0021231669961707667, + 0.002132416993845254, + 0.0021632919961120933, + 0.0021854590013390407, + 0.002189874998293817, + 0.002192458006902598, + 0.0021975830022711307, + 0.002277124993270263, + 0.002255749990581535, + 0.0023669170041102916, + 0.0022326670004986227, + 0.0023329579998971894, + 0.002471582993166521, + 0.002428916995995678, + 0.0024082910094875842, + 0.002529874996980652, + 0.0026383329968666658, + 0.002634917007526383, + 0.002581041000667028, + 0.002777291985694319, + 0.0027378750091884285, + 0.002777000001515262, + 0.00290116599353496, + 0.0028952499997103587, + 0.002959707999252714, + 0.0030780420056544244, + 0.002861957997083664, + 0.003058707996387966, + 0.003138125000987202, + 0.003243082988774404, + 0.0032361670018872246, + 0.0031367910123663023, + 0.0032425840036012232, + 0.003334833003464155, + 0.0032865410030353814, + 0.003423042013309896, + 0.0034267500013811514, + 0.003366832999745384, + 0.0037862079916521907, + 0.003637917005107738, + 0.003777082994929515, + 0.00375633301155176, + 0.004156125010922551, + 0.003851624991511926, + 0.004021291999379173, + 0.004034291006973945, + 0.004035624995594844, + 0.0040750829939497635, + 0.004126166008063592, + 0.004164208003203385, + 0.0041141669935313985, + 0.004267416996299289, + 0.004317416009143926, + 0.004241916001774371, + 0.004232666004099883, + 0.004333709002821706, + 0.004444040998350829, + 0.004351750001660548 + ], + [ + 0.0015231249999487773, + 0.0016522920050192624, + 0.0016302500007441267, + 0.0016790410008979961, + 0.0016064160008681938, + 0.0015961249882820994, + 0.0018674579914659262, + 0.0021004580048611388, + 0.0030507089977618307, + 0.0024341249954886734, + 0.002147415987565182, + 0.002103750011883676, + 0.002209415994002484, + 0.002212209001299925, + 0.0022046669910196215, + 0.0022165830014273524, + 0.0022215420030988753, + 0.0024371670006075874, + 0.002389041008427739, + 0.0024492910015396774, + 0.0025428330118302256, + 0.0026767499948618934, + 0.003382166993105784, + 0.002821666988893412, + 0.0026192920049652457, + 0.002718957999604754, + 0.0027947909984504804, + 0.0027668749971780926, + 0.002864250011043623, + 0.0029112089978298172, + 0.0030211249977583066, + 0.0031346250034403056, + 0.0029390419949777424, + 0.002994209004100412, + 0.003106290998402983, + 0.0031797909905435517, + 0.0031062079942785203, + 0.0032079999946290627, + 0.0032601249986328185, + 0.0034572500007925555, + 0.0032877080084290355, + 0.003426665993174538, + 0.0034608330024639145, + 0.003398374989046715, + 0.0034698329982347786, + 0.0036991670058341697, + 0.0036262089997762814, + 0.003681333997519687, + 0.0039118330023484305, + 0.0038049580034567043, + 0.004388459012261592, + 0.0041308750078314915, + 0.003906540994648822, + 0.004343957989476621, + 0.004131958005018532, + 0.004262917005689815, + 0.00419279100606218, + 0.004198082999209873, + 0.004225708995363675, + 0.004295624996302649, + 0.004206333993352018, + 0.004335417004767805, + 0.004536291002295911, + 0.004398624994792044 + ], + [ + 0.0016983749956125394, + 0.0016677499952493235, + 0.0014643330068793148, + 0.0014487089938484132, + 0.0014610420039389282, + 0.0015488749922951683, + 0.002054000011412427, + 0.0020947500015608966, + 0.00207274999411311, + 0.002326084009837359, + 0.002283750000060536, + 0.002223542003775947, + 0.0022842499893158674, + 0.002726333012105897, + 0.0028553330048453063, + 0.0028818330029025674, + 0.002853250000043772, + 0.003030500010936521, + 0.003074541993555613, + 0.0031791670044185594, + 0.00313129200367257, + 0.0033643329952610657, + 0.0033651249977992848, + 0.003424916008953005, + 0.0028969589911866933, + 0.0029787090024910867, + 0.003009624997503124, + 0.003035292000276968, + 0.003053249994991347, + 0.0030667920073028654, + 0.0033284579985775054, + 0.003192834003129974, + 0.002894375007599592, + 0.0029900840017944574, + 0.0030763329996261746, + 0.003150542004732415, + 0.0032959579984890297, + 0.0030350000015459955, + 0.0033968330099014565, + 0.004257666994817555, + 0.003327832993818447, + 0.0034903750056400895, + 0.0034886250068666413, + 0.0034881250030593947, + 0.0034806659969035536, + 0.0036008749884786084, + 0.0036350830050650984, + 0.0038249999925028533, + 0.003638833004515618, + 0.0038080410013208166, + 0.003985082992585376, + 0.003928999998606741, + 0.0038726250058971345, + 0.0047603330021956936, + 0.004099749989109114, + 0.004229166006552987, + 0.004093833995284513, + 0.0042613330006133765, + 0.00427408299583476, + 0.00419654200959485, + 0.004235749991494231, + 0.004344667002442293, + 0.004384291998576373, + 0.004394083996885456 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_mha": { + "first_token_ms": { + "count": 5, + "median": 1.7408749990863726, + "p95": 1.8137165956432, + "min": 1.641333001316525, + "max": 1.8263749952893704, + "iqr": 0.0364580046152696, + "mean": 1.739658197038807, + "cv": 0.03846149359124603 + }, + "decode_ms_per_step": { + "count": 315, + "median": 3.468209004495293, + "p95": 5.283992001204752, + "min": 1.7364160012220964, + "max": 11.70095799898263, + "iqr": 1.4876869972795248, + "mean": 3.668824425807947, + "cv": 0.32593751005238797 + }, + "total_ms": { + "count": 5, + "median": 226.7598320322577, + "p95": 252.07567216421012, + "min": 223.63462297653314, + "max": 257.489455936593, + "iqr": 4.34699998004362, + "mean": 232.87559702293947, + "cv": 0.06000032818090737 + }, + "tokens_per_second": { + "count": 5, + "median": 2257.8954809208253, + "p95": 2284.5089331131235, + "min": 1988.4309364732994, + "max": 2289.4487140916735, + "iqr": 42.725650674969074, + "mean": 2204.509819841735, + "cv": 0.05587502627515269 + }, + "median_step_ms_by_index": [ + 1.7408749990863726, + 1.7800830100895837, + 1.8673750018933788, + 1.8369160097790882, + 1.822499994887039, + 1.9455840083537623, + 2.403500009677373, + 2.4202079948736355, + 2.4829589965520427, + 2.499374997569248, + 2.5833339896053076, + 2.5575409963494167, + 2.512875013053417, + 2.8938330069649965, + 2.940625010523945, + 2.784459007671103, + 2.717208000831306, + 2.8632500034291297, + 2.889291994506493, + 2.893790995585732, + 2.8762910078512505, + 3.0690830026287585, + 3.016166010638699, + 3.063541997107677, + 3.034875000594184, + 3.157167011522688, + 3.1519160111201927, + 3.1812919914955273, + 3.1814159883651882, + 3.30191598914098, + 3.3481669961474836, + 3.3969590003835037, + 3.352540996274911, + 3.538082994055003, + 3.5315409913891926, + 3.5644170129671693, + 3.6385830026119947, + 3.7576670001726598, + 3.7911250110482797, + 3.8131670007715, + 3.8379580073524266, + 3.9998329884838313, + 4.106332999072038, + 4.075332995853387, + 4.06795900198631, + 4.2148330103373155, + 4.281916000763886, + 4.291999997803941, + 4.304250003769994, + 4.536833002930507, + 4.471209002076648, + 4.6034170081838965, + 4.666541004553437, + 4.71133399696555, + 4.715291011962108, + 4.799832997377962, + 4.8144170141313225, + 5.09504099318292, + 5.188583003473468, + 5.0473330047680065, + 5.070083003374748, + 5.167166993487626, + 5.262416001642123, + 5.227625006227754 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0017266249924432486, + 0.001772000003256835, + 0.0018051669903798029, + 0.0018369160097790882, + 0.001822499994887039, + 0.0018725000118138269, + 0.0022584999969694763, + 0.0022598750074394047, + 0.002293458004714921, + 0.0024029170017456636, + 0.0023862500092945993, + 0.0025575409963494167, + 0.002512875013053417, + 0.002550208999309689, + 0.0030426669982261956, + 0.002699959004530683, + 0.002717208000831306, + 0.002846749994205311, + 0.002889291994506493, + 0.0028477500018198043, + 0.002976124989800155, + 0.0030607499938923866, + 0.0030116249981801957, + 0.003063541997107677, + 0.003034875000594184, + 0.0031277079979190603, + 0.0031501249904977158, + 0.003177582999342121, + 0.0031786250037839636, + 0.00330191598914098, + 0.003323457989608869, + 0.0034088750107912347, + 0.003412791993469, + 0.003538082994055003, + 0.0035075839987257496, + 0.0035644170129671693, + 0.0036385830026119947, + 0.0037576670001726598, + 0.003928374993847683, + 0.0038131670007715, + 0.0038379580073524266, + 0.003999832988483831, + 0.004112832990358584, + 0.004075332995853387, + 0.00406795900198631, + 0.0042148330103373155, + 0.004281916000763886, + 0.004203916003461927, + 0.0043329999898560345, + 0.004438457996002398, + 0.004536624997854233, + 0.004550415993435308, + 0.004504791999352165, + 0.0046662499953527, + 0.004713333997642621, + 0.0046799580013612285, + 0.0048144170141313225, + 0.00509504099318292, + 0.004934500000672415, + 0.0050473330047680065, + 0.005033708002883941, + 0.0051217089931014925, + 0.00515487500524614, + 0.005138792010257021 + ], + [ + 0.001641333001316525, + 0.0018912500090664253, + 0.0020650420046877116, + 0.0019708750041900203, + 0.0019451659900369123, + 0.002034208009717986, + 0.002609292001579888, + 0.0026364170043962076, + 0.0026102499978151172, + 0.0027737079944927245, + 0.002780082999379374, + 0.0027725420077331364, + 0.0027837089874083176, + 0.0028938330069649965, + 0.0029243749886518344, + 0.0029416250035865232, + 0.002934708012617193, + 0.003155625003273599, + 0.0031217089999699965, + 0.003122291003819555, + 0.0028762910078512505, + 0.0029917919891886413, + 0.003016166010638699, + 0.0030433749925578013, + 0.002995874994667247, + 0.003157167011522688, + 0.003240207995986566, + 0.0031812919914955273, + 0.0031849170045461506, + 0.003317916998639703, + 0.0033132499956991524, + 0.0033794170012697577, + 0.003352540996274911, + 0.0037145000096643344, + 0.0035595000081229955, + 0.004185875004623085, + 0.003772915995796211, + 0.0037884159974055365, + 0.0037911250110482797, + 0.0037869590014452115, + 0.003818041004706174, + 0.004050874995300546, + 0.0039525829924969, + 0.004218916001264006, + 0.004013333003968, + 0.004192541993688792, + 0.004304708010749891, + 0.004291999997803941, + 0.0042988330096704885, + 0.0045404169941321015, + 0.00458233300014399, + 0.004472208995139226, + 0.004666541004553437, + 0.004700499994214624, + 0.004715291011962108, + 0.004799832997377962, + 0.004693708993727341, + 0.005188583003473468, + 0.0055604160006623715, + 0.005193709002924152, + 0.005070083003374748, + 0.005266292006126605, + 0.005343625001842156, + 0.005227625006227754 + ], + [ + 0.0017630829970585182, + 0.0017800830100895837, + 0.0017749589896993712, + 0.0017782080103643239, + 0.0018097090069204569, + 0.0019455840083537623, + 0.002403500009677373, + 0.0024202079948736355, + 0.0024829589965520427, + 0.002499374997569248, + 0.00253891600004863, + 0.0024663340009283274, + 0.002484874989022501, + 0.003468209004495293, + 0.002940625010523945, + 0.002784459007671103, + 0.0026773329882416874, + 0.0028632500034291297, + 0.002851917000953108, + 0.002893790995585732, + 0.002845041992259212, + 0.0030690830026287585, + 0.0029892909951740876, + 0.003027667000424117, + 0.003002917001140304, + 0.0031222499965224415, + 0.0031519160111201927, + 0.0031854160042712465, + 0.0031718750105937943, + 0.0033004580036504194, + 0.003416417006519623, + 0.003413582991925068, + 0.0033457920071668923, + 0.0035480830119922757, + 0.003522083003190346, + 0.003551417001290247, + 0.0035808749962598085, + 0.0036745830002473667, + 0.0038570409960811958, + 0.0037434160039993003, + 0.0038707079947926104, + 0.003953375009587035, + 0.00403729199024383, + 0.003974999999627471, + 0.004065833010827191, + 0.004119167002500035, + 0.004529833007836714, + 0.004928458991344087, + 0.004423416001372971, + 0.004536833002930507, + 0.004469124993192963, + 0.0046691250026924536, + 0.004744666992337443, + 0.00471133399696555, + 0.004706749998149462, + 0.004733209003461525, + 0.004812415994820185, + 0.005019165997509845, + 0.0050485419924370944, + 0.0050065830000676215, + 0.005021958990255371, + 0.005069792008725926, + 0.005262416001642123, + 0.00589824999042321 + ], + [ + 0.0017408749990863726, + 0.0017650420049903914, + 0.0018673750018933788, + 0.0017889169976115227, + 0.0017364160012220964, + 0.0018048750062007457, + 0.0022817919962108135, + 0.0022600840020459145, + 0.0022628330043517053, + 0.0023594999947817996, + 0.0025833339896053076, + 0.002450582993333228, + 0.002459125011228025, + 0.0025662910047685727, + 0.0025813749962253496, + 0.0026077910006279126, + 0.0025935409939847887, + 0.0027498330018715933, + 0.0027874590014107525, + 0.0028917500021634623, + 0.0028576670010806993, + 0.0033259580086451024, + 0.003546250009094365, + 0.003250791007303633, + 0.0030515410035150126, + 0.00376700000197161, + 0.0032513330079382285, + 0.003247832995839417, + 0.0032660840079188347, + 0.003371583006810397, + 0.003369792000739835, + 0.0033969590003835037, + 0.0034074589930241928, + 0.0035057920031249523, + 0.003622625008574687, + 0.003672167003969662, + 0.003650833008578047, + 0.003820917001576163, + 0.003741625012480654, + 0.0038335410063154995, + 0.0038291660021059215, + 0.003974040999310091, + 0.004106332999072038, + 0.004034875004435889, + 0.004095249998499639, + 0.004217499998048879, + 0.004199582996079698, + 0.004236708002281375, + 0.004298625004594214, + 0.004542458002106287, + 0.004428583008120768, + 0.0046034170081838965, + 0.0046422919986071065, + 0.004833707993384451, + 0.0053252919897204265, + 0.004844082999625243, + 0.004814750005607493, + 0.0050840830081142485, + 0.005188583003473468, + 0.004964292005752213, + 0.005100915994262323, + 0.005167166993487626, + 0.0052313749911263585, + 0.005213915996137075 + ], + [ + 0.0018263749952893704, + 0.0019086249958490953, + 0.001969917007954791, + 0.0019958750053774565, + 0.0019242080015828833, + 0.002199041991843842, + 0.0027561660099308938, + 0.0026306250074412674, + 0.0027393330092309043, + 0.002787833000184037, + 0.0027148749941261485, + 0.002834041995811276, + 0.002810000005410984, + 0.002919166989158839, + 0.0029935000056866556, + 0.003005459002451971, + 0.0029464580002240837, + 0.0032051250018412247, + 0.003126541996607557, + 0.003152165998471901, + 0.003152999997837469, + 0.0033372499892720953, + 0.0033220420009456575, + 0.0031597090128343552, + 0.0030742499948246405, + 0.0031679999956395477, + 0.003147625000565313, + 0.0031767500040587038, + 0.0031814159883651882, + 0.003295832997537218, + 0.0033481669961474836, + 0.0033689589909045026, + 0.0033502500009490177, + 0.003503250001813285, + 0.0035315409913891926, + 0.0035625829914351925, + 0.0035697909916052595, + 0.0037036670109955594, + 0.003787582987570204, + 0.0038441659999080002, + 0.003960791000281461, + 0.004104457999346778, + 0.004431166991707869, + 0.004076999990502372, + 0.004229832993587479, + 0.004231166996760294, + 0.004156916998908855, + 0.004331624993938021, + 0.004304250003769994, + 0.004528666991973296, + 0.004471209002076648, + 0.004994332994101569, + 0.005878541007405147, + 0.006393667004886083, + 0.006390916008967906, + 0.006563416987773962, + 0.006189624997205101, + 0.01170095799898263, + 0.006469208005000837, + 0.006655250006588176, + 0.008504792000167072, + 0.007074708002619445, + 0.006930917006684467, + 0.006884874994284473 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "speedup_cached_vs_uncached_sdpa": 4.5660108731756015, + "speedup_cached_vs_uncached_mha": 5.267620378073937 + }, + { + "batch": 8, + "prompt_len": 128, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_mha": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 7.51174999459181, + "p95": 7.91112559963949, + "min": 7.44391699845437, + "max": 7.998834000318311, + "iqr": 0.09024998871609569, + "mean": 7.596966999699362, + "cv": 0.030134578183822737 + }, + "decode_ms_per_step": { + "count": 315, + "median": 0.9290000016335398, + "p95": 1.0817035974469036, + "min": 0.8393339958274737, + "max": 1.2544579949462786, + "iqr": 0.08462501136818901, + "mean": 0.9471991815925798, + "cv": 0.07334856197098553 + }, + "total_ms": { + "count": 5, + "median": 66.59062500693835, + "p95": 68.68153961549979, + "min": 66.28112110774964, + "max": 68.74312201398425, + "iqr": 2.1327109716366976, + "mean": 67.27051544003189, + "cv": 0.018058944455317916 + }, + "tokens_per_second": { + "count": 5, + "median": 7688.770002483872, + "p95": 7724.175067790338, + "min": 7448.017852547417, + "max": 7724.673201705041, + "iqr": 240.65365483746154, + "mean": 7613.034493232384, + "cv": 0.017942711627892374 + }, + "median_step_ms_by_index": [ + 7.51174999459181, + 1.1411659943405539, + 0.9359159885207191, + 0.9080000018002465, + 0.8942910062614828, + 0.8897919906303287, + 0.8716660086065531, + 0.9356249938718975, + 0.8777500042924657, + 0.9019169956445694, + 0.9026659972732887, + 0.8952089992817491, + 0.9037910058395937, + 0.9013340022647753, + 0.9032090019900352, + 0.8965000015450642, + 0.8987499895738438, + 0.8923749992391095, + 0.9009589994093403, + 0.9390000050188974, + 0.9182079957099631, + 0.9276250057155266, + 0.8906670118449256, + 0.903125008335337, + 0.8895420032786205, + 0.9193750011036173, + 0.9031660010805354, + 0.9067079954547808, + 0.8967909961938858, + 0.9005830070236698, + 0.8915830112528056, + 0.9156660089502111, + 0.9011670044856146, + 0.9104160126298666, + 0.9682090021669865, + 0.9259170037694275, + 0.9249160066246986, + 0.9572499984642491, + 0.9330419998150319, + 0.9415840031579137, + 0.9078750008484349, + 0.9272500028600916, + 0.9235409961547703, + 0.9560000034980476, + 0.931167000089772, + 0.9658329945523292, + 0.9245410037692636, + 0.9409580088686198, + 0.9355829970445484, + 1.011707994621247, + 1.0186250001424924, + 0.9845000022323802, + 0.9578329918440431, + 0.9576669981470332, + 0.9540000028209761, + 0.9623340010875836, + 0.9724169940454885, + 0.970209002844058, + 0.9940840100171044, + 0.9968330123228952, + 0.9972499974537641, + 0.9882500016828999, + 0.9987079974962398, + 1.0097500053234398 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.007998834000318311, + 0.0011411659943405539, + 0.0009510409872746095, + 0.0009109999955398962, + 0.0008942910062614828, + 0.0008846249984344468, + 0.0008767079998506233, + 0.0009356249938718975, + 0.0008892080077202991, + 0.0009015419927891344, + 0.0008923330024117604, + 0.0008916249935282394, + 0.000984791011433117, + 0.0009013340022647753, + 0.0008587079937569797, + 0.0008474589994875714, + 0.0008571670041419566, + 0.0008597080013714731, + 0.0008414159965468571, + 0.0009390000050188974, + 0.0011171249934704974, + 0.0009870000067166984, + 0.0008892500045476481, + 0.0008672499970998615, + 0.0008576670079492033, + 0.0009744580020196736, + 0.0008858750079525635, + 0.0008689590031281114, + 0.0008560840069549158, + 0.0008861669921316206, + 0.0008869580051396042, + 0.0009074590052478015, + 0.0008858340006554499, + 0.0008922919951146469, + 0.0009190830023726448, + 0.0008982919971458614, + 0.0009249160066246986, + 0.0010262919968226925, + 0.000946375002968125, + 0.0009415840031579137, + 0.0009677909984020516, + 0.0009241669904440641, + 0.0009264170075766742, + 0.00098637500195764, + 0.0010314170067431405, + 0.0010038749896921217, + 0.001152084005298093, + 0.0012544579949462786, + 0.0010937909974018112, + 0.001011707994621247, + 0.0010186250001424924, + 0.0010498750052647665, + 0.000980874989181757, + 0.0009789169998839498, + 0.0010019169858423993, + 0.001089208002667874, + 0.001085583004169166, + 0.001040292001562193, + 0.0010228750034002587, + 0.001080791000276804, + 0.0010244580043945462, + 0.0010067920084111392, + 0.0010132089955732226, + 0.0010132090101251379 + ], + [ + 0.007560291996924207, + 0.0011336669995216653, + 0.0009335420036222786, + 0.0009037499985424802, + 0.0008912079938454553, + 0.0008897919906303287, + 0.0008716660086065531, + 0.0009470829972997308, + 0.0009814579971134663, + 0.0009323749982286245, + 0.0009055840055225417, + 0.0008968750044004992, + 0.000870000003487803, + 0.0008412090101046488, + 0.0008672499970998615, + 0.0010737090051406994, + 0.0010838329908438027, + 0.0008923749992391095, + 0.000863666005898267, + 0.0008589580102125183, + 0.0008892500045476481, + 0.0009276250057155266, + 0.0008697500015841797, + 0.0008564579911762848, + 0.0008434580086031929, + 0.0008393339958274737, + 0.000864541987539269, + 0.001008208011626266, + 0.0008879579982021824, + 0.0008630000083940104, + 0.0008452920010313392, + 0.0008645409980090335, + 0.0008657500002300367, + 0.0008590409997850657, + 0.0009010829962790012, + 0.0009137089946307242, + 0.0009056660055648535, + 0.000893083997652866, + 0.0010493750014575198, + 0.0009476249979343265, + 0.0009231250005541369, + 0.0009272500028600916, + 0.0009235409961547703, + 0.000936416006879881, + 0.0009268339927075431, + 0.0009806660091271624, + 0.0009245410037692636, + 0.0009409580088686198, + 0.0009167090029222891, + 0.0009617920004529878, + 0.0009547500085318461, + 0.0009845000022323802, + 0.0009744160051923245, + 0.0009589580004103482, + 0.0009540000028209761, + 0.0009623340010875836, + 0.0009571660048095509, + 0.0009807080059545115, + 0.0009940840100171044, + 0.0009968330123228952, + 0.0010507910046726465, + 0.0009882500016829, + 0.001059707996319048, + 0.0010097500053234398 + ], + [ + 0.007470042008208111, + 0.0011620829900493845, + 0.0009359159885207191, + 0.0009080000018002465, + 0.0008887909934855998, + 0.000876499994774349, + 0.0008746670064283535, + 0.0009513750119367614, + 0.0008762079960433766, + 0.0009019169956445694, + 0.0009015410032588989, + 0.0008941249980125576, + 0.0009037910058395937, + 0.0008995829994091764, + 0.0009032090019900352, + 0.000894041993888095, + 0.0008987499895738438, + 0.0008910420001484454, + 0.0009009589994093403, + 0.0009551249968353659, + 0.0009182079957099631, + 0.0009009580098791048, + 0.0008906670118449256, + 0.0009158750035567209, + 0.0008958329999586567, + 0.0009193750011036173, + 0.0009065420017577708, + 0.0009067079954547808, + 0.0009137079905485734, + 0.0009109170059673488, + 0.0008915830112528056, + 0.0009156660089502111, + 0.0009103750053327531, + 0.0009104160126298666, + 0.0009682090021669865, + 0.0009259170037694275, + 0.0009277080098399892, + 0.0009781249973457307, + 0.0009330419998150319, + 0.0009296659991377965, + 0.0009078750008484349, + 0.0009291669994127005, + 0.0009225840040016919, + 0.0009560000034980476, + 0.000931167000089772, + 0.0009366660087835044, + 0.0009290000016335398, + 0.0009427089971723035, + 0.0009355829970445484, + 0.0009669999999459833, + 0.000970374996541068, + 0.0009686249977676198, + 0.0009513330005574971, + 0.000934750001761131, + 0.0009569590038154274, + 0.0009540000028209761, + 0.0009569579997332767, + 0.0009666669939178973, + 0.000995875001535751, + 0.0010141249949811026, + 0.000997249997453764, + 0.0010316250118194148, + 0.0009910420048981905, + 0.0009979999886127189 + ], + [ + 0.00751174999459181, + 0.0011376659967936575, + 0.0009244170068996027, + 0.0008970420021796599, + 0.0009035830007633194, + 0.0008945000008679926, + 0.0008674579876242206, + 0.0008838339999783784, + 0.0008749580010771751, + 0.0009009999921545386, + 0.0009026659972732887, + 0.0009057080023922026, + 0.0009021669975481927, + 0.0009083340119104832, + 0.0009297500073444098, + 0.0008965000015450642, + 0.0008986670000012964, + 0.0008993750088848174, + 0.0009049999935086817, + 0.0009508749935775995, + 0.001055416010785848, + 0.0010579579975456, + 0.001109792006900534, + 0.0010669170005712658, + 0.0010472910071257502, + 0.0010556670022197068, + 0.0010527090053074062, + 0.00106654099363368, + 0.0009963750053429976, + 0.001017999995383434, + 0.0009891249937936664, + 0.0010160409874515608, + 0.001040457995259203, + 0.0010055419988930225, + 0.001033583001117222, + 0.0010476669995114207, + 0.00102862500352785, + 0.0009572499984642491, + 0.0008854159968905151, + 0.000909624999621883, + 0.0008952500065788627, + 0.0009072919929167256, + 0.0009717909997561947, + 0.0009799580002436414, + 0.000898582991794683, + 0.0009061250020749867, + 0.0008852499886415899, + 0.0009227920090779662, + 0.0009301669924752787, + 0.0011226659989915788, + 0.0012008750054519624, + 0.0010205830039922148, + 0.0009578329918440431, + 0.0009576669981470332, + 0.000935500007472001, + 0.0010560420050751418, + 0.0010040420020231977, + 0.000970209002844058, + 0.0009843750012805685, + 0.0009837500110734254, + 0.0009810000046854839, + 0.000974915994447656, + 0.000986750004813075, + 0.0009744580020196736 + ], + [ + 0.00744391699845437, + 0.0011565000022528693, + 0.0009944579942384735, + 0.0009545420034555718, + 0.0009089579980354756, + 0.000897624995559454, + 0.0008705829968675971, + 0.0009059579897439107, + 0.0008777500042924657, + 0.0009163749928120524, + 0.0009829170012380928, + 0.0008952089992817491, + 0.0009090410021599382, + 0.0009132919949479401, + 0.0009115840075537562, + 0.0009119579917751253, + 0.0009123750060098246, + 0.0009089590021176264, + 0.0009108329977607355, + 0.000906708010006696, + 0.0009043749887496233, + 0.0009054999973159283, + 0.000900708997505717, + 0.000903125008335337, + 0.0008895420032786205, + 0.0008940839907154441, + 0.0009031660010805354, + 0.0008995410025818273, + 0.0008967909961938858, + 0.0009005830070236698, + 0.0009000839927466586, + 0.0009559580066706985, + 0.0009011670044856146, + 0.0009166660020127892, + 0.0009803330030990764, + 0.0010196249932050705, + 0.0008922910055844113, + 0.0009362909913761541, + 0.0008796250040177256, + 0.001005833997623995, + 0.0008878329972503707, + 0.0009657079936005175, + 0.000887875008629635, + 0.0009305000130552799, + 0.0009687919955467805, + 0.0009658329945523292, + 0.0008784579986240715, + 0.0008970000053523108, + 0.0009677500056568533, + 0.0011749169934773818, + 0.001034000000800006, + 0.0009446249896427616, + 0.0009256670018658042, + 0.0009302920079790056, + 0.0009315420029452071, + 0.0009462500020163134, + 0.0009724169940454885, + 0.000960625009611249, + 0.0009895409893942997, + 0.0009649590065237135, + 0.000991875000181608, + 0.0009688340069260448, + 0.0009987079974962398, + 0.0010317920095985755 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + ] + } + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 7.247583998832852, + "p95": 7.634991794475354, + "min": 6.970166999963112, + "max": 7.729624994681217, + "iqr": 0.17354299779981375, + "mean": 7.257350196596235, + "cv": 0.0399381686658446 + }, + "decode_ms_per_step": { + "count": 315, + "median": 9.96845900954213, + "p95": 12.596295708499383, + "min": 7.315124996239319, + "max": 13.691749991266988, + "iqr": 2.8698329915641807, + "mean": 10.118037530189632, + "cv": 0.15845722950081534 + }, + "total_ms": { + "count": 5, + "median": 640.344876126619, + "p95": 653.2254897843814, + "min": 638.1576649000635, + "max": 653.3379559696186, + "iqr": 13.92317409045063, + "mean": 644.693714598543, + "cv": 0.011909181637290184 + }, + "tokens_per_second": { + "count": 5, + "median": 799.5691370204066, + "p95": 802.1350564337204, + "min": 783.6679245738004, + "max": 802.3095673076026, + "iqr": 17.09400079533725, + "mean": 794.265330796571, + "cv": 0.011861435787827106 + }, + "median_step_ms_by_index": [ + 7.247583998832852, + 7.51062500057742, + 7.713750004768372, + 7.730041004833765, + 7.945165998535231, + 8.192749999579974, + 7.949459002702497, + 7.907665989478119, + 7.948208003654145, + 8.858542001689784, + 8.332249999511987, + 8.35887499852106, + 8.347959010279737, + 8.406958004343323, + 8.44804100051988, + 8.52654199115932, + 8.405875007156283, + 8.661542000481859, + 8.75362500664778, + 8.93125000584405, + 8.976957993581891, + 8.857292006723583, + 9.29500001075212, + 9.189499993226491, + 9.098957991227508, + 9.144209005171433, + 9.340667005744763, + 9.226957990904339, + 9.514666002360173, + 9.525125002255663, + 9.475583006860688, + 9.53549999394454, + 9.462874993914738, + 9.956791996955872, + 10.483874997589737, + 10.550416001933627, + 10.381292013335042, + 10.166624997509643, + 10.241957992548123, + 10.33462499617599, + 10.1859999995213, + 10.654875004547648, + 10.779375006677583, + 11.805458008893766, + 11.213749996386468, + 11.322792008286342, + 11.856916011311114, + 11.50624999718275, + 11.243125001783483, + 11.634041991783306, + 11.877416996867396, + 11.781082997913472, + 11.563041000044905, + 11.626916995737702, + 11.875541997142136, + 11.947791004786268, + 11.830250005004928, + 12.510667002061382, + 12.0120420033345, + 12.071874996763654, + 11.816083992016502, + 12.012499995762482, + 12.483917002100497, + 12.813041001209058 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.006970166999963112, + 0.00751062500057742, + 0.007610874992678873, + 0.0077102909999666736, + 0.00794516599853523, + 0.008885124989319593, + 0.0082188749947818, + 0.007907665989478119, + 0.007948208003654145, + 0.007989624995389022, + 0.008000875008292496, + 0.008114833995932713, + 0.008315250001032837, + 0.00827829199261032, + 0.008419874997343868, + 0.00832333299331367, + 0.008580790992709808, + 0.008704791995114647, + 0.009284416999435052, + 0.009286832995712757, + 0.008976957993581891, + 0.00883229098690208, + 0.009032124988152646, + 0.00913962499180343, + 0.008864917006576434, + 0.009085917001357302, + 0.009172459002002142, + 0.0090812919952441, + 0.009154458006378263, + 0.009582166007021442, + 0.010207582992734388, + 0.009656958995037712, + 0.01017083300394006, + 0.009956791996955872, + 0.010011082995333709, + 0.010289875004673377, + 0.010023875001934357, + 0.009926792001351714, + 0.010022083995863795, + 0.010011000005761161, + 0.0101859999995213, + 0.010654875004547648, + 0.011804874986410141, + 0.011515707999933511, + 0.011015041993232444, + 0.010857583009055816, + 0.01110358300502412, + 0.010882375005166978, + 0.010742083002696745, + 0.011180624991538934, + 0.011877416996867396, + 0.012501124991104007, + 0.012249791994690895, + 0.011626916995737702, + 0.011632250010734424, + 0.01154212499386631, + 0.011366749997250736, + 0.012789749991497956, + 0.012161250007920898, + 0.012735500000417233, + 0.012615208004717715, + 0.013517166997189634, + 0.013581625011283904, + 0.012813041001209058 + ], + [ + 0.007247583998832852, + 0.007538249992649071, + 0.007745541006443091, + 0.007714334002230316, + 0.007615165988681838, + 0.010296083011780865, + 0.008625292000942864, + 0.008898875006707385, + 0.008652000004076399, + 0.009058124996954575, + 0.008332249999511987, + 0.008401375001994893, + 0.008347959010279737, + 0.008427166001638398, + 0.008370000010472722, + 0.008470958011457697, + 0.00834729100461118, + 0.008621375003713183, + 0.008702334002009593, + 0.008720042009372264, + 0.008692040995811112, + 0.008857292006723583, + 0.00887291700928472, + 0.009052958004758693, + 0.008879250002792105, + 0.00902166700689122, + 0.00996845900954213, + 0.009755041013704613, + 0.01002208401041571, + 0.009649500003433786, + 0.009501166990958154, + 0.010062041998025961, + 0.010388707989477552, + 0.01087129200459458, + 0.010483874997589737, + 0.010550416001933627, + 0.010381292013335042, + 0.010166624997509643, + 0.01021870800468605, + 0.01033462499617599, + 0.010143999999854714, + 0.010368707997258753, + 0.010614250000799075, + 0.011805458008893766, + 0.011480791989015415, + 0.010829416001797654, + 0.011091917011071928, + 0.011035250005079433, + 0.01084041599824559, + 0.011344999991706572, + 0.011065500002587214, + 0.011091584005043842, + 0.011518416999024339, + 0.012637917010579258, + 0.012523499986855313, + 0.011947791004786268, + 0.011830250005004928, + 0.012178165998193435, + 0.011845291999634355, + 0.01186474999121856, + 0.011816083992016502, + 0.012006083998130634, + 0.012379042003885843, + 0.012223333003930748 + ], + [ + 0.00708291599585209, + 0.007455750004737638, + 0.007564542000181973, + 0.008792124994215555, + 0.008361833999515511, + 0.008192749999579974, + 0.007780749991070479, + 0.008037624997086823, + 0.007765874994220212, + 0.008014374994672835, + 0.00812299999233801, + 0.007985834003193304, + 0.008208665996789932, + 0.008308333999593742, + 0.008449582994217053, + 0.008665290995850228, + 0.009453040998778306, + 0.009398084002896212, + 0.008875583996996284, + 0.009336874994914979, + 0.00938224999117665, + 0.009255500001017936, + 0.009295000010752119, + 0.009189499993226491, + 0.009098957991227508, + 0.009150249999947846, + 0.009177291009109467, + 0.009124542004428804, + 0.00923795800190419, + 0.009400708004250191, + 0.009343124998849817, + 0.009422165996511467, + 0.009351458997116424, + 0.00985516699438449, + 0.012346957999398, + 0.013081124998279847, + 0.01260904200898949, + 0.01105874999484513, + 0.010432459006551653, + 0.010583084003883414, + 0.011321124999085441, + 0.01082245800353121, + 0.013597333003417589, + 0.01211949999560602, + 0.010780833006720059, + 0.012566624995088205, + 0.01196791599795688, + 0.01150624999718275, + 0.011243125001783483, + 0.012174083007266745, + 0.012030375000904314, + 0.011795667000114918, + 0.011610292000113986, + 0.011721250004484318, + 0.011875541997142136, + 0.011697832989739254, + 0.011491458004456945, + 0.011957208000239916, + 0.012027500008116476, + 0.012923665999551304, + 0.012834458000725135, + 0.012263291995623149, + 0.012429833004716784, + 0.012332208003499545 + ], + [ + 0.007256458993651904, + 0.007516209007008001, + 0.007785665991832502, + 0.007730041004833765, + 0.007629999992786907, + 0.00794224999845028, + 0.00779550000152085, + 0.007798291990184225, + 0.008084208006039262, + 0.009101916992221959, + 0.008824709002510644, + 0.00835887499852106, + 0.008467167004710063, + 0.008470792003208771, + 0.00844804100051988, + 0.008565166994230822, + 0.008366958005353808, + 0.008573417013394646, + 0.008533875006833114, + 0.008622583001852036, + 0.008613792000687681, + 0.008852040991769172, + 0.009503082997980528, + 0.009837457997491583, + 0.009334290996775962, + 0.009144209005171433, + 0.009340667005744763, + 0.009226957990904339, + 0.009607957996195182, + 0.009376041998621076, + 0.009475583006860688, + 0.00948445800167974, + 0.009412625004188158, + 0.01096729100390803, + 0.010710167000070214, + 0.010307707998435944, + 0.010221582997473888, + 0.01004879099491518, + 0.010337125000660308, + 0.010117124998942018, + 0.010148790999664925, + 0.010327707990654744, + 0.010400165992905386, + 0.010734333001892082, + 0.011213749996386468, + 0.012193333997856826, + 0.011896332987817004, + 0.01206433298648335, + 0.011533957993378863, + 0.011634041991783306, + 0.01153537500067614, + 0.011565666005481035, + 0.011488541000289842, + 0.01160304099903442, + 0.011847500005387701, + 0.012526833001174964, + 0.01208737499837298, + 0.012590833008289337, + 0.011903917009476572, + 0.012071874996763654, + 0.011803082990809344, + 0.012012499995762482, + 0.012483917002100497, + 0.013394166002399288 + ], + [ + 0.007729624994681217, + 0.0073151249962393194, + 0.007713750004768372, + 0.009501791006186977, + 0.00816391700936947, + 0.007870875007938594, + 0.007949459002702497, + 0.007595208007842302, + 0.007717417000094429, + 0.008858542001689784, + 0.008971625007689, + 0.008599249995313585, + 0.008634333004010841, + 0.008406958004343323, + 0.008498791998135857, + 0.00852654199115932, + 0.008405875007156283, + 0.008661542000481859, + 0.00875362500664778, + 0.00893125000584405, + 0.009834750002482906, + 0.009553625000989996, + 0.009489083007792942, + 0.009893791997455992, + 0.009528082999167964, + 0.010018832996138372, + 0.00937362499826122, + 0.00941683299606666, + 0.009514666002360173, + 0.009525125002255663, + 0.009303124999860302, + 0.00953549999394454, + 0.009462874993914738, + 0.009705125004984438, + 0.009875042000203393, + 0.010897166997892782, + 0.01089254098769743, + 0.010618957996484824, + 0.010241957992548123, + 0.010651500007952563, + 0.010489292006241158, + 0.011782959001720883, + 0.010779375006677583, + 0.013518499996280298, + 0.012492209003539756, + 0.011322792008286342, + 0.011856916011311114, + 0.011922374993446283, + 0.011638707990641706, + 0.011685916004353203, + 0.01312079100171104, + 0.011781082997913472, + 0.011563041000044905, + 0.011336959010805003, + 0.012165749998530373, + 0.012515708993305452, + 0.012069584001437761, + 0.012510667002061382, + 0.0120120420033345, + 0.01206195900158491, + 0.011799415995483287, + 0.011945124992053024, + 0.012575000000651926, + 0.013691749991266988 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_mha": { + "first_token_ms": { + "count": 5, + "median": 9.053707995917648, + "p95": 9.1051249881275, + "min": 8.916000006138347, + "max": 9.116124987485819, + "iqr": 0.029499991796910763, + "mean": 9.03571659582667, + "cv": 0.00816717996307585 + }, + "decode_ms_per_step": { + "count": 315, + "median": 13.133334010490216, + "p95": 16.720341898326296, + "min": 9.149959005299024, + "max": 29.500916993129067, + "iqr": 3.535562500474043, + "mean": 13.209919425259743, + "cv": 0.18298366513727907 + }, + "total_ms": { + "count": 5, + "median": 835.8817919943249, + "p95": 862.8907953912858, + "min": 830.570367994369, + "max": 868.4977129887557, + "iqr": 9.572921044309624, + "mean": 841.2606403871905, + "cv": 0.018731372659777253 + }, + "tokens_per_second": { + "count": 5, + "median": 612.5268009229183, + "p95": 616.3963978538687, + "min": 589.5237170378465, + "max": 616.4438556077541, + "iqr": 7.018626559396694, + "mean": 608.7777761371553, + "cv": 0.018344972316444278 + }, + "median_step_ms_by_index": [ + 9.053707995917648, + 9.264624997740611, + 9.246125002391636, + 9.275292002712376, + 9.367332997499034, + 9.650208012317307, + 10.180124998441897, + 10.450165995280258, + 10.433917006594129, + 10.75608299288433, + 10.646249997080304, + 10.784375001094304, + 10.765332990558818, + 10.905834002187476, + 11.177458000020124, + 11.203917005332187, + 11.227333001443185, + 11.387207996449433, + 11.392750006052665, + 11.571583003387786, + 11.571832990739495, + 11.749833996873349, + 11.687958001857623, + 11.761333007598296, + 12.016666994895786, + 12.421917010215111, + 12.248208993696608, + 12.144000007538125, + 12.554332992294803, + 12.976625002920628, + 13.131957995938137, + 12.748541994369589, + 12.697208003373817, + 13.027291992329992, + 13.095625006826594, + 13.95620898983907, + 13.549458992201835, + 13.473291997797787, + 13.457374996505678, + 13.620957994135097, + 13.819333995343186, + 13.984750010422431, + 14.006375000462867, + 14.330332996905781, + 14.562500000465661, + 14.744250001967885, + 14.630082994699478, + 14.759415993466973, + 14.43433300300967, + 15.00220800517127, + 15.05508300033398, + 15.287249989341944, + 15.2649579977151, + 16.021250004996546, + 16.178250007214956, + 15.603791005560197, + 15.477000008104369, + 15.997833004803397, + 16.29908400354907, + 16.710791998775676, + 16.708750001271255, + 16.421374995843507, + 16.989333991659805, + 16.575417001149617 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.009053707995917648, + 0.00929362500028219, + 0.009246125002391636, + 0.009286042011808604, + 0.009250958013581112, + 0.009650208012317307, + 0.010906333001912571, + 0.010450165995280258, + 0.010433917006594129, + 0.010744625004008412, + 0.010455499999807216, + 0.0106620829901658, + 0.010576625005342066, + 0.010706125001888722, + 0.010747541004093364, + 0.010847915997146629, + 0.01104470799327828, + 0.011834207994979806, + 0.011888207998708822, + 0.011510500000440516, + 0.011571832990739495, + 0.011749833996873349, + 0.011687958001857623, + 0.011920125005417503, + 0.011876000004122034, + 0.013299875005031936, + 0.012248208993696608, + 0.013288832997204736, + 0.012554332992294803, + 0.012543000004370697, + 0.012386166999931447, + 0.012657625004067086, + 0.012477124997531064, + 0.012924915994517505, + 0.013063999998848885, + 0.014090666998527013, + 0.013740416994551197, + 0.013451166989398189, + 0.013457374996505678, + 0.013617666001664475, + 0.013540416999603622, + 0.013819041996612214, + 0.013974332992802374, + 0.0147502500039991, + 0.014725500004715286, + 0.014744250001967885, + 0.01451545799500309, + 0.014608999990741722, + 0.014360542001668364, + 0.014856916008284315, + 0.014773999995668419, + 0.01578070801042486, + 0.0152649579977151, + 0.015942457990604453, + 0.015776541986269876, + 0.015603791005560197, + 0.01536258299893234, + 0.015995791007298976, + 0.0167935000063153, + 0.016710791998775676, + 0.016249333013547584, + 0.016421374995843507, + 0.016505874998983927, + 0.016296707995934412 + ], + [ + 0.009031624998897314, + 0.009182374997180887, + 0.00918562499282416, + 0.009275292002712376, + 0.009220292005920783, + 0.009808874994632788, + 0.010180124998441897, + 0.01006937499914784, + 0.010229874998913147, + 0.010830041996086948, + 0.0105583749973448, + 0.010706458007916808, + 0.010490000000572763, + 0.010870416997931898, + 0.01079050000407733, + 0.010960542000248097, + 0.011227333001443185, + 0.011937957999180071, + 0.011946667000302114, + 0.011571583003387786, + 0.011379000003216788, + 0.011710832986864261, + 0.011628540989477187, + 0.01169445799314417, + 0.011718124995240942, + 0.011880291989655234, + 0.012485790997743607, + 0.012953124998603016, + 0.012798833995475434, + 0.012382000000798143, + 0.012560749994008802, + 0.012748541994369589, + 0.012477540993131697, + 0.01292216700676363, + 0.0130623330041999, + 0.01395620898983907, + 0.013671167005668394, + 0.013456000000587665, + 0.013389415995334275, + 0.013526124996133149, + 0.013483583999914117, + 0.01377433400193695, + 0.01400516601279378, + 0.014261832999181934, + 0.015350416986620985, + 0.014488625005469657, + 0.014630082994699478, + 0.014631291007390246, + 0.014359541004523635, + 0.014848666003672406, + 0.015135624998947605, + 0.015667125000618398, + 0.015617416996974498, + 0.015299250007956289, + 0.01560258299286943, + 0.015253707999363542, + 0.015301166000426747, + 0.015831833006814122, + 0.018526208004914224, + 0.017554040998220444, + 0.016742624997277744, + 0.016383583002607338, + 0.017091500005335547, + 0.016575417001149617 + ], + [ + 0.008916000006138347, + 0.009264624997740611, + 0.009273542003938928, + 0.009202749992255121, + 0.010032999998657033, + 0.009932542001479305, + 0.010000792011851445, + 0.010462083999300376, + 0.029500916993129067, + 0.01075608299288433, + 0.010646249997080304, + 0.010784375001094304, + 0.010765332990558818, + 0.010937792001641355, + 0.011739875000785105, + 0.011271625000517815, + 0.011359333002474159, + 0.011387207996449433, + 0.011392750006052665, + 0.011519417006638832, + 0.011401249998016283, + 0.011541209009010345, + 0.011565167005755939, + 0.011761333007598296, + 0.012515124995843507, + 0.01245879199996125, + 0.012532499997178093, + 0.012144000007538125, + 0.015156999987084419, + 0.015404874997329898, + 0.014425458997720852, + 0.013499791995855048, + 0.013655959002790041, + 0.01368779200129211, + 0.013333416995010339, + 0.013372290995903313, + 0.013504874994396232, + 0.013473291997797787, + 0.013337208001757972, + 0.013620957994135097, + 0.014707209003972821, + 0.01421358299558051, + 0.015172291998169385, + 0.01448908400197979, + 0.014562500000465661, + 0.014492707996396348, + 0.014516333001665771, + 0.015434375003678724, + 0.016022666997741908, + 0.015690874992287718, + 0.01505508300033398, + 0.015287249989341944, + 0.015881125000305474, + 0.017256291990634054, + 0.01629829200101085, + 0.016913166007725522, + 0.0156356249935925, + 0.016307958008837886, + 0.01629908400354907, + 0.016291499996441416, + 0.015866959001868963, + 0.016280791009194218, + 0.017530541997984983, + 0.01675383301335387 + ], + [ + 0.009061124990694225, + 0.00955345899274107, + 0.009301583995693363, + 0.00923645799048245, + 0.009376500005600974, + 0.009579000005032867, + 0.011058500007493421, + 0.0105689999909373, + 0.01043795800069347, + 0.010905874994932674, + 0.011177458000020124, + 0.011270250004599802, + 0.01118920900626108, + 0.010905834002187476, + 0.011177458000020124, + 0.011276374993030913, + 0.011464875002275221, + 0.011360791002516635, + 0.011243083004956134, + 0.011677834001602605, + 0.01209466700674966, + 0.012355291997664608, + 0.01180249999742955, + 0.011776833998737857, + 0.012035792009555735, + 0.011946082988288254, + 0.01197708300605882, + 0.012082916000508703, + 0.012331875012023374, + 0.012976625002920628, + 0.013220584005466662, + 0.01283987499482464, + 0.01273741599288769, + 0.013027291992329992, + 0.013223791000200436, + 0.014213541988283396, + 0.013549458992201835, + 0.013766457996098325, + 0.014036458000191487, + 0.014125292000244372, + 0.014490166999166831, + 0.013984750010422431, + 0.014392625002074055, + 0.014330332996905781, + 0.014100417000008747, + 0.014889291996951215, + 0.015096832998096943, + 0.014759415993466973, + 0.014456999997491948, + 0.01500220800517127, + 0.014762958002393134, + 0.014845999990939163, + 0.014877040986903012, + 0.016021250004996546, + 0.016178250007214956, + 0.015594458003761247, + 0.015477000008104369, + 0.015997833004803397, + 0.0157954169990262, + 0.016193250005017035, + 0.016838208000990562, + 0.01680987500003539, + 0.01648041700536851, + 0.01656433400057722 + ], + [ + 0.009116124987485819, + 0.009253625001292676, + 0.009149959005299024, + 0.009276500000851229, + 0.009367332997499034, + 0.009398708993103355, + 0.009677583002485335, + 0.009990541002480313, + 0.009927582999807782, + 0.01056862500263378, + 0.011246124995523132, + 0.011481374996947125, + 0.011797500003012829, + 0.011622499994700775, + 0.01123408300918527, + 0.011203917005332187, + 0.01119874999858439, + 0.011357915995176882, + 0.011343124991981313, + 0.012158124998677522, + 0.011809916992206126, + 0.012030375000904314, + 0.011908833010238595, + 0.01168850000249222, + 0.012016666994895786, + 0.012421917010215111, + 0.011995707987807691, + 0.011984833006863482, + 0.012283749994821846, + 0.013167166005587205, + 0.013131957995938137, + 0.012647333991480991, + 0.012697208003373817, + 0.013415791996521875, + 0.013095625006826594, + 0.013095833011902869, + 0.013133334010490216, + 0.013751624996075407, + 0.013895124997361563, + 0.014067332987906411, + 0.013819333995343186, + 0.014065584007767029, + 0.014006375000462867, + 0.014205665996996686, + 0.013867708999896422, + 0.014948041003663093, + 0.015210167010081932, + 0.014829667008598335, + 0.01443433300300967, + 0.015304999993531965, + 0.018331915998714976, + 0.015051084003061987, + 0.01526216699858196, + 0.016290375002427027, + 0.01637983300315682, + 0.016820624994579703, + 0.016320666996762156, + 0.01653749999240972, + 0.016206250002142042, + 0.016960625012870878, + 0.016708750001271255, + 0.01662945799762383, + 0.016989333991659805, + 0.016673833000822924 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "speedup_cached_vs_uncached_sdpa": 9.61614155235664, + "speedup_cached_vs_uncached_mha": 12.55254462482115 + }, + { + "batch": 8, + "prompt_len": 512, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_mha": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 43.4477499948116, + "p95": 48.61073319334537, + "min": 43.21574998903088, + "max": 49.54387499310542, + "iqr": 1.436415986972861, + "mean": 44.905458195717074, + "cv": 0.059585852105286385 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.6491659916937351, + "p95": 2.376071304024663, + "min": 1.5200829948298633, + "max": 4.90962500043679, + "iqr": 0.15283400716725737, + "mean": 1.7889361267739405, + "cv": 0.25694494167901366 + }, + "total_ms": { + "count": 5, + "median": 157.14546394883655, + "p95": 174.1412841482088, + "min": 146.78466599434614, + "max": 177.92991892201826, + "iqr": 11.791368058766238, + "mean": 157.60843418247532, + "cv": 0.0802980902120356 + }, + "tokens_per_second": { + "count": 5, + "median": 3258.12776986485, + "p95": 3486.1562381888925, + "min": 2877.5374209235456, + "max": 3488.102769670241, + "iqr": 257.97585971488206, + "mean": 3264.506465054151, + "cv": 0.07619193396875495 + }, + "median_step_ms_by_index": [ + 43.4477499948116, + 2.240749992779456, + 1.825665996875614, + 1.6819580050650984, + 1.638874993659556, + 1.6077910113381222, + 1.609333005035296, + 1.637542009120807, + 1.581332995556295, + 1.6054170118877664, + 1.647666998906061, + 1.5922089951345697, + 1.6277500108117238, + 1.6689589974703267, + 1.6103329980978742, + 1.6546660044696182, + 1.595666995854117, + 1.7134160007117316, + 1.6342919989256188, + 1.5947920037433505, + 1.6553750028833747, + 1.661500005866401, + 1.7067500011762604, + 1.7369580018566921, + 1.6386250063078478, + 1.5925839979900047, + 1.629500009585172, + 1.6282909928122535, + 1.7218340071849525, + 1.6354999970644712, + 1.6652499907650054, + 1.6350420046364889, + 1.5901670121820644, + 1.609583996469155, + 1.835957998991944, + 1.661875008721836, + 1.6341669979738072, + 1.6998339997371659, + 1.7117910028900951, + 1.6902089992072433, + 1.6093750018626451, + 1.622499999939464, + 1.6185830027097836, + 1.649167010327801, + 2.0068339945282787, + 1.6677499952493235, + 1.753917007590644, + 1.6259999974863604, + 1.6456670127809048, + 1.6314999957103282, + 1.6997500060824677, + 1.6553750028833747, + 1.7656249983701855, + 1.7102080018958077, + 1.7278339946642518, + 1.6575829940848053, + 1.6321669972967356, + 1.6635409992886707, + 1.85070899897255, + 1.704166003037244, + 1.6654589999234304, + 1.676375002716668, + 2.1777920046588406, + 1.753083008225076 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.04321574998903088, + 0.0021677499898942187, + 0.0021328329894458875, + 0.0016819580050650984, + 0.0018528749933466315, + 0.0016425000067101792, + 0.0018877080001402646, + 0.001637542009120807, + 0.0015995830035535619, + 0.001632750005228445, + 0.001812540998798795, + 0.0016047079989220947, + 0.0017616670083953068, + 0.0016769170033512637, + 0.004399666999233887, + 0.0017685420025372878, + 0.0018117500003427267, + 0.0017311249976046383, + 0.003934791995561682, + 0.0018435410020174459, + 0.004504208991420455, + 0.0019135410111630335, + 0.0038174589863047004, + 0.0017123330035246909, + 0.0015765419957460836, + 0.001584792000357993, + 0.004243332994519733, + 0.001727249997202307, + 0.0044291250087553635, + 0.0017691659886622801, + 0.00490962500043679, + 0.0017385420069331303, + 0.0015829579933779314, + 0.0016083340015029535, + 0.003788542002439499, + 0.0017740839975886047, + 0.003177957987645641, + 0.0017299579922109842, + 0.0025817090063355863, + 0.0017010000010486692, + 0.0015525839990004897, + 0.0015601250051986426, + 0.0019767089979723096, + 0.0016220419929595664, + 0.0020068339945282787, + 0.0016677499952493235, + 0.0019585000118240714, + 0.001688916003331542, + 0.001572833993122913, + 0.0016087080002762377, + 0.0021748750004917383, + 0.001770499991835095, + 0.0019669999892357737, + 0.0019508749974193051, + 0.0021547919895965606, + 0.001778833000571467, + 0.0016858750022947788, + 0.001655291998758912, + 0.002078833000268787, + 0.0019509999983711168, + 0.0020984580041840672, + 0.001768000001902692, + 0.002182457988965325, + 0.0018311669991817325 + ], + [ + 0.0434477499948116, + 0.002272583995363675, + 0.001825665996875614, + 0.0016144590044859797, + 0.0015955419949023053, + 0.001614875000086613, + 0.001609333005035296, + 0.0016542499943170696, + 0.0016019580070860684, + 0.0016054170118877664, + 0.0015913340030238032, + 0.0015922089951345697, + 0.0015855410019867122, + 0.0015902080049272627, + 0.0016052919963840395, + 0.0016546660044696182, + 0.001595666995854117, + 0.0017134160007117316, + 0.001587666993145831, + 0.0015772500046296045, + 0.0015807500021765009, + 0.0017629579961067066, + 0.0017387499974574894, + 0.0017369580018566921, + 0.0016819579905131832, + 0.0015925839979900047, + 0.0015897919947747141, + 0.0016052499995566905, + 0.0016491249989485368, + 0.001584792000357993, + 0.0015805829898454249, + 0.001592666987562552, + 0.0015976669965311885, + 0.0016043749928940088, + 0.0015969580126693472, + 0.001604083998245187, + 0.0015974999987520278, + 0.0016104170063044876, + 0.0016402909968746826, + 0.0016902089992072433, + 0.0016093750018626451, + 0.001622499999939464, + 0.0016185830027097836, + 0.0016190419992199168, + 0.0016181660030269995, + 0.0016215829964494333, + 0.0016091659927042201, + 0.0016157500067492947, + 0.0016456670127809048, + 0.001629917009267956, + 0.0016189579910133034, + 0.0016326670011039823, + 0.0016419999883510172, + 0.0016717499966034666, + 0.0016279580013360828, + 0.0016341670125257224, + 0.0016321669972967356, + 0.0016635409992886707, + 0.0016583329997956753, + 0.0016570410080021247, + 0.001650874997721985, + 0.001676375002716668, + 0.0016021659976104274, + 0.0016341670125257224 + ], + [ + 0.044878165994305164, + 0.0021998750016791746, + 0.001708582989522256, + 0.0016198750090552494, + 0.0015848329931031913, + 0.001575250003952533, + 0.0015567499940516427, + 0.0015753750049043447, + 0.0015724160039098933, + 0.0015803339920239523, + 0.001568000006955117, + 0.0015776670043123886, + 0.0016277500108117238, + 0.0016078750049928203, + 0.0016103329980978742, + 0.001586500002304092, + 0.0015877909900154918, + 0.001593332999618724, + 0.0016342919989256188, + 0.0015947920037433505, + 0.0015408750041387975, + 0.0015200829948298633, + 0.0015530839882558212, + 0.0015306249988498166, + 0.001524000006611459, + 0.0015382079873234034, + 0.0015384159923996776, + 0.001542707992484793, + 0.001573374989675358, + 0.0015587500092806295, + 0.0015562080079689622, + 0.0015462089941138402, + 0.001552999994601123, + 0.001609583996469155, + 0.0015915420081000775, + 0.0015970840031513944, + 0.001581749995239079, + 0.0015985830104909837, + 0.001588207989698276, + 0.00160116700863, + 0.001634125001146458, + 0.0017011250020004809, + 0.0016170830058399588, + 0.001652958002523519, + 0.001624999989871867, + 0.001864208999904804, + 0.001753917007590644, + 0.0016259999974863604, + 0.0016641670081298798, + 0.0017682920006336644, + 0.0016997500060824677, + 0.0016209169989451766, + 0.0016190000023925677, + 0.0017108339961851016, + 0.001687291995040141, + 0.0016453339922009036, + 0.0016501670033903793, + 0.0016374159895349294, + 0.0016511249996256083, + 0.001658667009905912, + 0.0016416250000474975, + 0.0016405000060331076, + 0.0016477089957334101, + 0.0016949160053627566 + ], + [ + 0.0434417500073323, + 0.002240749992779456, + 0.0018070410005748272, + 0.0017005830013658851, + 0.0017457500071031973, + 0.0016077910113381222, + 0.0017261659959331155, + 0.0016445420042145997, + 0.0015754170017316937, + 0.0016333750099875033, + 0.0016755840042605996, + 0.0016208749875659123, + 0.0017703749908832833, + 0.002009666000958532, + 0.0019437080045463517, + 0.0016696249949745834, + 0.0016730839997762814, + 0.0017690410022623837, + 0.0017456670029787347, + 0.001657332992181182, + 0.0018256670009577647, + 0.001661500005866401, + 0.0017067500011762604, + 0.0018115419952664524, + 0.0016386250063078478, + 0.0016090409917524084, + 0.0017643329920247197, + 0.0019020839972654358, + 0.001746875001117587, + 0.0016485420055687428, + 0.0017227920034201816, + 0.0016350420046364889, + 0.0015901670121820644, + 0.0016105830000014976, + 0.001835957998991944, + 0.001726709000649862, + 0.00196508300723508, + 0.0017459999944549054, + 0.0024625830119475722, + 0.0016421250038547441, + 0.0016038330068113282, + 0.0016239579999819398, + 0.0029394159937510267, + 0.001649167010327801, + 0.0027554159896681085, + 0.0017608339985599741, + 0.0022188330040080473, + 0.0016480829945066944, + 0.0016174999909708276, + 0.0016314999957103282, + 0.0023379170015687123, + 0.0016553750028833747, + 0.002261874993564561, + 0.0017102080018958077, + 0.0023880000080680475, + 0.0016713329969206825, + 0.0016284580051433295, + 0.0017263330082641914, + 0.0023709590022917837, + 0.0017247079958906397, + 0.001992374993278645, + 0.0016577909991610795, + 0.0027496660040924326, + 0.001753083008225076 + ], + [ + 0.04954387499310542, + 0.002388667009654455, + 0.0018304169934708625, + 0.0018577500013634562, + 0.001638874993659556, + 0.0016027920064516366, + 0.001580833995831199, + 0.0015745830023661256, + 0.001581332995556295, + 0.0016029580001486465, + 0.001647666998906061, + 0.001580416996148415, + 0.0016242080018855631, + 0.0016689589974703267, + 0.001600834002601914, + 0.0015916249976726249, + 0.0015835840022191405, + 0.0015881659928709269, + 0.0016105830000014976, + 0.001586500002304092, + 0.0016553750028833747, + 0.0015839580009924248, + 0.0015744169941172004, + 0.0018117919971700758, + 0.0017382079968228936, + 0.0017401670047547668, + 0.001629500009585172, + 0.0016282909928122535, + 0.0017218340071849525, + 0.0016354999970644712, + 0.0016652499907650054, + 0.001664209004957229, + 0.0016177080105990171, + 0.0017214580002473667, + 0.0021469170023920015, + 0.001661875008721836, + 0.0016341669979738072, + 0.0016998339997371659, + 0.0017117910028900951, + 0.0017477090004831553, + 0.0017367499967804179, + 0.0015783749986439943, + 0.0015975419955793768, + 0.0020094589999644086, + 0.0021749169973190874, + 0.001611291998415254, + 0.001585791993420571, + 0.0015852079959586263, + 0.0016474580042995512, + 0.0018087079952238128, + 0.0016491659916937351, + 0.0017010840092552826, + 0.0017656249983701855, + 0.0016334169922629371, + 0.0017278339946642518, + 0.0016575829940848053, + 0.001629041988053359, + 0.0017884160042740405, + 0.00185070899897255, + 0.001704166003037244, + 0.0016654589999234304, + 0.0017037499928846955, + 0.0021777920046588406, + 0.001881332995253615 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + ] + } + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 51.8514590075938, + "p95": 54.63706699956674, + "min": 43.48654100613203, + "max": 54.9515419988893, + "iqr": 9.179417000268586, + "mean": 49.573691803379916, + "cv": 0.10793607057167814 + }, + "decode_ms_per_step": { + "count": 315, + "median": 47.6197090029018, + "p95": 51.53884540341096, + "min": 43.066875005024485, + "max": 67.94258300215006, + "iqr": 3.007770501426421, + "mean": 47.76275131083648, + "cv": 0.05880522588106972 + }, + "total_ms": { + "count": 5, + "median": 3045.297582037165, + "p95": 3169.63956260297, + "min": 2953.5676300001796, + "max": 3196.7532870039577, + "iqr": 24.85270710894838, + "mean": 3058.627024386078, + "cv": 0.028688121104128078 + }, + "tokens_per_second": { + "count": 5, + "median": 168.12806834381533, + "p95": 172.4046457980106, + "min": 160.16250052247656, + "max": 173.34967880859693, + "iqr": 1.369004522881255, + "mean": 167.5040541326676, + "cv": 0.028283009427957646 + }, + "median_step_ms_by_index": [ + 51.8514590075938, + 53.64212499989662, + 49.14037500566337, + 49.30095901363529, + 49.409249986638315, + 49.97462499886751, + 46.15037501207553, + 45.45133298961446, + 44.37291699287016, + 44.01825000240933, + 44.38020799716469, + 44.67458400176838, + 45.71095800201874, + 45.310874993447214, + 45.53866600326728, + 44.96800000197254, + 45.40062499290798, + 45.21204199409112, + 46.66124998766463, + 44.53920800006017, + 45.76212500978727, + 45.94254199764691, + 46.448583001620136, + 46.53524998866487, + 46.352375007700175, + 46.091083000646904, + 47.20287500822451, + 46.99295898899436, + 46.29937499703374, + 47.26229100197088, + 47.10833299031947, + 46.00429200218059, + 47.603250000975095, + 47.894957999233156, + 47.61550000694115, + 46.79679099353962, + 47.685499986982904, + 47.801708002225496, + 47.1975420077797, + 46.928416995797306, + 47.591124995960854, + 48.14366600476205, + 48.19441700237803, + 48.70487500738818, + 47.75779201008845, + 48.56270900927484, + 48.32779199932702, + 48.57704100140836, + 48.34858400863595, + 48.20445900259074, + 48.87720900296699, + 47.82362500554882, + 48.90187499404419, + 48.76187500485685, + 49.056792006012984, + 48.907417003647424, + 48.52329099958297, + 49.77779199543875, + 49.47554200771265, + 49.47066599561367, + 49.18879200704396, + 50.251499997102655, + 49.934624999878, + 49.774500002968125 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.04419975000200793, + 0.043402916999184527, + 0.043920750002143905, + 0.044361709005897865, + 0.043066875005024485, + 0.04326200000650715, + 0.04483470800914802, + 0.04310187500959728, + 0.045668667007703334, + 0.043697792003513314, + 0.043418999994173646, + 0.04370024999661837, + 0.043528916998184286, + 0.0447819580003852, + 0.0440743339859182, + 0.04335537500446662, + 0.04598554200492799, + 0.0451062920037657, + 0.04500008298782632, + 0.04453920800006017, + 0.043670041995937936, + 0.04453954199561849, + 0.04463112499797717, + 0.047699167000246234, + 0.045660208008484915, + 0.04479308299778495, + 0.04763679200550541, + 0.044771625005523674, + 0.046188500011339784, + 0.04726229100197088, + 0.050096583989216015, + 0.04600429200218059, + 0.045350125001277775, + 0.04742370800522622, + 0.04568337500677444, + 0.04636887498782016, + 0.045597791991895065, + 0.04774708300828934, + 0.0471975420077797, + 0.04578337499697227, + 0.046669958988786675, + 0.045937000002595596, + 0.04723695799475536, + 0.04653558399877511, + 0.048422332998597994, + 0.04752429200743791, + 0.04656112499651499, + 0.04753962499671616, + 0.04718779200629797, + 0.04683916599606164, + 0.0476057089981623, + 0.04695437499321997, + 0.04890187499404419, + 0.04723162499431055, + 0.04803150000225287, + 0.048907417003647424, + 0.04852329099958297, + 0.04977779199543875, + 0.048060417000669986, + 0.04880029198830016, + 0.04783195799973328, + 0.0494176670035813, + 0.04796041699592024, + 0.047996332999900915 + ], + [ + 0.05337916700227652, + 0.05650733299262356, + 0.04914037500566337, + 0.04864508299215231, + 0.049878416990395635, + 0.05066041699319612, + 0.04962304199580103, + 0.04804733300989028, + 0.04377795901382342, + 0.043283291000989266, + 0.04438020799716469, + 0.04698812498827465, + 0.04571095800201874, + 0.04473741599940695, + 0.04553866600326728, + 0.04529524999088608, + 0.04362341699015815, + 0.04521204199409112, + 0.044975333992624655, + 0.04441308400419075, + 0.045959125010995194, + 0.044613082995056175, + 0.046448583001620136, + 0.0454464159993222, + 0.04651866600033827, + 0.04611920801107772, + 0.04720287500822451, + 0.04703995800809935, + 0.0474037079984555, + 0.04657349998888094, + 0.045715874992311, + 0.045439624998834915, + 0.04819741699611768, + 0.052217500007827766, + 0.04761550000694115, + 0.046278000008896925, + 0.04745895799715072, + 0.047801708002225496, + 0.04711529200721998, + 0.046928416995797306, + 0.0467117919906741, + 0.04849508299957961, + 0.04679137500352226, + 0.047864749998552725, + 0.04684200001065619, + 0.04678820900153369, + 0.04832779199932702, + 0.04855616600252688, + 0.048887166994973086, + 0.04798841600131709, + 0.06582158300443552, + 0.047716208995552734, + 0.04870637500425801, + 0.05030595800781157, + 0.049056792006012984, + 0.048924250004347414, + 0.047846207991824485, + 0.04925012499734294, + 0.048315209001884796, + 0.04947066599561367, + 0.04875750000064727, + 0.04926737499772571, + 0.049934624999878, + 0.04864870899473317 + ], + [ + 0.0549515419988893, + 0.057138459000270814, + 0.04972174999420531, + 0.04934524999407586, + 0.049409249986638315, + 0.04997462499886751, + 0.04451058400445618, + 0.044702749990392476, + 0.04429212499235291, + 0.04401825000240933, + 0.04307162499753758, + 0.04467458400176838, + 0.043447999996715225, + 0.045310874993447214, + 0.04455879099259619, + 0.04415970800619107, + 0.04540062499290798, + 0.04464100000041071, + 0.04666124998766463, + 0.04449516598833725, + 0.04480108298594132, + 0.0476197090029018, + 0.04641975001140963, + 0.04653524998866487, + 0.045240249994094484, + 0.046091083000646904, + 0.045819459002814256, + 0.04564162499445956, + 0.046258874994236976, + 0.04902754099748563, + 0.04710833299031947, + 0.0459926670009736, + 0.046061332992394455, + 0.047894957999233156, + 0.04771904100198299, + 0.0470763750054175, + 0.047685499986982904, + 0.04827512499468867, + 0.04696716699982062, + 0.04678587499074638, + 0.047695749992271885, + 0.048143666004762053, + 0.04819441700237803, + 0.04943837500468362, + 0.047619332995964214, + 0.04856270900927484, + 0.04859825001040008, + 0.04857704100140836, + 0.047343832993647084, + 0.04829758399864659, + 0.047927915991749614, + 0.04838962500798516, + 0.04888345900690183, + 0.04805700000724755, + 0.04883358400547877, + 0.04819920800218824, + 0.04850800000713207, + 0.04820774999097921, + 0.04969066700141411, + 0.04935416599619202, + 0.04918879200704396, + 0.050251499997102655, + 0.04908762499690056, + 0.049774500002968125 + ], + [ + 0.04348654100613203, + 0.04574733300250955, + 0.04598220899060834, + 0.04930095901363529, + 0.0455191249930067, + 0.04651116700551938, + 0.04615037501207553, + 0.04545133298961446, + 0.04437291699287016, + 0.0467309999949066, + 0.0458544580033049, + 0.044666707995929755, + 0.046222500008298084, + 0.045855000003939494, + 0.04648033301054966, + 0.04496800000197254, + 0.0449563330039382, + 0.04644499999994878, + 0.04849533300148323, + 0.04744008299894631, + 0.04576212500978727, + 0.04594254199764691, + 0.047524541994789615, + 0.04645808300119825, + 0.04713245799939614, + 0.045989958991413005, + 0.04674383299425244, + 0.04699295898899436, + 0.04629937499703374, + 0.04651787498733029, + 0.04562900000018999, + 0.049537999992026016, + 0.04771591699682176, + 0.047890708010527305, + 0.04726416700577829, + 0.046796790993539616, + 0.04809604199544992, + 0.0465817919903202, + 0.048268250000546686, + 0.04693170799873769, + 0.047591124995960854, + 0.04745783300313633, + 0.049084041005698964, + 0.04870487500738818, + 0.04775779201008845, + 0.04944233399874065, + 0.048110290998010896, + 0.049092458008090034, + 0.04834858400863595, + 0.04820445900259074, + 0.04887720900296699, + 0.04782362500554882, + 0.051179415997467004, + 0.04876187500485685, + 0.049794167003710754, + 0.04870954100624658, + 0.051946874998975545, + 0.05047683299926575, + 0.04947554200771265, + 0.049646665996988304, + 0.05043666700657923, + 0.0513024159881752, + 0.05081483400135767, + 0.05154529100400396 + ], + [ + 0.0518514590075938, + 0.05364212499989662, + 0.05527129200345371, + 0.05467729199153837, + 0.05633741698693484, + 0.05408750000060536, + 0.04957270799786784, + 0.04879812498984393, + 0.05114604100526776, + 0.05281258300237823, + 0.05035770800895989, + 0.04948154199519195, + 0.047173375001875684, + 0.046992332994705066, + 0.046581374990637414, + 0.04634854200412519, + 0.04733983299229294, + 0.04761187500844244, + 0.04888562500127591, + 0.04815287500969134, + 0.046826624995446764, + 0.04675966598733794, + 0.04720658301084768, + 0.047699750008177944, + 0.046352375007700175, + 0.04836095799691975, + 0.05025054101133719, + 0.04735245798656251, + 0.04819512501126155, + 0.047770542005309835, + 0.04714345801039599, + 0.04787225001200568, + 0.047603250000975095, + 0.04884245799621567, + 0.04817637499945704, + 0.048396750004030764, + 0.04836804200022016, + 0.048118749997229315, + 0.050504208003985696, + 0.050386667004204355, + 0.04988066598889418, + 0.0500026249937946, + 0.05008037500374485, + 0.04905362499994226, + 0.048477207994437777, + 0.05077120799978729, + 0.05153608300315682, + 0.06794258300215006, + 0.050627541000721976, + 0.050005332988803275, + 0.05042945800232701, + 0.050706292007816955, + 0.04951908299699426, + 0.04924016700533684, + 0.0496735000051558, + 0.04963462500018068, + 0.049071666988311335, + 0.05043912499968428, + 0.05024162499466911, + 0.05005554200033657, + 0.05212720799318049, + 0.053853416990023106, + 0.05124212500231806, + 0.05283374999999069 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_mha": { + "first_token_ms": { + "count": 5, + "median": 80.33195900497958, + "p95": 94.40575799671933, + "min": 77.9307089978829, + "max": 97.0115829986753, + "iqr": 4.942791987559758, + "mean": 83.6592749983538, + "cv": 0.09328452007851548 + }, + "decode_ms_per_step": { + "count": 315, + "median": 89.52562500780914, + "p95": 102.09650440519908, + "min": 78.55062501039356, + "max": 124.25379200431053, + "iqr": 9.669187995314132, + "mean": 90.26154751772992, + "cv": 0.07769266292296377 + }, + "total_ms": { + "count": 5, + "median": 5610.63291906612, + "p95": 6132.995251988177, + "min": 5524.615584974526, + "max": 6172.613625996746, + "iqr": 406.2217988685006, + "mean": 5770.136768615339, + "cv": 0.049795063260679354 + }, + "tokens_per_second": { + "count": 5, + "median": 91.25530174325172, + "p95": 92.53071904522545, + "min": 82.94703524673032, + "max": 92.67613142034764, + "iqr": 6.251833696567431, + "mean": 88.90495476064714, + "cv": 0.04863975379656711 + }, + "median_step_ms_by_index": [ + 80.33195900497958, + 79.34895899961703, + 84.38662499247584, + 82.19820899830665, + 83.68850000260863, + 81.7724999942584, + 80.37179199163802, + 80.63679200131446, + 80.33937501022592, + 82.20137500029523, + 83.18066700303461, + 82.2474579908885, + 83.90045800479129, + 83.2783340010792, + 82.90512500389013, + 82.25983400188852, + 81.77666700794362, + 84.70245900389273, + 84.25845800957177, + 82.90487500198651, + 87.35600000363775, + 84.77695799956564, + 89.21237499453127, + 85.56970801146235, + 84.89933300006669, + 85.23562499613035, + 85.17958300944883, + 87.10074999544304, + 86.52112500567455, + 87.7161660027923, + 87.06270800030325, + 86.89720800612122, + 91.52883299975656, + 87.96725000138395, + 88.21833399997558, + 87.72937500907574, + 87.86166699428577, + 88.36145799432416, + 88.49470899440348, + 90.15291700779926, + 90.43458300584462, + 89.52562500780914, + 89.678833988728, + 90.88524999970105, + 90.27066700218711, + 93.99204200599343, + 90.25012500933371, + 92.49262500088662, + 90.8232079964364, + 91.87258299789391, + 92.19870799279306, + 93.81770799518563, + 93.3788749971427, + 93.05095800664276, + 94.06954199948814, + 93.57158400234766, + 93.08783400047105, + 93.61208400514442, + 96.05404200556222, + 95.84150000591762, + 95.33758400357328, + 97.27116700378247, + 96.10112500377, + 98.45004099770449 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.08033195900497958, + 0.07934895899961703, + 0.08010229200590402, + 0.08007933299813885, + 0.07873583299806342, + 0.07995212500100024, + 0.07895458400889765, + 0.07930820799083449, + 0.08016570900508668, + 0.07993391698983032, + 0.08051854200311936, + 0.08056383300572634, + 0.08024725000723265, + 0.08237708298838697, + 0.08034020800550934, + 0.08084658299048897, + 0.0795617499970831, + 0.0823452499898849, + 0.08225470800243784, + 0.08221179099928122, + 0.08328237499517854, + 0.08148608299961779, + 0.09780899999896064, + 0.08369900000980124, + 0.08276987500721589, + 0.084760166995693, + 0.08384433400351554, + 0.08710074999544304, + 0.08352120898780413, + 0.08458475000225008, + 0.08472566699492745, + 0.08622433300479315, + 0.08400329199503176, + 0.08553862500411924, + 0.085495457999059, + 0.08634120799251832, + 0.08612120799080003, + 0.08830745800514705, + 0.08771304199763108, + 0.08750275000056718, + 0.09005324999452569, + 0.0877254160004668, + 0.08959033300925512, + 0.08832408300077077, + 0.08769195899367332, + 0.09399204200599343, + 0.08946258299693, + 0.0888197919994127, + 0.08932754100533202, + 0.08949729199230205, + 0.09110862499801442, + 0.09065066699986346, + 0.09158187499269843, + 0.09305095800664276, + 0.09316170899546705, + 0.09357158400234766, + 0.09234483299951535, + 0.09239870800229255, + 0.09333570899616461, + 0.09542974999931175, + 0.09233408400905319, + 0.09360112500144169, + 0.09610112500377, + 0.09845004099770449 + ], + [ + 0.08398245798889548, + 0.08521949998976197, + 0.08544204098870978, + 0.08543991700571496, + 0.08546566701261327, + 0.08467199999722652, + 0.08688054198864847, + 0.08702854199509602, + 0.08624458301346749, + 0.08616579200315755, + 0.08523637500184122, + 0.0869626670028083, + 0.08547608299704734, + 0.08688299999630544, + 0.08711037499597296, + 0.08676033299707342, + 0.08659083300153725, + 0.08807658399746288, + 0.08881862499401905, + 0.0910932499973569, + 0.08923629099444952, + 0.09104250000382308, + 0.08921237499453127, + 0.09133308300806675, + 0.10480145900510252, + 0.09210170799633488, + 0.09321341699978802, + 0.09292645800451282, + 0.09122387498791795, + 0.09513300000980962, + 0.09524424999835901, + 0.09371800000371877, + 0.09152883299975656, + 0.09420883399434388, + 0.09256304199516308, + 0.09467583399964496, + 0.09537104199989699, + 0.09673870900587644, + 0.09491829200123902, + 0.0958208329975605, + 0.09442612501152325, + 0.09492133399180602, + 0.0943939590069931, + 0.09589945900370367, + 0.09458512499986682, + 0.09602770899073221, + 0.09856458401191048, + 0.09668445799616165, + 0.09892945899628103, + 0.09929004199511837, + 0.10085824999259785, + 0.09836970799369738, + 0.09834741700615268, + 0.09998479200294241, + 0.09972870799538214, + 0.1001094999955967, + 0.09808966700802557, + 0.10035112500190735, + 0.10248937500000466, + 0.10262062499532476, + 0.10037479200400412, + 0.10113529099908192, + 0.10195774999738205, + 0.10182149999309331 + ], + [ + 0.07903966600133572, + 0.07867562498722691, + 0.07928345800610259, + 0.0801052499882644, + 0.08368850000260863, + 0.08005175000289455, + 0.08035879199451301, + 0.08032962500874419, + 0.07855062501039356, + 0.08090508299937937, + 0.08208533299330156, + 0.08045554200361948, + 0.08390045800479129, + 0.08171016701089684, + 0.0815105830115499, + 0.08159245800925419, + 0.08081379201030359, + 0.08203554099600296, + 0.08310479199280962, + 0.08290487500198651, + 0.08270658399851527, + 0.08294083400687668, + 0.08465966601215769, + 0.08392254200589377, + 0.08489933300006669, + 0.08523562499613035, + 0.08391970799129922, + 0.08468862499285024, + 0.08498929099005181, + 0.0877161660027923, + 0.08496837499842513, + 0.08451795800647233, + 0.09918104199459776, + 0.08796725000138395, + 0.08758937500533648, + 0.08755199999723118, + 0.08773574999941047, + 0.08836145799432416, + 0.08849470899440348, + 0.08828325000649784, + 0.08852291699440684, + 0.08952562500780914, + 0.08915187499951571, + 0.09088524999970105, + 0.09027066700218711, + 0.09013375001086388, + 0.0894655000010971, + 0.09249262500088662, + 0.0908232079964364, + 0.09074283299560193, + 0.09030687500489876, + 0.09338533300615381, + 0.09241816599387676, + 0.0926265419984702, + 0.0937857079989044, + 0.09271400001307484, + 0.09275837500172202, + 0.09361208400514442, + 0.09605404200556222, + 0.09469254199939314, + 0.09533758400357328, + 0.09727116700378247, + 0.09551362501224503, + 0.094378207999398 + ], + [ + 0.0779307089978829, + 0.0792918329971144, + 0.09679533299640752, + 0.08219820899830665, + 0.08057454200752545, + 0.0817724999942584, + 0.08037179199163802, + 0.08063679200131446, + 0.08033937501022592, + 0.08220137500029523, + 0.08318066700303461, + 0.0822474579908885, + 0.08205145799729507, + 0.0832783340010792, + 0.08290512500389013, + 0.08225983400188852, + 0.08177666700794362, + 0.08470245900389273, + 0.08425845800957177, + 0.08283600000140723, + 0.08735600000363775, + 0.08477695799956564, + 0.08876845800841693, + 0.08556970801146235, + 0.08409400000527967, + 0.08495420801045839, + 0.08517958300944883, + 0.08525937500235159, + 0.08652112500567455, + 0.08652858299319632, + 0.08706270800030325, + 0.08689720800612122, + 0.08637070799886715, + 0.08762350000324659, + 0.08821833399997558, + 0.08772937500907574, + 0.08786166699428577, + 0.08836087500094436, + 0.08843408299435396, + 0.09015291700779926, + 0.09043458300584462, + 0.08928949999972247, + 0.089678833988728, + 0.09023758400871884, + 0.09019137499853969, + 0.09310579199518543, + 0.09025012500933371, + 0.09149620900279842, + 0.08925237499352079, + 0.09187258299789391, + 0.09219870799279306, + 0.09381770799518563, + 0.0933788749971427, + 0.09247170799062587, + 0.09406954199948814, + 0.09339158399961889, + 0.09308783400047105, + 0.09305224999843631, + 0.09372495800198521, + 0.09584150000591762, + 0.09408187499502674, + 0.09588529101165477, + 0.09511599999677856, + 0.09537783300038427 + ], + [ + 0.0970115829986753, + 0.09093245799886063, + 0.08438662499247584, + 0.08498529200733174, + 0.08505495800636709, + 0.08638345799408853, + 0.08547700000053737, + 0.10565283300820738, + 0.08789445800357498, + 0.08857583299686667, + 0.09096091700484976, + 0.08807820799120236, + 0.08768725000845734, + 0.09059345800778829, + 0.08767649999936111, + 0.08884354199108202, + 0.08826441700512078, + 0.0903547920024721, + 0.09203104198968504, + 0.08967295799811836, + 0.09175604100164492, + 0.09241308300988749, + 0.09216508400277235, + 0.09392700000898913, + 0.09129829099401832, + 0.09367379199829884, + 0.09280212500016205, + 0.09498174999316689, + 0.09377062499697786, + 0.09562745899893343, + 0.09519129199907184, + 0.09494566699140705, + 0.09348258300451562, + 0.09816466699703597, + 0.09825279199867509, + 0.09642970800632611, + 0.09456066699931398, + 0.09768270800122991, + 0.09817283399752341, + 0.10014037501241546, + 0.09916258399607614, + 0.09892470799968578, + 0.09869087499100715, + 0.0980533339898102, + 0.10199416699470021, + 0.10315404100401793, + 0.10130358299647924, + 0.09928062499966472, + 0.09823520798818208, + 0.10296120899147354, + 0.1040172920038458, + 0.10083787499752361, + 0.09888225000759121, + 0.10163408399967011, + 0.10222037500352599, + 0.10204341700591613, + 0.10333674999128561, + 0.11482699999760371, + 0.12425379200431053, + 0.10579250000591855, + 0.10414166600094177, + 0.10868954099714756, + 0.10314012500748504, + 0.10708050000539515 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "speedup_cached_vs_uncached_sdpa": 19.37884496003431, + "speedup_cached_vs_uncached_mha": 35.70343539087346 + }, + { + "batch": 8, + "prompt_len": 1024, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_mha": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 120.839082999737, + "p95": 132.19392559840344, + "min": 118.2974160037702, + "max": 133.75345899839886, + "iqr": 7.159875996876508, + "mean": 123.52833320037462, + "cv": 0.05237547284636352 + }, + "decode_ms_per_step": { + "count": 315, + "median": 3.7055840075481683, + "p95": 6.1050873991916905, + "min": 2.739958988968283, + "max": 19.890084004146047, + "iqr": 1.3236660088296048, + "mean": 4.141538796599009, + "cv": 0.43787451653308873 + }, + "total_ms": { + "count": 5, + "median": 382.73279096756596, + "p95": 402.20824482967146, + "min": 368.6359629209619, + "max": 405.7344200555235, + "iqr": 11.08387486601714, + "mean": 384.44527738611214, + "cv": 0.03619753883329263 + }, + "tokens_per_second": { + "count": 5, + "median": 1337.747932978621, + "p95": 1382.727146193328, + "min": 1261.9092063471824, + "max": 1388.9040991634783, + "iqr": 38.783764288465136, + "mean": 1333.163228565254, + "cv": 0.03561376147277313 + }, + "median_step_ms_by_index": [ + 120.839082999737, + 4.191999993054196, + 3.201749990694225, + 2.8365839971229434, + 2.9175000090617687, + 2.800458009005524, + 2.989375003380701, + 2.750792002188973, + 2.841666995664127, + 2.792958010104485, + 2.9167499887989834, + 2.936292003141716, + 3.027290993486531, + 2.894875011406839, + 3.064291988266632, + 2.923958993051201, + 2.93279200559482, + 3.0822500120848417, + 3.339958013384603, + 3.283916012151167, + 3.29033299931325, + 3.154875012114644, + 3.5829580010613427, + 3.4046660002786666, + 3.2299169979523867, + 3.463792003458366, + 3.5809589899145067, + 3.6343750107334927, + 3.5744589986279607, + 3.4625830012373626, + 3.595208007027395, + 3.899540999555029, + 3.4667500003706664, + 3.806499997153878, + 3.64195900328923, + 4.039166000438854, + 4.539250003290363, + 3.5724580084206536, + 3.868125000735745, + 3.7944580108160153, + 3.728916999534704, + 4.367625006125309, + 4.508625002927147, + 4.444917009095661, + 4.450167005416006, + 4.337999998824671, + 4.554208993795328, + 4.477249996853061, + 4.477375012356788, + 4.575500002829358, + 4.465042002266273, + 4.494707987760194, + 4.485666999244131, + 4.507709003519267, + 4.461292002815753, + 4.916040998068638, + 4.523166004219092, + 4.644833999918774, + 4.601749998982996, + 4.762833006680012, + 4.6202090015867725, + 4.389291992993094, + 4.562040994642302, + 4.890500000328757 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.11879591600154527, + 0.004191999993054196, + 0.003370334001374431, + 0.002889084003982134, + 0.0032812499994179234, + 0.003071541999815963, + 0.0037335829983931035, + 0.003081624992773868, + 0.003130124998278916, + 0.0032980419928207994, + 0.0036566250055329874, + 0.0029978329985169694, + 0.0036149590014247224, + 0.0028770829958375543, + 0.003288833991973661, + 0.0032830419950187206, + 0.002931124996393919, + 0.003016292001120746, + 0.003339958013384603, + 0.0030072499939706177, + 0.00334324999130331, + 0.0029317080043256283, + 0.0035829580010613427, + 0.003307874998426996, + 0.0032299169979523867, + 0.0032161249982891604, + 0.0035809589899145067, + 0.0044091669988119975, + 0.005431832993053831, + 0.003098374989349395, + 0.006468084000516683, + 0.003925541997887194, + 0.0034405420010443777, + 0.005090499995276332, + 0.0059301250003045425, + 0.005176624996238388, + 0.007258166006067768, + 0.0035724580084206536, + 0.005926708996412344, + 0.003561874997103587, + 0.0035638750123325735, + 0.0038588750030612573, + 0.005070707993581891, + 0.004181583994068205, + 0.004032290991744958, + 0.003661166992969811, + 0.0038233329978538677, + 0.003918125003110617, + 0.0035689170035766438, + 0.0037789589987369254, + 0.0037587499973597005, + 0.0037055840075481683, + 0.003855249990010634, + 0.0035464169923216105, + 0.004650584000046365, + 0.004916040998068638, + 0.004497957997955382, + 0.004673707997426391, + 0.004721083008917049, + 0.004762833006680012, + 0.0046202090015867725, + 0.004389291992993094, + 0.0047626250016037375, + 0.004978500001016073 + ], + [ + 0.12595579199842177, + 0.0065822080068755895, + 0.007262458995683119, + 0.0028365839971229434, + 0.0027956249978160486, + 0.002739958988968283, + 0.0027567079960135743, + 0.002750792002188973, + 0.002754957997240126, + 0.002758375005214475, + 0.0027633749996311963, + 0.002748124999925494, + 0.002846332994522527, + 0.0028654579946305603, + 0.002915082994149998, + 0.002878500003134832, + 0.0031339999986812472, + 0.0030822500120848417, + 0.003522291997796856, + 0.003283916012151167, + 0.0032363749924115837, + 0.0030819170060567558, + 0.00312100000155624, + 0.0034046660002786666, + 0.003104040995822288, + 0.0032210829958785325, + 0.0031085840018931776, + 0.0031242910044966266, + 0.0034373750095255673, + 0.0034410410007694736, + 0.0035515839990694076, + 0.0034691659966483712, + 0.0034667500003706664, + 0.0034849170042434707, + 0.00364195900328923, + 0.0036345830012578517, + 0.003514708994771354, + 0.003468750001047738, + 0.003544915991369635, + 0.0035827499959850684, + 0.0034671670000534505, + 0.0038608329923590645, + 0.004390665999380872, + 0.00438158398901578, + 0.00462479199632071, + 0.004337999998824671, + 0.005256624994217418, + 0.019890084004146047, + 0.00877383399347309, + 0.009146999989752658, + 0.004465042002266273, + 0.004437291994690895, + 0.004578416992444545, + 0.004496333000133745, + 0.004448708001291379, + 0.00438795899390243, + 0.004459957999642938, + 0.004438916992512532, + 0.004408084001624957, + 0.004434750007931143, + 0.004421082994667813, + 0.0045906669984105974, + 0.004641999999876134, + 0.004890500000328757 + ], + [ + 0.13375345899839886, + 0.004161499993642792, + 0.003035500005353242, + 0.0029064170084893703, + 0.0029175000090617687, + 0.0029252910026116297, + 0.002989375003380701, + 0.0031317919929279014, + 0.0030340829980559647, + 0.003158874998916872, + 0.0034798749984474853, + 0.0031164579995675012, + 0.003027290993486531, + 0.003291250002803281, + 0.0033376249921275303, + 0.00335504200484138, + 0.003238667006371543, + 0.003228834000765346, + 0.0031721669947728515, + 0.003257542004575953, + 0.0032311250106431544, + 0.003198167003574781, + 0.003183499997248873, + 0.0032016670011216775, + 0.0031922089983709157, + 0.0035827920073643327, + 0.003611666994402185, + 0.0035841250064549968, + 0.0035744589986279607, + 0.003554041002644226, + 0.003595208007027395, + 0.0035705420013982803, + 0.003606790996855125, + 0.003610833009588532, + 0.003597374990931712, + 0.004270999997970648, + 0.004539250003290363, + 0.004459415999008343, + 0.0045275830052560195, + 0.004469499996048398, + 0.004434583010151982, + 0.004474542001844384, + 0.004508625002927147, + 0.004444917009095661, + 0.004450167005416006, + 0.0044382920023053885, + 0.00459858299291227, + 0.004477249996853061, + 0.004477375012356788, + 0.004452875000424683, + 0.004533958999672905, + 0.004494707987760194, + 0.004485666999244131, + 0.004510832994128577, + 0.004461292002815753, + 0.004567249998217449, + 0.004523166004219092, + 0.004580624998197891, + 0.004540666996035725, + 0.004577416999381967, + 0.004578167005092837, + 0.004565499999444, + 0.00440829100261908, + 0.004755125002702698 + ], + [ + 0.120839082999737, + 0.0041333749977638945, + 0.0030467920005321503, + 0.0028015419957228005, + 0.00290004099952057, + 0.0027643330104183406, + 0.0027604170027188957, + 0.002742749988101423, + 0.0027475829992908984, + 0.002792958010104485, + 0.0029167499887989834, + 0.002936292003141716, + 0.002908958005718887, + 0.002894875011406839, + 0.002874541998608038, + 0.002900583000155166, + 0.00293279200559482, + 0.0031192499882308766, + 0.003269000008003786, + 0.0032874169992282987, + 0.00329033299931325, + 0.01915612499578856, + 0.005024291996960528, + 0.008288334007374942, + 0.0034736249945126474, + 0.003607416001614183, + 0.003357583991601132, + 0.0036343750107334927, + 0.0033866660087369382, + 0.0034625830012373626, + 0.003307666993350722, + 0.004020791995571926, + 0.003393041988601908, + 0.003862625002511777, + 0.00317241701122839, + 0.003729083007783629, + 0.0035991249897051603, + 0.003485167006147094, + 0.0035613750078482553, + 0.005476374994032085, + 0.003728916999534704, + 0.004367625006125309, + 0.004086750006536022, + 0.005479459010530263, + 0.003730042008101009, + 0.00378662500588689, + 0.003724542009877041, + 0.004067083005793393, + 0.0036042080027982593, + 0.005367874997318722, + 0.0038852500001667067, + 0.004850541998166591, + 0.004453292000107467, + 0.004507709003519267, + 0.00440270800027065, + 0.009936290996847674, + 0.004537583998171613, + 0.011805332993390039, + 0.004601749998982996, + 0.013252083997940645, + 0.004709041997557506, + 0.004355208002380095, + 0.004412666996358894, + 0.012223500001709908 + ], + [ + 0.1182974160037702, + 0.004446125007234514, + 0.003201749990694225, + 0.0027866250020451844, + 0.003005041988217272, + 0.0028004580090055242, + 0.002996834009536542, + 0.0027502919983817264, + 0.002841666995664127, + 0.0027641670021694154, + 0.0028773749945685267, + 0.0028053330024704337, + 0.003056874993490055, + 0.002920208004070446, + 0.003064291988266632, + 0.002923958993051201, + 0.0028755840030498803, + 0.003078583991737105, + 0.0036246249946998432, + 0.004711374989710748, + 0.004852333004237153, + 0.003154875012114644, + 0.004200374998617917, + 0.005094499996630475, + 0.003450667005381547, + 0.003463792003458366, + 0.0048794169997563586, + 0.0037131660064915195, + 0.005043499986641109, + 0.0037712920020567253, + 0.003848083986667916, + 0.003899540999555029, + 0.003519457997754216, + 0.003806499997153878, + 0.005490458002896048, + 0.004039166000438854, + 0.00533374999940861, + 0.003577499999664724, + 0.003868125000735745, + 0.0037944580108160153, + 0.004128084008698352, + 0.004699083001469262, + 0.006065875000786036, + 0.0048686660011298954, + 0.00649433299258817, + 0.004371792005258612, + 0.004554208993795328, + 0.004608167000696994, + 0.004483708005864173, + 0.004575500002829358, + 0.0061965829954715446, + 0.005281582998577505, + 0.0066283329942962155, + 0.004995791998226196, + 0.005182207998586819, + 0.005236875003902242, + 0.0047036249889060855, + 0.004644833999918774, + 0.005943915995885618, + 0.0048035410000011325, + 0.006020500004524365, + 0.004389207999338396, + 0.004562040994642302, + 0.00466479200986214 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + ] + } + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 130.78458298696205, + "p95": 151.09292480628937, + "min": 118.3396250125952, + "max": 154.47508300712798, + "iqr": 13.117583002895117, + "mean": 133.122058401932, + "cv": 0.10455660508385849 + }, + "decode_ms_per_step": { + "count": 315, + "median": 138.1233330030227, + "p95": 148.15471249894475, + "min": 113.14333298651036, + "max": 164.28537499450613, + "iqr": 6.397354998625815, + "mean": 136.23866709517426, + "cv": 0.07412138897901872 + }, + "total_ms": { + "count": 5, + "median": 8932.197585963877, + "p95": 9095.99444437772, + "min": 7767.631794005865, + "max": 9122.698336970643, + "iqr": 220.0950379628921, + "mean": 8716.15808539791, + "cv": 0.06255100744940126 + }, + "tokens_per_second": { + "count": 5, + "median": 57.320720357167275, + "p95": 64.40903335344557, + "min": 56.12374552878385, + "max": 65.9145558875616, + "iqr": 1.4295717844751081, + "mean": 58.94066728460008, + "cv": 0.06756435235287546 + }, + "median_step_ms_by_index": [ + 130.78458298696205, + 125.49579198821448, + 127.27412499953061, + 127.79349999618717, + 129.90983400959522, + 134.33441700180992, + 133.95633299660403, + 137.29216699721292, + 133.71787501091603, + 134.9893750011688, + 136.78654200339224, + 135.3690830001142, + 136.68791600503027, + 136.6718750068685, + 137.12645799387246, + 136.6390840121312, + 136.34974999877159, + 138.25379199988674, + 139.08045800053515, + 137.22400000551715, + 136.10283299931325, + 136.19416700385045, + 136.24895899556577, + 137.1962910052389, + 137.16808400931768, + 137.0892920094775, + 137.49633300176356, + 136.4052089920733, + 138.29466699098703, + 137.01212500745896, + 137.35150000138674, + 137.3034170101164, + 138.13162500446197, + 139.34554198931437, + 137.5189170066733, + 139.43100000324193, + 138.7782910023816, + 137.15879200026393, + 138.73958299518563, + 138.73099999909755, + 139.2621249979129, + 139.2547079885844, + 140.85245800379198, + 139.76995900156908, + 139.36629200179596, + 139.99841699842364, + 139.3485840089852, + 141.01325000228826, + 142.1595419960795, + 141.6542910010321, + 140.3105829958804, + 141.11016600509174, + 140.1125420088647, + 142.64420900144614, + 141.09425000788178, + 141.51491700613406, + 142.95491699886043, + 141.71904200338759, + 141.87616600247566, + 140.02454200817738, + 142.03554100822657, + 143.35575001314282, + 143.94254201033618, + 143.12395799788646 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.1183396250125952, + 0.12366966699482873, + 0.1189581249927869, + 0.11463783300132491, + 0.11468099999183323, + 0.11428416699345689, + 0.13395633299660403, + 0.11459245899459347, + 0.1145470000046771, + 0.11447591701289639, + 0.11735508299898356, + 0.11838812500354834, + 0.11483437499555293, + 0.12261337498784997, + 0.11452116700820625, + 0.11466716600989457, + 0.11314333298651036, + 0.11595454199414235, + 0.11563320801360533, + 0.11497870899620466, + 0.11588241599383764, + 0.11535016600100789, + 0.11834983401058707, + 0.11594570800662041, + 0.11501216700708028, + 0.11589954199735075, + 0.11594933300511912, + 0.11623849999159575, + 0.13780766699346714, + 0.1192675830097869, + 0.11689487499825191, + 0.11679900001036003, + 0.11741679100668989, + 0.11849370900017675, + 0.12410329199337866, + 0.11998649999441113, + 0.12084529099229258, + 0.11973583399958443, + 0.11945412500062957, + 0.1198812079965137, + 0.11884745900169946, + 0.11917187500512227, + 0.12133070900745224, + 0.12038395900162868, + 0.12210654199589044, + 0.1224426249973476, + 0.11991066699556541, + 0.13018550000560936, + 0.12288570900273044, + 0.1416542910010321, + 0.12206470800447278, + 0.12199083299492486, + 0.12266875000204891, + 0.12201820898917504, + 0.12458508300187532, + 0.13749958299740683, + 0.12352774999453686, + 0.12404629201046191, + 0.12555054199765436, + 0.13344108300225344, + 0.12670995799999218, + 0.1262750000023516, + 0.14512854200438596, + 0.1496613749914104 + ], + [ + 0.13078458298696205, + 0.12549579198821448, + 0.1429153339995537, + 0.14365554199321195, + 0.14336133300093934, + 0.13929945799463894, + 0.14107854099711403, + 0.1405305420048535, + 0.14137149999442045, + 0.14692975000070874, + 0.13678654200339224, + 0.16078016700339504, + 0.14979537499311846, + 0.13913874998979736, + 0.14026591699803248, + 0.13937466600327753, + 0.13848979100293946, + 0.13825379199988674, + 0.13806870800908655, + 0.14036954200128093, + 0.13730862499505747, + 0.13959274999797344, + 0.1381233330030227, + 0.16166937501111533, + 0.1389571670006262, + 0.1369466670003021, + 0.14314716699300334, + 0.1387745419924613, + 0.1406219170021359, + 0.13853470799222123, + 0.13913420800236054, + 0.13966220799193252, + 0.1381691669957945, + 0.1391741249972256, + 0.1393701670021983, + 0.16271341699757613, + 0.1402900840039365, + 0.1455563330091536, + 0.14031049999175593, + 0.1403399580012774, + 0.1392621249979129, + 0.14479258299979847, + 0.14370716700796038, + 0.14180679200217128, + 0.13976800000818912, + 0.1418673750013113, + 0.16428537499450613, + 0.1397767919988837, + 0.1421595419960795, + 0.14306958399538416, + 0.14076537499204278, + 0.14173954099533148, + 0.14209945900074672, + 0.14264420900144614, + 0.14109425000788178, + 0.14591337500314694, + 0.1475090000021737, + 0.14310299999488052, + 0.1443760830006795, + 0.14299258300161455, + 0.1443292079929961, + 0.14335575001314282, + 0.14394254201033618, + 0.16319658400607295 + ], + [ + 0.15447508300712798, + 0.13193249999312684, + 0.1296830830106046, + 0.12779349999618717, + 0.12930499999492895, + 0.13433441700180992, + 0.1326455829985207, + 0.13729216699721292, + 0.13371787501091603, + 0.1349893750011688, + 0.13631400000303984, + 0.1353690830001142, + 0.13699137499497738, + 0.13529299999936484, + 0.16159645799780264, + 0.1366390840121312, + 0.13634974999877159, + 0.1397156669991091, + 0.13908045800053515, + 0.1385431249946123, + 0.13610283299931325, + 0.13701420799770858, + 0.13765329100715462, + 0.13754983400576748, + 0.13751575000060257, + 0.14022995899722446, + 0.13749633300176356, + 0.13908958299725782, + 0.13829466699098703, + 0.13796320800611284, + 0.13855470799899194, + 0.16020391600613948, + 0.14377974999661092, + 0.13934554198931437, + 0.14003591699292883, + 0.13947858300525695, + 0.1387782910023816, + 0.1392097920033848, + 0.14029937499435619, + 0.14115983399096876, + 0.14173158298945054, + 0.14588516700314358, + 0.14085245800379198, + 0.13976995900156908, + 0.14100991599843837, + 0.14127045900386292, + 0.1456429170066258, + 0.14142812498903368, + 0.16322937500081025, + 0.14214424999954645, + 0.14247312500083353, + 0.14111016600509174, + 0.14276654199056793, + 0.1432642500003567, + 0.1433337080088677, + 0.14151491700613406, + 0.14295491699886043, + 0.14586366699950304, + 0.14187616600247566, + 0.14234050000959542, + 0.14710770899546333, + 0.14366245799465105, + 0.1430066250031814, + 0.14312395799788646 + ], + [ + 0.12444670900003985, + 0.12795949999417644, + 0.1272741249995306, + 0.12662362499395385, + 0.13474749999295454, + 0.13606279098894447, + 0.13632483399123885, + 0.1376701670087641, + 0.1358676249947166, + 0.13688787499268074, + 0.13727195899991784, + 0.13950199999089818, + 0.13668791600503027, + 0.13987383400672115, + 0.1368357909959741, + 0.1372398340026848, + 0.15281204199709464, + 0.1394247499993071, + 0.13916083400545176, + 0.13722400000551715, + 0.14348883299680892, + 0.13619416700385045, + 0.13624895899556577, + 0.1371962910052389, + 0.13716808400931768, + 0.13818187499418855, + 0.13845941598992795, + 0.1364052089920733, + 0.14129108299675863, + 0.13701212500745896, + 0.13735150000138674, + 0.1373034170101164, + 0.13813162500446197, + 0.1635132500086911, + 0.1375189170066733, + 0.13943100000324193, + 0.13931929199316073, + 0.13715879200026393, + 0.13873958299518563, + 0.1383915829937905, + 0.14074420800898224, + 0.13925470798858441, + 0.14208270800008904, + 0.1398837500018999, + 0.13936629200179596, + 0.13999841699842364, + 0.13889191699854564, + 0.1425816249975469, + 0.16317508299835026, + 0.1397233339957893, + 0.1403105829958804, + 0.14191454200772569, + 0.1401125420088647, + 0.14009537499805447, + 0.1519204170035664, + 0.14738179100095294, + 0.14386054100759793, + 0.14171904200338759, + 0.14231958299933467, + 0.13994854099291842, + 0.14203554100822657, + 0.14324837499589194, + 0.14271508398815058, + 0.14051087498955894 + ], + [ + 0.13756429200293496, + 0.11571458300750237, + 0.12065895799605642, + 0.13107804099854548, + 0.12990983400959522, + 0.13020579199655913, + 0.13068987500446383, + 0.13018579200434033, + 0.12799983299919404, + 0.13385862500581425, + 0.13759300000674557, + 0.13488808300462551, + 0.13550599999143742, + 0.1366718750068685, + 0.13712645799387246, + 0.1363494580000406, + 0.13521449999825563, + 0.1378909999912139, + 0.161241166992113, + 0.13610958300705533, + 0.13582324999151751, + 0.13614920900727157, + 0.13572116701107007, + 0.13516570800857153, + 0.13559637501020916, + 0.1370892920094775, + 0.1355612920015119, + 0.13588100000924896, + 0.1353515829978278, + 0.13533804198959842, + 0.13544516600086354, + 0.13594279199605808, + 0.13752795799518935, + 0.14617966700461693, + 0.13626270899840165, + 0.13706858300429303, + 0.13631358300335705, + 0.13645437499508262, + 0.13668162499379832, + 0.13873099999909755, + 0.1365985830052523, + 0.13663045899011195, + 0.13587237500178162, + 0.13717858400195837, + 0.1374418329942273, + 0.1364441670011729, + 0.1393485840089852, + 0.14101325000228826, + 0.13744500000029802, + 0.1397265829873504, + 0.1383014169987291, + 0.13754095800686628, + 0.1364599999942584, + 0.15833758299413603, + 0.14037600001029205, + 0.13754462500219233, + 0.1382813329983037, + 0.13923991699994076, + 0.14064683399919886, + 0.14002454200817738, + 0.14163533400278538, + 0.14427383299334906, + 0.14592604199424386, + 0.14205487500294112 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_mha": { + "first_token_ms": { + "count": 5, + "median": 353.6117499897955, + "p95": 410.3735744021833, + "min": 344.9034999939613, + "max": 417.7005410019774, + "iqr": 32.0597079989966, + "mean": 369.25749979855027, + "cv": 0.08274447505947058 + }, + "decode_ms_per_step": { + "count": 315, + "median": 371.8506669974886, + "p95": 441.79702500114223, + "min": 305.4902080039028, + "max": 548.7232500017853, + "iqr": 35.5685204922338, + "mean": 376.0721083114197, + "cv": 0.09349696813074429 + }, + "total_ms": { + "count": 5, + "median": 24046.6801180155, + "p95": 25813.10715074942, + "min": 21734.459626095486, + "max": 26046.42100095225, + "iqr": 1278.2626278494718, + "mean": 24061.800323417992, + "cv": 0.06648396161664173 + }, + "tokens_per_second": { + "count": 5, + "median": 21.291920443371946, + "p95": 23.184340025533224, + "min": 19.65721125298871, + "max": 23.5570614042443, + "iqr": 1.114553754124966, + "mean": 21.355709673571564, + "cv": 0.06806855114169981 + }, + "median_step_ms_by_index": [ + 353.6117499897955, + 364.09187500248663, + 349.6400000003632, + 348.96962500351947, + 353.0976659967564, + 344.4885419885395, + 340.07658300106414, + 362.64916601066943, + 340.06020800734404, + 347.135290998267, + 347.8472090064315, + 352.5480000098469, + 360.3600839996943, + 362.3595000099158, + 367.0840830018278, + 368.82195800717454, + 355.93366700049955, + 355.70937499869615, + 360.96295800234657, + 364.0986660029739, + 362.5624590058578, + 366.6587920015445, + 361.77433299599215, + 368.33241701242514, + 368.53741700178944, + 366.61187501158565, + 370.564333992661, + 385.75258299533743, + 364.44295800174586, + 367.0849170011934, + 366.8859579920536, + 374.5277919952059, + 371.8506669974886, + 391.2202499923296, + 373.2790829963051, + 374.7327500022948, + 372.22708300396334, + 381.13083300413564, + 380.22966599965, + 383.86329100467265, + 381.50104200758506, + 377.4515830009477, + 388.0732499965234, + 379.9984590004897, + 377.000708002015, + 396.01379199302755, + 389.9070839979686, + 383.00229198648594, + 376.903583004605, + 374.8455839959206, + 382.75141699705273, + 386.6064159956295, + 391.2243749946356, + 389.9465419963235, + 394.8306250094902, + 393.1192500022007, + 390.7562090025749, + 394.98158301285, + 402.98737499688286, + 394.78520800184924, + 387.63599999947473, + 396.52395799930673, + 400.27199999894947, + 414.2231250007171 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.3490060000040103, + 0.32170050000422634, + 0.3110282499983441, + 0.30765275000885595, + 0.3065171670023119, + 0.31039300000702497, + 0.3054902080039028, + 0.3102307500084862, + 0.32560283299244475, + 0.3120739590085577, + 0.3106960419972893, + 0.3157157079986064, + 0.31670708299498074, + 0.33101383299799636, + 0.3176393750036368, + 0.3162560839991784, + 0.3129107499989914, + 0.3206454999890411, + 0.3326130000059493, + 0.32220116700045764, + 0.32114270800957456, + 0.32442004200129304, + 0.3220237080095103, + 0.3390041249949718, + 0.3212244160094997, + 0.32487749999563675, + 0.32782837499689776, + 0.33359399999608286, + 0.32973029201093595, + 0.3670849170011934, + 0.33319787500659004, + 0.33277325000381097, + 0.32950566700310446, + 0.33697583300818224, + 0.3554381249996368, + 0.3396911669988185, + 0.3383423750055954, + 0.3394006669986993, + 0.34179791700444184, + 0.3590163750050124, + 0.3401644169935025, + 0.34463575000700075, + 0.3497842920041876, + 0.3625470410042908, + 0.34697062498889863, + 0.3482540419936413, + 0.34618974999466445, + 0.3515253340010531, + 0.36279841599753127, + 0.351841499999864, + 0.35285354100051336, + 0.3542347089969553, + 0.3540674579999177, + 0.37636449999990873, + 0.35712600000260863, + 0.3584189170069294, + 0.3617245829955209, + 0.3948490830080118, + 0.3685806249995949, + 0.3679328330035787, + 0.36512558300455566, + 0.36682704200211447, + 0.39039416699961293, + 0.3880861250072485 + ], + [ + 0.3449034999939613, + 0.35981579100189265, + 0.35559820900380146, + 0.35401420899142977, + 0.3530976659967564, + 0.3581195409933571, + 0.35161337499448564, + 0.37512937499559484, + 0.34955291700316593, + 0.35481041700404603, + 0.35607612499734387, + 0.37421891700068954, + 0.360542374997749, + 0.47764220800308976, + 0.44081212500168476, + 0.4885044580005342, + 0.5487232500017853, + 0.35039929099730216, + 0.36096295800234657, + 0.4511101249954663, + 0.3625624590058578, + 0.3683922500058543, + 0.36871683300705627, + 0.3697894159995485, + 0.38617008400615305, + 0.3930544169998029, + 0.370564333992661, + 0.43328004199429415, + 0.4623277499922551, + 0.420913207999547, + 0.37647566699888557, + 0.40608616700046696, + 0.44491374999051914, + 0.4037840410019271, + 0.44806554200476967, + 0.4062035829992965, + 0.40589579098741524, + 0.4086666249932023, + 0.40254845800518524, + 0.4491885000024922, + 0.39966795800137334, + 0.43885304099239875, + 0.41069608301040716, + 0.40611408300173935, + 0.4090296249923995, + 0.4050310840102611, + 0.42203599998902064, + 0.3897952089901082, + 0.4184604590118397, + 0.42525016599392984, + 0.42761650000466034, + 0.4485941669991007, + 0.43091745900164824, + 0.44409512499987613, + 0.4272178339888342, + 0.4212471659993753, + 0.44965112500358373, + 0.4271418340067612, + 0.42303970899956767, + 0.4262587499979418, + 0.48215100000379607, + 0.41696224999031983, + 0.4291254999989178, + 0.4142231250007171 + ], + [ + 0.4177005410019774, + 0.3714943749946542, + 0.3478159579972271, + 0.3489266660035355, + 0.3554476249992149, + 0.3444885419885395, + 0.3399380420014495, + 0.3643746660090983, + 0.3386317909898935, + 0.3454150409961585, + 0.3435126249969471, + 0.3525480000098469, + 0.3603600839996943, + 0.3623595000099158, + 0.3800686670001596, + 0.36882195800717454, + 0.35593366700049955, + 0.360928750000312, + 0.36469862500962336, + 0.3640986660029739, + 0.38615129100799095, + 0.36654899999848567, + 0.36177433299599215, + 0.36833241701242514, + 0.35954620799748227, + 0.36661187501158565, + 0.38610800000606105, + 0.36565208400133997, + 0.36444295800174586, + 0.365638500006753, + 0.3668859579920536, + 0.37697683299484197, + 0.37699849999626167, + 0.4009996249951655, + 0.3732790829963051, + 0.3747327500022948, + 0.37222708300396334, + 0.38113083300413564, + 0.38022966599965, + 0.398309625001275, + 0.38063962500018533, + 0.3774515830009477, + 0.3907054580049589, + 0.3799984590004897, + 0.377000708002015, + 0.4174031250004191, + 0.38135933299781755, + 0.38300229198648594, + 0.376903583004605, + 0.37862295800005086, + 0.3838867919985205, + 0.3991013339982601, + 0.38943620899226516, + 0.38855845799844246, + 0.3866343749978114, + 0.38904083298984915, + 0.3863890829961747, + 0.4033525420090882, + 0.40027529100188985, + 0.388346082996577, + 0.38763599999947473, + 0.38988754199817777, + 0.40027199999894947, + 0.4306360409973422 + ], + [ + 0.3810657080030069, + 0.38872583300690167, + 0.3496400000003632, + 0.34896962500351947, + 0.33950745800393634, + 0.34012229200743604, + 0.34007658300106414, + 0.36264916601066943, + 0.34006020800734404, + 0.347135290998267, + 0.3478472090064315, + 0.34713700000429526, + 0.34435125000891276, + 0.34686425000836607, + 0.36185358300281223, + 0.35640083299949765, + 0.35296712499985006, + 0.35570937499869615, + 0.3577696249994915, + 0.3579788329952862, + 0.35616712500632275, + 0.3767051250033546, + 0.3547214999998687, + 0.3535510410001734, + 0.36853741700178944, + 0.36388487499789335, + 0.36925783299375325, + 0.39974000000802334, + 0.35948079099762253, + 0.3611782910011243, + 0.3612359170074342, + 0.35992587500368245, + 0.357788833003724, + 0.3829849159956211, + 0.3606664999970235, + 0.3629653340030927, + 0.3631412919930881, + 0.3643108329997631, + 0.37216645800799597, + 0.3630737079947721, + 0.38150104200758506, + 0.3693301250023069, + 0.3686219999945024, + 0.36732033400039654, + 0.36532512500707526, + 0.3678422090015374, + 0.39070287501090206, + 0.3732608749996871, + 0.3669837089983048, + 0.3695787909964565, + 0.3704159589979099, + 0.38645700000051875, + 0.3912243749946356, + 0.3899465419963235, + 0.39850316698721144, + 0.3931192500022007, + 0.3907562090025749, + 0.3884322920057457, + 0.40478704099950846, + 0.39568512499681674, + 0.38709074999496806, + 0.39652395799930673, + 0.39465295799891464, + 0.4152125000109663 + ], + [ + 0.3536117499897955, + 0.36409187500248663, + 0.3596657919988502, + 0.3605292080028448, + 0.3575331250031013, + 0.3771933340030955, + 0.36319849999563303, + 0.3594142920046579, + 0.35306358299567364, + 0.3664296250062762, + 0.3608782500086818, + 0.35992249999253545, + 0.37594820899539627, + 0.37467220801045187, + 0.3670840830018278, + 0.39931820800120477, + 0.4925315829896135, + 0.4577139999892097, + 0.5145214579970343, + 0.41030212500481866, + 0.4556689169985475, + 0.3666587920015445, + 0.38168437499552965, + 0.3812889169930713, + 0.3741135409945855, + 0.38281516599818133, + 0.38111191701318603, + 0.38575258299533743, + 0.38331658400420565, + 0.3720630839961814, + 0.38555291599186603, + 0.3745277919952059, + 0.3718506669974886, + 0.3912202499923296, + 0.38083266699686646, + 0.39912358300352935, + 0.3849877079919679, + 0.38127695900038816, + 0.38736475000041537, + 0.38386329100467265, + 0.38829858299868647, + 0.3845837919943733, + 0.3880732499965234, + 0.38333637500181794, + 0.3859240419988055, + 0.39601379199302755, + 0.3899070839979686, + 0.39652308299264405, + 0.408257290997426, + 0.3748455839959206, + 0.38275141699705273, + 0.3866064159956295, + 0.39127945799555164, + 0.40121325000654906, + 0.3948306250094902, + 0.39960970799438655, + 0.3912119580054423, + 0.39498158301285, + 0.40298737499688286, + 0.39478520800184924, + 0.38887029100442305, + 0.41010437499789987, + 0.402290958998492, + 0.4098680839961162 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "speedup_cached_vs_uncached_sdpa": 23.337946987460555, + "speedup_cached_vs_uncached_mha": 62.82889965404949 + } + ], + "window": { + "batch": 1, + "prompt_len": 32, + "context_length": 128, + "new_tokens": 224, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_mha": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 0.7685000018682331, + "p95": 0.791575395851396, + "min": 0.7455419981852174, + "max": 0.795208994532004, + "iqr": 0.0270830059889704, + "mean": 0.7672499981708825, + "cv": 0.026466333502472864 + }, + "decode_ms_per_step": { + "count": 1115, + "median": 1.4943339920137078, + "p95": 1.6392791963880882, + "min": 0.3577089955797419, + "max": 6.95116599672474, + "iqr": 1.1491250043036416, + "mean": 1.0607323956841836, + "cv": 0.6083223164505704 + }, + "total_ms": { + "count": 5, + "median": 231.61804297706112, + "p95": 249.2226830625441, + "min": 230.9325360110961, + "max": 251.1378140479792, + "iqr": 10.259840099024586, + "mean": 237.31057423574384, + "cv": 0.037598876677814634 + }, + "tokens_per_second": { + "count": 5, + "median": 967.1094579716504, + "p95": 969.6701127923583, + "min": 891.9405500487688, + "max": 969.9802542732091, + "iqr": 41.131989936707896, + "mean": 944.951473218966, + "cv": 0.03662257469184692 + }, + "median_step_ms_by_index": [ + 0.7685000018682331, + 0.4507089906837791, + 0.4020000051241368, + 0.3802499995799735, + 0.3772089985432103, + 0.37162500666454434, + 0.37304199940990657, + 0.37408299976959825, + 0.3741669934242964, + 0.36816699139308184, + 0.3612909931689501, + 0.3676249907584861, + 0.37104199873283505, + 0.37187499401625246, + 0.3660829970613122, + 0.36779200308956206, + 0.36979198921471834, + 0.3682500100694597, + 0.3676670021377504, + 0.36675001319963485, + 0.36383299448061734, + 0.36700000055134296, + 0.3695839986903593, + 0.3732080076588318, + 0.37349999183788896, + 0.36833299964200705, + 0.36912500218022615, + 0.37208299909252673, + 0.3684999974211678, + 0.3700829984154552, + 0.3744160057976842, + 0.3685830015456304, + 0.36874999932479113, + 0.3712920006364584, + 0.3699590015457943, + 0.37108299147803336, + 0.3683749964693561, + 0.3689159930218011, + 0.37149999116081744, + 0.37316700036171824, + 0.3726249997271225, + 0.3706659917952493, + 0.3721659886650741, + 0.3713749902090058, + 0.37199999496806413, + 0.37120799242984504, + 0.370624999050051, + 0.37891700048930943, + 0.3834590024780482, + 0.37437499850057065, + 0.37645800330210477, + 0.3720420063473284, + 0.37466699723154306, + 0.37370799691416323, + 0.37479199818335474, + 0.3778340033022687, + 0.3790829941863194, + 0.3748749877559021, + 0.37733299541287124, + 0.3766669979086146, + 0.3759579994948581, + 0.3776660014409572, + 0.37637499917764217, + 0.37487500230781734, + 0.37820899160578847, + 0.3787500027101487, + 0.3843329905066639, + 0.3750420000869781, + 0.3776669909711927, + 0.37879199953749776, + 0.38183300057426095, + 0.378249998902902, + 0.38245899486355484, + 0.37616600457113236, + 0.3793749929172918, + 0.3795829979935661, + 0.38079101068433374, + 0.38020800275262445, + 0.37954100116621703, + 0.38425000093411654, + 0.3814590018009767, + 0.3842080041067675, + 0.38337499427143484, + 0.3795829979935661, + 0.38174999644979835, + 0.3828750050161034, + 0.37741599953733385, + 0.3819999983534217, + 0.38133299676701427, + 0.37970799894537777, + 0.38649998896289617, + 0.38558300002478063, + 0.3802919964073226, + 0.385665989597328, + 0.3818339901044965, + 0.3817079996224493, + 0.39766699774190784, + 1.5454999957000837, + 1.5135419962462038, + 1.516750009614043, + 1.5231249999487773, + 1.5198750043055043, + 1.5205420058919117, + 1.5143749915296212, + 1.518500008387491, + 1.5082500030985102, + 1.5303750114981085, + 1.5761669928906485, + 1.5481250011362135, + 1.5205000090645626, + 1.5129590028664097, + 1.5125830104807392, + 1.5091249952092767, + 1.5087079955264926, + 1.5038340061437339, + 1.5058329881867394, + 1.5050000074552372, + 1.5029579953989014, + 1.5028329944470897, + 1.500458994996734, + 1.4980409905547276, + 1.4977910032030195, + 1.5025409957161173, + 1.5145409997785464, + 1.4979999978095293, + 1.5151250117924064, + 1.5080000011948869, + 1.5085000050021335, + 1.5042909944895655, + 1.5051249938551337, + 1.5071250090841204, + 1.5025409957161173, + 1.497917008236982, + 1.505749998614192, + 1.5009159978944808, + 1.5040829894132912, + 1.5081250021466985, + 1.4990420022513717, + 1.4965419977670535, + 1.5631659916834906, + 1.5251250006258488, + 1.5189999976428226, + 1.4998749975347891, + 1.501791994087398, + 1.51966699922923, + 1.5044579922687262, + 1.4943339920137078, + 1.5028329944470897, + 1.4968329924158752, + 1.5016249963082373, + 1.5302910032914951, + 1.5194160077953711, + 1.5076249983394518, + 1.503250008681789, + 1.4903330011293292, + 1.5314580086851493, + 1.4999999984866008, + 1.5042919985717162, + 1.5005829918663949, + 1.5058749995660037, + 1.4989580085966736, + 1.552041998365894, + 1.505332998931408, + 1.5005409950390458, + 1.5010829956736416, + 1.5219169872580096, + 1.574458001414314, + 1.5405419981107116, + 1.5037080011097714, + 1.5092079993337393, + 1.511250011390075, + 1.5186670061666518, + 1.5464169991901144, + 1.5657500043744221, + 1.5107499930309132, + 1.5052079979795963, + 1.5002909931354225, + 1.5282910026144236, + 1.536042007501237, + 1.5822499990463257, + 1.5118750015972182, + 1.5176249871728942, + 1.5293329925043508, + 1.5119589952519163, + 1.5159169997787103, + 1.5145409997785464, + 1.5851659991312772, + 1.5742910036351532, + 1.6072910075308755, + 1.5743749972898513, + 1.5635410090908408, + 1.5564999921480194, + 1.5625409869244322, + 1.5558750019408762, + 1.5611249982612208, + 1.5631249989382923, + 1.56095800048206, + 1.5842499997233972, + 1.5647499967599288, + 1.5836659877095371, + 1.6003749915398657, + 1.5728330035926774, + 1.5592080017086118, + 1.5622500068275258, + 1.5743749972898513, + 1.583250006660819, + 1.5572499978588894, + 1.5574580029351637, + 1.5591670089634135, + 1.561167009640485, + 1.5585410001222044, + 1.5650419954909012, + 1.5534999984083697, + 1.5545829955954105, + 1.5591659903293476, + 1.5706660051364452, + 1.5628330002073199, + 1.569791988003999, + 1.5591659903293476, + 1.5620839985786006, + 1.5706660051364452, + 1.5710000006947666, + 1.5660829958505929, + 1.5651660069124773 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0007685000018682331, + 0.0004507089906837791, + 0.0003900419978890568, + 0.0003787080058827996, + 0.00037720899854321033, + 0.0003711669996846467, + 0.0004271669895388186, + 0.00038987500010989606, + 0.000519416993483901, + 0.0004347909998614341, + 0.0004200420080451295, + 0.0003909579972969368, + 0.000374625000404194, + 0.00037245800194796175, + 0.00037237499782349914, + 0.00037141698703635484, + 0.00036791700404137373, + 0.0003674580075312406, + 0.0003640420036390424, + 0.00036950000503566116, + 0.00036383299448061734, + 0.00045116701221559197, + 0.00038712500827386975, + 0.0003787500027101487, + 0.00037474998680409044, + 0.0003689580044010654, + 0.000367375003406778, + 0.0003679999936139211, + 0.00036779200308956206, + 0.00036637501034419984, + 0.0003744160057976842, + 0.0003685830015456304, + 0.00036745799297932535, + 0.0003712920006364584, + 0.0003699590015457943, + 0.00037108299147803336, + 0.0003683749964693561, + 0.0003689159930218011, + 0.0003688340075314045, + 0.0003680829977383837, + 0.00037220900412648916, + 0.0003700420056702569, + 0.0003724590060301125, + 0.0003713749902090058, + 0.0003738329978659749, + 0.00037045800127089024, + 0.0003690830053528771, + 0.0003826669999398291, + 0.0003862919984385371, + 0.00039149999793153256, + 0.0003915410052286461, + 0.00038229099300224334, + 0.0003776660014409572, + 0.00037154200254008174, + 0.00037479199818335474, + 0.0003778340033022687, + 0.00038062500243540853, + 0.00037075000000186265, + 0.0003744999994523823, + 0.000374625000404194, + 0.000374249997548759, + 0.00038145799771882594, + 0.0003827089967671782, + 0.0003759579994948581, + 0.00037820899160578847, + 0.0004280419962015003, + 0.00040550000267103314, + 0.0003750420000869781, + 0.00037937500746920705, + 0.00038124999264255166, + 0.00038183300057426095, + 0.000378249998902902, + 0.0003929999948013574, + 0.0003830420027952641, + 0.0003793749929172918, + 0.0003795829979935661, + 0.00038195899105630815, + 0.00037991600402165204, + 0.0003780410042963922, + 0.000386583007639274, + 0.0003814590018009767, + 0.0003885000041918829, + 0.00037650000012945384, + 0.0003807909961324185, + 0.0003935409913538024, + 0.00039254099829122424, + 0.00037741599953733385, + 0.0003823750012088567, + 0.0003839590062852949, + 0.0003766250010812655, + 0.0003735409991350025, + 0.00038558300002478063, + 0.00037791699287481606, + 0.00038166700687725097, + 0.0003816250100499019, + 0.0003766669979086146, + 0.0004000409971922636, + 0.0015473749954253435, + 0.0015172909916145727, + 0.0015548749943263829, + 0.0015231249999487773, + 0.001508041998022236, + 0.0015205420058919117, + 0.0015080830053193495, + 0.0015038749988889322, + 0.0015082500030985102, + 0.001512417002231814, + 0.0015068329958012328, + 0.0015703749959357083, + 0.0015205000090645626, + 0.0015129590028664097, + 0.0015117919974727556, + 0.0014947500021662563, + 0.0015032499941298738, + 0.0014989579940447584, + 0.0015058329881867394, + 0.0015014170057838783, + 0.0015029579953989014, + 0.0014932920021237805, + 0.0014866249985061586, + 0.0014999169943621382, + 0.001502125000115484, + 0.0014901660033501685, + 0.001517374999821186, + 0.0014891669998178259, + 0.0015384589933091775, + 0.0015080000011948869, + 0.0015218750049825758, + 0.0014955829974496737, + 0.0015005000022938475, + 0.0014972499920986593, + 0.0014957079984014854, + 0.001497917008236982, + 0.001533333008410409, + 0.0014966669987188652, + 0.0016522499936399981, + 0.0015161659976001829, + 0.0015104170015547425, + 0.0015035419928608462, + 0.0016858750022947788, + 0.001859958007116802, + 0.0015525000053457916, + 0.0015119580057216808, + 0.0016602499963482842, + 0.00151966699922923, + 0.0015137910086195916, + 0.0015167079982347786, + 0.0015107920044101775, + 0.0015231669967761263, + 0.001521125013823621, + 0.0015842499997233972, + 0.0015277499915100634, + 0.0015092499961610883, + 0.001503250008681789, + 0.001629374994081445, + 0.0015314580086851493, + 0.0015120000025490299, + 0.001591250009369105, + 0.001511666996520944, + 0.0015479590074392036, + 0.0015454160020453855, + 0.0015062910097185522, + 0.0015019160055089742, + 0.0014991250063758343, + 0.0015010829956736416, + 0.0015219169872580096, + 0.0015050420042825863, + 0.0015012499934528023, + 0.0015037080011097714, + 0.0015010839997557923, + 0.0015145420038606972, + 0.0015047080087242648, + 0.0014976669917814434, + 0.0015553330013062805, + 0.0015084580081747845, + 0.0015043330058688298, + 0.0014993750082794577, + 0.0014947500021662563, + 0.0015012079966254532, + 0.0015035830001579598, + 0.00150079200102482, + 0.0015148749953368679, + 0.0015123329940252006, + 0.0015119589952519163, + 0.001504083993495442, + 0.0014951670018490404, + 0.0015963329933583736, + 0.0015135419962462038, + 0.0015087089996086434, + 0.0015030000067781657, + 0.0015023750020191073, + 0.001501332997577265, + 0.0015164580108830705, + 0.0015289170114556327, + 0.0014819579955656081, + 0.001504416999523528, + 0.0015012919902801514, + 0.0015086670027812943, + 0.0015062079910421744, + 0.001501332997577265, + 0.0016003749915398657, + 0.0015181670023594052, + 0.0015055419935379177, + 0.0015035840042401105, + 0.0018985830101883039, + 0.0016377079882659018, + 0.0015204170049401, + 0.0015159589966060594, + 0.0015501669986406341, + 0.0016171249881153926, + 0.0015187500102911144, + 0.00151012500282377, + 0.0015365419967565686, + 0.0015422919968841597, + 0.0015448329941136762, + 0.0015419160044984892, + 0.0015385409933514893, + 0.0015047499909996986, + 0.0015198750043055043, + 0.0015272499877028167, + 0.0016253339999821037, + 0.0015096669958438724, + 0.001511291993665509, + 0.0015311660099541768 + ], + [ + 0.0007455419981852174, + 0.00044220799463801086, + 0.0004020000051241368, + 0.00038291700184345245, + 0.00037525000516325235, + 0.00037162500666454434, + 0.0003683749964693561, + 0.00038991699693724513, + 0.00037475000135600567, + 0.00036816699139308184, + 0.00036145800550002605, + 0.0003676249907584861, + 0.00038020800275262445, + 0.00037208301364444196, + 0.0003660829970613122, + 0.00036779200308956206, + 0.00036979198921471834, + 0.0003727919975062832, + 0.00036962500598747283, + 0.00036675001319963485, + 0.0003677080094348639, + 0.00036700000055134296, + 0.0003693750040838495, + 0.0003701659879880026, + 0.00037070900725666434, + 0.0003654159954749048, + 0.0003692089958349243, + 0.000368666005670093, + 0.00036854100471828133, + 0.0003688750002766028, + 0.0003772919881157577, + 0.0003699999942909926, + 0.00036874999932479113, + 0.0003775410004891455, + 0.00036950000503566116, + 0.00037687500298488885, + 0.00036895899393130094, + 0.00036874999932479113, + 0.00037149999116081744, + 0.0003751250042114407, + 0.0003697909996844828, + 0.00036991700471844524, + 0.00036895900848321617, + 0.000367207991075702, + 0.00037641701055690646, + 0.00037162500666454434, + 0.0003693750040838495, + 0.00037891700048930943, + 0.0003748339950107038, + 0.00037437499850057065, + 0.00037525000516325235, + 0.0003720420063473284, + 0.0003727080038515851, + 0.00037370799691416323, + 0.0003770829935092479, + 0.0003828329936368391, + 0.0003765419969568029, + 0.00038508299621753395, + 0.00039433399797417223, + 0.00038741699245292693, + 0.0003777499950956553, + 0.00038174999644979835, + 0.00037775000964757055, + 0.00037487500230781734, + 0.0003996250015916303, + 0.0003798340039793402, + 0.0003843329905066639, + 0.00037900000461377203, + 0.0003799999976763502, + 0.00037724999128840864, + 0.00038825000228825957, + 0.0003804159932769835, + 0.000385290986741893, + 0.00037616600457113236, + 0.0003812079958152026, + 0.00038345799839589745, + 0.00038408400723710656, + 0.0003851670044241473, + 0.00038454199966508895, + 0.00037799999699927866, + 0.00039500001003034413, + 0.0003842080041067675, + 0.0003894170076819137, + 0.000378249998902902, + 0.0003860829892801121, + 0.00038341700565069914, + 0.00037679199886042625, + 0.0003792500065173954, + 0.00038133299676701427, + 0.00037970799894537777, + 0.0003869159991154447, + 0.0003896249982062727, + 0.0003841660072794184, + 0.000385665989597328, + 0.0003843750018859282, + 0.0003817079996224493, + 0.0003880420117639005, + 0.0015408340113935992, + 0.0015088750078575686, + 0.001516750009614043, + 0.001503040999523364, + 0.0015198750043055043, + 0.0016387499927077442, + 0.001531250003608875, + 0.001518500008387491, + 0.0015136250003706664, + 0.001887790989712812, + 0.0015979169984348118, + 0.0015481250011362135, + 0.001577584000187926, + 0.0016002909978851676, + 0.0015125830104807392, + 0.0015091249952092767, + 0.0015235410101013258, + 0.0015816670056665316, + 0.0015255000034812838, + 0.0015213339938782156, + 0.0015149999962886795, + 0.0015142920019570738, + 0.00153087500075344, + 0.0014980409905547276, + 0.0014977910032030195, + 0.001495499993325211, + 0.0014971250056987628, + 0.0014992079959483817, + 0.0016074169980129227, + 0.0015110840031411499, + 0.0015031659859232605, + 0.0015042909944895655, + 0.0015051249938551337, + 0.001513750001322478, + 0.001507041000877507, + 0.0014947910094633698, + 0.0015025830070953816, + 0.0015009159978944808, + 0.0015040829894132912, + 0.0015081250021466985, + 0.0014990420022513717, + 0.0014957079984014854, + 0.001575124988448806, + 0.0015251250006258488, + 0.0015152080013649538, + 0.0014998749975347891, + 0.0014940419932827353, + 0.0014944160066079348, + 0.0015038749988889322, + 0.001495415999670513, + 0.0015028329944470897, + 0.001509791996795684, + 0.0015016249963082373, + 0.0014969170006224886, + 0.0015048750065034255, + 0.001518958990345709, + 0.0015091669920366257, + 0.0015680830110795796, + 0.001545374994748272, + 0.0014948329917388037, + 0.0015042919985717162, + 0.0015005829918663949, + 0.0015058749995660037, + 0.0015010000061010942, + 0.00150079200102482, + 0.001505332998931408, + 0.0015005409950390458, + 0.00150154200673569, + 0.0014985409943619743, + 0.001574458001414314, + 0.0015835000085644424, + 0.00149983300070744, + 0.0015092079993337393, + 0.0015110420063138008, + 0.004766500002006069, + 0.00695116599672474, + 0.005962833005469292, + 0.00198358298803214, + 0.0047991659957915545, + 0.0018214159936178476, + 0.0018657500040717423, + 0.0018095830018864945, + 0.0017340000049443915, + 0.0016321250004693866, + 0.0015892920055193827, + 0.0015800420078448951, + 0.0015933750109979883, + 0.0015870829956838861, + 0.0015685840044170618, + 0.0015851659991312772, + 0.0015742910036351532, + 0.0016072910075308755, + 0.0015743749972898513, + 0.0015869999915594235, + 0.00157200000830926, + 0.0016701660060789436, + 0.0015995410067262128, + 0.0015682500088587403, + 0.0015631249989382923, + 0.0015693750028731301, + 0.0015842499997233972, + 0.0015790000034030527, + 0.0015805830043973401, + 0.0016247499879682437, + 0.0015728330035926774, + 0.0015694589965278283, + 0.0015731250023236498, + 0.0015743749972898513, + 0.0016053750005085021, + 0.0015705830010119826, + 0.0015574580029351637, + 0.0016287080070469528, + 0.00157816699356772, + 0.0015724169934401289, + 0.0015761250106152147, + 0.0015534999984083697, + 0.001567625004099682, + 0.0015849159972276539, + 0.0015706660051364452, + 0.0015628330002073199, + 0.001569791988003999, + 0.0015629159897798672, + 0.001570833002915606, + 0.0015706660051364452, + 0.0015710000006947666, + 0.0015660829958505929, + 0.0015746660064905882 + ], + [ + 0.0007770410011289641, + 0.0004698749980889261, + 0.00041566700383555144, + 0.0003980000037699938, + 0.000399791999370791, + 0.0003872499946737662, + 0.00037620899092871696, + 0.0003667499986477196, + 0.00036416700459085405, + 0.00036537498817779124, + 0.000360707999789156, + 0.0003701249952428043, + 0.00037104199873283505, + 0.000367750006262213, + 0.000364125007763505, + 0.00036433299828786403, + 0.00037308401078917086, + 0.0003682500100694597, + 0.0003676670021377504, + 0.0003677909990074113, + 0.00036691699642688036, + 0.00037266699655447155, + 0.0003695839986903593, + 0.0003732080076588318, + 0.0003693750040838495, + 0.00036833299964200705, + 0.00036912500218022615, + 0.00037208299909252673, + 0.0003668330027721822, + 0.0003700829984154552, + 0.00036954200186301023, + 0.0003656670014606789, + 0.00037254100607242435, + 0.0003684170078486204, + 0.0003703339898493141, + 0.0003697499923873693, + 0.00036754099710378796, + 0.0003706660063471645, + 0.00037475000135600567, + 0.00037316700036171824, + 0.0003726249997271225, + 0.0003775410004891455, + 0.00037416600389406085, + 0.0003733329940587282, + 0.00037199999496806413, + 0.00037120799242984504, + 0.00037741599953733385, + 0.00040920800529420376, + 0.0003834590024780482, + 0.0003713749902090058, + 0.00037645800330210477, + 0.000371000001905486, + 0.00037466699723154306, + 0.00037437499850057065, + 0.0003730830067070201, + 0.0003773749922402203, + 0.0003790829941863194, + 0.0003748749877559021, + 0.00037733299541287124, + 0.0003766669979086146, + 0.0003759579994948581, + 0.0003737500082934275, + 0.0003757920057978481, + 0.00037479199818335474, + 0.00038212499930523336, + 0.00037558299663942307, + 0.0003819999983534217, + 0.00037054100539535284, + 0.0003755000070668757, + 0.00039237500459421426, + 0.0003783330030273646, + 0.00037762499414384365, + 0.00037729200266767293, + 0.0003749589959625155, + 0.00037812499795109034, + 0.00037562500801868737, + 0.00038079101068433374, + 0.00038020800275262445, + 0.00037954100116621703, + 0.00038425000093411654, + 0.0003807909961324185, + 0.0003792080096900463, + 0.0003785840090131387, + 0.0003795829979935661, + 0.00038174999644979835, + 0.0003828750050161034, + 0.00037933299608994275, + 0.0003819999983534217, + 0.00038070899609010667, + 0.0003817079996224493, + 0.00037895800778642297, + 0.0003787500027101487, + 0.0003794579970417544, + 0.0004024170048069209, + 0.0004093329916941002, + 0.00038150000909809023, + 0.00039766699774190784, + 0.00153779199172277, + 0.0015135419962462038, + 0.001502541999798268, + 0.00199966698710341, + 0.0016066249954747036, + 0.0015319170051952824, + 0.0015559170133201405, + 0.0015365419967565686, + 0.0015305420092772692, + 0.001518709002994001, + 0.0015761669928906485, + 0.0015220420027617365, + 0.0015072079986566678, + 0.001512458999059163, + 0.0016067499964265153, + 0.0015910420042928308, + 0.001516375006758608, + 0.0015077920106705278, + 0.001505500011262484, + 0.0015376250084955245, + 0.001496624987339601, + 0.0015028329944470897, + 0.001500458994996734, + 0.0014928330056136474, + 0.0015007079928182065, + 0.0015107499930309132, + 0.001501791994087398, + 0.0014908330049365759, + 0.0014985410089138895, + 0.00150079200102482, + 0.0015085000050021335, + 0.0014937920059310272, + 0.0014970409974921495, + 0.0014957909879740328, + 0.0015025409957161173, + 0.001503040999523364, + 0.001499416001024656, + 0.0015915830008452758, + 0.001494916999945417, + 0.001495791002525948, + 0.0014934160135453567, + 0.0014835829933872446, + 0.0014928750024409965, + 0.0014914589992258698, + 0.0014874170010443777, + 0.0014837079943390563, + 0.001501791994087398, + 0.0014879999944241717, + 0.0014992910000728443, + 0.0014848330029053614, + 0.0015004580054664984, + 0.0014968329924158752, + 0.0015845000016270205, + 0.001573416986502707, + 0.0015194160077953711, + 0.0014821660006418824, + 0.0014944159920560196, + 0.0014903330011293292, + 0.0014974590012570843, + 0.0014999999984866008, + 0.0014826249971520156, + 0.0014775000017834827, + 0.0014831250009592623, + 0.0014888750010868534, + 0.0016195000061998144, + 0.001517333002993837, + 0.0015047079941723496, + 0.0014907080039847642, + 0.0018917910056188703, + 0.0016672920028213412, + 0.0015405419981107116, + 0.001574499998241663, + 0.001531874993816018, + 0.001511250011390075, + 0.0015186670061666518, + 0.0015279590006684884, + 0.0015773339982843027, + 0.0015107499930309132, + 0.0015663749945815653, + 0.0015136670117499307, + 0.001615457993466407, + 0.0015166660014074296, + 0.0015916249976726249, + 0.0015118750015972182, + 0.0015176249871728942, + 0.0015293329925043508, + 0.0014905830030329525, + 0.001495292002800852, + 0.0014935420040274039, + 0.0014935420040274039, + 0.0015067920030560344, + 0.0015176250017248094, + 0.0014970000047469512, + 0.0014995420060586184, + 0.0014929169992683455, + 0.0014876249915687367, + 0.0014887919969623908, + 0.001494875003118068, + 0.0014775419986108318, + 0.0014822920056758448, + 0.0014849169965600595, + 0.0014891669998178259, + 0.0015843750006752089, + 0.0014974999940022826, + 0.0014973330107750371, + 0.0014888330042595044, + 0.0014792090078117326, + 0.0014855419867672026, + 0.0015024169988464564, + 0.0014847499987808987, + 0.0014979589905124158, + 0.0014834580069873482, + 0.0014964169968152419, + 0.0015394589863717556, + 0.0015690000000176951, + 0.0015511249948758632, + 0.0015705409896327183, + 0.0015547919902019203, + 0.0015959580050548539, + 0.0015769169986015186, + 0.0015524159971391782, + 0.0015591659903293476, + 0.001561165991006419, + 0.0015517500141868368, + 0.0016056250024121255, + 0.0015465830074390396, + 0.0015559160092379898 + ], + [ + 0.000795208994532004, + 0.00047900000936351717, + 0.0004072500014444813, + 0.0003802499995799735, + 0.0004420829936861992, + 0.00042337499326094985, + 0.00037304199940990657, + 0.0003644579992396757, + 0.0003741669934242964, + 0.0003639170026872307, + 0.0003612909931689501, + 0.0003627499972935766, + 0.0003652079903986305, + 0.00037187499401625246, + 0.00036854199424851686, + 0.0003717499930644408, + 0.0003753749915631488, + 0.00037120799242984504, + 0.00036537498817779124, + 0.0003640000068116933, + 0.00036287499824538827, + 0.00036408400046639144, + 0.00036416700459085405, + 0.0003815829986706376, + 0.0003871249937219545, + 0.00037229199369903654, + 0.0003625000099418685, + 0.00037437499850057065, + 0.0003684999974211678, + 0.000370624999050051, + 0.000366291991667822, + 0.0003677909990074113, + 0.00036433299828786403, + 0.0003639999922597781, + 0.0003643749951152131, + 0.00036787500721402466, + 0.00036533400998450816, + 0.00036508298944681883, + 0.0003627090045483783, + 0.00036458300019148737, + 0.00037404200702439994, + 0.00037195799814071506, + 0.0003656250046333298, + 0.0003660420043161139, + 0.00036612499388866127, + 0.0003713329933816567, + 0.000370624999050051, + 0.0003683340037241578, + 0.000403750003897585, + 0.0003776250086957589, + 0.0003673329920275137, + 0.000366666994523257, + 0.0003705830022227019, + 0.00037104199873283505, + 0.00037029200757388026, + 0.0003772910131374374, + 0.0003726249997271225, + 0.00036941700091119856, + 0.00037024999619461596, + 0.0003672919992823154, + 0.00037029200757388026, + 0.00037516700103878975, + 0.00037637499917764217, + 0.000374625000404194, + 0.000371540998457931, + 0.0003701250097947195, + 0.0003760410036193207, + 0.0003738329978659749, + 0.0003742910048458725, + 0.00037370801146607846, + 0.00038212499930523336, + 0.0003767080052057281, + 0.0003742920089280233, + 0.0003715839993674308, + 0.00036929101042915136, + 0.00037387499469332397, + 0.00037454100674949586, + 0.0003764160064747557, + 0.00037708300806116313, + 0.0003769169998122379, + 0.0003767500020330772, + 0.00037399999564513564, + 0.00038337499427143484, + 0.0003750409960048273, + 0.0003760410036193207, + 0.0003766250010812655, + 0.0003770000039367005, + 0.0003842080041067675, + 0.000378249998902902, + 0.00037583400262519717, + 0.00038649998896289617, + 0.00037895800778642297, + 0.0003802919964073226, + 0.0003912090032827109, + 0.0003817079996224493, + 0.00043229199945926666, + 0.0003942500043194741, + 0.0015454999957000837, + 0.0015079169970704243, + 0.0015008330083219334, + 0.001493374991696328, + 0.0014969170006224886, + 0.00150079200102482, + 0.0015052080125315115, + 0.0014935829967726022, + 0.0014966660091886297, + 0.0015303750114981085, + 0.0015140000032261014, + 0.001488541005528532, + 0.0015006670000730082, + 0.0014893750048941001, + 0.0014852080057607964, + 0.0015358330128947273, + 0.0014964580041123554, + 0.0015038340061437339, + 0.0014945409930078313, + 0.0014934159989934415, + 0.001491583010647446, + 0.0014999999984866008, + 0.001493374991696328, + 0.0014819170028204098, + 0.0014828330022282898, + 0.0015036249969853088, + 0.0016242499987129122, + 0.001528042004792951, + 0.0015151250117924064, + 0.00149049999890849, + 0.0016985419933917, + 0.0017938329983735457, + 0.001557333001983352, + 0.0015121250035008416, + 0.0015811250050319359, + 0.0015129159874049947, + 0.0015065829938976094, + 0.00151012500282377, + 0.0015761249960632995, + 0.0015341250109486282, + 0.001527291999082081, + 0.0014997079997556284, + 0.0015631659916834906, + 0.001549582986626774, + 0.001530499997898005, + 0.0015094999980647117, + 0.0015039590070955455, + 0.0015382499987026677, + 0.0015044579922687262, + 0.0014842920063529164, + 0.001487292000092566, + 0.0014937090018065646, + 0.0014887500001350418, + 0.0015302910032914951, + 0.001498083001933992, + 0.0014948749885661528, + 0.0015557080041617155, + 0.0014897920045768842, + 0.0014943339920137078, + 0.0014947500021662563, + 0.001488249996327795, + 0.001498833007644862, + 0.0014930409961380064, + 0.001498458004789427, + 0.0016017499874578789, + 0.0015046660118969157, + 0.0014961249980842695, + 0.0014830000000074506, + 0.001490166992880404, + 0.0014927080046618357, + 0.0014919169916538522, + 0.0014854170003673062, + 0.0014826249971520156, + 0.0014965839945944026, + 0.001485375003539957, + 0.0015464169991901144, + 0.0014942080015316606, + 0.00149049999890849, + 0.0015052079979795963, + 0.0014854580076644197, + 0.0014915409992681816, + 0.001536042007501237, + 0.0014964579895604402, + 0.0014894170017214492, + 0.0014972920034779236, + 0.0014863749966025352, + 0.0015118750015972182, + 0.0015159169997787103, + 0.0015145409997785464, + 0.0015495410043513402, + 0.0015776250074850395, + 0.0016726659960113466, + 0.0015953749971231446, + 0.0015635410090908408, + 0.0015564999921480194, + 0.0015625409869244322, + 0.0015558750019408762, + 0.0015611249982612208, + 0.001570124994032085, + 0.00156095800048206, + 0.0016391669923905283, + 0.0015647499967599288, + 0.0015836659877095371, + 0.001598541988641955, + 0.001575459013110958, + 0.0015592080017086118, + 0.0015622500068275258, + 0.001564165999297984, + 0.001583250006660819, + 0.0015572499978588894, + 0.0015627499960828573, + 0.0015662080113543198, + 0.001561167009640485, + 0.0015585410001222044, + 0.0015595000004395843, + 0.001559249998535961, + 0.0015545829955954105, + 0.0015591659903293476, + 0.0015440409915754572, + 0.0015525409980909899, + 0.001580000011017546, + 0.0015588329988531768, + 0.0015620839985786006, + 0.0015550829994026572, + 0.0015703749959357083, + 0.0015738329966552556, + 0.0015651660069124773 + ], + [ + 0.0007499579951399937, + 0.0004446669918252155, + 0.0003892499953508377, + 0.0003672500024549663, + 0.0003577089955797419, + 0.0003633749875007197, + 0.00035775000287685543, + 0.00037408299976959825, + 0.0003682079986901954, + 0.0003750829928321764, + 0.0003592919965740293, + 0.00036520800495054573, + 0.0003702089888975024, + 0.0003627500118454918, + 0.0003625410026870668, + 0.0003677909990074113, + 0.0003668330027721822, + 0.00036241699126549065, + 0.0003693750040838495, + 0.0003626249963417649, + 0.0003617080074036494, + 0.00036212499253451824, + 0.00037220900412648916, + 0.0003719580126926303, + 0.00037349999183788896, + 0.00036816699139308184, + 0.00040187500417232513, + 0.00040429100045002997, + 0.0004075420001754537, + 0.00041341599717270583, + 0.000400500008254312, + 0.00039754199679009616, + 0.0004047090042149648, + 0.00039862500852905214, + 0.0003987500094808638, + 0.0003887080092681572, + 0.00037229199369903654, + 0.0003691669990075752, + 0.00042308299452997744, + 0.00038741700700484216, + 0.0003742909902939573, + 0.0003706659917952493, + 0.0003721659886650741, + 0.00037295800575520843, + 0.0003668330027721822, + 0.00037112500285729766, + 0.00037183400127105415, + 0.0003682500100694597, + 0.00037216600321698934, + 0.0003732920013135299, + 0.00041449999844189733, + 0.000414624999393709, + 0.0004080420039827004, + 0.0007388749945675954, + 0.0004288749914849177, + 0.00040695900679565966, + 0.0003808750043390319, + 0.0003766250010812655, + 0.00039254198782145977, + 0.00038533400220330805, + 0.00037995900493115187, + 0.0003776660014409572, + 0.0003751250042114407, + 0.0003807500033872202, + 0.00037275000067893416, + 0.0003787500027101487, + 0.0003892090026056394, + 0.00037866600905545056, + 0.0003776669909711927, + 0.00037879199953749776, + 0.00037583398807328194, + 0.0003859999997075647, + 0.00038245899486355484, + 0.0003768330061575398, + 0.00040220799564849585, + 0.0003839999990304932, + 0.0003774170036194846, + 0.00038225000025704503, + 0.00038212499930523336, + 0.00040029099909588695, + 0.0003913330001523718, + 0.0004070830036653206, + 0.00038475000474136323, + 0.0003840829886030406, + 0.0003792079951381311, + 0.0003748330054804683, + 0.0003777089877985418, + 0.00037974999577272683, + 0.0003835409879684448, + 0.0004002920031780377, + 0.0003876249975292012, + 0.0003879160067299381, + 0.00038837500324007124, + 0.0003847920015687123, + 0.0003818339901044965, + 0.00039454099896829575, + 0.0005269590037642047, + 0.0018627079989528283, + 0.0015988329978426918, + 0.0016059170011430979, + 0.0015267499984474853, + 0.0015408330073114485, + 0.0015094170084921643, + 0.0015143749915296212, + 0.0015266660047927871, + 0.0015046250045998022, + 0.0015337500080931932, + 0.0015991670079529285, + 0.0015652080037398264, + 0.0015978329902281985, + 0.0016320830036420375, + 0.0015187499957391992, + 0.0015085420018294826, + 0.0015087079955264926, + 0.0015002920117694885, + 0.0015080000011948869, + 0.0015050000074552372, + 0.0015212909929687157, + 0.0016084169910755008, + 0.0015136250003706664, + 0.0015100829914445058, + 0.0014924160059308633, + 0.0015025409957161173, + 0.0015145409997785464, + 0.0014979999978095293, + 0.0015033749950816855, + 0.0015083750040503219, + 0.0014962919958634302, + 0.0016020839975681156, + 0.0015180000045802444, + 0.0015071250090841204, + 0.0014991250063758343, + 0.0014973330107750371, + 0.001505749998614192, + 0.001499416001024656, + 0.0015005000022938475, + 0.0014890830061631277, + 0.0014919160021236166, + 0.0014965419977670535, + 0.001496166005381383, + 0.0015073749964358285, + 0.0015189999976428226, + 0.001496208002208732, + 0.0014958750107325613, + 0.0015450000064447522, + 0.001572583001689054, + 0.0014943339920137078, + 0.0015294580080080777, + 0.0014964159927330911, + 0.0014941670087864622, + 0.001497584002208896, + 0.0016395410057157278, + 0.0015076249983394518, + 0.0014935420040274039, + 0.0014827499981038272, + 0.001869334009825252, + 0.0016307499899994582, + 0.001530499997898005, + 0.0015226249961415306, + 0.0015370000037364662, + 0.0014989580085966736, + 0.001552041998365894, + 0.0015264170069713145, + 0.001607084006536752, + 0.0015656659961678088, + 0.0016380409942939878, + 0.0015943750040605664, + 0.0015901669976301491, + 0.0016533750022063032, + 0.001611583007615991, + 0.0015227080002659932, + 0.001520249992609024, + 0.0015639159973943606, + 0.0015657500043744221, + 0.0015130000101635233, + 0.0015050000074552372, + 0.0015002909931354225, + 0.0015282910026144236, + 0.0015670000138925388, + 0.0015822499990463257, + 0.0015663749945815653, + 0.0015638330078218132, + 0.0015689170104451478, + 0.0017427080019842833, + 0.0018068339995807037, + 0.001800375001039356, + 0.0018586249934742227, + 0.0019187919970136136, + 0.001816375006455928, + 0.001778791003744118, + 0.0017482080002082512, + 0.0017967499879887328, + 0.0018074580002576113, + 0.001757083009579219, + 0.001749500006553717, + 0.0017551670025568455, + 0.0017347910034004599, + 0.001788374996976927, + 0.0017693340050755069, + 0.0017792909929994494, + 0.0017869159928523004, + 0.0018419999978505075, + 0.0017880829982459545, + 0.0015991660038707778, + 0.0015757090004626662, + 0.0015792499907547608, + 0.0016259580006590113, + 0.0015742090035928413, + 0.0015591670089634135, + 0.0015593749994877726, + 0.0015609170077368617, + 0.0015650419954909012, + 0.001556624993099831, + 0.0015431670035468414, + 0.00178283300192561, + 0.0017627090128371492, + 0.0017857500060927123, + 0.0017588750051800162, + 0.0017667920037638396, + 0.0017734169960021973, + 0.0017669999942881986, + 0.0017995420057559386, + 0.0017755419976310804, + 0.0017710829997668043 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 0.7787089998601004, + "p95": 0.8133996016113088, + "min": 0.7621669938089326, + "max": 0.8134160016197711, + "iqr": 0.04050100687891245, + "mean": 0.788091798312962, + "cv": 0.030237407725713015 + }, + "decode_ms_per_step": { + "count": 1115, + "median": 1.5513749967794865, + "p95": 1.6553536930587143, + "min": 0.7910840067779645, + "max": 2.1964589977869764, + "iqr": 0.26293799601262435, + "mean": 1.441040743563132, + "cv": 0.15636864677880885 + }, + "total_ms": { + "count": 5, + "median": 321.35886294418015, + "p95": 326.299217788619, + "min": 318.47308414580766, + "max": 326.7105449922383, + "iqr": 5.149421966052614, + "mean": 322.1401776128914, + "cv": 0.010777118639921058 + }, + "tokens_per_second": { + "count": 5, + "median": 697.0400565516958, + "p95": 702.9020372577529, + "min": 685.6221919782895, + "max": 703.3561426416974, + "iqr": 11.12010534876947, + "mean": 695.4139034533727, + "cv": 0.010747838866318273 + }, + "median_step_ms_by_index": [ + 0.7787089998601004, + 0.82562500028871, + 0.8305829978780821, + 0.8469169988529757, + 0.8410830050706863, + 0.8430420130025595, + 0.8606670016888529, + 0.8697919984115288, + 0.880457999301143, + 0.8853750041453168, + 0.8911249897209927, + 1.0041659988928586, + 0.9969169914256781, + 1.0005829972214997, + 1.0108750138897449, + 1.0217079980066046, + 1.00637499417644, + 0.999541996861808, + 1.0326249903300777, + 1.006167003652081, + 1.0117920028278604, + 1.0190839966526255, + 1.0310409998055547, + 1.0260420094709843, + 1.0269159974996, + 1.078583998605609, + 1.0987909918185323, + 1.0860839975066483, + 1.0941660002572462, + 1.1348330008331686, + 1.1100830015493557, + 1.1100000119768083, + 1.1076670052716509, + 1.2284999975236133, + 1.2377910024952143, + 1.2682910019066185, + 1.2450409994926304, + 1.2574160064104944, + 1.2550840037874877, + 1.27966700529214, + 1.2222079967614263, + 1.236958007211797, + 1.296458998695016, + 1.271499990252778, + 1.2651249999180436, + 1.2380830012261868, + 1.2790409964509308, + 1.254125003470108, + 1.2418749975040555, + 1.2827500031562522, + 1.2998750025872141, + 1.3080409989925101, + 1.2946670030942187, + 1.274125010240823, + 1.293374996748753, + 1.2796249939128757, + 1.281207994907163, + 1.3111250009387732, + 1.3082500081509352, + 1.2983330088900402, + 1.303083001403138, + 1.271874993108213, + 1.3012920098844916, + 1.2795410002581775, + 1.3284999877214432, + 1.422500004991889, + 1.4316660090116784, + 1.4562500000465661, + 1.4176670083543286, + 1.4646659983554855, + 1.415208011167124, + 1.4346250100061297, + 1.4719170139869675, + 1.4126250025583431, + 1.4140419953037053, + 1.4387080009328201, + 1.4213340036803856, + 1.4287079975474626, + 1.43183401087299, + 1.4385829999810085, + 1.44824999733828, + 1.5177079912973568, + 1.452415992389433, + 1.4700409956276417, + 1.4645000046584755, + 1.4671249955426902, + 1.509582987637259, + 1.479625003412366, + 1.4704579953104258, + 1.473750002332963, + 1.4849169965600595, + 1.5525409980909899, + 1.5194580046227202, + 1.5039160061860457, + 1.5070000081323087, + 1.514541989308782, + 1.5098329895408824, + 1.5060829900903627, + 1.4986250025685877, + 1.5717499918537214, + 1.5877080004429445, + 1.5387909952551126, + 1.5100000018719584, + 1.5812500059837475, + 1.5856670070206746, + 1.582500000949949, + 1.5753750049043447, + 1.5866670000832528, + 1.5837499959161505, + 1.5641249920008704, + 1.5917080017970875, + 1.57216600200627, + 1.5760840033181012, + 1.5945839986670762, + 1.5616250020684674, + 1.5668330015614629, + 1.5644580125808716, + 1.5607499954057857, + 1.5921659942250699, + 1.5635410090908408, + 1.5676659968448803, + 1.5659580094506964, + 1.5619580080965534, + 1.5699579962529242, + 1.57816699356772, + 1.595624999026768, + 1.5668330015614629, + 1.564375008456409, + 1.5683329984312877, + 1.5652919973945245, + 1.5643749939044937, + 1.5510829980485141, + 1.577332994202152, + 1.5550829994026572, + 1.5566670044790953, + 1.552666988573037, + 1.5693339955760166, + 1.5613329887855798, + 1.559459007694386, + 1.5579580067424104, + 1.5566250076517463, + 1.552000001538545, + 1.567457991768606, + 1.558709001983516, + 1.5675000031478703, + 1.5745420096209273, + 1.561750003020279, + 1.5622500068275258, + 1.5644580125808716, + 1.5720840019639581, + 1.5740840026410297, + 1.5852499927859753, + 1.5547500079264864, + 1.5704580000601709, + 1.5605000080540776, + 1.5514589904341847, + 1.5535830025328323, + 1.5574999997625127, + 1.5597080055158585, + 1.5540419990429655, + 1.557167008286342, + 1.5537079889327288, + 1.5654169983463362, + 1.564582998980768, + 1.564375008456409, + 1.5743749972898513, + 1.5778749948367476, + 1.581374992383644, + 1.5693750028731301, + 1.5711250016465783, + 1.5761249960632995, + 1.5724160039098933, + 1.57200000830926, + 1.5751250030007213, + 1.6351670055883005, + 1.6340840084012598, + 1.5690839936723933, + 1.5637079923180863, + 1.559790995088406, + 1.5817920066183433, + 1.5724999975645915, + 1.56095800048206, + 1.5626660024281591, + 1.5733750042272732, + 1.629291000426747, + 1.5770830068504438, + 1.5629999979864806, + 1.567084007547237, + 1.575832997332327, + 1.5710419975221157, + 1.5871669893385842, + 1.5649169945390895, + 1.5644160012016073, + 1.5566660003969446, + 1.5767500008223578, + 1.6342079907190055, + 1.5623749932274222, + 1.5720419905846938, + 1.5775419888086617, + 1.5816669911146164, + 1.5740420058136806, + 1.5704589895904064, + 1.567583007272333, + 1.591666994499974, + 1.585332996910438, + 1.5766249998705462, + 1.5584170032525435, + 1.5632910071872175, + 1.5777090011397377, + 1.5726249985164031, + 1.5707499987911433, + 1.56787499145139, + 1.568042003782466, + 1.5632080030627549, + 1.5647499967599288, + 1.5610420086886734, + 1.5634169976692647, + 1.5639590128557757, + 1.558916992507875, + 1.5574160061078146, + 1.5581249899696559, + 1.573166999150999, + 1.5627079992555082, + 1.5548329974990338 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0007787089998601004, + 0.0008234160050051287, + 0.0008332089928444475, + 0.0008469169988529757, + 0.0008424999978160486, + 0.0008427920110989362, + 0.0008684169879416004, + 0.000873625001986511, + 0.000880457999301143, + 0.0008863750117598101, + 0.0008911249897209927, + 0.0010078749910462648, + 0.0009942080068867654, + 0.0010005829972214997, + 0.0010082910011988133, + 0.0010217079980066046, + 0.00100637499417644, + 0.000999541996861808, + 0.0011262500047450885, + 0.0010245830053463578, + 0.0010132919996976852, + 0.0010190839966526255, + 0.0010323749884264544, + 0.0010388750088168308, + 0.001330125000094995, + 0.001250875007826835, + 0.0010987909918185323, + 0.0010927080002147704, + 0.0011077500093961135, + 0.0011664999910863116, + 0.001156042009824887, + 0.0011923749989364296, + 0.0011167079937877133, + 0.0012339589884504676, + 0.0012384160072542727, + 0.0013694579974981025, + 0.0012591659906320274, + 0.0012769579916493967, + 0.0012670419964706525, + 0.0013456669985316694, + 0.0012639170017791912, + 0.001278208990697749, + 0.001296458998695016, + 0.001271499990252778, + 0.0012740840029437095, + 0.0012723749969154596, + 0.0012830419873353094, + 0.0012835829984396696, + 0.0012947500072186813, + 0.0013943340018158779, + 0.0013121670053806156, + 0.0013157919893274084, + 0.0013262919965200126, + 0.0013275840028654784, + 0.0012999580067116767, + 0.0013057909964118153, + 0.0013030419941060245, + 0.0013111250009387732, + 0.0013185830030124635, + 0.0013214160135248676, + 0.0013265409943414852, + 0.0013767920027021319, + 0.0013431669940473512, + 0.0013470840058289468, + 0.001397708008880727, + 0.0014681659959023818, + 0.0014556249952875078, + 0.0014734579890500754, + 0.0014762089995201677, + 0.0014646659983554855, + 0.0014831250009592623, + 0.0015037090051919222, + 0.001488042005803436, + 0.0014963749999878928, + 0.0014783750084461644, + 0.0014935829967726022, + 0.0014997079997556284, + 0.0014868749858578667, + 0.0015707080019637942, + 0.0015085000050021335, + 0.0015015410026535392, + 0.001522334001492709, + 0.0015172499988693744, + 0.0015136250003706664, + 0.0015241250075632706, + 0.0015255829930538312, + 0.0015324999985750765, + 0.001534332986921072, + 0.0015332500042859465, + 0.0015484160103369504, + 0.0015432499931193888, + 0.0015525409980909899, + 0.001560165997943841, + 0.0015697500057285652, + 0.0015700000076321885, + 0.00159441601135768, + 0.0015890409995336086, + 0.0015870830102358013, + 0.0015823749999981374, + 0.0015824580041226, + 0.0015877080004429445, + 0.0015824589936528355, + 0.0015843329892959446, + 0.001596375004737638, + 0.0015856670070206746, + 0.001582500000949949, + 0.0015843329892959446, + 0.0015866670000832528, + 0.0015837499959161505, + 0.0015779160021338612, + 0.0015776250074850395, + 0.0015664170059608296, + 0.0016646250005578622, + 0.0015945839986670762, + 0.0015769580058986321, + 0.0015668330015614629, + 0.0018966670031659305, + 0.0018957499996758997, + 0.0018459170096321031, + 0.0017128329927800223, + 0.0016295829991577193, + 0.0016065840027295053, + 0.0016821669996716082, + 0.0015799170068930835, + 0.00157816699356772, + 0.0017253339901799336, + 0.0016138329956447706, + 0.001571625005453825, + 0.00156883399176877, + 0.0016142090025823563, + 0.0016092079895315692, + 0.0015478340064873919, + 0.001577332994202152, + 0.0015550829994026572, + 0.0015566670044790953, + 0.0015649589913664386, + 0.0019428339874139056, + 0.0016150830051628873, + 0.0015706669946666807, + 0.001564916005008854, + 0.001579292002134025, + 0.0015717920032329857, + 0.0016272499924525619, + 0.001575457994476892, + 0.0015720840019639581, + 0.0015745420096209273, + 0.0016267920000245795, + 0.001576208000187762, + 0.001563459009048529, + 0.0015720840019639581, + 0.0015740840026410297, + 0.0015852499927859753, + 0.0015593330026604235, + 0.0015704580000601709, + 0.001566667007864453, + 0.0015602909988956526, + 0.0015733750042272732, + 0.0016616659995634109, + 0.001663250004639849, + 0.0015700420044595376, + 0.0015570000105071813, + 0.0015537079889327288, + 0.0015629590052412823, + 0.0015615829906892031, + 0.001564375008456409, + 0.0016367499920306727, + 0.0015778749948367476, + 0.001581374992383644, + 0.0015693750028731301, + 0.0015711250016465783, + 0.0015761249960632995, + 0.001558666001074016, + 0.00157200000830926, + 0.0015780410030856729, + 0.0016848749946802855, + 0.0015659579948987812, + 0.0015690839936723933, + 0.0016174579941434786, + 0.0019322920124977827, + 0.0016299160051858053, + 0.0015790420002304018, + 0.0016563749959459528, + 0.001595708992681466, + 0.0015961250028340146, + 0.001629291000426747, + 0.0016001670010155067, + 0.0015753330080769956, + 0.001567084007547237, + 0.0016705420130165294, + 0.0015690830041421577, + 0.001569708008901216, + 0.0015649169945390895, + 0.0015754589985590428, + 0.0015817909879842773, + 0.0015767500008223578, + 0.001593540990143083, + 0.0015623749932274222, + 0.0015614169969921932, + 0.001609333005035296, + 0.0016125419933814555, + 0.0015740420058136806, + 0.0015715000045020133, + 0.0015578750026179478, + 0.0016027499950723723, + 0.001585332996910438, + 0.0015795840008649975, + 0.0015565000066999346, + 0.001557374998810701, + 0.001581584001542069, + 0.001622707990463823, + 0.0016011249972507358, + 0.0015745830023661256, + 0.001560459000756964, + 0.0015632080030627549, + 0.001565333004691638, + 0.0015610420086886734, + 0.0015634169976692647, + 0.0015639590128557757, + 0.001571291999425739, + 0.0015695000038249418, + 0.0015701660013291985, + 0.0015588750102324411, + 0.001567082988913171, + 0.0015764169947942719 + ], + [ + 0.0007728329946985468, + 0.00082562500028871, + 0.0008305829978780821, + 0.0008458749944111332, + 0.0008392500021727756, + 0.0008477919909637421, + 0.0008582919981563464, + 0.0008686669898452237, + 0.0008725420047994703, + 0.0008839579968480393, + 0.0008996249962365255, + 0.0010041659988928586, + 0.0010080000065499917, + 0.0009983750060200691, + 0.0010149589943466708, + 0.0010805420024553314, + 0.0010601669928291813, + 0.001047917001415044, + 0.0010326249903300777, + 0.001006167003652081, + 0.0010117920028278604, + 0.0010233330103801563, + 0.0010310409998055547, + 0.0010260420094709843, + 0.0010269159974996, + 0.001078583998605609, + 0.0012461659935070202, + 0.0011169589997734874, + 0.0010941660002572462, + 0.001253041991731152, + 0.0011384999961592257, + 0.0012282910029171035, + 0.0014052089973120019, + 0.0012934999977005646, + 0.0011765829985961318, + 0.0011867920111399144, + 0.0012450409994926304, + 0.0012574160064104944, + 0.001254707996849902, + 0.0012339999957475811, + 0.001205000007757917, + 0.001210917005664669, + 0.0012268749997019768, + 0.0012517080031102523, + 0.0012253749882802367, + 0.0012174590083304793, + 0.0012403750006342307, + 0.0012385000009089708, + 0.0012232080043759197, + 0.0012827500031562522, + 0.001252250003744848, + 0.0012716659985017031, + 0.0012524169869720936, + 0.0012579580070450902, + 0.0012490829976741225, + 0.0012468330096453428, + 0.001281207994907163, + 0.0012874590029241517, + 0.0012532079999800771, + 0.0012576669978443533, + 0.0012575420114444569, + 0.0012607920070877299, + 0.0012702499952865764, + 0.0012795410002581775, + 0.001266291001229547, + 0.001422500004991889, + 0.0013862500054528937, + 0.0014562500000465661, + 0.0014176670083543286, + 0.0014562909927917644, + 0.0014094999933149666, + 0.0014190830115694553, + 0.0014719170139869675, + 0.0014126250025583431, + 0.0014073749916860834, + 0.0014387080009328201, + 0.0014213340036803856, + 0.0014229160005925223, + 0.0014212909882189706, + 0.0014324160001706332, + 0.001434916994185187, + 0.0014443330001085997, + 0.0014415829937206581, + 0.0014485839928966016, + 0.0014492909976979718, + 0.0014489579916698858, + 0.001509582987637259, + 0.0014682909968541935, + 0.001460999992559664, + 0.0014675420097773895, + 0.001469207985792309, + 0.001491082992288284, + 0.0014865420089336112, + 0.0014893339975969866, + 0.0014999999984866008, + 0.0015043329913169146, + 0.0015098329895408824, + 0.001495625008828938, + 0.0014947500021662563, + 0.0015362919948529452, + 0.001592792003066279, + 0.0014982910070102662, + 0.0014989160117693245, + 0.0015787500014994293, + 0.001969832999748178, + 0.001621707997401245, + 0.0015753750049043447, + 0.001654082996537909, + 0.0015846669994061813, + 0.0016205830033868551, + 0.0015917080017970875, + 0.00157216600200627, + 0.0015760840033181012, + 0.0016452909912914038, + 0.001584792000357993, + 0.0015710000006947666, + 0.0015644580125808716, + 0.0015657920012017712, + 0.0016055000014603138, + 0.001565291007864289, + 0.0015676659968448803, + 0.0015611249982612208, + 0.0015610410046065226, + 0.0015699579962529242, + 0.0016687080060364679, + 0.0015696670016041026, + 0.0015665420069126412, + 0.0015660839999327436, + 0.0015683329984312877, + 0.0015652919973945245, + 0.0015643749939044937, + 0.0015510829980485141, + 0.0015580829931423068, + 0.0014896249922458082, + 0.0015620409976691008, + 0.0015094580012373626, + 0.0015693339955760166, + 0.0014945830043870956, + 0.0015009579947218299, + 0.0015009589988039806, + 0.001515625001047738, + 0.0015126250073080882, + 0.001567457991768606, + 0.001558709001983516, + 0.0015564579953206703, + 0.0015964580088621005, + 0.001582500000949949, + 0.0015711250016465783, + 0.0015644580125808716, + 0.0015702499949838966, + 0.0015800410037627444, + 0.0015915420081000775, + 0.0015547500079264864, + 0.0015554580022580922, + 0.0015630420093657449, + 0.001569374988321215, + 0.0015535830025328323, + 0.0015595420118188486, + 0.0015627920074621215, + 0.001559084004838951, + 0.0015619170007994398, + 0.0015519579901592806, + 0.0015627909888280556, + 0.001564582998980768, + 0.0015627919929102063, + 0.0015658339980291203, + 0.001564958001836203, + 0.0015550829994026572, + 0.0015630839916411787, + 0.0015610830014338717, + 0.001564833990414627, + 0.0015724160039098933, + 0.0015078750002430752, + 0.0014837499911664054, + 0.00153012499504257, + 0.0014912909973645583, + 0.0014971250056987628, + 0.0015244579990394413, + 0.001501791994087398, + 0.0014913750055711716, + 0.001553500012960285, + 0.0015530410018982366, + 0.0015626660024281591, + 0.0015589999966323376, + 0.0017002079985104501, + 0.0015770830068504438, + 0.0015629999979864806, + 0.001562124991323799, + 0.0021964589977869764, + 0.0017420840013073757, + 0.0015871669893385842, + 0.0015675419999752194, + 0.0016321250004693866, + 0.0015811249904800206, + 0.0016108750132843852, + 0.0016029590042307973, + 0.0015839160041650757, + 0.0015800410037627444, + 0.0015844160079723224, + 0.0015816669911146164, + 0.0015817910025361925, + 0.0015717080095782876, + 0.0015688749990658835, + 0.0015811659977771342, + 0.00156879099085927, + 0.001563832993269898, + 0.0015584170032525435, + 0.0015567090013064444, + 0.0015496669948333874, + 0.0016914580046432093, + 0.00156004200107418, + 0.0015595000004395843, + 0.001568042003782466, + 0.0015699999930802733, + 0.0015598750032950193, + 0.0015701670054113492, + 0.0015586250083288178, + 0.0015584170032525435, + 0.001558916992507875, + 0.0015695000038249418, + 0.0016249170002993196, + 0.0015838749968679622, + 0.0015687910054111853, + 0.0015548329974990338 + ], + [ + 0.0007621669938089326, + 0.0008305420051328838, + 0.0008297079912154004, + 0.0007974590116646141, + 0.0007910840067779645, + 0.0008032920013647527, + 0.0008245000062743202, + 0.0008286249940283597, + 0.0008345829992322251, + 0.000836124992929399, + 0.0008491250046063215, + 0.0009500840096734464, + 0.0009528750088065863, + 0.0009521669999230653, + 0.0009645420068409294, + 0.0009618329931981862, + 0.0009663329983595759, + 0.0009664170065661892, + 0.000959749988396652, + 0.0009575409931130707, + 0.0009691250015748665, + 0.000985000006039627, + 0.0009905829938361421, + 0.0009846659959293902, + 0.0009842910076258704, + 0.0010244159930152819, + 0.0010389579983893782, + 0.0010367499926360324, + 0.0010424579959362745, + 0.0010454169969307259, + 0.0010649589967215434, + 0.0010689999908208847, + 0.0010526670084800571, + 0.0011622919992078096, + 0.0012599159963428974, + 0.0013842500047758222, + 0.0013387920043896884, + 0.0013852910051355138, + 0.0012636250030482188, + 0.0012380419939290732, + 0.001207875000545755, + 0.001236958007211797, + 0.0015802910056663677, + 0.0014778750046389177, + 0.0013077499897917733, + 0.0012380830012261868, + 0.0012782919948222116, + 0.001254125003470108, + 0.0012418749975040555, + 0.0012599999899975955, + 0.0013319159916136414, + 0.0013080409989925101, + 0.0012946670030942187, + 0.0012717500067083165, + 0.0013076670002192259, + 0.0012796249939128757, + 0.0012637909967452288, + 0.0016307920013787225, + 0.001322916999924928, + 0.0012983330088900402, + 0.001275874994462356, + 0.0012660420034080744, + 0.0013012920098844916, + 0.001277417002711445, + 0.001333374995738268, + 0.0014122910070000216, + 0.001400583001668565, + 0.0013927919935667887, + 0.0013968749990453944, + 0.0015002919972175732, + 0.001415208011167124, + 0.0014346250100061297, + 0.001397540996549651, + 0.0014052499900572002, + 0.0014135829987935722, + 0.0014190000074449927, + 0.001420666987542063, + 0.0014287079975474626, + 0.0014274999994086102, + 0.001429582989658229, + 0.00144824999733828, + 0.0015586670051561669, + 0.001452415992389433, + 0.0014700409956276417, + 0.0014645000046584755, + 0.0014671249955426902, + 0.0014681250031571835, + 0.001479625003412366, + 0.0014704579953104258, + 0.001473750002332963, + 0.0014785829989705235, + 0.001593708002474159, + 0.0015194580046227202, + 0.0015039160061860457, + 0.0015070000081323087, + 0.001514541989308782, + 0.00149125000461936, + 0.0015060829900903627, + 0.0014890419988660142, + 0.001486499997554347, + 0.0014982499997131526, + 0.0015387909952551126, + 0.0015100000018719584, + 0.001523415994597599, + 0.0015140829927986488, + 0.0014930420002201572, + 0.0015608750109095126, + 0.0015737909998279065, + 0.0016258329997071996, + 0.0015104999911272898, + 0.0014980419946368784, + 0.0014956249942770228, + 0.0018835840019164607, + 0.001611457992112264, + 0.001512666989583522, + 0.0015473750099772587, + 0.0015652500005671754, + 0.0015607499954057857, + 0.0015921659942250699, + 0.0015635410090908408, + 0.0016633750055916607, + 0.0015653750015189871, + 0.0015619580080965534, + 0.0015775409992784262, + 0.001662499998928979, + 0.001595624999026768, + 0.0015763749979669228, + 0.001564375008456409, + 0.001575208007125184, + 0.0015938750002533197, + 0.0015764999989187345, + 0.0015676249895477667, + 0.0017149170016637072, + 0.001572792010847479, + 0.0016336250118911266, + 0.0015774579951539636, + 0.0015629999979864806, + 0.0015613329887855798, + 0.0015450840000994503, + 0.001567625004099682, + 0.0015566250076517463, + 0.0015550829994026572, + 0.0015638339973520488, + 0.001557333001983352, + 0.0016499999910593033, + 0.0015769170131534338, + 0.0015617500030202791, + 0.0015622500068275258, + 0.0015665829996578395, + 0.00157891699927859, + 0.0015667910047341138, + 0.0015653750015189871, + 0.0015547079965472221, + 0.0015618749894201756, + 0.0015113330009626225, + 0.0014866669953335077, + 0.0014917499938746914, + 0.0014867080026306212, + 0.0014908749872120097, + 0.001484207998146303, + 0.001485165994381532, + 0.001529583998490125, + 0.0015669999993406236, + 0.0015475839900318533, + 0.0015631249989382923, + 0.0018339160014875233, + 0.001815208001062274, + 0.0017800420027924702, + 0.0017725420038914308, + 0.0017598339909454808, + 0.0018539580050855875, + 0.001768333007930778, + 0.001835999995819293, + 0.0017522080015623942, + 0.0018280830117873847, + 0.00163929199334234, + 0.001577167000505142, + 0.0015516670100623742, + 0.001559790995088406, + 0.0015553330013062805, + 0.0015724999975645915, + 0.00156095800048206, + 0.0015660000062780455, + 0.0015733750042272732, + 0.0015627079992555082, + 0.0015584999928250909, + 0.0015985829959390685, + 0.0015743339899927378, + 0.001575832997332327, + 0.0015557500009890646, + 0.0016967090050457045, + 0.0015757079963805154, + 0.0015532079996773973, + 0.0015522080066148192, + 0.0019517499895300716, + 0.00169237500813324, + 0.0015789159951964393, + 0.0017039579979609698, + 0.0015775419888086617, + 0.0015696670016041026, + 0.0015637500036973506, + 0.0015704589895904064, + 0.001567583007272333, + 0.0016239169926848263, + 0.0015937910065986216, + 0.0015658750053262338, + 0.0015701250085840002, + 0.0016829160013003275, + 0.0015777090011397377, + 0.0015662499936297536, + 0.0015735409979242831, + 0.001566916995216161, + 0.001617332993191667, + 0.0015660409990232438, + 0.001545749997603707, + 0.0015534999984083697, + 0.0015661670040572062, + 0.0015652919973945245, + 0.001579625008162111, + 0.0015574160061078146, + 0.0015476249973289669, + 0.0015583749918732792, + 0.0015533340047113597, + 0.0015509590011788532 + ], + [ + 0.0008134160016197711, + 0.0008155839896062389, + 0.000884250010130927, + 0.0008714159921510145, + 0.0008410830050706863, + 0.0008481660042889416, + 0.0008651669922983274, + 0.0008748330001253635, + 0.0008881670073606074, + 0.0008936249942053109, + 0.0009014580136863515, + 0.0010087500122608617, + 0.0009969169914256781, + 0.0010051249992102385, + 0.0010125839908141643, + 0.0010330410004826263, + 0.001010167005006224, + 0.0009573749994160607, + 0.0009618329931981862, + 0.0009619580087019131, + 0.0009682079980848357, + 0.0009738330118125305, + 0.000984459009487182, + 0.000984082987997681, + 0.0009967500081984326, + 0.0010508749983273447, + 0.001149582996731624, + 0.0010544999968260527, + 0.0010468750115251169, + 0.0010442499915370718, + 0.0010645000002114102, + 0.0010539169888943434, + 0.001078208995750174, + 0.0011687919904943556, + 0.001180459003080614, + 0.0011883330007549375, + 0.0011758340115193278, + 0.001192249997984618, + 0.001200832994072698, + 0.001285791993723251, + 0.0012222079967614263, + 0.001205000007757917, + 0.0012292909959796816, + 0.0012370839976938441, + 0.0012072919926140457, + 0.001230333000421524, + 0.0012819590047001839, + 0.0012360830005491152, + 0.0012414999946486205, + 0.0012565000070026144, + 0.001251625013537705, + 0.001255833005416207, + 0.001260834003915079, + 0.001274125010240823, + 0.0012533330009318888, + 0.0012460830039344728, + 0.0012529590021586046, + 0.0012433329975465313, + 0.001258584001334384, + 0.0012794159993063658, + 0.001303083001403138, + 0.001271874993108213, + 0.0012653329904424027, + 0.00126937500317581, + 0.0012705830013146624, + 0.0013927919935667887, + 0.0014316660090116784, + 0.0013929160049883649, + 0.001383416005410254, + 0.001390875011566095, + 0.001400624998495914, + 0.001425209004082717, + 0.0013950420106993988, + 0.0013959999923827127, + 0.0014140419953037053, + 0.0014113750075921416, + 0.001407625008141622, + 0.0014187080087140203, + 0.00143183401087299, + 0.0014385829999810085, + 0.0014383750094566494, + 0.0014416669873753563, + 0.0014464170089922845, + 0.001466875008190982, + 0.0014527500024996698, + 0.0014604999887524173, + 0.0014448750007431954, + 0.001458124999771826, + 0.001466083005652763, + 0.0014727090019732714, + 0.0014849169965600595, + 0.0014837920025456697, + 0.0014779589982936159, + 0.0014991249918239191, + 0.0014979999978095293, + 0.0015014579985290766, + 0.0015032090013846755, + 0.0014922500122338533, + 0.0014986250025685877, + 0.0015882500010775402, + 0.0015792909980518743, + 0.001502500002970919, + 0.0014970000047469512, + 0.0016224169958150014, + 0.0020983750000596046, + 0.0016820839955471456, + 0.0017347499961033463, + 0.0016922920040087774, + 0.0015719579969299957, + 0.001553333000629209, + 0.0016262080025626346, + 0.0016229169996222481, + 0.0015566670044790953, + 0.0015680829965276644, + 0.0015616250020684674, + 0.0015932499954942614, + 0.0015514999977312982, + 0.0015461660077562556, + 0.0015453750093001872, + 0.001556750008603558, + 0.0015636249881936237, + 0.001617542002350092, + 0.0016609589947620407, + 0.0015567079972242936, + 0.0015629160043317825, + 0.0015514999977312982, + 0.0015668330015614629, + 0.0015576669975416735, + 0.0015555840072920546, + 0.0015543330082437024, + 0.0015511660021729767, + 0.0015463330055354163, + 0.0015883750020293519, + 0.0015540000022156164, + 0.0015435829991474748, + 0.001552666988573037, + 0.0016272920038318262, + 0.0015577919984934852, + 0.0015624590014340356, + 0.0015579580067424104, + 0.0015624170046066865, + 0.00155050000466872, + 0.0015639159973943606, + 0.0015579579921904951, + 0.0015675000031478703, + 0.001544166007079184, + 0.0015509579970967025, + 0.001561375000164844, + 0.0016059999907156453, + 0.001556083996547386, + 0.0015584169887006283, + 0.0015534579870291054, + 0.0015502919995924458, + 0.0015856249956414104, + 0.0015605000080540776, + 0.001549374996102415, + 0.0015667089901398867, + 0.0015485000039916486, + 0.0015597080055158585, + 0.0015540419990429655, + 0.001557167008286342, + 0.0015586249937769026, + 0.0016887910023797303, + 0.0015718750073574483, + 0.0016193750052480027, + 0.0015743749972898513, + 0.0019328330090502277, + 0.0016703750006854534, + 0.0015857089892961085, + 0.0016292079963022843, + 0.0016549159918213263, + 0.0015614170115441084, + 0.0015631670103175566, + 0.0015751250030007213, + 0.0016351670055883005, + 0.0016340840084012598, + 0.0015664170059608296, + 0.0015637079923180863, + 0.0015862079890212044, + 0.0015817920066183433, + 0.001552834000904113, + 0.0015445419994648546, + 0.001548416999867186, + 0.0015534999984083697, + 0.0016392500110669062, + 0.001589749997947365, + 0.0015520830056630075, + 0.001555459006340243, + 0.0015655830065952614, + 0.0015710419975221157, + 0.0015467920020455495, + 0.0015437499969266355, + 0.0015644160012016073, + 0.0015566660003969446, + 0.0015598750032950193, + 0.0016342079907190055, + 0.0015620420017512515, + 0.0015720419905846938, + 0.0015697920025559142, + 0.0016488339897478, + 0.0015731660096207634, + 0.0015671669971197844, + 0.0015574999997625127, + 0.0015583750064251944, + 0.0015607090026605874, + 0.0015766249998705462, + 0.001557374998810701, + 0.0015632910071872175, + 0.0015687080012867227, + 0.0015726249985164031, + 0.0015707499987911433, + 0.0015918749995762482, + 0.0015567090013064444, + 0.0015629159897798672, + 0.001566499995533377, + 0.001559832991915755, + 0.0015625410014763474, + 0.00157216600200627, + 0.0015574160061078146, + 0.0015528750082012266, + 0.0015581249899696559, + 0.0015731669991509989, + 0.0015627079992555082, + 0.0015718750073574483 + ], + [ + 0.0008133340015774593, + 0.0008409580041188747, + 0.0008238750015152618, + 0.0008545000018784776, + 0.0008687910012667999, + 0.0008430420130025595, + 0.0008606670016888529, + 0.0008697919984115288, + 0.0008824579999782145, + 0.0008853750041453168, + 0.0008904580026865005, + 0.0010032910067820922, + 0.0010887500102398917, + 0.0010761250014184043, + 0.0010108750138897449, + 0.0010202920093433931, + 0.0009969579987227917, + 0.0010055000020656735, + 0.0011304579966235906, + 0.0013983749959152192, + 0.0011082079872721806, + 0.0010312079975847155, + 0.00104674999602139, + 0.0010744589962996542, + 0.0011755829909816384, + 0.0010889580007642508, + 0.0010926670074695721, + 0.0010860839975066483, + 0.0011124160082545131, + 0.0011348330008331686, + 0.0011100830015493557, + 0.0011100000119768083, + 0.0011076670052716509, + 0.0012284999975236133, + 0.0012377910024952143, + 0.0012682910019066185, + 0.0012374580110190436, + 0.0012409999908413738, + 0.0012550840037874877, + 0.00127966700529214, + 0.00124937501095701, + 0.001265041995793581, + 0.001331874998868443, + 0.0013041249912930652, + 0.0012651249999180436, + 0.0012672500015469268, + 0.0012790409964509308, + 0.0012851670035161078, + 0.0012829579936806113, + 0.001299124996876344, + 0.0012998750025872141, + 0.0013105830003041774, + 0.0013104160025250167, + 0.0013225409929873422, + 0.001293374996748753, + 0.0013152909959899262, + 0.0013166659919079393, + 0.0013136659981682897, + 0.0013082500081509352, + 0.001309540995862335, + 0.0013707080070162192, + 0.0013523749948944896, + 0.0013346250052563846, + 0.0013315410033101216, + 0.0013284999877214432, + 0.0014454579941229895, + 0.0014735000004293397, + 0.0014627499913331121, + 0.0014550420019077137, + 0.0014659160078736022, + 0.0014719159953529015, + 0.001487082990934141, + 0.0014777089963899925, + 0.0015271250013029203, + 0.0014915829960955307, + 0.0014782920043217018, + 0.0014778340118937194, + 0.001485375003539957, + 0.0014909580058883876, + 0.0015386250015581027, + 0.0014913329941919073, + 0.0015177079912973568, + 0.0015100829914445058, + 0.001513374998467043, + 0.0015150830004131421, + 0.0015445419994648546, + 0.0015327919973060489, + 0.0015300419909181073, + 0.0016372500103898346, + 0.0016041249909903854, + 0.0015639170014765114, + 0.0015737089997855946, + 0.0018448750051902607, + 0.0018092500104103237, + 0.0015991660038707778, + 0.0016063329967437312, + 0.001721500011626631, + 0.0015925000043353066, + 0.0015811250050319359, + 0.0015717499918537214, + 0.0016542079974897206, + 0.0015911249938653782, + 0.0015910420042928308, + 0.0015812500059837475, + 0.001573416986502707, + 0.0015486250049434602, + 0.001549374996102415, + 0.0015690830041421577, + 0.0015577910089632496, + 0.0015641249920008704, + 0.0015948749933158979, + 0.0016343330062227324, + 0.0015712079912191257, + 0.0015551670076092705, + 0.0015542499895673245, + 0.0015593330026604235, + 0.0015495419938815758, + 0.0015602920029778033, + 0.001552207992062904, + 0.0015586249937769026, + 0.0015473750099772587, + 0.0015659580094506964, + 0.0015613339928677306, + 0.0015488749922951683, + 0.0015525420021731406, + 0.0016510839923284948, + 0.0015515410050284117, + 0.0015465420001419261, + 0.0015512079990003258, + 0.001557333001983352, + 0.0015524579939665273, + 0.0015905839973129332, + 0.0015707080019637942, + 0.0015592500130878761, + 0.0015472910017706454, + 0.0015516249986831099, + 0.0015481669979635626, + 0.0015979169984348118, + 0.001559459007694386, + 0.0015576659934595227, + 0.0015479590074392036, + 0.001552000001538545, + 0.0015786250005476177, + 0.0015735000051790848, + 0.0015436249959748238, + 0.001560999997309409, + 0.0015470420039491728, + 0.0015560840110993013, + 0.0015456659893970937, + 0.0016294999950332567, + 0.0016161250096047297, + 0.00168379201204516, + 0.001560750009957701, + 0.0015982920012902468, + 0.0015578750026179478, + 0.0015514589904341847, + 0.0015502500027650967, + 0.0015574999997625127, + 0.0015491249941987917, + 0.0015466250042663887, + 0.0015752499894006178, + 0.0016781250014901161, + 0.0015654169983463362, + 0.0016249170002993196, + 0.0015652079891879112, + 0.0015509160002693534, + 0.0015623329964000732, + 0.0015592920099152252, + 0.0015621670027030632, + 0.0015511249948758632, + 0.0015639170014765114, + 0.0017165000026579946, + 0.001604375007445924, + 0.0015618330071447417, + 0.0015917920100037009, + 0.0017677919968264177, + 0.0019088750123046339, + 0.0015892499941401184, + 0.0015585000073770061, + 0.0015867909969529137, + 0.001638874993659556, + 0.001585332996910438, + 0.0015587910020258278, + 0.0015739579976070672, + 0.0016264589939964935, + 0.0015618339966749772, + 0.0015500419976888224, + 0.001570124994032085, + 0.0015593330026604235, + 0.001584833997185342, + 0.0016097079933388159, + 0.0015445419994648546, + 0.0015471669903490692, + 0.0015525000053457916, + 0.0015654169983463362, + 0.0016800829907879233, + 0.0015488329954678193, + 0.0015442500007338822, + 0.0015632910071872175, + 0.001557708004838787, + 0.0015803749993210658, + 0.0015696669870521873, + 0.0015905420004855841, + 0.001591666994499974, + 0.0016004999924916774, + 0.0016130000003613532, + 0.0015971660031937063, + 0.0016181660030269995, + 0.0015792499907547608, + 0.001559625001391396, + 0.0015629999979864806, + 0.00156787499145139, + 0.0016234170034294948, + 0.0015542499895673245, + 0.0015647499967599288, + 0.0015613329887855798, + 0.0015921250014798716, + 0.00155050000466872, + 0.0015513749967794865, + 0.0015489170036744326, + 0.0015495419938815758, + 0.0015979579911800101, + 0.0015442500007338822, + 0.0015443750016856939 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_mha": { + "first_token_ms": { + "count": 5, + "median": 0.9215000027325004, + "p95": 0.9392586041940376, + "min": 0.8792500011622906, + "max": 0.9411670034751296, + "iqr": 0.014375007594935596, + "mean": 0.9181584027828649, + "cv": 0.025754112265939728 + }, + "decode_ms_per_step": { + "count": 1115, + "median": 1.8593749991850927, + "p95": 2.000521401350852, + "min": 0.8832500025164336, + "max": 2.3693330003879964, + "iqr": 0.4322294989833608, + "mean": 1.7158277847776984, + "cv": 0.18061277092249087 + }, + "total_ms": { + "count": 5, + "median": 383.6469170637429, + "p95": 385.96277044853196, + "min": 379.7072661545826, + "max": 386.236072081374, + "iqr": 1.5906110929790884, + "mean": 383.5477544082096, + "cv": 0.006360492765649297 + }, + "tokens_per_second": { + "count": 5, + "median": 583.8701942775743, + "p95": 588.8286562735398, + "min": 579.9561879160956, + "max": 589.9281366630665, + "iqr": 2.4153689895737216, + "mean": 584.0401238596057, + "cv": 0.006385211914787705 + }, + "median_step_ms_by_index": [ + 0.9215000027325004, + 0.9686249977676198, + 0.9394160006195307, + 0.937249991693534, + 0.9479160071350634, + 0.9436669934075326, + 0.9519160084892064, + 0.9532089898129925, + 0.9689580037957057, + 0.973166010226123, + 0.9745419956743717, + 1.1018329969374463, + 1.091124999220483, + 1.0990839946316555, + 1.1034999915864319, + 1.092666992917657, + 1.082791990484111, + 1.1293749994365498, + 1.0996660130331293, + 1.1114169901702553, + 1.1413330066716298, + 1.1295420117676258, + 1.1230000091018155, + 1.1080839904025197, + 1.114958999096416, + 1.1671659885905683, + 1.1875829950440675, + 1.1806670081568882, + 1.1833750031655654, + 1.1975000088568777, + 1.196958008222282, + 1.1980839917669073, + 1.1972920037806034, + 1.327833000686951, + 1.3237920065876096, + 1.3323750026756898, + 1.3349159999052063, + 1.3415830035228282, + 1.3511249999282882, + 1.373749997583218, + 1.3653329951921478, + 1.367250006296672, + 1.4084169961279258, + 1.4150830102153122, + 1.4170829963404685, + 1.442499997210689, + 1.4344159862957895, + 1.4450830058194697, + 1.4536249946104363, + 1.459416002035141, + 1.4632910024374723, + 1.47237500641495, + 1.4814589958405122, + 1.4915829960955307, + 1.4725410001119599, + 1.4692499971715733, + 1.4768749970244244, + 1.4826249971520156, + 1.4929589960956946, + 1.5654159942641854, + 1.5057090058689937, + 1.5114159905351698, + 1.522874998045154, + 1.5254999889293686, + 1.5204170049401, + 1.648042001761496, + 1.6459159960504621, + 1.668375000008382, + 1.658875000430271, + 1.6668339958414435, + 1.6814159898785874, + 1.677749998634681, + 1.6837919974932447, + 1.7000829975586385, + 1.7255409911740571, + 1.710875003482215, + 1.7181250004796311, + 1.7452079919166863, + 1.7597910045878962, + 1.7602089938009158, + 1.7442079988541082, + 1.7714999994495884, + 1.7895829951157793, + 1.7869169969344512, + 1.785208994988352, + 1.796374999685213, + 1.8065829935949296, + 1.8339160014875233, + 1.8135840073227882, + 1.8387080053798854, + 1.832374997320585, + 1.8423330038785934, + 1.8562920013209805, + 1.8957919965032488, + 1.873917004559189, + 1.8814579962054268, + 1.8694169993977994, + 1.8844170117517933, + 1.8720420048339292, + 1.907000012579374, + 1.8724580004345626, + 1.8612910062074661, + 1.8735830090008676, + 1.863999990746379, + 1.9290409982204437, + 1.9049160036956891, + 1.866791004431434, + 1.8620420014485717, + 1.8888329941546544, + 1.8726660055108368, + 1.861875003669411, + 1.8724160036072135, + 1.8842920107999817, + 1.8680000066524372, + 1.870082996902056, + 1.8692919984459877, + 1.869458006694913, + 1.8673750018933788, + 1.8631249986356124, + 1.8810839974321425, + 1.864749996457249, + 1.8633330037118867, + 1.8675419996725395, + 1.8689169955905527, + 1.8661659996723756, + 1.861124997958541, + 1.853875000961125, + 1.856916001997888, + 1.8663749942788854, + 1.8659169872989878, + 1.8596669979160652, + 1.9067080138484016, + 1.9245000003138557, + 1.9190420134691522, + 1.8764580017887056, + 1.9497920002322644, + 1.943916009622626, + 1.9325419998494908, + 1.9367089989827946, + 1.877500006230548, + 1.8742499960353598, + 1.9635000062407926, + 1.8708340066950768, + 1.9460000039543957, + 1.947541008121334, + 1.924750002217479, + 1.8877499969676137, + 1.87462501344271, + 1.8729169969446957, + 1.9324169988976792, + 1.938832996529527, + 1.942874994711019, + 1.95387500571087, + 1.9519159977789968, + 1.9514160085236654, + 1.9462909986032173, + 2.028167000389658, + 1.9552919984562322, + 1.886082987766713, + 1.89462500566151, + 1.878291994216852, + 1.887249993160367, + 1.8669590062927455, + 1.8576250004116446, + 1.868915991508402, + 1.9479579932522029, + 1.9460000039543957, + 1.9500839989632368, + 1.972708007087931, + 1.937124994583428, + 1.9380419980734587, + 1.9365000043762848, + 1.9376249983906746, + 1.866667007561773, + 1.896416986710392, + 1.945833006175235, + 1.9454169960226864, + 1.9430410029599443, + 1.9526660034898669, + 1.9407500076340511, + 1.9433339912211522, + 1.8805830040946603, + 1.8857500108424574, + 1.9076660100836307, + 1.8810409965226427, + 1.859790994785726, + 1.8894160020863637, + 1.905500001157634, + 1.8909579957835376, + 1.9368750072317198, + 1.8733330070972443, + 1.9450000108918175, + 1.9138339994242415, + 1.9419580057729036, + 1.9397919968469068, + 1.9424589991103858, + 1.9400829914957285, + 1.9602919928729534, + 1.9639159872895107, + 1.9608750008046627, + 1.9483750074869022, + 1.9529169949237257, + 1.8850839987862855, + 1.8610409897519276, + 1.8762500112643465, + 1.8591669941088185, + 1.8774579948512837, + 1.8582079937914386, + 1.896583999041468, + 1.8717499915510416, + 1.8716250051511452, + 1.8701670051086694, + 1.8719170038821176, + 1.9411670073168352, + 1.9520839996403083, + 1.9415830029174685, + 1.9450830004643649, + 1.9477910100249574, + 1.9571660086512566, + 1.948917008121498, + 1.891583000542596, + 1.8599169998196885, + 1.973499995074235, + 1.8679159984458238 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0009316250070696697, + 0.001077166001778096, + 0.0010167500004172325, + 0.0010129169968422502, + 0.0009912499954225495, + 0.0009900829900288954, + 0.0009940840100171044, + 0.000996457994915545, + 0.0010209589963778853, + 0.0010190409957431257, + 0.0010647080052876845, + 0.0012001249997410923, + 0.0011864999978570268, + 0.0012075830018147826, + 0.0012041250010952353, + 0.0011985409946646541, + 0.0012136669975006953, + 0.0012819169933209196, + 0.0011700000031851232, + 0.00115833300515078, + 0.0011497919913381338, + 0.0011970829946221784, + 0.0012883339950349182, + 0.0015250419965013862, + 0.0012331670004641637, + 0.0012306249991524965, + 0.0012407089961925521, + 0.0013407920050667599, + 0.0012513749970821664, + 0.001255082999705337, + 0.001257209005416371, + 0.0012702079984592274, + 0.001315667002927512, + 0.0013942080113338307, + 0.0013919999910285696, + 0.0014037079963600263, + 0.0014800840144744143, + 0.0014844169927528128, + 0.0014232499961508438, + 0.0014256659924285486, + 0.0014392909943126142, + 0.0014655419945484027, + 0.0014436659985221922, + 0.0014288329984992743, + 0.0014170829963404685, + 0.0014272910048021004, + 0.0014344159862957895, + 0.0014492909976979718, + 0.0014612080121878535, + 0.0014700000028824434, + 0.0014669579977635294, + 0.00147237500641495, + 0.0014730000111740083, + 0.0014991250063758343, + 0.0014723340136697516, + 0.0014681250031571835, + 0.001476207995438017, + 0.0014780829951632768, + 0.0014929589960956946, + 0.001594290995853953, + 0.0015057090058689937, + 0.0015114159905351698, + 0.0015162909985519946, + 0.0015261250082403421, + 0.0015349579916801304, + 0.0016559579962631688, + 0.0016459159960504621, + 0.001656375010497868, + 0.001654707986745052, + 0.0016668339958414435, + 0.0016839169984450564, + 0.001677749998634681, + 0.0016837919974932447, + 0.0016997920029098168, + 0.0017255409911740571, + 0.001727415990899317, + 0.0017181250004796311, + 0.0017247079958906397, + 0.0017308750102529302, + 0.0018240000063087791, + 0.0017410830041626468, + 0.0017686659994069487, + 0.0017711249965941533, + 0.001799582998501137, + 0.001785208994988352, + 0.0017959580000024289, + 0.0018065829935949296, + 0.0018802909908117726, + 0.0018182079948019236, + 0.0018387080053798854, + 0.001832374997320585, + 0.0018423330038785934, + 0.0018562920013209805, + 0.0018810420006047934, + 0.0018657920008990914, + 0.0018814579962054268, + 0.001867666986072436, + 0.0018891249928856269, + 0.0018769169982988387, + 0.0018641670030774549, + 0.0018663750088308007, + 0.0019943329971283674, + 0.0018830409971997142, + 0.0018629160040291026, + 0.002010208001593128, + 0.002212416991824284, + 0.00193087499064859, + 0.0019775410037254915, + 0.0018888329941546544, + 0.00186191700049676, + 0.001861875003669411, + 0.0019127909909002483, + 0.001908167003421113, + 0.0018680000066524372, + 0.001870082996902056, + 0.00200470800336916, + 0.001869458006694913, + 0.0018637500033946708, + 0.0018789159948937595, + 0.0019036250014323741, + 0.001862499993876554, + 0.0018621670024003834, + 0.0018675419996725395, + 0.001868416991783306, + 0.0019188750011380762, + 0.0018605829973239452, + 0.0018532079993747175, + 0.0018562499899417162, + 0.001861500000813976, + 0.0018590410036267713, + 0.001854832997196354, + 0.0018627499957801774, + 0.0019595000048866495, + 0.0019190420134691522, + 0.0018570420070318505, + 0.0018592920096125454, + 0.001862124991021119, + 0.0018698329949984327, + 0.001858457995695062, + 0.0018593749991850927, + 0.0019800830050371587, + 0.0019635000062407926, + 0.0019526670075720176, + 0.0019521669892128557, + 0.001947541008121334, + 0.001967042000615038, + 0.0018836250092135742, + 0.0018525420018704608, + 0.0018551250104792416, + 0.0018595419969642535, + 0.0018606249941512942, + 0.0018524999904911965, + 0.0019107920088572428, + 0.00209312500373926, + 0.0019971660076407716, + 0.001956374995643273, + 0.002031999989412725, + 0.0023675830016145483, + 0.0019425000064074993, + 0.001958958004252054, + 0.0018782919942168519, + 0.0018677080079214647, + 0.0018822910060407594, + 0.0018815829971572384, + 0.002070333997835405, + 0.0020665840129368007, + 0.0019727089966181666, + 0.002013833000091836, + 0.001972708007087931, + 0.0019541250076144934, + 0.0019627500005299225, + 0.0019169580045854673, + 0.0018585000070743263, + 0.0018595000001369044, + 0.0018575830035842955, + 0.0018643340008566156, + 0.0019079579942626879, + 0.0018703750101849437, + 0.001854667003499344, + 0.0018562500044936314, + 0.0018684579990804195, + 0.001869332991191186, + 0.0018665410025278106, + 0.001891624997369945, + 0.001989875003346242, + 0.002052540992735885, + 0.001951957994606346, + 0.001952416991116479, + 0.001953917002538219, + 0.0019512910075718537, + 0.0018621659983182326, + 0.0018600829935166985, + 0.0018554999987827614, + 0.0018580830073915422, + 0.0018532919930294156, + 0.001857958995969966, + 0.0019349579961271957, + 0.0019602919928729534, + 0.0019847910007229075, + 0.001948749995790422, + 0.001950040998053737, + 0.0019493750005494803, + 0.0018850839987862855, + 0.0018610409897519276, + 0.0018606250087032095, + 0.0018530410015955567, + 0.0018602499912958592, + 0.0018582079937914386, + 0.0018551659886725247, + 0.0018717499915510416, + 0.0018716250051511452, + 0.0018627499957801774, + 0.0018614160071592778, + 0.0018551249959273264, + 0.0018532499962020665, + 0.0018736249912763014, + 0.0019450830004643649, + 0.0019477910100249574, + 0.001957125001354143, + 0.001984000002266839, + 0.0019672909984365106, + 0.0019478750036796555, + 0.001973499995074235, + 0.0018669170094653964 + ], + [ + 0.0008792500011622906, + 0.0009264580003218725, + 0.0008837080094963312, + 0.0008832500025164336, + 0.0008834590116748586, + 0.0008922910055844113, + 0.0009008339984575287, + 0.0009021250007208437, + 0.000909792011952959, + 0.0009123330091824755, + 0.0009257499914383516, + 0.0010867090022657067, + 0.001068832993041724, + 0.001057083994965069, + 0.0010527080012252554, + 0.0010607500007608905, + 0.001061375005519949, + 0.0010716250108089298, + 0.0010919169872067869, + 0.0010817500005941838, + 0.0010791660024551675, + 0.0011512500059325248, + 0.0011152079969178885, + 0.0011032910115318373, + 0.0011064999998779967, + 0.0011671659885905683, + 0.0011875829950440675, + 0.001178666003397666, + 0.0011833750031655654, + 0.0011819589999504387, + 0.0011892909969901666, + 0.0011951659980695695, + 0.0012020409922115505, + 0.001327833000686951, + 0.0013191670004744083, + 0.0013310840004123747, + 0.0013349159999052063, + 0.0013415830035228282, + 0.0013623749982798472, + 0.0013652089983224869, + 0.0013614159979624674, + 0.001367250006296672, + 0.0014084169961279258, + 0.0014150830102153122, + 0.001415040998836048, + 0.0014571249921573326, + 0.0014390830037882552, + 0.0014450830058194697, + 0.0014380000066012144, + 0.001459416002035141, + 0.001469374998123385, + 0.00148787499347236, + 0.0014844169927528128, + 0.0014915829960955307, + 0.0014725410001119599, + 0.0014710830000694841, + 0.0014742499915882945, + 0.0014826249971520156, + 0.0014975829981267452, + 0.001507291992311366, + 0.0014991250063758343, + 0.0015080409939400852, + 0.0015287499991245568, + 0.0015254999889293686, + 0.0015182920033112168, + 0.001648042001761496, + 0.0016467500099679455, + 0.0016733750089770183, + 0.0016585410048719496, + 0.0016600829985691234, + 0.0016741669969633222, + 0.0016867080121301115, + 0.0016926660027820617, + 0.0018375409999862313, + 0.0017376250034430996, + 0.0017076670046662912, + 0.0017337080062134191, + 0.0021175830042921007, + 0.0018422499997541308, + 0.0017535410006530583, + 0.001774832999217324, + 0.001788540990673937, + 0.0018643340008566156, + 0.0017871250020107254, + 0.0017872090102173388, + 0.0018058749992633238, + 0.0018103329930454493, + 0.0018224590021418408, + 0.0018169579998357221, + 0.0018546659994171932, + 0.0018459580023773015, + 0.002016958998865448, + 0.0019003750057891011, + 0.0019720830023288727, + 0.0019080839992966503, + 0.0019135409966111183, + 0.0018694169993977994, + 0.001868542007287033, + 0.001865875005023554, + 0.001907000012579374, + 0.001903375014080666, + 0.0018612910062074661, + 0.0018735830090008676, + 0.001863999990746379, + 0.0018597919988678768, + 0.001890041006845422, + 0.001866791004431434, + 0.0018620420014485717, + 0.0018973749974975362, + 0.0018726660055108368, + 0.0018636669992702082, + 0.001857499999459833, + 0.0018706249975366518, + 0.0018672910082386807, + 0.0018622079951455817, + 0.0018692919984459877, + 0.0018589579995023087, + 0.0019130409928038716, + 0.0018713329918682575, + 0.0018654160085134208, + 0.0018659580091480166, + 0.0018730829906417057, + 0.0018919579888461158, + 0.0018562499899417162, + 0.0018623329960973933, + 0.001861124997958541, + 0.0019937500037485734, + 0.001892708009108901, + 0.0019099159981124103, + 0.002246957999886945, + 0.0019970409921370447, + 0.0019487500103423372, + 0.0019245000003138557, + 0.0018676250037970021, + 0.0018764580017887056, + 0.0019497920002322644, + 0.0019744589953916147, + 0.002119791999575682, + 0.0021828750032000244, + 0.001891166000859812, + 0.0018726249982137233, + 0.0020301670010667294, + 0.0018708340066950768, + 0.00232883301214315, + 0.001970458004507236, + 0.001868915991508402, + 0.001870833002612926, + 0.00187462501344271, + 0.0018729169969446957, + 0.0019324169988976792, + 0.001938832996529527, + 0.0018778339872369543, + 0.00195387500571087, + 0.0018723750108620152, + 0.0018580000032670796, + 0.0018782499973895028, + 0.0018783330015139654, + 0.0018649580015335232, + 0.0018626660021254793, + 0.001867040991783142, + 0.0019238329987274483, + 0.0019419590098550543, + 0.0018669590062927455, + 0.0018576250004116446, + 0.0018677500047488138, + 0.0018598750029923394, + 0.0018618330068420619, + 0.0018598750029923394, + 0.0018870829953812063, + 0.00189462500566151, + 0.0018730410083662719, + 0.0018602909985929728, + 0.0018589579995023087, + 0.001865541998995468, + 0.0018712500022957101, + 0.0018607500096550211, + 0.0018635829910635948, + 0.0018610419938340783, + 0.001905707991681993, + 0.0018732079915935174, + 0.0018766669963952154, + 0.0018695410108193755, + 0.0018845000013243407, + 0.0019909169932361692, + 0.0018810409965226427, + 0.001859790994785726, + 0.0018768329900922254, + 0.0019040000042878091, + 0.0018792080081766471, + 0.0018853750079870224, + 0.0018733330070972443, + 0.0018633750005392358, + 0.0018834579968824983, + 0.0019463339995127171, + 0.0018656669999472797, + 0.002007583010708913, + 0.0018910829967353493, + 0.0018636250024428591, + 0.002042958003585227, + 0.0021687909902539104, + 0.0018974169943248853, + 0.002007208007853478, + 0.001932958999532275, + 0.0018610830011311918, + 0.001975582999875769, + 0.0019110830035060644, + 0.0018774579948512837, + 0.001865290993009694, + 0.001896583999041468, + 0.0018657910113688558, + 0.001859583004261367, + 0.0018701670051086694, + 0.0018593749991850927, + 0.001913125001010485, + 0.0018682919908314943, + 0.0018559169984655455, + 0.0018632919964147732, + 0.001862709003034979, + 0.0018555409915279597, + 0.0018582499906187877, + 0.0018565000063972548, + 0.0018599169998196885, + 0.0019262910063844174, + 0.0018679159984458238 + ], + [ + 0.0009411670034751296, + 0.0009686249977676198, + 0.0009394160006195307, + 0.0009340409887954593, + 0.0009672080050222576, + 0.0009429579949937761, + 0.00100449999445118, + 0.0009668330021668226, + 0.0009689580037957057, + 0.000970791996223852, + 0.0009745419956743717, + 0.0011018329969374463, + 0.001091124999220483, + 0.0010990839946316555, + 0.0011034999915864319, + 0.00116879100096412, + 0.0011229999945499003, + 0.0011293749994365498, + 0.0011298750032437965, + 0.0011416250054026023, + 0.0011413330066716298, + 0.0011295420117676258, + 0.0011230000091018155, + 0.0011080839904025197, + 0.0011132920044474304, + 0.0011627079948084429, + 0.0011705829965649173, + 0.0011806670081568882, + 0.0011756669991882518, + 0.0011990830098511651, + 0.0012319999950705096, + 0.0011980839917669073, + 0.0011935000075027347, + 0.001326500001596287, + 0.0013237920065876096, + 0.0013323750026756898, + 0.0013339580036699772, + 0.0013409169914666563, + 0.0013436669978545979, + 0.001436958002159372, + 0.001385374998790212, + 0.001366875003441237, + 0.001399667002260685, + 0.0014017499925103039, + 0.0017974170041270554, + 0.001536000010673888, + 0.001432999997632578, + 0.0015000419953139499, + 0.0014816250040894374, + 0.001476207995438017, + 0.0014632910024374723, + 0.0014677500003017485, + 0.0014792499860050157, + 0.0014982080028858036, + 0.0015182079951046035, + 0.0015309590089600533, + 0.001487250003265217, + 0.0014843339886283502, + 0.0014922499976819381, + 0.0015904589963611215, + 0.0015259170031640679, + 0.0015214590093819425, + 0.0015222080110106617, + 0.001527667001937516, + 0.0015532920078840107, + 0.0016387080104323104, + 0.001641540991840884, + 0.0016444580105599016, + 0.001658875000430271, + 0.001707250004983507, + 0.0016814159898785874, + 0.0016774579999037087, + 0.00167891698947642, + 0.0016891669947654009, + 0.001704750000499189, + 0.001706041002762504, + 0.0017717500013532117, + 0.0017317079909844324, + 0.0017358749901177362, + 0.0017910410097101703, + 0.0017442079988541082, + 0.0017685839993646368, + 0.0017703749908832833, + 0.0017740420007612556, + 0.0017797079926822335, + 0.0018322499963687733, + 0.0018006669997703284, + 0.0018096250132657588, + 0.0018135840073227882, + 0.0018309580045752227, + 0.001825917002861388, + 0.0018418749968986958, + 0.0018492079980205745, + 0.0018957919965032488, + 0.0018659169872989878, + 0.0018750000017462298, + 0.001860333009972237, + 0.0018844170117517933, + 0.0018720420048339292, + 0.0019993329915450886, + 0.0018551249959273264, + 0.0018557910079834983, + 0.0022101249924162403, + 0.001984665999771096, + 0.002003417001105845, + 0.0019506660028127953, + 0.0019504590018186718, + 0.0018665420066099614, + 0.0018852499924832955, + 0.0018900420109275728, + 0.0018914579995907843, + 0.0019307500042486936, + 0.0018842920107999817, + 0.0018627500103320926, + 0.0018723750108620152, + 0.0019352910021552816, + 0.0018718749925028533, + 0.0018670410063350573, + 0.0018631249986356124, + 0.0018810839974321425, + 0.001864749996457249, + 0.0018584170029498637, + 0.0018564170022727922, + 0.0018689169955905527, + 0.0018882500007748604, + 0.0018674580060178414, + 0.001853875000961125, + 0.0018560829921625555, + 0.0018663749942788854, + 0.001958499997272156, + 0.0018567499937489629, + 0.0018692500016186386, + 0.0019170419982401654, + 0.0018854579975595698, + 0.0018587079975986853, + 0.0018582089978735894, + 0.0018611250125104561, + 0.0019325419998494908, + 0.001964583992958069, + 0.0019503330113366246, + 0.0019490840059006587, + 0.002002417008043267, + 0.0019646250002551824, + 0.0019460000039543957, + 0.0018793749914038926, + 0.001870124993729405, + 0.0018877499969676137, + 0.0018535840063123032, + 0.0018537909927545115, + 0.001854832997196354, + 0.0019064580119447783, + 0.0019488750112941489, + 0.0019486660021357238, + 0.0019519159977789968, + 0.001941875001648441, + 0.0019443750061327592, + 0.0020338750036899, + 0.0018919169961009175, + 0.0018808330059982836, + 0.00189462500566151, + 0.0018659159977687523, + 0.0018579579918878153, + 0.0018678750057006255, + 0.0019595839985413477, + 0.0019487909885356203, + 0.0019479579932522029, + 0.0019460000039543957, + 0.0019964579987572506, + 0.002080374993965961, + 0.0018629580008564517, + 0.0018642499926500022, + 0.0022668330057058483, + 0.001968917000340298, + 0.001866667007561773, + 0.001896416986710392, + 0.001954667008249089, + 0.001999209009227343, + 0.001952749997144565, + 0.0019526660034898669, + 0.0019447079976089299, + 0.0019516669999575242, + 0.0020242919999873266, + 0.0019490420090733096, + 0.0019500839989632368, + 0.0020682500035036355, + 0.0019766250043176115, + 0.0019566670089261606, + 0.0019568749994505197, + 0.0018909579957835376, + 0.0018667499971343204, + 0.0018711249867919832, + 0.0020560830016620457, + 0.0019138339994242415, + 0.0019065829983446747, + 0.0019525419920682907, + 0.0019457919988781214, + 0.0019420830067247152, + 0.0020131659985054284, + 0.0019639159872895107, + 0.001948374992934987, + 0.0019483750074869022, + 0.001955625004484318, + 0.0019436670118011534, + 0.0019446670048637316, + 0.0019406250066822395, + 0.001943040988408029, + 0.0018929579964606091, + 0.001855249996879138, + 0.00185333400440868, + 0.0018669999990379438, + 0.0018575409922050312, + 0.0018554999987827614, + 0.0018719170038821176, + 0.0019411670073168352, + 0.0019649580062832683, + 0.0019545840041246265, + 0.0019416669965721667, + 0.0019456669979263097, + 0.0020144169975537807, + 0.0019444579957053065, + 0.0018517919961595908, + 0.001847624997026287, + 0.0022122500085970387, + 0.0019567909912439063 + ], + [ + 0.0009172499994747341, + 0.0009715839987620711, + 0.0009635829919716343, + 0.000937249991693534, + 0.0009479160071350634, + 0.0009907089988701046, + 0.0009514580015093088, + 0.0009528330119792372, + 0.0011295419972157106, + 0.001110333003452979, + 0.0010936670005321503, + 0.0011955829977523535, + 0.0011816659971373156, + 0.0011930839973501861, + 0.001131083001382649, + 0.001092666992917657, + 0.001082791990484111, + 0.0011615840048762038, + 0.0010996660130331293, + 0.0011114169901702553, + 0.0011644170008366928, + 0.001108666998334229, + 0.0011694590066326782, + 0.0011910419998457655, + 0.0011269579990766943, + 0.0011694999993778765, + 0.001191874995129183, + 0.0012447500048438087, + 0.0011946669983444735, + 0.0011975000088568777, + 0.001196958008222282, + 0.001208082991070114, + 0.0011972920037806034, + 0.0013367500068852678, + 0.0017063749983208254, + 0.0014255419955588877, + 0.0013631660112878308, + 0.0013492080033756793, + 0.0013511249999282882, + 0.001373749997583218, + 0.0013653329951921478, + 0.0014087090094108135, + 0.0014504169957945123, + 0.0014296250010374933, + 0.0014363749942276627, + 0.0014424999972106889, + 0.0015180000045802444, + 0.001436624996131286, + 0.0014536249946104363, + 0.0014551669883076102, + 0.0014616669941460714, + 0.0014989579940447584, + 0.0014814589958405122, + 0.0014839159994153306, + 0.0014742080093128607, + 0.0014692499971715733, + 0.0014835840120213106, + 0.00161029098671861, + 0.0015026250039227307, + 0.0015654159942641854, + 0.0015242090012179688, + 0.00151012500282377, + 0.001522874998045154, + 0.0015199579938780516, + 0.0015204170049401, + 0.0016394580015912652, + 0.0016422909975517541, + 0.001668375000008382, + 0.0016682499990565702, + 0.001659084009588696, + 0.0016656670049997047, + 0.0016742080042604357, + 0.0016820000018924475, + 0.001771415991242975, + 0.0017054159980034456, + 0.001710875003482215, + 0.001717291001114063, + 0.0017452079919166863, + 0.0017597910045878962, + 0.0017602089938009158, + 0.0017441249947296456, + 0.0017949589964700863, + 0.0018237500044051558, + 0.0017869169969344512, + 0.0017870419978862628, + 0.001796374999685213, + 0.0018193750001955777, + 0.0018339160014875233, + 0.0018068750068778172, + 0.0019914580043405294, + 0.002024166999035515, + 0.002054208001936786, + 0.0020420000073499978, + 0.0019749999919440597, + 0.001873917004559189, + 0.0019290839991299435, + 0.0022315000096568838, + 0.0019895830046152696, + 0.00187462501344271, + 0.0019332500087330118, + 0.0019231670012231916, + 0.001883624994661659, + 0.0018640829948708415, + 0.001868209001258947, + 0.0019290409982204437, + 0.0019049160036956891, + 0.001865290993009694, + 0.001856708011473529, + 0.0018989170057466254, + 0.0018584170029498637, + 0.0018517499993322417, + 0.0018507079948903993, + 0.0018954999977722764, + 0.0018977080035256222, + 0.0018785839929478243, + 0.0018572499975562096, + 0.0018570829997770488, + 0.0018673750018933788, + 0.0018515419942559674, + 0.0018566249927971512, + 0.0018533750117057934, + 0.0018633330037118867, + 0.0018672090081963688, + 0.001940374990226701, + 0.0018661659996723756, + 0.001869542000349611, + 0.0018503750034142286, + 0.001856916001997888, + 0.001858082992839627, + 0.0018522909958846867, + 0.0018934160034405068, + 0.0019067080138484016, + 0.0019462909986032173, + 0.0019474580039968714, + 0.0019600410014390945, + 0.001983999987714924, + 0.001953166996827349, + 0.0019086660031462088, + 0.0018515000119805336, + 0.001877500006230548, + 0.001862124991021119, + 0.0018526670028222725, + 0.0018524999904911965, + 0.0018515419942559674, + 0.001960291003342718, + 0.0020870410080533475, + 0.001984125003218651, + 0.0019469160033622757, + 0.0023604579910170287, + 0.002072125003905967, + 0.0019537920015864074, + 0.0020732919947477058, + 0.001976707993890159, + 0.002005707996431738, + 0.0019515000021783635, + 0.0019646250002551824, + 0.002028167000389658, + 0.0019552919984562322, + 0.0019533330050762743, + 0.001953958999365568, + 0.0019903329957742244, + 0.0019252500060247257, + 0.0018568329978734255, + 0.0018515419942559674, + 0.0018532919930294156, + 0.001972041995031759, + 0.0018551249959273264, + 0.0018567920051282272, + 0.0018533339898567647, + 0.001937124994583428, + 0.0019387499924050644, + 0.0019365000043762848, + 0.0019376249983906746, + 0.0019804999901680276, + 0.001998417006689124, + 0.001945833006175235, + 0.0019476660090731457, + 0.0019430410029599443, + 0.0019540830107871443, + 0.0019407500076340511, + 0.0019433339912211522, + 0.0018805830040946603, + 0.0018857500108424574, + 0.0019076660100836307, + 0.0018790000031003729, + 0.0018536249990575016, + 0.0018894160020863637, + 0.001905500001157634, + 0.0019443329947534949, + 0.0019368750072317198, + 0.001938790999702178, + 0.0019450000108918175, + 0.001944208997883834, + 0.0019419580057729036, + 0.0019421659962972626, + 0.001940749993082136, + 0.0019412080000620335, + 0.0019475419976515695, + 0.0019566249975468963, + 0.0019608750008046627, + 0.0018585000070743263, + 0.0018522080063121393, + 0.0018534159898990765, + 0.0018567920051282272, + 0.0018582089978735894, + 0.0018591669941088185, + 0.0019470830011414364, + 0.0019478750036796555, + 0.0019861670007230714, + 0.001956541003892198, + 0.0019711249915417284, + 0.0019481250055832788, + 0.0019534590101102367, + 0.0020584580051945522, + 0.0019568330026231706, + 0.0019456250010989606, + 0.0023566669988213107, + 0.002060957995126955, + 0.0019571660086512566, + 0.0020453750039450824, + 0.0019645419961307198, + 0.001984165995963849, + 0.0019882080086972564, + 0.0019505410018609837 + ], + [ + 0.0009215000027325004, + 0.0009425000025657937, + 0.0009256670018658042, + 0.0009423329902347177, + 0.0009355410002171993, + 0.0009436669934075326, + 0.0009519160084892064, + 0.0009532089898129925, + 0.0009567919914843515, + 0.000973166010226123, + 0.0009663749951869249, + 0.001089083991246298, + 0.001084791001630947, + 0.0010731670045061037, + 0.0010536250047152862, + 0.0010655840014806017, + 0.001051500003086403, + 0.0010856250009965152, + 0.001094749997719191, + 0.0010869169927900657, + 0.001123833004385233, + 0.0010988750000251457, + 0.0011025410058209673, + 0.0011064580030506477, + 0.001114958999096416, + 0.0011595420073717833, + 0.0011712080013239756, + 0.001171124997199513, + 0.0011764579976443201, + 0.0011843749962281436, + 0.001182165986392647, + 0.0011923749989364296, + 0.0011957079987041652, + 0.0013133339962223545, + 0.001323124990449287, + 0.0013245000009192154, + 0.0013284580054460093, + 0.001338749993010424, + 0.0013418329908745363, + 0.001357750006718561, + 0.001360790993203409, + 0.0013663330028066412, + 0.0013869160029571503, + 0.001412208002875559, + 0.001412249999702908, + 0.001414875005139038, + 0.0014199999859556556, + 0.0014362910005729645, + 0.001429041993105784, + 0.001446334004867822, + 0.001455416000680998, + 0.0014696670114062726, + 0.0015058749995660037, + 0.001481540995882824, + 0.0014725410001119599, + 0.001467416004743427, + 0.0014768749970244244, + 0.0014781659992877394, + 0.0014924579882062972, + 0.0014913750055711716, + 0.001495040996815078, + 0.001549332999275066, + 0.0015953749971231446, + 0.0015251669974531978, + 0.00151966699922923, + 0.0017047909932443872, + 0.0022488329996122047, + 0.0018934170075226575, + 0.001974250000785105, + 0.0017756669985828921, + 0.0016832080000312999, + 0.0017589579947525635, + 0.0016963749949354678, + 0.0017000829975586385, + 0.0017688329971861094, + 0.001718499988783151, + 0.0017165000026579946, + 0.0018278750067111105, + 0.001779625003109686, + 0.0017481660033809021, + 0.0017566660098964348, + 0.0017714999994495884, + 0.0017895829951157793, + 0.0017837499908637255, + 0.0017762079951353371, + 0.0017852500022854656, + 0.0017958750104298815, + 0.0018898330017691478, + 0.0018111249955836684, + 0.0018192089919466525, + 0.0018245829996885732, + 0.001840791999711655, + 0.0018383330025244504, + 0.0018593749991850927, + 0.0019765000033658, + 0.0018764999986160547, + 0.0019135829934384674, + 0.0018559579912107438, + 0.0018559169984655455, + 0.0018780840036924928, + 0.0018724580004345626, + 0.0018570829997770488, + 0.0018493750103516504, + 0.0018458330014254898, + 0.0018662079964997247, + 0.0018556669965619221, + 0.0018508339999243617, + 0.0018447080074111, + 0.0018514580006012693, + 0.0018924169999081641, + 0.001861208991613239, + 0.0018724160036072135, + 0.0018754580087261274, + 0.0018778749945340678, + 0.0018639589980011806, + 0.001851249995524995, + 0.0019997089984826744, + 0.0018796250078594312, + 0.0018601669871713966, + 0.0019362499879207462, + 0.002278041996760294, + 0.0018812499911291525, + 0.001990667005884461, + 0.0018814579962054268, + 0.0018541250028647482, + 0.0018566249927971512, + 0.0018771249888231978, + 0.0018886250036302954, + 0.0019227500015404075, + 0.0018659169872989878, + 0.0018596669979160652, + 0.0019681249978020787, + 0.001886708996607922, + 0.0019494169973768294, + 0.0019560840009944513, + 0.0019838750013150275, + 0.001943916009622626, + 0.0019463749922579154, + 0.0019367089989827946, + 0.0018685830000322312, + 0.0018742499960353598, + 0.001946667005540803, + 0.0018577920127427205, + 0.0018599589966470376, + 0.0018492500093998387, + 0.001924750002217479, + 0.0019960420031566173, + 0.001948584002093412, + 0.0019879169994965196, + 0.0019619160011643544, + 0.0019467499951133505, + 0.001942874994711019, + 0.00195575000543613, + 0.0019484999938867986, + 0.0019514160085236654, + 0.0019462909986032173, + 0.0019469159888103604, + 0.0019568749994505197, + 0.001886082987766713, + 0.0018469579954398796, + 0.001854499991168268, + 0.001887249993160367, + 0.0018576669972389936, + 0.0018507909990148619, + 0.001868915991508402, + 0.001934832995175384, + 0.0019484590011416003, + 0.0019500839989632368, + 0.002072165996651165, + 0.001959000001079403, + 0.0019380419980734587, + 0.0023693330003879964, + 0.0020151659991825, + 0.0019883339991793036, + 0.0020579590054694563, + 0.001955084007931873, + 0.0019454169960226864, + 0.001970291996258311, + 0.0019728750048670918, + 0.002018041006522253, + 0.0019820830057142302, + 0.0019473749998724088, + 0.0020444580004550517, + 0.0018829159962479025, + 0.0018644170049810782, + 0.0018589999963296577, + 0.0018793750059558079, + 0.0018530419911257923, + 0.0018709580035647377, + 0.0019526249961927533, + 0.0019505829986883327, + 0.0019859579915646464, + 0.0019432919943938032, + 0.0019667080050567165, + 0.0019397919968469068, + 0.0019424589991103858, + 0.0019400829914957285, + 0.0020139590051257983, + 0.0019438329909462482, + 0.0019848749943776056, + 0.001956167005118914, + 0.0019529169949237257, + 0.0018467500049155205, + 0.0018560829921625555, + 0.0018762500112643465, + 0.001847042003646493, + 0.0018451659998390824, + 0.0018458749982528389, + 0.0019220829999540001, + 0.0019404170016059652, + 0.0019347920024301857, + 0.0019337919948156923, + 0.0019526669930201024, + 0.0019689580076374114, + 0.0019520839996403083, + 0.0019415830029174685, + 0.0019482920033624396, + 0.0019786249904427677, + 0.001958666995051317, + 0.001948917008121498, + 0.001891583000542596, + 0.0018560419994173571, + 0.0018478749989299104, + 0.001859042007708922 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "speedup_cached_vs_uncached_sdpa": 1.387451766769339, + "speedup_cached_vs_uncached_mha": 1.6563775089911208 + }, + "memory": [ + { + "batch": 1, + "prompt_len": 128, + "steps": [ + { + "step": 0, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 1, + "cached_positions": 129, + "cache_bytes": 792576 + }, + { + "step": 2, + "cached_positions": 130, + "cache_bytes": 798720 + }, + { + "step": 3, + "cached_positions": 131, + "cache_bytes": 804864 + }, + { + "step": 4, + "cached_positions": 132, + "cache_bytes": 811008 + }, + { + "step": 5, + "cached_positions": 133, + "cache_bytes": 817152 + }, + { + "step": 6, + "cached_positions": 134, + "cache_bytes": 823296 + }, + { + "step": 7, + "cached_positions": 135, + "cache_bytes": 829440 + }, + { + "step": 8, + "cached_positions": 136, + "cache_bytes": 835584 + }, + { + "step": 9, + "cached_positions": 137, + "cache_bytes": 841728 + }, + { + "step": 10, + "cached_positions": 138, + "cache_bytes": 847872 + }, + { + "step": 11, + "cached_positions": 139, + "cache_bytes": 854016 + }, + { + "step": 12, + "cached_positions": 140, + "cache_bytes": 860160 + }, + { + "step": 13, + "cached_positions": 141, + "cache_bytes": 866304 + }, + { + "step": 14, + "cached_positions": 142, + "cache_bytes": 872448 + }, + { + "step": 15, + "cached_positions": 143, + "cache_bytes": 878592 + }, + { + "step": 16, + "cached_positions": 144, + "cache_bytes": 884736 + }, + { + "step": 17, + "cached_positions": 145, + "cache_bytes": 890880 + }, + { + "step": 18, + "cached_positions": 146, + "cache_bytes": 897024 + }, + { + "step": 19, + "cached_positions": 147, + "cache_bytes": 903168 + }, + { + "step": 20, + "cached_positions": 148, + "cache_bytes": 909312 + }, + { + "step": 21, + "cached_positions": 149, + "cache_bytes": 915456 + }, + { + "step": 22, + "cached_positions": 150, + "cache_bytes": 921600 + }, + { + "step": 23, + "cached_positions": 151, + "cache_bytes": 927744 + }, + { + "step": 24, + "cached_positions": 152, + "cache_bytes": 933888 + }, + { + "step": 25, + "cached_positions": 153, + "cache_bytes": 940032 + }, + { + "step": 26, + "cached_positions": 154, + "cache_bytes": 946176 + }, + { + "step": 27, + "cached_positions": 155, + "cache_bytes": 952320 + }, + { + "step": 28, + "cached_positions": 156, + "cache_bytes": 958464 + }, + { + "step": 29, + "cached_positions": 157, + "cache_bytes": 964608 + }, + { + "step": 30, + "cached_positions": 158, + "cache_bytes": 970752 + }, + { + "step": 31, + "cached_positions": 159, + "cache_bytes": 976896 + }, + { + "step": 32, + "cached_positions": 160, + "cache_bytes": 983040 + }, + { + "step": 33, + "cached_positions": 161, + "cache_bytes": 989184 + }, + { + "step": 34, + "cached_positions": 162, + "cache_bytes": 995328 + }, + { + "step": 35, + "cached_positions": 163, + "cache_bytes": 1001472 + }, + { + "step": 36, + "cached_positions": 164, + "cache_bytes": 1007616 + }, + { + "step": 37, + "cached_positions": 165, + "cache_bytes": 1013760 + }, + { + "step": 38, + "cached_positions": 166, + "cache_bytes": 1019904 + }, + { + "step": 39, + "cached_positions": 167, + "cache_bytes": 1026048 + }, + { + "step": 40, + "cached_positions": 168, + "cache_bytes": 1032192 + }, + { + "step": 41, + "cached_positions": 169, + "cache_bytes": 1038336 + }, + { + "step": 42, + "cached_positions": 170, + "cache_bytes": 1044480 + }, + { + "step": 43, + "cached_positions": 171, + "cache_bytes": 1050624 + }, + { + "step": 44, + "cached_positions": 172, + "cache_bytes": 1056768 + }, + { + "step": 45, + "cached_positions": 173, + "cache_bytes": 1062912 + }, + { + "step": 46, + "cached_positions": 174, + "cache_bytes": 1069056 + }, + { + "step": 47, + "cached_positions": 175, + "cache_bytes": 1075200 + }, + { + "step": 48, + "cached_positions": 176, + "cache_bytes": 1081344 + }, + { + "step": 49, + "cached_positions": 177, + "cache_bytes": 1087488 + }, + { + "step": 50, + "cached_positions": 178, + "cache_bytes": 1093632 + }, + { + "step": 51, + "cached_positions": 179, + "cache_bytes": 1099776 + }, + { + "step": 52, + "cached_positions": 180, + "cache_bytes": 1105920 + }, + { + "step": 53, + "cached_positions": 181, + "cache_bytes": 1112064 + }, + { + "step": 54, + "cached_positions": 182, + "cache_bytes": 1118208 + }, + { + "step": 55, + "cached_positions": 183, + "cache_bytes": 1124352 + }, + { + "step": 56, + "cached_positions": 184, + "cache_bytes": 1130496 + }, + { + "step": 57, + "cached_positions": 185, + "cache_bytes": 1136640 + }, + { + "step": 58, + "cached_positions": 186, + "cache_bytes": 1142784 + }, + { + "step": 59, + "cached_positions": 187, + "cache_bytes": 1148928 + }, + { + "step": 60, + "cached_positions": 188, + "cache_bytes": 1155072 + }, + { + "step": 61, + "cached_positions": 189, + "cache_bytes": 1161216 + }, + { + "step": 62, + "cached_positions": 190, + "cache_bytes": 1167360 + }, + { + "step": 63, + "cached_positions": 191, + "cache_bytes": 1173504 + } + ] + }, + { + "batch": 8, + "prompt_len": 128, + "steps": [ + { + "step": 0, + "cached_positions": 128, + "cache_bytes": 6291456 + }, + { + "step": 1, + "cached_positions": 129, + "cache_bytes": 6340608 + }, + { + "step": 2, + "cached_positions": 130, + "cache_bytes": 6389760 + }, + { + "step": 3, + "cached_positions": 131, + "cache_bytes": 6438912 + }, + { + "step": 4, + "cached_positions": 132, + "cache_bytes": 6488064 + }, + { + "step": 5, + "cached_positions": 133, + "cache_bytes": 6537216 + }, + { + "step": 6, + "cached_positions": 134, + "cache_bytes": 6586368 + }, + { + "step": 7, + "cached_positions": 135, + "cache_bytes": 6635520 + }, + { + "step": 8, + "cached_positions": 136, + "cache_bytes": 6684672 + }, + { + "step": 9, + "cached_positions": 137, + "cache_bytes": 6733824 + }, + { + "step": 10, + "cached_positions": 138, + "cache_bytes": 6782976 + }, + { + "step": 11, + "cached_positions": 139, + "cache_bytes": 6832128 + }, + { + "step": 12, + "cached_positions": 140, + "cache_bytes": 6881280 + }, + { + "step": 13, + "cached_positions": 141, + "cache_bytes": 6930432 + }, + { + "step": 14, + "cached_positions": 142, + "cache_bytes": 6979584 + }, + { + "step": 15, + "cached_positions": 143, + "cache_bytes": 7028736 + }, + { + "step": 16, + "cached_positions": 144, + "cache_bytes": 7077888 + }, + { + "step": 17, + "cached_positions": 145, + "cache_bytes": 7127040 + }, + { + "step": 18, + "cached_positions": 146, + "cache_bytes": 7176192 + }, + { + "step": 19, + "cached_positions": 147, + "cache_bytes": 7225344 + }, + { + "step": 20, + "cached_positions": 148, + "cache_bytes": 7274496 + }, + { + "step": 21, + "cached_positions": 149, + "cache_bytes": 7323648 + }, + { + "step": 22, + "cached_positions": 150, + "cache_bytes": 7372800 + }, + { + "step": 23, + "cached_positions": 151, + "cache_bytes": 7421952 + }, + { + "step": 24, + "cached_positions": 152, + "cache_bytes": 7471104 + }, + { + "step": 25, + "cached_positions": 153, + "cache_bytes": 7520256 + }, + { + "step": 26, + "cached_positions": 154, + "cache_bytes": 7569408 + }, + { + "step": 27, + "cached_positions": 155, + "cache_bytes": 7618560 + }, + { + "step": 28, + "cached_positions": 156, + "cache_bytes": 7667712 + }, + { + "step": 29, + "cached_positions": 157, + "cache_bytes": 7716864 + }, + { + "step": 30, + "cached_positions": 158, + "cache_bytes": 7766016 + }, + { + "step": 31, + "cached_positions": 159, + "cache_bytes": 7815168 + }, + { + "step": 32, + "cached_positions": 160, + "cache_bytes": 7864320 + }, + { + "step": 33, + "cached_positions": 161, + "cache_bytes": 7913472 + }, + { + "step": 34, + "cached_positions": 162, + "cache_bytes": 7962624 + }, + { + "step": 35, + "cached_positions": 163, + "cache_bytes": 8011776 + }, + { + "step": 36, + "cached_positions": 164, + "cache_bytes": 8060928 + }, + { + "step": 37, + "cached_positions": 165, + "cache_bytes": 8110080 + }, + { + "step": 38, + "cached_positions": 166, + "cache_bytes": 8159232 + }, + { + "step": 39, + "cached_positions": 167, + "cache_bytes": 8208384 + }, + { + "step": 40, + "cached_positions": 168, + "cache_bytes": 8257536 + }, + { + "step": 41, + "cached_positions": 169, + "cache_bytes": 8306688 + }, + { + "step": 42, + "cached_positions": 170, + "cache_bytes": 8355840 + }, + { + "step": 43, + "cached_positions": 171, + "cache_bytes": 8404992 + }, + { + "step": 44, + "cached_positions": 172, + "cache_bytes": 8454144 + }, + { + "step": 45, + "cached_positions": 173, + "cache_bytes": 8503296 + }, + { + "step": 46, + "cached_positions": 174, + "cache_bytes": 8552448 + }, + { + "step": 47, + "cached_positions": 175, + "cache_bytes": 8601600 + }, + { + "step": 48, + "cached_positions": 176, + "cache_bytes": 8650752 + }, + { + "step": 49, + "cached_positions": 177, + "cache_bytes": 8699904 + }, + { + "step": 50, + "cached_positions": 178, + "cache_bytes": 8749056 + }, + { + "step": 51, + "cached_positions": 179, + "cache_bytes": 8798208 + }, + { + "step": 52, + "cached_positions": 180, + "cache_bytes": 8847360 + }, + { + "step": 53, + "cached_positions": 181, + "cache_bytes": 8896512 + }, + { + "step": 54, + "cached_positions": 182, + "cache_bytes": 8945664 + }, + { + "step": 55, + "cached_positions": 183, + "cache_bytes": 8994816 + }, + { + "step": 56, + "cached_positions": 184, + "cache_bytes": 9043968 + }, + { + "step": 57, + "cached_positions": 185, + "cache_bytes": 9093120 + }, + { + "step": 58, + "cached_positions": 186, + "cache_bytes": 9142272 + }, + { + "step": 59, + "cached_positions": 187, + "cache_bytes": 9191424 + }, + { + "step": 60, + "cached_positions": 188, + "cache_bytes": 9240576 + }, + { + "step": 61, + "cached_positions": 189, + "cache_bytes": 9289728 + }, + { + "step": 62, + "cached_positions": 190, + "cache_bytes": 9338880 + }, + { + "step": 63, + "cached_positions": 191, + "cache_bytes": 9388032 + } + ] + }, + { + "batch": 1, + "prompt_len": 32, + "steps": [ + { + "step": 0, + "cached_positions": 32, + "cache_bytes": 196608 + }, + { + "step": 1, + "cached_positions": 33, + "cache_bytes": 202752 + }, + { + "step": 2, + "cached_positions": 34, + "cache_bytes": 208896 + }, + { + "step": 3, + "cached_positions": 35, + "cache_bytes": 215040 + }, + { + "step": 4, + "cached_positions": 36, + "cache_bytes": 221184 + }, + { + "step": 5, + "cached_positions": 37, + "cache_bytes": 227328 + }, + { + "step": 6, + "cached_positions": 38, + "cache_bytes": 233472 + }, + { + "step": 7, + "cached_positions": 39, + "cache_bytes": 239616 + }, + { + "step": 8, + "cached_positions": 40, + "cache_bytes": 245760 + }, + { + "step": 9, + "cached_positions": 41, + "cache_bytes": 251904 + }, + { + "step": 10, + "cached_positions": 42, + "cache_bytes": 258048 + }, + { + "step": 11, + "cached_positions": 43, + "cache_bytes": 264192 + }, + { + "step": 12, + "cached_positions": 44, + "cache_bytes": 270336 + }, + { + "step": 13, + "cached_positions": 45, + "cache_bytes": 276480 + }, + { + "step": 14, + "cached_positions": 46, + "cache_bytes": 282624 + }, + { + "step": 15, + "cached_positions": 47, + "cache_bytes": 288768 + }, + { + "step": 16, + "cached_positions": 48, + "cache_bytes": 294912 + }, + { + "step": 17, + "cached_positions": 49, + "cache_bytes": 301056 + }, + { + "step": 18, + "cached_positions": 50, + "cache_bytes": 307200 + }, + { + "step": 19, + "cached_positions": 51, + "cache_bytes": 313344 + }, + { + "step": 20, + "cached_positions": 52, + "cache_bytes": 319488 + }, + { + "step": 21, + "cached_positions": 53, + "cache_bytes": 325632 + }, + { + "step": 22, + "cached_positions": 54, + "cache_bytes": 331776 + }, + { + "step": 23, + "cached_positions": 55, + "cache_bytes": 337920 + }, + { + "step": 24, + "cached_positions": 56, + "cache_bytes": 344064 + }, + { + "step": 25, + "cached_positions": 57, + "cache_bytes": 350208 + }, + { + "step": 26, + "cached_positions": 58, + "cache_bytes": 356352 + }, + { + "step": 27, + "cached_positions": 59, + "cache_bytes": 362496 + }, + { + "step": 28, + "cached_positions": 60, + "cache_bytes": 368640 + }, + { + "step": 29, + "cached_positions": 61, + "cache_bytes": 374784 + }, + { + "step": 30, + "cached_positions": 62, + "cache_bytes": 380928 + }, + { + "step": 31, + "cached_positions": 63, + "cache_bytes": 387072 + }, + { + "step": 32, + "cached_positions": 64, + "cache_bytes": 393216 + }, + { + "step": 33, + "cached_positions": 65, + "cache_bytes": 399360 + }, + { + "step": 34, + "cached_positions": 66, + "cache_bytes": 405504 + }, + { + "step": 35, + "cached_positions": 67, + "cache_bytes": 411648 + }, + { + "step": 36, + "cached_positions": 68, + "cache_bytes": 417792 + }, + { + "step": 37, + "cached_positions": 69, + "cache_bytes": 423936 + }, + { + "step": 38, + "cached_positions": 70, + "cache_bytes": 430080 + }, + { + "step": 39, + "cached_positions": 71, + "cache_bytes": 436224 + }, + { + "step": 40, + "cached_positions": 72, + "cache_bytes": 442368 + }, + { + "step": 41, + "cached_positions": 73, + "cache_bytes": 448512 + }, + { + "step": 42, + "cached_positions": 74, + "cache_bytes": 454656 + }, + { + "step": 43, + "cached_positions": 75, + "cache_bytes": 460800 + }, + { + "step": 44, + "cached_positions": 76, + "cache_bytes": 466944 + }, + { + "step": 45, + "cached_positions": 77, + "cache_bytes": 473088 + }, + { + "step": 46, + "cached_positions": 78, + "cache_bytes": 479232 + }, + { + "step": 47, + "cached_positions": 79, + "cache_bytes": 485376 + }, + { + "step": 48, + "cached_positions": 80, + "cache_bytes": 491520 + }, + { + "step": 49, + "cached_positions": 81, + "cache_bytes": 497664 + }, + { + "step": 50, + "cached_positions": 82, + "cache_bytes": 503808 + }, + { + "step": 51, + "cached_positions": 83, + "cache_bytes": 509952 + }, + { + "step": 52, + "cached_positions": 84, + "cache_bytes": 516096 + }, + { + "step": 53, + "cached_positions": 85, + "cache_bytes": 522240 + }, + { + "step": 54, + "cached_positions": 86, + "cache_bytes": 528384 + }, + { + "step": 55, + "cached_positions": 87, + "cache_bytes": 534528 + }, + { + "step": 56, + "cached_positions": 88, + "cache_bytes": 540672 + }, + { + "step": 57, + "cached_positions": 89, + "cache_bytes": 546816 + }, + { + "step": 58, + "cached_positions": 90, + "cache_bytes": 552960 + }, + { + "step": 59, + "cached_positions": 91, + "cache_bytes": 559104 + }, + { + "step": 60, + "cached_positions": 92, + "cache_bytes": 565248 + }, + { + "step": 61, + "cached_positions": 93, + "cache_bytes": 571392 + }, + { + "step": 62, + "cached_positions": 94, + "cache_bytes": 577536 + }, + { + "step": 63, + "cached_positions": 95, + "cache_bytes": 583680 + }, + { + "step": 64, + "cached_positions": 96, + "cache_bytes": 589824 + }, + { + "step": 65, + "cached_positions": 97, + "cache_bytes": 595968 + }, + { + "step": 66, + "cached_positions": 98, + "cache_bytes": 602112 + }, + { + "step": 67, + "cached_positions": 99, + "cache_bytes": 608256 + }, + { + "step": 68, + "cached_positions": 100, + "cache_bytes": 614400 + }, + { + "step": 69, + "cached_positions": 101, + "cache_bytes": 620544 + }, + { + "step": 70, + "cached_positions": 102, + "cache_bytes": 626688 + }, + { + "step": 71, + "cached_positions": 103, + "cache_bytes": 632832 + }, + { + "step": 72, + "cached_positions": 104, + "cache_bytes": 638976 + }, + { + "step": 73, + "cached_positions": 105, + "cache_bytes": 645120 + }, + { + "step": 74, + "cached_positions": 106, + "cache_bytes": 651264 + }, + { + "step": 75, + "cached_positions": 107, + "cache_bytes": 657408 + }, + { + "step": 76, + "cached_positions": 108, + "cache_bytes": 663552 + }, + { + "step": 77, + "cached_positions": 109, + "cache_bytes": 669696 + }, + { + "step": 78, + "cached_positions": 110, + "cache_bytes": 675840 + }, + { + "step": 79, + "cached_positions": 111, + "cache_bytes": 681984 + }, + { + "step": 80, + "cached_positions": 112, + "cache_bytes": 688128 + }, + { + "step": 81, + "cached_positions": 113, + "cache_bytes": 694272 + }, + { + "step": 82, + "cached_positions": 114, + "cache_bytes": 700416 + }, + { + "step": 83, + "cached_positions": 115, + "cache_bytes": 706560 + }, + { + "step": 84, + "cached_positions": 116, + "cache_bytes": 712704 + }, + { + "step": 85, + "cached_positions": 117, + "cache_bytes": 718848 + }, + { + "step": 86, + "cached_positions": 118, + "cache_bytes": 724992 + }, + { + "step": 87, + "cached_positions": 119, + "cache_bytes": 731136 + }, + { + "step": 88, + "cached_positions": 120, + "cache_bytes": 737280 + }, + { + "step": 89, + "cached_positions": 121, + "cache_bytes": 743424 + }, + { + "step": 90, + "cached_positions": 122, + "cache_bytes": 749568 + }, + { + "step": 91, + "cached_positions": 123, + "cache_bytes": 755712 + }, + { + "step": 92, + "cached_positions": 124, + "cache_bytes": 761856 + }, + { + "step": 93, + "cached_positions": 125, + "cache_bytes": 768000 + }, + { + "step": 94, + "cached_positions": 126, + "cache_bytes": 774144 + }, + { + "step": 95, + "cached_positions": 127, + "cache_bytes": 780288 + }, + { + "step": 96, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 97, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 98, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 99, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 100, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 101, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 102, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 103, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 104, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 105, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 106, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 107, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 108, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 109, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 110, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 111, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 112, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 113, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 114, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 115, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 116, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 117, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 118, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 119, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 120, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 121, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 122, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 123, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 124, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 125, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 126, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 127, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 128, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 129, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 130, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 131, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 132, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 133, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 134, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 135, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 136, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 137, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 138, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 139, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 140, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 141, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 142, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 143, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 144, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 145, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 146, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 147, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 148, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 149, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 150, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 151, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 152, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 153, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 154, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 155, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 156, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 157, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 158, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 159, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 160, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 161, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 162, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 163, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 164, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 165, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 166, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 167, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 168, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 169, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 170, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 171, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 172, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 173, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 174, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 175, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 176, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 177, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 178, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 179, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 180, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 181, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 182, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 183, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 184, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 185, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 186, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 187, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 188, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 189, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 190, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 191, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 192, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 193, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 194, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 195, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 196, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 197, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 198, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 199, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 200, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 201, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 202, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 203, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 204, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 205, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 206, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 207, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 208, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 209, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 210, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 211, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 212, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 213, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 214, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 215, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 216, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 217, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 218, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 219, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 220, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 221, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 222, + "cached_positions": 128, + "cache_bytes": 786432 + }, + { + "step": 223, + "cached_positions": 128, + "cache_bytes": 786432 + } + ] + } + ] + } +} diff --git a/experiments/records/kv_cache/20261022T053346Z_kv_mps_float32_57be26d.json b/experiments/records/kv_cache/20261022T053346Z_kv_mps_float32_57be26d.json new file mode 100644 index 0000000..31f2f64 --- /dev/null +++ b/experiments/records/kv_cache/20261022T053346Z_kv_mps_float32_57be26d.json @@ -0,0 +1,35862 @@ +{ + "record_version": 2, + "created_utc": "20261022T053346Z", + "experiment": "kv_cache_scaling", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "mps", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "57be26dc620ddc37c1928c7d7a674e4ff0d3d39f", + "branch": "experiments/sdpa-training-v2", + "dirty": false + }, + "command": "python scripts/benchmark_kv_scaling.py --device mps --dtype float32" + }, + "settings": { + "dtype": "float32", + "position": "learned", + "prompt_lens": [ + 1, + 16, + 128, + 512, + 1024 + ], + "batch_sizes": [ + 1, + 8 + ], + "new_tokens": 64, + "warmup": 1, + "repeats": 5, + "seed": 0, + "modes": { + "cached": { + "attention_backend": "sdpa", + "use_cache": true + }, + "uncached_sdpa": { + "attention_backend": "sdpa", + "use_cache": false + }, + "uncached_mha": { + "attention_backend": "mha", + "use_cache": false + } + }, + "weights": "seeded random initialization (timing does not depend on trained values)", + "grid_model_config": { + "vocab_size": 256, + "context_length": 1088, + "embedding_dim": 192, + "num_layers": 4, + "num_heads": 6, + "dropout": 0.0, + "positional_encoding": "learned", + "norm": "rmsnorm", + "activation": "swiglu", + "rope_theta": 10000.0 + }, + "window_model_config": { + "vocab_size": 256, + "context_length": 128, + "embedding_dim": 192, + "num_layers": 4, + "num_heads": 6, + "dropout": 0.0, + "positional_encoding": "learned", + "norm": "rmsnorm", + "activation": "swiglu", + "rope_theta": 10000.0 + }, + "decoding": "greedy argmax; step 0 is the prompt prefill", + "timer": "time.perf_counter", + "synchronization": "device synchronize before and after every step (no-op on CPU); backend/mode order rotated each repeat", + "units": { + "raw": "seconds per step, one list per measured repeat", + "summaries": "milliseconds (ms fields) or tokens/second" + } + }, + "results": { + "grid": [ + { + "batch": 1, + "prompt_len": 1, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_mha": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 1.4654580008937046, + "p95": 1.5496746054850519, + "min": 1.4300829934654757, + "max": 1.5618330071447417, + "iqr": 0.03662500239443034, + "mean": 1.4845661993604153, + "cv": 0.03364811783580293 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.6781250014901161, + "p95": 1.784874900477007, + "min": 1.5408750041387975, + "max": 2.6302920014131814, + "iqr": 0.0826459945528768, + "mean": 1.6768103143447892, + "cv": 0.06132288390254612 + }, + "total_ms": { + "count": 5, + "median": 106.3483339821687, + "p95": 109.6495830337517, + "min": 106.12782699172385, + "max": 110.4504160466604, + "iqr": 0.2009989693760872, + "mean": 107.12361600308213, + "cv": 0.017395837114014198 + }, + "tokens_per_second": { + "count": 5, + "median": 601.7959812208323, + "p95": 602.9130617312363, + "min": 579.4455312233757, + "max": 603.0463622419302, + "iqr": 1.137454159758363, + "mean": 597.5820279806604, + "cv": 0.01700316796859964 + }, + "median_step_ms_by_index": [ + 1.4654580008937046, + 1.725708003505133, + 1.6922920040087774, + 1.7049580055754632, + 1.6070830024546012, + 1.6904169897316024, + 1.6075409948825836, + 1.6262080025626346, + 1.6934999875957146, + 1.6690829943399876, + 1.6923329967539757, + 1.6907499957596883, + 1.6909999976633117, + 1.6137500060722232, + 1.674125000135973, + 1.674458006164059, + 1.6825409984448925, + 1.683583002886735, + 1.6717079997761175, + 1.6542079974897206, + 1.625875011086464, + 1.6591669991612434, + 1.6781250014901161, + 1.6907920071389526, + 1.6715410019969568, + 1.6408329975092784, + 1.6510419955011457, + 1.694874998065643, + 1.6777089913375676, + 1.6726660105632618, + 1.6883339994819835, + 1.7101669945986941, + 1.691832992946729, + 1.649708006880246, + 1.6842920013004914, + 1.6835420101415366, + 1.6844579949975014, + 1.6854589921422303, + 1.700375010841526, + 1.688958000158891, + 1.5876670076977462, + 1.788791996659711, + 1.6987919952953234, + 1.6907909885048866, + 1.702709007076919, + 1.6353329992853105, + 1.6800829907879233, + 1.6972919984254986, + 1.7094590002670884, + 1.7010419978760183, + 1.6954999882727861, + 1.6307079931721091, + 1.630124999792315, + 1.672209007665515, + 1.6938750050030649, + 1.6088749980553985, + 1.6875410074135289, + 1.7158749978989363, + 1.6627080040052533, + 1.6857089940458536, + 1.606916994205676, + 1.6860829928191379, + 1.6863749915501103, + 1.6754999960539863 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0015010409988462925, + 0.0017934579955181107, + 0.0016974579921225086, + 0.0017049580055754632, + 0.0015995419962564483, + 0.0017412919987691566, + 0.0016075409948825836, + 0.0015842919965507463, + 0.0016938750050030649, + 0.0015679170028306544, + 0.0017505410069134086, + 0.0017038749938365072, + 0.0015793750062584877, + 0.0015880409919191152, + 0.0015568750095553696, + 0.001674458006164059, + 0.0016825409984448925, + 0.0016602499963482842, + 0.0016104160022223368, + 0.0016542079974897206, + 0.0016572500026086345, + 0.0016307500045513734, + 0.0017050830065272748, + 0.001694709004368633, + 0.0015721249947091565, + 0.0016408329975092784, + 0.0016787909989943728, + 0.001694874998065643, + 0.0016777089913375676, + 0.0016697919927537441, + 0.0016530829889234155, + 0.0015619170007994398, + 0.0017199579888256267, + 0.0015617919998476282, + 0.0015585840010317042, + 0.001593708002474159, + 0.0019492090068524703, + 0.0015859580016694963, + 0.0017097080126404762, + 0.0017282080079894513, + 0.0015876670076977462, + 0.0019042499916395172, + 0.0016987919952953234, + 0.0016907909885048866, + 0.001685708004515618, + 0.0016353329992853105, + 0.0016800829907879233, + 0.0017020000086631626, + 0.0017094590002670884, + 0.001576916009071283, + 0.0016954999882727861, + 0.0016307079931721091, + 0.0015778749948367476, + 0.001672209007665515, + 0.0016735829995013773, + 0.0015927499916870147, + 0.0016059580084402114, + 0.001724750007269904, + 0.0016085000097518787, + 0.0016922080103540793, + 0.001606916994205676, + 0.0016860829928191379, + 0.0016754160023992881, + 0.0016183749976335093 + ], + [ + 0.0014644159964518622, + 0.001893958993605338, + 0.0016922920040087774, + 0.0016739169950596988, + 0.001665791991399601, + 0.001598666000063531, + 0.0015727909922134131, + 0.0016503749939147383, + 0.0016366670024581254, + 0.0017288749950239435, + 0.0016606660065008327, + 0.0017656670097494498, + 0.0016977920022327453, + 0.0016385000053560361, + 0.0016972919984254986, + 0.0015661250072298571, + 0.0017335829907096922, + 0.001683583002886735, + 0.001613749991520308, + 0.0016354170074919239, + 0.0016670420009177178, + 0.0017511249898234382, + 0.0016781250014901161, + 0.0016652500053169206, + 0.001711040997179225, + 0.001611250001587905, + 0.0016510419955011457, + 0.001697917003184557, + 0.0016938339977059513, + 0.0017081249970942736, + 0.0016719160048523918, + 0.0017101669945986941, + 0.0015982500044628978, + 0.001672624988714233, + 0.0017037079960573465, + 0.0016891660052351654, + 0.0016844579949975014, + 0.0016854589921422303, + 0.001686333998804912, + 0.001586083002621308, + 0.001572583001689054, + 0.0017016670026350766, + 0.0017017089994624257, + 0.0016969580028671771, + 0.001702709007076919, + 0.0016071249992819503, + 0.0016345409967470914, + 0.001617542002350092, + 0.0016207910084631294, + 0.0016982499946607277, + 0.0017041250102920458, + 0.0017157500114990398, + 0.001583416000357829, + 0.0015965830098139122, + 0.0016938750050030649, + 0.0015537089930148795, + 0.0015408750041387975, + 0.0017158749978989363, + 0.0016627080040052533, + 0.0016857089940458536, + 0.0015766669966978952, + 0.0017087089945562184, + 0.0016863749915501103, + 0.0016749579954193905 + ], + [ + 0.0014300829934654757, + 0.001725708003505133, + 0.001596999994944781, + 0.0017154999950435013, + 0.0016070830024546012, + 0.0016875000001164153, + 0.0016754169919295236, + 0.001689749988145195, + 0.0016934999875957146, + 0.0015529170050285757, + 0.0015953750116750598, + 0.001675999999861233, + 0.0016570839943597093, + 0.0016634999919915572, + 0.001677708001807332, + 0.0016776669945102185, + 0.0015856249956414104, + 0.00169237500813324, + 0.0016717079997761175, + 0.0016869160026544705, + 0.001612207997823134, + 0.0017369169945595786, + 0.0016245000006165355, + 0.0017056249926099554, + 0.0016841250035213307, + 0.0016767919878475368, + 0.0016938339977059513, + 0.0016787500062491745, + 0.001575583009980619, + 0.0015795409999554977, + 0.0016971250006463379, + 0.0017540000117151067, + 0.001691832992946729, + 0.0016536250041099265, + 0.0016842920013004914, + 0.0016186249995371327, + 0.0017192090017488226, + 0.0016984170069918036, + 0.001700375010841526, + 0.001688958000158891, + 0.0015842499997233972, + 0.0016423330089310184, + 0.0015988329978426918, + 0.001680000001215376, + 0.0017029160080710426, + 0.0017034170014085248, + 0.001714916987111792, + 0.0016972919984254986, + 0.0016411249962402508, + 0.0017010419978760183, + 0.0015858340047998354, + 0.0016820419987197965, + 0.0016862499905982986, + 0.0016781250014901161, + 0.0016370000084862113, + 0.0016088749980553985, + 0.0016875410074135289, + 0.001672959013376385, + 0.0016579579969402403, + 0.0016848749946802855, + 0.0015644590021111071, + 0.001600000003236346, + 0.0017010420124279335, + 0.0016994170000543818 + ], + [ + 0.0014654580008937046, + 0.0016812909889267758, + 0.0016184170090127736, + 0.0017930000030901283, + 0.0016018329915823415, + 0.0016904169897316024, + 0.0016181250102818012, + 0.0016262080025626346, + 0.0015817089879419655, + 0.0016690829943399876, + 0.0017510000034235418, + 0.0016907499957596883, + 0.0016949589917203411, + 0.001549125008750707, + 0.0015749580052215606, + 0.001661875008721836, + 0.001676416999544017, + 0.0016742080042604357, + 0.001709374992060475, + 0.001705125003354624, + 0.001625875011086464, + 0.0016229159955400974, + 0.0015641249920008704, + 0.0016124999965541065, + 0.0016661660047248006, + 0.0015832920034881681, + 0.0015644999948563054, + 0.0016960410139290616, + 0.001583416000357829, + 0.0020416250044945627, + 0.0017480000096838921, + 0.0016772920062066987, + 0.0017854580073617399, + 0.001649708006880246, + 0.0016985830006888136, + 0.0016960420034592971, + 0.0016215000068768859, + 0.0017403749952791259, + 0.0019027499947696924, + 0.0026302920014131814, + 0.002228166995337233, + 0.0020096249936614186, + 0.001950833000591956, + 0.0020065839926246554, + 0.0021682080114260316, + 0.002153916997485794, + 0.001673291000770405, + 0.001741875006700866, + 0.0017193749954458326, + 0.0017359170014970005, + 0.0015604999935021624, + 0.0015883330052020028, + 0.001630124999792315, + 0.001740584004437551, + 0.0017059169913409278, + 0.0017176250112242997, + 0.0016965420072665438, + 0.0017231670062756166, + 0.0016882920026546344, + 0.0016309579950757325, + 0.001684750008280389, + 0.0015971249958965927, + 0.0016794169932836667, + 0.0016754999960539863 + ], + [ + 0.0015618330071447417, + 0.0017231249948963523, + 0.00171162499464117, + 0.0016690839984221384, + 0.0016709579940652475, + 0.0016967089904937893, + 0.0015707499987911433, + 0.0015938750002533197, + 0.0017846249975264072, + 0.0016787919885246083, + 0.0016923329967539757, + 0.0015809580072527751, + 0.0016909999976633117, + 0.0016137500060722232, + 0.001674125000135973, + 0.001684582995949313, + 0.0016847079969011247, + 0.0016846660000737756, + 0.001682709000306204, + 0.0015818329993635416, + 0.0015786250005476177, + 0.0016591669991612434, + 0.0017101669945986941, + 0.0016907920071389526, + 0.0016715410019969568, + 0.0016616669890936464, + 0.0016153750038938597, + 0.0015916249976726249, + 0.0016984160029096529, + 0.0016726660105632618, + 0.0016883339994819835, + 0.0017760409973561764, + 0.0016861250041984022, + 0.0015703749959357083, + 0.0016711250063963234, + 0.0016835420101415366, + 0.0015882089937804267, + 0.001610916995559819, + 0.0015930410008877516, + 0.0015758330118842423, + 0.0016599170048721135, + 0.001788791996659711, + 0.001559041003929451, + 0.0016336249973392114, + 0.0016449999966425821, + 0.0015871669893385842, + 0.0016896250017452985, + 0.0015770830068504438, + 0.0017493750056019053, + 0.0017143340082839131, + 0.0017113339999923483, + 0.0016262499993899837, + 0.0017774589941836894, + 0.0015766249998705462, + 0.0017143750010291114, + 0.0016793340037111193, + 0.0017044999985955656, + 0.0015579589962726459, + 0.001698124993708916, + 0.0016934159939410165, + 0.0016968330019153655, + 0.001702458001091145, + 0.0016983749956125394, + 0.001681708003161475 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + ] + } + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 1.454749988624826, + "p95": 1.5086833969689906, + "min": 1.4140419953037053, + "max": 1.5096249990165234, + "iqr": 0.05216698627918959, + "mean": 1.4672167948447168, + "cv": 0.027288109888805253 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.4573329972336069, + "p95": 1.5918374949251302, + "min": 1.304792007431388, + "max": 1.738416001899168, + "iqr": 0.11464600538602099, + "mean": 1.4635725974729876, + "cv": 0.057536720219284944 + }, + "total_ms": { + "count": 5, + "median": 93.78691602614708, + "p95": 93.87382422573864, + "min": 93.13196010771208, + "max": 93.87570901890285, + "iqr": 0.16570308071095496, + "mean": 93.67229043564294, + "cv": 0.0033116016097632824 + }, + "tokens_per_second": { + "count": 5, + "median": 682.3979581774208, + "p95": 686.3629106706097, + "min": 681.7525073191504, + "max": 687.1969614510485, + "iqr": 1.205753797379998, + "mean": 683.2390176495898, + "cv": 0.0033245685607934 + }, + "median_step_ms_by_index": [ + 1.454749988624826, + 1.4755420124856755, + 1.447124988771975, + 1.5694169997004792, + 1.5662920050090179, + 1.6085420065792277, + 1.61929200112354, + 1.6888329992070794, + 1.452833996154368, + 1.4304579963209108, + 1.3960409996798262, + 1.3853340060450137, + 1.451250005629845, + 1.4835830079391599, + 1.4187920023687184, + 1.3815830025123432, + 1.4057090011192486, + 1.374749990645796, + 1.4099999971222132, + 1.4033329935045913, + 1.3793750113109127, + 1.4450419985223562, + 1.3814999983878806, + 1.3451249978970736, + 1.3915839954279363, + 1.3777909916825593, + 1.3622080005006865, + 1.3722919975407422, + 1.3983750104671344, + 1.4250829990487546, + 1.404208000167273, + 1.4001249946886674, + 1.4968330069677904, + 1.4861670060781762, + 1.4353750011650845, + 1.380583009449765, + 1.4439579972531646, + 1.5018330013845116, + 1.4496660005534068, + 1.4983750006649643, + 1.42058300843928, + 1.4638749998994172, + 1.483667001593858, + 1.5042919985717162, + 1.4972080098232254, + 1.5014160017017275, + 1.4697079895995557, + 1.5376249939436093, + 1.464957997086458, + 1.4785420062253252, + 1.4928750024409965, + 1.5526250062976032, + 1.5642080106772482, + 1.5292080061044544, + 1.531958012492396, + 1.5413340006489307, + 1.5394170040963218, + 1.548458996694535, + 1.498208992416039, + 1.4745420048711821, + 1.5176250017248094, + 1.556957999127917, + 1.5056249976623803, + 1.4598749985452741 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0015049169887788594, + 0.0014755420124856755, + 0.0014400829968508333, + 0.0015837079990888014, + 0.001574875001097098, + 0.0016085420065792277, + 0.00161929200112354, + 0.0017000829975586385, + 0.0014704999921377748, + 0.0014196660049492493, + 0.0013147920108167455, + 0.0013117500056978315, + 0.001451250005629845, + 0.0014835830079391599, + 0.001423458001227118, + 0.0013187089934945107, + 0.0015880830032983795, + 0.0013382920005824417, + 0.0014099999971222132, + 0.0014788749977014959, + 0.0013092080043861642, + 0.0014798330084886402, + 0.0014171660004649311, + 0.0013299170095706359, + 0.0013915839954279363, + 0.0013122910022502765, + 0.0013622080005006865, + 0.0013142090028850362, + 0.0013983750104671344, + 0.0014298330061137676, + 0.0013491249992512167, + 0.0014001249946886674, + 0.0015029999922262505, + 0.0014031250029802322, + 0.001378833010676317, + 0.0014743750070920214, + 0.0014439579972531646, + 0.0014210420049494132, + 0.0014934580103727058, + 0.0015115419955691323, + 0.00142058300843928, + 0.0014249160012695938, + 0.001483667001593858, + 0.0015042919985717162, + 0.0015046659973450005, + 0.0014567089965566993, + 0.001531499990960583, + 0.0015376249939436093, + 0.0014573329972336069, + 0.0014679159939987585, + 0.001554916991153732, + 0.0015526250062976032, + 0.0015887079935055226, + 0.0015459169953828678, + 0.0015010420029284433, + 0.0015603329957230017, + 0.0015202500071609393, + 0.001548458996694535, + 0.0014982089924160391, + 0.0014569170016329736, + 0.0015309999871533364, + 0.00157891699927859, + 0.0014649160002591088, + 0.0014562919968739152 + ], + [ + 0.001454749988624826, + 0.0015854170051170513, + 0.0015393749927170575, + 0.0015694169997004792, + 0.0014895829954184592, + 0.001519249999546446, + 0.0016654579958412796, + 0.001738416001899168, + 0.0014614590036217123, + 0.0014304579963209108, + 0.001436999998986721, + 0.0014545829908456653, + 0.0015459160058526322, + 0.001415874998201616, + 0.0014187920023687184, + 0.0013815830025123432, + 0.0013202080008341, + 0.001374749990645796, + 0.0014222500030882657, + 0.0014033329935045913, + 0.0013440420007100329, + 0.001447832997655496, + 0.0014078750100452453, + 0.0013451249978970736, + 0.0014190420042723417, + 0.00141891700332053, + 0.0013414580025710166, + 0.0013722919975407422, + 0.0013898339966544881, + 0.0014250829990487546, + 0.0013776249979855493, + 0.0014465840067714453, + 0.0014997499965829775, + 0.0014885419950587675, + 0.0014458329969784245, + 0.0015033329982543364, + 0.0014240000018617138, + 0.0014148749905871227, + 0.0013856250006938353, + 0.0014667920040665194, + 0.001504416999523528, + 0.001429167008609511, + 0.001489208996645175, + 0.0014989159972174093, + 0.0015254580066539347, + 0.0014115829981165007, + 0.0013917500036768615, + 0.0016097079933388159, + 0.001590500003658235, + 0.0014785420062253252, + 0.0014423329994315282, + 0.0014067919983062893, + 0.0015901250008028, + 0.001425750000635162, + 0.0014477079967036843, + 0.0015413340006489307, + 0.0015901250008028, + 0.0014687919901916757, + 0.0014537910028593615, + 0.001454124998417683, + 0.0015176250017248094, + 0.0014356660103658214, + 0.0014532079949276522, + 0.0015516249986831099 + ], + [ + 0.0015096249990165234, + 0.0014482919941656291, + 0.0014770830021006986, + 0.0015770000027259812, + 0.0015804579888936132, + 0.0016658749955240637, + 0.0015934999973978847, + 0.0016888329992070794, + 0.001452833996154368, + 0.0014970420015743002, + 0.0014437500067288056, + 0.0013657079980475828, + 0.0014044580020708963, + 0.001392042002407834, + 0.0013262499996926636, + 0.0013761669979430735, + 0.0014208749926183373, + 0.0013132500025676563, + 0.00138208300631959, + 0.001440207997802645, + 0.001410917000612244, + 0.0013262919965200126, + 0.0013150000013411045, + 0.0013650409964611754, + 0.0013082090008538216, + 0.0013777909916825593, + 0.0014231670065782964, + 0.0013535420002881438, + 0.0014459170051850379, + 0.0014404579997062683, + 0.0014205420011421666, + 0.0013505829992936924, + 0.0014595829998143017, + 0.0015189580008154735, + 0.0014333340077428147, + 0.001380583009449765, + 0.001480624996474944, + 0.0015018330013845116, + 0.0014496660005534068, + 0.0015119999879971147, + 0.001504833999206312, + 0.001475333992857486, + 0.001422125002136454, + 0.0015149590035434812, + 0.0014972080098232254, + 0.0015014160017017275, + 0.0014697079895995557, + 0.001516375006758608, + 0.0015718750073574483, + 0.0015600830083712935, + 0.0014663339970866218, + 0.0014401250082300976, + 0.0014827920094830915, + 0.0014814589958405122, + 0.001531958012492396, + 0.0014635830011684448, + 0.001520624995464459, + 0.0015565420035272837, + 0.0015691669977968559, + 0.0015626669919583946, + 0.0015572080010315403, + 0.001556957999127917, + 0.0015631249989382923, + 0.0014598749985452741 + ], + [ + 0.0014527500024996698, + 0.0014748749963473529, + 0.0013892499991925433, + 0.0014591249928344041, + 0.0015662920050090179, + 0.0017119590047514066, + 0.0015414579975185916, + 0.0015943750040605664, + 0.0014267500082496554, + 0.0014825420075794682, + 0.001334125001449138, + 0.0013853340060450137, + 0.0014937910018488765, + 0.0014849170111119747, + 0.0013564580003730953, + 0.001381750000291504, + 0.0014057090011192486, + 0.001381750000291504, + 0.0013935000024503097, + 0.001401333007379435, + 0.0014090829936321825, + 0.0013202090049162507, + 0.0013814999983878806, + 0.0013269580085761845, + 0.0013197920052334666, + 0.0013560410006903112, + 0.0013926250103395432, + 0.0014259999879868701, + 0.0013319999998202547, + 0.0013843749911757186, + 0.0014475829957518727, + 0.0013762919988948852, + 0.0014968330069677904, + 0.0014861670060781762, + 0.0014353750011650845, + 0.0013477909960784018, + 0.0014921670081093907, + 0.0015113749977899715, + 0.0014332080027088523, + 0.0014948340103728697, + 0.0013755409891018644, + 0.0014638749998994172, + 0.0014060419925954193, + 0.001363625007797964, + 0.0014852080057607964, + 0.0015238750056596473, + 0.001458458005799912, + 0.0014885419950587675, + 0.0014298750029411167, + 0.0015174169966485351, + 0.0014928750024409965, + 0.001556957999127917, + 0.0015642080106772482, + 0.0015622920036548749, + 0.0015566250076517463, + 0.001499457997852005, + 0.0015394170040963218, + 0.0015557919978164136, + 0.0015782919945195317, + 0.0014745420048711821, + 0.001413209000020288, + 0.0015220420027617365, + 0.00160283301374875, + 0.0014127079921308905 + ], + [ + 0.0014140419953037053, + 0.0015617500030202791, + 0.001447124988771975, + 0.0014997079997556284, + 0.0014860410010442138, + 0.0015193750004982576, + 0.001662250011577271, + 0.0015896249969955534, + 0.0014168329944368452, + 0.001421416993252933, + 0.0013960409996798262, + 0.001462875006836839, + 0.0014010409940965474, + 0.0015119590098038316, + 0.0014307080127764493, + 0.0014042920083738863, + 0.0013371660024859011, + 0.0013984999968670309, + 0.001419250009348616, + 0.0013550000003306195, + 0.0013793750113109127, + 0.0014450419985223562, + 0.0013200419925851747, + 0.0013872499985154718, + 0.0014206670020939782, + 0.0013863330095773563, + 0.001333791995421052, + 0.0013914580049458891, + 0.001433750003343448, + 0.0013554159959312528, + 0.001404208000167273, + 0.0014249170053517446, + 0.0013845409994246438, + 0.001377915992634371, + 0.0014609580102842301, + 0.0013783750036964193, + 0.001304792007431388, + 0.0015403750003315508, + 0.0015178750036284328, + 0.0014983750006649643, + 0.0013577499921666458, + 0.0016837080038385466, + 0.0015390000044135377, + 0.001504666986875236, + 0.001378500004648231, + 0.0016385000053560361, + 0.0015022500010672957, + 0.001540083991130814, + 0.001464957997086458, + 0.0014557499962393194, + 0.0015548749943263829, + 0.0015574580029351637, + 0.001434332996723242, + 0.0015292080061044544, + 0.001588416998856701, + 0.0015990410029189661, + 0.0015621659986209124, + 0.0014822079974692315, + 0.0014979580009821802, + 0.0015515410050284117, + 0.001492417010013014, + 0.0015911249938653782, + 0.0015056249976623803, + 0.0015742910036351532 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_mha": { + "first_token_ms": { + "count": 5, + "median": 1.569959000335075, + "p95": 1.5923417959129438, + "min": 1.5223329974105582, + "max": 1.595666995854117, + "iqr": 0.022999985958449543, + "mean": 1.5646081999875605, + "cv": 0.017683955078859177 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.5609169931849465, + "p95": 1.7394998954841867, + "min": 1.363250004942529, + "max": 1.982250003493391, + "iqr": 0.12968699593329802, + "mean": 1.5702173397469792, + "cv": 0.061887505792683685 + }, + "total_ms": { + "count": 5, + "median": 100.63437798817176, + "p95": 101.10078180441633, + "min": 99.63058603170794, + "max": 101.18299900204875, + "iqr": 0.5502860294654965, + "mean": 100.48830060404725, + "cv": 0.005868735514660635 + }, + "tokens_per_second": { + "count": 5, + "median": 635.9655743837593, + "p95": 641.6153573053497, + "min": 632.5173263415935, + "max": 642.373015648344, + "iqr": 3.487124950800421, + "mean": 636.9076478579282, + "cv": 0.005881096396965074 + }, + "median_step_ms_by_index": [ + 1.569959000335075, + 1.6441670013591647, + 1.6823750047478825, + 1.7094580107368529, + 1.7528749885968864, + 1.7060420068446547, + 1.7339580081170425, + 1.7923330015037209, + 1.526124993688427, + 1.5810000040801242, + 1.5163329953793436, + 1.4497920055873692, + 1.5405830054078251, + 1.5671669971197844, + 1.508249988546595, + 1.4452909963438287, + 1.4385829999810085, + 1.4328329998534173, + 1.5032909868750721, + 1.4584169985027984, + 1.4938329986762255, + 1.503457999206148, + 1.5134580025915056, + 1.4584590098820627, + 1.4772919967072085, + 1.5127910010050982, + 1.4730410039192066, + 1.4922499976819381, + 1.4548329927492887, + 1.4893329935148358, + 1.5246660041157156, + 1.5031250077299774, + 1.5117919974727556, + 1.5760840033181012, + 1.6453749994980171, + 1.5960830060066655, + 1.54974999895785, + 1.5094999980647117, + 1.5478749992325902, + 1.596000001882203, + 1.5874579985393211, + 1.549332999275066, + 1.5684579993830994, + 1.614709006389603, + 1.6022089985199273, + 1.5914999967208132, + 1.5803330024937168, + 1.5594589931424707, + 1.6641669935779646, + 1.6219169920077547, + 1.636708009755239, + 1.5970420063240454, + 1.5919580037007108, + 1.596041998709552, + 1.6579999937675893, + 1.6085409879451618, + 1.6657500091241673, + 1.6012499982025474, + 1.6052909923018888, + 1.6005420038709417, + 1.6631249891361222, + 1.6620420065009966, + 1.6512909933226183, + 1.6001670010155067 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.001569959000335075, + 0.0018064169998979196, + 0.0016931249992921948, + 0.0016857500013429672, + 0.0017221669986611232, + 0.0017060420068446547, + 0.0018265000107930973, + 0.0017293339915340766, + 0.0015276669873856008, + 0.0015810000040801242, + 0.0014591250073863193, + 0.001435291997040622, + 0.0015405830054078251, + 0.0015876249963184819, + 0.0015234160091495141, + 0.0014531250053551048, + 0.0014107909955782816, + 0.0015058340068208054, + 0.0015305830020224676, + 0.0014584169985027984, + 0.0015307080029742792, + 0.0015062080055940896, + 0.0015134580025915056, + 0.0014584590098820627, + 0.0015317080105887726, + 0.00151983299292624, + 0.0015326660068240017, + 0.001396583000314422, + 0.0014936669904273003, + 0.0015123749908525497, + 0.0015402499993797392, + 0.0014615830004913732, + 0.0015117919974727556, + 0.0015760840033181012, + 0.001598499991814606, + 0.0016122499946504831, + 0.0015685419930377975, + 0.0015094999980647117, + 0.00144058300065808, + 0.0017767090030247346, + 0.0016319579881383106, + 0.001549332999275066, + 0.0015684579993830994, + 0.0015261660009855404, + 0.0015521669993177056, + 0.0016050410049501806, + 0.0015616669988958165, + 0.0016325839969795197, + 0.0016679160034982488, + 0.0016219169920077547, + 0.0016989580035442486, + 0.0015536250139120966, + 0.001588792001712136, + 0.0015759999951114878, + 0.0016608330042799935, + 0.0016694160003680736, + 0.0016602919931756333, + 0.0016012499982025474, + 0.001640375005081296, + 0.0017559580010129139, + 0.0016728329937905073, + 0.0015917919954517856, + 0.0016512909933226183, + 0.0016001670010155067 + ], + [ + 0.0015223329974105582, + 0.0015391669876407832, + 0.0016662910056766123, + 0.0016858750022947788, + 0.0017528749885968864, + 0.0016722500004107133, + 0.0018169170070905238, + 0.0017427080019842833, + 0.001526124993688427, + 0.0016149579896591604, + 0.0015339169913204387, + 0.00141816699760966, + 0.0015433340013260022, + 0.0015730840095784515, + 0.0014853749889880419, + 0.0014423749962588772, + 0.0014863749966025352, + 0.0015250829892465845, + 0.0015258750063367188, + 0.001447667003958486, + 0.0014938329986762255, + 0.0015140000032261014, + 0.0014356249885167927, + 0.0014303750067483634, + 0.001480999999330379, + 0.0015231659926939756, + 0.0015052499948069453, + 0.00154283300798852, + 0.001415250007994473, + 0.001457915990613401, + 0.0015481250011362135, + 0.0015031250077299774, + 0.0015239170024869964, + 0.0015597500023432076, + 0.0017456660134484991, + 0.001662082999246195, + 0.0014738340105395764, + 0.0015020839928183705, + 0.0015452499937964603, + 0.0015228340052999556, + 0.0015642920043319464, + 0.0015452500083483756, + 0.001629416990908794, + 0.001548458996694535, + 0.0014567500038538128, + 0.0015760419919388369, + 0.001586166996276006, + 0.0016340000001946464, + 0.001677332998951897, + 0.001618499998585321, + 0.0016367080097552389, + 0.0015970420063240454, + 0.001633374995435588, + 0.0016587499994784594, + 0.0016288330079987645, + 0.0016566670092288405, + 0.0016601670067757368, + 0.0015635830059181899, + 0.0015963749901857227, + 0.0016785420011729002, + 0.0016631249891361222, + 0.0016812079993542284, + 0.0015927089989418164, + 0.001644417003262788 + ], + [ + 0.001595666995854117, + 0.0016441670013591647, + 0.0016806250059744343, + 0.0017335840093437582, + 0.001738124992698431, + 0.0017444590048398823, + 0.0016901250055525452, + 0.001982250003493391, + 0.0014922500122338533, + 0.001616042005480267, + 0.0015157500019995496, + 0.0015616659948136657, + 0.0015615830052411184, + 0.0015133339911699295, + 0.001508249988546595, + 0.0014452909963438287, + 0.0014385829999810085, + 0.0014024160045664757, + 0.0014688749943161383, + 0.0015034590032882988, + 0.0014773340080864727, + 0.0013860409962944686, + 0.001519291996373795, + 0.0015376660012407228, + 0.0014258749870350584, + 0.0014091249904595315, + 0.0014730410039192066, + 0.0015279590006684884, + 0.0014212920068530366, + 0.0014893329935148358, + 0.0015111249958863482, + 0.0015241670043906197, + 0.0014573340013157576, + 0.0014390419964911416, + 0.0015576659934595227, + 0.0015960830060066655, + 0.0015826669987291098, + 0.001612000007298775, + 0.001557666008011438, + 0.001501959006418474, + 0.00161179099814035, + 0.0015826250019017607, + 0.0015290410083252937, + 0.001614709006389603, + 0.0016022089985199273, + 0.001519041004939936, + 0.0015803330024937168, + 0.0015594589931424707, + 0.0016641669935779646, + 0.0014600419963244349, + 0.0014754170115338638, + 0.001632915998925455, + 0.0016873749991646037, + 0.0015805409930180758, + 0.0016157089994521812, + 0.0015069170040078461, + 0.0016941659996518865, + 0.001603708995389752, + 0.001493917006882839, + 0.0015567090013064444, + 0.001522499995189719, + 0.0015609169931849465, + 0.0015781670081196353, + 0.001555041002575308 + ], + [ + 0.0015560410101898015, + 0.0017081670084735379, + 0.0016823750047478825, + 0.0017094580107368529, + 0.0017970829940168187, + 0.0016585839912295341, + 0.0017022910033119842, + 0.0017923330015037209, + 0.0015690830041421577, + 0.0015620000049239025, + 0.00152916599472519, + 0.0014497920055873692, + 0.0014852500025881454, + 0.0015671669971197844, + 0.0015263749955920503, + 0.0014437920035561547, + 0.0014338339969981462, + 0.001400417007971555, + 0.0015032909868750721, + 0.001419124993844889, + 0.0014823749952483922, + 0.001503457999206148, + 0.0015154169959714636, + 0.001407958014169708, + 0.0014772919967072085, + 0.0015127910010050982, + 0.0014200000005075708, + 0.001363250004942529, + 0.0014999999984866008, + 0.0015311659954022616, + 0.0015246660041157156, + 0.0014388330018846318, + 0.001473333002650179, + 0.0016049169935286045, + 0.0016453749994980171, + 0.0015873750089667737, + 0.00154974999895785, + 0.001633666004636325, + 0.0016014590073609725, + 0.0016122919914778322, + 0.0015874579985393211, + 0.001658791006775573, + 0.0014149580092635006, + 0.0017124999867519364, + 0.0016125419933814555, + 0.00161945799482055, + 0.0015605840017087758, + 0.0015214159939205274, + 0.0015746669960208237, + 0.0016672920028213412, + 0.001688375006779097, + 0.001577332994202152, + 0.0015609169931849465, + 0.001596041998709552, + 0.0016579999937675893, + 0.0016085409879451618, + 0.0016866250080056489, + 0.00157200000830926, + 0.0016052909923018888, + 0.0016005420038709417, + 0.0017275419959332794, + 0.0016620420065009966, + 0.0017593340016901493, + 0.0016586660058237612 + ], + [ + 0.001579040996148251, + 0.00161070900503546, + 0.0017010839947033674, + 0.0017165830067824572, + 0.0017692499968688935, + 0.0017679159936960787, + 0.0017339580081170425, + 0.0018438330007484183, + 0.0015016670076875016, + 0.0015114169946173206, + 0.0015163329953793436, + 0.001556041999720037, + 0.0015083750040503219, + 0.0015203749935608357, + 0.0014942910056561232, + 0.001498500001616776, + 0.0015038749988889322, + 0.0014328329998534173, + 0.0014166670007398352, + 0.0015075419942149892, + 0.0015102920006029308, + 0.001495124990469776, + 0.0013777920103166252, + 0.0015110420063138008, + 0.0014379579952219501, + 0.0014695830031996593, + 0.001389375000144355, + 0.0014922499976819381, + 0.0014548329927492887, + 0.0014488750020973384, + 0.0014972079952713102, + 0.001507082997704856, + 0.001562709003337659, + 0.0015935000119498, + 0.0016631670005153865, + 0.0014882090035825968, + 0.001483457992435433, + 0.0014954590005800128, + 0.0015478749992325902, + 0.001596000001882203, + 0.0014793750015087426, + 0.001495499993325211, + 0.0015687089908169582, + 0.0016292920045088977, + 0.00163237500237301, + 0.0015914999967208132, + 0.001604792007128708, + 0.0015545829955954105, + 0.0015933339891489595, + 0.001652042003115639, + 0.001631832987186499, + 0.0016681670094840229, + 0.0015919580037007108, + 0.0016156249912455678, + 0.0016703330038581043, + 0.0015598750032950193, + 0.0016657500091241673, + 0.0016608750011073425, + 0.001608207996468991, + 0.0015864579909248278, + 0.0015262500091921538, + 0.0016657920059515163, + 0.0016643749986542389, + 0.0015925419866107404 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "speedup_cached_vs_uncached_sdpa": 0.8818842055567342, + "speedup_cached_vs_uncached_mha": 0.9462713163428118 + }, + { + "batch": 1, + "prompt_len": 16, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_mha": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 1.382374990498647, + "p95": 1.4357417996507138, + "min": 1.3144160038791597, + "max": 1.4447499997913837, + "iqr": 0.08058399544097483, + "mean": 1.3720749993808568, + "cv": 0.04038174328174239 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.6892090061446652, + "p95": 1.8191035953350365, + "min": 1.5591249975841492, + "max": 2.4290000001201406, + "iqr": 0.0944794956012629, + "mean": 1.6851895051779195, + "cv": 0.05457113684737875 + }, + "total_ms": { + "count": 5, + "median": 107.68162697786465, + "p95": 108.65211161435582, + "min": 106.11458704806864, + "max": 108.8841620075982, + "iqr": 0.43312698835507035, + "mean": 107.53901382558979, + "cv": 0.009245980641471908 + }, + "tokens_per_second": { + "count": 5, + "median": 594.344660237684, + "p95": 601.7992369899106, + "min": 587.7806176763698, + "max": 603.1216044878803, + "iqr": 2.3983949227704215, + "mean": 595.1736042950454, + "cv": 0.00925732178101926 + }, + "median_step_ms_by_index": [ + 1.382374990498647, + 1.8452499934937805, + 1.653958999668248, + 1.6789159999461845, + 1.7670419911155477, + 1.6261670098174363, + 1.7069580062525347, + 1.7194999963976443, + 1.7161249998025596, + 1.685916999122128, + 1.681708003161475, + 1.6928330005612224, + 1.684375005424954, + 1.678833010373637, + 1.575958012836054, + 1.68395800574217, + 1.7147500038845465, + 1.7135829984908924, + 1.6953750018728897, + 1.6841670003486797, + 1.686083007371053, + 1.7035000055329874, + 1.6325000033248216, + 1.6892090061446652, + 1.6890000115381554, + 1.6832080000313, + 1.6935000021476299, + 1.5965839993441477, + 1.6771249938756227, + 1.6887919919099659, + 1.7217499989783391, + 1.708750001853332, + 1.6129579889820889, + 1.6794169932836667, + 1.672582991886884, + 1.6979169886326417, + 1.6786669875727966, + 1.6933750011958182, + 1.7130839987657964, + 1.698916996247135, + 1.670665995334275, + 1.6307079931721091, + 1.7004580004140735, + 1.6835840069688857, + 1.7351249989587814, + 1.706499999272637, + 1.7330830014543608, + 1.742250009556301, + 1.618458001757972, + 1.7527080053696409, + 1.6991250013234094, + 1.6073329898063093, + 1.6802910104161128, + 1.6552909946767613, + 1.664750001509674, + 1.672041995334439, + 1.680209010373801, + 1.7005830013658851, + 1.7132919892901555, + 1.7164580058306456, + 1.629708000109531, + 1.5953330002957955, + 1.6740840073907748, + 1.6430410032626241 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.001382374990498647, + 0.0024290000001201406, + 0.001653958999668248, + 0.001582665994646959, + 0.001853875000961125, + 0.001725916998111643, + 0.0017063330014934763, + 0.0017256669962080196, + 0.0016305839963024482, + 0.0017332079878542572, + 0.0015978750016074628, + 0.0017146669997600839, + 0.001684375005424954, + 0.001695832994300872, + 0.001695208004093729, + 0.00168395800574217, + 0.0016289580089505762, + 0.0016051669954322278, + 0.0016207080043386668, + 0.0016841670003486797, + 0.001686083007371053, + 0.0016727909969631582, + 0.0016325000033248216, + 0.001629332997254096, + 0.0016759590071160346, + 0.0017046249995473772, + 0.0016935000021476299, + 0.001720915999612771, + 0.0015914999967208132, + 0.0016100840002764016, + 0.0017362080106977373, + 0.0017086250009015203, + 0.0017278750019613653, + 0.0015938330034259707, + 0.001672582991886884, + 0.0016272089997073635, + 0.0015747500001452863, + 0.0016714579978724942, + 0.0016454169963253662, + 0.0016716670070309192, + 0.0016745829925639555, + 0.001618334004888311, + 0.0016254999936791137, + 0.0016835840069688857, + 0.0017440840019844472, + 0.0015862089931033552, + 0.0019346249900991097, + 0.0017522499983897433, + 0.0015705410041846335, + 0.0017954159993678331, + 0.0016824589984025806, + 0.001726916991174221, + 0.0015962909965310246, + 0.001673917009611614, + 0.0017970420012716204, + 0.001597958995262161, + 0.001619459013454616, + 0.0017171249928651378, + 0.0016609159938525409, + 0.0017172079969896004, + 0.0017135419911937788, + 0.0015951249952195212, + 0.0016740840073907748, + 0.0016430410032626241 + ], + [ + 0.0014447499997913837, + 0.002368542001931928, + 0.0016726250032661483, + 0.001663791001192294, + 0.001676375002716668, + 0.0016261670098174363, + 0.001710875003482215, + 0.00157049999688752, + 0.0017300830077147111, + 0.0016958330088527873, + 0.0017106250015785918, + 0.00169012499100063, + 0.001583082994329743, + 0.0016134170000441372, + 0.001575958012836054, + 0.0017118749965447932, + 0.0017179160058731213, + 0.0017779159970814362, + 0.0017228329961653799, + 0.0016605419950792566, + 0.0016169580048881471, + 0.0017157080001197755, + 0.0016973330057226121, + 0.001708584008156322, + 0.0016940420027822256, + 0.0016783329920144752, + 0.001729208990582265, + 0.0015742500108899549, + 0.0016771249938756227, + 0.0017104159924201667, + 0.0017199160065501928, + 0.001708750001853332, + 0.0016129579889820889, + 0.0016794169932836667, + 0.001727249997202307, + 0.0017122500139521435, + 0.0016786669875727966, + 0.0016933750011958182, + 0.0017525830044178292, + 0.001698916996247135, + 0.0016454580036224797, + 0.0016307079931721091, + 0.0017004580004140735, + 0.0016981660010060295, + 0.001690082994173281, + 0.0016978339990600944, + 0.0017330830014543608, + 0.0016260000120382756, + 0.001689333003014326, + 0.0016357499989680946, + 0.001565708007547073, + 0.0015969590021995828, + 0.0015958750009303913, + 0.0016552909946767613, + 0.0017172500083688647, + 0.0017132080101873726, + 0.0017005830013658851, + 0.0015858749975450337, + 0.0017425419937353581, + 0.001688042000751011, + 0.001629708000109531, + 0.0015953330002957955, + 0.0017554579972056672, + 0.001725333000649698 + ], + [ + 0.001399708999088034, + 0.00177408401214052, + 0.0016192079929169267, + 0.0016841250035213307, + 0.001691417011898011, + 0.0017185420001624152, + 0.0016598749934928492, + 0.0017194999963976443, + 0.0015737500070827082, + 0.0015593330026604235, + 0.001681708003161475, + 0.0016529999993508682, + 0.0016820839955471456, + 0.001678833010373637, + 0.001564792008139193, + 0.001560417003929615, + 0.0017322089988738298, + 0.0017135829984908924, + 0.001598083006683737, + 0.0016975000035017729, + 0.0015850829950068146, + 0.0016808340005809441, + 0.001614082997548394, + 0.0017286249931203201, + 0.0016903749929042533, + 0.0016832080000312999, + 0.001672416998189874, + 0.0016336669941665605, + 0.0016332919913111255, + 0.0016887919919099659, + 0.0017217499989783391, + 0.0017156250105472282, + 0.0015993749984772876, + 0.001685708004515618, + 0.0016421670006820932, + 0.0016979169886326417, + 0.0017214160034200177, + 0.001697917003184557, + 0.0017130839987657964, + 0.0017187500052386895, + 0.0015816249942872673, + 0.0015833749930607155, + 0.0017070839967345819, + 0.0016343340103048831, + 0.0016984170069918036, + 0.001706499999272637, + 0.0017214580002473667, + 0.0016537089977646247, + 0.001618458001757972, + 0.0017126669990830123, + 0.0017225829942617565, + 0.0016073329898063093, + 0.0016802910104161128, + 0.0015711250016465783, + 0.0015901670121820644, + 0.001672041995334439, + 0.001680209010373801, + 0.0017005830013658851, + 0.0016652080084895715, + 0.0016780409932835028, + 0.001609333005035296, + 0.0016795830015325919, + 0.001575250003952533, + 0.0015793750062584877 + ], + [ + 0.0013144160038791597, + 0.0018452499934937805, + 0.0015948329964885488, + 0.0018397079984424636, + 0.0017670419911155477, + 0.0015692080050939694, + 0.0017069580062525347, + 0.0017269579984713346, + 0.0017161249998025596, + 0.0015960410091793165, + 0.0016116249898914248, + 0.0017109580076066777, + 0.0017104169965023175, + 0.0017157500114990398, + 0.0017767499957699329, + 0.0015697909984737635, + 0.0015981670003384352, + 0.0015591249975841492, + 0.0017107080057030544, + 0.0017158749978989363, + 0.001727625000057742, + 0.0017139999981736764, + 0.001597750000655651, + 0.001618124995729886, + 0.0016890000115381554, + 0.0017007920105243102, + 0.0017134999943664297, + 0.0015965839993441477, + 0.0017434579931432381, + 0.0015849580086069182, + 0.0017218749999301508, + 0.001623208008822985, + 0.0016063330112956464, + 0.0016898330068215728, + 0.0015847500035306439, + 0.001607583006261848, + 0.0015721670060884207, + 0.0016302080039167777, + 0.0017874579934868962, + 0.0017167919868370518, + 0.0017016250058077276, + 0.0017304590001003817, + 0.0016170000017154962, + 0.0017102499987231568, + 0.0017351249989587814, + 0.001818040997022763, + 0.0018538330041337758, + 0.0018446659960318357, + 0.0018325000128243119, + 0.001835624992963858, + 0.0020144589943811297, + 0.0018607079982757568, + 0.0017336659948341548, + 0.0019359169964445755, + 0.001664750001509674, + 0.001730457996018231, + 0.0017434170003980398, + 0.0017142500000772998, + 0.0017132919892901555, + 0.0017767089884728193, + 0.0017084999999497086, + 0.0016227909945882857, + 0.0015962080069584772, + 0.0018082089954987168 + ], + [ + 0.0013191250036470592, + 0.001823583006625995, + 0.0016938329936238006, + 0.0016789159999461845, + 0.00180170800013002, + 0.001593125009094365, + 0.0017449160077376291, + 0.0016541660006623715, + 0.0017886249988805503, + 0.0016859169991221279, + 0.0016927090036915615, + 0.0016928330005612224, + 0.0016974580066744238, + 0.0016097909974632785, + 0.0015643749939044937, + 0.0017264170019188896, + 0.0017147500038845465, + 0.0017209590005222708, + 0.0016953750018728897, + 0.0015911660011624917, + 0.001688165997620672, + 0.0017035000055329874, + 0.0016979579959297553, + 0.0016892090061446652, + 0.0016860000032465905, + 0.0016711249918444082, + 0.0015776660002302378, + 0.0015759999951114878, + 0.0016797499993117526, + 0.0017073750059353188, + 0.0016969999996945262, + 0.0017102080018958077, + 0.0016936250030994415, + 0.0016140420048031956, + 0.0017066250002244487, + 0.0017132499924628064, + 0.0017085830040741712, + 0.001710416006972082, + 0.001700000007986091, + 0.001573749992530793, + 0.001670665995334275, + 0.0017069159948732704, + 0.0017267079965677112, + 0.0015877919940976426, + 0.0019994580070488155, + 0.0018215829913970083, + 0.0016184170090127736, + 0.001742250009556301, + 0.001608000005944632, + 0.0017527080053696409, + 0.0016991250013234094, + 0.0015665000100852922, + 0.0017490419995738193, + 0.0015889170026639476, + 0.001638916990486905, + 0.001590083003975451, + 0.0016223330021603033, + 0.0017004999972414225, + 0.0017152079963125288, + 0.0017164580058306456, + 0.0015909579960862175, + 0.0015797909873072058, + 0.0016842500044731423, + 0.0016202080005314201 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + ] + } + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 1.3467499957187101, + "p95": 1.4116330014076084, + "min": 1.287874998524785, + "max": 1.4140830026008189, + "iqr": 0.08366699330508709, + "mean": 1.353741399361752, + "cv": 0.03979149071093693 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.486499997554347, + "p95": 1.6141957996296696, + "min": 1.313916000071913, + "max": 1.901958996313624, + "iqr": 0.12641750072361901, + "mean": 1.4818231456927837, + "cv": 0.05879497672801276 + }, + "total_ms": { + "count": 5, + "median": 94.93508396553807, + "p95": 95.27326741081197, + "min": 93.43283298949245, + "max": 95.29029202531092, + "iqr": 0.5255489959381521, + "mean": 94.70859957800712, + "cv": 0.007944788613715094 + }, + "tokens_per_second": { + "count": 5, + "median": 674.1448717023554, + "p95": 683.1800061609044, + "min": 671.6319012119344, + "max": 684.9840463169676, + "iqr": 3.731437318160147, + "mean": 675.7914145972802, + "cv": 0.00801024779381828 + }, + "median_step_ms_by_index": [ + 1.3467499957187101, + 1.3919170014560223, + 1.4086669980315492, + 1.4094999933149666, + 1.3685409940080717, + 1.3927079999120906, + 1.4117079990683123, + 1.405584000167437, + 1.378500004648231, + 1.3619579985970631, + 1.4157080004224554, + 1.357375003863126, + 1.4150420029181987, + 1.4323329960461706, + 1.4451669994741678, + 1.4191669906722382, + 1.3936670002294704, + 1.5105829952517524, + 1.4979580009821802, + 1.4776249881833792, + 1.4298750029411167, + 1.4704589993925765, + 1.4902080001775175, + 1.5002089930931106, + 1.4866249985061586, + 1.4734580036019906, + 1.4176249969750643, + 1.499791003880091, + 1.4802500081714243, + 1.499292004154995, + 1.4568750048056245, + 1.4967089955462143, + 1.5154159918893129, + 1.4762500068172812, + 1.5104579942999408, + 1.5494589897571132, + 1.5402089920826256, + 1.5463330055354163, + 1.4900420064805076, + 1.48695899406448, + 1.5651249996153638, + 1.4939169923309237, + 1.5440840070368722, + 1.462542000808753, + 1.5333750052377582, + 1.5523329930147156, + 1.5005409950390458, + 1.4688749943161383, + 1.445540998247452, + 1.5789159951964393, + 1.5452910010935739, + 1.5117920120246708, + 1.5179579932009801, + 1.5786250005476177, + 1.6014169959817082, + 1.5699159994255751, + 1.536791998660192, + 1.557374998810701, + 1.5390420012408867, + 1.5645420062355697, + 1.488084002630785, + 1.6144590044859797, + 1.610541992704384, + 1.5935839910525829 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0013181660033296794, + 0.0013925410021329299, + 0.0014018330111866817, + 0.00141833299130667, + 0.0013851249968865886, + 0.0014141659921733662, + 0.001334290995146148, + 0.001405584000167437, + 0.0013173329934943467, + 0.0013204999995650724, + 0.0014459579979302362, + 0.0014202500024111941, + 0.0014150420029181987, + 0.001367541990475729, + 0.0014586249890271574, + 0.0014191669906722382, + 0.0013936670002294704, + 0.0015105829952517524, + 0.0014992910000728443, + 0.0014848749997327104, + 0.0014870000013615936, + 0.0014704589993925765, + 0.0013882079947507009, + 0.001466125002480112, + 0.001397375002852641, + 0.0014605829928768799, + 0.001487625006120652, + 0.001499791003880091, + 0.0014336669992189854, + 0.001499292004154995, + 0.0014942090056138113, + 0.0013463340001180768, + 0.0016809170047054067, + 0.0014762500068172812, + 0.0013624159910250455, + 0.001367709002806805, + 0.0013732920051552355, + 0.0013764160103164613, + 0.0013659999967785552, + 0.0013862089981557801, + 0.001901958996313624, + 0.001374125000438653, + 0.0013616659998660907, + 0.0013632089976454154, + 0.0013642909907503054, + 0.001360916008707136, + 0.0014669590018456802, + 0.0014729169924976304, + 0.0013827919901814312, + 0.001542500001960434, + 0.0014314580039354041, + 0.0014322089991765097, + 0.0014903330011293292, + 0.0016139580111484975, + 0.0017077080119634047, + 0.0015699159994255751, + 0.001443667002604343, + 0.0015894170064711943, + 0.0016132919990923256, + 0.00160133300232701, + 0.0014652919926447794, + 0.0015914999967208132, + 0.0016710420022718608, + 0.0016776249976828694 + ], + [ + 0.0014140830026008189, + 0.0013488749973475933, + 0.0014086669980315492, + 0.0013887499953852966, + 0.0013685409940080717, + 0.0013927079999120906, + 0.0014246250066207722, + 0.0013422079937299713, + 0.0013961670047137886, + 0.0013360830052988604, + 0.0014157080004224554, + 0.0013508340052794665, + 0.0014082919951761141, + 0.0014323329960461706, + 0.0013827919901814312, + 0.0014311670092865825, + 0.0013665830047102645, + 0.001516999996965751, + 0.001405584000167437, + 0.0014571250067092478, + 0.0014298750029411167, + 0.001390125005855225, + 0.001492790994234383, + 0.0015172499988693744, + 0.0015062919992487878, + 0.001448040988179855, + 0.0014789999986533076, + 0.0015107499930309132, + 0.001498124998761341, + 0.0014570839994121343, + 0.0014518750103889033, + 0.0015122080076253042, + 0.001436582999303937, + 0.001459040999179706, + 0.001579625008162111, + 0.001552916000946425, + 0.0015612080023856834, + 0.001535832998342812, + 0.0014900420064805076, + 0.00148695899406448, + 0.0015774999919813126, + 0.0014939169923309237, + 0.0015440840070368722, + 0.001462542000808753, + 0.0015392079949378967, + 0.0015803749993210658, + 0.0015005409950390458, + 0.0014915420033503324, + 0.0015235000028042123, + 0.0016137079946929589, + 0.0016353750106645748, + 0.001536208001198247, + 0.0015179579932009801, + 0.0015786250005476177, + 0.0016014169959817082, + 0.0014822919911239296, + 0.001536791998660192, + 0.0016303749871440232, + 0.0015216250030789524, + 0.0015191669954219833, + 0.001488084002630785, + 0.0015234589955070987, + 0.001610541992704384, + 0.0016425000067101792 + ], + [ + 0.001287874998524785, + 0.0013919170014560223, + 0.0014163750020088628, + 0.0014094999933149666, + 0.0013414999993983656, + 0.0013818330044159666, + 0.001422875007847324, + 0.0013600419915746897, + 0.001378500004648231, + 0.0013619579985970631, + 0.0013524579990189523, + 0.0014334580046124756, + 0.0014227499923435971, + 0.001313916000071913, + 0.001317375004873611, + 0.0015024159947643057, + 0.0014372080040629953, + 0.001429542011464946, + 0.001528541004518047, + 0.0015056249976623803, + 0.0013776249979855493, + 0.0014164169988362119, + 0.0015041249862406403, + 0.0014855829940643162, + 0.0014294580032583326, + 0.0014734580036019906, + 0.0013982499949634075, + 0.0016542910016141832, + 0.0014802500081714243, + 0.0014460419915849343, + 0.0015102500037755817, + 0.0014967089955462143, + 0.0015945829945849255, + 0.0014979169936850667, + 0.0015104579942999408, + 0.0015494589897571132, + 0.0014950839977245778, + 0.0015672089939471334, + 0.00148974999319762, + 0.0015698750066803768, + 0.0015651249996153638, + 0.0015458749985555187, + 0.001429167008609511, + 0.0015335409989347681, + 0.001549291002447717, + 0.0015523329930147156, + 0.0014614579995395616, + 0.0014214999973773956, + 0.00139666699396912, + 0.0015789159951964393, + 0.0015452910010935739, + 0.0015147500089369714, + 0.0015661250072298571, + 0.001486499997554347, + 0.0015780000103404745, + 0.0015374999929917976, + 0.0015344160055974498, + 0.001490749986260198, + 0.0015671670116716996, + 0.0015212909929687157, + 0.0015298340003937483, + 0.0016517080075573176, + 0.001614082997548394, + 0.0015658749907743186 + ], + [ + 0.0013467499957187101, + 0.0013633749913424253, + 0.0013817089929943904, + 0.0014130000054137781, + 0.0013618750090245157, + 0.001429041993105784, + 0.0014117079990683123, + 0.0014167089975671843, + 0.0013359580043470487, + 0.0013784999900963157, + 0.001450334006221965, + 0.001357375003863126, + 0.001323292002780363, + 0.001524084000266157, + 0.0014451669994741678, + 0.0014092080091359094, + 0.001360958005534485, + 0.001687249998212792, + 0.0014979580009821802, + 0.001411499993992038, + 0.001396958003169857, + 0.001475291996030137, + 0.0014902080001775175, + 0.0015182090137386695, + 0.0015230830031214282, + 0.0015054160030558705, + 0.0014176249969750643, + 0.0014924999995855615, + 0.0015074999973876402, + 0.0015182499919319525, + 0.0014251249958761036, + 0.0013833749981131405, + 0.0014460000093095005, + 0.0014237499999580905, + 0.0014804999955231324, + 0.0015984159981599078, + 0.0015455410029971972, + 0.0015641250065527856, + 0.0015473329985979944, + 0.0015375829971162602, + 0.0015611660055583343, + 0.001557333001983352, + 0.0015570830000797287, + 0.0014857500063953921, + 0.001527667001937516, + 0.0015379160031443462, + 0.0015576669975416735, + 0.0014688749943161383, + 0.001445540998247452, + 0.0015714170003775507, + 0.0016103329980978742, + 0.0015117920120246708, + 0.0014811250002821907, + 0.0015797079977346584, + 0.001589749997947365, + 0.0016449999966425821, + 0.0016067499964265153, + 0.0015435840032296255, + 0.0015390420012408867, + 0.0015645420062355697, + 0.001485375003539957, + 0.0016144590044859797, + 0.0015522920002695173, + 0.0015935839910525829 + ], + [ + 0.0014018329966347665, + 0.0013960419892100617, + 0.0014299169997684658, + 0.0014063330017961562, + 0.001531292000436224, + 0.0013297499972395599, + 0.0013831249962095171, + 0.001422707995516248, + 0.0013948749983683228, + 0.00144058300065808, + 0.001362334005534649, + 0.0013210829929448664, + 0.001430167001672089, + 0.0014417089987546206, + 0.0014457910001510754, + 0.0013639160024467856, + 0.0014133750082692131, + 0.001361375005217269, + 0.0013948330015409738, + 0.0014776249881833792, + 0.001485000000684522, + 0.0014814579917583615, + 0.0014140419953037053, + 0.0015002089930931106, + 0.0014866249985061586, + 0.0014940409892005846, + 0.0013952079898444936, + 0.001496208002208732, + 0.001456042009522207, + 0.0015271250013029203, + 0.0014568750048056245, + 0.0015089590015122667, + 0.0015154159918893129, + 0.0015602920029778033, + 0.0015456660039490089, + 0.0015100839955266565, + 0.0015402089920826256, + 0.0015463330055354163, + 0.001542541998787783, + 0.0013805839989800006, + 0.0015378749958472326, + 0.001465749999624677, + 0.0015736659988760948, + 0.0014382500085048378, + 0.0015333750052377582, + 0.00158079199900385, + 0.0016017919988371432, + 0.001443625005776994, + 0.001531292000436224, + 0.001608582999324426, + 0.0015093339898157865, + 0.001480417005950585, + 0.0015274170000338927, + 0.001540833996841684, + 0.0016069580015027896, + 0.0016209580062422901, + 0.001654916995903477, + 0.001557374998810701, + 0.0015376249939436093, + 0.0016098330088425428, + 0.0015177919995039701, + 0.0016359169967472553, + 0.001554665999719873, + 0.0015445419994648546 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_mha": { + "first_token_ms": { + "count": 5, + "median": 1.5430420025950298, + "p95": 1.6456999932415783, + "min": 1.5255830076057464, + "max": 1.6671249904902652, + "iqr": 0.019459010218270123, + "mean": 1.5672581997932866, + "cv": 0.036463032328451266 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.608958002179861, + "p95": 1.7530124066979624, + "min": 1.4020829985383898, + "max": 1.9118330092169344, + "iqr": 0.1281034928979352, + "mean": 1.606870238393885, + "cv": 0.05831140575596345 + }, + "total_ms": { + "count": 5, + "median": 102.46258405095432, + "p95": 105.37216414522845, + "min": 101.31087605259381, + "max": 106.06053893570788, + "iqr": 1.0709129128372297, + "mean": 102.80008321860805, + "cv": 0.018561319106432026 + }, + "tokens_per_second": { + "count": 5, + "median": 624.6182505818222, + "p95": 631.4242309419309, + "min": 603.4289533338665, + "max": 631.718947596263, + "iqr": 6.577145580882302, + "mean": 622.7359469160549, + "cv": 0.018207760557522674 + }, + "median_step_ms_by_index": [ + 1.5430420025950298, + 1.5399160038214177, + 1.5198330074781552, + 1.4967910101404414, + 1.5374579961644486, + 1.5270419971784577, + 1.5198339970083907, + 1.516042000730522, + 1.5155000000959262, + 1.5260829968610778, + 1.532207999844104, + 1.4311249979073182, + 1.5027080080471933, + 1.5127079968806356, + 1.5467080083908513, + 1.5138330054469407, + 1.544999991892837, + 1.5775000065332279, + 1.5874580130912364, + 1.575832997332327, + 1.6040409973356873, + 1.6133750032167882, + 1.5954999980749562, + 1.5283340035239235, + 1.589416991919279, + 1.5620840131305158, + 1.5812079946044832, + 1.603666998562403, + 1.5489170036744326, + 1.5587079979013652, + 1.612207997823134, + 1.6270829946734011, + 1.6211250040214509, + 1.6659169923514128, + 1.5903749881545082, + 1.6303750016959384, + 1.5812499914318323, + 1.6694579971954226, + 1.6105000104289502, + 1.6103749949252233, + 1.601125011802651, + 1.6209169989451766, + 1.6510410059709102, + 1.5975830028764904, + 1.6737920086598024, + 1.585374993737787, + 1.608958002179861, + 1.5869159979047254, + 1.6127499984577298, + 1.7043749976437539, + 1.6864579956745729, + 1.6613750049145892, + 1.6447910020360723, + 1.6277079994324595, + 1.630124999792315, + 1.6317090048687533, + 1.6635829961160198, + 1.6522920050192624, + 1.7022079991875216, + 1.7080830002669245, + 1.6984170069918036, + 1.71312500606291, + 1.714874990284443, + 1.7228329961653799 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0016671249904902652, + 0.0015399160038214177, + 0.001585209000040777, + 0.0016286250029224902, + 0.0016938329936238006, + 0.0016268329927697778, + 0.001618875001440756, + 0.0015686249971622601, + 0.0017679589946055785, + 0.0017046670109266415, + 0.0017480829992564395, + 0.0017090000037569553, + 0.0016326670011039823, + 0.001664458992308937, + 0.0015743330004625022, + 0.0016109999996842816, + 0.0016479579935548827, + 0.0017179580027004704, + 0.0018688330019358546, + 0.001819625002099201, + 0.001800375001039356, + 0.0017097090021707118, + 0.0016765830077929422, + 0.0017273330013267696, + 0.0017626659973757342, + 0.001742334003210999, + 0.0016791660018498078, + 0.0016439169994555414, + 0.001531292000436224, + 0.0015587079979013652, + 0.0016590000013820827, + 0.0016501659993082285, + 0.001636165994568728, + 0.0017559580010129139, + 0.0015903749881545082, + 0.0015928750071907416, + 0.0016903749929042533, + 0.0016643330018268898, + 0.0016787499916972592, + 0.001639417008846067, + 0.0015830420015845448, + 0.0015843330038478598, + 0.001666500000283122, + 0.001673708000453189, + 0.0016737920086598024, + 0.001585374993737787, + 0.0015321249957196414, + 0.0015620829944964498, + 0.0015455829998245463, + 0.0016719580016797408, + 0.0016866660007508472, + 0.0016613750049145892, + 0.0016404159978264943, + 0.0016158340004039928, + 0.0016751249931985512, + 0.0017012079915730283, + 0.0015925419866107404, + 0.0016067919932538643, + 0.0016511249996256083, + 0.0016263750003417954, + 0.0016864170029293746, + 0.0017109170003095642, + 0.001714874990284443, + 0.001627291989279911 + ], + [ + 0.0015430420025950298, + 0.001556249990244396, + 0.0015198330074781552, + 0.0015231669967761263, + 0.0015627499960828573, + 0.0015270419971784577, + 0.0015122500044526532, + 0.0014817499904893339, + 0.0015074169932631776, + 0.0015766249998705462, + 0.0015408329927595332, + 0.0014283750060712919, + 0.0014020829985383898, + 0.0015127079968806356, + 0.0015467080083908513, + 0.0015228330012178048, + 0.0014572079962817952, + 0.0015775000065332279, + 0.0015478749992325902, + 0.0016570000007050112, + 0.0015100000018719584, + 0.0015241250075632706, + 0.0015954999980749562, + 0.0015925420011626557, + 0.001589416991919279, + 0.0015620840131305158, + 0.0014917079970473424, + 0.001601374999154359, + 0.0016104160022223368, + 0.0016219579993048683, + 0.001553708003484644, + 0.0015478749992325902, + 0.001680624991422519, + 0.0016781250014901161, + 0.0016840829921420664, + 0.0015752909966977313, + 0.001570916996570304, + 0.0016694579971954226, + 0.0015663750091334805, + 0.0015520830056630075, + 0.001601125011802651, + 0.0016209169989451766, + 0.0016428339877165854, + 0.0015677079936722293, + 0.001720416999887675, + 0.0016826249920995906, + 0.0016828340012580156, + 0.0016757920093368739, + 0.00171237500035204, + 0.0017043749976437539, + 0.0016864579956745729, + 0.0016469999973196536, + 0.001661332993535325, + 0.0016277079994324595, + 0.001630124999792315, + 0.001664541006903164, + 0.0016815419949125499, + 0.0017772499995771796, + 0.0017202500021085143, + 0.0016874169959919527, + 0.0016984170069918036, + 0.0017701250035315752, + 0.0017517500091344118, + 0.0017228329961653799 + ], + [ + 0.0015405409940285608, + 0.0014935420040274039, + 0.0015285420086001977, + 0.0014127910108072683, + 0.0014789170090807602, + 0.0014443749969359487, + 0.0015487909986404702, + 0.001516042000730522, + 0.0015155000000959262, + 0.0014547920000040904, + 0.001532207999844104, + 0.0014271670079324394, + 0.0015027080080471933, + 0.0014562500000465661, + 0.001524459003121592, + 0.0014048750017536804, + 0.0017184159951284528, + 0.0015355420036939904, + 0.001545374994748272, + 0.0014765830128453672, + 0.0016040409973356873, + 0.001563666999572888, + 0.0014761659986106679, + 0.0015283340035239235, + 0.0016310419887304306, + 0.0016136249905684963, + 0.0016265419981209561, + 0.001603666998562403, + 0.0015489170036744326, + 0.0016391669923905283, + 0.001612207997823134, + 0.0016299169947160408, + 0.0016211250040214509, + 0.001574875001097098, + 0.0015712909953435883, + 0.0016434999997727573, + 0.0017056250071618706, + 0.0017500420071883127, + 0.0015904169995337725, + 0.0016103749949252233, + 0.001585749996593222, + 0.0017092080088332295, + 0.0016510410059709102, + 0.0017330419941572472, + 0.0016877500020200387, + 0.0015687499981140718, + 0.001608958002179861, + 0.0015767089935252443, + 0.001657791988691315, + 0.0017234160040970892, + 0.0015679170028306544, + 0.0015440840070368722, + 0.0016447910020360723, + 0.0018143330089515075, + 0.0017243749898625538, + 0.001576791997649707, + 0.0019118330092169344, + 0.0016435419966001064, + 0.0015687499981140718, + 0.001788374996976927, + 0.0017305000073974952, + 0.00171312500606291, + 0.0017250000091735274, + 0.001804792002076283 + ], + [ + 0.0015255830076057464, + 0.0014408330025617033, + 0.0014808749983785674, + 0.001458500002627261, + 0.0015374579961644486, + 0.001530541994725354, + 0.0015198339970083907, + 0.0014635829866165295, + 0.0015483750030398369, + 0.0015260829968610778, + 0.0014332499995362014, + 0.0014311249979073182, + 0.0015547919902019203, + 0.0015008330083219334, + 0.0015628340042894706, + 0.0014294170105131343, + 0.0014176250115269795, + 0.001554125003167428, + 0.0016052919963840395, + 0.001575832997332327, + 0.001517708005849272, + 0.0016133750032167882, + 0.0014398750063264742, + 0.001466500005335547, + 0.0015018330013845116, + 0.0015085420018294826, + 0.001568207997479476, + 0.0015997500013327226, + 0.0015941669989842921, + 0.0015411250060424209, + 0.0016432920092483982, + 0.0016257080133073032, + 0.001534208990051411, + 0.0016659169923514128, + 0.0016194999916478992, + 0.0016303750016959384, + 0.0015552910044789314, + 0.0017309590039076284, + 0.0017957499949261546, + 0.001570667009218596, + 0.001800582991563715, + 0.0016020000039134175, + 0.0016114169993670657, + 0.0015975830028764904, + 0.0016457500023534521, + 0.0017007499991450459, + 0.0015797920059412718, + 0.0015869159979047254, + 0.0016127499984577298, + 0.0015867080073803663, + 0.0017246250063180923, + 0.0017041670071193948, + 0.0016278749972116202, + 0.0016627500008326024, + 0.0015632080030627549, + 0.0016317090048687533, + 0.0016635829961160198, + 0.001725583992083557, + 0.001710875003482215, + 0.0017080830002669245, + 0.001601125011802651, + 0.0016174170013982803, + 0.0016969169955700636, + 0.0017660419980529696 + ], + [ + 0.001560000004246831, + 0.001545959006762132, + 0.0014291669940575957, + 0.0014967910101404414, + 0.001471834009862505, + 0.0014157079858705401, + 0.0014219580043572932, + 0.001523291997727938, + 0.0015067920030560344, + 0.001520792007795535, + 0.0014214999973773956, + 0.0015028339985292405, + 0.001446166992536746, + 0.0015368750027846545, + 0.0014597080007661134, + 0.0015138330054469407, + 0.001544999991892837, + 0.0015981670003384352, + 0.0015874580130912364, + 0.0015265830006683245, + 0.001634125001146458, + 0.0016272920038318262, + 0.001628375001018867, + 0.0014857499918434769, + 0.0015012079966254532, + 0.0015321249957196414, + 0.0015812079946044832, + 0.0016113330057123676, + 0.0015336670039687306, + 0.0014384170062839985, + 0.001537167001515627, + 0.0016270829946734011, + 0.0015647079999325797, + 0.00159104100021068, + 0.0015638750046491623, + 0.0016340000001946464, + 0.0015812499914318323, + 0.0016080839995993301, + 0.0016105000104289502, + 0.0016134170000441372, + 0.0016622919938527048, + 0.0016519579949090257, + 0.0016590419982094318, + 0.0015807079907972366, + 0.0014738340105395764, + 0.0015812080091563985, + 0.0016375829873140901, + 0.0016609159938525409, + 0.0014780000055907294, + 0.0019042909989366308, + 0.0016623749979771674, + 0.0017025000124704093, + 0.0017285000067204237, + 0.0016185839922400191, + 0.0016205830033868551, + 0.0016267080063698813, + 0.0016344590112566948, + 0.0016522920050192624, + 0.0017022079991875216, + 0.0017327920068055391, + 0.0017398750060237944, + 0.0017399999924236909, + 0.0016527919942745939, + 0.0016723330045351759 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "speedup_cached_vs_uncached_sdpa": 0.8816275035020895, + "speedup_cached_vs_uncached_mha": 0.9515326516381186 + }, + { + "batch": 1, + "prompt_len": 128, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_mha": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 1.7315000004600734, + "p95": 1.749658805783838, + "min": 1.6629589954391122, + "max": 1.7540840053698048, + "iqr": 0.007916009053587914, + "mean": 1.7209086014190689, + "cv": 0.019924743602011588 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.7164160090032965, + "p95": 1.824566299910657, + "min": 1.5577090089209378, + "max": 2.37462499353569, + "iqr": 0.07375051063718274, + "mean": 1.711589241093616, + "cv": 0.050187805590072555 + }, + "total_ms": { + "count": 5, + "median": 109.68245298136026, + "p95": 110.39402259048074, + "min": 108.14283002400771, + "max": 110.51937199954409, + "iqr": 0.37475096178241074, + "mean": 109.55103079031687, + "cv": 0.007978366781627156 + }, + "tokens_per_second": { + "count": 5, + "median": 583.5026320105763, + "p95": 590.3238382571813, + "min": 579.083999864422, + "max": 591.8099238367629, + "iqr": 1.9928250712021054, + "mean": 584.2325445036538, + "cv": 0.00802312629332474 + }, + "median_step_ms_by_index": [ + 1.7315000004600734, + 1.8318750080652535, + 1.6951669967966154, + 1.7221250018337741, + 1.742833002936095, + 1.7099579999921843, + 1.713167002890259, + 1.7304159991908818, + 1.6229999891947955, + 1.6519999917363748, + 1.7263750050915405, + 1.6839170129969716, + 1.7342089995509014, + 1.6854169953148812, + 1.7290419928031042, + 1.720416999887675, + 1.7046249995473772, + 1.6644580027787015, + 1.738292005029507, + 1.714874990284443, + 1.7124160076491535, + 1.6957499901764095, + 1.7228329961653799, + 1.6855419962666929, + 1.721917011309415, + 1.7196660046465695, + 1.7097500094678253, + 1.7198749992530793, + 1.719958003377542, + 1.6409170057158917, + 1.710833006654866, + 1.6793750110082328, + 1.7155410023406148, + 1.721583990729414, + 1.7108329921029508, + 1.7028749862220138, + 1.7029580048983917, + 1.7070419999072328, + 1.7251660028705373, + 1.657749991863966, + 1.7137910035671666, + 1.6358330030925572, + 1.7003749962896109, + 1.6778749995864928, + 1.7129580082837492, + 1.7225420015165582, + 1.7162080039270222, + 1.733749988488853, + 1.7114169895648956, + 1.7188329948112369, + 1.6543329984415323, + 1.7340830090688542, + 1.7556660022819415, + 1.6704169975128025, + 1.7272910044994205, + 1.6972909943433478, + 1.6774169926065952, + 1.7345420055789873, + 1.7263750050915405, + 1.723291992675513, + 1.720165993901901, + 1.698124993708916, + 1.7222500027855858, + 1.7268750088987872 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.001731958007439971, + 0.0019701250130310655, + 0.0016426250076619908, + 0.001784374995622784, + 0.0017688329971861094, + 0.0017099579999921843, + 0.001720000000204891, + 0.001734124991344288, + 0.001600000003236346, + 0.0016053329891292378, + 0.001746458001434803, + 0.0017961249977815896, + 0.0017342089995509014, + 0.0017258749867323786, + 0.0017290419928031042, + 0.0015757090004626662, + 0.0017004580004140735, + 0.001723207999020815, + 0.0018079590081470087, + 0.0021562920010183007, + 0.00189124999451451, + 0.0017967079911613837, + 0.0016744159947847947, + 0.0016855419962666929, + 0.0016397919971495867, + 0.0017500840040156618, + 0.00166079199698288, + 0.0016793329996289685, + 0.0018400829867459834, + 0.0017850830045063049, + 0.0017487080040154979, + 0.0016793750110082328, + 0.0018123749905498698, + 0.0017303329950664192, + 0.0016865829966263846, + 0.0016089170094346628, + 0.0017029580048983917, + 0.001625749995582737, + 0.00174079199496191, + 0.0017302080086665228, + 0.0017160829884232953, + 0.0017539580003358424, + 0.001654874999076128, + 0.001642083007027395, + 0.0017156660032924265, + 0.0017233749967999756, + 0.0017183330055559054, + 0.001733749988488853, + 0.0017515000072307885, + 0.001654833002248779, + 0.0016360830049961805, + 0.0018249169952468947, + 0.0017605000030016527, + 0.0016369580116588622, + 0.0017734580032993108, + 0.0017221250018337741, + 0.0017258340085390955, + 0.0016603749973000959, + 0.0016340829897671938, + 0.0017125839949585497, + 0.0017332920106127858, + 0.001698124993708916, + 0.0017779580084607005, + 0.0017268750088987872 + ], + [ + 0.0016629589954391122, + 0.0017916659999173135, + 0.0016951669967966154, + 0.0017221250018337741, + 0.0017176249966723844, + 0.0016118749917950481, + 0.001601708005182445, + 0.0017158749978989363, + 0.0016702080029062927, + 0.0017355839954689145, + 0.0017288329981965944, + 0.0016018750029616058, + 0.0020787499961443245, + 0.0016823340120026842, + 0.0017094169888878241, + 0.0017555410013301298, + 0.0017046249995473772, + 0.0016644580027787015, + 0.001738292005029507, + 0.0017121250130003318, + 0.0016576670022914186, + 0.0016678329993737862, + 0.001734082994516939, + 0.0017280410102102906, + 0.001721917011309415, + 0.0017196660046465695, + 0.0017349579866277054, + 0.0017121249984484166, + 0.001612583000678569, + 0.0016100000066217035, + 0.001710833006654866, + 0.0017179999995278195, + 0.0017155410023406148, + 0.001721583990729414, + 0.0017108329921029508, + 0.0016619999951217324, + 0.001640375005081296, + 0.0017164160090032965, + 0.0017217909917235374, + 0.0016183749976335093, + 0.0016197079967241734, + 0.0016358330030925572, + 0.0017311249976046383, + 0.0017425419937353581, + 0.00237462499353569, + 0.001637374996789731, + 0.0017803329974412918, + 0.0017422499950043857, + 0.0017605419998290017, + 0.0017367089894833043, + 0.0016225830040639266, + 0.00199387499014847, + 0.0017556660022819415, + 0.0016613750049145892, + 0.0016079170018201694, + 0.0016972909943433478, + 0.0016774169926065952, + 0.0017678330041235313, + 0.0016200829995796084, + 0.0015808750031283125, + 0.0016014580032788217, + 0.0017497500084573403, + 0.0017222500027855858, + 0.0017273749981541187 + ], + [ + 0.0017315000004600734, + 0.0017615410033613443, + 0.001716708007734269, + 0.0017242080066353083, + 0.0017215840052813292, + 0.00171050000062678, + 0.0015577090089209378, + 0.001702875000773929, + 0.0016229999891947955, + 0.0017139999981736764, + 0.0015669999993406236, + 0.0019808340002782643, + 0.0018244160019094124, + 0.001616042005480267, + 0.0017477090004831553, + 0.0016632920014671981, + 0.0017088750028051436, + 0.001574833004269749, + 0.001720999993267469, + 0.001714874990284443, + 0.0017124160076491535, + 0.0016957499901764095, + 0.0016879169997991994, + 0.001723291992675513, + 0.001734667006530799, + 0.001731666998239234, + 0.0017097500094678253, + 0.0017267910006921738, + 0.0017000420048134401, + 0.0016409170057158917, + 0.0016101670044008642, + 0.0016707920003682375, + 0.0017237080028280616, + 0.001706665992969647, + 0.0017248329968424514, + 0.0017028749862220138, + 0.0016563749959459528, + 0.0016024170035962015, + 0.0017251660028705373, + 0.0017204159958055243, + 0.0016044579970184714, + 0.0016053750005085021, + 0.001611042011063546, + 0.0016142499953275546, + 0.0017129580082837492, + 0.001618082998902537, + 0.00171237500035204, + 0.0016659579996485263, + 0.0016147080023074523, + 0.0016127080016303807, + 0.0016543329984415323, + 0.001665791991399601, + 0.0019130409928038716, + 0.001701875007711351, + 0.001734499994199723, + 0.0016165419947355986, + 0.001611415995284915, + 0.0017345420055789873, + 0.0017263750050915405, + 0.0017256249993806705, + 0.001720165993901901, + 0.00161929200112354, + 0.0017133750079665333, + 0.001644916002987884 + ], + [ + 0.0017540840053698048, + 0.001965374991414137, + 0.0016719580016797408, + 0.001622541996766813, + 0.0018061250011669472, + 0.0016844579949975014, + 0.0017678750009508803, + 0.0017632910021347925, + 0.0017187499906867743, + 0.0016272079956252128, + 0.0017263750050915405, + 0.0016839170129969716, + 0.0017309579998254776, + 0.0017398329946445301, + 0.0017267079965677112, + 0.001720416999887675, + 0.0017305420042248443, + 0.0016598340007476509, + 0.0016192499897442758, + 0.0016144170076586306, + 0.0017232500104000792, + 0.0017365829990012571, + 0.001738541992381215, + 0.0016511669964529574, + 0.0017061669932445511, + 0.0016290420026052743, + 0.0017363749939249828, + 0.0017198749992530793, + 0.0017327079985989258, + 0.0016220420075114816, + 0.0017261670000152662, + 0.0016678749962011352, + 0.0016212909977184609, + 0.0018936249980470166, + 0.0018597500020405278, + 0.0017610409995540977, + 0.0017839159991126508, + 0.001764375003403984, + 0.0017549579933984205, + 0.001657749991863966, + 0.0017137910035671666, + 0.0016282080032397062, + 0.0017507500015199184, + 0.001716957995085977, + 0.0016526669933227822, + 0.0017657919961493462, + 0.0017162080039270222, + 0.0017265000060433522, + 0.0015959589945850894, + 0.001752791999024339, + 0.001723832989227958, + 0.0017340830090688542, + 0.0017401669902028516, + 0.0017497080116299912, + 0.0017272910044994205, + 0.0016690420015947893, + 0.0016475419979542494, + 0.0017186670011142269, + 0.0017502080008853227, + 0.0017239169974345714, + 0.0017326249944744632, + 0.0017284999921685085, + 0.0017102919955505058, + 0.0016967090050457045 + ], + [ + 0.001724041998386383, + 0.0018318750080652535, + 0.0017680830060271546, + 0.0017164590099127963, + 0.001742833002936095, + 0.001745917004882358, + 0.001713167002890259, + 0.0017304159991908818, + 0.0015932920068735257, + 0.0016519999917363748, + 0.001674125000135973, + 0.0016713329969206825, + 0.001670083001954481, + 0.0016854169953148812, + 0.001820749996113591, + 0.001813958995626308, + 0.0017003329994622618, + 0.0017112909990828484, + 0.001759624996338971, + 0.0017702500044833869, + 0.0016833330009831116, + 0.001656166001339443, + 0.0017228329961653799, + 0.0016609159938525409, + 0.0019529580022208393, + 0.001628458994673565, + 0.0016427910013590008, + 0.0020374169980641454, + 0.001719958003377542, + 0.0016744589956942946, + 0.0016165409906534478, + 0.001750499999616295, + 0.0016628339944873005, + 0.0016035000007832423, + 0.0016146669950103387, + 0.0017175829998450354, + 0.0017193749954458326, + 0.0017070419999072328, + 0.0016755420074332505, + 0.0015940409939503297, + 0.0017188750061905012, + 0.0017264590132981539, + 0.0017003749962896109, + 0.0016778749995864928, + 0.0016765000036684796, + 0.0017225420015165582, + 0.0016259999974863604, + 0.001759958002367057, + 0.0017114169895648956, + 0.0017188329948112369, + 0.0017288329981965944, + 0.0017179580027004704, + 0.0017165839963126928, + 0.0016704169975128025, + 0.001638083005673252, + 0.0017123329889727756, + 0.0017372920119669288, + 0.001794833005988039, + 0.0017302499909419566, + 0.001723291992675513, + 0.001630124999792315, + 0.0016543750098207965, + 0.0017347500106552616, + 0.001753749995259568 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + ] + } + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 1.7391250003129244, + "p95": 1.7774333973648027, + "min": 1.7139999981736764, + "max": 1.786124994396232, + "iqr": 0.011625001206994057, + "mean": 1.7425918020308018, + "cv": 0.015339436848932061 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.9383749895496294, + "p95": 2.047320503334049, + "min": 1.7056249926099554, + "max": 2.17479100683704, + "iqr": 0.10879149340325966, + "mean": 1.9304902126480425, + "cv": 0.04192674022776731 + }, + "total_ms": { + "count": 5, + "median": 123.09825100237504, + "p95": 124.12711280630901, + "min": 122.69883298722561, + "max": 124.18488098774105, + "iqr": 0.9566691442159936, + "mean": 123.36347519885749, + "cv": 0.005206712464894358 + }, + "tokens_per_second": { + "count": 5, + "median": 519.9099051274513, + "p95": 521.3982440286783, + "min": 515.3606420601053, + "max": 521.6023530285993, + "iqr": 4.019697098128404, + "mean": 518.8033638352032, + "cv": 0.005198675548885689 + }, + "median_step_ms_by_index": [ + 1.7391250003129244, + 1.8942089955089614, + 1.8235419993288815, + 1.8724580004345626, + 1.8210830021416768, + 1.8734169862000272, + 1.8051249935524538, + 1.8051670049317181, + 1.8987079965882003, + 1.8829580076271668, + 1.8706669943640009, + 1.8672500009415671, + 1.902040996355936, + 1.8909999926108867, + 1.8869580089813098, + 1.894874993013218, + 1.9082499929936603, + 1.8822080019162968, + 1.9041670020669699, + 1.9358329882379621, + 1.8627499957801774, + 1.8645410018507391, + 1.9189159938832745, + 1.9358330027898774, + 1.935207998030819, + 1.8789160094456747, + 1.9029169925488532, + 1.8684579990804195, + 1.8787919980240986, + 1.9039170001633465, + 1.9279169937362894, + 1.8764999986160547, + 1.9006249931408092, + 2.0150829950580373, + 1.9461670017335564, + 1.9349580106791109, + 1.991458993870765, + 2.014042009250261, + 1.98304200603161, + 2.0183749875286594, + 2.0124590082559735, + 1.9629170128609985, + 1.909292011987418, + 1.9687499880092219, + 1.951957994606346, + 1.9651670008897781, + 2.022250002482906, + 1.9656670046970248, + 2.012958997511305, + 1.9950410060118884, + 1.9615829951362684, + 1.9584580004448071, + 1.9934169977204874, + 1.9829160009976476, + 1.9810420053545386, + 1.9927909888792783, + 1.9936660100938752, + 1.97199999820441, + 1.9617920042946935, + 1.9892910058842972, + 1.9809160003205761, + 1.9646250002551824, + 1.9710829947143793, + 1.9985420076409355 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.001731042008032091, + 0.0018129170057363808, + 0.0018147920054616407, + 0.0018724580004345626, + 0.0018210830021416768, + 0.0018734169862000272, + 0.0019050420087296516, + 0.0018921660084743053, + 0.0018987079965882003, + 0.0018829580076271668, + 0.0018747499998426065, + 0.0017857080092653632, + 0.001847832987550646, + 0.0018909999926108867, + 0.0018248330015921965, + 0.0018756659992504865, + 0.0018186249944847077, + 0.0018677920015761629, + 0.001855459006037563, + 0.001861541997641325, + 0.0018337500077905133, + 0.001819249999243766, + 0.0018984589987667277, + 0.0019358330027898774, + 0.0019135830079903826, + 0.0019236250082030892, + 0.001940208996529691, + 0.0019249169999966398, + 0.0019472499989205971, + 0.0019011660042451695, + 0.0019304169982206076, + 0.001944374991580844, + 0.0019353339885128662, + 0.0020509999885689467, + 0.0019461670017335564, + 0.0019909170077880844, + 0.0020119170076213777, + 0.0020061249961145222, + 0.0019388330110814422, + 0.0019775840046349913, + 0.0020124590082559735, + 0.0020086250005988404, + 0.0020092080085305497, + 0.0019418339943513274, + 0.0019451250118436292, + 0.002063084000837989, + 0.0020291660039220005, + 0.002059582999208942, + 0.0020591249922290444, + 0.002003708010306582, + 0.0018783330015139654, + 0.00217479100683704, + 0.002035040990449488, + 0.0019829160009976476, + 0.0019810420053545386, + 0.002002583001740277, + 0.0019816249987343326, + 0.0019998329953523353, + 0.0019522919901646674, + 0.001989291005884297, + 0.001980916000320576, + 0.001993541998672299, + 0.001986208008020185, + 0.002047208006842993 + ], + [ + 0.0017391250003129244, + 0.0018942089955089614, + 0.0018235419993288815, + 0.001830082997912541, + 0.0018412920035189018, + 0.0017610420036362484, + 0.0018249170097988099, + 0.001759999999194406, + 0.0019052499992540106, + 0.0017923330015037209, + 0.001786875000107102, + 0.0019302090076962486, + 0.0019272909994469956, + 0.0019212079932913184, + 0.0018995420105056837, + 0.001894874993013218, + 0.0019085410021943972, + 0.0018327089928789064, + 0.0019041670020669699, + 0.00204454199410975, + 0.001898832997540012, + 0.0018585839861771092, + 0.0018098750006174669, + 0.0017833749880082905, + 0.0019701670098584145, + 0.0018509579967940226, + 0.001890667001134716, + 0.0018657089967746288, + 0.001826499996241182, + 0.0020804579980904236, + 0.0018568329978734255, + 0.0018622079951455817, + 0.0018492079980205745, + 0.0018969999946421012, + 0.0019894169963663444, + 0.0020147500035818666, + 0.002060332990367897, + 0.002014042009250261, + 0.002012750002904795, + 0.0020215830008964986, + 0.0020263750047888607, + 0.0019232920021750033, + 0.00190395800746046, + 0.00198920899129007, + 0.001951957994606346, + 0.0019444170029601082, + 0.002022250002482906, + 0.0019580000080168247, + 0.0020337500027380884, + 0.0019461659976514056, + 0.0019646250002551824, + 0.0020089580066269264, + 0.001979958004085347, + 0.0019802500028163195, + 0.0019850840035360307, + 0.001995957994950004, + 0.0019936660100938752, + 0.0019483329961076379, + 0.0019831670069834217, + 0.001995625003473833, + 0.002004791997023858, + 0.0019646250002551824, + 0.0019710829947143793, + 0.001987750001717359 + ], + [ + 0.0017139999981736764, + 0.0019291670032544062, + 0.0018943749892059714, + 0.0018867920007323846, + 0.0017174169915961102, + 0.002068750007310882, + 0.0017917090008268133, + 0.001713500008918345, + 0.0017482499970356002, + 0.00171929199132137, + 0.0018103329930454493, + 0.0017175419925479218, + 0.00203237499226816, + 0.0017056249926099554, + 0.0017242920002900064, + 0.0018867500039050356, + 0.001935207998030819, + 0.0019762919982895255, + 0.001953083003172651, + 0.002024750006967224, + 0.0021200839983066544, + 0.002076791992294602, + 0.002103749997331761, + 0.00203483400400728, + 0.0019678329990711063, + 0.0019394999981159344, + 0.0019029169925488532, + 0.001984583999728784, + 0.0019328330090502277, + 0.0019450839899946004, + 0.0019325419998494908, + 0.001805916996090673, + 0.0019006249931408092, + 0.0020523750135907903, + 0.002020500003709458, + 0.001876333000836894, + 0.0019473330030450597, + 0.002062250001472421, + 0.002010249998420477, + 0.0020183749875286594, + 0.0019428750092629343, + 0.0019629170128609985, + 0.002047582995146513, + 0.0020391250000102445, + 0.0020406249968800694, + 0.0019637090008473024, + 0.002014166006119922, + 0.001944791991263628, + 0.002012958997511305, + 0.00195295900630299, + 0.0019425000064074993, + 0.0019383749895496294, + 0.001977125008124858, + 0.0020280419994378462, + 0.001973083009943366, + 0.0019472919957479462, + 0.002041083003859967, + 0.001980125001864508, + 0.0019509170087985694, + 0.00194133298646193, + 0.002031625001109205, + 0.001950833000591956, + 0.0019482500065350905, + 0.002028375005465932 + ], + [ + 0.001742667009239085, + 0.001821665995521471, + 0.0017600840074010193, + 0.0018817920063156635, + 0.0018112909892806783, + 0.0017632500093895942, + 0.001770416012732312, + 0.0018051670049317181, + 0.0019134169997414574, + 0.0019145420083077624, + 0.0019182920077582821, + 0.0019172079919371754, + 0.001816750009311363, + 0.0018957079882966354, + 0.001931083999807015, + 0.001923417003126815, + 0.0018054579995805398, + 0.0019126249972032383, + 0.0018483339954400435, + 0.0018554999987827614, + 0.0018627499957801774, + 0.0018645410018507391, + 0.0019189159938832745, + 0.0018535409908508882, + 0.00185087499266956, + 0.0018789160094456747, + 0.0019071659917244688, + 0.0018462080042809248, + 0.0018313339969608933, + 0.0019039170001633465, + 0.0019279169937362894, + 0.0019436250004218891, + 0.0019489999976940453, + 0.0020150829950580373, + 0.0019278749969089404, + 0.0019266249873908237, + 0.001991458993870765, + 0.002023749999352731, + 0.00198304200603161, + 0.0020037079957546666, + 0.0020446249982342124, + 0.001926041004480794, + 0.001909292011987418, + 0.0019328750058775768, + 0.0019430419924901798, + 0.002041291998466477, + 0.0020265420025680214, + 0.001966333991731517, + 0.0019395420094951987, + 0.002006832990446128, + 0.0019615829951362684, + 0.0019375830015633255, + 0.0020222079911036417, + 0.0019679590041050687, + 0.0019437089940765873, + 0.0019099159981124103, + 0.0020297500013839453, + 0.00197199999820441, + 0.0019617920042946935, + 0.002022416010731831, + 0.001854457994340919, + 0.0020780000049853697, + 0.0019495839951559901, + 0.0019325410103192553 + ], + [ + 0.001786124994396232, + 0.0019108749984297901, + 0.0018949590012198314, + 0.0017700830067042261, + 0.0018452499934937805, + 0.0019092080037808046, + 0.0018051249935524538, + 0.001814333998481743, + 0.0017924999992828816, + 0.0019161250093020499, + 0.0018706669943640009, + 0.0018672500009415671, + 0.001902040996355936, + 0.0018202080100309104, + 0.0018869580089813098, + 0.0019110839930363, + 0.0019082499929936603, + 0.0018822080019162968, + 0.001936540997121483, + 0.0019358329882379621, + 0.001861500000813976, + 0.0019263750000391155, + 0.0019382090104045346, + 0.0019467909878585488, + 0.001935207998030819, + 0.0018430409982101992, + 0.001887249993160367, + 0.0018684579990804195, + 0.0018787919980240986, + 0.0018569999956525862, + 0.0018833329959306866, + 0.0018764999986160547, + 0.001826457999413833, + 0.0019596250058384612, + 0.0018989169911947101, + 0.0019349580106791109, + 0.0018759580125333741, + 0.0019634999916888773, + 0.001952332997461781, + 0.002146834012819454, + 0.001979791995836422, + 0.0019663340062834322, + 0.0019082079961663112, + 0.001968749988009222, + 0.0019640829996205866, + 0.001965167000889778, + 0.0019533330050762743, + 0.001965667004697025, + 0.0019629589951364323, + 0.0019950410060118884, + 0.001998457999434322, + 0.001958458000444807, + 0.0019934169977204874, + 0.0019894589931936935, + 0.0019961669895565137, + 0.0019927909888792783, + 0.001987540992558934, + 0.001964708004379645, + 0.001995915998122655, + 0.0019505829986883327, + 0.0019286669994471595, + 0.0019430829997872934, + 0.0019855829887092113, + 0.0019985420076409355 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_mha": { + "first_token_ms": { + "count": 5, + "median": 1.915417000418529, + "p95": 1.975207804935053, + "min": 1.893334003398195, + "max": 1.9891660049324855, + "iqr": 0.018875012756325305, + "mean": 1.923558401176706, + "cv": 0.01985287788844724 + }, + "decode_ms_per_step": { + "count": 315, + "median": 2.065458000288345, + "p95": 2.239262600778602, + "min": 1.8542079924372956, + "max": 2.6914579939329997, + "iqr": 0.15806249575689435, + "mean": 2.073881479621392, + "cv": 0.05040240840930097 + }, + "total_ms": { + "count": 5, + "median": 132.42058300238568, + "p95": 133.4710664465092, + "min": 131.4552920666756, + "max": 133.5704160592286, + "iqr": 0.7031690329313278, + "mean": 132.5780916173244, + "cv": 0.006032866726530549 + }, + "tokens_per_second": { + "count": 5, + "median": 483.3085502942317, + "p95": 486.18431452677805, + "min": 479.14801711496307, + "max": 486.8575391208935, + "iqr": 2.5547968778930112, + "mean": 482.74842839056566, + "cv": 0.0060389876934192795 + }, + "median_step_ms_by_index": [ + 1.915417000418529, + 1.975000006495975, + 1.957125001354143, + 1.9713340007001534, + 1.9589580042520538, + 1.9865000067511573, + 2.0012090099044144, + 1.9972080044681206, + 1.9419159943936393, + 1.996209000935778, + 1.9814160041278228, + 1.971790989045985, + 2.016208993154578, + 1.9648339948616922, + 1.9476669986033812, + 1.9480000046314672, + 2.0109170000068843, + 2.0293749985285103, + 2.016917002038099, + 1.9809170044027269, + 2.000749998842366, + 2.0287920051487163, + 2.0154579979134724, + 2.0259579905541614, + 1.9867080118274316, + 2.004791997023858, + 2.0284999918658286, + 2.0088750025024638, + 1.9998749921796843, + 1.9747079932130873, + 1.9736249960260466, + 1.9692500063683838, + 2.00395799765829, + 2.1010420023230836, + 2.093124989187345, + 2.1109579975018278, + 2.1162500052014366, + 2.1413330105133355, + 2.093667004373856, + 2.1024169982410967, + 2.0928330050082877, + 2.1499590075109154, + 2.147915991372429, + 2.151791995856911, + 2.1552500111283734, + 2.122624995536171, + 2.1465829922817647, + 2.13425001129508, + 2.1749169973190874, + 2.17162499029655, + 2.140291006071493, + 2.133250003680587, + 2.1863750007469207, + 2.1880419953959063, + 2.158292001695372, + 2.1732499881181866, + 2.1362919942475855, + 2.200707996962592, + 2.2057500027585775, + 2.211375001934357, + 2.226249998784624, + 2.1733749890699983, + 2.190415994846262, + 2.160125004593283 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.001893334003398195, + 0.001959375003934838, + 0.0019149580039083958, + 0.0019713340007001534, + 0.001958958004252054, + 0.0019789579964708537, + 0.001978250002139248, + 0.0019627080037025735, + 0.0019762500014621764, + 0.0019716250099008903, + 0.0020157910039415583, + 0.0020169999916106462, + 0.002016208993154578, + 0.0019937080069212243, + 0.0019934590090997517, + 0.0019322500011185184, + 0.001979833992663771, + 0.0020293749985285103, + 0.0019814579864032567, + 0.001980917004402727, + 0.0020195000106468797, + 0.0020334579894552007, + 0.0020179579878458753, + 0.0020037909998791292, + 0.0019567090057535097, + 0.00197124999249354, + 0.0019833330006804317, + 0.002008708004723303, + 0.001957916989340447, + 0.0019716250099008903, + 0.0018898749985964969, + 0.0019692500063683838, + 0.001942583010531962, + 0.001962249996722676, + 0.002227999997558072, + 0.0021198749891482294, + 0.002036542005953379, + 0.0023447500134352595, + 0.002149707986973226, + 0.0021591249969787896, + 0.0021712919988203794, + 0.0020904999983031303, + 0.002147915991372429, + 0.002151791995856911, + 0.0021648749971063808, + 0.002143999998224899, + 0.0021357079967856407, + 0.00213425001129508, + 0.002194667002186179, + 0.0022524170053657144, + 0.002229833000455983, + 0.002133250003680587, + 0.0021863750007469207, + 0.0021850000048289075, + 0.002169083003536798, + 0.002125832994352095, + 0.0021487920021172613, + 0.002200707996962592, + 0.00223333400208503, + 0.002244915987830609, + 0.0022555420000571758, + 0.002202667004894465, + 0.0021257499902276322, + 0.002160125004593283 + ], + [ + 0.0019193750049453229, + 0.0019370000081835315, + 0.002006291993893683, + 0.002101541991578415, + 0.0019104999955743551, + 0.0019865000067511573, + 0.0020012090099044144, + 0.0019214999920222908, + 0.0019419159943936393, + 0.0019979580101789907, + 0.001930084006744437, + 0.0019545830000424758, + 0.001998249994358048, + 0.0019265000009909272, + 0.0019476669986033812, + 0.00196770801267121, + 0.0019987079867860302, + 0.002073083000141196, + 0.002016917002038099, + 0.00204695800493937, + 0.0020307919912738726, + 0.0020287920051487163, + 0.0020307080121710896, + 0.0020716659928439185, + 0.002054291995591484, + 0.0021263329981593415, + 0.0020935830107191578, + 0.002024875007919036, + 0.002192333005950786, + 0.00212433299748227, + 0.002039375001913868, + 0.0019629590096883476, + 0.002065458000288345, + 0.0021467499900609255, + 0.002093124989187345, + 0.0021109579975018278, + 0.0021162500052014366, + 0.002068875008262694, + 0.002093667004373856, + 0.002090916008455679, + 0.0021086250053485855, + 0.0021499590075109154, + 0.002093457995215431, + 0.0022941670031286776, + 0.0022522920044139028, + 0.0022092910076025873, + 0.002154625006369315, + 0.0021498750138562173, + 0.002178041002480313, + 0.0021421249984996393, + 0.002140291006071493, + 0.002190667000832036, + 0.0022410419915104285, + 0.0022461250046035275, + 0.002149166990420781, + 0.0021732499881181866, + 0.0021798330126330256, + 0.0021713339956477284, + 0.002164374993299134, + 0.0021847089956281707, + 0.0022385000047506765, + 0.0021733749890699983, + 0.0022903749922988936, + 0.0021446250029839575 + ], + [ + 0.0019004999921889976, + 0.001995750004425645, + 0.001949500001501292, + 0.001923750009154901, + 0.0020210840011714026, + 0.001862499993876554, + 0.002023749999352731, + 0.0019972080044681206, + 0.0019095840107183903, + 0.0020350410050014034, + 0.001981416004127823, + 0.00195295900630299, + 0.0020465410052565858, + 0.001964833994861692, + 0.0019357089913683012, + 0.0018739579973043874, + 0.0020789589907508343, + 0.0019907079986296594, + 0.001997125000343658, + 0.001980332992388867, + 0.0019975409959442914, + 0.0020080420072190464, + 0.001986832998227328, + 0.0019849579985020682, + 0.0019867080118274316, + 0.001979624998057261, + 0.001976375002413988, + 0.001964209004654549, + 0.0019693330104928464, + 0.0019747079932130873, + 0.0019736249960260466, + 0.001966249998076819, + 0.0019901670020772144, + 0.0021010420023230836, + 0.002068917005090043, + 0.002063749998342246, + 0.0022205830027814955, + 0.0021413330105133355, + 0.0021035419922554865, + 0.002081124999676831, + 0.002085500003886409, + 0.0021756659989478067, + 0.0021576250001089647, + 0.0020789590053027496, + 0.0021552500111283734, + 0.002114041999448091, + 0.0021152500121388584, + 0.0021482079901034012, + 0.0021749169973190874, + 0.002124708000337705, + 0.002090832989779301, + 0.0020749159884871915, + 0.0021101250022184104, + 0.00204341700009536, + 0.002123167010722682, + 0.0022069170081522316, + 0.002086667009280063, + 0.0023809999984223396, + 0.0022097500041127205, + 0.002211375001934357, + 0.0022082919895183295, + 0.0021392500057118014, + 0.002143916004570201, + 0.0021356670040404424 + ], + [ + 0.0019891660049324855, + 0.0019844170019496232, + 0.0019924160005757585, + 0.0019431249966146424, + 0.0018542079924372956, + 0.0020736669976031408, + 0.002010333992075175, + 0.002043000000412576, + 0.0018895829998655245, + 0.001996209000935778, + 0.001966625000932254, + 0.002024000001256354, + 0.0020183330052532256, + 0.0019562499946914613, + 0.001931042002979666, + 0.0019480000046314672, + 0.0020222079911036417, + 0.0020940419926773757, + 0.0020435839978745207, + 0.002029916999163106, + 0.002000749998842366, + 0.0020296669972594827, + 0.0020035420020576566, + 0.0020362499926704913, + 0.0020273330010240898, + 0.0020286250073695555, + 0.002039041995885782, + 0.0020088750025024638, + 0.0021250420104479417, + 0.0020227500062901527, + 0.002031416996032931, + 0.0020274580019759014, + 0.002029666007729247, + 0.0021269590070005506, + 0.00210416701156646, + 0.0021495830005733296, + 0.0021352500043576583, + 0.0021215409942669794, + 0.0020893749897368252, + 0.002130624998244457, + 0.002078083998640068, + 0.002081458005704917, + 0.002092166992952116, + 0.002134958005626686, + 0.0021287089912220836, + 0.002082582999719307, + 0.002238208006019704, + 0.0020756670128321275, + 0.0021554999984800816, + 0.0022507499961648136, + 0.002179374991101213, + 0.0021767089929198846, + 0.0021292080054990947, + 0.0023769580002408475, + 0.0021905409957980737, + 0.0021682920050807297, + 0.0020604999881470576, + 0.0022465840011136606, + 0.0022057500027585775, + 0.002157334005460143, + 0.002226249998784624, + 0.0021964159968774766, + 0.002190415994846262, + 0.0021732080058427528 + ], + [ + 0.001915417000418529, + 0.001975000006495975, + 0.001957125001354143, + 0.001973458012798801, + 0.001969875011127442, + 0.0020045830024173483, + 0.0019897080055670813, + 0.002006666996749118, + 0.0019791669910773635, + 0.0019793340034084395, + 0.001988541000173427, + 0.001971790989045985, + 0.0020004580001113936, + 0.0019785839976975694, + 0.001976957995793782, + 0.001966790994629264, + 0.0020109170000068843, + 0.0020217499986756593, + 0.0020289169915486127, + 0.001974291997612454, + 0.0019793340034084395, + 0.002017125007114373, + 0.0020154579979134724, + 0.0020259579905541614, + 0.001981916997465305, + 0.002004791997023858, + 0.0020284999918658286, + 0.00202358300157357, + 0.0019998749921796843, + 0.0019291249918751419, + 0.0019292919896543026, + 0.001994709004065953, + 0.00200395799765829, + 0.002013292003539391, + 0.002016167010879144, + 0.0020852499874308705, + 0.0020959160028724, + 0.002146250000805594, + 0.0020314160065026954, + 0.0021024169982410967, + 0.0020928330050082877, + 0.002263707996462472, + 0.0022044579964131117, + 0.0021644170046783984, + 0.0020845830003963783, + 0.002122624995536171, + 0.0021465829922817647, + 0.002131375003955327, + 0.0021726249979110435, + 0.00217162499029655, + 0.0021282919915392995, + 0.002050292008789256, + 0.0021944580075796694, + 0.0021880419953959063, + 0.002158292001695372, + 0.002336542005650699, + 0.0021362919942475855, + 0.0020667500066338107, + 0.002137832998414524, + 0.0026914579939329997, + 0.0020886659913230687, + 0.002165833007893525, + 0.0022101250069681555, + 0.002169125000364147 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "speedup_cached_vs_uncached_sdpa": 1.1223148977466313, + "speedup_cached_vs_uncached_mha": 1.2073087299104224 + }, + { + "batch": 1, + "prompt_len": 512, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_mha": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 3.652833998785354, + "p95": 3.96474999433849, + "min": 3.4722919954219833, + "max": 3.98549999226816, + "iqr": 0.3770000039367005, + "mean": 3.6994251975556836, + "cv": 0.06145366467863395 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.8113340047420934, + "p95": 2.3640624014660716, + "min": 1.6562919918214902, + "max": 5.693333005183376, + "iqr": 0.40068799717118964, + "mean": 1.964166958258295, + "cv": 0.2071194981419471 + }, + "total_ms": { + "count": 5, + "median": 124.93279491900466, + "p95": 140.16791061731055, + "min": 118.90300102822948, + "max": 143.50216902676038, + "iqr": 3.790001093875617, + "mean": 127.44194356782828, + "cv": 0.07411746437834088 + }, + "tokens_per_second": { + "count": 5, + "median": 512.2754200887919, + "p95": 534.6335617185476, + "min": 445.98629020070933, + "max": 538.2538661476287, + "iqr": 15.543359785084988, + "mean": 504.25538093129836, + "cv": 0.06918470073381114 + }, + "median_step_ms_by_index": [ + 3.652833998785354, + 2.5923750072252005, + 2.335416997084394, + 2.278416999615729, + 1.9146249978803098, + 2.133166999556124, + 2.10529200558085, + 2.2397089924197644, + 2.2057919995859265, + 1.861500000813976, + 2.250958001241088, + 1.753749995259568, + 1.799999998183921, + 1.7802080110413954, + 1.8002500000875443, + 1.820959005272016, + 2.1814170031575486, + 2.0360829948913306, + 2.166291989851743, + 1.810457993997261, + 1.7843329987954348, + 1.8169579998357221, + 2.1819590037921444, + 1.808957997127436, + 2.271750010550022, + 1.80095799441915, + 1.8189580005127937, + 1.799834004486911, + 1.8012079963227734, + 1.8011659994954243, + 1.8479590071365237, + 1.7989590123761445, + 1.829875007388182, + 1.8149170064134523, + 1.8030830105999485, + 1.8062909948639572, + 1.771000010194257, + 1.7438750073779374, + 1.7324590007774532, + 1.801875012461096, + 1.7294579884037375, + 1.7298750026384369, + 1.7888750007841736, + 1.8617910100147128, + 1.859249998233281, + 1.8164579960284755, + 1.807666994864121, + 1.8230830028187484, + 1.8280000076629221, + 1.875458998256363, + 1.811291993362829, + 1.798874989617616, + 1.7964580038096756, + 1.778457997716032, + 1.7204579926328734, + 1.7966669984161854, + 1.7948329914361238, + 1.7894170014187694, + 1.7861670057754964, + 1.8437089893268421, + 1.796416996512562, + 1.8369580066064373, + 1.783415995305404, + 1.7977500101551414 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.003652833998785354, + 0.0025923750072252005, + 0.002335416997084394, + 0.0019330829964019358, + 0.0023212089872686192, + 0.002273833000799641, + 0.00210529200558085, + 0.0019376669952180237, + 0.0022057919995859265, + 0.002273625010275282, + 0.002250958001241088, + 0.0018413750076433644, + 0.0023093749914551154, + 0.0022796249977545813, + 0.0018002500000875443, + 0.0017401249933755025, + 0.0017313749995082617, + 0.002205917000537738, + 0.002166291989851743, + 0.0018423330038785934, + 0.0017118749965447932, + 0.0022263330029090866, + 0.0021569579985225573, + 0.0021109579975018278, + 0.002301874992554076, + 0.0023887919960543513, + 0.002223000003141351, + 0.0017133750079665333, + 0.001847999999881722, + 0.0018256670009577647, + 0.002215582993812859, + 0.0018954580009449273, + 0.0018138330051442608, + 0.0017458749935030937, + 0.0018542089965194464, + 0.0018270000000484288, + 0.001771000010194257, + 0.0017101660050684586, + 0.001739250001264736, + 0.0022941670031286776, + 0.0017794169980334118, + 0.0017298750026384369, + 0.0016958330088527873, + 0.0018617910100147128, + 0.0018473749951226637, + 0.0018042080046143383, + 0.0017905840068124235, + 0.0018100000015692785, + 0.0017964159924304113, + 0.0024091670056805015, + 0.001811291993362829, + 0.00180004199501127, + 0.0017976249946514145, + 0.001778457997716032, + 0.0017204579926328734, + 0.0017619169957470149, + 0.0017065419960999861, + 0.001745125002344139, + 0.0018442500004312024, + 0.0018437089893268421, + 0.001831666988437064, + 0.0018369580066064373, + 0.0017996250098804012, + 0.0018564170022727922 + ], + [ + 0.00398549999226816, + 0.002662083992618136, + 0.002412667003227398, + 0.002278416999615729, + 0.0017028329893946648, + 0.0023231249942909926, + 0.002271042001666501, + 0.0022972090082475916, + 0.002181125004426576, + 0.0022592500026803464, + 0.0022696249943692237, + 0.0022800419974373654, + 0.001799999998183921, + 0.0021886670001549646, + 0.0017827919946284965, + 0.0017955410003196448, + 0.0019015840080101043, + 0.0022899590112501755, + 0.0022446670045610517, + 0.0016890830011107028, + 0.0017843329987954348, + 0.0017945419967873022, + 0.0022760829888284206, + 0.001808957997127436, + 0.002271750010550022, + 0.001797707998775877, + 0.0017737500020302832, + 0.0023053329932736233, + 0.0022387079952750355, + 0.0016940420027822256, + 0.00225387500540819, + 0.0017820840002968907, + 0.001829875007388182, + 0.001910208011395298, + 0.0018030830105999485, + 0.001785542001016438, + 0.0017875000048661605, + 0.0017809580022003502, + 0.0016621660033706576, + 0.0017968330066651106, + 0.0016562919918214902, + 0.0017107079911511391, + 0.0017850000003818423, + 0.00178133299050387, + 0.0019400830060476437, + 0.005693333005183376, + 0.004356667006504722, + 0.004639791994122788, + 0.0022621659882133827, + 0.004512042010901496, + 0.002129125001374632, + 0.0017635000112932175, + 0.0019052919960813597, + 0.0023467079881811514, + 0.002387041997280903, + 0.0024432920035906136, + 0.0023118330136639997, + 0.0023518749949289486, + 0.0026557920064078644, + 0.002303542001754977, + 0.002310666997800581, + 0.0019144590041832998, + 0.0017755419976310804, + 0.0018135409918613732 + ], + [ + 0.00350474999868311, + 0.002517999993870035, + 0.0017100419936468825, + 0.0023442079982487485, + 0.0018262499943375587, + 0.0018151249969378114, + 0.001816375006455928, + 0.0023208329948829487, + 0.0016922919894568622, + 0.0017330420087091625, + 0.0018376250081928447, + 0.0017239169974345714, + 0.0017393339949194342, + 0.001696541003184393, + 0.0016696250095264986, + 0.0019530419958755374, + 0.0021814170031575486, + 0.0018317500071134418, + 0.0018428330076858401, + 0.001810457993997261, + 0.001673749997280538, + 0.0018209169938927516, + 0.0018163339991588145, + 0.001703166009974666, + 0.0017473749903729185, + 0.0017177089903270826, + 0.001810874993680045, + 0.001799834004486911, + 0.0018012079963227734, + 0.0018011659994954243, + 0.0018479590071365237, + 0.001796333002857864, + 0.0018017910042544827, + 0.0017943749990081415, + 0.0022986670082900673, + 0.00178116699680686, + 0.0016935840103542432, + 0.0017438750073779374, + 0.0016947080002864823, + 0.0017596250108908862, + 0.001796791999367997, + 0.0017856250051409006, + 0.001803249993827194, + 0.0024690419959370047, + 0.001859249998233281, + 0.0018408329924568534, + 0.001835208007832989, + 0.0018230830028187484, + 0.0018280000076629221, + 0.001829500004532747, + 0.0018060419970424846, + 0.0017294579884037375, + 0.0017277919978369027, + 0.001709707998088561, + 0.0016976250044535846, + 0.0017978749965550378, + 0.0018046250042971224, + 0.0018059159920085222, + 0.0017861670057754964, + 0.0017790419951779768, + 0.0016951249999692664, + 0.0017169580096378922, + 0.001706458002445288, + 0.0017977500101551414 + ], + [ + 0.0038817500026198104, + 0.003092416998697445, + 0.0017951249901670963, + 0.0021189169929130003, + 0.0019146249978803098, + 0.0017497910012025386, + 0.0022918330068932846, + 0.0022397089924197644, + 0.0022913340071681887, + 0.0017134169902419671, + 0.00235912500647828, + 0.001673542006756179, + 0.002299458996276371, + 0.0017802080110413954, + 0.0021733339963248, + 0.002124500009813346, + 0.0023154579976107925, + 0.001815790994442068, + 0.0018481250008335337, + 0.0018078749999403954, + 0.0021989160013617948, + 0.0018169579998357221, + 0.0023027080023894086, + 0.0017931659967871383, + 0.0023271249956451356, + 0.0022345839970512316, + 0.0018189580005127937, + 0.0018001249991357327, + 0.001793749994249083, + 0.0019143749959766865, + 0.0018114169943146408, + 0.0017989590123761445, + 0.0022796249977545813, + 0.0022475419973488897, + 0.001790041002095677, + 0.0018113340047420934, + 0.0018007089965976775, + 0.0017963329883059487, + 0.0017324590007774532, + 0.0018456669931765646, + 0.0017294579884037375, + 0.0016909159894566983, + 0.0017897920042742044, + 0.0017998749972321093, + 0.0018077080021612346, + 0.0017925830034073442, + 0.001807666994864121, + 0.0018726659909589216, + 0.0018852089997380972, + 0.001875458998256363, + 0.0018397079984424636, + 0.001859208001405932, + 0.0017043340048985556, + 0.0017431250016670674, + 0.0017037499928846955, + 0.0016858339949976653, + 0.0016737919941078871, + 0.0016573750035604462, + 0.0017258750012842938, + 0.0018907500052591786, + 0.0017620419966988266, + 0.0019349999929545447, + 0.001783415995305404, + 0.0017161669966299087 + ], + [ + 0.0034722919954219833, + 0.001972041995031759, + 0.0023400000063702464, + 0.002320125000551343, + 0.002244416988105513, + 0.002133166999556124, + 0.001686832998530008, + 0.0017618749989196658, + 0.002234625004348345, + 0.001861500000813976, + 0.002246165997348726, + 0.001753749995259568, + 0.0017514999926788732, + 0.001656916007050313, + 0.002375582989770919, + 0.001820959005272016, + 0.0022137499909149483, + 0.0020360829948913306, + 0.0021823749993927777, + 0.001977667008759454, + 0.0023331660049734637, + 0.0018074580002576113, + 0.0021819590037921444, + 0.0019503749936120585, + 0.002200167000410147, + 0.00180095799441915, + 0.002292499993927777, + 0.0016941250069066882, + 0.001788749999832362, + 0.0017797920008888468, + 0.0017607089976081625, + 0.0018383329879725352, + 0.0018573749985080212, + 0.0018149170064134523, + 0.001791999995475635, + 0.0018062909948639572, + 0.0017027919966494665, + 0.0016892499988898635, + 0.0018014169909292832, + 0.001801875012461096, + 0.001688042000751011, + 0.0017966250015888363, + 0.0017888750007841736, + 0.0022311249922495335, + 0.0019002500048372895, + 0.0018164579960284755, + 0.001807666994864121, + 0.0018001669959630817, + 0.0018034999957308173, + 0.0017003329994622618, + 0.0017020419909385964, + 0.001798874989617616, + 0.0017964580038096756, + 0.0018562079931143671, + 0.0018042499868897721, + 0.0017966669984161854, + 0.0017948329914361238, + 0.0017894170014187694, + 0.0016828329971758649, + 0.0018329169979551807, + 0.001796416996512562, + 0.0017411659937351942, + 0.0017977500101551414, + 0.0017821670044213533 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + ] + } + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 3.749124996829778, + "p95": 4.212258002371527, + "min": 3.465332993073389, + "max": 4.282458001398481, + "iqr": 0.4234580119373277, + "mean": 3.7872747983783484, + "cv": 0.0884729745439641 + }, + "decode_ms_per_step": { + "count": 315, + "median": 3.903375007212162, + "p95": 4.293112402956467, + "min": 3.4785839961841702, + "max": 4.685583000537008, + "iqr": 0.3771460033021867, + "mean": 3.9416914313776594, + "cv": 0.05589050240326445 + }, + "total_ms": { + "count": 5, + "median": 247.6499169570161, + "p95": 261.11508941394277, + "min": 244.9977970100008, + "max": 261.2380380160175, + "iqr": 14.56316711846739, + "mean": 252.1138349751709, + "cv": 0.032154893631806264 + }, + "tokens_per_second": { + "count": 5, + "median": 258.42932146473646, + "p95": 261.00127647987193, + "min": 244.98729391037574, + "max": 261.22683869433945, + "iqr": 14.533871987644261, + "mean": 254.06152746516221, + "cv": 0.031818853558642265 + }, + "median_step_ms_by_index": [ + 3.749124996829778, + 3.946583005017601, + 3.903375007212162, + 3.761083004064858, + 3.7071669939905405, + 3.721458007930778, + 3.6846250004600734, + 3.763540997169912, + 3.710999997565523, + 3.6723340017488226, + 3.6864169960608706, + 3.715250000823289, + 3.7791249924339354, + 3.900750001776032, + 3.7313339998945594, + 3.7357910041464493, + 3.8012919976608828, + 3.754040997591801, + 3.8232920051086694, + 3.727000002982095, + 3.7247500004014, + 3.7574169982690364, + 4.097041004570201, + 3.840290999505669, + 3.7277090013958514, + 3.71970800915733, + 3.895750007359311, + 3.9279170014197007, + 3.7387499905889854, + 3.7156670005060732, + 3.809709000051953, + 3.979540997534059, + 3.916708010365255, + 3.834792005363852, + 3.8685829931637272, + 4.176540998741984, + 4.032791999634355, + 3.822833998128772, + 3.9398749941028655, + 4.2390829912619665, + 3.9056669920682907, + 3.8197079993551597, + 3.8880829961271957, + 3.9580410084454343, + 3.838249991531484, + 3.8000420026946813, + 3.8917499914532527, + 3.963333001593128, + 4.258500004652888, + 4.128208995098248, + 3.8817919994471595, + 4.029500007163733, + 4.091374998097308, + 4.285332994186319, + 3.948375000618398, + 4.014459002064541, + 4.202082986012101, + 3.879208001308143, + 4.248625002219342, + 4.065874993102625, + 4.094792006071657, + 3.928209000150673, + 3.9047919999575242, + 3.945125004975125 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.004282458001398481, + 0.004339374994742684, + 0.003903375007212162, + 0.004333709002821706, + 0.004292750003514811, + 0.0041082499956246465, + 0.0036660830082837492, + 0.004168416999164037, + 0.0041569580062059686, + 0.004161541990470141, + 0.0037240829988149926, + 0.004122667000046931, + 0.00440345800598152, + 0.004070458002388477, + 0.0037361249997047707, + 0.0040852090023690835, + 0.004206416007946245, + 0.003754040997591801, + 0.0040250000020023435, + 0.00410587499209214, + 0.004138834003242664, + 0.003831834008451551, + 0.004097041004570201, + 0.004037708000396378, + 0.004067500005476177, + 0.0040358330006711185, + 0.004056250007124618, + 0.004106207998120226, + 0.004020458000013605, + 0.004126083003939129, + 0.004163917008554563, + 0.00418270799855236, + 0.004182499993476085, + 0.004189874991425313, + 0.004240249996655621, + 0.004271208992577158, + 0.004226125005516224, + 0.004239792004227638, + 0.0042283750080969185, + 0.0038588750030612573, + 0.003831791997072287, + 0.004193624990875833, + 0.00436158299271483, + 0.0038685419858666137, + 0.003747791997739114, + 0.0039990829973248765, + 0.0037621660012518987, + 0.003945959004340693, + 0.004258500004652888, + 0.003898042006767355, + 0.004247500008204952, + 0.003875415990478359, + 0.004274708000593819, + 0.004285332994186319, + 0.00424270800431259, + 0.0038990409957477823, + 0.004202082986012101, + 0.004273124999599531, + 0.004293958001653664, + 0.0038699159922543913, + 0.004225166005198844, + 0.003928209000150673, + 0.0039047919999575242, + 0.0039013750065350905 + ], + [ + 0.003749124996829778, + 0.003946583005017601, + 0.003946541997720487, + 0.003761083004064858, + 0.0036890829942421988, + 0.003721458007930778, + 0.003696540996315889, + 0.0036242920032236725, + 0.003710999997565523, + 0.004068416994414292, + 0.003601791991968639, + 0.003715250000823289, + 0.0037791249924339354, + 0.0037576250033453107, + 0.0037125419912626967, + 0.003659999987576157, + 0.0038012919976608828, + 0.0037180419894866645, + 0.003633833010098897, + 0.003710832999786362, + 0.0036606670037144795, + 0.0037240420060697943, + 0.0036907920002704486, + 0.003668167002615519, + 0.00371558299229946, + 0.0037162499938858673, + 0.0037389159988379106, + 0.00367316699703224, + 0.0037048330123070627, + 0.0037143749941606075, + 0.0037426249909913167, + 0.0037063749914523214, + 0.003686875003040768, + 0.0038218340050661936, + 0.0037842500023543835, + 0.0038053750031394884, + 0.003793208990828134, + 0.0038228339981287718, + 0.003830832996754907, + 0.0038644579908577725, + 0.0037869589868932962, + 0.003849665998131968, + 0.004200333991320804, + 0.0038408749969676137, + 0.003838249991531484, + 0.0037897919974057004, + 0.004235707994666882, + 0.00396474999433849, + 0.0042028750031022355, + 0.003902249998645857, + 0.0038817919994471595, + 0.004149749991483986, + 0.004262374990503304, + 0.004386332991998643, + 0.003985874995123595, + 0.004014459002064541, + 0.003922959003830329, + 0.003879208001308143, + 0.004248625002219342, + 0.0038345000066328794, + 0.003906459009158425, + 0.003967249998822808, + 0.004239916001097299, + 0.0038992500049062073 + ], + [ + 0.003465332993073389, + 0.0036587499926099554, + 0.0036951249931007624, + 0.00367391700274311, + 0.0036712089931825176, + 0.0036303750093793496, + 0.0036846250004600734, + 0.003763540997169912, + 0.003684582989080809, + 0.003620083996793255, + 0.0034785839961841702, + 0.003698874992551282, + 0.003694625003845431, + 0.003900750001776032, + 0.0037313339998945594, + 0.0037357910041464493, + 0.0037543329963227734, + 0.003773709002416581, + 0.003775459001190029, + 0.003707959011080675, + 0.0037247500004014, + 0.0036005419970024377, + 0.004474708010093309, + 0.0037042920012027025, + 0.0037277090013958514, + 0.003695999999763444, + 0.0037150830030441284, + 0.0037180000072112307, + 0.0036681669880636036, + 0.0036910829949192703, + 0.003809709000051953, + 0.0037071669939905405, + 0.003640750001068227, + 0.0038134589995024726, + 0.0038685829931637272, + 0.0038206250028451905, + 0.0037754160002805293, + 0.0038078750076238066, + 0.003986166004324332, + 0.0042390829912619665, + 0.003906500001903623, + 0.0038100830133771524, + 0.0038811670092400163, + 0.004089082998689264, + 0.003792208997765556, + 0.0036789589939871803, + 0.0038917499914532527, + 0.003963333001593128, + 0.0038532500038854778, + 0.004350084011093713, + 0.003845833009108901, + 0.004029500007163733, + 0.00391633300750982, + 0.003968749995692633, + 0.003948375000618398, + 0.003959749999921769, + 0.004431749999639578, + 0.0038591249904129654, + 0.00393991700548213, + 0.004065874993102625, + 0.004094792006071657, + 0.003877541996189393, + 0.003805291999015026, + 0.004050417002872564 + ], + [ + 0.0039314580062637106, + 0.004081916005816311, + 0.0040945000073406845, + 0.004068374997586943, + 0.004109125002287328, + 0.0041194579971488565, + 0.004116374999284744, + 0.0040973330033011734, + 0.004077750010765158, + 0.0036723340017488226, + 0.004062041989527643, + 0.00402945899986662, + 0.0041040420037461445, + 0.004063457992742769, + 0.00403991699567996, + 0.004067417001351714, + 0.004033292003441602, + 0.003704540999024175, + 0.004139458003919572, + 0.004167624996625818, + 0.004203000004054047, + 0.004181083000730723, + 0.004139041993767023, + 0.004071666990057565, + 0.004157207993557677, + 0.00371970800915733, + 0.00416887499159202, + 0.0040585000097053126, + 0.0040130419947672635, + 0.003770667011849582, + 0.004135750001296401, + 0.004268832999514416, + 0.004109834000701085, + 0.004140584002016112, + 0.004243791991029866, + 0.004290374999982305, + 0.004197290996671654, + 0.0038012080040061846, + 0.0038300830055959523, + 0.004297499996027909, + 0.0042399580124765635, + 0.003787209003348835, + 0.0038382089987862855, + 0.004209250007988885, + 0.004236166991177015, + 0.003805875006946735, + 0.004244666997692548, + 0.004208458005450666, + 0.004269332988769747, + 0.004128208995098248, + 0.0038436670001829043, + 0.00414045799698215, + 0.0037879589945077896, + 0.003970417004893534, + 0.0037874160043429583, + 0.004193333996227011, + 0.004685583000537008, + 0.0037728750030510128, + 0.0042681659979280084, + 0.0042440839897608384, + 0.004267708995030262, + 0.0039130000077420846, + 0.0039303329976974055, + 0.004273041995475069 + ], + [ + 0.003507999994326383, + 0.0036014579964103177, + 0.003640041992184706, + 0.003671249985927716, + 0.0037071669939905405, + 0.003676957989227958, + 0.0036168749938951805, + 0.0036684999940916896, + 0.003675625004689209, + 0.00366625000606291, + 0.0036864169960608706, + 0.0036325419932836667, + 0.0037048329977551475, + 0.003692540994961746, + 0.003719000000273809, + 0.0037311250052880496, + 0.0036965000035706908, + 0.003783749998547137, + 0.0038232920051086694, + 0.003727000002982095, + 0.003712375008035451, + 0.0037574169982690364, + 0.004072291005286388, + 0.003840290999505669, + 0.0037134579906705767, + 0.003726666996954009, + 0.003895750007359311, + 0.003927917001419701, + 0.0037387499905889854, + 0.0037156670005060732, + 0.0037027500075055286, + 0.003979540997534059, + 0.003916708010365255, + 0.003834792005363852, + 0.0038155420043040067, + 0.004176540998741984, + 0.004032791999634355, + 0.0038646250031888485, + 0.0039398749941028655, + 0.004379082995001227, + 0.0039056669920682907, + 0.0038197079993551597, + 0.0038880829961271957, + 0.003958041008445434, + 0.0039766669942764565, + 0.0038000420026946813, + 0.0037469170056283474, + 0.0038412499998230487, + 0.004502416995819658, + 0.00430512499588076, + 0.0039554999966640025, + 0.003916374989785254, + 0.004091374998097308, + 0.004315416008466855, + 0.0039319169882219285, + 0.004037125007016584, + 0.004092832998139784, + 0.003949167003156617, + 0.0039245830121217296, + 0.0043332089990144596, + 0.003976334002800286, + 0.0042564589966787025, + 0.0038086249987827614, + 0.003945125004975125 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_mha": { + "first_token_ms": { + "count": 5, + "median": 4.099040990695357, + "p95": 4.573391197482124, + "min": 3.9829169982112944, + "max": 4.6104159991955385, + "iqr": 0.3411259822314605, + "mean": 4.240366397425532, + "cv": 0.06259305923834246 + }, + "decode_ms_per_step": { + "count": 315, + "median": 4.649374997825362, + "p95": 5.8094375883229095, + "min": 4.145041006268002, + "max": 9.924708007019944, + "iqr": 0.5957914909231476, + "mean": 4.739296069462961, + "cv": 0.13776498436947504 + }, + "total_ms": { + "count": 5, + "median": 295.2882939571282, + "p95": 332.26766053412575, + "min": 289.0127510181628, + "max": 341.1705849284772, + "iqr": 4.703461949247867, + "mean": 302.8160187735921, + "cv": 0.0714830082544664 + }, + "tokens_per_second": { + "count": 5, + "median": 216.73734214906577, + "p95": 220.99756172112038, + "min": 187.58944301548425, + "max": 221.44351685015434, + "iqr": 3.4756203119380302, + "mean": 212.14443282254712, + "cv": 0.06554534566226183 + }, + "median_step_ms_by_index": [ + 4.099040990695357, + 4.230167003697716, + 4.265916999429464, + 4.283000002033077, + 4.268999997293577, + 4.274374994565733, + 4.26625000545755, + 4.279083004803397, + 4.237542001646943, + 4.263875001925044, + 4.260707995854318, + 4.333459000918083, + 4.341209001722746, + 4.315291007515043, + 4.448167004738934, + 4.418541997438297, + 4.283082991605625, + 4.34849999146536, + 4.383000006782822, + 4.311791999498382, + 4.288416996132582, + 4.332166994572617, + 4.452665991266258, + 4.332166994572617, + 4.299209002056159, + 4.319291998399422, + 4.310207994421944, + 4.351167008280754, + 4.323375003878027, + 4.750124993734062, + 4.771040999912657, + 4.736292001325637, + 4.761499993037432, + 4.490958992391825, + 4.886875001830049, + 4.672458002460189, + 4.916624995530583, + 4.863875001319684, + 4.576833991450258, + 4.8070000048028305, + 4.6165419917088, + 4.892375000054017, + 4.47691700537689, + 4.89283399656415, + 4.9620420031715184, + 4.783374999533407, + 4.999708995455876, + 4.742582998005673, + 4.940708007779904, + 4.978833996574394, + 4.931542003760114, + 5.01879099465441, + 4.961000013281591, + 4.924708991893567, + 5.004958002245985, + 4.958042001817375, + 4.897416001767851, + 4.94495801103767, + 4.968416003976017, + 5.000834004022181, + 4.7961250093067065, + 4.985999999917112, + 4.900290994555689, + 4.9542499909875914 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.004425291990628466, + 0.004230167003697716, + 0.004265916999429464, + 0.0042142909951508045, + 0.004291916993679479, + 0.004443208992597647, + 0.004233999992720783, + 0.004253957988112234, + 0.00423654199403245, + 0.004184875011560507, + 0.0042061249987455085, + 0.004333459000918083, + 0.004289958000299521, + 0.004315291007515043, + 0.004675915988627821, + 0.004673916002502665, + 0.004749375002575107, + 0.004804083000635728, + 0.004779416995006613, + 0.004255916006513871, + 0.004343417007476091, + 0.004332166994572617, + 0.004347292007878423, + 0.004332166994572617, + 0.004299209002056159, + 0.004280916997231543, + 0.004310207994421944, + 0.004351167008280754, + 0.004287375006242655, + 0.0048281670024152845, + 0.004869374999543652, + 0.004888874987955205, + 0.004863249996560626, + 0.004433209003764205, + 0.004876042003161274, + 0.004672458002460189, + 0.004833041995880194, + 0.004863875001319684, + 0.004936458994052373, + 0.004912457996397279, + 0.004495375003898516, + 0.004892375000054017, + 0.004465749996597879, + 0.00489283399656415, + 0.0049620420031715184, + 0.0048324160015909, + 0.004999708995455876, + 0.004521499999100342, + 0.004940708007779904, + 0.005006874998798594, + 0.004945374996168539, + 0.00501879099465441, + 0.005027125007472932, + 0.00493170898698736, + 0.004822457995032892, + 0.004934083000989631, + 0.004873292011325248, + 0.0043664999975590035, + 0.005010332999518141, + 0.005179750005481765, + 0.004768833998241462, + 0.004985999999917112, + 0.00505841699487064, + 0.004998959004296921 + ], + [ + 0.004099040990695357, + 0.004273415994248353, + 0.004229333993862383, + 0.004303250010707416, + 0.0042638329905457795, + 0.004274374994565733, + 0.00426625000545755, + 0.004321625005104579, + 0.0042729160049930215, + 0.004236834007315338, + 0.004148041000007652, + 0.004286708004656248, + 0.0042177499999525025, + 0.004364458000054583, + 0.004320125008234754, + 0.004418541997438297, + 0.0042545000032987446, + 0.004301541994209401, + 0.004290707991458476, + 0.004344834000221454, + 0.00428420800017193, + 0.00430699999560602, + 0.004331874995841645, + 0.004339416991570033, + 0.004359917002147995, + 0.004319291998399422, + 0.004327374990680255, + 0.004338665996328928, + 0.004323375003878027, + 0.004353916010586545, + 0.00471283400838729, + 0.004342291998909786, + 0.004761499993037432, + 0.004490958992391825, + 0.004401916987262666, + 0.004566042000078596, + 0.004916624995530583, + 0.004799749993253499, + 0.004461415999685414, + 0.0048070000048028305, + 0.004446042003110051, + 0.0044683750020340085, + 0.00447691700537689, + 0.004808000012417324, + 0.004940499988151714, + 0.00448829100059811, + 0.005011708999518305, + 0.004502584008150734, + 0.004907875001663342, + 0.004978833996574394, + 0.004931542003760114, + 0.0048868329904507846, + 0.004961000013281591, + 0.004924708991893567, + 0.005004958002245985, + 0.005078500005765818, + 0.005056500012869947, + 0.0050185410073027015, + 0.004968416003976017, + 0.005000834004022181, + 0.00495766599487979, + 0.0050788340013241395, + 0.005007250001654029, + 0.005014333000872284 + ], + [ + 0.0046104159991955385, + 0.004235333006363362, + 0.004268374992534518, + 0.004283000002033077, + 0.004593166988343, + 0.00438291700265836, + 0.004339915991295129, + 0.004355750003014691, + 0.004552290993160568, + 0.004696999996667728, + 0.004773207998368889, + 0.004850667013670318, + 0.00466254199272953, + 0.004295833001378924, + 0.004448167004738934, + 0.004377083998406306, + 0.0042830829916056246, + 0.00434849999146536, + 0.004795333006768487, + 0.004700499994214624, + 0.004743375000543892, + 0.004809541002032347, + 0.004478249989915639, + 0.004829917001188733, + 0.004281040994101204, + 0.004372291994513944, + 0.004229792000842281, + 0.004406207997817546, + 0.004377749995910563, + 0.004750124993734062, + 0.005612916997051798, + 0.0050433750002412125, + 0.004819124995265156, + 0.005897084003663622, + 0.005756167010986246, + 0.005679249996319413, + 0.006824833006248809, + 0.0060269169916864485, + 0.005442958005005494, + 0.005773499986389652, + 0.005567875006818213, + 0.006452041998272762, + 0.006536041997605935, + 0.006505584009573795, + 0.008045125010539778, + 0.0070968749932944775, + 0.009924708007019944, + 0.008067333998042159, + 0.008171833003871143, + 0.006567959004314616, + 0.0066826249967562035, + 0.006325624999590218, + 0.006276583997532725, + 0.005893291992833838, + 0.005391041995608248, + 0.005445208007586189, + 0.005302457997458987, + 0.005195874997298233, + 0.005000541001209058, + 0.005122959002619609, + 0.004896792001090944, + 0.004840166991925798, + 0.004900290994555689, + 0.0049542499909875914 + ], + [ + 0.0040841660083970055, + 0.0042002080008387566, + 0.004289042000891641, + 0.004269166995072737, + 0.004268999997293577, + 0.004208542013657279, + 0.004317708007874899, + 0.004234332998748869, + 0.004213791005895473, + 0.004282750000129454, + 0.004260707995854318, + 0.004741291995742358, + 0.004650458999094553, + 0.004680124999140389, + 0.004661833998397924, + 0.00476441701175645, + 0.00422962500306312, + 0.004281249988707714, + 0.004273792001185939, + 0.004223958996590227, + 0.004277415995602496, + 0.004288874988560565, + 0.004649374997825362, + 0.004268959004548378, + 0.004264040995622054, + 0.004312834003940225, + 0.0042895420046988875, + 0.004770874991663732, + 0.004771916996105574, + 0.0047927080013323575, + 0.004771040999912657, + 0.004736292001325637, + 0.004303624999010935, + 0.004863124995608814, + 0.004886875001830049, + 0.004761292002513073, + 0.004939792008372024, + 0.00477820799278561, + 0.004576833991450258, + 0.004388667002785951, + 0.004806957993423566, + 0.004856125000515021, + 0.004693082999438047, + 0.005110041005536914, + 0.0051907500019297, + 0.004375208998681046, + 0.0046411669900408015, + 0.004742582998005673, + 0.005301124998368323, + 0.004940833008731715, + 0.0049114169960375875, + 0.0049443329917266965, + 0.004874707999988459, + 0.0049221669905819, + 0.005035166992456652, + 0.004958042001817375, + 0.004565416995319538, + 0.00494495801103767, + 0.004866582996328361, + 0.004898750004940666, + 0.0047961250093067065, + 0.0050426249945303425, + 0.0045369579893304035, + 0.004504708995227702 + ], + [ + 0.003982916998211294, + 0.004220791000989266, + 0.004219125010422431, + 0.004493458996876143, + 0.004145041006268002, + 0.0042545000032987446, + 0.004244042007485405, + 0.004279083004803397, + 0.004237542001646943, + 0.004263875001925044, + 0.004283209011191502, + 0.004302041998016648, + 0.004341209001722746, + 0.004277665997506119, + 0.004286208000849001, + 0.0042573329992592335, + 0.004293958001653664, + 0.004358249992947094, + 0.004383000006782822, + 0.004311791999498382, + 0.004288416996132582, + 0.004754166991915554, + 0.004452665991266258, + 0.004329792005592026, + 0.0044164169958094135, + 0.004724375001387671, + 0.0045578330027638, + 0.004258917004335672, + 0.0043108749960083514, + 0.004353833006462082, + 0.004364624997833744, + 0.00433375001011882, + 0.004363207990536466, + 0.0044787090009776875, + 0.0049654999893391505, + 0.00447149999672547, + 0.004914374992949888, + 0.004891290998784825, + 0.0045132910017855465, + 0.004442666991963051, + 0.0046165419917088, + 0.004928084003040567, + 0.0043329999898560345, + 0.004858499989495613, + 0.004800875001819804, + 0.004783374999533407, + 0.004352666001068428, + 0.004895542006124742, + 0.004875875005382113, + 0.004347915994003415, + 0.0048212920082733035, + 0.005170666001504287, + 0.004912042000796646, + 0.004847250005695969, + 0.004426291998242959, + 0.004442290999577381, + 0.004897416001767851, + 0.0049128750106319785, + 0.004958500008797273, + 0.004934166994644329, + 0.004438959003891796, + 0.004456542010302655, + 0.004442833989742212, + 0.004937999998219311 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "speedup_cached_vs_uncached_sdpa": 1.9822650819392165, + "speedup_cached_vs_uncached_mha": 2.3635771067842266 + }, + { + "batch": 1, + "prompt_len": 1024, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_mha": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 8.2872920029331, + "p95": 11.886416803463362, + "min": 6.783624994568527, + "max": 12.418875005096197, + "iqr": 2.058333993772976, + "mean": 8.988925200537778, + "cv": 0.24484253221737687 + }, + "decode_ms_per_step": { + "count": 315, + "median": 2.3341249907389283, + "p95": 2.9222045995993544, + "min": 1.8260419892612845, + "max": 4.217000008793548, + "iqr": 0.5107709948788397, + "mean": 2.3303097777811455, + "cv": 0.1473856433093008 + }, + "total_ms": { + "count": 5, + "median": 158.6982509761583, + "p95": 172.20617422135547, + "min": 135.37604092562106, + "max": 175.4554570070468, + "iqr": 8.955629062256776, + "mean": 155.79844120074995, + "cv": 0.0937998663482195 + }, + "tokens_per_second": { + "count": 5, + "median": 403.2810670964162, + "p95": 463.395183553031, + "min": 364.7649442868544, + "max": 472.75721436678134, + "iqr": 23.959844230109866, + "mean": 413.7475004232003, + "cv": 0.09575232345496175 + }, + "median_step_ms_by_index": [ + 8.2872920029331, + 3.6951669899281114, + 2.5325830065412447, + 2.949209010694176, + 2.4680420028744265, + 2.4005830055102706, + 2.411708002910018, + 2.5907079980242997, + 2.4478340055793524, + 2.3687909997534007, + 2.6007080014096573, + 2.355917007662356, + 2.35929100017529, + 2.5215000059688464, + 2.5002079928526655, + 2.2777080012019724, + 2.4415840016445145, + 2.2331660002237186, + 2.45612500293646, + 2.2568330023204908, + 2.2304580052150413, + 2.558207997935824, + 2.4125830095726997, + 2.3819590132916346, + 2.2998329950496554, + 2.369666995946318, + 2.4571669928263873, + 2.483332995325327, + 2.3684579937253147, + 2.0943329873261973, + 2.3959169920999557, + 2.187833990319632, + 2.350832990487106, + 2.271582998218946, + 2.166249993024394, + 2.329333990928717, + 2.3580410052090883, + 2.3316250008065253, + 2.435875008814037, + 2.3826660035410896, + 2.3005830007605255, + 2.3146659950725734, + 2.056375000393018, + 2.4482080043526366, + 2.111374997184612, + 2.331542011233978, + 2.1432080102385953, + 2.2567920095752925, + 2.2439169988501817, + 2.303750006831251, + 2.063624997390434, + 2.2598330106120557, + 2.064875006908551, + 2.379917001235299, + 2.035208002780564, + 1.9799589936155826, + 2.2869159874971956, + 2.066541012027301, + 2.141292003216222, + 2.02433300728444, + 1.9633339979918674, + 2.013292003539391, + 2.1724160033045337, + 2.351374991121702 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.006783624994568527, + 0.0030815420032013208, + 0.0024506660120096058, + 0.002097250006045215, + 0.002005874994210899, + 0.0019409170054132119, + 0.0019646659930003807, + 0.0020045830024173483, + 0.0020204999891575426, + 0.0020067080040462315, + 0.0028310000052442774, + 0.001992332996451296, + 0.0018828750035027042, + 0.0019220829999540001, + 0.0019267499883426353, + 0.0019066249951720238, + 0.0018793330091284588, + 0.0020010000007459894, + 0.002590666990727186, + 0.001963207992957905, + 0.0019181249954272062, + 0.00193349999608472, + 0.001938291999977082, + 0.002159791998565197, + 0.0019261660054326057, + 0.0019201249961042777, + 0.0026484169939067215, + 0.002032290998613462, + 0.0019939999911002815, + 0.001990333999856375, + 0.0018761250103125349, + 0.0019911249983124435, + 0.0019862089975504205, + 0.002086834007059224, + 0.0020358750043669716, + 0.0020225409971317276, + 0.0020179579878458753, + 0.0019852079858537763, + 0.002013916993746534, + 0.002007417002459988, + 0.0019927920075133443, + 0.0020324579963926226, + 0.0020444580004550517, + 0.00210266700014472, + 0.0018461250001564622, + 0.0020252079993952066, + 0.002005083006224595, + 0.0019304169982206076, + 0.0021965829946566373, + 0.0020508750021690503, + 0.002063624997390434, + 0.0018260419892612845, + 0.002028000002610497, + 0.0018945830088341609, + 0.002035208002780564, + 0.001885332996607758, + 0.0019918749894713983, + 0.001983249996555969, + 0.0019955419993493706, + 0.001965542003745213, + 0.0019633339979918674, + 0.0018999169988092035, + 0.0018950829980894923, + 0.001985583992791362 + ], + [ + 0.012418875005096197, + 0.0037402500020107254, + 0.002952249997179024, + 0.003019458003109321, + 0.002735332993324846, + 0.002659584002685733, + 0.0024965839984361082, + 0.0025907079980242997, + 0.002693000002182089, + 0.0028291250055190176, + 0.0026007080014096573, + 0.002545457988162525, + 0.002534915998694487, + 0.0025215000059688464, + 0.0026293749979231507, + 0.002721791999647394, + 0.0026255839911755174, + 0.0025387089990545064, + 0.00245612500293646, + 0.002601499989395961, + 0.0019750829960685223, + 0.003007583989528939, + 0.0026004169922089204, + 0.002668208006070927, + 0.002423917001578957, + 0.0024919999996200204, + 0.002583083012723364, + 0.002559458007453941, + 0.002944459003629163, + 0.0026053750043502077, + 0.0023959169920999557, + 0.002402750003966503, + 0.0025531249993946403, + 0.002747584003373049, + 0.002601832995424047, + 0.0025064999936148524, + 0.002645582993864082, + 0.002438625000650063, + 0.00251745899731759, + 0.002511165992473252, + 0.002541791007388383, + 0.0023306659859372303, + 0.0025610000011511147, + 0.0024784580018604174, + 0.002530042009311728, + 0.002551875004428439, + 0.0024659589980728924, + 0.0025965420063585043, + 0.0025771660002646968, + 0.0025512499996693805, + 0.0024042080040089786, + 0.0026310420071240515, + 0.0025692909985082224, + 0.0023354580043815076, + 0.0026198329869657755, + 0.0023754160065436736, + 0.002406250001513399, + 0.0025042079942068085, + 0.002624875007313676, + 0.00202433300728444, + 0.002620541999931447, + 0.0028765000024577603, + 0.0026252920069964603, + 0.0025624999980209395 + ], + [ + 0.009756583996932022, + 0.0036951669899281114, + 0.0023941670078784227, + 0.0025476669979980215, + 0.0024680420028744265, + 0.0024005830055102706, + 0.002411708002910018, + 0.002733583009103313, + 0.0024478340055793524, + 0.0021086670021759346, + 0.0023308750096475706, + 0.002355917007662356, + 0.00235929100017529, + 0.0025664999993750826, + 0.0025002079928526655, + 0.002112166999722831, + 0.002241124995634891, + 0.0020560420089168474, + 0.0024157089937943965, + 0.0022040829935576767, + 0.002567584000644274, + 0.002402750003966503, + 0.0024125830095726997, + 0.0023819590132916346, + 0.0022998329950496554, + 0.002369666995946318, + 0.0023341249907389283, + 0.0025064159999601543, + 0.002403375008725561, + 0.0020943329873261973, + 0.0025971249997382984, + 0.002367791996221058, + 0.0023788329999661073, + 0.002271582998218946, + 0.002166249993024394, + 0.002329333990928717, + 0.0023731670080451295, + 0.002071166003588587, + 0.002535749998060055, + 0.0023845410032663494, + 0.0025000419991556555, + 0.002360290993237868, + 0.002340541992452927, + 0.0024482080043526366, + 0.002111374997184612, + 0.002331542011233978, + 0.0021432080102385953, + 0.0022567920095752925, + 0.0023149170010583475, + 0.002386666994425468, + 0.0023977499949978665, + 0.0022598330106120557, + 0.002357499994104728, + 0.0024410830083070323, + 0.002311291013029404, + 0.002425625003525056, + 0.0023381250066449866, + 0.0023223330063046888, + 0.002345042012166232, + 0.0026142499991692603, + 0.0020083340059500188, + 0.0023164169979281723, + 0.0021724160033045337, + 0.002351374991121702 + ], + [ + 0.007698250003159046, + 0.004217000008793548, + 0.0036778329958906397, + 0.003257332995417528, + 0.00313204200938344, + 0.003040583003894426, + 0.003009917010786012, + 0.0029593750077765435, + 0.002871290998882614, + 0.0030098749994067475, + 0.002912666997872293, + 0.002809666999382898, + 0.0027883750008186325, + 0.0027032919897465035, + 0.0027171249967068434, + 0.0025999999925261363, + 0.0026552920026006177, + 0.002670000001671724, + 0.0024842499988153577, + 0.002636832999996841, + 0.002486792000127025, + 0.0026975840009981766, + 0.002501415990991518, + 0.002426999999443069, + 0.002542207992519252, + 0.0024050830106716603, + 0.0018868339975597337, + 0.002483332995325327, + 0.0019067500106757507, + 0.0020895839988952503, + 0.00258716699318029, + 0.0018865419988287613, + 0.0019557499908842146, + 0.0023443749960279092, + 0.0018808749882737175, + 0.0025649580056779087, + 0.0019487500103423372, + 0.002642959007062018, + 0.002435875008814037, + 0.0019752920052269474, + 0.0019414160051383078, + 0.001957374988705851, + 0.002035041994531639, + 0.002516541993827559, + 0.0019071669958066195, + 0.0024291250010719523, + 0.002356458004214801, + 0.001943458992172964, + 0.0019424169877311215, + 0.0019573750032577664, + 0.001955166007974185, + 0.002685624989680946, + 0.0020004590041935444, + 0.002379917001235299, + 0.001841916993726045, + 0.0018342080002184957, + 0.001834082999266684, + 0.00194358300359454, + 0.0019364999898243695, + 0.0019794580002781004, + 0.0019541660003596917, + 0.0019703330035554245, + 0.0019464999932097271, + 0.001949833007529378 + ], + [ + 0.0082872920029331, + 0.002795666005113162, + 0.0025325830065412447, + 0.002949209010694176, + 0.0022588749998249114, + 0.0022238329984247684, + 0.0022363339958246797, + 0.002088332999846898, + 0.0022071249986765906, + 0.0023687909997534007, + 0.0023203330056276172, + 0.002245875002699904, + 0.0022406670032069087, + 0.002222249997430481, + 0.0021781250106869265, + 0.0022777080012019724, + 0.0024415840016445145, + 0.0022331660002237186, + 0.002266666997456923, + 0.0022568330023204908, + 0.0022304580052150413, + 0.002558207997935824, + 0.0020247909997124225, + 0.002368166999076493, + 0.0022013330017216504, + 0.0019230829930165783, + 0.0024571669928263873, + 0.002317667007446289, + 0.0023684579937253147, + 0.0023659590078750625, + 0.0023118340031942353, + 0.002187833990319632, + 0.002350832990487106, + 0.0022467919916380197, + 0.0022979589994065464, + 0.0019139590003760532, + 0.0023580410052090883, + 0.0023316250008065253, + 0.0024143329937942326, + 0.0023826660035410896, + 0.0023005830007605255, + 0.0023146659950725734, + 0.002056375000393018, + 0.002334165998036042, + 0.0023007919953670353, + 0.0020261250028852373, + 0.001894333996460773, + 0.0022951249993639067, + 0.0022439169988501817, + 0.002303750006831251, + 0.0019380419980734587, + 0.0021998750016791746, + 0.002064875006908551, + 0.0025247919984394684, + 0.0020303329947637394, + 0.0019799589936155826, + 0.0022869159874971956, + 0.002066541012027301, + 0.002141292003216222, + 0.002147958002751693, + 0.0019542079971870407, + 0.002013292003539391, + 0.002197791007347405, + 0.002395290997810662 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + ] + } + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 8.380416998988949, + "p95": 9.735291794640943, + "min": 6.976875010877848, + "max": 9.973374995752238, + "iqr": 0.9280839876737446, + "mean": 8.393700199667364, + "cv": 0.13246184729726573 + }, + "decode_ms_per_step": { + "count": 315, + "median": 8.90804099617526, + "p95": 11.326566303614527, + "min": 6.627042006584816, + "max": 11.888707987964153, + "iqr": 2.354499498324003, + "mean": 9.018269793355897, + "cv": 0.15920645226554117 + }, + "total_ms": { + "count": 5, + "median": 564.5820820063818, + "p95": 686.862773145549, + "min": 453.2874500291655, + "max": 701.4269139326643, + "iqr": 93.78538005694281, + "mean": 576.5446971810888, + "cv": 0.16312858665763352 + }, + "tokens_per_second": { + "count": 5, + "median": 113.35818482329478, + "p95": 136.88585758775562, + "min": 91.24257813429709, + "max": 141.19076095286138, + "iqr": 17.85369601347206, + "mean": 113.45406323032925, + "cv": 0.1670506111320792 + }, + "median_step_ms_by_index": [ + 8.380416998988949, + 8.643332999781705, + 8.69925000006333, + 9.016416996018961, + 8.599541994044557, + 8.855916006723419, + 8.705083004315384, + 8.96354200085625, + 8.759042000747286, + 9.211583994328976, + 8.917459010262974, + 8.937082995544188, + 9.082584001589566, + 8.924416993977502, + 9.121625000261702, + 8.928124996600673, + 9.121166993281804, + 9.0922910021618, + 9.445292002055794, + 8.67362500866875, + 9.042833000421524, + 8.758042007684708, + 9.026125000673346, + 8.920832988223992, + 9.074207991943695, + 9.006874999613501, + 9.034709000843577, + 9.375749999890104, + 9.230875002685934, + 8.893916994566098, + 8.90804099617526, + 8.59324999328237, + 9.15420800447464, + 8.927583010517992, + 9.825083005125634, + 9.610749999410473, + 9.281834005378187, + 9.02874999155756, + 9.433707993593998, + 8.633165998617187, + 9.339791009551845, + 8.329832999152131, + 8.70054199185688, + 8.118541998555884, + 8.93924999400042, + 8.27754101192113, + 8.670124996569939, + 8.144124993123114, + 8.651292009744793, + 8.098874997813255, + 8.672541996929795, + 7.99120900046546, + 8.739915996557102, + 8.065125002758577, + 8.712041992112063, + 8.116666009300388, + 8.694957999978215, + 8.300708999740891, + 8.592375001171604, + 8.18108300154563, + 8.593584003392607, + 8.22729199717287, + 8.862749993568286, + 8.566207994590513 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.006976875010877848, + 0.007479000007151626, + 0.0072355419979430735, + 0.007075750007061288, + 0.006716749994666316, + 0.006627042006584816, + 0.007367832993622869, + 0.006793291002395563, + 0.007270041998708621, + 0.0068362909951247275, + 0.007005207997281104, + 0.00690575000771787, + 0.006923458000528626, + 0.007031500004814006, + 0.006999667006311938, + 0.006944749999092892, + 0.006979166995733976, + 0.0069859580107731745, + 0.007039292002446018, + 0.006675708005786873, + 0.007092250001733191, + 0.006807041994761676, + 0.007039415999315679, + 0.00700124999275431, + 0.006959291000384837, + 0.007053584005916491, + 0.007033874993794598, + 0.00704816599318292, + 0.007089584003551863, + 0.006833290986833163, + 0.007232541000121273, + 0.0069260419986676425, + 0.007403374998830259, + 0.007051207998301834, + 0.007321957993553951, + 0.007031957997241989, + 0.007074042005115189, + 0.007062457996653393, + 0.007097583002177998, + 0.007172165991505608, + 0.007204541994724423, + 0.007140250003430992, + 0.007160790992202237, + 0.007065124998916872, + 0.007157125000958331, + 0.007102499992470257, + 0.007395624998025596, + 0.007101916999090463, + 0.00725733301078435, + 0.007168915995862335, + 0.007063916011247784, + 0.00714575000165496, + 0.007122499999240972, + 0.0072002500091912225, + 0.007065333003993146, + 0.007207957998616621, + 0.007132707993150689, + 0.007112375009455718, + 0.007177000006777234, + 0.0070916660042712465, + 0.007453875005012378, + 0.007166708004660904, + 0.007317625000723638, + 0.0070757079956820235 + ], + [ + 0.009973374995752238, + 0.011021832993719727, + 0.01085054200666491, + 0.010399708000477403, + 0.01124695799080655, + 0.01036658299562987, + 0.010655707999831066, + 0.01109145800000988, + 0.010695416000089608, + 0.010742250000475906, + 0.011350832995958626, + 0.010924792004516348, + 0.011168582990649156, + 0.01105312499566935, + 0.010888707998674363, + 0.010869916994124651, + 0.011418082998716272, + 0.01038262499787379, + 0.011135458000353537, + 0.011259916005656123, + 0.010917708001215942, + 0.010998541009030305, + 0.011292208000668325, + 0.010639250001986511, + 0.011292291994323023, + 0.011230541989789344, + 0.011035542003810406, + 0.011094332992797717, + 0.01149220899969805, + 0.010673249998944812, + 0.01140004199987743, + 0.01096562499878928, + 0.010949040995910764, + 0.011144334013806656, + 0.011583542000153102, + 0.01072041699080728, + 0.011323041006107815, + 0.011546916997758672, + 0.010819375005667098, + 0.010986249995767139, + 0.011877375000040047, + 0.011161624992382713, + 0.011017708995495923, + 0.01123558399558533, + 0.010567292003543116, + 0.010846292003407143, + 0.01174416700087022, + 0.010460875011631288, + 0.010849415994016454, + 0.011241375003010035, + 0.011068375009926967, + 0.010215874994173646, + 0.010100458996021189, + 0.010402707994217053, + 0.011208416995941661, + 0.010984374996041879, + 0.01012300000002142, + 0.011531374999321997, + 0.010953749995678663, + 0.010281166003551334, + 0.011297958000795916, + 0.010538957998505794, + 0.010229749997961335, + 0.011888707987964153 + ], + [ + 0.008782958990195766, + 0.008816917004878633, + 0.008995249998406507, + 0.009428833000129089, + 0.00886545900721103, + 0.009134207997703925, + 0.008705083004315384, + 0.009385540994117036, + 0.008708042005309835, + 0.009211583994328976, + 0.008917459010262974, + 0.009151541002211161, + 0.00912412500474602, + 0.008928082999773324, + 0.00918391699087806, + 0.00909141699958127, + 0.009719041001517326, + 0.0090922910021618, + 0.009445292002055794, + 0.00867362500866875, + 0.009042833000421524, + 0.008758042007684708, + 0.009026125000673346, + 0.008920832988223992, + 0.009074207991943695, + 0.009310624998761341, + 0.009034709000843577, + 0.009375749999890104, + 0.009230875002685934, + 0.008893916994566098, + 0.00890804099617526, + 0.00859324999328237, + 0.008824500007904135, + 0.0087263330060523, + 0.009289499997976236, + 0.009141958013060503, + 0.00921704201027751, + 0.00902874999155756, + 0.009433707993593998, + 0.008633165998617187, + 0.009339791009551845, + 0.008329832999152131, + 0.00870054199185688, + 0.008096667006611824, + 0.00893924999400042, + 0.0077922079944983125, + 0.008670124996569939, + 0.007942666008602828, + 0.008651292009744793, + 0.007978416993864812, + 0.008672541996929795, + 0.00799120900046546, + 0.008739915996557102, + 0.008065125002758577, + 0.008712041992112063, + 0.008116666009300388, + 0.008694957999978215, + 0.008300708999740891, + 0.008592375001171604, + 0.00818108300154563, + 0.008593584003392607, + 0.00822729199717287, + 0.008862749993568286, + 0.008566207994590513 + ], + [ + 0.008380416998988949, + 0.008643332999781705, + 0.00869925000006333, + 0.008483957994030789, + 0.00857308300328441, + 0.008398542006034404, + 0.008712915994692594, + 0.00872075000370387, + 0.008759042000747286, + 0.008522125004674308, + 0.009204750007484108, + 0.008925332993385382, + 0.009082584001589566, + 0.008908415999030694, + 0.009121625000261702, + 0.008800665993476287, + 0.008901207998860627, + 0.00800708300084807, + 0.00900129099318292, + 0.008033000005525537, + 0.008883250004146248, + 0.00752762499905657, + 0.00823508299072273, + 0.0074018339946633205, + 0.00815941700420808, + 0.008295541003462858, + 0.008801875010249205, + 0.008239791000960395, + 0.00839849999465514, + 0.008205291000194848, + 0.008647707989439368, + 0.008078624989138916, + 0.00915420800447464, + 0.008927583010517992, + 0.01016883300326299, + 0.009610749999410473, + 0.009281834005378187, + 0.008429582987446338, + 0.00891158400918357, + 0.008382792002521455, + 0.00859233399387449, + 0.008241208997787908, + 0.00823491699702572, + 0.008118541998555884, + 0.007975249987794086, + 0.00827754101192113, + 0.007755999991786666, + 0.008144124993123114, + 0.007742624991806224, + 0.008098874997813255, + 0.007551540998974815, + 0.007593208996695466, + 0.007308582993573509, + 0.00755324999045115, + 0.007615916998474859, + 0.007584375009173527, + 0.007763041998259723, + 0.007805125002050772, + 0.007767124989186414, + 0.00798558299720753, + 0.007899709002231248, + 0.007910999993328005, + 0.007902208002633415, + 0.007777666003676131 + ], + [ + 0.007854875002522022, + 0.008283500006655231, + 0.008513833992765285, + 0.009016416996018961, + 0.008599541994044557, + 0.008855916006723419, + 0.008578791006584652, + 0.00896354200085625, + 0.008882541995262727, + 0.009359999996377155, + 0.008595791005063802, + 0.008937082995544188, + 0.008685540989972651, + 0.008924416993977502, + 0.008905333001166582, + 0.008928124996600673, + 0.009121166993281804, + 0.009243624997907318, + 0.009793249992071651, + 0.009116291999816895, + 0.009504583998932503, + 0.00879704199905973, + 0.009735958999954164, + 0.009087499987799674, + 0.009699417001684196, + 0.009006874999613501, + 0.009687917001429014, + 0.009690042003057897, + 0.009775833008461632, + 0.009484624999458902, + 0.0092228330031503, + 0.009553584008244798, + 0.009564417006913573, + 0.009791208998649381, + 0.009825083005125634, + 0.010214833004283719, + 0.009940916992491111, + 0.01003758399747312, + 0.009982916002627462, + 0.010663374996511266, + 0.010176584008149803, + 0.009293958006310277, + 0.010873624996747822, + 0.009760167013155296, + 0.010602957991068251, + 0.010349709002184682, + 0.01032058299460914, + 0.011525458001415245, + 0.010079582993057556, + 0.010852042003534734, + 0.011334791997796856, + 0.009992375009460375, + 0.011236667007324286, + 0.010615999999572523, + 0.010510416002944112, + 0.011540667008375749, + 0.010613541002385318, + 0.010849833008251153, + 0.011546457986696623, + 0.01109741599066183, + 0.011108791994047351, + 0.011465208008303307, + 0.01103724999120459, + 0.011396000001695938 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_mha": { + "first_token_ms": { + "count": 5, + "median": 8.870707999449223, + "p95": 10.385033409693278, + "min": 8.241208008257672, + "max": 10.686375011573546, + "iqr": 0.583333006943576, + "mean": 9.114858403336257, + "cv": 0.1035825958041225 + }, + "decode_ms_per_step": { + "count": 315, + "median": 10.190917004365474, + "p95": 15.167471102904528, + "min": 8.424416009802371, + "max": 17.244707996724173, + "iqr": 2.068729496386368, + "mean": 10.738363788169366, + "cv": 0.17300103611278728 + }, + "total_ms": { + "count": 5, + "median": 679.2926700873068, + "p95": 777.5729336339282, + "min": 606.9987120426958, + "max": 791.1068750254344, + "iqr": 96.11370800121222, + "mean": 685.6317770580063, + "cv": 0.10860125417700384 + }, + "tokens_per_second": { + "count": 5, + "median": 94.21564933384948, + "p95": 104.75358447253197, + "min": 80.89930958815441, + "max": 105.43679703145447, + "iqr": 13.554170967876956, + "mean": 94.20781069185304, + "cv": 0.10577748356887853 + }, + "median_step_ms_by_index": [ + 8.870707999449223, + 9.133375002420507, + 9.43212499259971, + 9.181957997498102, + 9.568583991494961, + 9.166332994936965, + 9.493957986705936, + 9.297958997194655, + 9.430125006474555, + 9.3592500052182, + 9.405749995494261, + 9.979916998418048, + 9.314834009273909, + 9.484833994065411, + 9.607707994291559, + 10.125874992809258, + 10.382542008301243, + 10.302166992914863, + 10.787541003082879, + 10.622625006362796, + 10.337790998164564, + 10.899792003328912, + 10.51108300453052, + 9.589041001163423, + 9.42412500444334, + 9.652541994000785, + 10.057332998258062, + 9.491500008152798, + 9.891291992971674, + 9.235333011019975, + 9.379584007547237, + 9.412333005457185, + 9.394624998094514, + 9.84266700106673, + 10.095958001329564, + 10.113083000760525, + 10.128708003321663, + 9.923082994646393, + 10.868500001379289, + 11.46550000703428, + 11.447209006291814, + 11.481541994726285, + 11.512416996993124, + 11.445583993918262, + 11.566917004529387, + 11.645208011032082, + 11.657249997369945, + 11.659709009109065, + 10.804208999616094, + 10.245540994219482, + 10.809917002916336, + 10.482707992196083, + 10.804917008499615, + 10.896750012761913, + 11.012040995410644, + 11.825249996036291, + 11.318834003759548, + 10.768749998533167, + 11.301582999294624, + 11.773999998695217, + 11.199750006198883, + 11.619958997471258, + 11.577708995901048, + 11.677625006996095 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.008596333995228633, + 0.008677625010022894, + 0.009032958987518214, + 0.008823458992992528, + 0.008993792012915947, + 0.00875662500038743, + 0.009000667007057928, + 0.008787874990957789, + 0.008879332992364652, + 0.008722209007828496, + 0.009001582991913892, + 0.010232417000224814, + 0.009314834009273909, + 0.009279582998715341, + 0.009607707994291559, + 0.010453292008605786, + 0.0108967090054648, + 0.012466167012462392, + 0.015154209002503194, + 0.014868833008222282, + 0.012790833992767148, + 0.011540459003299475, + 0.010784250000142492, + 0.009295124997152016, + 0.009412540995981544, + 0.009137291010119952, + 0.00937233300646767, + 0.009450124998693354, + 0.009401125003932975, + 0.008975040997029282, + 0.009379584007547237, + 0.009412333005457185, + 0.009394624998094514, + 0.009462250003707595, + 0.009746125011588447, + 0.00917404200299643, + 0.010020832996815443, + 0.009869542001979426, + 0.010868500001379289, + 0.012417917008860968, + 0.01352383301127702, + 0.015361083002062514, + 0.016653874990879558, + 0.01717441700748168, + 0.0167819999915082, + 0.016155707999132574, + 0.013174624997191131, + 0.012103959001251496, + 0.010804208999616094, + 0.010142458995687775, + 0.00957704101165291, + 0.010482707992196083, + 0.010128583002369851, + 0.010178708005696535, + 0.009834917000262067, + 0.009544165994157083, + 0.009459958004299551, + 0.010102666012244299, + 0.009534332988550887, + 0.009430207996047102, + 0.009668416998465545, + 0.009657875008997507, + 0.010446833999594674, + 0.009919000003719702 + ], + [ + 0.008870707999449223, + 0.009133375002420507, + 0.00943212499259971, + 0.009181957997498102, + 0.009750416997121647, + 0.009634707996156067, + 0.009766792005393654, + 0.009161167006823234, + 0.010216666996711865, + 0.009569292000378482, + 0.009840916987741366, + 0.009979916998418048, + 0.010227040998870507, + 0.010240333009278402, + 0.010130624999874271, + 0.010125874992809258, + 0.010382542008301243, + 0.010302166992914863, + 0.010787541003082879, + 0.011689874998410232, + 0.010337790998164564, + 0.010899792003328912, + 0.01087891700444743, + 0.010770375010906719, + 0.011292582988971844, + 0.01085254100325983, + 0.011132125000585802, + 0.011257791993557476, + 0.011564500004169531, + 0.0113043339952128, + 0.01224137500685174, + 0.01183425000635907, + 0.011677167000016198, + 0.012015084008453414, + 0.01256604200170841, + 0.012227125000208616, + 0.01218025000707712, + 0.012669917006860487, + 0.01326566599891521, + 0.012503457997809164, + 0.012799707998055965, + 0.012826499994844198, + 0.013156207991414703, + 0.013437333007459529, + 0.013875582997570746, + 0.01341320900246501, + 0.013307333996635862, + 0.014203750004526228, + 0.015198416003840975, + 0.014740458005690016, + 0.014522833007504232, + 0.014528583997162059, + 0.014748333996976726, + 0.014940041000954807, + 0.015443457989022136, + 0.0158731660048943, + 0.015663459009374492, + 0.015593375006574206, + 0.01625420899654273, + 0.016367083007935435, + 0.016722041997127235, + 0.017244707996724173, + 0.017129332991316915, + 0.01722262499970384 + ], + [ + 0.010686375011573546, + 0.011354250003932975, + 0.01097633301105816, + 0.010909625008935109, + 0.011474625003756955, + 0.010835667009814642, + 0.01078845800657291, + 0.010873874998651445, + 0.010764999999082647, + 0.010852833001990803, + 0.010852833001990803, + 0.01076995799667202, + 0.010935458005405962, + 0.010794875008286908, + 0.010600333000184037, + 0.011110792009276338, + 0.010654625002644025, + 0.010707749999710359, + 0.011254666009335779, + 0.010622625006362796, + 0.010957000005873851, + 0.010911374993156642, + 0.01051108300453052, + 0.011058624993893318, + 0.011093957989942282, + 0.010887124997680075, + 0.01121350000903476, + 0.011046792002161965, + 0.010787291001179256, + 0.011237207989324816, + 0.011071208005887456, + 0.010788875006255694, + 0.01138416699541267, + 0.011122958007035777, + 0.011168582990649156, + 0.01198287500301376, + 0.011415916989790276, + 0.011275374999968335, + 0.011721917006070726, + 0.01146550000703428, + 0.011447209006291814, + 0.011481541994726285, + 0.011512416996993124, + 0.011445583993918262, + 0.011566917004529387, + 0.011645208011032082, + 0.011657249997369945, + 0.011659709009109065, + 0.01166695800202433, + 0.011697708003339358, + 0.011763374990550801, + 0.011708958991221152, + 0.012485958999604918, + 0.011767374991904944, + 0.011829167007817887, + 0.011825249996036291, + 0.01191745899268426, + 0.011897042000782676, + 0.011714041989762336, + 0.012100458989152685, + 0.011899165998329408, + 0.011817582999356091, + 0.011652166998828761, + 0.012356375009403564 + ], + [ + 0.00917966700217221, + 0.009668291997513734, + 0.009871166010270827, + 0.009272167008020915, + 0.009568583991494961, + 0.009166332994936965, + 0.009493957986705936, + 0.009431500002392568, + 0.009430125006474555, + 0.0093592500052182, + 0.009363083008793183, + 0.009301416997914203, + 0.009132958002737723, + 0.009484833994065411, + 0.008431416004896164, + 0.009353959001600742, + 0.009058291994733736, + 0.009016750002047047, + 0.009196166007313877, + 0.008898290994693525, + 0.009379833994898945, + 0.009200291999150068, + 0.009503458000835963, + 0.008990375004941598, + 0.00942412500444334, + 0.008901124994736165, + 0.00938375000259839, + 0.009232791999238543, + 0.00903945800382644, + 0.009235333011019975, + 0.009066541999345645, + 0.009041749988682568, + 0.008997833996545523, + 0.009155749998171814, + 0.009300500008976087, + 0.009734040999319404, + 0.010128708003321663, + 0.009923082994646393, + 0.009790209005586803, + 0.009632624991354533, + 0.009181000001262873, + 0.009579207995557226, + 0.00985650000802707, + 0.009488791998592205, + 0.009627167004509829, + 0.009072208995348774, + 0.009557042008964345, + 0.010230999992927536, + 0.009423375013284385, + 0.009003625003970228, + 0.00958995899418369, + 0.009420166999916546, + 0.009162917005596682, + 0.009371874999487773, + 0.009915709000779316, + 0.013311084010638297, + 0.010564374999376014, + 0.009808749993680976, + 0.009566417007590644, + 0.009550749993650243, + 0.009796874990570359, + 0.00963441601197701, + 0.009993332991143689, + 0.009552375006023794 + ], + [ + 0.008241208008257672, + 0.008613458994659595, + 0.008424416009802371, + 0.008840833004796878, + 0.008559833993786015, + 0.00875741700292565, + 0.008813916996587068, + 0.009297958997194655, + 0.009385541998199187, + 0.009348833002150059, + 0.009405749995494261, + 0.00904991700372193, + 0.008902375004254282, + 0.00865883300139103, + 0.008841458009555936, + 0.009068833009223454, + 0.009347333005280234, + 0.00908279100258369, + 0.009383542012074031, + 0.009489459000178613, + 0.00940637500025332, + 0.009520709005300887, + 0.0090099169901805, + 0.009589041001163423, + 0.008996541000669822, + 0.009652541994000785, + 0.010057332998258062, + 0.009491500008152798, + 0.009891291992971674, + 0.009011999994982034, + 0.00933566600724589, + 0.009053499990841374, + 0.00929533300222829, + 0.00984266700106673, + 0.010095958001329564, + 0.010113083000760525, + 0.009405166987562552, + 0.009450375000596978, + 0.009735166997415945, + 0.009496084006968886, + 0.01071054200292565, + 0.01016508300381247, + 0.00973366599646397, + 0.009952624997822568, + 0.009665749996202067, + 0.010255208006128669, + 0.010190917004365474, + 0.010380500010796823, + 0.009999709000112489, + 0.010245540994219482, + 0.010809917002916336, + 0.010448041997733526, + 0.010804917008499615, + 0.010896750012761913, + 0.011012040995410644, + 0.010850082995602861, + 0.011318834003759548, + 0.010768749998533167, + 0.011301582999294624, + 0.011773999998695217, + 0.011199750006198883, + 0.011619958997471258, + 0.011577708995901048, + 0.011677625006996095 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "speedup_cached_vs_uncached_sdpa": 3.557582257735157, + "speedup_cached_vs_uncached_mha": 4.280404263493483 + }, + { + "batch": 8, + "prompt_len": 1, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_mha": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 1.8127090006601065, + "p95": 2.5262587994802743, + "min": 1.6969999996945262, + "max": 2.565458999015391, + "iqr": 0.5730420089093968, + "mean": 2.0482083986280486, + "cv": 0.19112195856800235 + }, + "decode_ms_per_step": { + "count": 315, + "median": 2.0187080081086606, + "p95": 4.492996392946225, + "min": 1.7917910008691251, + "max": 5.757916995207779, + "iqr": 0.2120829958585091, + "mean": 2.4226663872555254, + "cv": 0.35552402571176794 + }, + "total_ms": { + "count": 5, + "median": 129.18866795371287, + "p95": 231.05211033253, + "min": 126.52199805597775, + "max": 254.9141279014293, + "iqr": 8.451920046354644, + "mean": 154.67619079572614, + "cv": 0.3630156882810548 + }, + "tokens_per_second": { + "count": 5, + "median": 3963.1959064973485, + "p95": 4042.7162774484655, + "min": 2008.5195128846728, + "max": 4046.727113600224, + "iqr": 250.97421630585268, + "mean": 3564.1628364718513, + "cv": 0.2458312903988477 + }, + "median_step_ms_by_index": [ + 1.8127090006601065, + 2.120374992955476, + 1.9971670117229223, + 2.4363330012420192, + 2.13199999416247, + 1.997125000343658, + 2.0158330007689074, + 2.3000420042080805, + 1.982667003176175, + 1.9756660040002316, + 1.8820829864125699, + 2.0099999965168536, + 2.113874987117015, + 2.001290995394811, + 1.977665990125388, + 1.9821659952867776, + 2.0461250096559525, + 1.985499999136664, + 2.037833008216694, + 1.9700830016518012, + 1.9812919927062467, + 1.953917002538219, + 2.063833991996944, + 2.033665994531475, + 1.9699580006999895, + 2.040542007307522, + 1.9771660008700565, + 1.9757499976549298, + 1.9837919971905649, + 1.9875420111929998, + 2.0021660020574927, + 2.03479100309778, + 2.0114579965593293, + 1.9985419930890203, + 2.004374997341074, + 1.9948749977629632, + 1.959417000762187, + 2.1605829970212653, + 2.013250006712042, + 2.084416992147453, + 1.9846669893013313, + 1.9856250000884756, + 2.0107919990550727, + 1.9809579971479252, + 2.0532910129986703, + 2.006833004998043, + 1.9937920005759224, + 2.0004580001113936, + 1.9703750003827736, + 2.0970420009689406, + 2.054666998446919, + 2.055958000710234, + 2.011541000683792, + 2.024208995862864, + 2.0244159968569875, + 2.1032910008216277, + 2.0082499977434054, + 2.0520830003079027, + 2.1422919962788, + 2.126082996255718, + 2.07070900069084, + 1.995957994950004, + 2.154207992134616, + 2.1209999977145344 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0016969999996945262, + 0.0023472920001950115, + 0.002720333999604918, + 0.002541791007388383, + 0.00213199999416247, + 0.0024991249956656247, + 0.0020292499975766987, + 0.002389250003034249, + 0.0021068330097477883, + 0.002071041992167011, + 0.0018690410070121288, + 0.002061874998616986, + 0.002014541008975357, + 0.00201575001119636, + 0.001977665990125388, + 0.001990999997360632, + 0.0020461250096559525, + 0.0020944999996572733, + 0.002037833008216694, + 0.00275708400295116, + 0.002081667000311427, + 0.00212604200351052, + 0.0021672500006388873, + 0.0022061670024413615, + 0.0021344999986467883, + 0.002040542007307522, + 0.0021277079940773547, + 0.001971417004824616, + 0.0020145000016782433, + 0.002041749990894459, + 0.002069833004497923, + 0.00203479100309778, + 0.0020457500068005174, + 0.002063916006591171, + 0.002109082997776568, + 0.0021366250002756715, + 0.0021078330028103665, + 0.0021605829970212653, + 0.0021025419991929084, + 0.002084416992147453, + 0.00214304200198967, + 0.002121917001204565, + 0.002098832992487587, + 0.0018773749907268211, + 0.0020532910129986703, + 0.0020391250000102445, + 0.0020007090060971677, + 0.0021777920046588406, + 0.0020886249985778704, + 0.002100875004543923, + 0.002054666998446919, + 0.002077500001178123, + 0.0021356250072130933, + 0.0020414169994182885, + 0.0020497920049820095, + 0.0021628749964293092, + 0.0021916660043643788, + 0.0021827080054208636, + 0.002150250002159737, + 0.0021292499877745286, + 0.00207070900069084, + 0.0021545409981627017, + 0.0021542079921346158, + 0.0021209999977145344 + ], + [ + 0.002565458999015391, + 0.002883916997234337, + 0.002843916998244822, + 0.0026744999922811985, + 0.0024870420020306483, + 0.003997915991931222, + 0.0036367920110933483, + 0.0028894999995827675, + 0.002715499998885207, + 0.0027381249965401366, + 0.004262667003786191, + 0.003679166009533219, + 0.0034991669963346794, + 0.0028725409938488156, + 0.003158499996061437, + 0.003783791995374486, + 0.003923874988686293, + 0.003947125005652197, + 0.003065624987357296, + 0.0028219160012668, + 0.003924041986465454, + 0.0038432500005001202, + 0.0037197499914327636, + 0.003766207999433391, + 0.0038083339895820245, + 0.0035147500020684674, + 0.0036206670047249645, + 0.0034949590044561774, + 0.0041154999926220626, + 0.0036499579873634502, + 0.004179042007308453, + 0.004318624996813014, + 0.004463208009838127, + 0.00539658400521148, + 0.0037290840118657798, + 0.004084250002051704, + 0.0042137499985983595, + 0.003963958995882422, + 0.004079040998476557, + 0.004126667001401074, + 0.004138707998208702, + 0.004261124995537102, + 0.004197582995402627, + 0.004339207996963523, + 0.004281290996004827, + 0.004521334005403332, + 0.004501541989156976, + 0.004419959004735574, + 0.004456833004951477, + 0.004722375000710599, + 0.004489333994570188, + 0.0041419159970246255, + 0.00457112499861978, + 0.004596458995365538, + 0.004678124998463318, + 0.004516415996477008, + 0.005757916995207779, + 0.004521041002590209, + 0.00477149999642279, + 0.004827333003049716, + 0.004842333990382031, + 0.004878125007962808, + 0.004873832993325777, + 0.005150041994056664 + ], + [ + 0.0018127090006601065, + 0.0020190829964121804, + 0.0019489999976940453, + 0.0019436250004218891, + 0.002303167013451457, + 0.001997125000343658, + 0.0020158330007689074, + 0.0023000420042080805, + 0.001878082999610342, + 0.0018639579939190298, + 0.002074249990982935, + 0.0020099999965168536, + 0.0018412920035189018, + 0.002001290995394811, + 0.00195424999401439, + 0.00197108399879653, + 0.002392416005022824, + 0.0019200840033590794, + 0.001859208001405932, + 0.0018557500006863847, + 0.001960167006473057, + 0.0019353340030647814, + 0.0019486249948386103, + 0.0019768749916693196, + 0.0019699580006999895, + 0.0017917910008691251, + 0.0019740419957088307, + 0.001965290997759439, + 0.001983791997190565, + 0.0019875420111929998, + 0.0019259999971836805, + 0.0020162500004516914, + 0.001989292009966448, + 0.0017961249977815896, + 0.001889958992251195, + 0.001994874997762963, + 0.0019303329900139943, + 0.0023499160015489906, + 0.0020025840058224276, + 0.0021646670065820217, + 0.0019573750032577664, + 0.0019608330039773136, + 0.0020047909929417074, + 0.001980957997147925, + 0.0019480829942040145, + 0.001971750010852702, + 0.0019860830070683733, + 0.001963166010682471, + 0.0019703750003827736, + 0.002013082994380966, + 0.001999583008000627, + 0.0020065829885425046, + 0.002011541000683792, + 0.0020187080081086606, + 0.0020040420058649033, + 0.002002124994760379, + 0.0019405000057304278, + 0.001999415995669551, + 0.002003333007451147, + 0.0018943749892059714, + 0.0019983329984825104, + 0.001993833007873036, + 0.0019401249883230776, + 0.0020674580009654164 + ], + [ + 0.002369458001339808, + 0.002120374992955476, + 0.0019971670117229223, + 0.0024363330012420192, + 0.001979166001547128, + 0.0019484999938867986, + 0.0019462089985609055, + 0.0019342080049682409, + 0.0018227919936180115, + 0.0018568750092526898, + 0.0018820829864125699, + 0.0018097499996656552, + 0.0026746250077849254, + 0.0018256250041304156, + 0.0018983750051120296, + 0.0019821659952867776, + 0.0020076660002814606, + 0.0019551250006770715, + 0.0020996670064050704, + 0.001970083001651801, + 0.0019812919927062467, + 0.001953917002538219, + 0.002063833991996944, + 0.0019656250078696758, + 0.0019375829870114103, + 0.001927291988977231, + 0.0019771660008700565, + 0.0019757499976549298, + 0.0019745420140679926, + 0.001985124996281229, + 0.0019827499927487224, + 0.0020509169989963993, + 0.0020114579965593293, + 0.0019985419930890203, + 0.0019804579933406785, + 0.0019661670085042715, + 0.001959417000762187, + 0.0018305419944226742, + 0.0019792920065810904, + 0.001972333004232496, + 0.0019846669893013313, + 0.0019552089943317696, + 0.0020107919990550727, + 0.0021438749972730875, + 0.0020877499919151887, + 0.002006833004998043, + 0.001960167006473057, + 0.0020004580001113936, + 0.001918082998599857, + 0.0020958329987479374, + 0.0020900830131722614, + 0.002055958000710234, + 0.0020074999920325354, + 0.002024208995862864, + 0.0020244159968569875, + 0.0020627920021070167, + 0.0019715419912245125, + 0.002032584001426585, + 0.0021422919962788, + 0.002126082996255718, + 0.0020593329973053187, + 0.0019412080000620335, + 0.0023471669992432, + 0.0021495840046554804 + ], + [ + 0.0017964159924304113, + 0.0019929580012103543, + 0.0018539580050855875, + 0.0018645829986780882, + 0.0019415410060901195, + 0.0019410420063650236, + 0.0019409579981584102, + 0.0019899999897461385, + 0.001982667003176175, + 0.0019756660040002316, + 0.0018297500064363703, + 0.0019794999971054494, + 0.002113874987117015, + 0.001997125000343658, + 0.002004791997023858, + 0.0019187500001862645, + 0.002027584007009864, + 0.001985499999136664, + 0.0018838329997379333, + 0.0018152090051444247, + 0.0019627500005299225, + 0.0018084589974023402, + 0.0019781669980147853, + 0.002033665994531475, + 0.0019336669938638806, + 0.0022480829939013347, + 0.0019240420078858733, + 0.0020122499990975484, + 0.0019512909930199385, + 0.001864917008788325, + 0.0020021660020574927, + 0.00201424999977462, + 0.001901499999803491, + 0.00189462500566151, + 0.002004374997341074, + 0.001991041994187981, + 0.0018425000016577542, + 0.002008041992667131, + 0.002013250006712042, + 0.0020102089911233634, + 0.001980750006623566, + 0.0019856250000884756, + 0.0019844580092467368, + 0.001976375002413988, + 0.001982834000955336, + 0.0019340420112712309, + 0.0019937920005759224, + 0.001992707999306731, + 0.0019406670035095885, + 0.0020970420009689406, + 0.0020335420122137293, + 0.002004750000196509, + 0.002004291003686376, + 0.001973791004274972, + 0.0019877910090144724, + 0.0021032910008216277, + 0.0020082499977434054, + 0.0020520830003079027, + 0.0020347499958006665, + 0.002069082998787053, + 0.002072500006761402, + 0.001995957994950004, + 0.0020617500122170895, + 0.002016917002038099 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + ] + } + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 1.7669579974608496, + "p95": 3.5128920048009604, + "min": 1.55237500439398, + "max": 3.7606670084642246, + "iqr": 0.8870009914971888, + "mean": 2.2473165998235345, + "cv": 0.4135182664033946 + }, + "decode_ms_per_step": { + "count": 315, + "median": 2.701000004890375, + "p95": 3.9133374026278034, + "min": 1.5172499988693744, + "max": 5.5155000009108335, + "iqr": 1.0782704994198866, + "mean": 2.7206939427987744, + "cv": 0.26265923903829347 + }, + "total_ms": { + "count": 5, + "median": 169.91175200382713, + "p95": 195.94181723950896, + "min": 156.87462699133903, + "max": 199.58845904329792, + "iqr": 20.830163106438704, + "mean": 173.65103499614634, + "cv": 0.09973593745533842 + }, + "tokens_per_second": { + "count": 5, + "median": 3013.32894259408, + "p95": 3248.908746920335, + "min": 2565.278586017485, + "max": 3263.752780290386, + "iqr": 366.3444237889007, + "mean": 2971.016222398663, + "cv": 0.09545478105201588 + }, + "median_step_ms_by_index": [ + 1.7669579974608496, + 1.7229160002898425, + 1.7996670067077503, + 1.8636659951880574, + 1.8946669879369438, + 1.9415830029174685, + 2.3023340036161244, + 1.8045420001726598, + 1.567250001244247, + 1.6387499927077442, + 1.6927089891396463, + 1.7751670093275607, + 1.8136659928131849, + 1.8326669960515574, + 1.848540996434167, + 1.8014579982263967, + 2.000458989641629, + 1.9551659934222698, + 1.9142090022796765, + 1.9377499993424863, + 2.096041993354447, + 2.133250003680587, + 1.984542002901435, + 2.145916994777508, + 2.1757079957751557, + 2.12245799775701, + 2.14416699600406, + 2.207584009738639, + 2.4805409921100363, + 2.342624997254461, + 2.5256660010199994, + 2.403332997346297, + 2.6037500065285712, + 2.501082999515347, + 2.506291988538578, + 2.49483399966266, + 2.656124997884035, + 2.754707995336503, + 2.5341250002384186, + 2.561040993896313, + 2.6557920064078644, + 2.7130829985253513, + 2.922958999988623, + 2.9358339961618185, + 2.85987499228213, + 2.817833999870345, + 2.905999994254671, + 2.9036249907221645, + 3.3374579943483695, + 3.1291249906644225, + 3.0057080002734438, + 2.989041997352615, + 3.1132080039242283, + 3.2515409984625876, + 3.2854579912964255, + 3.2425829995190725, + 3.193958997144364, + 3.208624999388121, + 3.3432089985581115, + 3.3704590023262426, + 3.3947080082725734, + 3.3797500072978437, + 3.338875001645647, + 3.459458996076137 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0025217919901479036, + 0.0021968749933876097, + 0.002252166988910176, + 0.002313292003236711, + 0.002242834001663141, + 0.002382666993071325, + 0.0023550409969175234, + 0.0022604160039918497, + 0.0023142500140238553, + 0.001755249992129393, + 0.0023101249971659854, + 0.002427041996270418, + 0.0018136659928131849, + 0.0022480420011561364, + 0.002228083001682535, + 0.002230583006166853, + 0.0024392920022364706, + 0.0019964170060120523, + 0.002384332998190075, + 0.0019880830077454448, + 0.00210808300471399, + 0.0024348749866476282, + 0.001984542002901435, + 0.0027476669929455966, + 0.002227624994702637, + 0.0026259590085828677, + 0.0021265420073177665, + 0.002207584009738639, + 0.0026824579981621355, + 0.002324334011063911, + 0.00267929099209141, + 0.0022851660032756627, + 0.0029186669999035075, + 0.002501082999515347, + 0.002506291988538578, + 0.0025160420045722276, + 0.002656124997884035, + 0.002993999994941987, + 0.0026153750077355653, + 0.0025974169984692708, + 0.002649250003742054, + 0.0027130829985253513, + 0.0028165830008219928, + 0.003358875008416362, + 0.00285987499228213, + 0.002819500004989095, + 0.0028089170082239434, + 0.0029036249907221645, + 0.003387292003026232, + 0.0031291249906644225, + 0.003388791999896057, + 0.003053166001336649, + 0.0031188330031000078, + 0.0031540409981971607, + 0.00350399999297224, + 0.003325707992189564, + 0.003114125007414259, + 0.0031339169945567846, + 0.0031555420137010515, + 0.00320008400012739, + 0.003745875001186505, + 0.0034586249967105687, + 0.0033132500102510676, + 0.0034002920001512393 + ], + [ + 0.0037606670084642246, + 0.003912375002983026, + 0.004014042002381757, + 0.003751999989617616, + 0.0039775409968569875, + 0.004336124999099411, + 0.00427391700213775, + 0.0038038329948903993, + 0.003255499992519617, + 0.0032980419928207994, + 0.0034040409955196083, + 0.003598874987801537, + 0.0035417920007603243, + 0.003308084007585421, + 0.0033201250043930486, + 0.0031277499947464094, + 0.003161167012876831, + 0.002701000004890375, + 0.002697792006074451, + 0.0030602909973822534, + 0.0030193330021575093, + 0.0030930829962017015, + 0.0029270000086398795, + 0.0026522499974817038, + 0.0031026670039864257, + 0.0028123329975642264, + 0.0026043749967357144, + 0.0029441250080708414, + 0.0024805409921100363, + 0.0024112500104820356, + 0.002414333008346148, + 0.002771291008684784, + 0.0026037500065285712, + 0.002620457991724834, + 0.0025392920069862157, + 0.0024073340027825907, + 0.0029034170001978055, + 0.0024697920016478747, + 0.0025093330041272566, + 0.0025443330087000504, + 0.0030510840006172657, + 0.00268845800019335, + 0.0031964999943738803, + 0.002735542002483271, + 0.0027898749976884574, + 0.002713000008952804, + 0.0032780839974293485, + 0.00283145799767226, + 0.0033374579943483695, + 0.0032601249986328185, + 0.0029577499954029918, + 0.002969416993437335, + 0.0030565839988412336, + 0.003479833001620136, + 0.0030885419982951134, + 0.0031625830015400425, + 0.0031484999926760793, + 0.003481541993096471, + 0.0038486670091515407, + 0.0031290409970097244, + 0.0031555830064462498, + 0.0033271669963141903, + 0.003491250012302771, + 0.0032751670078141615 + ], + [ + 0.0016347909986507148, + 0.0017229160002898425, + 0.0017425000114599243, + 0.0017591659998288378, + 0.0017351669957861304, + 0.0018911249935626984, + 0.0019480409973766655, + 0.0017709999956423417, + 0.0015382919955300167, + 0.0015467920020455495, + 0.0015813750069355592, + 0.001570874999742955, + 0.001575250003952533, + 0.0016277499962598085, + 0.001600458999746479, + 0.0016557500057388097, + 0.0017069169989554211, + 0.0017291249969275668, + 0.0017268750088987872, + 0.0017978749965550378, + 0.0018849170010071248, + 0.0018520419980632141, + 0.0018738340004347265, + 0.0019726250029634684, + 0.0021617919992422685, + 0.00212245799775701, + 0.002139166012057103, + 0.004191957996226847, + 0.00391558300179895, + 0.003932124993298203, + 0.004068041991558857, + 0.003190500006894581, + 0.0029900829977123067, + 0.0028222500113770366, + 0.0029587089957203716, + 0.002848625008482486, + 0.003228041998227127, + 0.0037927920056972653, + 0.0036861670087091625, + 0.003459250001469627, + 0.0036515840038191527, + 0.0039918330003274605, + 0.004695374998846091, + 0.004678124998463318, + 0.004242625000188127, + 0.004077290999703109, + 0.004145083003095351, + 0.0036391670000739396, + 0.003721874993061647, + 0.0036520830035442486, + 0.003608708007959649, + 0.0035958749940618873, + 0.0036295830068411306, + 0.0035008749982807785, + 0.0032259169965982437, + 0.0032425829995190725, + 0.003393458988284692, + 0.0033955410035559908, + 0.003209332993719727, + 0.003389167002751492, + 0.0033725420071277767, + 0.00354529099422507, + 0.003338875001645647, + 0.003459458996076137 + ], + [ + 0.0017669579974608496, + 0.001720999993267469, + 0.0017682500038063154, + 0.0018636659951880574, + 0.001847790990723297, + 0.0018114160047844052, + 0.0023023340036161244, + 0.0018045420001726598, + 0.0015172499988693744, + 0.0015778749948367476, + 0.0016927089891396463, + 0.0017203329916810617, + 0.0017619169957470149, + 0.0018099580047419295, + 0.0018428339972160757, + 0.0018014579982263967, + 0.002000458989641629, + 0.0018385000003036112, + 0.0018994580022990704, + 0.0017988340114243329, + 0.0018247920088469982, + 0.0020104169961996377, + 0.0019497920002322644, + 0.002108000000589527, + 0.0021757079957751557, + 0.0020347499958006665, + 0.0023979580000741407, + 0.0021851670026080683, + 0.0022310839995043352, + 0.0022647089936072007, + 0.0025256660010199994, + 0.002403332997346297, + 0.0024771250027697533, + 0.002409082997473888, + 0.0024893329973565415, + 0.00249483399966266, + 0.0025679160025902092, + 0.002754707995336503, + 0.0025341250002384186, + 0.0025243749987566844, + 0.0026554999931249768, + 0.0026996669912477955, + 0.002852250006981194, + 0.0029117909871274605, + 0.0028818749997299165, + 0.0027318339998601004, + 0.0028217090002726763, + 0.0028415419947123155, + 0.0031012499966891482, + 0.0031146249966695905, + 0.0030057080002734438, + 0.002989041997352615, + 0.003027374987141229, + 0.0030744579999009147, + 0.0032854579912964255, + 0.0032959170057438314, + 0.0032488750002812594, + 0.0031979170016711578, + 0.0033432089985581115, + 0.0033704590023262426, + 0.003500167003949173, + 0.0033797500072978437, + 0.005168792005861178, + 0.0055155000009108335 + ], + [ + 0.00155237500439398, + 0.001612000007298775, + 0.0017996670067077503, + 0.001811958005419001, + 0.0018946669879369438, + 0.0019415830029174685, + 0.0018706669943640009, + 0.0018002919969148934, + 0.001567250001244247, + 0.0016387499927077442, + 0.001650541991693899, + 0.0017751670093275607, + 0.0018343749979976565, + 0.0018326669960515574, + 0.001848540996434167, + 0.0017729999963194132, + 0.001937875000294298, + 0.00195516599342227, + 0.0019142090022796765, + 0.0019377499993424863, + 0.002096041993354447, + 0.002133250003680587, + 0.0021871250064577907, + 0.002145916994777508, + 0.0020389579876791686, + 0.0020635839900933206, + 0.00214416699600406, + 0.002197334004449658, + 0.0023627080081496388, + 0.002342624997254461, + 0.002360500002396293, + 0.0021560419991146773, + 0.0024443750007776543, + 0.0024782500113360584, + 0.0024828329915180802, + 0.002483500007656403, + 0.002489082995452918, + 0.0026455000042915344, + 0.002487041987478733, + 0.002561040993896313, + 0.0026557920064078644, + 0.0027898330008611083, + 0.002922958999988623, + 0.0029358339961618185, + 0.0028374999965308234, + 0.002817833999870345, + 0.002905999994254671, + 0.002934375006589107, + 0.003103875002125278, + 0.003091292004683055, + 0.0029866250115446746, + 0.0029809160041622818, + 0.0031132080039242283, + 0.0032515409984625876, + 0.003355416003614664, + 0.0032205419993260875, + 0.003193958997144364, + 0.003208624999388121, + 0.0034412499953759834, + 0.003468542010523379, + 0.0033947080082725734, + 0.003269166001700796, + 0.0032207079930230975, + 0.003525749998516403 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_mha": { + "first_token_ms": { + "count": 5, + "median": 2.4428329925285652, + "p95": 3.7148664006963372, + "min": 1.9048749963985756, + "max": 3.963208000641316, + "iqr": 0.8162089943652973, + "mean": 2.5875413994072005, + "cv": 0.3269500939611309 + }, + "decode_ms_per_step": { + "count": 315, + "median": 2.8676249930867925, + "p95": 4.054487698886078, + "min": 1.7762500065146014, + "max": 8.053417011979036, + "iqr": 1.2317294967942871, + "mean": 2.959307251102661, + "cv": 0.3208003129000166 + }, + "total_ms": { + "count": 5, + "median": 184.006834984757, + "p95": 209.85568743490148, + "min": 176.35958202299662, + "max": 214.04145403357688, + "iqr": 15.513622027356178, + "mean": 189.02389821887482, + "cv": 0.08190116901732643 + }, + "tokens_per_second": { + "count": 5, + "median": 2782.505334882879, + "p95": 2899.107444587979, + "min": 2392.0599974978777, + "max": 2903.159522873201, + "iqr": 231.59650170639816, + "mean": 2722.3853232883484, + "cv": 0.07710017726175633 + }, + "median_step_ms_by_index": [ + 2.4428329925285652, + 2.037500002188608, + 2.0631670049624518, + 1.9935829914174974, + 2.131375003955327, + 2.256292005768046, + 2.206582998041995, + 2.0193749951431528, + 1.9261669949628413, + 1.8656250031199306, + 1.8757910002022982, + 1.9847910007229075, + 2.032375006820075, + 2.055000004475005, + 2.1060830040369183, + 2.061292005237192, + 2.184708006097935, + 2.1671669965144247, + 2.1764589910162613, + 2.201082999818027, + 2.2444999922299758, + 2.2965829994063824, + 2.488874990376644, + 2.421249999315478, + 2.478207999956794, + 2.4835829972289503, + 2.531333011575043, + 2.6192500081378967, + 2.536458006943576, + 2.873042001738213, + 2.6611249923007563, + 2.5541250070091337, + 2.8440000023692846, + 2.8268750029383227, + 2.8676249930867925, + 2.8754160011885688, + 2.960208003059961, + 2.945207990705967, + 2.947708999272436, + 2.9575000080512837, + 3.130875003989786, + 3.1394169927807525, + 3.253125003539026, + 3.1691250042058527, + 3.1431660027010366, + 3.2826250098878518, + 3.3620000031078234, + 3.4008750080829486, + 3.4597080084495246, + 3.461707994574681, + 3.5295830020913854, + 3.5382079950068146, + 3.647916004410945, + 3.596333001041785, + 3.7562079960480332, + 3.692750004120171, + 3.7766250025015324, + 3.771708987187594, + 3.783666994422674, + 3.8047910056775436, + 3.9314580062637106, + 3.860875003738329, + 3.8267080090008676, + 3.882083998178132 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0027215000009164214, + 0.002037500002188608, + 0.002447791994200088, + 0.0018452919903211296, + 0.00240808300441131, + 0.0024181250046240166, + 0.0025779159914236516, + 0.0020193749951431528, + 0.0026845420070458204, + 0.0024105000047711655, + 0.0018757910002022982, + 0.0018941669986816123, + 0.001957707994733937, + 0.0019816249987343326, + 0.0021060830040369183, + 0.0020672920072684065, + 0.0021806250006193295, + 0.0021671669965144247, + 0.0021945420012343675, + 0.0020890000014333054, + 0.002248541990411468, + 0.002343792002648115, + 0.002819874993292615, + 0.0023021669912850484, + 0.00286641699494794, + 0.0024835829972289503, + 0.002500125003280118, + 0.002564875001553446, + 0.002536458006943576, + 0.0030287090048659593, + 0.0026851250004256144, + 0.0025413330004084855, + 0.002778374997433275, + 0.00276845799817238, + 0.0028676249930867925, + 0.00294191601278726, + 0.002970583998830989, + 0.002945207990705967, + 0.0029322089976631105, + 0.0029399999912129715, + 0.003520792000927031, + 0.0031394169927807525, + 0.003253417002269998, + 0.0031691250042058527, + 0.0031431660027010366, + 0.003496459001326002, + 0.0033690409909468144, + 0.0034008750080829486, + 0.003352915999130346, + 0.003290500011644326, + 0.003916625006240793, + 0.0035280829906696454, + 0.0037157089973334223, + 0.003392375001567416, + 0.0035317080037202686, + 0.004075459000887349, + 0.0038153329951455817, + 0.003771708987187594, + 0.003597875009290874, + 0.003912291998858564, + 0.0039314580062637106, + 0.0038730000087525696, + 0.003753541997866705, + 0.0039059580012690276 + ], + [ + 0.0019052910065511242, + 0.0020282500045141205, + 0.0020141249988228083, + 0.002038082995568402, + 0.0020025419944431633, + 0.0020059590024175122, + 0.002244332994450815, + 0.002198207992478274, + 0.0019261669949628413, + 0.00196154200239107, + 0.001982583009521477, + 0.0019847910007229075, + 0.002032375006820075, + 0.0020103749993722886, + 0.0021689160057576373, + 0.002159333002055064, + 0.002184708006097935, + 0.0021787500008940697, + 0.002171041996916756, + 0.0022050419938750565, + 0.0022087920078774914, + 0.0022965829994063824, + 0.002488874990376644, + 0.002460542003973387, + 0.002452874992741272, + 0.0024438330001430586, + 0.0025348330091219395, + 0.002521999995224178, + 0.0025213750050170347, + 0.0028967919934075326, + 0.0026611249923007563, + 0.0025541250070091337, + 0.002887833004933782, + 0.002762165997410193, + 0.0028013340051984414, + 0.003317125010653399, + 0.0029784170037601143, + 0.0027748330030590296, + 0.003046750003704801, + 0.0029730829992331564, + 0.003002333003678359, + 0.0031397919956361875, + 0.003275249997386709, + 0.003114749997621402, + 0.002959833000204526, + 0.0031750420021126047, + 0.0033620000031078234, + 0.003509749993099831, + 0.003506541994283907, + 0.003427292002015747, + 0.004674999989219941, + 0.005221667001023889, + 0.005095207990962081, + 0.006997083008172922, + 0.005742958004702814, + 0.005826708002132364, + 0.006794083004933782, + 0.005317332994309254, + 0.005416333005996421, + 0.00697191599465441, + 0.006680416001472622, + 0.006295125000178814, + 0.008053417011979036, + 0.007495917001506314 + ], + [ + 0.0024428329925285652, + 0.0021423750004032627, + 0.0019729999912669882, + 0.0019935829914174974, + 0.0019584589899750426, + 0.0022562920057680458, + 0.002192707994254306, + 0.0021242079965304583, + 0.002358999990974553, + 0.001810292000300251, + 0.001859541007434018, + 0.0018296250054845586, + 0.0019150000007357448, + 0.0020655840053223073, + 0.002086207998218015, + 0.0019671250047395006, + 0.0020616249967133626, + 0.0021666660031769425, + 0.0021764589910162613, + 0.0021562500041909516, + 0.0021758329967269674, + 0.0022132080048322678, + 0.0022535419993801042, + 0.0023301670007640496, + 0.002478207999956794, + 0.0025232920015696436, + 0.002527708013076335, + 0.0026192500081378967, + 0.002475707995472476, + 0.0024973340041469783, + 0.002503749987226911, + 0.0026898330106632784, + 0.0028477500018198043, + 0.0028268750029383227, + 0.002792083003441803, + 0.0027217500028200448, + 0.0028562090010382235, + 0.0030321249942062423, + 0.0032525420101592317, + 0.0029464999970514327, + 0.002966125000966713, + 0.0030180000030668452, + 0.003253125003539026, + 0.0032302920008078218, + 0.00326120899990201, + 0.003147249997709878, + 0.0032062499958556145, + 0.0033225410006707534, + 0.0034597080084495246, + 0.003461707994574681, + 0.003457375001744367, + 0.003398332992219366, + 0.003647916004410945, + 0.0036776670021936297, + 0.0037562079960480332, + 0.0034492499980842695, + 0.0034300000115763396, + 0.0036675420124083757, + 0.003783666994422674, + 0.0037823330058017746, + 0.00365445799252484, + 0.003804167005000636, + 0.00383595799212344, + 0.003827416992862709 + ], + [ + 0.003963208000641316, + 0.0042182080069324, + 0.0029712499963352457, + 0.0020402920054038987, + 0.002344499996979721, + 0.0022686669981339946, + 0.002163833996746689, + 0.0019345830078236759, + 0.0018710000003920868, + 0.0017762500065146014, + 0.0018555829883553088, + 0.002532124999561347, + 0.002040584004134871, + 0.002178624999942258, + 0.0022299580014077947, + 0.0019857080042129382, + 0.0025094169977819547, + 0.0022461250046035275, + 0.0026419159985380247, + 0.002910582988988608, + 0.0037179579958319664, + 0.0029601659916806966, + 0.003515834003337659, + 0.0030163750052452087, + 0.0026308329979656264, + 0.002770374994724989, + 0.002879708001273684, + 0.0029188749904278666, + 0.002959833000204526, + 0.002873042001738213, + 0.002797958004521206, + 0.0027179159951629117, + 0.0028440000023692846, + 0.002958334007416852, + 0.0029660420113941655, + 0.0028754160011885688, + 0.002960208003059961, + 0.002998334006406367, + 0.002947708999272436, + 0.003046375000849366, + 0.003130875003989786, + 0.0031501660123467445, + 0.0032264579931506887, + 0.003187666996382177, + 0.0030618750024586916, + 0.0034686250000959262, + 0.00356550001015421, + 0.003765707995626144, + 0.0034661660029087216, + 0.003565167004126124, + 0.003273749985964969, + 0.0035382079950068146, + 0.00360174999514129, + 0.003539625002304092, + 0.003567583000403829, + 0.003464583001914434, + 0.0037766250025015324, + 0.0037827500054845586, + 0.003800541002419777, + 0.003621875002863817, + 0.00404549999802839, + 0.003860875003738329, + 0.0038267080090008676, + 0.003786333996686153 + ], + [ + 0.0019048749963985756, + 0.0018284999969182536, + 0.0020631670049624518, + 0.0019185409910278395, + 0.002131375003955327, + 0.0021154999994905666, + 0.002206582998041995, + 0.001960374996997416, + 0.0018841249984689057, + 0.0018656250031199306, + 0.0019406250066822395, + 0.001995250000618398, + 0.002035624987911433, + 0.002055000004475005, + 0.002096374999382533, + 0.002061292005237192, + 0.0022099579946370795, + 0.0021304999972926453, + 0.0021214169973973185, + 0.002201082999818027, + 0.0022444999922299758, + 0.0022657499939668924, + 0.002413041002000682, + 0.002421249999315478, + 0.002326250003534369, + 0.0023432500020135194, + 0.002531333011575043, + 0.002654416995937936, + 0.0025973750016419217, + 0.0026226250047329813, + 0.0025062920030904934, + 0.0024618750030640513, + 0.0027550840022740886, + 0.0028531249990919605, + 0.002891124997404404, + 0.0028599579964065924, + 0.002818666005623527, + 0.0028781250002793968, + 0.002943707993836142, + 0.0029575000080512837, + 0.003132457990432158, + 0.0029748329980066046, + 0.0030751670128665864, + 0.0030986250058049336, + 0.0032706250058254227, + 0.0032826250098878518, + 0.003348417012603022, + 0.003147041003103368, + 0.003227082997909747, + 0.00360345900116954, + 0.0035295830020913854, + 0.003668416989967227, + 0.003446665999945253, + 0.003596333001041785, + 0.0037636670022038743, + 0.003692750004120171, + 0.0035817089956253767, + 0.0035084999981336296, + 0.0036561250017257407, + 0.0038047910056775436, + 0.0037972499994793907, + 0.003592374996514991, + 0.0036079589917790145, + 0.003882083998178132 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "speedup_cached_vs_uncached_sdpa": 1.3152217968893756, + "speedup_cached_vs_uncached_mha": 1.4243264358967227 + }, + { + "batch": 8, + "prompt_len": 16, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_mha": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 2.3229590005939826, + "p95": 2.661366597749293, + "min": 1.7895830096676946, + "max": 2.7439999976195395, + "iqr": 0.06883298920001835, + "mean": 2.289875003043562, + "cv": 0.14810527335383702 + }, + "decode_ms_per_step": { + "count": 315, + "median": 2.1135829883860424, + "p95": 3.6786793032661085, + "min": 1.8091660022037104, + "max": 5.6661250127945095, + "iqr": 0.3480415034573525, + "mean": 2.3349117743597914, + "cv": 0.2547240130267979 + }, + "total_ms": { + "count": 5, + "median": 135.73858090967406, + "p95": 190.81736364169046, + "min": 133.231712054112, + "max": 203.12866305175703, + "iqr": 8.296704079839401, + "mean": 149.3893167877104, + "cv": 0.20237601401725244 + }, + "tokens_per_second": { + "count": 5, + "median": 3771.956333775918, + "p95": 3842.676624195611, + "min": 2520.569929953917, + "max": 3842.928925149978, + "iqr": 225.13731802137454, + "mean": 3518.7305423229445, + "cv": 0.16072672566322224 + }, + "median_step_ms_by_index": [ + 2.3229590005939826, + 2.54566699732095, + 2.0394580060383305, + 2.5805409968597814, + 2.1145000064279884, + 2.4980829912237823, + 2.0257080032024533, + 2.4015829985728487, + 1.982040994334966, + 2.060958999209106, + 2.1097499993629754, + 2.4954999971669167, + 2.040916995611042, + 2.052624986390583, + 2.364665997447446, + 2.0539160032058135, + 2.3773330030962825, + 2.010208001593128, + 2.14304200198967, + 2.104540995787829, + 2.073166993795894, + 2.2955419990466908, + 2.0477910002227873, + 2.0257500000298023, + 1.9992919987998903, + 2.755291003268212, + 2.17312500171829, + 2.081749989883974, + 2.077958997688256, + 2.5637910002842546, + 2.0760000043082982, + 2.0272920082788914, + 2.08454200765118, + 2.2510830021928996, + 2.0753339922521263, + 2.0828749984502792, + 2.029499999480322, + 2.13425001129508, + 2.0307500089984387, + 2.144124999176711, + 2.139999996870756, + 2.0660419977502897, + 2.067749999696389, + 2.0181669970043004, + 2.0124589937040582, + 2.041499988990836, + 2.1163750061532483, + 2.12170799204614, + 2.1215410088188946, + 2.1440419950522482, + 2.1929159993305802, + 2.1044999884907156, + 2.163291006581858, + 2.1451659995364025, + 2.1167920058360323, + 2.139834003173746, + 2.118999997037463, + 2.1302920067682862, + 2.117541996994987, + 2.119207987561822, + 2.1015420061303303, + 2.116584000759758, + 2.0825420069741085, + 2.0661250018747523 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0023308329982683063, + 0.00254566699732095, + 0.001861500000813976, + 0.0025805409968597814, + 0.0019578339997678995, + 0.0024980829912237823, + 0.001989458003663458, + 0.0024015829985728487, + 0.0019707499886862934, + 0.0024045419995673, + 0.0019358749996172264, + 0.0025852920080069453, + 0.0019512920116540045, + 0.002370208007050678, + 0.002364665997447446, + 0.0018091660022037104, + 0.0023773330030962825, + 0.0019829160009976476, + 0.00214304200198967, + 0.00221741599671077, + 0.0019314160017529503, + 0.0022955419990466908, + 0.0022380000009434298, + 0.0020257500000298023, + 0.001953833008883521, + 0.001873707995400764, + 0.002313457996933721, + 0.0021135829883860424, + 0.002005125003051944, + 0.0020240829908289015, + 0.002036958991084248, + 0.0018780000100377947, + 0.002234832994872704, + 0.0020616249967133626, + 0.001967624993994832, + 0.0021551249956246465, + 0.002029499999480322, + 0.0020505829888861626, + 0.001921125003718771, + 0.0021587499941233546, + 0.0019133750029141083, + 0.0019907499954570085, + 0.002187083999160677, + 0.0019760409923037514, + 0.0019818339933408424, + 0.002041499988990836, + 0.0020833339949604124, + 0.002220374997705221, + 0.0021019580017309636, + 0.0020475000055739656, + 0.002225583986728452, + 0.002100917001371272, + 0.002220665992354043, + 0.0021452499931911007, + 0.0019879169994965196, + 0.002140209006029181, + 0.002152125001884997, + 0.0021302920067682862, + 0.0021022089931648225, + 0.0021615409932564944, + 0.0021015420061303303, + 0.002112540998496115, + 0.002001291999476962, + 0.0020661250018747523 + ], + [ + 0.0023229590005939826, + 0.002133832997060381, + 0.001904959004605189, + 0.001967958000022918, + 0.002494915999704972, + 0.0024107499921228737, + 0.0020257080032024533, + 0.0025274170038755983, + 0.0019709159969352186, + 0.0019644579879241064, + 0.002045166998868808, + 0.0020270829991204664, + 0.002040916995611042, + 0.0020337500027380884, + 0.0020412499870872125, + 0.002520124995498918, + 0.0019688329921336845, + 0.0018925420008599758, + 0.0019863749912474304, + 0.0019967089901911095, + 0.001984208996873349, + 0.001985875001992099, + 0.001999750005779788, + 0.0019112089939881116, + 0.0019992919987998903, + 0.002892999997129664, + 0.002067791996523738, + 0.002081749989883974, + 0.002077958997688256, + 0.002125874991179444, + 0.0021105409978190437, + 0.0019168340077158064, + 0.0019642499973997474, + 0.001977124993572943, + 0.0020186249894322827, + 0.001974333994439803, + 0.002003750007133931, + 0.0020127909956499934, + 0.0020089580066269264, + 0.0020977079984731972, + 0.002057125006103888, + 0.002037708996795118, + 0.002067749999696389, + 0.0020181669970043004, + 0.002234667001175694, + 0.002024791989242658, + 0.0021163750061532483, + 0.002120666002156213, + 0.0020464999979594722, + 0.0021440419950522482, + 0.0020884580007987097, + 0.0020977499953005463, + 0.0021473340020747855, + 0.0021451659995364025, + 0.0020556250092340633, + 0.002056917001027614, + 0.002032708012848161, + 0.0020996670064050704, + 0.0020518329984042794, + 0.0020010000007459894, + 0.0020213749958202243, + 0.001998874999117106, + 0.0020825420069741085, + 0.002040167004452087 + ], + [ + 0.0017895830096676946, + 0.0021597920131171122, + 0.0020394580060383305, + 0.0023838749912101775, + 0.0019179999944753945, + 0.0024549590016249567, + 0.001868542007287033, + 0.001887042002636008, + 0.001982040994334966, + 0.0020090839971089736, + 0.0021097499993629754, + 0.0019249160104664043, + 0.001839042000938207, + 0.0018554589914856479, + 0.001979624998057261, + 0.001977082996745594, + 0.0019795420084847137, + 0.002019458988797851, + 0.0020461249951040372, + 0.0019999169890070334, + 0.0025063339999178424, + 0.0020700000022770837, + 0.0020477910002227873, + 0.0020608749910024926, + 0.0019817499996861443, + 0.0019365000043762848, + 0.00217312500171829, + 0.002067583001917228, + 0.0019212500046705827, + 0.0025637910002842546, + 0.0020760000043082982, + 0.0020272920082788914, + 0.00208454200765118, + 0.0022510830021928996, + 0.0020753339922521263, + 0.0020828749984502792, + 0.0020139590051257983, + 0.00213425001129508, + 0.002099374993122183, + 0.002023666995228268, + 0.0022436250001192093, + 0.0020660419977502897, + 0.002011541000683792, + 0.002005334012210369, + 0.0019960420031566173, + 0.0022272500063991174, + 0.0021270830038702115, + 0.0020979999972041696, + 0.002144500002032146, + 0.002250250006909482, + 0.0021929159993305802, + 0.00217387500742916, + 0.002163291006581858, + 0.002008416995522566, + 0.0021526670025195926, + 0.0020727920054923743, + 0.0020670410012826324, + 0.0020980830013286322, + 0.002117541996994987, + 0.0020914579945383593, + 0.0020985000010114163, + 0.002160916992579587, + 0.0020803340012207627, + 0.0021635419980157167 + ], + [ + 0.002262000009068288, + 0.002650041991728358, + 0.002120167002431117, + 0.0027282910014037043, + 0.0021145000064279884, + 0.0025674170028651133, + 0.00254470799700357, + 0.0019205830030841753, + 0.002169459010474384, + 0.002060958999209106, + 0.0025118329940596595, + 0.0024954999971669167, + 0.002132708003045991, + 0.0020526249863905832, + 0.0025793750101001933, + 0.0020539160032058135, + 0.002508874997147359, + 0.002010208001593128, + 0.0025862080074148253, + 0.002104540995787829, + 0.002073166993795894, + 0.0025434580020373687, + 0.0019347080087754875, + 0.0019509590056259185, + 0.0023607909970451146, + 0.002755291003268212, + 0.0019210840109735727, + 0.00205275000189431, + 0.0020948750025127083, + 0.0025787089980440214, + 0.0020705420029116794, + 0.002090000009047799, + 0.002078874997096136, + 0.00238379200163763, + 0.0021213750005699694, + 0.00201929199101869, + 0.0020512089977273718, + 0.0021361660037655383, + 0.0020307500089984387, + 0.002144124999176711, + 0.002139999996870756, + 0.002673249997314997, + 0.0020337909954832867, + 0.00206058299227152, + 0.0020124589937040582, + 0.0020097499946132302, + 0.0020634999964386225, + 0.00212170799204614, + 0.0021215410088188946, + 0.0027529579965630546, + 0.002170624997233972, + 0.0021044999884907156, + 0.0021547079959418625, + 0.0021272089943522587, + 0.0021167920058360323, + 0.002139834003173746, + 0.002160290998290293, + 0.00222733301052358, + 0.0022098339977674186, + 0.0021922079904470593, + 0.0021455420064739883, + 0.002116584000759758, + 0.0020857909985352308, + 0.002065541993943043 + ], + [ + 0.0027439999976195395, + 0.003242124992539175, + 0.0032037080090958625, + 0.0029084999987389892, + 0.002961582998977974, + 0.003116375009994954, + 0.0032442500087199733, + 0.0029182919970480725, + 0.0032326660002581775, + 0.002835666004102677, + 0.0036722920049214736, + 0.003293665999080986, + 0.0036994580004829913, + 0.0038516670028911904, + 0.005274916999042034, + 0.004403333005029708, + 0.004446959006600082, + 0.004752334003569558, + 0.0056661250127945095, + 0.004590917000314221, + 0.00456904100428801, + 0.004432250003446825, + 0.00472945898945909, + 0.0041969580051954836, + 0.0036935829994035885, + 0.003997291001724079, + 0.003924707998521626, + 0.003767792004509829, + 0.003473332995781675, + 0.003518291996442713, + 0.0034001250023720786, + 0.00335520799853839, + 0.0034330000053159893, + 0.0031899169989628717, + 0.003090124999289401, + 0.003105208001215942, + 0.0030348330037668347, + 0.003089000005275011, + 0.002824041002895683, + 0.0025934169971151277, + 0.002912459007347934, + 0.0028127499972470105, + 0.00280808299430646, + 0.002875625010346994, + 0.002262624999275431, + 0.0020566250022966415, + 0.0023097500088624656, + 0.002184750002925284, + 0.0026005830004578456, + 0.00212508300319314, + 0.002626000001328066, + 0.00288529199315235, + 0.002208500009146519, + 0.0021857919928152114, + 0.0021844589937245473, + 0.0021685829997295514, + 0.002118999997037463, + 0.0027946659974986687, + 0.002327041991520673, + 0.002119207987561822, + 0.0021153329871594906, + 0.0021717909985454753, + 0.0020878750074189156, + 0.002710375003516674 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + ] + } + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 2.2936249879421666, + "p95": 2.431633201194927, + "min": 1.8082080059684813, + "max": 2.4538750003557652, + "iqr": 0.5342910008039325, + "mean": 2.141349800513126, + "cv": 0.14455058467182716 + }, + "decode_ms_per_step": { + "count": 315, + "median": 3.0770830053370446, + "p95": 4.22079550626222, + "min": 1.8581659969640896, + "max": 4.518458008533344, + "iqr": 1.0647289818734862, + "mean": 3.106620476261476, + "cv": 0.2194420223605527 + }, + "total_ms": { + "count": 5, + "median": 196.81399794353638, + "p95": 203.33385883714072, + "min": 192.4549960240256, + "max": 204.34675103751943, + "iqr": 2.8881260514026508, + "mean": 197.85843980498612, + "cv": 0.022111075167881654 + }, + "tokens_per_second": { + "count": 5, + "median": 2601.4409815855006, + "p95": 2649.6901961389285, + "min": 2505.5450962662644, + "max": 2660.3622175445275, + "iqr": 37.78233735721551, + "mean": 2588.7140358144284, + "cv": 0.021960115744739155 + }, + "median_step_ms_by_index": [ + 2.2936249879421666, + 2.1002080029575154, + 2.0182910084258765, + 1.977541993255727, + 1.9629159942269325, + 2.014832993154414, + 2.0831250003539026, + 2.2486250090878457, + 2.2627500002272427, + 2.2819580044597387, + 2.2807920031482354, + 2.3519999958807603, + 2.3666670022066683, + 2.3427910055033863, + 2.4218339967774227, + 2.359833990340121, + 2.3498340015066788, + 2.5655000063125044, + 2.5937089958461, + 2.600457999506034, + 2.603540997370146, + 2.6990420010406524, + 2.7556669956538826, + 2.701582998270169, + 2.8192919999128208, + 2.758917005849071, + 2.901540996390395, + 2.99220799934119, + 2.976791001856327, + 3.01370800298173, + 3.1369589996756986, + 3.0327499989653006, + 2.997417002916336, + 3.094708008575253, + 3.1027080112835392, + 3.1753330113133416, + 3.0770830053370446, + 3.231499998946674, + 3.293957997811958, + 3.3762500097509474, + 3.367249999428168, + 3.3537089911988005, + 3.3750830043572932, + 3.421332992729731, + 3.5030830040341243, + 3.469791990937665, + 3.521666003507562, + 3.495083990856074, + 3.5160410043317825, + 3.9752080047037452, + 3.8347919908119366, + 3.7785409949719906, + 3.8412080029956996, + 3.9813330076867715, + 3.9231669943546876, + 3.962625007261522, + 3.9351250015897676, + 4.119958000956103, + 4.051333002280444, + 4.05095900350716, + 4.161459000897594, + 4.256750005879439, + 4.1802499908953905, + 4.230332997394726 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0023426660045515746, + 0.0022833340044599026, + 0.002259750006487593, + 0.002482584008248523, + 0.002370207992498763, + 0.0019966249965364113, + 0.002077707991702482, + 0.002448707993607968, + 0.0025064999936148524, + 0.002155207999749109, + 0.002661040998646058, + 0.0025668330054031685, + 0.0024107499921228737, + 0.0023046669957693666, + 0.0022887920058565214, + 0.002662582992343232, + 0.002275791994179599, + 0.0029742079932475463, + 0.002883875000406988, + 0.0025630840100347996, + 0.0024727919953875244, + 0.0030340410012286156, + 0.0027556669956538826, + 0.0027502500015543774, + 0.0029959160019643605, + 0.0027178750024177134, + 0.0027301669906591997, + 0.0031945829978212714, + 0.0029537079972214997, + 0.0032072909962153062, + 0.003177333011990413, + 0.002844292001100257, + 0.0028384589968482032, + 0.0030588749941671267, + 0.0030681669886689633, + 0.0033743329986464232, + 0.002982332996907644, + 0.003084999989368953, + 0.0031422920001205057, + 0.003328958002384752, + 0.003138166997814551, + 0.003185083987773396, + 0.0031856249988777563, + 0.0031754170049680397, + 0.0034599159989738837, + 0.0033504579914733768, + 0.0034751250059343874, + 0.003277208990766667, + 0.003327000013086945, + 0.003975208004703745, + 0.0037585410027531907, + 0.003621291005401872, + 0.003971666999859735, + 0.004317333005019464, + 0.0038111250032670796, + 0.0037023750046500936, + 0.003742667002370581, + 0.003964666000683792, + 0.004023166999104433, + 0.00405095900350716, + 0.0038545419956790283, + 0.003966999996919185, + 0.004153415997279808, + 0.004098791992873885 + ], + [ + 0.001808375003747642, + 0.0018885419995058328, + 0.0018726670095929876, + 0.001901499999803491, + 0.0018581659969640896, + 0.002014832993154414, + 0.0020810419955523685, + 0.002049500006251037, + 0.002052709009149112, + 0.002187833990319632, + 0.0021504580072360113, + 0.0021177500020712614, + 0.0023666670022066683, + 0.0023427910055033863, + 0.0023395000025629997, + 0.0022764169989386573, + 0.0023006249975878745, + 0.002532541999244131, + 0.002471957996021956, + 0.002456332993460819, + 0.0025641250103944913, + 0.002593583005364053, + 0.00265929099987261, + 0.0026183329900959507, + 0.002568458003224805, + 0.002676040996448137, + 0.0028527079994091764, + 0.002891208991059102, + 0.0029554169886978343, + 0.0029793749999953434, + 0.0028840830054832622, + 0.0030327499989653006, + 0.002997417002916336, + 0.003094708008575253, + 0.0030504589958582073, + 0.003113165992544964, + 0.0030738749919692054, + 0.0032193750084843487, + 0.0032847090042196214, + 0.003300834010588005, + 0.0033191659895237535, + 0.0033537089911988005, + 0.003369041995028965, + 0.003421332992729731, + 0.0035030830040341243, + 0.00339033298951108, + 0.003574250004021451, + 0.0035048749996349216, + 0.0035160410043317825, + 0.0037985000089975074, + 0.00377520800975617, + 0.0037785409949719906, + 0.0038412080029956996, + 0.0038467499980470166, + 0.003876165996189229, + 0.003962625007261522, + 0.00403783300134819, + 0.0044424579973565415, + 0.003947791992686689, + 0.004034708006656729, + 0.004193375003524125, + 0.004216708010062575, + 0.004128500004298985, + 0.004142667006817646 + ], + [ + 0.0018082080059684813, + 0.0019370829977560788, + 0.0020182910084258765, + 0.001977541993255727, + 0.0019456669979263097, + 0.0020122499990975484, + 0.0020831250003539026, + 0.0021503340103663504, + 0.0022326670004986227, + 0.0022819580044597387, + 0.002158832998247817, + 0.0021677079930668697, + 0.002486957993824035, + 0.0022907499951543286, + 0.0024475419922964647, + 0.0023512920015491545, + 0.002550083998357877, + 0.0025815409899223596, + 0.00256920799438376, + 0.002600457999506034, + 0.002603540997370146, + 0.0026990420010406524, + 0.0027742499951273203, + 0.002765833996818401, + 0.002734041991061531, + 0.0028154169995104894, + 0.002901540996390395, + 0.00299220799934119, + 0.003004041005624458, + 0.00301370800298173, + 0.0029179999983171, + 0.0029357500025071204, + 0.002981708006700501, + 0.0031394159886986017, + 0.0031949999975040555, + 0.0031424589978996664, + 0.0030770830053370446, + 0.003231499998946674, + 0.003293957997811958, + 0.0036190829996485263, + 0.0033939999993890524, + 0.003345290999277495, + 0.0033750830043572932, + 0.0034795420069713145, + 0.003643333009677008, + 0.003469791990937665, + 0.0034570839925436303, + 0.003481917010503821, + 0.003565291000995785, + 0.003987375006545335, + 0.0038347919908119366, + 0.0037791250069858506, + 0.003726666996954009, + 0.0039813330076867715, + 0.00404066700139083, + 0.0040027080103755, + 0.003935125001589768, + 0.004119958000956103, + 0.004160374999628402, + 0.004101709011592902, + 0.004161459000897594, + 0.004290457989554852, + 0.004285999995772727, + 0.004285999995772727 + ], + [ + 0.0022936249879421666, + 0.002111708003212698, + 0.0025427910004509613, + 0.0020467499998630956, + 0.0020130420016357675, + 0.00220320800144691, + 0.0026083749980898574, + 0.0022486250090878457, + 0.0022627500002272427, + 0.002296708000358194, + 0.0026899590011453256, + 0.0028215000056661665, + 0.0023581669956911355, + 0.0024949170037871227, + 0.0027662499924190342, + 0.002359833990340121, + 0.0027171249967068434, + 0.002544709001085721, + 0.0026590420020511374, + 0.0026770419935928658, + 0.0027481669967528433, + 0.0026355830050306395, + 0.003042209005798213, + 0.002701582998270169, + 0.0028192919999128208, + 0.002935416006948799, + 0.0029752919945167378, + 0.002953000002889894, + 0.003323458004160784, + 0.0029645419999724254, + 0.003375250002136454, + 0.0036584170011337847, + 0.0030142079922370613, + 0.0030989579972811043, + 0.00314062500547152, + 0.003253041999414563, + 0.003269917011493817, + 0.003692750004120171, + 0.003300208001746796, + 0.0033762500097509474, + 0.003380375012056902, + 0.0035410839918768033, + 0.0033800410019466653, + 0.0034663749975152314, + 0.003586499995435588, + 0.0035633329971460626, + 0.003521666003507562, + 0.003495083990856074, + 0.00351670800591819, + 0.0043189169955439866, + 0.004328708004322834, + 0.003755541009013541, + 0.0037891659885644913, + 0.003995459002908319, + 0.004466750004212372, + 0.003952707993448712, + 0.0038330419920384884, + 0.004518458008533344, + 0.0041227090114261955, + 0.003987707998021506, + 0.004061458006617613, + 0.004256750005879439, + 0.004283583999495022, + 0.004230332997394726 + ], + [ + 0.0024538750003557652, + 0.0021002080029575154, + 0.0019579579966375604, + 0.0019383750041015446, + 0.0019629159942269325, + 0.002084374995320104, + 0.002198041998781264, + 0.002285875001689419, + 0.002290792006533593, + 0.0022948750120121986, + 0.0022807920031482354, + 0.0023519999958807603, + 0.0022919170005479828, + 0.002522791997762397, + 0.0024218339967774227, + 0.002375750002101995, + 0.0023498340015066788, + 0.0025655000063125044, + 0.0025937089958461, + 0.0027357920043868944, + 0.002636208009789698, + 0.0027039999986300245, + 0.0027282499941065907, + 0.0025990419962909073, + 0.0029080000094836578, + 0.002758917005849071, + 0.0031500000040978193, + 0.0030051670037209988, + 0.002976791001856327, + 0.0030239579937187955, + 0.0031369589996756986, + 0.003105541007244028, + 0.0030747910059290007, + 0.0030534590041497722, + 0.0031027080112835392, + 0.0031753330113133416, + 0.0032219169952441007, + 0.0035035000037169084, + 0.0033055829990189523, + 0.003412249992834404, + 0.003367249999428168, + 0.0036277079925639555, + 0.0035152909986209124, + 0.003388750003068708, + 0.0034214170009363443, + 0.003510665992507711, + 0.0036570410011336207, + 0.0036313749878900126, + 0.0035075839987257496, + 0.003722833003848791, + 0.0038349579990608618, + 0.003937583009246737, + 0.0038535839994437993, + 0.003821708000032231, + 0.003923166994354688, + 0.004508166995947249, + 0.004063625005073845, + 0.0041045830002985895, + 0.004051333002280444, + 0.004153417001361959, + 0.004254790997947566, + 0.004288458003429696, + 0.0041802499908953905, + 0.004317165992688388 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_mha": { + "first_token_ms": { + "count": 5, + "median": 2.216708002379164, + "p95": 2.26587459910661, + "min": 1.9406250066822395, + "max": 2.2749159979866818, + "iqr": 0.10095900506712496, + "mean": 2.158141601830721, + "cv": 0.061456806887378826 + }, + "decode_ms_per_step": { + "count": 315, + "median": 3.4354999952483922, + "p95": 4.777370700321626, + "min": 2.0932079933118075, + "max": 5.460333006340079, + "iqr": 1.3314789975993335, + "mean": 3.4680869077361884, + "cv": 0.22843077728227218 + }, + "total_ms": { + "count": 5, + "median": 219.92549703281838, + "p95": 228.30097780388314, + "min": 211.56474798044655, + "max": 228.59800200967584, + "iqr": 11.075925038312562, + "mean": 220.6476167892106, + "cv": 0.03278244116041478 + }, + "tokens_per_second": { + "count": 5, + "median": 2328.061124825362, + "p95": 2410.0433068328603, + "min": 2239.7396105777366, + "max": 2420.062911649726, + "iqr": 115.57932480428735, + "mean": 2322.442819475866, + "cv": 0.032856474314059476 + }, + "median_step_ms_by_index": [ + 2.216708002379164, + 2.4322499957634136, + 2.1965830092085525, + 2.251041994895786, + 2.6608339976519346, + 2.2434159909607843, + 2.2239169920794666, + 2.326166009879671, + 2.403165999567136, + 2.5025840004673228, + 2.693834001547657, + 2.5787079939618707, + 2.548333999584429, + 2.6322079938836396, + 2.6169579941779375, + 2.766416990198195, + 2.747833001194522, + 2.7982920000795275, + 2.8737079992424697, + 2.8912499983562157, + 2.817790998960845, + 3.17070800520014, + 3.3532079978613183, + 3.034915993339382, + 3.0614160059485584, + 3.126916999462992, + 3.1982499931473285, + 3.2726670033298433, + 3.2985840080073103, + 3.2610839989501983, + 3.3067500044126064, + 3.375625004991889, + 3.3639589964877814, + 3.4294159995624796, + 3.502457999275066, + 3.4916670119855553, + 3.5022079973714426, + 3.622124990215525, + 3.708125004777685, + 3.698291999171488, + 3.7591669970424846, + 3.8066660054028034, + 3.765874993405305, + 3.784209009609185, + 3.836125004454516, + 3.895916001056321, + 3.8687089981976897, + 3.86495899874717, + 3.8615829980699345, + 4.320167005062103, + 4.07075000111945, + 4.177625000011176, + 4.229832993587479, + 4.277374988305382, + 4.274459002772346, + 4.365166998468339, + 4.4579170062206686, + 4.624166991561651, + 4.464249999728054, + 4.580042004818097, + 4.639833001419902, + 4.608999995980412, + 5.075000008218922, + 4.945667009451427 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.002128749998519197, + 0.0024322499957634136, + 0.0021965830092085525, + 0.0026149170007556677, + 0.0026608339976519346, + 0.002172166990931146, + 0.00216358300531283, + 0.0022061669878894463, + 0.002311541000381112, + 0.002391208996414207, + 0.0029127919988241047, + 0.0024851250054780394, + 0.0024532910028938204, + 0.002914374999818392, + 0.0024907499901019037, + 0.00284754199674353, + 0.0026870409928960726, + 0.0026860829966608435, + 0.002517082990380004, + 0.0026952079933835194, + 0.0027798329974757507, + 0.00280545798887033, + 0.0033532079978613183, + 0.0029277919966261834, + 0.0030545829940820113, + 0.0029936250066384673, + 0.0029952080076327547, + 0.00311091699404642, + 0.003396999993128702, + 0.0033051249920390546, + 0.0033067500044126064, + 0.0033499580022180453, + 0.0033054170053219423, + 0.0033989160001510754, + 0.0034628340072231367, + 0.003461333006271161, + 0.003435999999055639, + 0.0036040830018464476, + 0.003599792005843483, + 0.003698291999171488, + 0.003736416998435743, + 0.004200957991997711, + 0.003765874993405305, + 0.003784209009609185, + 0.0037661669921362773, + 0.0038257080013863742, + 0.00414679100504145, + 0.003850624998449348, + 0.0037994590093148872, + 0.003921041003195569, + 0.003979625005740672, + 0.004159125004662201, + 0.004201750009087846, + 0.004165667007328011, + 0.004197666989057325, + 0.0043651669984683394, + 0.004290374999982305, + 0.004440666001755744, + 0.004305665992433205, + 0.004486083998926915, + 0.004583791000186466, + 0.004523625000729226, + 0.005120499990880489, + 0.005106582990265451 + ], + [ + 0.0019406250066822395, + 0.0020932079933118075, + 0.002110666988301091, + 0.0022248340101214126, + 0.002264082999317907, + 0.0022221670078579336, + 0.0021824169962201267, + 0.0022571659937966615, + 0.0022815830016043037, + 0.0024514999968232587, + 0.0025609579897718504, + 0.0025787079939618707, + 0.002548333999584429, + 0.002523540999391116, + 0.0025770830106921494, + 0.00234529199951794, + 0.002747833001194522, + 0.0028965419915039092, + 0.002915542005212046, + 0.002701958001125604, + 0.0027133330004289746, + 0.002790207989164628, + 0.002996415991219692, + 0.0029243750032037497, + 0.0030898749973857775, + 0.00302100001135841, + 0.003081458999076858, + 0.0031635829946026206, + 0.0032985840080073103, + 0.003289875006885268, + 0.003204166001523845, + 0.003189167007803917, + 0.0031334590021288022, + 0.003409124998142943, + 0.003509083006065339, + 0.0034807079937309027, + 0.0032854159944690764, + 0.0034407089988235384, + 0.003679624991491437, + 0.003677917004097253, + 0.003691541001899168, + 0.00351408300048206, + 0.003731958000571467, + 0.0037334160006139427, + 0.0035573330096667632, + 0.0037290420004865155, + 0.0035770420072367415, + 0.004110666995984502, + 0.0038417500036302954, + 0.004058833990711719, + 0.0039012089982861653, + 0.004240249996655621, + 0.004229832993587479, + 0.004210166996927001, + 0.004135583993047476, + 0.00432829100464005, + 0.0043580410128924996, + 0.0045160000008763745, + 0.0044327500072540715, + 0.004546583004412241, + 0.004480624993448146, + 0.004539749992545694, + 0.004660124992369674, + 0.004637749996618368 + ], + [ + 0.002216708002379164, + 0.0025682090054033324, + 0.00233220899826847, + 0.002251041994895786, + 0.0027378750091884285, + 0.002402207988779992, + 0.0027803330012829974, + 0.00240545799897518, + 0.002448374987579882, + 0.002553958009229973, + 0.002693834001547657, + 0.003085000003920868, + 0.0026106670120498165, + 0.0026971670013153926, + 0.0028406659985193983, + 0.002766416990198195, + 0.0027644170040730387, + 0.0029525419959099963, + 0.002947708999272436, + 0.002903665998019278, + 0.003026041988050565, + 0.00317070800520014, + 0.003416958003072068, + 0.0032830419950187206, + 0.003468208000413142, + 0.003126916999462992, + 0.0032435419998364523, + 0.003368750010849908, + 0.003452291988651268, + 0.0032506669958820567, + 0.003506499997456558, + 0.0034445000055711716, + 0.003431208009715192, + 0.0035567909944802523, + 0.003502457999275066, + 0.0034916670119855553, + 0.003620041999965906, + 0.00419858300301712, + 0.004128625005250797, + 0.003764749999390915, + 0.0038078749930718914, + 0.003923416996258311, + 0.00377408298663795, + 0.003776958998059854, + 0.004144833001191728, + 0.004133832990191877, + 0.0038687089981976897, + 0.00386495899874717, + 0.004341374995419756, + 0.004320167005062103, + 0.004320959007600322, + 0.004126207990339026, + 0.004502833995502442, + 0.004336375001003034, + 0.004518000001553446, + 0.004303583002183586, + 0.004503250005654991, + 0.004624166991561651, + 0.005147083997144364, + 0.004580042004818097, + 0.005169166994164698, + 0.004769042003317736, + 0.005075000008218922, + 0.0047702500014565885 + ], + [ + 0.002229709003586322, + 0.002666417000000365, + 0.0027635830047074705, + 0.00241670900140889, + 0.0026904160040430725, + 0.0022434159909607843, + 0.002731959000811912, + 0.002452458007610403, + 0.002403165999567136, + 0.0025025840004673228, + 0.0030401249969145283, + 0.002889915995183401, + 0.0030074589885771275, + 0.0026322079938836396, + 0.0026719590096035972, + 0.003052499989280477, + 0.002720832999330014, + 0.002793583000311628, + 0.002820042005623691, + 0.002902542008087039, + 0.0029554170032497495, + 0.0035190830094506964, + 0.0034354999952483922, + 0.003034915993339382, + 0.0030431670020334423, + 0.0031517500028712675, + 0.0032319169986294582, + 0.003285957995103672, + 0.0032562920096097514, + 0.0032610839989501983, + 0.0033534579997649416, + 0.003457333004917018, + 0.0034538749896455556, + 0.0035195420059608296, + 0.0033970829972531646, + 0.003540917008649558, + 0.004071916991961189, + 0.003939083006116562, + 0.0037100839981576428, + 0.004073000003700145, + 0.003800833990680985, + 0.0038066660054028034, + 0.0037470410024980083, + 0.0038082499959273264, + 0.004511082996032201, + 0.004500916998949833, + 0.003851333007332869, + 0.003871834007441066, + 0.004590291995555162, + 0.004638292011804879, + 0.00407075000111945, + 0.004246457989211194, + 0.004266375006409362, + 0.004277374988305382, + 0.004274459002772346, + 0.004972750000888482, + 0.005460333006340079, + 0.0049083329940913245, + 0.004464249999728054, + 0.004666875000111759, + 0.004639833001419902, + 0.004608999995980412, + 0.00509320899436716, + 0.005198500002734363 + ], + [ + 0.0022749159979866818, + 0.0021733339963248, + 0.002159583003958687, + 0.0022322500008158386, + 0.0022503329964820296, + 0.0022907079983269796, + 0.0022239169920794666, + 0.002326166009879671, + 0.0024547499924665317, + 0.002526958007365465, + 0.002511041995603591, + 0.0024607910017948598, + 0.002500541988410987, + 0.002547500000218861, + 0.0026169579941779375, + 0.002675915995496325, + 0.0027740840014303103, + 0.0027982920000795275, + 0.0028737079992424697, + 0.0028912499983562157, + 0.002817790998960845, + 0.0032022500090533867, + 0.003053084001294337, + 0.0030529159994330257, + 0.0030614160059485584, + 0.003150707998429425, + 0.0031982499931473285, + 0.0032726670033298433, + 0.0031564579985570163, + 0.0032415830064564943, + 0.0032374170114053413, + 0.003375625004991889, + 0.0033639589964877814, + 0.0034294159995624796, + 0.0035354580031707883, + 0.003496333010843955, + 0.0035022079973714426, + 0.003622124990215525, + 0.003708125004777685, + 0.0036656250013038516, + 0.0037591669970424846, + 0.003804209001827985, + 0.004192791995592415, + 0.00383391699870117, + 0.003836125004454516, + 0.003895916001056321, + 0.0038876250036992133, + 0.0038255830004345626, + 0.0038615829980699345, + 0.004353459007688798, + 0.004171291991951875, + 0.004177625000011176, + 0.003948209006921388, + 0.004680166995967738, + 0.0043196250044275075, + 0.004888791998382658, + 0.0044579170062206686, + 0.004703915998106822, + 0.00531974999466911, + 0.004780666989972815, + 0.004806791010196321, + 0.004992333007976413, + 0.004775958004756831, + 0.0049456670094514266 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "speedup_cached_vs_uncached_sdpa": 1.4499488400759426, + "speedup_cached_vs_uncached_mha": 1.6202136161948364 + }, + { + "batch": 8, + "prompt_len": 128, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_mha": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 10.391125004389323, + "p95": 12.234058199101128, + "min": 6.0186659975443035, + "max": 12.40045799931977, + "iqr": 5.105750999064185, + "mean": 9.368283199728467, + "cv": 0.31459074108100016 + }, + "decode_ms_per_step": { + "count": 315, + "median": 3.338707989314571, + "p95": 4.647675095475279, + "min": 2.331750001758337, + "max": 7.415166997816414, + "iqr": 1.165207999292761, + "mean": 3.3819415650445053, + "cv": 0.2337555781871384 + }, + "total_ms": { + "count": 5, + "median": 242.56945998058654, + "p95": 270.1699003664544, + "min": 169.4473790266784, + "max": 276.52091697382275, + "iqr": 65.91641486738808, + "mean": 222.4306017975323, + "cv": 0.20769282291153124 + }, + "tokens_per_second": { + "count": 5, + "median": 2110.7356220398756, + "p95": 2989.8182977598394, + "min": 1851.5778321698144, + "max": 3021.5870138622145, + "iqr": 770.9482192690193, + "mean": 2387.687823100713, + "cv": 0.21753962573619653 + }, + "median_step_ms_by_index": [ + 10.391125004389323, + 4.05241698899772, + 4.119500008528121, + 3.781541992793791, + 3.6614579876186326, + 3.314166999189183, + 3.338124995934777, + 3.7113330035936087, + 3.6010409967275336, + 3.678249995573424, + 3.1633330072509125, + 3.64287500269711, + 3.418334003072232, + 3.69391699496191, + 3.806583001278341, + 3.4081670019077137, + 3.7279170064721256, + 3.7235000054351985, + 3.640708004240878, + 3.38004098739475, + 3.4738329995889217, + 3.154291000100784, + 3.0740410002181306, + 3.4330420021433383, + 3.5627500037662685, + 3.4769169869832695, + 3.3227910025743768, + 3.1433749973075464, + 3.2186250027734786, + 3.4962500067194924, + 3.3247499959543347, + 2.9607080068672076, + 3.2902499951887876, + 3.414250008063391, + 3.4037500008707866, + 3.023292010766454, + 2.7149169909534976, + 3.272916001151316, + 3.5419580090092495, + 3.2738749869167805, + 3.4262910048710182, + 2.9568330064648762, + 3.4486670047044754, + 3.216832992620766, + 3.2479999936185777, + 3.338707989314571, + 3.358916990691796, + 3.283666999777779, + 3.3937090047402307, + 3.074334003031254, + 3.278874995885417, + 2.9856250039301813, + 3.378041001269594, + 3.2877500052563846, + 3.364792006323114, + 3.3515840041218325, + 3.1051669939188287, + 2.9648750060005113, + 3.320417003124021, + 3.610874991863966, + 3.3826250000856817, + 3.1795830000191927, + 3.073750005569309, + 3.3278339979005978 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.01156845899822656, + 0.00405241698899772, + 0.004119500008528121, + 0.003781541992793791, + 0.0036614579876186326, + 0.003314166999189183, + 0.003338124995934777, + 0.0037113330035936087, + 0.0036010409967275336, + 0.003678249995573424, + 0.0031633330072509125, + 0.00364287500269711, + 0.003418334003072232, + 0.00369391699496191, + 0.004095291995326988, + 0.0034081670019077137, + 0.004139166994718835, + 0.003737665989319794, + 0.003661583992652595, + 0.00338004098739475, + 0.0034738329995889217, + 0.0036474579974310473, + 0.0031275829969672486, + 0.0034330420021433383, + 0.0038705000042682514, + 0.0038077079952927306, + 0.003890958003466949, + 0.004072916999575682, + 0.0035448330017970875, + 0.0037930420076008886, + 0.0036808749864576384, + 0.0035726659989450127, + 0.004099625002709217, + 0.003414250008063391, + 0.003955041000153869, + 0.003755249999812804, + 0.003693749997182749, + 0.0035025830002268776, + 0.0036409169988473877, + 0.0038202919968171045, + 0.003752290998818353, + 0.0037052499974379316, + 0.0034486670047044754, + 0.0038238749984884635, + 0.003770082999835722, + 0.0037349589983932674, + 0.003618084010668099, + 0.003555458999471739, + 0.003696375002618879, + 0.003447666997089982, + 0.0038185419980436563, + 0.0033242909994442016, + 0.0036270830023568124, + 0.0038024169916752726, + 0.003793875002884306, + 0.003984750001109205, + 0.0035152909986209124, + 0.003841458004899323, + 0.004156249997322448, + 0.003975958999944851, + 0.0037819999997736886, + 0.0039509170019300655, + 0.0038548750017071143, + 0.0038216249959077686 + ], + [ + 0.01240045799931977, + 0.007415166997816414, + 0.006166708000819199, + 0.006007417003274895, + 0.005386916993302293, + 0.00530366699967999, + 0.004390625006635673, + 0.004281332992832176, + 0.004400624995469116, + 0.004390458998386748, + 0.004201000003376976, + 0.00417054099671077, + 0.004155083996010944, + 0.004099041994777508, + 0.003806583001278341, + 0.0038179580005817115, + 0.0037279170064721256, + 0.0037235000054351985, + 0.003640708004240878, + 0.00356137499329634, + 0.0035282919998280704, + 0.003154291000100784, + 0.0030740410002181306, + 0.0035975829960079864, + 0.0035627500037662685, + 0.0034769169869832695, + 0.0033227910025743768, + 0.0031433749973075464, + 0.0032186250027734786, + 0.0034962500067194924, + 0.0033247499959543347, + 0.0029536670044763014, + 0.0032902499951887876, + 0.0034751670027617365, + 0.0034037500008707866, + 0.0029289590020198375, + 0.0027149169909534976, + 0.003272916001151316, + 0.0035419580090092495, + 0.0032738749869167805, + 0.0034262910048710182, + 0.00273929099785164, + 0.003611417007050477, + 0.003216832992620766, + 0.0032479999936185777, + 0.003338707989314571, + 0.003358916990691796, + 0.003283666999777779, + 0.0033937090047402307, + 0.003074334003031254, + 0.003278874995885417, + 0.0029856250039301813, + 0.003378041001269594, + 0.0032877500052563846, + 0.003364792006323114, + 0.0033515840041218325, + 0.0031051669939188287, + 0.0029648750060005113, + 0.003320417003124021, + 0.0030751669983146712, + 0.0033826250000856817, + 0.0031795830000191927, + 0.003073750005569309, + 0.0033278339979005978 + ], + [ + 0.006462707999162376, + 0.003241917002014816, + 0.003129083997919224, + 0.0025178750074701384, + 0.0028354160021990538, + 0.0027994170086458325, + 0.0028865830099675804, + 0.002842041998519562, + 0.003044583005248569, + 0.0024021249992074445, + 0.0028739170084008947, + 0.002798791989334859, + 0.0028241250111022964, + 0.002731083004618995, + 0.002983875005156733, + 0.0025232499901903793, + 0.002834707993315533, + 0.0028147920093033463, + 0.002404791986919008, + 0.002926332992501557, + 0.0029608750046463683, + 0.0024744999973336235, + 0.0024024580052355304, + 0.0029159999976400286, + 0.0029312499973457307, + 0.0023892089957371354, + 0.002671167007065378, + 0.0029141660052118823, + 0.0024323750112671405, + 0.002730583000811748, + 0.0024199580075219274, + 0.0029607080068672076, + 0.0029131670016795397, + 0.0029822920041624457, + 0.0025610000011511147, + 0.002888166010961868, + 0.002339791986742057, + 0.0029646249895449728, + 0.0026746250077849254, + 0.0028328339976724237, + 0.0023746670049149543, + 0.0029568330064648762, + 0.002556916995672509, + 0.002401667006779462, + 0.0029175420058891177, + 0.002833834005286917, + 0.0032360829936806113, + 0.0029091250034980476, + 0.002435750007862225, + 0.0024333750043297186, + 0.0027093749959021807, + 0.002956666998215951, + 0.0024333750043297186, + 0.00249725001049228, + 0.0027526249905349687, + 0.002409124994301237, + 0.0024272500013466924, + 0.002908249996835366, + 0.0023764169891364872, + 0.003610874991863966, + 0.0029407080000964925, + 0.0025723750004544854, + 0.00244941700657364, + 0.0025127499975496903 + ], + [ + 0.0060186659975443035, + 0.003076791006606072, + 0.0024436669918941334, + 0.002518749999580905, + 0.0023981249978533015, + 0.0025958749902201816, + 0.002508707999368198, + 0.0027821249968837947, + 0.002471875006449409, + 0.0023847499978728592, + 0.002462834003381431, + 0.0024771250027697533, + 0.0025004589988384396, + 0.0025084169901674613, + 0.0028683749987976626, + 0.002520082998671569, + 0.00250791700091213, + 0.002492125000571832, + 0.002459458992234431, + 0.002518459004932083, + 0.0024864999868441373, + 0.002331750001758337, + 0.002500332993804477, + 0.0024992499966174364, + 0.0024952920066425577, + 0.0025350420037284493, + 0.0032321670005330816, + 0.0030243329965742305, + 0.00267558399355039, + 0.0025546670076437294, + 0.0030390419997274876, + 0.002510959006031044, + 0.002528875003918074, + 0.0025284170114900917, + 0.0026539170066826046, + 0.003023292010766454, + 0.002489040998625569, + 0.0026803750079125166, + 0.003119082990451716, + 0.002599915998871438, + 0.002530957994167693, + 0.002527208998799324, + 0.0025826670025708154, + 0.0026327079976908863, + 0.0025025420036399737, + 0.002594958001282066, + 0.0025594999897293746, + 0.0026441670052008703, + 0.0025095000019064173, + 0.002584415997262113, + 0.0026453750033397228, + 0.0025219169910997152, + 0.002573250007117167, + 0.002533625010983087, + 0.0025269170000683516, + 0.00255762500455603, + 0.0025287500029662624, + 0.002540083005442284, + 0.0026082080003106967, + 0.0026430419966345653, + 0.0025419590092496946, + 0.0024420830013696104, + 0.002538208005717024, + 0.0025552919978508726 + ], + [ + 0.010391125004389323, + 0.004199792005238123, + 0.00461337500018999, + 0.0039970419893506914, + 0.004162249999353662, + 0.004254790997947566, + 0.005146875002537854, + 0.0046958339953562245, + 0.004860665998421609, + 0.006978499994147569, + 0.005942875010077842, + 0.0058107079967157915, + 0.004961583006661385, + 0.0046670419978909194, + 0.004801667004358023, + 0.004845250005018897, + 0.004419250006321818, + 0.004301541994209401, + 0.004369333997601643, + 0.004269666998879984, + 0.004301041000871919, + 0.003919750000932254, + 0.004060499995830469, + 0.004223374999128282, + 0.004022916007670574, + 0.003923542011762038, + 0.003920874994946644, + 0.004697041993495077, + 0.0035450420109555125, + 0.003735291989869438, + 0.0037318750109989196, + 0.004047249996801838, + 0.004009083000710234, + 0.0036680839984910563, + 0.0037720830005127937, + 0.004052874995977618, + 0.0040614169993205, + 0.0038934170006541535, + 0.0037784169981023297, + 0.004203333010082133, + 0.003848374995868653, + 0.004064833003212698, + 0.003806041000643745, + 0.003922916992451064, + 0.0037158750055823475, + 0.004031707998365164, + 0.0038404579972848296, + 0.003606499987654388, + 0.004207541991490871, + 0.003863749996526167, + 0.004295042002922855, + 0.004048582995892502, + 0.004639374994440004, + 0.0038530000019818544, + 0.0036765419936273247, + 0.003840167002636008, + 0.004211958002997562, + 0.0038813330029370263, + 0.003708125004777685, + 0.003974499995820224, + 0.004194666995317675, + 0.0038769999955547974, + 0.0042577080021146685, + 0.0038985409919405356 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + ] + } + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 6.4185830124188215, + "p95": 11.043633008375764, + "min": 5.679459005477838, + "max": 11.916583011043258, + "iqr": 1.7254580016015097, + "mean": 7.4785666045499966, + "cv": 0.3460586684019925 + }, + "decode_ms_per_step": { + "count": 315, + "median": 8.226707999710925, + "p95": 17.045037499337923, + "min": 5.968375000520609, + "max": 23.789667000528425, + "iqr": 4.054145494592376, + "mean": 9.890887567806724, + "cv": 0.3629906366254196 + }, + "total_ms": { + "count": 5, + "median": 524.4824169785716, + "p95": 931.7266753670992, + "min": 471.03904094547033, + "max": 1007.3837519594235, + "iqr": 108.0795309972018, + "mean": 630.6044833763735, + "cv": 0.34624602208975824 + }, + "tokens_per_second": { + "count": 5, + "median": 976.2005043934932, + "p95": 1066.1049871262148, + "min": 508.2472285304666, + "max": 1086.9587348265502, + "iqr": 168.82681493457414, + "mean": 873.5919290931367, + "cv": 0.25914267128711327 + }, + "median_step_ms_by_index": [ + 6.4185830124188215, + 6.05599999835249, + 6.604000009247102, + 6.604458001675084, + 7.924374993308447, + 6.969750000280328, + 6.867458010674454, + 6.958457990549505, + 6.775249989004806, + 6.945083005120978, + 7.271458991453983, + 7.081250005285256, + 7.1347080083796754, + 7.289083005161956, + 7.456332998117432, + 7.251957998960279, + 7.133708000765182, + 7.434415994794108, + 7.643208009540103, + 7.716000007349066, + 7.748457996058278, + 8.008083008462563, + 7.8940829989733174, + 7.887750005465932, + 7.806416993844323, + 7.829791997210123, + 7.969709011376835, + 8.089583992841654, + 8.077249993220903, + 8.111832998110913, + 7.983417002833448, + 8.073417004197836, + 7.951207997393794, + 8.12208399293013, + 8.393457988859154, + 8.25083399831783, + 8.19270798820071, + 8.318042004248127, + 8.167207997757941, + 8.48483300069347, + 8.226707999710925, + 8.851167003740557, + 8.506625003064983, + 8.470583008602262, + 8.22145800339058, + 8.244750002631918, + 8.535291999578476, + 8.478625008137897, + 8.66333300655242, + 9.527207992505282, + 9.863958999630995, + 11.75649999640882, + 12.057917003403418, + 11.672459004330449, + 12.043082999298349, + 11.287083005299792, + 12.380166997900233, + 11.4131250011269, + 9.968250000383705, + 9.843457999522798, + 9.260374994482845, + 9.834083000896499, + 9.219999992637895, + 9.22862500010524 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0064185830124188215, + 0.006018957996275276, + 0.006604000009247102, + 0.006581333000212908, + 0.006560291993082501, + 0.006794042012188584, + 0.006684874999336898, + 0.006815208005718887, + 0.006775249989004806, + 0.0068281249987194315, + 0.007271458991453983, + 0.006818750000093132, + 0.00710554099350702, + 0.007009833992924541, + 0.007104000003891997, + 0.007062749995384365, + 0.007133708000765182, + 0.007267499997396953, + 0.007006707994150929, + 0.007276291988091543, + 0.007059541996568441, + 0.007227458001580089, + 0.007251499991980381, + 0.007506916997954249, + 0.007327291998080909, + 0.00747399999818299, + 0.007685208998736925, + 0.007510041992645711, + 0.007900208001956344, + 0.007568875007564202, + 0.0077665419958066195, + 0.007715083003859036, + 0.007870957997511141, + 0.00738279199867975, + 0.007937750007840805, + 0.008045624999795109, + 0.008005458003026433, + 0.008225500001572073, + 0.008005708004930057, + 0.00848483300069347, + 0.008226707999710925, + 0.008851167003740557, + 0.008350916992640123, + 0.00797433299885597, + 0.007562459009932354, + 0.008244750002631918, + 0.007950583007186651, + 0.008478625008137897, + 0.008217499998863786, + 0.009527207992505282, + 0.009863958999630995, + 0.01175649999640882, + 0.012057917003403418, + 0.011672459004330449, + 0.012043082999298349, + 0.011287083005299792, + 0.012380166997900233, + 0.0114131250011269, + 0.009968250000383705, + 0.009843457999522798, + 0.009260374994482845, + 0.009834083000896499, + 0.009219999992637895, + 0.009409207996213809 + ], + [ + 0.007551832997705787, + 0.007731959005468525, + 0.008118665995425545, + 0.007570665999082848, + 0.007924374993308447, + 0.008174374990630895, + 0.0077097500034142286, + 0.00746574999357108, + 0.008336541999597102, + 0.007593709000502713, + 0.00801062500977423, + 0.00772520799364429, + 0.008040083994274028, + 0.007706250005867332, + 0.007994083003723063, + 0.007754582999041304, + 0.007850958005292341, + 0.007772249999106862, + 0.007652750005945563, + 0.0077160000073490664, + 0.007905874997959472, + 0.008198957992135547, + 0.007894082998973317, + 0.007887750005465932, + 0.007806416993844323, + 0.007829791997210123, + 0.007969709011376835, + 0.008089583992841654, + 0.008077249993220903, + 0.008111832998110913, + 0.007983417002833448, + 0.008073417004197836, + 0.007951207997393794, + 0.00812208399293013, + 0.008393457988859154, + 0.00825083399831783, + 0.00819270798820071, + 0.008318042004248127, + 0.008167207997757941, + 0.008336541999597102, + 0.007740000000922009, + 0.00789062499825377, + 0.008306584000820294, + 0.008470583008602262, + 0.008029624994378537, + 0.008079584004008211, + 0.008535291999578476, + 0.008109834001515992, + 0.00866333300655242, + 0.008219333001761697, + 0.008490249994792975, + 0.00826670900278259, + 0.00849137500335928, + 0.008430000001681037, + 0.008578499997383915, + 0.008254167012637481, + 0.008449957997072488, + 0.008418166995397769, + 0.00881962499988731, + 0.008927166010835208, + 0.008630209005787037, + 0.009256292003556155, + 0.008772417000727728, + 0.00922862500010524 + ], + [ + 0.005679459005477838, + 0.005968375000520609, + 0.006180500000482425, + 0.006604458001675084, + 0.008062000008067116, + 0.006969750000280328, + 0.006867458010674454, + 0.006958457990549505, + 0.006688791996566579, + 0.006661375009571202, + 0.006811957995523699, + 0.007081250005285256, + 0.0071347080083796754, + 0.007289083005161956, + 0.007456332998117432, + 0.007251957998960279, + 0.007132541999453679, + 0.007434415994794108, + 0.007643208009540103, + 0.007891916000517085, + 0.007748457996058278, + 0.008008083008462563, + 0.00834374999976717, + 0.008249374994193204, + 0.008989166002720594, + 0.009328874992206693, + 0.00885458399716299, + 0.00874150000163354, + 0.00855362499714829, + 0.008727583001018502, + 0.009349500003736466, + 0.00896483298856765, + 0.009333750000223517, + 0.009571000002324581, + 0.010273583000525832, + 0.010102041997015476, + 0.00994637499388773, + 0.009642374992836267, + 0.010461375000886619, + 0.010546624995185994, + 0.010451417008880526, + 0.010886332995141856, + 0.023789667000528425, + 0.012281290997634642, + 0.010590374993626028, + 0.01066904200706631, + 0.010519125004066154, + 0.01129462500102818, + 0.010985582994180731, + 0.011715667002135888, + 0.011979832997894846, + 0.013577540987171233, + 0.013111584004946053, + 0.01349166699219495, + 0.014748208006494679, + 0.015490292003960349, + 0.014766624997719191, + 0.01574595800775569, + 0.01555916600045748, + 0.013883040999644436, + 0.011461208006949164, + 0.010473666989128105, + 0.00925120800093282, + 0.008870791993103921 + ], + [ + 0.005826374996104278, + 0.00605599999835249, + 0.006500082992715761, + 0.006551875005243346, + 0.006500499992398545, + 0.006572833997779526, + 0.006706582993501797, + 0.006801583003834821, + 0.006651207993854769, + 0.006945083005120978, + 0.006314833997748792, + 0.006605167000088841, + 0.006620957996346988, + 0.006410709000192583, + 0.006839416004368104, + 0.006159124997793697, + 0.006571457997779362, + 0.007029542000964284, + 0.006663000007392839, + 0.007094916989444755, + 0.006730499997502193, + 0.0071292499924311414, + 0.006878415995743126, + 0.007115250002243556, + 0.007099000009475276, + 0.006997249991400167, + 0.0071604580007260665, + 0.007206000009318814, + 0.007383749994914979, + 0.006989916000748053, + 0.007103625001036562, + 0.007040166005026549, + 0.007232332995044999, + 0.007346707992837764, + 0.00728258398885373, + 0.0072538340027676895, + 0.007310999993933365, + 0.007672040999750607, + 0.007822333995136432, + 0.007624583013239317, + 0.00799962499877438, + 0.007754334001219831, + 0.008506625003064983, + 0.0076495840039569885, + 0.00822145800339058, + 0.007901499993749894, + 0.008252291998360306, + 0.007416499996907078, + 0.008413541989284568, + 0.00797045900253579, + 0.008461082994472235, + 0.007884624996222556, + 0.008586958996602334, + 0.00805962500453461, + 0.008752333000302315, + 0.008110333001241088, + 0.0084640830027638, + 0.008199125004466623, + 0.007964791991980746, + 0.0077560420031659305, + 0.00806612501037307, + 0.008415999996941537, + 0.008637915991130285, + 0.007793833006871864 + ], + [ + 0.011916583011043258, + 0.01151670799299609, + 0.011276666991761886, + 0.012381041990011, + 0.011705750002874993, + 0.011572499992325902, + 0.012616416002856568, + 0.013192540995078161, + 0.012251666994416155, + 0.012482500009355135, + 0.013059125005383976, + 0.013304708001669496, + 0.013418542002909817, + 0.01377975000650622, + 0.01415524999902118, + 0.013748333003604785, + 0.01351224999234546, + 0.014697958002216183, + 0.01500254200072959, + 0.014731291012139991, + 0.014957292005419731, + 0.01584629199351184, + 0.015618167002685368, + 0.01501412499055732, + 0.013724124990403652, + 0.015451583007234149, + 0.01592112499929499, + 0.016286500002024695, + 0.0162222499930067, + 0.016535916991415434, + 0.017080999998142943, + 0.01644000000669621, + 0.016468083005747758, + 0.01688145799562335, + 0.016709542003809474, + 0.016991041993605904, + 0.01726937500643544, + 0.017010292009217665, + 0.016747540998039767, + 0.016552500004763715, + 0.01640058400516864, + 0.017029624999850057, + 0.01701399999728892, + 0.01748079201206565, + 0.016597249996266328, + 0.016562999997404404, + 0.0166472499986412, + 0.016541749995667487, + 0.01589783300005365, + 0.01657337498909328, + 0.017211166996276006, + 0.016629124991595745, + 0.016686499991919845, + 0.017089832996134646, + 0.017435500005376525, + 0.017738791997544467, + 0.01858370899572037, + 0.01923145899490919, + 0.019039541992242448, + 0.019921792001696303, + 0.018446709000272676, + 0.01948775000346359, + 0.019490582999424078, + 0.01959549999446608 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_mha": { + "first_token_ms": { + "count": 5, + "median": 7.188957999460399, + "p95": 11.361150600714609, + "min": 6.791166990296915, + "max": 12.396459002047777, + "iqr": 0.17183400632347912, + "mean": 8.128916795249097, + "cv": 0.2942121149479543 + }, + "decode_ms_per_step": { + "count": 315, + "median": 9.584083003574051, + "p95": 16.370287699101027, + "min": 6.894500009366311, + "max": 17.82462499977555, + "iqr": 5.778124992502853, + "mean": 10.827097105137664, + "cv": 0.3007682113030828 + }, + "total_ms": { + "count": 5, + "median": 667.3485799547052, + "p95": 934.1273274330888, + "min": 536.5155009785667, + "max": 995.273252032348, + "iqr": 127.04441894311458, + "mean": 690.2360344189219, + "cv": 0.264693607282118 + }, + "tokens_per_second": { + "count": 5, + "median": 767.2152386010184, + "p95": 945.4895572298042, + "min": 514.4315884652742, + "max": 954.3060714297123, + "iqr": 167.70340679120807, + "mean": 777.739298513028, + "cv": 0.22215509943172554 + }, + "median_step_ms_by_index": [ + 7.188957999460399, + 7.037625007797033, + 7.045749996905215, + 7.190041986177675, + 7.694874992012046, + 7.5501250103116035, + 7.692915998632088, + 7.783542008837685, + 7.520582992583513, + 8.019999993848614, + 7.43866601260379, + 7.927166996523738, + 8.034791011596099, + 8.298207991174422, + 7.890749999205582, + 8.50745900243055, + 8.288999990327284, + 8.309790995554067, + 8.408666995819658, + 8.576416003052145, + 8.300041998154484, + 9.118415997363627, + 8.895584003766999, + 9.4799999933457, + 9.69666700984817, + 9.306874999310821, + 9.83029200870078, + 9.13083300110884, + 9.755624996614642, + 9.853415991528891, + 9.882874990580603, + 10.120624996488914, + 9.716166008729488, + 10.127084009582177, + 9.821582993026823, + 10.499541996978223, + 10.322917005396448, + 10.97495800058823, + 10.463625003467314, + 10.90233400464058, + 10.667750000720844, + 11.175708001246676, + 11.488207994261757, + 11.26808300614357, + 11.512541997944936, + 11.858709010994062, + 11.630708002485335, + 11.818250000942498, + 11.953249995713122, + 12.436750010238029, + 11.968250008067116, + 12.31954200193286, + 13.084375008475035, + 13.814541991450824, + 13.870916998712346, + 13.84983399475459, + 13.309375004610047, + 14.174083000398241, + 14.917166001396254, + 14.402083004824817, + 13.573832999099977, + 14.58141599141527, + 15.771124992170371, + 16.738874997827224 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.007188957999460399, + 0.008025499992072582, + 0.007498166989535093, + 0.007991375008714385, + 0.0081848749978235, + 0.008676834011566825, + 0.007877958007156849, + 0.008471124994684942, + 0.00803233300393913, + 0.008019999993848614, + 0.008012624995899387, + 0.0087157080124598, + 0.008163416991010308, + 0.009065624995855615, + 0.00807291700039059, + 0.00850745900243055, + 0.008326583993039094, + 0.00901999999769032, + 0.008408666995819658, + 0.009220583000569604, + 0.008300041998154484, + 0.009485583999776281, + 0.008895584003766999, + 0.009833584001171403, + 0.00969666700984817, + 0.00952766700356733, + 0.00983029200870078, + 0.00913083300110884, + 0.009755624996614642, + 0.009964083990780637, + 0.010184333004872315, + 0.010120624996488914, + 0.010181500008911826, + 0.010127084009582177, + 0.01040416699834168, + 0.010499541996978223, + 0.011055374998250045, + 0.01097495800058823, + 0.010463625003467314, + 0.010964666988002136, + 0.010667750000720844, + 0.011175708001246676, + 0.011488207994261757, + 0.01126808300614357, + 0.011836666002636775, + 0.012669916002778336, + 0.011630708002485335, + 0.012270791004993953, + 0.011998749992926605, + 0.012436750010238029, + 0.012916249994304962, + 0.01231954200193286, + 0.013084375008475035, + 0.013814541991450824, + 0.013870916998712346, + 0.01384983399475459, + 0.014164834006805904, + 0.014537416005623527, + 0.015375000002677552, + 0.01559020800050348, + 0.016130874995724298, + 0.016270250009256415, + 0.01759208300791215, + 0.017707624996546656 + ], + [ + 0.012396459002047777, + 0.012544083001557738, + 0.01284454099368304, + 0.01386962499236688, + 0.013437292000162415, + 0.013785457995254546, + 0.013933916998212226, + 0.014076499996008351, + 0.014801291006733663, + 0.014413458004128188, + 0.01454441700479947, + 0.015133708002394997, + 0.015151583007536829, + 0.015975375004927628, + 0.015093708003405482, + 0.01528087499900721, + 0.015607000008458272, + 0.015422374999616295, + 0.0154528749990277, + 0.015507666990743019, + 0.015811416000360623, + 0.015525834009167738, + 0.015927833010209724, + 0.0159365839936072, + 0.015865499997744337, + 0.016380500004743226, + 0.01678037500823848, + 0.015926207997836173, + 0.01583345900871791, + 0.016011250001611188, + 0.016031834005843848, + 0.016078583998023532, + 0.01593249999859836, + 0.016367749994969927, + 0.01782462499977555, + 0.015964917009114288, + 0.016042625007685274, + 0.016423041990492493, + 0.01643958299246151, + 0.01585645899467636, + 0.016556458998820744, + 0.016079624998383224, + 0.016467166002257727, + 0.01605337498767767, + 0.015914000003249384, + 0.015966833001584746, + 0.015951124994899146, + 0.015428458005771972, + 0.015844624998862855, + 0.015629250003257766, + 0.015382666999357753, + 0.015826957998797297, + 0.015838917010114528, + 0.015465499993297271, + 0.015957374998833984, + 0.015787665994139388, + 0.01637620900874026, + 0.01442608400247991, + 0.015519541993853636, + 0.015944333004881628, + 0.01629737499752082, + 0.016498750002938323, + 0.01691312500042841, + 0.016914749998250045 + ], + [ + 0.007219916995381936, + 0.007036584007437341, + 0.006980999998631887, + 0.0071807089989306405, + 0.006959500009543262, + 0.007417000000714324, + 0.006966375003685243, + 0.007541458995547146, + 0.006894500009366311, + 0.0070732500025769696, + 0.006975583004532382, + 0.007111500002793036, + 0.007195708996732719, + 0.007065209007123485, + 0.007215249992441386, + 0.007236290999571793, + 0.007274666000739671, + 0.007307084000785835, + 0.007337041010032408, + 0.007516917001339607, + 0.007380374998319894, + 0.007365083001786843, + 0.007483958004740998, + 0.007624541991390288, + 0.007470625001587905, + 0.007939207993331365, + 0.007314041999052279, + 0.007799917002557777, + 0.00785179200465791, + 0.007904583995696157, + 0.00796312499733176, + 0.007842874998459592, + 0.007998416011105292, + 0.008113708012388088, + 0.008226624995586462, + 0.008211958993342705, + 0.008122583996737376, + 0.008100541002932005, + 0.008578207998652942, + 0.008291125006508082, + 0.008210332991438918, + 0.008403125000768341, + 0.009052458000951447, + 0.008452334004687145, + 0.007763999994494952, + 0.00818395899841562, + 0.009001541999168694, + 0.008725708001293242, + 0.010116208010003902, + 0.01094208299764432, + 0.011968250008067116, + 0.014951916004065424, + 0.016026625002268702, + 0.01721200000611134, + 0.017605707995244302, + 0.014532875007716939, + 0.012139750004280359, + 0.011466583993751556, + 0.009556750010233372, + 0.009843874999205582, + 0.009136166001553647, + 0.00921383399690967, + 0.008860708010615781, + 0.0100435830099741 + ], + [ + 0.007048082989058457, + 0.006940749997738749, + 0.007045749996905215, + 0.007190041986177675, + 0.007004041995969601, + 0.0075501250103116035, + 0.007293959002709016, + 0.0076960419974057, + 0.007152959005907178, + 0.00743545799923595, + 0.007245875007356517, + 0.007247040994116105, + 0.007358957998803817, + 0.007291208996321075, + 0.007641499993042089, + 0.007282167003722861, + 0.007551415998023003, + 0.00755866699910257, + 0.007639584000571631, + 0.007232291012769565, + 0.007541833998402581, + 0.007841458995244466, + 0.0076825419964734465, + 0.008105708009679802, + 0.007864332990720868, + 0.008336875005625188, + 0.007820124999852851, + 0.008410708003793843, + 0.008007625001482666, + 0.008356457998161204, + 0.008090292001725174, + 0.00843400000303518, + 0.008017083004233427, + 0.008676333003677428, + 0.008134083996992558, + 0.008819124996080063, + 0.008244750002631918, + 0.009418084009666927, + 0.00838891600142233, + 0.008831166996969841, + 0.00855158299964387, + 0.009101957999519072, + 0.008805249992292374, + 0.009349041996756569, + 0.008989541995106265, + 0.009410583996213973, + 0.009348166000563651, + 0.00916187500115484, + 0.009584083003574051, + 0.00887420799699612, + 0.009397791000083089, + 0.009525707995635457, + 0.009511333992122672, + 0.009613207992515527, + 0.009574415991664864, + 0.009560916005284525, + 0.009565209009451792, + 0.009232417010935023, + 0.009235000004991889, + 0.00935570799629204, + 0.009755749997566454, + 0.009822749998420477, + 0.009954541994375177, + 0.009807042006286792 + ], + [ + 0.006791166990296915, + 0.007037625007797033, + 0.006950334005523473, + 0.007001874997513369, + 0.007694874992012046, + 0.007221499996376224, + 0.007692915998632088, + 0.007783542008837685, + 0.007520582992583513, + 0.008118208002997562, + 0.00743866601260379, + 0.007927166996523738, + 0.008034791011596099, + 0.008298207991174422, + 0.007890749999205582, + 0.008634833997348323, + 0.008288999990327284, + 0.008309790995554067, + 0.008477249997667968, + 0.008576416003052145, + 0.008497916991473176, + 0.009118415997363627, + 0.009374917004606687, + 0.0094799999933457, + 0.009712415994727053, + 0.009306874999310821, + 0.010235249996185303, + 0.009446082985959947, + 0.010620125001878478, + 0.009853415991528891, + 0.009882874990580603, + 0.010499625001102686, + 0.009716166008729488, + 0.010494125002878718, + 0.009821582993026823, + 0.0109923329873709, + 0.010322917005396448, + 0.011063917001592927, + 0.010857000001124106, + 0.01090233400464058, + 0.010711750001064502, + 0.01129741600016132, + 0.011916666990146041, + 0.012082624991307966, + 0.011512541997944936, + 0.011858709010994062, + 0.011655709007754922, + 0.011818250000942498, + 0.011953249995713122, + 0.013685415993677452, + 0.011884667008416727, + 0.012244709010701627, + 0.012392709002597257, + 0.012900500005343929, + 0.013307250002981164, + 0.012770667002769187, + 0.013309375004610047, + 0.014174083000398241, + 0.014917166001396254, + 0.014402083004824817, + 0.013573832999099977, + 0.01458141599141527, + 0.01577112499217037, + 0.016738874997827224 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "speedup_cached_vs_uncached_sdpa": 2.162194766895005, + "speedup_cached_vs_uncached_mha": 2.751164882867426 + }, + { + "batch": 8, + "prompt_len": 512, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_mha": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 31.555875000776723, + "p95": 33.13592500053346, + "min": 28.50729199417401, + "max": 33.23349999845959, + "iqr": 3.989292003097944, + "mean": 30.95972500159405, + "cv": 0.07147042849084892 + }, + "decode_ms_per_step": { + "count": 315, + "median": 4.512624989729375, + "p95": 6.979199901979882, + "min": 3.5840830096276477, + "max": 9.531333009363152, + "iqr": 0.7515210018027574, + "mean": 4.870782688487365, + "cv": 0.19807725949994962 + }, + "total_ms": { + "count": 5, + "median": 339.8436678980943, + "p95": 357.4364161962876, + "min": 315.92341703071725, + "max": 361.742666005739, + "iqr": 8.837412970024161, + "mean": 337.8190343762981, + "cv": 0.049158330152537906 + }, + "tokens_per_second": { + "count": 5, + "median": 1506.575076612958, + "p95": 1605.533138663983, + "min": 1415.370781813935, + "max": 1620.6459299920089, + "iqr": 40.13541812653739, + "mean": 1518.5240633992248, + "cv": 0.04895120350481568 + }, + "median_step_ms_by_index": [ + 31.555875000776723, + 7.40633399982471, + 5.2179580088704824, + 4.741041004308499, + 4.847874995903112, + 4.692708011134528, + 4.773125008796342, + 4.707125001004897, + 5.458792002173141, + 4.771291991346516, + 4.540542009635828, + 4.8087499890243635, + 4.689167006290518, + 4.4824999931734055, + 4.700457997387275, + 4.63337499240879, + 4.96870800270699, + 5.001875004381873, + 5.185458998312242, + 4.8467909946339205, + 4.655333003029227, + 4.598999992595054, + 5.561083002248779, + 4.668708992539905, + 4.421917008585297, + 4.688082990469411, + 4.419165998115204, + 4.967166998540051, + 4.4571250036824495, + 4.467000006115995, + 4.476040994632058, + 4.554333005216904, + 4.399709010613151, + 4.39541699597612, + 4.6831660001771525, + 4.718458993011154, + 4.470290994504467, + 4.415958013851196, + 4.7438339970540255, + 4.666584005462937, + 4.2924169974867254, + 4.588625000906177, + 4.163042001891881, + 4.509708000114188, + 4.434166010469198, + 4.274542006896809, + 4.232208986650221, + 4.721542005427182, + 4.591749995597638, + 4.447833998710848, + 4.087374996743165, + 4.526125005213544, + 4.159749994869344, + 4.494541994063184, + 4.272209000191651, + 4.187415994238108, + 4.315833997679874, + 4.381791004561819, + 4.35970799298957, + 4.404916995554231, + 4.2924589943140745, + 4.31225000647828, + 4.372082999907434, + 4.1813749994616956 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.028756333005730994, + 0.007790542003931478, + 0.006428291992051527, + 0.006035375001374632, + 0.005784583991044201, + 0.006264290990657173, + 0.006065875000786036, + 0.005835083007696085, + 0.0056990830053109676, + 0.005236917000729591, + 0.00516158300160896, + 0.0048087499890243635, + 0.004513084000791423, + 0.0044824999931734055, + 0.004455916001461446, + 0.004327292001107708, + 0.005483749991981313, + 0.004441625002073124, + 0.004341833002399653, + 0.004501750008785166, + 0.004239959002006799, + 0.004504249998717569, + 0.004579625005135313, + 0.004267040989361703, + 0.004319667001254857, + 0.004370541995740496, + 0.004276208012015559, + 0.004967166998540051, + 0.00440458400407806, + 0.004361291998066008, + 0.004476040994632058, + 0.004554333005216904, + 0.0042164170008618385, + 0.004064749999088235, + 0.004236374996253289, + 0.004718458993011154, + 0.004532709004706703, + 0.004338250000728294, + 0.005114250001497567, + 0.005063333999714814, + 0.0042924169974867254, + 0.004588625000906177, + 0.004163042001891881, + 0.004018292005639523, + 0.004316125006880611, + 0.004211749997921288, + 0.004082792002009228, + 0.005027624996728264, + 0.004641458988771774, + 0.004476458008866757, + 0.004873333004070446, + 0.004936582990922034, + 0.004899499996099621, + 0.004790124992723577, + 0.005099124988191761, + 0.0044942920067114756, + 0.004780000002938323, + 0.004945334003423341, + 0.0048390000010840595, + 0.005195292003918439, + 0.004454624999198131, + 0.004943625011947006, + 0.004418707991135307, + 0.004866165996645577 + ], + [ + 0.03323349999845959, + 0.0048464579886058345, + 0.0037934169959044084, + 0.003712917008670047, + 0.0042840839887503535, + 0.0041925410041585565, + 0.003871165987220593, + 0.0038742910110158846, + 0.0042229170067003, + 0.0043771670025307685, + 0.004485458994167857, + 0.004151750006712973, + 0.004689167006290518, + 0.004352542004198767, + 0.00430249999044463, + 0.00463337499240879, + 0.00496870800270699, + 0.005134666993399151, + 0.005722417001379654, + 0.006627833005040884, + 0.00742483300564345, + 0.006798707996495068, + 0.007965332988533191, + 0.0071740419953130186, + 0.007036541996058077, + 0.0068321669969009236, + 0.007147041993448511, + 0.007138625005609356, + 0.006198582996148616, + 0.006044541994924657, + 0.00569916600943543, + 0.005900749994907528, + 0.005372332991100848, + 0.005357375004678033, + 0.00523862500267569, + 0.004967458007740788, + 0.004425666993483901, + 0.004940625003655441, + 0.0047438339970540255, + 0.004739166994113475, + 0.004176040994934738, + 0.004050834002555348, + 0.003790834001847543, + 0.003910416999133304, + 0.0038881660002516583, + 0.004035999998450279, + 0.004077707999385893, + 0.003948124998714775, + 0.00411224999697879, + 0.004458458992303349, + 0.004087374996743165, + 0.004000708999228664, + 0.004051207986776717, + 0.004181749987765215, + 0.004061458006617613, + 0.004187415994238108, + 0.004300790998968296, + 0.004512624989729375, + 0.00435970799298957, + 0.004247582997777499, + 0.004237916989950463, + 0.004115917006856762, + 0.0042467090097488835, + 0.0041813749994616956 + ], + [ + 0.02850729199417401, + 0.00740633399982471, + 0.0052179580088704824, + 0.004741041004308499, + 0.004847874995903112, + 0.004692708011134528, + 0.004773125008796342, + 0.004707125001004897, + 0.005458792002173141, + 0.004771291991346516, + 0.004469999999855645, + 0.004771416992298327, + 0.0046729159948881716, + 0.004475916997762397, + 0.004700457997387275, + 0.0041873339941957965, + 0.004496917012147605, + 0.0045367499988060445, + 0.004407000000355765, + 0.004475749999983236, + 0.004444916005013511, + 0.004430875007528812, + 0.004498958005569875, + 0.00446737500897143, + 0.004375000004074536, + 0.004422792000696063, + 0.004266749994712882, + 0.004568916992866434, + 0.004456042006495409, + 0.004273499987903051, + 0.004270791992894374, + 0.004421207995619625, + 0.004399709010613151, + 0.004370875001768582, + 0.004507124991505407, + 0.004470750005566515, + 0.004470290994504467, + 0.004307959010475315, + 0.004381458988063969, + 0.004440667005837895, + 0.0042567920027067885, + 0.004340125000453554, + 0.004553499995381571, + 0.004509708000114188, + 0.004434166010469198, + 0.008924750000005588, + 0.007103207986801863, + 0.006291541998507455, + 0.007017791009275243, + 0.0040714999922784045, + 0.0036093750095460564, + 0.0037949580000713468, + 0.0037084590003360063, + 0.003619208000600338, + 0.003648957994300872, + 0.0035840830096276477, + 0.0036518330016406253, + 0.003750999996555038, + 0.003696959000080824, + 0.003739166000741534, + 0.003861666002194397, + 0.0036867500020889565, + 0.0036392920010257512, + 0.0038646670000161976 + ], + [ + 0.03155587500077672, + 0.009531333009363152, + 0.006991332993493415, + 0.0069199590070638806, + 0.006700833007926121, + 0.006564042007084936, + 0.006849124998552725, + 0.006574291997822002, + 0.006266292009968311, + 0.005765249996329658, + 0.0054742089996580034, + 0.005037208000430837, + 0.004779875001986511, + 0.004892875003861263, + 0.005550624991883524, + 0.0048276249872287735, + 0.004745041995192878, + 0.005001875004381873, + 0.005185458998312242, + 0.0048467909946339205, + 0.004655333003029227, + 0.004598999992595054, + 0.005561083002248779, + 0.004668708992539905, + 0.004421917008585297, + 0.004688082990469411, + 0.004419165998115204, + 0.004458666997379623, + 0.0044571250036824495, + 0.004467000006115995, + 0.004401791011332534, + 0.004291290999390185, + 0.0041917909984476864, + 0.00439541699597612, + 0.0046831660001771525, + 0.004236166991177015, + 0.0044387090019881725, + 0.004415958013851196, + 0.0045939589908812195, + 0.004555541003355756, + 0.004535207990556955, + 0.004641166000510566, + 0.004129500011913478, + 0.004600540996761993, + 0.004711832996690646, + 0.004274542006896809, + 0.004232208986650221, + 0.004431582987308502, + 0.004591749995597638, + 0.0043184999958612025, + 0.004079959006048739, + 0.004583417001413181, + 0.004159749994869344, + 0.005471791999298148, + 0.004272209000191651, + 0.004169291991274804, + 0.004315833997679874, + 0.0043312909983797, + 0.004230291000567377, + 0.004404916995554231, + 0.0042924589943140745, + 0.00431225000647828, + 0.004452542008948512, + 0.004008790987427346 + ], + [ + 0.03274562500882894, + 0.004296708997571841, + 0.004198791008093394, + 0.004336583006079309, + 0.004223042007652111, + 0.004007582989288494, + 0.0042577080021146685, + 0.004530999998678453, + 0.0039875419897725806, + 0.004275166997103952, + 0.0045405420096358284, + 0.00494645799335558, + 0.005139374989084899, + 0.005272708003758453, + 0.007322165998630226, + 0.006902125009219162, + 0.006931249998160638, + 0.006816624998464249, + 0.007526415996835567, + 0.006738957992638461, + 0.006852958002127707, + 0.006977750002988614, + 0.006982582999626175, + 0.006667207999271341, + 0.006579374996363185, + 0.0065110839932458475, + 0.006322250002995133, + 0.005772000004071742, + 0.005921249990933575, + 0.0052509579982142895, + 0.006633166995015927, + 0.0059341250016586855, + 0.005493500008014962, + 0.005841833000886254, + 0.005748000010498799, + 0.004843834001803771, + 0.004903583001578227, + 0.004928041991661303, + 0.004785958008142188, + 0.004666584005462937, + 0.004677374992752448, + 0.004699541997979395, + 0.004685791995143518, + 0.004678249999415129, + 0.0055851659999461845, + 0.004424250000738539, + 0.004450833992450498, + 0.004721542005427182, + 0.004486999998334795, + 0.004447833998710848, + 0.0047557500074617565, + 0.004526125005213544, + 0.004548416996840388, + 0.004494541994063184, + 0.004370624999864958, + 0.004427500010933727, + 0.004520667003816925, + 0.004381791004561819, + 0.005566624997300096, + 0.004678790995967574, + 0.004746250007883646, + 0.004411332993186079, + 0.004372082999907434, + 0.004472166998311877 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + ] + } + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 27.388916991185397, + "p95": 29.577424994204193, + "min": 24.755374994128942, + "max": 29.758374992525205, + "iqr": 2.540875007980503, + "mean": 27.413808394339867, + "cv": 0.07261379323165014 + }, + "decode_ms_per_step": { + "count": 315, + "median": 32.11979100888129, + "p95": 36.690333001024555, + "min": 24.891584005672485, + "max": 38.865333001012914, + "iqr": 4.056271005538292, + "mean": 32.164559647531455, + "cv": 0.08830941404969957 + }, + "total_ms": { + "count": 5, + "median": 2054.856922055478, + "p95": 2158.908798993798, + "min": 1976.0839959199075, + "max": 2180.51966499479, + "iqr": 87.48592200572602, + "mean": 2053.7810661888216, + "cv": 0.04013913582193563 + }, + "tokens_per_second": { + "count": 5, + "median": 249.16576648453227, + "p95": 258.86607517224127, + "min": 234.80641253525374, + "max": 259.09829797576674, + "iqr": 10.888419688934903, + "mean": 249.6112850445793, + "cv": 0.03931789152637387 + }, + "median_step_ms_by_index": [ + 27.388916991185397, + 28.093208005884662, + 28.40258300420828, + 29.08074999868404, + 28.844833999755792, + 29.10866700403858, + 30.172750004567206, + 29.83558300184086, + 29.92529199400451, + 30.309499998111278, + 29.970582996611483, + 30.806583003140986, + 29.621708003105596, + 30.038624987355433, + 29.482500001904555, + 28.04491600545589, + 28.701499992166646, + 30.171207996318117, + 29.70741700846702, + 29.770499997539446, + 30.145374999847263, + 31.170749993179925, + 31.326333002652973, + 31.431416995474137, + 30.489124997984618, + 30.631000001449138, + 29.998124999110587, + 31.092542005353607, + 31.278125010430813, + 31.070249999174848, + 30.989708000561222, + 31.574042004649527, + 31.714416996692307, + 32.457916997373104, + 33.64470800443087, + 33.309999998891726, + 34.55424999992829, + 34.273625002242625, + 33.6533329973463, + 33.55658300279174, + 32.68266600207426, + 33.50283399049658, + 33.85295800399035, + 33.63987499324139, + 34.36641600274015, + 35.43425000680145, + 33.227374995476566, + 33.794875009334646, + 33.32541600684635, + 34.40616698935628, + 33.63179200096056, + 34.00033300567884, + 33.58979200129397, + 34.48808399843983, + 35.18191599869169, + 33.313584004645236, + 34.21145900210831, + 34.331792005104944, + 34.22166699601803, + 33.64041699387599, + 34.76754200528376, + 33.87629200005904, + 34.21974999946542, + 33.70412500225939 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.028853625000920147, + 0.029496417002519593, + 0.02992637499119155, + 0.029911125006037764, + 0.030084625002928078, + 0.030273458003648557, + 0.032520082997507416, + 0.03151312500995118, + 0.03159212500031572, + 0.03253199999744538, + 0.03251224999257829, + 0.03292304198839702, + 0.03268374998879153, + 0.033961999986786395, + 0.0334946660004789, + 0.033681250002700835, + 0.03440633398713544, + 0.03471850001369603, + 0.03186529199592769, + 0.032539916996029206, + 0.032644374994561076, + 0.03259683299984317, + 0.032985541998641565, + 0.033965208000154234, + 0.03362341599131469, + 0.036670832996605895, + 0.035967292002169415, + 0.03576554199389648, + 0.03627429199696053, + 0.03601095799240284, + 0.03241699999489356, + 0.03318362499703653, + 0.033591957995668054, + 0.03616320800210815, + 0.038378375000320375, + 0.03515341700403951, + 0.03455424999992829, + 0.035345790995052084, + 0.0355392089986708, + 0.03518695900856983, + 0.03678954100178089, + 0.038865333001012914, + 0.03653066600963939, + 0.03421766600513365, + 0.03649691700411495, + 0.03543425000680145, + 0.03626333299325779, + 0.035746959008974954, + 0.03332541600684635, + 0.03306558300391771, + 0.033040833004633896, + 0.03400033300567884, + 0.03358979200129397, + 0.034488083998439834, + 0.03518191599869169, + 0.033313584004645236, + 0.03421145900210831, + 0.034618040997884236, + 0.0355837909883121, + 0.037318166010663845, + 0.035345041993423365, + 0.03674441701150499, + 0.03544045900343917, + 0.035400042004766874 + ], + [ + 0.027388916991185397, + 0.028093208005884662, + 0.02840258300420828, + 0.02908074999868404, + 0.028844833999755792, + 0.02910866700403858, + 0.03126787500514183, + 0.02983558300184086, + 0.02992529199400451, + 0.030309499998111278, + 0.029970582996611483, + 0.030896583004505374, + 0.030354000002262183, + 0.03148437499476131, + 0.028330332992482, + 0.02804491600545589, + 0.028701499992166646, + 0.030171207996318117, + 0.03094649998820387, + 0.029770499997539446, + 0.030145374999847263, + 0.031170749993179925, + 0.030041666992474347, + 0.03143141699547414, + 0.029771208006422967, + 0.030631000001449138, + 0.029547334008384496, + 0.031196542011457495, + 0.029954625002574176, + 0.03164504200685769, + 0.029523125005653128, + 0.030692333995830268, + 0.029531499996664934, + 0.031731750001199543, + 0.03049066598759964, + 0.03208708399324678, + 0.031103957997402176, + 0.031101625005248934, + 0.03044387500267476, + 0.03355658300279174, + 0.029642957990290597, + 0.03145774999575224, + 0.03385295800399035, + 0.03207737499906216, + 0.03331574999901932, + 0.0329718330031028, + 0.03141029200924095, + 0.03377212499617599, + 0.032341749989427626, + 0.03467750000709202, + 0.03390266600763425, + 0.032148874990525655, + 0.03218754200497642, + 0.03344391599239316, + 0.032065583000076, + 0.031942540997988544, + 0.031698958002380095, + 0.03197224999894388, + 0.03133187499770429, + 0.03225499999825843, + 0.03476754200528376, + 0.03163991600740701, + 0.03211979100888129, + 0.03125749999890104 + ], + [ + 0.024755374994128942, + 0.024891584005672485, + 0.02601104200584814, + 0.025992666007368825, + 0.026909874999546446, + 0.02601600000343751, + 0.02656112500699237, + 0.025880250002956018, + 0.025278791988966987, + 0.025683999992907047, + 0.02582987499772571, + 0.02676537500519771, + 0.026389624996227212, + 0.026948250000714324, + 0.028796250000596046, + 0.02772654200089164, + 0.0275499589915853, + 0.028561041995999403, + 0.029637874991749413, + 0.029463958999258466, + 0.031051041994942352, + 0.03283462500257883, + 0.031882291004876606, + 0.031506875006016344, + 0.02975516598962713, + 0.030008583009475842, + 0.029998124999110587, + 0.030736917004105635, + 0.030376708004041575, + 0.031044292001752183, + 0.030989708000561222, + 0.03157404200464953, + 0.03171441699669231, + 0.032457916997373104, + 0.03270808300294448, + 0.033309999998891726, + 0.03460774999985006, + 0.034273625002242625, + 0.03420824999921024, + 0.03520658299385104, + 0.03528816699690651, + 0.036101500008953735, + 0.03635879200010095, + 0.036086584004806355, + 0.036569917007000186, + 0.037668582997866906, + 0.038124208003864624, + 0.03631920900079422, + 0.03622399999585468, + 0.037163542001508176, + 0.036854209000011906, + 0.03756108299421612, + 0.036539291992085055, + 0.03536829199583735, + 0.036807833006605506, + 0.03843129200686235, + 0.037015750000136904, + 0.036386208012118004, + 0.036556333012413234, + 0.036441917007323354, + 0.03618691700103227, + 0.03673583301133476, + 0.035741667001275346, + 0.03643133399600629 + ], + [ + 0.029758374992525205, + 0.029739125006017275, + 0.02992487500887364, + 0.031120665997150354, + 0.03096233299584128, + 0.02974966701003723, + 0.030172750004567206, + 0.031207166000967845, + 0.0322939170000609, + 0.03325483399385121, + 0.03138800000306219, + 0.030806583003140986, + 0.029621708003105596, + 0.029292124992934987, + 0.029482500001904555, + 0.030140000002575107, + 0.029652832992724143, + 0.029816334004863165, + 0.02970741700846702, + 0.03030079198651947, + 0.03012208399013616, + 0.03030600000056438, + 0.030769208999117836, + 0.0306422090070555, + 0.030489124997984618, + 0.030742791990633123, + 0.031204290993628092, + 0.031092542005353607, + 0.03135608400043566, + 0.031064499999047257, + 0.03081275000295136, + 0.031458375000511296, + 0.032031457987613976, + 0.03459487500367686, + 0.035121124994475394, + 0.03673654100566637, + 0.03466358300647698, + 0.037728291004896164, + 0.0336533329973463, + 0.03217649999714922, + 0.03268266600207426, + 0.0322535000013886, + 0.03226212499430403, + 0.03363987499324139, + 0.03436641600274015, + 0.03590308300044853, + 0.033227374995476566, + 0.033794875009334646, + 0.033879833994433284, + 0.03440616698935628, + 0.03363179200096056, + 0.03470029200252611, + 0.035417792008956894, + 0.034640375000890344, + 0.035359041998162866, + 0.03456920900498517, + 0.03616420799517073, + 0.034331792005104944, + 0.03332800000498537, + 0.033235625000088476, + 0.03447612500167452, + 0.03387629200005904, + 0.033548583000083454, + 0.03361262498947326 + ], + [ + 0.026312749992939644, + 0.026781666005263105, + 0.026780207990668714, + 0.02715383299801033, + 0.02657862499472685, + 0.026129000005312264, + 0.0270037079899339, + 0.028512999997474253, + 0.028970999992452562, + 0.028298790988628753, + 0.028558000005432405, + 0.02881287499621976, + 0.02847766700142529, + 0.030038624987355433, + 0.03210974999819882, + 0.026915916008874774, + 0.02761070799897425, + 0.030230790987843648, + 0.02926774999650661, + 0.029655416001332924, + 0.030090874992311, + 0.02948066701355856, + 0.03132633300265297, + 0.030849707996821962, + 0.0319155830075033, + 0.03046395799901802, + 0.028441666989238, + 0.03049187500437256, + 0.03127812501043081, + 0.031070249999174848, + 0.03112104200408794, + 0.032223874994087964, + 0.030022917009773664, + 0.03139962500426918, + 0.03364470800443087, + 0.03274733299622312, + 0.0325286250008503, + 0.032347915999707766, + 0.03336200000194367, + 0.0325486249930691, + 0.03246524999849498, + 0.03350283399049658, + 0.032939000011538155, + 0.03279991699673701, + 0.03259516699472442, + 0.03449316701153293, + 0.03212299999722745, + 0.03210637500160374, + 0.03165279199311044, + 0.032674292000592686, + 0.03357183300249744, + 0.03201545799674932, + 0.03326704200298991, + 0.031335291991126724, + 0.029968540999107063, + 0.03133512499334756, + 0.03180187499674503, + 0.03246449999278411, + 0.03422166699601803, + 0.03364041699387599, + 0.03222916599770542, + 0.03340162499807775, + 0.03421974999946542, + 0.03370412500225939 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_mha": { + "first_token_ms": { + "count": 5, + "median": 35.74041600222699, + "p95": 39.31985000381246, + "min": 28.407166013494134, + "max": 39.32100000383798, + "iqr": 9.425832991837524, + "mean": 34.53464980702847, + "cv": 0.1493021778151898 + }, + "decode_ms_per_step": { + "count": 315, + "median": 38.488292004331015, + "p95": 45.773312397068366, + "min": 28.78658399276901, + "max": 47.1373339969432, + "iqr": 3.02429201110499, + "mean": 38.64571506347847, + "cv": 0.09390419057362043 + }, + "total_ms": { + "count": 5, + "median": 2456.481878005434, + "p95": 2505.4482314008055, + "min": 2442.253867906402, + "max": 2510.9001219825586, + "iqr": 30.84371201111935, + "mean": 2469.2146988061722, + "cv": 0.0112822455107304 + }, + "tokens_per_second": { + "count": 5, + "median": 208.42816085243166, + "p95": 209.46219028127166, + "min": 203.91093835932216, + "max": 209.64241544590405, + "iqr": 2.5923058444498963, + "mean": 207.37435761173842, + "cv": 0.011210935090246472 + }, + "median_step_ms_by_index": [ + 35.74041600222699, + 38.739458002964966, + 36.593625001842156, + 37.06858400255442, + 37.84783399896696, + 37.9999169963412, + 36.348291992908344, + 36.15183400688693, + 36.138666997430846, + 38.42675000487361, + 36.92683299595956, + 38.07662500184961, + 38.80445798859, + 34.710042004007846, + 35.95849999692291, + 37.02216599776875, + 37.38216699275654, + 37.91191699565388, + 36.652750000939704, + 37.71550000237767, + 37.702708010328934, + 37.757750003947876, + 37.684957991587, + 37.817792006535456, + 37.36416700121481, + 37.355499996920116, + 37.34799999801908, + 37.84795799583662, + 37.34495800745208, + 37.992958998074755, + 37.39179199328646, + 36.9483329996001, + 36.59670799970627, + 37.59920800803229, + 37.68570801184978, + 38.35558299033437, + 38.033000004361384, + 38.99266700318549, + 38.41249999823049, + 37.989750009728596, + 37.786041997605935, + 38.51041600864846, + 38.932958006625995, + 39.174833000288345, + 39.53625001304317, + 40.200041999924, + 40.91675000381656, + 40.000791996135376, + 40.16950000368524, + 40.79279200232122, + 42.68858399882447, + 40.80670900293626, + 42.02412499580532, + 42.156916999374516, + 41.3833339989651, + 41.17729200515896, + 41.651709005236626, + 41.48499999428168, + 41.370541002834216, + 42.56270799669437, + 42.02516601071693, + 43.34795799513813, + 42.70341600931715, + 42.84920899954159 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.03931525000371039, + 0.03932524999254383, + 0.03784612500749063, + 0.03957412499585189, + 0.03784783399896696, + 0.0379999169963412, + 0.03817650000564754, + 0.03871974999492522, + 0.0389554579887772, + 0.03842675000487361, + 0.03916816599667072, + 0.03970724999089725, + 0.03880445798859, + 0.03920687500794884, + 0.04088287500781007, + 0.03846245800377801, + 0.03738216699275654, + 0.03835875001095701, + 0.03976008300378453, + 0.03982766599801835, + 0.037959708002745174, + 0.039433208003174514, + 0.03819170800852589, + 0.039114790997700766, + 0.038004332993295975, + 0.03941920799843501, + 0.03856412500317674, + 0.039369582998915575, + 0.04024370800470933, + 0.039359125003102235, + 0.03811774999485351, + 0.04139670799486339, + 0.03659670799970627, + 0.03759920800803229, + 0.03731970800436102, + 0.03835558299033437, + 0.0389465000043856, + 0.03905754200241063, + 0.03570041700731963, + 0.0361229169939179, + 0.037401666995720007, + 0.03797387499071192, + 0.0395929160004016, + 0.04054816700227093, + 0.038529999990714714, + 0.041624041987233795, + 0.03770420800719876, + 0.03855016699526459, + 0.04016950000368524, + 0.04036562499823049, + 0.04268858399882447, + 0.038059625003370456, + 0.039356540990411304, + 0.040895458005252294, + 0.0413833339989651, + 0.04117729200515896, + 0.041651709005236626, + 0.04148499999428168, + 0.0407848750037374, + 0.041422708993195556, + 0.04003108300094027, + 0.0396631250041537, + 0.04037116600375157, + 0.04284920899954159 + ], + [ + 0.028407166013494134, + 0.02878658399276901, + 0.02989016599894967, + 0.03057874999649357, + 0.03279729199130088, + 0.032443040996440686, + 0.03209183299622964, + 0.03158941700530704, + 0.03255333300330676, + 0.04365316600888036, + 0.03272829198976979, + 0.029450582995195873, + 0.03008224999939557, + 0.030149124999297783, + 0.03035854100016877, + 0.031068958996911533, + 0.03214808300253935, + 0.030427083998802118, + 0.030872500006807968, + 0.030889874993590638, + 0.03149858399410732, + 0.03170095800305717, + 0.031709041009889916, + 0.032905625004787, + 0.03403629099193495, + 0.03499629200086929, + 0.036234624989447184, + 0.034289542003534734, + 0.034067625005263835, + 0.034105707993148826, + 0.03500008399714716, + 0.035131582990288734, + 0.035564874997362494, + 0.03720466599042993, + 0.03720579198852647, + 0.03746508399490267, + 0.038521208000020124, + 0.03936966600304004, + 0.03978183299477678, + 0.04138545799651183, + 0.04159091600740794, + 0.04311620800581295, + 0.044112832998507656, + 0.04547233298944775, + 0.045262291008839384, + 0.04551424999954179, + 0.04665733300498687, + 0.04684687498956919, + 0.04631633299868554, + 0.04627433299901895, + 0.04590891698899213, + 0.0469897500006482, + 0.04712649999419227, + 0.046159874997101724, + 0.0471373339969432, + 0.046078499988652766, + 0.04673008300596848, + 0.04701458400813863, + 0.045874082992668264, + 0.04587620800884906, + 0.045730124998954125, + 0.04704795799625572, + 0.045618165997439064, + 0.04465749999508262 + ], + [ + 0.03932100000383798, + 0.038739458002964966, + 0.03903008300403599, + 0.039592209010152146, + 0.03793412499362603, + 0.03887699999904726, + 0.039164417001302354, + 0.03762520900636446, + 0.039575749993673526, + 0.03846745799819473, + 0.037838709002244286, + 0.03849150000314694, + 0.0388827080023475, + 0.03754762500466313, + 0.03757266599859577, + 0.038304000001517124, + 0.03802258300129324, + 0.03791191699565388, + 0.037721208995208144, + 0.03771550000237767, + 0.037934791995212436, + 0.037590332998661324, + 0.037684957991587, + 0.037817792006535456, + 0.03736416700121481, + 0.037355499996920116, + 0.03734799999801908, + 0.037958332992275245, + 0.03734495800745208, + 0.0380181249929592, + 0.03739179199328646, + 0.037273624999215826, + 0.0380754590005381, + 0.03780604200437665, + 0.03810137500113342, + 0.03888383400044404, + 0.038033000004361384, + 0.038027832997613586, + 0.0381257090048166, + 0.037989750009728596, + 0.037786041997605935, + 0.03833366700564511, + 0.0376666660013143, + 0.03828462499950547, + 0.03910308299236931, + 0.039385458003380336, + 0.038937999997870065, + 0.03840137500083074, + 0.038151708999066614, + 0.038542916998267174, + 0.03842104200157337, + 0.0385010419995524, + 0.038838917011162266, + 0.03856687500956468, + 0.038980041994364, + 0.0392002079897793, + 0.038948083005379885, + 0.03889520799566526, + 0.0394297079910757, + 0.039380791000439785, + 0.039561375000630505, + 0.03975250000075903, + 0.03924041699792724, + 0.03971162499510683 + ], + [ + 0.03574041600222699, + 0.03924508299678564, + 0.036593625001842156, + 0.03706858400255442, + 0.036482625000644475, + 0.039699333006865345, + 0.036348291992908344, + 0.03615183400688693, + 0.036138666997430846, + 0.036755000008270144, + 0.03692683299595956, + 0.03701933300180826, + 0.03892704100871924, + 0.034710042004007846, + 0.03595849999692291, + 0.03702216599776875, + 0.03738241699466016, + 0.03673958400031552, + 0.036652750000939704, + 0.03789491699717473, + 0.037702708010328934, + 0.037757750003947876, + 0.03725045900500845, + 0.03784816699044313, + 0.03831641700526234, + 0.0368980829953216, + 0.039305083992076106, + 0.03784795799583662, + 0.040766833000816405, + 0.037992958998074755, + 0.03834654200181831, + 0.0369483329996001, + 0.03579725000599865, + 0.03615658399940003, + 0.03768570801184978, + 0.0376620000024559, + 0.03784375000395812, + 0.03899266700318549, + 0.039239584002643824, + 0.03933858301024884, + 0.039281000004848465, + 0.03960375000315253, + 0.038932958006625995, + 0.039174833000288345, + 0.039594125002622604, + 0.040200041999924, + 0.04091675000381656, + 0.040000791996135376, + 0.03946204201201908, + 0.04079279200232122, + 0.04092270799446851, + 0.04080670900293626, + 0.04202412499580532, + 0.042156916999374516, + 0.03979904200241435, + 0.041066458012210205, + 0.0407066249899799, + 0.04064091600594111, + 0.041370541002834216, + 0.04256270799669437, + 0.04202516601071693, + 0.04334795799513813, + 0.04630516699398868, + 0.042792083986569196 + ], + [ + 0.029889417011872865, + 0.030542458000127226, + 0.032113624998601153, + 0.03264991700416431, + 0.04055233400140423, + 0.03355520799232181, + 0.03099741699406877, + 0.031240083000739105, + 0.03196545799437445, + 0.03350983400014229, + 0.03504691700800322, + 0.03807662500184961, + 0.033813959002145566, + 0.034105375001672655, + 0.03483191700070165, + 0.03544220799813047, + 0.03718612500233576, + 0.03864445899671409, + 0.03578754101181403, + 0.03635566700540949, + 0.037251041998388246, + 0.03794329198717605, + 0.04061116700177081, + 0.03686624999681953, + 0.037061708004330285, + 0.04119749998790212, + 0.036568249997799285, + 0.03696416600723751, + 0.03684149999753572, + 0.03686974999436643, + 0.0359260830009589, + 0.03587295800389256, + 0.03719049999199342, + 0.038488292004331015, + 0.03904200000397395, + 0.03886412500287406, + 0.03748533299949486, + 0.03679304198885802, + 0.03841249999823049, + 0.03714441700140014, + 0.037546999999904074, + 0.03851041600864846, + 0.03852979101066012, + 0.03899041700060479, + 0.03953625001304317, + 0.039823832994443364, + 0.041328166000312194, + 0.04081795799720567, + 0.041744957998162135, + 0.042228499994962476, + 0.04288829199504107, + 0.04245895901112817, + 0.04496029201254714, + 0.04265708300226834, + 0.04403525000088848, + 0.044572541999514215, + 0.04305279100663029, + 0.04512312500446569, + 0.04419945900735911, + 0.04399783299595583, + 0.04523958300705999, + 0.04461579099006485, + 0.04270341600931715, + 0.0435348330065608 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "speedup_cached_vs_uncached_sdpa": 6.046477001512498, + "speedup_cached_vs_uncached_mha": 7.228270260848396 + }, + { + "batch": 8, + "prompt_len": 1024, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_mha": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 66.45250000292435, + "p95": 72.94707499095239, + "min": 61.58579200564418, + "max": 74.3722499901196, + "iqr": 4.0680409874767065, + "mean": 66.5670501999557, + "cv": 0.07424041081774262 + }, + "decode_ms_per_step": { + "count": 315, + "median": 5.6098339991876855, + "p95": 8.207691098505167, + "min": 4.46629201178439, + "max": 12.485416009440087, + "iqr": 1.279750002140645, + "mean": 6.063945479469285, + "cv": 0.19389448532742032 + }, + "total_ms": { + "count": 5, + "median": 446.5818759490503, + "p95": 470.0057456007926, + "min": 429.5572890114272, + "max": 471.7313699948136, + "iqr": 31.098953972104937, + "mean": 448.59561540652066, + "cv": 0.041517818758935024 + }, + "tokens_per_second": { + "count": 5, + "median": 1146.4862941692984, + "p95": 1190.5745473853017, + "min": 1085.363477111198, + "max": 1191.924833072451, + "iqr": 79.58841428335904, + "mean": 1142.9065998685992, + "cv": 0.04128624917408886 + }, + "median_step_ms_by_index": [ + 66.45250000292435, + 10.910750002949499, + 8.454625000013039, + 7.896958006313071, + 8.134458010317758, + 7.961666997289285, + 7.33695799135603, + 7.462374996975996, + 7.1975830069277436, + 7.656165995285846, + 7.249540998600423, + 7.792916992912069, + 7.282417005626485, + 6.944374996237457, + 6.51283300248906, + 6.217582995304838, + 5.972915998427197, + 5.819625002914108, + 5.567749991314486, + 5.934040993452072, + 5.5775829969206825, + 5.40066699613817, + 5.665208009304479, + 5.60141600726638, + 5.630333005683497, + 5.623332996037789, + 5.951792001724243, + 5.895624999538995, + 5.893583002034575, + 5.543999999645166, + 6.24029099708423, + 6.673540992778726, + 6.413999988581054, + 6.189542007632554, + 6.378125006449409, + 5.521374987438321, + 5.9474999870872125, + 5.64891699468717, + 5.866999999852851, + 5.883000005269423, + 5.502790998434648, + 5.7982089929282665, + 5.927542006247677, + 5.437749990960583, + 5.870665991096757, + 5.325291000190191, + 6.250291989999823, + 5.90237500728108, + 5.422792004537769, + 5.531792005058378, + 6.081040992285125, + 5.455665988847613, + 5.19900000654161, + 5.4836659983266145, + 5.321000004187226, + 5.358833994250745, + 5.43691600614693, + 5.223333006142639, + 5.409917008364573, + 5.5406669998774305, + 5.348125007003546, + 5.521083003259264, + 5.523166008060798, + 5.388709003455006 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.06158579200564418, + 0.005321083008311689, + 0.00513150000188034, + 0.00461708300281316, + 0.004846208001254126, + 0.004807416000403464, + 0.00470662499719765, + 0.004747749990201555, + 0.004889959003776312, + 0.0048245000070892274, + 0.00446629201178439, + 0.005001583995181136, + 0.0047447079996345565, + 0.004636625002603978, + 0.004810291007743217, + 0.005077540990896523, + 0.0046495419956045225, + 0.004865499999141321, + 0.005250500005786307, + 0.005017584000597708, + 0.005013959002099, + 0.00540066699613817, + 0.005471166005008854, + 0.0053488749981625006, + 0.005630333005683497, + 0.0053200830006971955, + 0.005951792001724243, + 0.006274249986745417, + 0.007981208007549867, + 0.008427833003224805, + 0.008131333001074381, + 0.0081491669989191, + 0.00785870899562724, + 0.0072960410034283996, + 0.007132166996598244, + 0.007464916998287663, + 0.006880667002405971, + 0.006917542006704025, + 0.0067313750041648746, + 0.006387667002854869, + 0.005959041998721659, + 0.006904958994709887, + 0.007369375001871958, + 0.0072145840094890445, + 0.006601874993066303, + 0.006670917005976662, + 0.006250291989999823, + 0.006524124997667968, + 0.006877500010887161, + 0.005884541998966597, + 0.006095542004914023, + 0.0059965830005239695, + 0.005480291991261765, + 0.005447625007946044, + 0.005578958996920846, + 0.005472582997754216, + 0.00543691600614693, + 0.005575416987994686, + 0.005409917008364573, + 0.005510207993211225, + 0.005450875003589317, + 0.005521083003259264, + 0.005488916009198874, + 0.005514332995517179 + ], + [ + 0.06317833400680684, + 0.005243374995188788, + 0.005004833001294173, + 0.0049149999977089465, + 0.004882875000475906, + 0.004829500001505949, + 0.00498479200177826, + 0.005035000009229407, + 0.004866750008659437, + 0.004822999995667487, + 0.004821916998480447, + 0.004790542006958276, + 0.004935290999128483, + 0.005039249997935258, + 0.0046940409956732765, + 0.005510499991942197, + 0.004768625003634952, + 0.004919958999380469, + 0.0050386249931761995, + 0.004894375000731088, + 0.005186333000892773, + 0.005113167004310526, + 0.005206041008932516, + 0.00560141600726638, + 0.00538720800250303, + 0.0053844999929424375, + 0.005526624998310581, + 0.005448583004181273, + 0.0057786249963101, + 0.005895541995414533, + 0.00624029099708423, + 0.008072374999755993, + 0.008277458007796668, + 0.0088675000006333, + 0.008475416005239822, + 0.007332332999794744, + 0.008177790994523093, + 0.006902957989950664, + 0.007364874996710569, + 0.006456500006606802, + 0.006252374994801357, + 0.006442167010391131, + 0.006502958000055514, + 0.006373666998115368, + 0.006696707991068251, + 0.005915583999012597, + 0.006286917006946169, + 0.006398375000571832, + 0.006298249994870275, + 0.005794041993794963, + 0.006081040992285125, + 0.0058202499931212515, + 0.0059354160039220005, + 0.005887249993975274, + 0.005277000003843568, + 0.0055745000136084855, + 0.0056098339991876855, + 0.005155457998625934, + 0.005692624996299855, + 0.005550417001359165, + 0.005176000006031245, + 0.0057850000011967495, + 0.0057551249919924885, + 0.005424209011835046 + ], + [ + 0.06645250000292435, + 0.010910750002949499, + 0.008454625000013039, + 0.00789695800631307, + 0.008136167001794092, + 0.007961666997289285, + 0.00733695799135603, + 0.007462374996975996, + 0.0076233339932514355, + 0.007887374988058582, + 0.007832709001377225, + 0.008288750002975576, + 0.007704666000790894, + 0.006944374996237457, + 0.006850874997326173, + 0.006395709002390504, + 0.005972915998427197, + 0.005819625002914108, + 0.005567749991314486, + 0.005934040993452072, + 0.0055775829969206825, + 0.0053882500069448724, + 0.005936333996942267, + 0.005564624996623024, + 0.005424417002359405, + 0.005971834005322307, + 0.005612374996417202, + 0.006090249997214414, + 0.005893583002034575, + 0.005230583003140055, + 0.005376582994358614, + 0.005539666992262937, + 0.005226291003054939, + 0.005326458995114081, + 0.005392083010519855, + 0.005225000000791624, + 0.005191292002564296, + 0.005355249988497235, + 0.0052753329946426675, + 0.005146957992110401, + 0.005209082999499515, + 0.005469292009365745, + 0.005388165998738259, + 0.005437749990960583, + 0.0054123339941725135, + 0.005325291000190191, + 0.006356541998684406, + 0.00590237500728108, + 0.005209374998230487, + 0.005416957996203564, + 0.005279792007058859, + 0.005432042002212256, + 0.00505545899795834, + 0.0054836659983266145, + 0.005449332995340228, + 0.005358833994250745, + 0.005073707987321541, + 0.005223333006142639, + 0.005028958010370843, + 0.0055406669998774305, + 0.005348125007003546, + 0.005062583994003944, + 0.005550625006435439, + 0.005388709003455006 + ], + [ + 0.06724637499428354, + 0.011332459005643614, + 0.008629957999801263, + 0.008353791999979876, + 0.008134458010317758, + 0.008166458006598987, + 0.008155415998771787, + 0.008060749998549, + 0.007678708992898464, + 0.0079846250009723, + 0.007728665994363837, + 0.007792916992912069, + 0.007668374993954785, + 0.008035374994506128, + 0.007307708001462743, + 0.00764987499860581, + 0.0070491249935003, + 0.008089584007393569, + 0.00739770800282713, + 0.0071193329931702465, + 0.006926166010089219, + 0.005960666996543296, + 0.005665208009304479, + 0.006966832996113226, + 0.0068089169944869354, + 0.005778999999165535, + 0.006022540997946635, + 0.005454083002405241, + 0.006701165999402292, + 0.005543999999645166, + 0.00647791700612288, + 0.006673540992778726, + 0.005447874995297752, + 0.006189542007632554, + 0.006378125006449409, + 0.005204832996241748, + 0.005262791994027793, + 0.005350542007363401, + 0.0051532080105971545, + 0.005317375005688518, + 0.005223125001066364, + 0.005117458000313491, + 0.005927542006247677, + 0.005265333005809225, + 0.005179957996006124, + 0.005126041010953486, + 0.005319417003192939, + 0.004958208010066301, + 0.004841959002078511, + 0.004912208998575807, + 0.005043416997068562, + 0.005455665988847613, + 0.0051477919914759696, + 0.004997291005565785, + 0.0051808749994961545, + 0.005033958004787564, + 0.005174624995561317, + 0.005185458998312242, + 0.005243916995823383, + 0.005221291998168454, + 0.005100708003737964, + 0.005184458990697749, + 0.005209042006754316, + 0.005187500006286427 + ], + [ + 0.0743722499901196, + 0.012485416009440087, + 0.008611666999058798, + 0.008523708005668595, + 0.009423124996828847, + 0.008885416988050565, + 0.0075516670040087774, + 0.007693000006838702, + 0.0071975830069277436, + 0.007656165995285846, + 0.007249540998600423, + 0.008285957999760285, + 0.007282417005626485, + 0.007103583004209213, + 0.00651283300248906, + 0.006217582995304838, + 0.006214875000296161, + 0.006451124994782731, + 0.006101292005041614, + 0.006165041006170213, + 0.006222249998245388, + 0.006863250004244037, + 0.006509208003990352, + 0.005908041013753973, + 0.0058841660065809265, + 0.005623332996037789, + 0.006250083999475464, + 0.005895624999538995, + 0.005791541989310645, + 0.005216290999669582, + 0.005954791995463893, + 0.005976458007353358, + 0.006413999988581054, + 0.006166207997011952, + 0.005843916005687788, + 0.005521374987438321, + 0.0059474999870872125, + 0.00564891699468717, + 0.005866999999852851, + 0.005883000005269423, + 0.005502790998434648, + 0.0057982089929282665, + 0.005234083000686951, + 0.005231999995885417, + 0.005870665991096757, + 0.005264417006401345, + 0.005422625006758608, + 0.0054995839891489595, + 0.005422792004537769, + 0.005531792005058378, + 0.006113458002801053, + 0.005323541990946978, + 0.00519900000654161, + 0.005484291992615908, + 0.005321000004187226, + 0.005106124997837469, + 0.0056897920003393665, + 0.005700749999959953, + 0.006026000002748333, + 0.005797958001494408, + 0.005602332996204495, + 0.005571250003413297, + 0.005523166008060798, + 0.005122542002936825 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + ] + } + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 54.65975000697654, + "p95": 65.81694159540348, + "min": 50.50112499156967, + "max": 66.45137499435805, + "iqr": 11.473916005343199, + "mean": 57.33934999734629, + "cv": 0.1242038404123478 + }, + "decode_ms_per_step": { + "count": 315, + "median": 69.6848340012366, + "p95": 76.77111249795416, + "min": 52.005249992362224, + "max": 85.98879100463819, + "iqr": 4.735583504952956, + "mean": 69.1030470605862, + "cv": 0.07464914432057611 + }, + "total_ms": { + "count": 5, + "median": 4391.672703961376, + "p95": 4481.784632633207, + "min": 4339.00891803205, + "max": 4481.85779202322, + "iqr": 121.36683009157423, + "mean": 4410.831314814277, + "cv": 0.015265116790650644 + }, + "tokens_per_second": { + "count": 5, + "median": 116.58427995742164, + "p95": 117.88500154917584, + "min": 114.23834127697093, + "max": 117.99929653802525, + "iqr": 3.1801557354255863, + "mean": 116.09948104490972, + "cv": 0.015227456168239438 + }, + "median_step_ms_by_index": [ + 54.65975000697654, + 57.62112500087824, + 59.630584000842646, + 58.346082994830795, + 58.48300000070594, + 58.34562500240281, + 61.30924999888521, + 61.33570800011512, + 64.01591699977871, + 64.4013750134036, + 64.03583299834281, + 64.16629100567661, + 64.81204200827051, + 66.34487499832176, + 66.97895799879916, + 67.01470800908282, + 65.3195000049891, + 67.50712500070222, + 69.79916599811986, + 67.95741600217298, + 68.0542920017615, + 70.68041698948946, + 70.50820800941437, + 68.72170900169294, + 69.81270899996161, + 69.61962500645313, + 70.4694589949213, + 71.26066600903869, + 68.81695799529552, + 70.34279199433513, + 70.06312500743661, + 71.52504099940415, + 70.71862500743009, + 70.80308299919125, + 68.6005409952486, + 69.61016599962022, + 70.59787500475068, + 70.07816599798389, + 69.62824999936856, + 71.94237499788869, + 70.3218749986263, + 72.22087500849739, + 69.13845799863338, + 71.96025000303052, + 70.97408300614916, + 72.88045799941756, + 70.84429101087153, + 71.18383300257847, + 69.53758301096968, + 68.94916600140277, + 70.85712499974761, + 72.39358298829757, + 70.93924999935552, + 70.2552089933306, + 70.30116699752398, + 71.38670799031388, + 71.34862498787697, + 73.55733298754785, + 71.81733399920631, + 72.41220799915027, + 71.51337500545196, + 69.69016599759925, + 71.99812500039116, + 70.74041600571945 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.06645137499435805, + 0.06856962500023656, + 0.07171554199885577, + 0.07103058299981058, + 0.0693797920102952, + 0.06860187499842141, + 0.06857283299905248, + 0.06739300000481308, + 0.06758800000534393, + 0.06763045900152065, + 0.07414370799961034, + 0.08196595800109208, + 0.07784699999319855, + 0.07216695901297498, + 0.07585358299547806, + 0.0740700000023935, + 0.07594941700517666, + 0.07350433300598525, + 0.07113341699005105, + 0.0714704999991227, + 0.07356550000258721, + 0.07311275000392925, + 0.07050820800941437, + 0.06872170900169294, + 0.06982695800252259, + 0.07230770800379105, + 0.06845500000054017, + 0.07126066600903869, + 0.06761316698975861, + 0.07034279199433513, + 0.06708058300137054, + 0.0727334580005845, + 0.06785870899329893, + 0.06711379101034254, + 0.06751083300332539, + 0.06879762500466313, + 0.066028124987497, + 0.06381704199884553, + 0.0644484160002321, + 0.06605074999970384, + 0.06973558400932234, + 0.0666667080076877, + 0.06465495898737572, + 0.07196025000303052, + 0.06316591599897947, + 0.06939187500393018, + 0.06660270900465548, + 0.0688659169973107, + 0.06754654100222979, + 0.06913079199148342, + 0.07085712499974761, + 0.07251483399886638, + 0.06795112499094103, + 0.07279037500848062, + 0.06907387499813922, + 0.07129470798827242, + 0.06890949999797158, + 0.06870966700080317, + 0.06831266700464766, + 0.07241220799915027, + 0.07151337500545196, + 0.07105470799433533, + 0.07199812500039116, + 0.074552499994752 + ], + [ + 0.05465975000697654, + 0.05762112500087824, + 0.059630584000842646, + 0.058346082994830795, + 0.05757425000774674, + 0.05834562500240281, + 0.06130924999888521, + 0.06133570800011512, + 0.06438325000635814, + 0.06449433299712837, + 0.06403583299834281, + 0.06416629100567661, + 0.06481204200827051, + 0.06717754200508352, + 0.07264595800370444, + 0.06718420799006708, + 0.06428187500569038, + 0.06750712500070222, + 0.06979916599811986, + 0.07396220900409389, + 0.07348237499536481, + 0.07000900000275578, + 0.07617154100444168, + 0.07135383400600404, + 0.07128016700153239, + 0.06978304201038554, + 0.07057829199766275, + 0.07548333299928345, + 0.0713074169907486, + 0.07275491599284578, + 0.0734693749982398, + 0.07021350000286475, + 0.07071862500743009, + 0.07080308299919125, + 0.06804104099865071, + 0.06853112499811687, + 0.07166629200219177, + 0.06879200000548735, + 0.06962824999936856, + 0.07194237499788869, + 0.06993266599602066, + 0.07335908300592564, + 0.06864945800043643, + 0.07191204201080836, + 0.07152375001169275, + 0.07288045799941756, + 0.07424658301169984, + 0.07118383300257847, + 0.06953758301096968, + 0.06778658399707638, + 0.07573066699842457, + 0.08124245899671223, + 0.08598879100463819, + 0.08428316601202823, + 0.08284420799463987, + 0.07650612499855924, + 0.07134862498787697, + 0.07355733298754785, + 0.077937624999322, + 0.07695775000320282, + 0.07614900000044145, + 0.06856820799293928, + 0.0687184169946704, + 0.07136579100915696 + ], + [ + 0.05050112499156967, + 0.052005249992362224, + 0.056496125005651265, + 0.05605195800308138, + 0.05848300000070594, + 0.05657483299728483, + 0.057259749999502674, + 0.06044379100785591, + 0.058946874996763654, + 0.06053895800141618, + 0.06375975000264589, + 0.059300790991983376, + 0.06076312500226777, + 0.06492833400261588, + 0.06331170799967367, + 0.06701470800908282, + 0.06489229199360125, + 0.06820462500036228, + 0.0705683330015745, + 0.06795741600217298, + 0.06729445898963604, + 0.07068041698948946, + 0.06809449999127537, + 0.06610441701195668, + 0.06981270899996161, + 0.06912520799960475, + 0.07269666600041091, + 0.07175258299685083, + 0.07496787499985658, + 0.07205666700610891, + 0.0757069999963278, + 0.07382949999009725, + 0.07258200000796933, + 0.07275258300069254, + 0.0701749169966206, + 0.06973225000547245, + 0.07043579198943917, + 0.07132375000219326, + 0.07254579200525768, + 0.07277095800964162, + 0.07294716699107084, + 0.07222087500849739, + 0.07306129198695999, + 0.07488625000405591, + 0.07116462499834597, + 0.07531891700637061, + 0.07498204200237524, + 0.07127245899755508, + 0.07214341599319596, + 0.07035820800228976, + 0.06983341599698178, + 0.06610204199387226, + 0.06648841699643526, + 0.0687130000005709, + 0.07020512499730103, + 0.07248737500049174, + 0.07290408300468698, + 0.07669112499570474, + 0.07349237499875017, + 0.07090741601132322, + 0.06938600000285078, + 0.06969016599759925, + 0.07362220900540706, + 0.0688043749978533 + ], + [ + 0.0632792079995852, + 0.06361437498708256, + 0.06360441599099431, + 0.06390354200266302, + 0.0643809580069501, + 0.06521120799880009, + 0.06629904199508019, + 0.06584108299284708, + 0.06401591699977871, + 0.0644013750134036, + 0.06485837500076741, + 0.06582020899804775, + 0.06581458399887197, + 0.06634487499832176, + 0.06697895799879916, + 0.06690770800923929, + 0.06728124999790452, + 0.06749212498834822, + 0.06713454199780244, + 0.06758587498916313, + 0.06797808299597818, + 0.06992637499934062, + 0.06879658300022129, + 0.06830987500143237, + 0.06903745800082106, + 0.06795641699864063, + 0.06651129199599382, + 0.06712366700230632, + 0.06710741700953804, + 0.06749012500222307, + 0.06769741600146517, + 0.06803799999761395, + 0.0674775410006987, + 0.0685215830017114, + 0.0686005409952486, + 0.06961016599962022, + 0.07059787500475068, + 0.07007816599798389, + 0.06894241599366069, + 0.06951079200371169, + 0.0703218749986263, + 0.07015137501002755, + 0.06913845799863338, + 0.0696848340012366, + 0.06882208300521597, + 0.06990245899942238, + 0.06979895799304359, + 0.07026920799398795, + 0.06962670900975354, + 0.06894916600140277, + 0.06829162499343511, + 0.07024387500132434, + 0.07098541600862518, + 0.0702552089933306, + 0.07030116699752398, + 0.07138670799031388, + 0.07143233300303109, + 0.07486287500069011, + 0.07181733399920631, + 0.07576975000847597, + 0.07737249998899642, + 0.0739352910022717, + 0.079316999996081, + 0.06895508299930952 + ], + [ + 0.051805291994242, + 0.053428750004968606, + 0.0527496249997057, + 0.053707084007328376, + 0.05373929200868588, + 0.05469312499917578, + 0.055921542007126845, + 0.057430541011854075, + 0.058219124999595806, + 0.059465584010467865, + 0.05999933400016744, + 0.06223675000364892, + 0.06233700001030229, + 0.06414662499446422, + 0.0638759589928668, + 0.06439908299944364, + 0.0653195000049891, + 0.06609070798731409, + 0.06693824999092612, + 0.06657258300401736, + 0.0680542920017615, + 0.07068187498953193, + 0.07084991699957754, + 0.06973716701031663, + 0.06922591698821634, + 0.06961962500645313, + 0.0704694589949213, + 0.07078754199028481, + 0.06881695799529552, + 0.06872141599887982, + 0.07006312500743661, + 0.07152504099940415, + 0.0709594170039054, + 0.07135712500894442, + 0.07080804200086277, + 0.07329225000285078, + 0.07874108300893568, + 0.07700641600240488, + 0.07010308399912901, + 0.07219104199612048, + 0.07750479200331029, + 0.07731266699556727, + 0.07380620799085591, + 0.07320791600795928, + 0.07097408300614916, + 0.07730170799186453, + 0.07084429101087153, + 0.0734506250009872, + 0.06807895899692085, + 0.06827754199912306, + 0.07731024999520741, + 0.07239358298829757, + 0.07093924999935552, + 0.0692282499949215, + 0.0712569169991184, + 0.06962229200871661, + 0.0685573749942705, + 0.07350225000118371, + 0.07026595799834467, + 0.07068345900916029, + 0.07038408299558796, + 0.0678998329967726, + 0.06937966600526124, + 0.07074041600571945 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_mha": { + "first_token_ms": { + "count": 5, + "median": 81.52908299234696, + "p95": 83.60306639806367, + "min": 65.58362499345094, + "max": 83.91495799878612, + "iqr": 15.902665996691212, + "mean": 75.96719999564812, + "cv": 0.12015207850834211 + }, + "decode_ms_per_step": { + "count": 315, + "median": 88.17291700688656, + "p95": 97.23791260039434, + "min": 69.34141600504518, + "max": 101.53512499528006, + "iqr": 5.646770994644612, + "mean": 88.35230398750383, + "cv": 0.05834918572372402 + }, + "total_ms": { + "count": 5, + "median": 5630.708496959414, + "p95": 5698.5784992342815, + "min": 5595.896796949091, + "max": 5708.366332037258, + "iqr": 43.01420594856609, + "mean": 5642.16235120839, + "cv": 0.0077335905188344175 + }, + "tokens_per_second": { + "count": 5, + "median": 90.92994252436976, + "p95": 91.42876685323772, + "min": 89.69291216060978, + "max": 91.49561162013295, + "iqr": 0.6928677748386889, + "mean": 90.7496748203175, + "cv": 0.007702146449804141 + }, + "median_step_ms_by_index": [ + 81.52908299234696, + 81.72499999636784, + 80.51808399613947, + 79.641290998552, + 79.658875009045, + 81.70016700751148, + 81.0633330111159, + 81.25804099836387, + 84.11712499218993, + 85.0990000035381, + 86.03604199015535, + 86.28870800021105, + 86.3696250016801, + 85.0692079984583, + 86.59816700674128, + 85.91654199699406, + 87.16883399756625, + 86.99866700044367, + 86.99479200004134, + 88.24570800061338, + 88.75850000185892, + 89.28112499415874, + 89.09479100839235, + 90.11445900250692, + 90.13891700305976, + 90.52308299578726, + 89.20420899812598, + 91.5989999921294, + 88.64508398983162, + 94.23333298764192, + 93.78820800338872, + 91.66775000630878, + 87.32787499320693, + 88.58729200437665, + 85.59108299959917, + 85.56841699464712, + 86.06058300938457, + 86.70020899444353, + 87.54833399143536, + 88.65187500487082, + 87.40400000533555, + 88.92999999807216, + 87.515874998644, + 88.2999169989489, + 88.68216699920595, + 88.36604100361001, + 87.63483400980476, + 89.3302920012502, + 91.39241700177081, + 90.75400000438094, + 94.37454100407194, + 96.74612499657087, + 94.34150000743102, + 95.30699999595527, + 93.23262500402052, + 92.87454199511558, + 89.39433400519192, + 88.70320800633635, + 91.28445800160989, + 92.31654100585729, + 88.02833300433122, + 88.14999999594875, + 86.80879199528135, + 88.9152500021737 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.08391495799878612, + 0.08473279199097306, + 0.08194316599110607, + 0.08784404200559948, + 0.08558845800871495, + 0.08578645800298546, + 0.08251229098823387, + 0.08236116600164678, + 0.08494758400775027, + 0.08797620800032746, + 0.08619808399816975, + 0.09048895799787715, + 0.088614291002159, + 0.08786529200733639, + 0.08659816700674128, + 0.08287849999032915, + 0.08295795900630765, + 0.08384966700396035, + 0.08365374999993946, + 0.08466287500050385, + 0.0848303329985356, + 0.0853237910050666, + 0.08631070899718907, + 0.08667950000381097, + 0.08654316599131562, + 0.08633187500527129, + 0.09126341699447948, + 0.09163941600127146, + 0.09224858300876804, + 0.09423333298764192, + 0.09397508300025947, + 0.09166775000630878, + 0.08488733299600426, + 0.0856757080036914, + 0.08484004200727213, + 0.08556841699464712, + 0.08606058300938457, + 0.08670020899444353, + 0.08754833399143536, + 0.08789504200103693, + 0.09464412499801256, + 0.09736941699520685, + 0.10083916599978693, + 0.09477041699574329, + 0.0949383750121342, + 0.09106070800044108, + 0.09886533299868461, + 0.09118699999817181, + 0.08893766600522213, + 0.08975545900466386, + 0.09437454100407194, + 0.09581158299988601, + 0.08965158300998155, + 0.0962119169998914, + 0.09149229199101683, + 0.09792358400591183, + 0.09840383299160749, + 0.09731879200262483, + 0.09568162499635946, + 0.09644904200104065, + 0.08937237500504125, + 0.08594933401036542, + 0.08561249999911524, + 0.08614837500499561 + ], + [ + 0.06645283399848267, + 0.07111691699537914, + 0.06934141600504518, + 0.07338870799867436, + 0.07563758299511392, + 0.07594270800473168, + 0.07820299999730196, + 0.08019966699066572, + 0.08259483400615864, + 0.08307608400355093, + 0.08381533301144373, + 0.08050008399004582, + 0.07994004200736526, + 0.08164133300306275, + 0.08290970799862407, + 0.08503558300435543, + 0.08641158300451934, + 0.08699866700044367, + 0.08699479200004134, + 0.08826516699627973, + 0.08875850000185892, + 0.08928112499415874, + 0.09003395799663849, + 0.09095545799937099, + 0.09084716699726414, + 0.09152591699967161, + 0.08894454099936411, + 0.09436149999964982, + 0.09720324999943841, + 0.09959508400061168, + 0.09378820800338872, + 0.09494149999227375, + 0.09780929199769162, + 0.09744995798973832, + 0.0894163339980878, + 0.08942908399330918, + 0.08910562499659136, + 0.08814808299939614, + 0.09029583299707156, + 0.08865187500487082, + 0.08740400000533555, + 0.0878062090050662, + 0.087515874998644, + 0.0887557500100229, + 0.08952954200503882, + 0.08999924999079667, + 0.08978149999165908, + 0.0898381249862723, + 0.0892810420045862, + 0.08944433399301488, + 0.09366162499645725, + 0.08906612500140909, + 0.09540479099086951, + 0.09530699999595527, + 0.09549433398933616, + 0.09287454199511558, + 0.08795004199782852, + 0.08817291700688656, + 0.08738675000495277, + 0.08775620900269132, + 0.08907000000181142, + 0.08814999999594875, + 0.08832325000548735, + 0.0889152500021737 + ], + [ + 0.08152908299234696, + 0.08674341699224897, + 0.08221683399460744, + 0.079641290998552, + 0.079658875009045, + 0.08048499999858905, + 0.0810633330111159, + 0.08125804099836387, + 0.0820966660103295, + 0.08164683399081696, + 0.08268650001264177, + 0.08250270799908321, + 0.08476504200370982, + 0.08414645900484174, + 0.08865204200264998, + 0.08877045899862424, + 0.09596858400618657, + 0.09009816699835937, + 0.08976783300749958, + 0.09090650000143796, + 0.09011987499252427, + 0.09265541700005997, + 0.08909479100839235, + 0.09011445900250692, + 0.09301029199559707, + 0.09261158401204739, + 0.09129774999746587, + 0.09159899999212939, + 0.08864508398983162, + 0.09617954200075474, + 0.09087037498829886, + 0.08451704200706445, + 0.08380141700035892, + 0.08492975000990555, + 0.08559108299959917, + 0.08472408400848508, + 0.08431933300744276, + 0.08319200000551064, + 0.08349745800660457, + 0.08374883400392719, + 0.08389520899800118, + 0.08423175000643823, + 0.08508829199126922, + 0.08587808300217148, + 0.0863477080129087, + 0.08756716699281242, + 0.08763483400980476, + 0.0893302920012502, + 0.09656874999927823, + 0.10153512499528006, + 0.09850812499644235, + 0.09674612499657087, + 0.09434150000743102, + 0.09536954200302716, + 0.09601954100071453, + 0.09370166600274388, + 0.08845804100565147, + 0.08728416600206401, + 0.0861880000011297, + 0.08772825000050943, + 0.08702279199496843, + 0.08551204200193752, + 0.08538533298997208, + 0.08694779100187588 + ], + [ + 0.08235549999517389, + 0.08172499999636784, + 0.08051808399613947, + 0.08076079199963715, + 0.0811764999962179, + 0.08231524999428075, + 0.08237183399614878, + 0.0837602919928031, + 0.08411712499218993, + 0.0850990000035381, + 0.08603604199015535, + 0.08628870800021105, + 0.0863696250016801, + 0.0850692079984583, + 0.08505595799942967, + 0.08624241600045934, + 0.08716883399756625, + 0.08692754201183561, + 0.08671958399645519, + 0.08656924999377225, + 0.08653324999613687, + 0.08724616600375157, + 0.08772933299769647, + 0.08889095800986979, + 0.08801116699760314, + 0.08774558299046475, + 0.08797966699057724, + 0.08847733300353866, + 0.08774916600668803, + 0.08738041599281132, + 0.08658400000422262, + 0.08752112500951625, + 0.08732787499320693, + 0.08858729200437665, + 0.08783320800284855, + 0.08971937499882188, + 0.088319832997513, + 0.08809345800545998, + 0.08816083300916944, + 0.0889783750026254, + 0.08867183300026227, + 0.08892999999807216, + 0.08926695799164008, + 0.0882999169989489, + 0.08868216699920595, + 0.08836604100361001, + 0.08685804200649727, + 0.08807324999361299, + 0.0921599159919424, + 0.09075400000438094, + 0.09680920898972545, + 0.09689070799504407, + 0.09364799999457318, + 0.09056233300361782, + 0.09323262500402052, + 0.09271691700269002, + 0.08939433400519192, + 0.08870320800633635, + 0.09495970800344367, + 0.09746645799896214, + 0.08802833300433122, + 0.09051420800096821, + 0.08680879199528135, + 0.09539658299763687 + ], + [ + 0.06558362499345094, + 0.07237020799948368, + 0.07448379100242164, + 0.07553579199884553, + 0.07674724998651072, + 0.08170016700751148, + 0.07887945800030138, + 0.08093795800232328, + 0.08633658399048727, + 0.08687733300030231, + 0.08798508399922866, + 0.08884820800449234, + 0.0867998330068076, + 0.0909672080015298, + 0.09884612500900403, + 0.08591654199699406, + 0.09515579200524371, + 0.08902820899675135, + 0.09092262500780635, + 0.08824570800061338, + 0.09253941700444557, + 0.09291395799664315, + 0.0960486249969108, + 0.09109095799794886, + 0.09013891700305976, + 0.09052308299578726, + 0.08920420899812598, + 0.09084716699726414, + 0.08852329199726228, + 0.08926974999485537, + 0.09500808299344499, + 0.09607675000734162, + 0.09139537499868311, + 0.09043841699894983, + 0.08411795900610741, + 0.08511204199749045, + 0.08569483300379943, + 0.08456862499588169, + 0.08736820900230668, + 0.0902155829971889, + 0.08740004200080875, + 0.0916254579933593, + 0.08561433300201315, + 0.08324074999836739, + 0.08446816699870396, + 0.0841417920019012, + 0.08688954201352317, + 0.08828529099992011, + 0.09139241700177081, + 0.0925838339899201, + 0.09166445799928624, + 0.09845424999366514, + 0.10018933299579658, + 0.08985237500746734, + 0.09148470900254324, + 0.09241562501119915, + 0.08970554199186154, + 0.09242625000479165, + 0.09128445800160989, + 0.09231654100585729, + 0.08800604099815246, + 0.09018587500031572, + 0.09468625001318287, + 0.0978510830027517 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "speedup_cached_vs_uncached_sdpa": 9.833969850720978, + "speedup_cached_vs_uncached_mha": 12.608457262161288 + } + ], + "window": { + "batch": 1, + "prompt_len": 32, + "context_length": 128, + "new_tokens": 224, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_mha": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 1.5260420041158795, + "p95": 1.627757801907137, + "min": 1.335499997367151, + "max": 1.6295410023303702, + "iqr": 0.13262499123811722, + "mean": 1.5199416026007384, + "cv": 0.07869323020410306 + }, + "decode_ms_per_step": { + "count": 1115, + "median": 1.7934589996002614, + "p95": 2.3547081000288017, + "min": 1.5459999995073304, + "max": 9.45700000738725, + "iqr": 0.12674999743467197, + "mean": 1.8642772097691827, + "cv": 0.20575595644427977 + }, + "total_ms": { + "count": 5, + "median": 408.8941658992553, + "p95": 459.1469702369068, + "min": 395.13270097086206, + "max": 470.01773605006747, + "iqr": 19.10361998307053, + "mean": 417.2537593811285, + "cv": 0.07361582679229603 + }, + "tokens_per_second": { + "count": 5, + "median": 547.8190169511732, + "p95": 566.4899993153099, + "min": 476.5777604105964, + "max": 566.8981571244801, + "iqr": 25.960446223732447, + "mean": 539.0098448839551, + "cv": 0.0683002506398672 + }, + "median_step_ms_by_index": [ + 1.5260420041158795, + 2.3769999970681965, + 1.9724579906323925, + 1.8517090065870434, + 1.8311250023543835, + 1.7754580039763823, + 1.8065420008497313, + 1.769500013324432, + 1.7883330001495779, + 1.764041997375898, + 1.7261659959331155, + 1.7613340023672208, + 1.790290989447385, + 1.7587500042282045, + 1.7673749971436337, + 1.757167003233917, + 1.7517090018372983, + 1.738874998409301, + 1.7327089881291613, + 1.7207079945364967, + 1.7423339886590838, + 1.730832998873666, + 1.7490839964011684, + 1.7560420092195272, + 1.7287090013269335, + 1.7557919927639887, + 1.7439169896533713, + 1.7340000049443915, + 1.7508750024717301, + 1.744791996316053, + 1.7612499941606075, + 1.7490000027464703, + 1.7548329924466088, + 1.7537920066388324, + 1.756792000378482, + 1.8572499975562096, + 1.6656250081723556, + 1.716207989375107, + 1.7562919965712354, + 1.7532080091768876, + 1.7490839964011684, + 1.797542005078867, + 1.7745829973137006, + 1.7512909980723634, + 1.7067500011762604, + 1.7092920024879277, + 1.8019170092884451, + 1.7629999929340556, + 1.7965000006370246, + 1.7333750001853332, + 1.7534579965285957, + 1.7687499930616468, + 1.7617090052226558, + 1.757125006406568, + 1.7748750105965883, + 1.7585000023245811, + 1.7562080029165372, + 1.6742089937906712, + 1.752791999024339, + 1.7391250003129244, + 1.7377920012222603, + 1.7528749885968864, + 1.7367499967804179, + 1.7346669919788837, + 1.8061670125462115, + 1.793458010070026, + 1.725666006677784, + 1.801125006750226, + 1.7423329991288483, + 1.7454170010751113, + 1.741665997542441, + 1.756375000695698, + 1.7759170004865155, + 1.7241249879589304, + 1.787250002962537, + 1.7932920018211007, + 1.774457996361889, + 1.781500002834946, + 1.7104159924201667, + 1.789125002687797, + 1.7622080049477518, + 1.7391250003129244, + 1.7114999936893582, + 1.7519160028314218, + 1.7729170067468658, + 1.6962090012384579, + 1.8143329943995923, + 1.827208005124703, + 1.756708006723784, + 1.7972920031752437, + 1.7078329983633012, + 1.7597499972907826, + 1.7984590085688978, + 1.7400420038029552, + 1.8084579933201894, + 1.7548750038258731, + 1.7519169923616573, + 1.8154169956687838, + 1.7749580001691356, + 1.7874999903142452, + 1.8702079978538677, + 1.8760829989332706, + 1.825166997150518, + 1.8198339967057109, + 1.7942910053534433, + 1.7876250058179721, + 1.8090000085067004, + 1.7194579995702952, + 1.799417004804127, + 1.8078749999403954, + 1.8114160047844052, + 1.8650419951882213, + 1.7874170007416978, + 1.8286660051671788, + 1.8580839969217777, + 1.808957997127436, + 1.7441249947296456, + 1.7984589940169826, + 1.7824170063249767, + 1.816165997297503, + 1.8138329905923456, + 1.825707993702963, + 1.8791249895002693, + 1.7989170009968802, + 1.7932500049937516, + 1.8023329903371632, + 1.8180000042775646, + 1.7490000027464703, + 1.7535410006530583, + 1.7681250028545037, + 1.7950000037671998, + 1.7760830087354407, + 1.7850000003818423, + 1.7780420021153986, + 1.7961249977815896, + 1.8018330010818318, + 1.8149579991586506, + 1.791083996067755, + 1.750499999616295, + 1.8105420022038743, + 1.8185829976573586, + 1.8291659944225103, + 1.8391249905107543, + 1.7741250048857182, + 1.789166999515146, + 1.8397499952698126, + 1.927666991832666, + 1.8863330042222515, + 1.8819999968400225, + 1.817541997297667, + 1.8810840119840577, + 1.864374993601814, + 1.8769589951261878, + 1.7653340037213638, + 1.7902499967021868, + 1.8629579863045365, + 1.7606659966986626, + 1.822833000915125, + 1.844333004555665, + 1.8387909949524328, + 1.8234999879496172, + 1.8415829981677234, + 1.7915839998750016, + 1.8703749956330284, + 1.7728749953676015, + 1.8530840025050566, + 1.7907079891301692, + 1.8204999942099676, + 1.77425000583753, + 1.8505420011933893, + 1.8578329909360036, + 1.8471670045983046, + 1.7743749922374263, + 1.8124999915016815, + 1.8070410005748272, + 1.891417006845586, + 1.804792002076283, + 1.8500419973861426, + 1.7677499999990687, + 1.7657499993219972, + 1.7772919964045286, + 1.8509159999666736, + 1.8004160083364695, + 1.8191249982919544, + 1.8497919954825193, + 1.8484159954823554, + 1.8117080035153776, + 1.7989999905694276, + 1.80283299414441, + 1.8751669995253906, + 1.8043340096483007, + 1.8349169986322522, + 1.8189999973401427, + 1.8115419952664524, + 1.8204999942099676, + 1.8560419994173571, + 1.7542080022394657, + 1.7856250051409006, + 1.8304169934708625, + 1.7650000081630424, + 1.8351250037085265, + 1.7665829946054146, + 1.7988329927902669, + 1.8448329938109964, + 1.8386250012554228, + 1.8592910055303946, + 1.8315420020371675, + 1.7974579968722537, + 1.8274169997312129, + 1.788749999832362, + 1.7953750066226348, + 1.8429170013405383, + 1.8262079975102097, + 1.8200839986093342, + 1.8637910106917843, + 1.8278749921591952, + 1.8062499875668436, + 1.787999994121492, + 1.806874992325902, + 1.751666990458034, + 1.7741670017130673, + 1.808375003747642, + 1.8402500136289746, + 1.77945799077861 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.001488000008976087, + 0.0025602499954402447, + 0.0016281249991152436, + 0.0018517090065870434, + 0.001686667004832998, + 0.0017450830055167899, + 0.0018065420008497313, + 0.0017346660024486482, + 0.001759083999786526, + 0.001764041997375898, + 0.0017261659959331155, + 0.0016392920078942552, + 0.0019198329973733053, + 0.0017587500042282045, + 0.0016750000067986548, + 0.0017400420038029552, + 0.0017517090018372983, + 0.001738874998409301, + 0.0017327089881291613, + 0.0017207079945364967, + 0.0016351669910363853, + 0.0016334579995600507, + 0.0017404999962309375, + 0.0017560420092195272, + 0.001727415990899317, + 0.0016786669875727966, + 0.0015883329906500876, + 0.001709332995233126, + 0.0016457090096082538, + 0.0016982910019578412, + 0.001704209003946744, + 0.0016285000019706786, + 0.0017051659960998222, + 0.0017144169978564605, + 0.00158845899568405, + 0.002201290990342386, + 0.001644207994104363, + 0.001658250010223128, + 0.0017562919965712354, + 0.0017532080091768876, + 0.0017389580025337636, + 0.0018782499973895028, + 0.0016801250021671876, + 0.0017512909980723634, + 0.0016635409992886707, + 0.0015944170008879155, + 0.0018019170092884451, + 0.0017629999929340556, + 0.0017965000006370246, + 0.001706833005300723, + 0.0016223330021603033, + 0.0017687499930616468, + 0.0016922909999266267, + 0.0016460839979117736, + 0.0017346669919788837, + 0.0016802919999463484, + 0.0017562080029165372, + 0.001764957996783778, + 0.001752791999024339, + 0.0016679169930284843, + 0.001662082999246195, + 0.0017400829965481535, + 0.0017367499967804179, + 0.0017366249958286062, + 0.0018477080011507496, + 0.001773707990651019, + 0.001725666006677784, + 0.0017527909949421883, + 0.0016813749971333891, + 0.0016380839952034876, + 0.0017416659975424409, + 0.0016828329971758649, + 0.0017759170004865155, + 0.0017217919958056882, + 0.001716915998258628, + 0.0017932920018211007, + 0.0016788329958217219, + 0.0016215409996220842, + 0.0016226250008912757, + 0.0016275419911835343, + 0.0017818749911384657, + 0.0016152089956449345, + 0.0016975000035017729, + 0.0016054579900810495, + 0.0016499160119565204, + 0.0016495000018039718, + 0.0018143329943995923, + 0.0017926659929798916, + 0.0016162499960046262, + 0.0016925840027397498, + 0.0016715829988243058, + 0.0017597499972907826, + 0.0019166669953847304, + 0.0018597089947434142, + 0.0018667090043891221, + 0.0019078749901382253, + 0.002006958005949855, + 0.0022307499893940985, + 0.0029222500015748665, + 0.0021467500046128407, + 0.0020633329986594617, + 0.002049832997727208, + 0.0018894999957410619, + 0.0018198339967057109, + 0.001800750003894791, + 0.0017654589901212603, + 0.0017465830023866147, + 0.0016909170080907643, + 0.001799417004804127, + 0.0018119169981218874, + 0.0018897080008173361, + 0.0019069170084549114, + 0.0018750840099528432, + 0.0018704170070122927, + 0.0019027090020244941, + 0.0017112909990828484, + 0.001714124999125488, + 0.0017202500021085143, + 0.0017207909986609593, + 0.001816165997297503, + 0.0018138329905923456, + 0.001853875000961125, + 0.001886291996925138, + 0.0018320829985896125, + 0.0017932500049937516, + 0.0018242079968331382, + 0.0018023750017164275, + 0.0017161249998025596, + 0.001723666995530948, + 0.0017681250028545037, + 0.001864374993601814, + 0.0017760830087354407, + 0.0018270830041728914, + 0.0017780420021153986, + 0.0017961249977815896, + 0.0018192089919466525, + 0.0018149579991586506, + 0.001791083996067755, + 0.001750499999616295, + 0.0017443329998059198, + 0.0018185829976573586, + 0.0018291659944225103, + 0.0018391249905107543, + 0.001716792001388967, + 0.001789166999515146, + 0.001647042008698918, + 0.004517374996794388, + 0.004002750007202849, + 0.003404999995836988, + 0.003576207993319258, + 0.0018810840119840577, + 0.001831792003940791, + 0.0038954159972490743, + 0.0019682909914990887, + 0.001789125002687797, + 0.001689666009042412, + 0.0015826669987291098, + 0.0018240839999634773, + 0.001822833000915125, + 0.0018387909949524328, + 0.0018234999879496172, + 0.0018933749961433932, + 0.001894874993013218, + 0.0018703749956330284, + 0.0017728749953676015, + 0.0017128750041592866, + 0.0017512500053271651, + 0.001839417003793642, + 0.0016871250118128955, + 0.0018505420011933893, + 0.0018998750019818544, + 0.001875999994808808, + 0.0016819169977679849, + 0.001758333994075656, + 0.0018070410005748272, + 0.001779790996806696, + 0.0017506660078652203, + 0.001741125000989996, + 0.0017360420024488121, + 0.0017657499993219972, + 0.0018973330006701872, + 0.0018509159999666736, + 0.0018005830061156303, + 0.0017693749978207052, + 0.0018223329971078783, + 0.0017329580005025491, + 0.0017267920047743246, + 0.0017165000026579946, + 0.0017624169995542616, + 0.0018751669995253906, + 0.0018918749992735684, + 0.0018349169986322522, + 0.0018189999973401427, + 0.0018115419952664524, + 0.0018204999942099676, + 0.001732042001094669, + 0.001753542004735209, + 0.001723084002151154, + 0.0018304169934708625, + 0.0017650000081630424, + 0.0018351250037085265, + 0.0017150000057881698, + 0.001788749999832362, + 0.001808916000300087, + 0.0017537500098114833, + 0.001845332997618243, + 0.0016991250013234094, + 0.0017974579968722537, + 0.0018274169997312129, + 0.0017822910012910143, + 0.0017953750066226348, + 0.0017620419966988266, + 0.0018262079975102097, + 0.001636375003727153, + 0.001938708999659866, + 0.0018278749921591952, + 0.0017406670085620135, + 0.0016917500033741817, + 0.001806874992325902, + 0.001751666990458034, + 0.0017741670017130673, + 0.0017172500083688647, + 0.0018217920005554333, + 0.0017782089998945594 + ], + [ + 0.0015260420041158795, + 0.0022729159973096102, + 0.00245462499151472, + 0.0023243330069817603, + 0.0021170420077396557, + 0.0016157499921973795, + 0.001710875003482215, + 0.0017344169900752604, + 0.0017597499972907826, + 0.0017128330073319376, + 0.0016932500002440065, + 0.001622875002794899, + 0.0016437919985037297, + 0.0017471669998485595, + 0.0017459579976275563, + 0.001757167003233917, + 0.001721125008771196, + 0.0017186659970320761, + 0.001654457999393344, + 0.0016859579918673262, + 0.0017423339886590838, + 0.001730832998873666, + 0.0017490839964011684, + 0.001747250003973022, + 0.0017287090013269335, + 0.0017557919927639887, + 0.0017439169896533713, + 0.0016411249962402508, + 0.0016794169932836667, + 0.001744791996316053, + 0.0017288329981965944, + 0.0017869169969344512, + 0.0017548329924466088, + 0.001776791992597282, + 0.001756792000378482, + 0.0017015830089803785, + 0.0016427500086138025, + 0.0017391250003129244, + 0.0017390410066582263, + 0.00174079199496191, + 0.0017490839964011684, + 0.0016182090039364994, + 0.0017745829973137006, + 0.001686499992501922, + 0.0016495000018039718, + 0.0017706669896142557, + 0.0017307919915765524, + 0.0017303329950664192, + 0.00174245800008066, + 0.0017333750001853332, + 0.0016205839929170907, + 0.0016380409942939878, + 0.0017477500077802688, + 0.001757125006406568, + 0.001749584000208415, + 0.0017366670072078705, + 0.0015849170013098046, + 0.0016562499949941412, + 0.001746875001117587, + 0.0017391250003129244, + 0.0017377920012222603, + 0.0017528749885968864, + 0.0017456250061513856, + 0.0017346669919788837, + 0.0017310419934801757, + 0.0016283330041915178, + 0.0017115420050686225, + 0.0017302500054938719, + 0.0017423329991288483, + 0.001678499989793636, + 0.0016719580016797408, + 0.0015568749950034544, + 0.0016954160091700032, + 0.0016422920016339049, + 0.0017872500029625371, + 0.0015762499970151111, + 0.0022009160020388663, + 0.0018263750098412856, + 0.0016352079983334988, + 0.001797125005396083, + 0.0015837920072954148, + 0.001704000009340234, + 0.0016507499967701733, + 0.0016745000029914081, + 0.0017880829982459545, + 0.0016089169948827475, + 0.0018486669869162142, + 0.001827208005124703, + 0.0017782080103643239, + 0.0018760410021059215, + 0.0017078329983633012, + 0.0016394999984186143, + 0.0017435000045225024, + 0.0017053750052582473, + 0.0017313749995082617, + 0.0017428339924663305, + 0.0016386250063078478, + 0.0017791670106817037, + 0.0017552079953020439, + 0.0017406660044798627, + 0.0018702079978538677, + 0.0018760829989332706, + 0.001857166993431747, + 0.0018728330032899976, + 0.0017942910053534433, + 0.0018349169986322522, + 0.0018090000085067004, + 0.0017194579995702952, + 0.0017295410070801154, + 0.0018078749999403954, + 0.0018114160047844052, + 0.0018297089991392568, + 0.0017729579994920641, + 0.0017282499902648851, + 0.001774874996044673, + 0.001808957997127436, + 0.0018010409985436127, + 0.0017568329931236804, + 0.0019041250052396208, + 0.0018318330112379044, + 0.0018207910034107044, + 0.001825707993702963, + 0.0018220000056317076, + 0.0017273749981541187, + 0.0017808330012485385, + 0.0018023329903371632, + 0.001808165994589217, + 0.001700166001683101, + 0.001775415992597118, + 0.0018094169936375692, + 0.0017829999997047707, + 0.0016966250113910064, + 0.0018134580022888258, + 0.001764792003086768, + 0.0017764579970389605, + 0.0018158750026486814, + 0.0017768329998943955, + 0.0017027499998221174, + 0.0018620420014485717, + 0.0018105420022038743, + 0.0018105829949490726, + 0.0016715829988243058, + 0.001778499994543381, + 0.001707250004983507, + 0.0016579160001128912, + 0.0017997499962802976, + 0.0017875409976113588, + 0.001879249990452081, + 0.001735625002766028, + 0.001788041990948841, + 0.0017989589978242293, + 0.0017648329958319664, + 0.0017090829933295026, + 0.0017653340037213638, + 0.0017903749976539984, + 0.0018629579863045365, + 0.0017606659966986626, + 0.0017321250052191317, + 0.001844333004555665, + 0.001764915999956429, + 0.0017012919997796416, + 0.0017479590023867786, + 0.0017158330010715872, + 0.001779415993951261, + 0.0017164170130854473, + 0.0018530840025050566, + 0.0017907079891301692, + 0.0017717080045258626, + 0.0017676660063443705, + 0.001738124992698431, + 0.0017508340097265318, + 0.0017122909921454266, + 0.0017743749922374263, + 0.0018124999915016815, + 0.001717167004244402, + 0.001891417006845586, + 0.0018609580001793802, + 0.0018605000077513978, + 0.001731666998239234, + 0.001716750004561618, + 0.0017490409954916686, + 0.0017169170023407787, + 0.0017065830033970997, + 0.0016717079997761175, + 0.0018497919954825193, + 0.001864917008788325, + 0.001833749993238598, + 0.0018868339975597337, + 0.0016819169977679849, + 0.0017202910094056278, + 0.0017450000013923272, + 0.0017394160095136613, + 0.0017863749962998554, + 0.0017179169954033569, + 0.0020154160010861233, + 0.0018753329932224005, + 0.0018576669972389936, + 0.0017831669974839315, + 0.0018818340031430125, + 0.0017520420078653842, + 0.001730541989672929, + 0.0016946669929893687, + 0.001730541989672929, + 0.001858874995377846, + 0.0018714579928200692, + 0.0019248330063419417, + 0.001731042008032091, + 0.00177237500611227, + 0.0018129579984815791, + 0.001788749999832362, + 0.0017179999995278195, + 0.0019041659979848191, + 0.001715999998850748, + 0.001786250009899959, + 0.0017654590046731755, + 0.001651625003432855, + 0.0016798330034362152, + 0.0017084170103771612, + 0.0018048750062007457, + 0.0017110420012613758, + 0.0017017500067595392, + 0.001808375003747642, + 0.0018402500136289746, + 0.0017794579907786101 + ], + [ + 0.001335499997367151, + 0.0019515419990057126, + 0.0016349579964298755, + 0.0016606669960310683, + 0.0016681670094840229, + 0.0017754580039763823, + 0.0018825419974746183, + 0.0017831659934017807, + 0.0017883330001495779, + 0.0017429159925086424, + 0.0016885000077309087, + 0.0017613340023672208, + 0.0017052920011337847, + 0.0016331660008290783, + 0.0017673749971436337, + 0.0017559170082677156, + 0.0016029999969759956, + 0.0015459999995073304, + 0.0017014999903040007, + 0.0016609580052318051, + 0.0017040840029949322, + 0.0017066250002244487, + 0.0017036659992299974, + 0.0017382499936502427, + 0.001706124996417202, + 0.0017024590051732957, + 0.0017374169983668253, + 0.0017340000049443915, + 0.0017508750024717301, + 0.0017598749982425943, + 0.0017612499941606075, + 0.0017490000027464703, + 0.0017607089976081625, + 0.0017493330087745562, + 0.0017333750001853332, + 0.0018572499975562096, + 0.0016656250081723556, + 0.0016604580014245585, + 0.0015978340088622645, + 0.0016690409975126386, + 0.0016776249976828694, + 0.001949374985997565, + 0.0017373750015394762, + 0.0017262910114368424, + 0.0019502909999573603, + 0.001697875006357208, + 0.0016027499950723723, + 0.002576124999905005, + 0.001743250002618879, + 0.0016631659964332357, + 0.0017971249908441678, + 0.0017518329987069592, + 0.0017617090052226558, + 0.0017454999906476587, + 0.0017972920031752437, + 0.0017585000023245811, + 0.0016679169930284843, + 0.0016543329984415323, + 0.0015918749995762482, + 0.001591291991644539, + 0.0018522080063121393, + 0.0017618749989196658, + 0.001732375007122755, + 0.0016624169948045164, + 0.0018061670125462115, + 0.0020574170048348606, + 0.0016821669996716082, + 0.001801125006750226, + 0.00173216599796433, + 0.0017864170076791197, + 0.001799999998183921, + 0.001756375000695698, + 0.0017582079890416935, + 0.0017241249879589304, + 0.0017840000073192641, + 0.0018078749999403954, + 0.0016973339952528477, + 0.001781500002834946, + 0.0017104159924201667, + 0.0018079169967677444, + 0.0016182499966816977, + 0.0017947500018635765, + 0.0017114999936893582, + 0.0017519160028314218, + 0.0017113339999923483, + 0.0016962090012384579, + 0.00168208300601691, + 0.0016969159914879128, + 0.0016284170123981312, + 0.0015696670016041026, + 0.0016757499979576096, + 0.0015514999977312982, + 0.0016704159934306517, + 0.0017400420038029552, + 0.0017252079996978864, + 0.0016091249999590218, + 0.001607250000233762, + 0.0018154169956687838, + 0.001673332997597754, + 0.0017874999903142452, + 0.0018394579965388402, + 0.0017913329938892275, + 0.0017399999924236909, + 0.001826499996241182, + 0.001811459005693905, + 0.0017876250058179721, + 0.0018402499990770593, + 0.0017795829917304218, + 0.0018423330038785934, + 0.0018020420102402568, + 0.001773916999809444, + 0.0018650419951882213, + 0.0018745410052360967, + 0.0018640839989529923, + 0.0018580839969217777, + 0.0017598330014152452, + 0.0017441249947296456, + 0.001808375003747642, + 0.0017824170063249767, + 0.0017118749965447932, + 0.0017854159959824756, + 0.0018725000118138269, + 0.0018791249895002693, + 0.0018740000086836517, + 0.0018758750084089115, + 0.0018749169976217672, + 0.0018738749931799248, + 0.0017962499987334013, + 0.0017535410006530583, + 0.001758416008669883, + 0.0017950000037671998, + 0.0017428329883841798, + 0.001750082999933511, + 0.0018616250017657876, + 0.001861500000813976, + 0.0016962500085355714, + 0.0018341669929213822, + 0.001819583005271852, + 0.001724207992083393, + 0.0017459999944549054, + 0.001844915997935459, + 0.0018429170013405383, + 0.0018267499981448054, + 0.0017741250048857182, + 0.001742082997225225, + 0.001881290998426266, + 0.0018438750121276826, + 0.001803042003302835, + 0.0018166249938076362, + 0.0017576250102138147, + 0.0022863339981995523, + 0.001864374993601814, + 0.00184229199658148, + 0.0016619579982943833, + 0.0017335000011371449, + 0.0017822090012487024, + 0.0017210410005645826, + 0.0017998749972321093, + 0.0018170420080423355, + 0.0018951249949168414, + 0.0018818329990608618, + 0.0017879590013762936, + 0.0017396670009475201, + 0.0017780000052880496, + 0.0017505830037407577, + 0.0017364170053042471, + 0.001741332991514355, + 0.0017463750118622556, + 0.0018454170058248565, + 0.0018602499912958592, + 0.0018578329909360036, + 0.0018471670045983046, + 0.0017407080013072118, + 0.0018294170004082844, + 0.0017887080030050129, + 0.0017512500053271651, + 0.001804792002076283, + 0.0018069580110022798, + 0.001819790995796211, + 0.0017774590087356046, + 0.0017772919964045286, + 0.0017867080023279414, + 0.0017904579872265458, + 0.0018191249982919544, + 0.0017987920000450686, + 0.0017547080060467124, + 0.0018117080035153776, + 0.0017989999905694276, + 0.00180283299414441, + 0.0017325840017292649, + 0.0016909590049181134, + 0.0017844999965745956, + 0.0018138750019716099, + 0.0018081250018440187, + 0.0017939999961527064, + 0.0018560419994173571, + 0.0017542080022394657, + 0.0017856250051409006, + 0.0018001669959630817, + 0.0017327499954262748, + 0.0017165000026579946, + 0.0017665829946054146, + 0.0017988329927902669, + 0.0017968330066651106, + 0.0018496250122552738, + 0.0018684579990804195, + 0.0018330000020796433, + 0.0017738750029820949, + 0.001807291992008686, + 0.0017122909921454266, + 0.0017384169914294034, + 0.0017182500014314428, + 0.0017724999925121665, + 0.0018481250008335337, + 0.0018637910106917843, + 0.0018867079925257713, + 0.0018762910040095448, + 0.0018188340036431327, + 0.0016994580073514953, + 0.0017225420015165582, + 0.001701875007711351, + 0.0017340409976895899, + 0.0018604579963721335, + 0.0018415829981677234 + ], + [ + 0.0016295410023303702, + 0.0023769999970681965, + 0.0019724579906323925, + 0.0018145409994758666, + 0.0018717909988481551, + 0.0018082919996231794, + 0.00178116699680686, + 0.001769500013324432, + 0.0019476659945212305, + 0.001912374995299615, + 0.0018169999966630712, + 0.001898457994684577, + 0.001790290989447385, + 0.0018758749938569963, + 0.001833875008742325, + 0.0018726670095929876, + 0.001873915985925123, + 0.0024478330014972016, + 0.0024462919973302633, + 0.0019164160039508715, + 0.0019518750050337985, + 0.0020522080012597144, + 0.0020402499940246344, + 0.0019846249924739823, + 0.0017333330033579841, + 0.001774082993506454, + 0.0025505419907858595, + 0.001777541998308152, + 0.001906833000248298, + 0.0017083330021705478, + 0.0023083749983925372, + 0.0023967089946381748, + 0.0017284159985138103, + 0.0017537920066388324, + 0.0019324159948155284, + 0.0017400409997208044, + 0.0018608749960549176, + 0.001716207989375107, + 0.0017809999990276992, + 0.0018390000041108578, + 0.0018752920004772022, + 0.001767583002219908, + 0.0018517499993322417, + 0.0018924999894807115, + 0.0018757919897325337, + 0.0018457080004736781, + 0.0018602499912958592, + 0.0017562909924890846, + 0.00187145899690222, + 0.0018171249976148829, + 0.0017901670071296394, + 0.0018449170020176098, + 0.0019247079908382148, + 0.001759582999511622, + 0.0017748750105965883, + 0.0018229589913971722, + 0.0017989159969147295, + 0.0016742089937906712, + 0.0017850409931270406, + 0.0017934589996002614, + 0.0018134590063709766, + 0.0016954160091700032, + 0.0016446660010842606, + 0.0016698340041330084, + 0.00177254099980928, + 0.001793458010070026, + 0.0021253750019241124, + 0.0018739160004770383, + 0.0018368750024819747, + 0.0018110829987563193, + 0.0017290830001002178, + 0.002051666990155354, + 0.001879624993307516, + 0.0018517919961595908, + 0.001830041001085192, + 0.002095541000016965, + 0.00185591600893531, + 0.0018983330082846805, + 0.0017822079971665516, + 0.0017149999912362546, + 0.0018850000051315874, + 0.0019361250015208498, + 0.00184116700256709, + 0.0018134590063709766, + 0.001779583006282337, + 0.001884915996924974, + 0.00191516600898467, + 0.0018376669904682785, + 0.0019409170054132119, + 0.0019177500071236864, + 0.0018818329990608618, + 0.001891417006845586, + 0.0019387500069569796, + 0.001933666004333645, + 0.0018684999959077686, + 0.0018701250082813203, + 0.0018497499986551702, + 0.0018549579981481656, + 0.0018154999997932464, + 0.0018656250031199306, + 0.0019477500027278438, + 0.0019524999952409416, + 0.001825166997150518, + 0.0017947500018635765, + 0.0017724999925121665, + 0.0017978749965550378, + 0.0018548750085756183, + 0.0018774999916786328, + 0.001852000001235865, + 0.0018345420103287324, + 0.0019110410066787153, + 0.0018762500112643465, + 0.0017277080041822046, + 0.0018286660051671788, + 0.0018649580015335232, + 0.0018169170070905238, + 0.001785500004189089, + 0.0017984589940169826, + 0.0018901250005001202, + 0.0019404170016059652, + 0.0019324169988976792, + 0.0017182079900521785, + 0.001799582998501137, + 0.0017989170009968802, + 0.0018612499989103526, + 0.0017454999906476587, + 0.0019595000048866495, + 0.0018697909981710836, + 0.0017585409950697795, + 0.0018329580052522942, + 0.0018026249890681356, + 0.0018834999937098473, + 0.0017850000003818423, + 0.0018609160033520311, + 0.0019107499974779785, + 0.0018018330010818318, + 0.001820916004362516, + 0.0018560419994173571, + 0.0019235840009059757, + 0.0018299999937880784, + 0.0017774160078261048, + 0.0018147079972550273, + 0.0018623750074766576, + 0.0018042909941868857, + 0.0018499579891795292, + 0.0018397499952698126, + 0.001927666991832666, + 0.0018863330042222515, + 0.0018819999968400225, + 0.001817541997297667, + 0.0017882919928524643, + 0.0018686660041566938, + 0.0018769589951261878, + 0.0017623750027269125, + 0.0017902499967021868, + 0.0018705840047914535, + 0.0018574169953353703, + 0.001822833000915125, + 0.0018454999953974038, + 0.0017940840043593198, + 0.0017909580055857077, + 0.0018415829981677234, + 0.0017915839998750016, + 0.0019162920070812106, + 0.0018362080008955672, + 0.00186191700049676, + 0.0018817090021912009, + 0.0018204999942099676, + 0.00177425000583753, + 0.0017094170034397393, + 0.00171929199132137, + 0.00181275000795722, + 0.0018054579995805398, + 0.0017844160029198974, + 0.001905500001157634, + 0.0019266250019427389, + 0.001779000012902543, + 0.0018500419973861426, + 0.0017677499999990687, + 0.0017513330094516277, + 0.0017198749992530793, + 0.00192116700054612, + 0.0018004160083364695, + 0.0021297500061336905, + 0.001899166003568098, + 0.0018484159954823554, + 0.0017718339950079098, + 0.0017309590039076284, + 0.0018046250042971224, + 0.001876750000519678, + 0.0018147499940823764, + 0.0018710830045165494, + 0.0018607909878483042, + 0.001903417010908015, + 0.0018120419990736991, + 0.001800416997866705, + 0.0017273339908570051, + 0.0018767090077744797, + 0.001808375003747642, + 0.0017882079991977662, + 0.0019497920002322644, + 0.0019208329904358834, + 0.0018786250002449378, + 0.0018509579967940226, + 0.0018386250012554228, + 0.0017655830015428364, + 0.0018315420020371675, + 0.0018735829944489524, + 0.0018505840125726536, + 0.0018930420046672225, + 0.0018727920105447993, + 0.0018429170013405383, + 0.0018782919942168519, + 0.0018200839986093342, + 0.001755999997840263, + 0.001790250011254102, + 0.0018062499875668436, + 0.001787999994121492, + 0.0018214580049971119, + 0.001971000005141832, + 0.0019076669996138662, + 0.0018396670056972653, + 0.0017862919921753928, + 0.0017733749991748482 + ], + [ + 0.0016206250002142042, + 0.00265929099987261, + 0.002356917000724934, + 0.0025415420095669106, + 0.0018311250023543835, + 0.0019177080102963373, + 0.002032500007771887, + 0.0019142499950248748, + 0.0019122910016449168, + 0.0018830409971997142, + 0.0018968750082422048, + 0.0018598749884404242, + 0.0018165829969802871, + 0.0018932499951915815, + 0.002425625003525056, + 0.0018794999923557043, + 0.0018212080030934885, + 0.0019135829934384674, + 0.0018954999977722764, + 0.0019132080051349476, + 0.001919207992614247, + 0.001845332997618243, + 0.0018855829985113814, + 0.0019170419982401654, + 0.0019180840026820078, + 0.0019258749962318689, + 0.0019142079981975257, + 0.0018643340008566156, + 0.00189124999451451, + 0.001881666001281701, + 0.002353791001951322, + 0.0017083329876186326, + 0.0020379999914439395, + 0.0023451250017387792, + 0.0019462079944787547, + 0.0019093750015599653, + 0.0019143329991493374, + 0.0018336670036660507, + 0.0017757080058800057, + 0.0018777910008793697, + 0.0018282499950146303, + 0.001797542005078867, + 0.001842416008003056, + 0.001799666992155835, + 0.0017067500011762604, + 0.0017092920024879277, + 0.0018589999963296577, + 0.0018612080020830035, + 0.0018438330007484183, + 0.001804041996365413, + 0.0017534579965285957, + 0.0018675419996725395, + 0.0024618750030640513, + 0.0018314170010853559, + 0.0018612080020830035, + 0.0018037919944617897, + 0.0018630410049809143, + 0.0018120419990736991, + 0.001782874998752959, + 0.001825749990530312, + 0.0017314160068053752, + 0.0018312909960513934, + 0.001797707998775877, + 0.0017424589896108955, + 0.0018648749974090606, + 0.0018541250028647482, + 0.0018591670086607337, + 0.001851415989222005, + 0.001826124993385747, + 0.0017454170010751113, + 0.0018416250095469877, + 0.0019217499939259142, + 0.0018387080053798854, + 0.0018040409922832623, + 0.0019151250016875565, + 0.0017820000066421926, + 0.001774457996361889, + 0.0017138749972218648, + 0.0017849170108092949, + 0.001789125002687797, + 0.0017622080049477518, + 0.0017391250003129244, + 0.0018639159970916808, + 0.001779415993951261, + 0.0017729170067468658, + 0.0017591670039109886, + 0.0016631250036880374, + 0.0020107500022277236, + 0.001756708006723784, + 0.0017972920031752437, + 0.0017737500020302832, + 0.0018687080009840429, + 0.0017984590085688978, + 0.0016957920015556738, + 0.0018084579933201894, + 0.0017548750038258731, + 0.0017519169923616573, + 0.001786250009899959, + 0.0017749580001691356, + 0.001757125006406568, + 0.0017625000036787242, + 0.0017569170013302937, + 0.0017783329967642203, + 0.0017500419926363975, + 0.00174229200638365, + 0.001725708003505133, + 0.001695541010121815, + 0.0016417909937445074, + 0.0016569580038776621, + 0.0017016660131048411, + 0.0017099160031648353, + 0.0017922499973792583, + 0.0017874170007416978, + 0.0017223329923581332, + 0.001709125004708767, + 0.0018136250000679865, + 0.0016845000063767657, + 0.002088958994136192, + 0.0017187909979838878, + 0.0017525419971207157, + 0.0017686250066617504, + 0.0017484999989392236, + 0.0019398750009713694, + 0.0017416659975424409, + 0.0017641660087974742, + 0.0017182500014314428, + 0.0018180000042775646, + 0.0017490000027464703, + 0.00170266701024957, + 0.0016993329918477684, + 0.0017437080095987767, + 0.0018054579995805398, + 0.0017520830006105825, + 0.0017238750006072223, + 0.0016844999918248504, + 0.0017413339955965057, + 0.001738416001899168, + 0.001713167002890259, + 0.0017134999943664297, + 0.0025917919992934912, + 0.0025753340014489368, + 0.0027083750028396025, + 0.0026895000046351925, + 0.0028845829947385937, + 0.0028667089936789125, + 0.002244041010271758, + 0.002164290999644436, + 0.0021955419942969456, + 0.002203165990067646, + 0.0021980410092510283, + 0.0021172080014366657, + 0.0021289580035954714, + 0.002077290992019698, + 0.0023567500029457733, + 0.0029713330004597083, + 0.0023526669974671677, + 0.002353832998778671, + 0.002366250002523884, + 0.002335957993636839, + 0.0023999169934540987, + 0.0030272500007413328, + 0.0027704999956768006, + 0.00270979201013688, + 0.002721957993344404, + 0.0026550000038696453, + 0.002856999999494292, + 0.0036404159909579903, + 0.0035922909883083776, + 0.003638499998487532, + 0.0033118339924840257, + 0.004602250002790242, + 0.0030332089954754338, + 0.0030214580037863925, + 0.003037041999050416, + 0.0030336670024553314, + 0.0030732500017620623, + 0.0026817080070031807, + 0.00255687499884516, + 0.002528166995034553, + 0.002551875004428439, + 0.0025661249965196475, + 0.002552541991462931, + 0.0023204589961096644, + 0.0022347089980030432, + 0.0021650419948855415, + 0.002159333002055064, + 0.0020336669986136258, + 0.0020402499940246344, + 0.0020774169970536605, + 0.0021850000048289075, + 0.0018043340096483007, + 0.001963375005288981, + 0.0019820419984171167, + 0.0019607919966802, + 0.0019775830005528405, + 0.00198320799972862, + 0.0019325000030221418, + 0.0018432920041959733, + 0.00945700000738725, + 0.005585457998677157, + 0.003010500004165806, + 0.003253875009249896, + 0.0038399589975597337, + 0.0018448329938109964, + 0.0018029159982688725, + 0.0018592910055303946, + 0.001881666001281701, + 0.001917125002364628, + 0.001932915998622775, + 0.0018645000091055408, + 0.0018917080014944077, + 0.0018806250009220093, + 0.0019275000086054206, + 0.0019113339949399233, + 0.001875624991953373, + 0.0018849170010071248, + 0.0019250000041211024, + 0.001862625009380281, + 0.0018893750093411654, + 0.0018627499957801774, + 0.0018693330057431012, + 0.0018562920013209805, + 0.0018959579902002588, + 0.001861124997958541 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 1.5377920062746853, + "p95": 1.5435413923114538, + "min": 1.3997499918332323, + "max": 1.5443329903064296, + "iqr": 0.03512500552460551, + "mean": 1.5054999967105687, + "cv": 0.04060806261550587 + }, + "decode_ms_per_step": { + "count": 1115, + "median": 1.7591669893590733, + "p95": 2.0227374028763734, + "min": 1.381582987960428, + "max": 2.2740830027032644, + "iqr": 0.15754149353597313, + "mean": 1.763646369379728, + "cv": 0.08238093167520712 + }, + "total_ms": { + "count": 5, + "median": 383.6360550048994, + "p95": 423.7196711153956, + "min": 380.6027130485745, + "max": 430.4301699012285, + "iqr": 14.43108805688098, + "mean": 394.7986403683899, + "cv": 0.053019957019963794 + }, + "tokens_per_second": { + "count": 5, + "median": 583.8867256549682, + "p95": 587.9727070496493, + "min": 520.4096173170241, + "max": 588.5402082549316, + "iqr": 21.297058974936817, + "mean": 568.5889793418055, + "cv": 0.05023737086588878 + }, + "median_step_ms_by_index": [ + 1.5377920062746853, + 1.4895420026732609, + 1.517750002676621, + 1.502917002653703, + 1.4427499991143122, + 1.5354169881902635, + 1.5373339992947876, + 1.531832996988669, + 1.5242090012179688, + 1.5463330055354163, + 1.4844999968772754, + 1.5318330115405843, + 1.5370829933090135, + 1.5368329914053902, + 1.551666995510459, + 1.4789999986533076, + 1.5799579996382818, + 1.6054589941632003, + 1.6172080067917705, + 1.5520409942837432, + 1.5425409947056323, + 1.5975839924067259, + 1.596041998709552, + 1.592999993590638, + 1.570292006363161, + 1.5790420002304018, + 1.5411249914905056, + 1.5651249996153638, + 1.5915830008452758, + 1.5699579962529242, + 1.5537079889327288, + 1.5936249983496964, + 1.6012920095818117, + 1.6918339970288798, + 1.6389999946113676, + 1.6727910115150735, + 1.6518749907845631, + 1.6461670020362362, + 1.657917004195042, + 1.6012499982025474, + 1.6275420057354495, + 1.652582999668084, + 1.618292008060962, + 1.6617090004729107, + 1.5900839935056865, + 1.6204160056076944, + 1.6949160053627566, + 1.6679580003255978, + 1.6761669976403937, + 1.7464170086896047, + 1.6389999946113676, + 1.6944169910857454, + 1.5984579949872568, + 1.6241249977611005, + 1.662124996073544, + 1.6853749984875321, + 1.6768750065239146, + 1.7479170055594295, + 1.764957996783778, + 1.759999999194406, + 1.7265420028707013, + 1.6803749895188957, + 1.7210419900948182, + 1.7259579908568412, + 1.702709007076919, + 1.8213339935755357, + 1.8141250038752332, + 1.7302909982390702, + 1.771583003574051, + 1.8284590041730553, + 1.78041699109599, + 1.76304200431332, + 1.7973340000025928, + 1.8467500049155205, + 1.8465839966665953, + 1.8166250083595514, + 1.7684589984128252, + 1.800750003894791, + 1.7527499876450747, + 1.8189999973401427, + 1.7786249954951927, + 1.819249999243766, + 1.8102909962181002, + 1.8712500022957101, + 1.8468330090399832, + 1.7952920024981722, + 1.8048339989036322, + 1.763790991390124, + 1.7516250081826001, + 1.767459005350247, + 1.8019590061157942, + 1.8644170049810782, + 1.8797499942593277, + 1.8624590011313558, + 1.8086249910993502, + 1.7903749976539984, + 1.7452920001232997, + 1.7876250058179721, + 1.8149589886888862, + 1.8173749995185062, + 1.8263749952893704, + 1.808375003747642, + 1.769541995599866, + 1.8067079945467412, + 1.8145420035580173, + 1.7778339970391244, + 1.8067499913740903, + 1.7086660081986338, + 1.7570000054547563, + 1.7928750021383166, + 1.8724170076893643, + 1.910040999064222, + 1.8900419963756576, + 1.82154199865181, + 1.791291986592114, + 1.7959590040845796, + 1.7899579979712144, + 1.833749993238598, + 1.7852919991128147, + 1.796167009160854, + 1.7761670023901388, + 1.7680840101093054, + 1.738874998409301, + 1.7925830034073442, + 1.7537919920869172, + 1.792416995158419, + 1.7646670021349564, + 1.8067909986712039, + 1.7847079870989546, + 1.74004198925104, + 1.7865840054582804, + 1.7255840066354722, + 1.8066669872496277, + 1.78958399919793, + 1.7780839989427477, + 1.81387500197161, + 1.9173749897163361, + 1.7591670039109886, + 1.8434999947203323, + 1.785749991540797, + 1.8215409945696592, + 1.741125000989996, + 1.7709999956423417, + 1.7880420055007562, + 1.8408750038361177, + 1.745125002344139, + 1.8017499969573691, + 1.8160420004278421, + 1.8346249999012798, + 1.7844579997472465, + 1.817541997297667, + 1.7897080106195062, + 1.7985419981414452, + 1.785125001333654, + 1.8298329960089177, + 1.847375009674579, + 1.7777500033844262, + 1.8065420008497313, + 1.7959169927053154, + 1.8322079995414242, + 1.737541999318637, + 1.7867090064100921, + 1.796750002540648, + 1.7836249899119139, + 1.7185420001624152, + 1.7727910017129034, + 1.8345000135013834, + 1.7774169973563403, + 1.7918749945238233, + 1.8094159895554185, + 1.7806249961722642, + 1.7523340065963566, + 1.7773750005289912, + 1.7673330003162846, + 1.7118329997174442, + 1.8296249909326434, + 1.8550829990999773, + 1.7816249892348424, + 1.8045420001726598, + 1.7696249997243285, + 1.8021660071099177, + 1.784791995305568, + 1.820000004954636, + 1.7135829984908924, + 1.866667007561773, + 1.8214580049971119, + 1.7994579975493252, + 1.832041991292499, + 1.7947500018635765, + 1.7142080032499507, + 1.7647089989623055, + 1.7994169902522117, + 1.7499160021543503, + 1.785125001333654, + 1.7554170044604689, + 1.7949159955605865, + 1.806832995498553, + 1.7985419981414452, + 1.8016670073848218, + 1.7720830073812976, + 1.7205830081366003, + 1.7821669898694381, + 1.7642500024521723, + 1.8561670003691688, + 1.79237499833107, + 1.75183400278911, + 1.8048749916488305, + 1.7772499995771796, + 1.7242080066353083, + 1.7360839992761612, + 1.7315000004600734, + 1.787834000424482, + 1.8012090004049242, + 1.793749994249083, + 1.8204170046374202, + 1.8625410011736676, + 1.7597500118426979, + 1.79220900463406, + 1.785749991540797, + 1.7448329890612513, + 1.7167909973068163, + 1.7641669983277097, + 1.8353330087848008, + 1.8077079876093194 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0013997499918332323, + 0.0014369159907801077, + 0.001517750002676621, + 0.0015272080054273829, + 0.0013921250065322965, + 0.0014370840071933344, + 0.0015436670073540881, + 0.001531832996988669, + 0.001493917006882839, + 0.0014485419960692525, + 0.0014844999968772754, + 0.0015349159948527813, + 0.0014659160078736022, + 0.0015368329914053902, + 0.0015394999936688691, + 0.001381582987960428, + 0.0015104170015547425, + 0.0016054589941632003, + 0.0015837500104680657, + 0.0016031670093070716, + 0.0015515419945586473, + 0.0014877079956931993, + 0.0015622080100001767, + 0.0015839999978197739, + 0.00157108400890138, + 0.001508207991719246, + 0.0014531250053551048, + 0.0015517919964622706, + 0.0015915830008452758, + 0.0015699579962529242, + 0.0015149590035434812, + 0.00168058299459517, + 0.0016121670050779358, + 0.0017058340017683804, + 0.0016389999946113676, + 0.0016727910115150735, + 0.0016101250075735152, + 0.0015959159936755896, + 0.001657917004195042, + 0.0016012499982025474, + 0.0015658750053262338, + 0.0015572920092381537, + 0.0015682500088587403, + 0.0016617090004729107, + 0.0016960829962044954, + 0.00169237500813324, + 0.0016011660045478493, + 0.001618667010916397, + 0.0016495840100105852, + 0.0017464170086896047, + 0.001582083001267165, + 0.0016944169910857454, + 0.0016762919985922053, + 0.0016439589962828904, + 0.001662124996073544, + 0.0017104589933296666, + 0.0017429580038879067, + 0.0017822910012910143, + 0.0016299999988405034, + 0.0016290409985231236, + 0.0016319169953931123, + 0.001647583005251363, + 0.001744166002026759, + 0.001677167005254887, + 0.0016676659870427102, + 0.0017418749921489507, + 0.001850291999289766, + 0.0017299160099355504, + 0.001712416997179389, + 0.0017486249998910353, + 0.0017218339926330373, + 0.0017106250015785918, + 0.0018557910079834983, + 0.0018933749961433932, + 0.0018674169987207279, + 0.0017999169940594584, + 0.001719790991046466, + 0.001800750003894791, + 0.0017359579942421988, + 0.0017684589984128252, + 0.0017362500075250864, + 0.00167966699518729, + 0.0018102909962181002, + 0.0018712500022957101, + 0.0018468330090399832, + 0.001679624998359941, + 0.001800624988391064, + 0.001763790991390124, + 0.0017302909982390702, + 0.001735583005938679, + 0.0018019590061157942, + 0.0018644170049810782, + 0.0018797499942593277, + 0.0018624590011313558, + 0.0018086249910993502, + 0.0017903749976539984, + 0.0018050840008072555, + 0.001718458006507717, + 0.001720416999887675, + 0.0018173749995185062, + 0.0017474999913247302, + 0.0017562499997438863, + 0.001769541995599866, + 0.0017508750024717301, + 0.0018145420035580173, + 0.0017778339970391244, + 0.0018067499913740903, + 0.0017086660081986338, + 0.001756999990902841, + 0.0017928750021383166, + 0.0018040839931927621, + 0.001733291006530635, + 0.0017591669893590733, + 0.0017033329932019114, + 0.0018430409982101992, + 0.0017943340062629431, + 0.00170041699311696, + 0.0017034579941537231, + 0.0017559170082677156, + 0.001796167009160854, + 0.0017689999949652702, + 0.0017680840101093054, + 0.001738874998409301, + 0.0017925830034073442, + 0.0017219169967574999, + 0.0017120410047937185, + 0.0016765000036684796, + 0.0016547919949516654, + 0.0017847079870989546, + 0.00174004198925104, + 0.0017865840054582804, + 0.0017255840066354722, + 0.0016894169966690242, + 0.001772207993781194, + 0.0017074170027626678, + 0.0017076250078389421, + 0.0017247079958906397, + 0.0017591670039109886, + 0.0020063340052729473, + 0.0017506250005681068, + 0.0017907500005094334, + 0.001741125000989996, + 0.0016507079999428242, + 0.0017267920047743246, + 0.001739457991789095, + 0.0017312500131083652, + 0.001737166996463202, + 0.001848750005592592, + 0.0018346249999012798, + 0.0017844579997472465, + 0.0018005839956458658, + 0.0017259580054087564, + 0.0018187089881394058, + 0.001733917000819929, + 0.0017270419921260327, + 0.0017439580115024, + 0.0017777500033844262, + 0.0018065420008497313, + 0.0017959169927053154, + 0.0017926250002346933, + 0.0017515840008854866, + 0.0018065420008497313, + 0.0016972500015981495, + 0.0017089169996324927, + 0.0017147500038845465, + 0.0017176670080516487, + 0.0018345000135013834, + 0.0017774169973563403, + 0.0017918749945238233, + 0.0018000830023083836, + 0.0016951249999692664, + 0.0016705840098438784, + 0.001734457997372374, + 0.0017267079965677112, + 0.0017118329997174442, + 0.001727084003505297, + 0.0018712909950409085, + 0.0017816249892348424, + 0.0018022910080617294, + 0.0017344160005450249, + 0.001708957992377691, + 0.0017127080063801259, + 0.0017288340022787452, + 0.001680041998042725, + 0.001797542005078867, + 0.0018332919862587005, + 0.0017994579975493252, + 0.0018104580085491762, + 0.0017947500018635765, + 0.0017006669950205833, + 0.0016840839962242171, + 0.0016757920093368739, + 0.0016684580041328445, + 0.001785125001333654, + 0.0018845839949790388, + 0.0017949159955605865, + 0.001806832995498553, + 0.0017985419981414452, + 0.0017197499983012676, + 0.0017441659874748439, + 0.0016886669909581542, + 0.0017054170020855963, + 0.0017642500024521723, + 0.0018561670003691688, + 0.00179237499833107, + 0.0016385000053560361, + 0.0019082500075455755, + 0.0017401670047547668, + 0.0016466250090161338, + 0.0017360839992761612, + 0.0017214169929502532, + 0.001787834000424482, + 0.0018012090004049242, + 0.001703291985904798, + 0.0019562499946914613, + 0.0017952920024981722, + 0.0017873340111691505, + 0.0017295840079896152, + 0.001785749991540797, + 0.0017448329890612513, + 0.0016980420041363686, + 0.001708209005300887, + 0.0018353330087848008, + 0.0017925839929375798 + ], + [ + 0.0015052499948069453, + 0.0014895420026732609, + 0.0014285840006778017, + 0.0014375419996213168, + 0.001508834000560455, + 0.0015381660050479695, + 0.0014427910064114258, + 0.0014321670023491606, + 0.0015242090012179688, + 0.0015776250074850395, + 0.0015663749945815653, + 0.001472582996939309, + 0.0015370829933090135, + 0.001441750006051734, + 0.0015233329904731363, + 0.0014364160015247762, + 0.0015710000006947666, + 0.0014760409976588562, + 0.0015478749992325902, + 0.0015520409942837432, + 0.0015425409947056323, + 0.0015696250047767535, + 0.001596041998709552, + 0.001592999993590638, + 0.0014707920054206625, + 0.0015790420002304018, + 0.0015411249914905056, + 0.0015934580005705357, + 0.0015066249907249585, + 0.0015267080016201362, + 0.0016856250003911555, + 0.0015936249983496964, + 0.0015987500082701445, + 0.001545708000776358, + 0.0015549999952781945, + 0.0016603749973000959, + 0.0016518749907845631, + 0.0016461670020362362, + 0.0014858750073472038, + 0.001581208998686634, + 0.0016199160018004477, + 0.001646292002988048, + 0.001488583002355881, + 0.001822499994887039, + 0.0015900839935056865, + 0.0015554589917883277, + 0.0016949160053627566, + 0.0016679580003255978, + 0.0016761669976403937, + 0.001824166000005789, + 0.001610707986401394, + 0.0019005419890163466, + 0.0015984579949872568, + 0.001599834009539336, + 0.0016049999976530671, + 0.0015534170088358223, + 0.0016768750065239146, + 0.0016555000038351864, + 0.0016276249953079969, + 0.001786833003279753, + 0.0017619999998714775, + 0.0017350830021314323, + 0.0017210419900948182, + 0.0017259579908568412, + 0.001702709007076919, + 0.001829707995057106, + 0.0017742079944582656, + 0.0017302909982390702, + 0.0017115829978138208, + 0.0019545419927453622, + 0.0018010409985436127, + 0.001803624996682629, + 0.0017973340000025928, + 0.0017039999947883189, + 0.001766916990163736, + 0.0017328749963780865, + 0.0017302500054938719, + 0.0017444170080125332, + 0.0017527499876450747, + 0.0018189999973401427, + 0.0017786249954951927, + 0.001819249999243766, + 0.0017584170127520338, + 0.0017819169879658148, + 0.0017577499966137111, + 0.0017349170084344223, + 0.0017293749988311902, + 0.001733917000819929, + 0.0017516250081826001, + 0.0018315000052098185, + 0.0017217919958056882, + 0.001786875000107102, + 0.001812333008274436, + 0.0017318750033155084, + 0.0017509169992990792, + 0.0016720420098863542, + 0.0017130000051110983, + 0.0017876250058179721, + 0.0018149589886888862, + 0.0016118330095196143, + 0.001826874999096617, + 0.001808375003747642, + 0.0017281250038649887, + 0.0018067079945467412, + 0.0017816670006141067, + 0.0017652079986874014, + 0.001713707999442704, + 0.0016922920040087774, + 0.0019037079910049215, + 0.0017768750112736598, + 0.001782874998752959, + 0.001734082994516939, + 0.0017646250053076074, + 0.0017273749981541187, + 0.0017551659984746948, + 0.0017007499991450459, + 0.0017204579926328734, + 0.001833749993238598, + 0.0017852919991128147, + 0.001780165999662131, + 0.001699790998827666, + 0.001654499996220693, + 0.0017121669952757657, + 0.0017022079991875216, + 0.0017537919920869172, + 0.001792416995158419, + 0.0018459579878253862, + 0.0018067909986712039, + 0.0016787499916972592, + 0.0016906249948078766, + 0.0017158749978989363, + 0.0017087079904740676, + 0.0017700000025797635, + 0.0018029170023510233, + 0.0017797920008888468, + 0.0018138750019716099, + 0.0019173749897163361, + 0.001790250011254102, + 0.0017848749994300306, + 0.0017268339870497584, + 0.0016485000087413937, + 0.0016028340032789856, + 0.0017395840113749728, + 0.0017880420055007562, + 0.0017849169962573797, + 0.001745125002344139, + 0.0018017499969573691, + 0.0017198330024257302, + 0.0017614579992368817, + 0.0017227079952135682, + 0.0016964579990599304, + 0.0017494170024292544, + 0.0017985419981414452, + 0.001785125001333654, + 0.0018298329960089177, + 0.001847375009674579, + 0.0016959579952526838, + 0.0017152090003946796, + 0.0017356669995933771, + 0.0016692920034984127, + 0.0016922499926295131, + 0.0017867090064100921, + 0.001796750002540648, + 0.0017836249899119139, + 0.001854832997196354, + 0.0017444580007577315, + 0.0016726670000934973, + 0.0017089580069296062, + 0.0017035829951055348, + 0.0016711670032236725, + 0.0017839999927673489, + 0.0018092090031132102, + 0.001782457999070175, + 0.0017616249970160425, + 0.0016790420049801469, + 0.0018303749966435134, + 0.0017464590055169538, + 0.001718083003652282, + 0.0017094169888878241, + 0.0017958750104298815, + 0.0018021660071099177, + 0.001784791995305568, + 0.0017059169913409278, + 0.0016730000061215833, + 0.0019734590023290366, + 0.00174245800008066, + 0.0016984590038191527, + 0.0017032089963322505, + 0.0017383749946020544, + 0.001702709007076919, + 0.0017647089989623055, + 0.0017994169902522117, + 0.0017499160021543503, + 0.0017738330061547458, + 0.0017273749981541187, + 0.0017530420009279624, + 0.0017851670127129182, + 0.0016771249938756227, + 0.001764207991072908, + 0.0017329580005025491, + 0.0017205830081366003, + 0.0016828340012580156, + 0.0017386249965056777, + 0.0017543750000186265, + 0.001706416005617939, + 0.00175183400278911, + 0.0017132079956354573, + 0.0017955420044017956, + 0.0016832080000312999, + 0.0017147499893326312, + 0.0017315000004600734, + 0.0017217919958056882, + 0.0017687079962342978, + 0.0017947919986909255, + 0.001777875004336238, + 0.0017846660048235208, + 0.0017597500118426979, + 0.00179220900463406, + 0.001728041999740526, + 0.0017154170054709539, + 0.0017260000022361055, + 0.00171050000062678, + 0.00183275000017602, + 0.0017710829997668043 + ], + [ + 0.0015403750003315508, + 0.0015740840026410297, + 0.0014807919942541048, + 0.0015022500010672957, + 0.0013855830038664863, + 0.001526667008874938, + 0.001411709003150463, + 0.001437750004697591, + 0.0015113340050447732, + 0.0015463330055354163, + 0.0014715839934069663, + 0.0014523340068990365, + 0.0015510420053033158, + 0.0015618750039720908, + 0.001551666995510459, + 0.0014789999986533076, + 0.0015890420036157593, + 0.001591459003975615, + 0.0016172080067917705, + 0.0014593750092899427, + 0.001418707994162105, + 0.001611667001270689, + 0.001588041996001266, + 0.0015958750009303913, + 0.001478916994528845, + 0.0015074580005602911, + 0.0014677909930469468, + 0.0015264169924193993, + 0.001528417007648386, + 0.0014376659964909777, + 0.001463416003389284, + 0.0015706249978393316, + 0.0015471660008188337, + 0.0015860000130487606, + 0.001646292002988048, + 0.0016455829900223762, + 0.0016434169956482947, + 0.001617707996047102, + 0.0015874580130912364, + 0.0015579589962726459, + 0.0016275420057354495, + 0.001652582999668084, + 0.001618292008060962, + 0.001566874998388812, + 0.0015836249949643388, + 0.0015947499923640862, + 0.001698791005765088, + 0.0015761249960632995, + 0.0016691659984644502, + 0.0015365829895017669, + 0.0016721660067560151, + 0.0016158750077011064, + 0.0015899580030236393, + 0.001577749993884936, + 0.001611084007890895, + 0.0016427500086138025, + 0.0016432080010417849, + 0.0017452920001232997, + 0.0017840829968918115, + 0.001733166995109059, + 0.0017265420028707013, + 0.0016702499997336417, + 0.00169070900301449, + 0.0016375830018660054, + 0.0016995000041788444, + 0.0018213339935755357, + 0.0018141250038752332, + 0.001617583999177441, + 0.001771583003574051, + 0.0017484999989392236, + 0.0017512920021545142, + 0.00176304200431332, + 0.0017673749971436337, + 0.0017809999990276992, + 0.0017812919977586716, + 0.0018166250083595514, + 0.0017684589984128252, + 0.0017599169950699434, + 0.0017537910025566816, + 0.0016714999946998432, + 0.0017337080062134191, + 0.0017869590083137155, + 0.0017897090001497418, + 0.0018054169922834262, + 0.0017128749896073714, + 0.0017952920024981722, + 0.0018048339989036322, + 0.0018216250027762726, + 0.0017506250005681068, + 0.00173554100911133, + 0.0017607079935260117, + 0.0017460840026615188, + 0.0017809580022003502, + 0.0018269580032210797, + 0.0017560410051373765, + 0.0016532499867025763, + 0.0017097920062951744, + 0.0016231670015258715, + 0.0016190419992199168, + 0.0016097920015454292, + 0.0018049999926006421, + 0.0016281670104945078, + 0.0015892080118646845, + 0.0016236660012509674, + 0.0016410000098403543, + 0.0016211670008488, + 0.0017059999954653904, + 0.0016530830034753308, + 0.0016832500114105642, + 0.0016890830011107028, + 0.0018724170076893643, + 0.0020749169925693423, + 0.0018900419963756576, + 0.00182154199865181, + 0.001791291986592114, + 0.0017959590040845796, + 0.0017899579979712144, + 0.0017309160029981285, + 0.0017727499944157898, + 0.0017289999959757552, + 0.0017761670023901388, + 0.0017466669960413128, + 0.0016296249959850684, + 0.0016372499958379194, + 0.001713374993414618, + 0.0016614999913144857, + 0.0017263750050915405, + 0.001721125008771196, + 0.0016864580102264881, + 0.0017335840093437582, + 0.0017618749989196658, + 0.0017004999972414225, + 0.001821917001507245, + 0.00178958399919793, + 0.0017780839989427477, + 0.0017827080009737983, + 0.0017077499942388386, + 0.0017190420039696619, + 0.0017372919974150136, + 0.001785749991540797, + 0.0018473329982953146, + 0.0016899579932214692, + 0.0017709999956423417, + 0.00171066599432379, + 0.0018523750040913, + 0.0017372500005876645, + 0.0017014579934766516, + 0.0017038749938365072, + 0.001702875000773929, + 0.0017750419938238338, + 0.001817541997297667, + 0.0017897080106195062, + 0.0017858329956652597, + 0.0017923330015037209, + 0.0017846670089056715, + 0.001668999990215525, + 0.0017085419967770576, + 0.0017436670023016632, + 0.0017778749897843227, + 0.0018322079995414242, + 0.0016285839956253767, + 0.0017787499964470044, + 0.0017975410009967163, + 0.0018284999969182536, + 0.00167350000992883, + 0.0017727910017129034, + 0.0017656670097494498, + 0.0017279160092584789, + 0.0020366249955259264, + 0.0018094159895554185, + 0.0016858749877428636, + 0.0017055829957826063, + 0.0017773750005289912, + 0.0018037080008070916, + 0.0017105419974541292, + 0.0017102919955505058, + 0.001742833002936095, + 0.0017812919977586716, + 0.0018309169972781092, + 0.0017696249997243285, + 0.0018005000019911677, + 0.0016736249963287264, + 0.001820000004954636, + 0.0017135829984908924, + 0.0017177920090034604, + 0.0016754579992266372, + 0.001718833009363152, + 0.001832041991292499, + 0.0018058330024359748, + 0.0017911670001922175, + 0.0017420830117771402, + 0.0017372909933328629, + 0.0017174160020658746, + 0.0016679589898558334, + 0.001717374994768761, + 0.0017350410053040832, + 0.001836374998674728, + 0.0018703749956330284, + 0.0018016670073848218, + 0.0018009999912464991, + 0.0016854999994393438, + 0.0017821669898694381, + 0.001655208005104214, + 0.001674458006164059, + 0.0016970410069916397, + 0.001699790998827666, + 0.0018048749916488305, + 0.0017772499995771796, + 0.0017242080066353083, + 0.001713374993414618, + 0.0017221669986611232, + 0.001731666998239234, + 0.0016942079964792356, + 0.001768000001902692, + 0.0018204170046374202, + 0.0018625410011736676, + 0.00170358300965745, + 0.0018019590061157942, + 0.0017024590051732957, + 0.001670292011112906, + 0.0016725000023143366, + 0.0017641669983277097, + 0.001794083000277169, + 0.0018077079876093194 + ], + [ + 0.0015377920062746853, + 0.0014809590065851808, + 0.0015831249911570922, + 0.001502917002653703, + 0.0014427499991143122, + 0.0015681249933550134, + 0.0015373339992947876, + 0.001596458998392336, + 0.0015909579960862175, + 0.0014929589960956946, + 0.0014833329914836213, + 0.0015318330115405843, + 0.0014818750059930608, + 0.0015199159970507026, + 0.0015925420011626557, + 0.0016069159901235253, + 0.0015799579996382818, + 0.001606874997378327, + 0.0016589589940849692, + 0.0015256670012604445, + 0.0015176249871728942, + 0.0015975839924067259, + 0.0016372079990105703, + 0.0015449169877683744, + 0.001570292006363161, + 0.0016650839970679954, + 0.0016432910051662475, + 0.0015651249996153638, + 0.0016026660014176741, + 0.0016246250015683472, + 0.0015537079889327288, + 0.0015167499950621277, + 0.0016012920095818117, + 0.0016918339970288798, + 0.00155237500439398, + 0.0017157080001197755, + 0.001698290987405926, + 0.0017382079968228936, + 0.0016754169919295236, + 0.0016718330007279292, + 0.0017305829969700426, + 0.0017053750052582473, + 0.0016881249903235584, + 0.0016424589994130656, + 0.0015535419952357188, + 0.0016204160056076944, + 0.0016547909908695146, + 0.0016769590001786128, + 0.0017058749945135787, + 0.0016850829997565597, + 0.0016389999946113676, + 0.0015506249910686165, + 0.0015894999960437417, + 0.0016241249977611005, + 0.0017520840046927333, + 0.0016853749984875321, + 0.0016669580072630197, + 0.0017479170055594295, + 0.0018154999997932464, + 0.001759999999194406, + 0.001701707995380275, + 0.0016803749895188957, + 0.001632915998925455, + 0.0017940000107046217, + 0.001759042002959177, + 0.0017800409987103194, + 0.001797917007934302, + 0.0018774579948512837, + 0.0018222090002382174, + 0.0018284590041730553, + 0.00178041699109599, + 0.0017230829980690032, + 0.0017874580080388114, + 0.0018467500049155205, + 0.0018912500090664253, + 0.0018638750043464825, + 0.001858667004853487, + 0.0018390000041108578, + 0.0017047090077539906, + 0.0018292500026291236, + 0.0018404580041533336, + 0.0018417920073261485, + 0.0018165420042350888, + 0.0018886250036302954, + 0.0018497919954825193, + 0.0018989999953191727, + 0.0018308750004507601, + 0.0017404590034857392, + 0.0017650840018177405, + 0.001767459005350247, + 0.0018218749901279807, + 0.001969874996575527, + 0.0019224590068915859, + 0.0019252089987276122, + 0.001862709003034979, + 0.0018443329900037497, + 0.0017452920001232997, + 0.0018143340130336583, + 0.0018392920028418303, + 0.001820416990085505, + 0.0018263749952893704, + 0.0018496669945307076, + 0.0018168749957112595, + 0.0018829579930752516, + 0.0019052499992540106, + 0.0017915829957928509, + 0.0018320830131415278, + 0.0017809589917305857, + 0.0017570000054547563, + 0.0019096250034635887, + 0.0018749590090010315, + 0.001910040999064222, + 0.0019037919992115349, + 0.0018332500039832667, + 0.0017835829930845648, + 0.001837125004385598, + 0.0017992909997701645, + 0.0018468330090399832, + 0.0019788750068983063, + 0.0018505829939385876, + 0.0018442919972585514, + 0.0018764590058708563, + 0.0018053750100079924, + 0.0018814159993780777, + 0.001779208003426902, + 0.0018145420035580173, + 0.0017646670021349564, + 0.0019467919919406995, + 0.0018290410080226138, + 0.0018798749952111393, + 0.0019118750060442835, + 0.0019038750033359975, + 0.0018066669872496277, + 0.0017760409973561764, + 0.0017541669949423522, + 0.001836999988881871, + 0.002055000004475005, + 0.0017330830014543608, + 0.0018434999947203323, + 0.0018469579954398796, + 0.0018215409945696592, + 0.0018350419995840639, + 0.001817166994442232, + 0.0018539580050855875, + 0.0018408750038361177, + 0.0019782090093940496, + 0.001830250010243617, + 0.0018160420004278421, + 0.0018686669936869293, + 0.0018861669959733263, + 0.0019187500001862645, + 0.0018405000009806827, + 0.0017751250125002116, + 0.0017486249998910353, + 0.001876750000519678, + 0.0019006249931408092, + 0.0019008749950444326, + 0.0019039159960811958, + 0.0018374170031165704, + 0.0018937499989988282, + 0.001737541999318637, + 0.0016604999982519075, + 0.0017826660041464493, + 0.0017324579966953024, + 0.0017185420001624152, + 0.0017909169982885942, + 0.001908957987325266, + 0.0019323339947732165, + 0.0017850829899543896, + 0.0018416659877402708, + 0.0017806249961722642, + 0.0017523340065963566, + 0.0017191250080941245, + 0.0017673330003162846, + 0.0019269589975010604, + 0.0018296249909326434, + 0.0018550829990999773, + 0.0018194589938502759, + 0.0018045420001726598, + 0.001742874999763444, + 0.0018708749994402751, + 0.0019268750038463622, + 0.0018781670078169554, + 0.001833917005569674, + 0.001866667007561773, + 0.0018214580049971119, + 0.0018587500089779496, + 0.0018745830020634457, + 0.001744416993460618, + 0.0017142080032499507, + 0.0018171249976148829, + 0.0018397910025669262, + 0.0018898749985964969, + 0.0018888339982368052, + 0.0017554170044604689, + 0.001869500003522262, + 0.0017926670116139576, + 0.0017515420040581375, + 0.0018100409943144768, + 0.0017720830073812976, + 0.0018470829963916913, + 0.0018098330037901178, + 0.001876333000836894, + 0.001898457994684577, + 0.0018853749934351072, + 0.0017974160000449046, + 0.0017954580107470974, + 0.0017462919931858778, + 0.0018123329937225208, + 0.0017848340066848323, + 0.0019123340025544167, + 0.0018468750058673322, + 0.001836750001530163, + 0.001793749994249083, + 0.0018067500059260055, + 0.001910292005049996, + 0.001712416997179389, + 0.0017043750121956691, + 0.0018310419982299209, + 0.0018308339931536466, + 0.0017167909973068163, + 0.0020139999978709966, + 0.0019097500044154003, + 0.0019242499984102324 + ], + [ + 0.0015443329903064296, + 0.0015334170020651072, + 0.001564999998663552, + 0.0015531669923802838, + 0.0014958749961806461, + 0.0015354169881902635, + 0.0015999159950297326, + 0.0015731250023236498, + 0.0015419169940287247, + 0.0015797090018168092, + 0.0016115840116981417, + 0.001616416993783787, + 0.0016120419895742089, + 0.0016167080029845238, + 0.0016570839943597093, + 0.0016445419896626845, + 0.00165125000057742, + 0.001677332998951897, + 0.0016230420005740598, + 0.0016716660029487684, + 0.0016579169896431267, + 0.001657166998484172, + 0.0016497920005349442, + 0.0019022079941350967, + 0.0017430000007152557, + 0.0017395839968230575, + 0.0017334579897578806, + 0.0017444590048398823, + 0.0017069579917006195, + 0.0016717500111553818, + 0.0017134590016212314, + 0.0017323339998256415, + 0.001709666001261212, + 0.0018525000050431117, + 0.0017360420024488121, + 0.0017886249988805503, + 0.0017929169989656657, + 0.0017351249989587814, + 0.001745125002344139, + 0.001699084008578211, + 0.001748374997987412, + 0.001862499993876554, + 0.0017944999999599531, + 0.0018101669993484393, + 0.001760083992849104, + 0.0017473750049248338, + 0.0017957080126507208, + 0.001896791989565827, + 0.0017631669907132164, + 0.0017687919898889959, + 0.0017333750001853332, + 0.001774124990333803, + 0.0017109170003095642, + 0.001819249999243766, + 0.0018366669974057004, + 0.001844041995354928, + 0.001796374999685213, + 0.0017930000030901283, + 0.001764957996783778, + 0.001820749996113591, + 0.001865875005023554, + 0.0018924589967355132, + 0.001931958002387546, + 0.0019264580041635782, + 0.0019158329960191622, + 0.0019686669984366745, + 0.0019146249978803098, + 0.0019127910054521635, + 0.0019073339935857803, + 0.0019745000026887283, + 0.0019565419934224337, + 0.001973167003598064, + 0.001975207997020334, + 0.00196358299581334, + 0.0018465839966665953, + 0.0019443339988356456, + 0.0019434580026427284, + 0.001959165994776413, + 0.001952542006620206, + 0.0020023329998366535, + 0.001972332989680581, + 0.0019473749998724088, + 0.0018835000082617626, + 0.00197108399879653, + 0.001919041998917237, + 0.0019674579962156713, + 0.002054249998764135, + 0.001983291993383318, + 0.001975916005903855, + 0.0020202080049784854, + 0.0019227500015404075, + 0.002056666009593755, + 0.001970249999430962, + 0.001992165998672135, + 0.0020185829926049337, + 0.001995207989239134, + 0.0019936669996241108, + 0.0019993329915450886, + 0.0019204159907530993, + 0.0019551669975044206, + 0.0019305419991724193, + 0.001909500002511777, + 0.001959250002983026, + 0.0020439590007299557, + 0.001971417004824616, + 0.002043333006440662, + 0.0019185839919373393, + 0.0019022079941350967, + 0.001950250007212162, + 0.0019237499946029857, + 0.001906833000248298, + 0.002002207998884842, + 0.0020400000066729262, + 0.001980707995244302, + 0.0020202919986331835, + 0.0019611659954534844, + 0.002048708003712818, + 0.0020072500046808273, + 0.0019897500023944303, + 0.002099207995343022, + 0.002018166007474065, + 0.0020334169967100024, + 0.0020006249978905544, + 0.0019229169993195683, + 0.001950457997736521, + 0.0019633329939097166, + 0.0019469590042717755, + 0.0019663749990286306, + 0.00202358300157357, + 0.0019897909951396286, + 0.001986250004847534, + 0.0020412079902598634, + 0.0019288750045234337, + 0.0019446660007815808, + 0.0019265830051153898, + 0.0019479170005070046, + 0.0019757499976549298, + 0.0019991250010207295, + 0.0020266659994376823, + 0.001988000003620982, + 0.0019092499860562384, + 0.0019179170049028471, + 0.001901499999803491, + 0.0018905830074800178, + 0.0019552919984562322, + 0.0019833340047625825, + 0.0019392090034671128, + 0.0021975409908918664, + 0.002024208995862864, + 0.0019409999949857593, + 0.00200545800908003, + 0.002013542005443014, + 0.0020048750011483207, + 0.0019615830096881837, + 0.00226545799523592, + 0.0020304999925429, + 0.002064749991404824, + 0.002070791000733152, + 0.0020651670056395233, + 0.0020431250013643876, + 0.0020730840042233467, + 0.002141916993423365, + 0.0021255829924484715, + 0.0021281660010572523, + 0.002117166994139552, + 0.0019324999884702265, + 0.0019213750056223944, + 0.0019297919934615493, + 0.0019307080074213445, + 0.0020150829950580373, + 0.0020063340052729473, + 0.0020172090007690713, + 0.0019499580084811896, + 0.0019962500082328916, + 0.0019343329913681373, + 0.0019137909985147417, + 0.0019287909963168204, + 0.001938250003149733, + 0.0020027080026920885, + 0.0020212079980410635, + 0.0020092500053578988, + 0.001987625000765547, + 0.001968957993085496, + 0.001994999998714775, + 0.0019451670086709782, + 0.0019496669992804527, + 0.0020198750135023147, + 0.00209612499747891, + 0.0020747500093420967, + 0.0020456250058487058, + 0.002077083001495339, + 0.0020016249909531325, + 0.0019873340061167255, + 0.001995625003473833, + 0.002016541999182664, + 0.0020366669923532754, + 0.002006833004998043, + 0.0022740830027032644, + 0.0021164589998079464, + 0.0020933749910909683, + 0.002031957992585376, + 0.0020255830022506416, + 0.0020591670036083087, + 0.002210999999078922, + 0.0020509169989963993, + 0.0020488329901127145, + 0.002042124993749894, + 0.002120584002113901, + 0.0020109170000068843, + 0.0019949580018874258, + 0.0020400000066729262, + 0.0020482920081121847, + 0.0020482500112848356, + 0.0021827500022482127, + 0.002062667001155205, + 0.0021066659974167123, + 0.0020223750034347177, + 0.0020836670009884983, + 0.0020550829940475523, + 0.0020888750004814938, + 0.0020804170053452253, + 0.002069083013338968, + 0.002055999997537583, + 0.002091750007821247 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_mha": { + "first_token_ms": { + "count": 5, + "median": 1.728291012113914, + "p95": 1.8189581925980747, + "min": 1.6554160101804882, + "max": 1.832957990700379, + "iqr": 0.06470899097621441, + "mean": 1.7355748044792563, + "cv": 0.03876376901064423 + }, + "decode_ms_per_step": { + "count": 1115, + "median": 1.9092919974355027, + "p95": 2.181241300422698, + "min": 1.462917003664188, + "max": 4.545916002825834, + "iqr": 0.1837285017245449, + "mean": 1.9277657371618881, + "cv": 0.1433587795851763 + }, + "total_ms": { + "count": 5, + "median": 432.32287088176236, + "p95": 460.45646051934455, + "min": 412.40083501907066, + "max": 467.2231678850949, + "iqr": 20.589464940712787, + "mean": 431.6273341915803, + "cv": 0.051737786558450606 + }, + "tokens_per_second": { + "count": 5, + "median": 518.1312743023087, + "p95": 543.0557912147342, + "min": 479.4282805237277, + "max": 543.1608788804746, + "iqr": 25.779512425336293, + "mean": 520.0423604769439, + "cv": 0.05003494590007879 + }, + "median_step_ms_by_index": [ + 1.728291012113914, + 1.7135829984908924, + 1.7033750045811757, + 1.6079999913927168, + 1.6189579910133034, + 1.6362909955205396, + 1.6572080057812855, + 1.5713750035502017, + 1.6332499944837764, + 1.6034169966587797, + 1.699208005447872, + 1.665791991399601, + 1.657374989008531, + 1.6259169933618978, + 1.650874997721985, + 1.6862499905982986, + 1.5992920089047402, + 1.7088750028051436, + 1.707834002445452, + 1.6290420026052743, + 1.7486249998910353, + 1.676750005572103, + 1.6709579940652475, + 1.7329999973298982, + 1.6214589995797724, + 1.713749996270053, + 1.6762089944677427, + 1.6551250009797513, + 1.665666000917554, + 1.719249994494021, + 1.757875012117438, + 1.7212090024258941, + 1.6966660041362047, + 1.7697080038487911, + 1.7654999974183738, + 1.7968329921131954, + 1.748041991959326, + 1.8065829935949296, + 1.7709579988149926, + 1.7667500069364905, + 1.7641250015003607, + 1.7884580011013895, + 1.7019170045387, + 1.6709159972378984, + 1.7510000034235418, + 1.76320799801033, + 1.79237499833107, + 1.7988329927902669, + 1.8532499962020665, + 1.8235000025015324, + 1.7497920052846894, + 1.7492500046500936, + 1.8090839876094833, + 1.8836250092135742, + 1.8413340003462508, + 1.886666999780573, + 1.8322910036658868, + 1.8556669965619221, + 1.833166999858804, + 1.8315829947823659, + 1.7933339986484498, + 1.87404100142885, + 1.8566670041764155, + 1.8650409911060706, + 1.8597080052131787, + 1.8905409961007535, + 1.9080829952144995, + 1.9198749942006543, + 1.9602919928729534, + 1.969666001969017, + 1.9491660059429705, + 1.9335830002091825, + 1.9284159934613854, + 1.8697920022532344, + 1.8579579918878153, + 1.8704999965848401, + 1.8633330037118867, + 1.9053750002058223, + 1.9586249982239679, + 1.9579999934649095, + 1.9317910046083853, + 1.8964999908348545, + 1.8853750079870224, + 2.0190829964121804, + 1.891207997687161, + 2.0059999951627105, + 2.0187500049360096, + 2.0442079985514283, + 1.9983329984825104, + 1.973291000467725, + 1.911458995891735, + 1.939417008543387, + 1.926625001942739, + 1.9484170043142512, + 1.961125002708286, + 2.03237499226816, + 1.9170000014128163, + 2.0076660002814606, + 1.9365000043762848, + 1.9177500071236864, + 1.8345829885220155, + 1.909083002828993, + 1.9804589974228293, + 1.9190000020898879, + 1.9625419954536483, + 1.9840839959215373, + 1.876915994216688, + 1.9098339980700985, + 1.885500008938834, + 1.8969999946421012, + 2.003000001423061, + 2.0039170049130917, + 1.9608750008046627, + 1.9565830007195473, + 1.9907090027118102, + 1.9006249931408092, + 1.9218329980503768, + 1.860749995103106, + 1.9777919951593503, + 1.970666999113746, + 1.9886660011252388, + 1.9405000057304278, + 1.9972919981228188, + 2.077874989481643, + 1.9315419922349975, + 1.8669169949134812, + 1.9631670002127066, + 1.953665996552445, + 1.9470830011414364, + 2.047165995463729, + 1.979042004677467, + 2.0747500093420967, + 1.968750002561137, + 1.9242080015828833, + 1.9242499984102324, + 1.979249995201826, + 1.993875004700385, + 1.9482080097077414, + 1.9993749883724377, + 1.9999170035589486, + 1.9568330026231706, + 1.929334001033567, + 1.9512500002747402, + 2.023749999352731, + 1.9586249982239679, + 1.9340830040164292, + 2.0355419983388856, + 2.009667005040683, + 1.9335420074639842, + 1.9179579976480454, + 1.981499997782521, + 1.9480420014588162, + 1.9596670026658103, + 2.0405830000527203, + 2.0465420093387365, + 2.0500419923337176, + 2.05462500161957, + 1.9764580065384507, + 1.978124986635521, + 1.9399169977987185, + 1.9619999948190525, + 2.0014170004287735, + 1.9827079959213734, + 1.9625420100055635, + 1.9003750057891011, + 1.8867079925257713, + 1.8846249877242371, + 1.908167003421113, + 1.9215829961467534, + 1.9140830117976293, + 1.9229169993195683, + 1.9124170066788793, + 1.9187090074410662, + 1.9485000084387138, + 1.9071670103585348, + 1.9871249969583005, + 1.977707986952737, + 1.9767500052694231, + 1.9786249904427677, + 1.9395419949432835, + 1.9104170060018077, + 1.9632500043371692, + 1.9175829947926104, + 1.988291012821719, + 1.991208002436906, + 2.0040410017827526, + 1.9734169909497723, + 1.9734169909497723, + 1.971958001377061, + 1.9849579985020682, + 1.938625006005168, + 1.982082991162315, + 1.933541992912069, + 1.9964590028394014, + 1.9880409963661805, + 1.9463339995127171, + 1.8397499952698126, + 1.953541999682784, + 1.9732920045498759, + 1.961125002708286, + 1.8929169891634956, + 1.988749994779937, + 1.9783749885391444, + 1.9377089920453727, + 1.9197919900761917, + 1.9920420018024743, + 1.9313340017106384, + 1.9400000019231811, + 1.9182079995516688, + 1.893583990749903, + 1.9722500001080334, + 1.9690000044647604, + 1.9258329994045198, + 1.9775830005528405, + 1.9995839975308627, + 2.0079159876331687, + 1.9899999897461385, + 1.8893339874921367, + 1.9988749991171062, + 1.975042003323324, + 1.9337919948156923, + 1.9914160075131804, + 1.907958008814603, + 1.9170409941580147 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0016554160101804882, + 0.001649208992603235, + 0.0015617079916410148, + 0.0015279170038411394, + 0.0016761250008130446, + 0.0016362909955205396, + 0.0016572080057812855, + 0.0015477499982807785, + 0.001601374999154359, + 0.001636999993934296, + 0.001634083004319109, + 0.0015858749975450337, + 0.0016550410073250532, + 0.0015782089903950691, + 0.001625416989554651, + 0.0016862499905982986, + 0.0015683340025134385, + 0.0016798330034362152, + 0.0016665830044075847, + 0.0016144590044859797, + 0.0015846669994061813, + 0.0016535830072825775, + 0.0016709579940652475, + 0.0016488340042997152, + 0.0016030830011004582, + 0.0015962920006131753, + 0.0016762089944677427, + 0.0016769169887993485, + 0.0016376659914385527, + 0.0016728330083424225, + 0.0017030419985530898, + 0.0016447079979116097, + 0.00165200000628829, + 0.0017697080038487911, + 0.001803459002985619, + 0.0017968329921131954, + 0.001616416993783787, + 0.0018211670103482902, + 0.0019348750065546483, + 0.0017365000094287097, + 0.0017641250015003607, + 0.001753707998432219, + 0.001683624999714084, + 0.001657707995036617, + 0.0017987909959629178, + 0.0017460000090068206, + 0.0018017499969573691, + 0.0018280420044902712, + 0.0016290000057779253, + 0.0019182499963790178, + 0.0017879170045489445, + 0.0017424169927835464, + 0.0018090839876094833, + 0.0019594579935073853, + 0.0018869170016841963, + 0.00189479099935852, + 0.001872917011496611, + 0.0019082500075455755, + 0.001897499998449348, + 0.0019189159938832745, + 0.0018977500003529713, + 0.0018330829916521907, + 0.0018566670041764155, + 0.0019135409966111183, + 0.0018563339981483296, + 0.0018905409961007535, + 0.001896000001579523, + 0.0019028750102734193, + 0.001925290998769924, + 0.0019118330092169344, + 0.0018801249971147627, + 0.0018893340020440519, + 0.0018406250019324943, + 0.0018840410048142076, + 0.0018579579918878153, + 0.0018704999965848401, + 0.0017700839962344617, + 0.001793458010070026, + 0.0020329999970272183, + 0.001969458011444658, + 0.0018910840008175, + 0.0018886670004576445, + 0.001860041986219585, + 0.002047333007794805, + 0.0018669999990379438, + 0.0020059999951627105, + 0.0020284170022932813, + 0.001959249988431111, + 0.0018332500039832667, + 0.0017582089931238443, + 0.001854292000643909, + 0.0018668340053409338, + 0.0018313339969608933, + 0.001796540993382223, + 0.001961125002708286, + 0.0018984580092364922, + 0.0018697080085985363, + 0.0018722920067375526, + 0.001844165992224589, + 0.001740833991789259, + 0.0017230419907718897, + 0.0017812080041039735, + 0.0020376250031404197, + 0.001772000003256835, + 0.0017677079886198044, + 0.0017284999921685085, + 0.001741125000989996, + 0.0018438339902786538, + 0.0018461659929016605, + 0.0018969999946421012, + 0.004545916002825834, + 0.0037239170051179826, + 0.002430624997941777, + 0.0023234999971464276, + 0.0022735409875167534, + 0.0024175839935196564, + 0.00268733300617896, + 0.0026706659991759807, + 0.0027035829989472404, + 0.0028866669890703633, + 0.002783125004498288, + 0.002768124992144294, + 0.0027852920029545203, + 0.002773040992906317, + 0.0029593749932246283, + 0.0027277909975964576, + 0.0026999580004485324, + 0.002684625011170283, + 0.002952708993689157, + 0.0029573749925475568, + 0.0030818749946774915, + 0.003488582995487377, + 0.00313558301422745, + 0.0037323339929571375, + 0.003230083006201312, + 0.0034629159927135333, + 0.003527249995386228, + 0.0034136250033043325, + 0.0038640840066364035, + 0.003562500001862645, + 0.0033971249940805137, + 0.003334124994580634, + 0.0034296669909963384, + 0.003527666995069012, + 0.0034081250050803646, + 0.003387874996406026, + 0.0029992080089868978, + 0.002780084003461525, + 0.0027993749972665682, + 0.002747874998021871, + 0.002692834008485079, + 0.0023839580098865554, + 0.0024582920013926923, + 0.002467750004143454, + 0.002517499990062788, + 0.0022067090030759573, + 0.002184917000704445, + 0.0022185419948073104, + 0.0021796660003019497, + 0.002191916006268002, + 0.0021937080018687993, + 0.00227370900392998, + 0.0017842500092228875, + 0.001979084001504816, + 0.0019003750057891011, + 0.0018867079925257713, + 0.0018804159917635843, + 0.001908167003421113, + 0.0019215829961467534, + 0.001914833002956584, + 0.0019229169993195683, + 0.0019124170066788793, + 0.0019187090074410662, + 0.0019180000090273097, + 0.0019071670103585348, + 0.0019406250066822395, + 0.0019517500040819868, + 0.0019892919954145327, + 0.0019786249904427677, + 0.001859208001405932, + 0.0019029999966733158, + 0.0019632500043371692, + 0.0019021670013898984, + 0.001955333005753346, + 0.002000041989958845, + 0.001981290988624096, + 0.0019734169909497723, + 0.0019734169909497723, + 0.001987999989069067, + 0.002000208987738006, + 0.001938625006005168, + 0.001982082991162315, + 0.0019160830124747008, + 0.0019964590028394014, + 0.0019880409963661805, + 0.0019288749899715185, + 0.0018397499952698126, + 0.0021096249984111637, + 0.0019880420004483312, + 0.001997125000343658, + 0.0018465000030118972, + 0.0019747500045923516, + 0.0019337499979883432, + 0.0019169169972883537, + 0.0018652080034371465, + 0.001994792008190416, + 0.001979042004677467, + 0.0019955839961767197, + 0.0019182079995516688, + 0.0018896659894380718, + 0.0019230829930165783, + 0.00193237500207033, + 0.0019258329994045198, + 0.0019775830005528405, + 0.0019995839975308627, + 0.0020079159876331687, + 0.0019899999897461385, + 0.0018335839995415881, + 0.0018882920121541247, + 0.0018781249964376912, + 0.0018333339976379648, + 0.001970083001651801, + 0.001910208011395298, + 0.0019170409941580147 + ], + [ + 0.001698250009212643, + 0.0017182500014314428, + 0.0017033750045811757, + 0.0016079999913927168, + 0.0016189579910133034, + 0.0016905829979805276, + 0.0015800830005900934, + 0.001462917003664188, + 0.0016332499944837764, + 0.0016034169966587797, + 0.001675624997005798, + 0.0016620000096736476, + 0.0016947499971138313, + 0.001698791005765088, + 0.0016681249981047586, + 0.0016227919986704364, + 0.0015972080000210553, + 0.0015582920023007318, + 0.001634083004319109, + 0.0016290420026052743, + 0.0016269159968942404, + 0.001676750005572103, + 0.0016644999996060506, + 0.0017329999973298982, + 0.0016007499943953007, + 0.0017213340033777058, + 0.0016027499950723723, + 0.001573542002006434, + 0.0016047090030042455, + 0.0016839589952724054, + 0.0017075409996323287, + 0.0017212090024258941, + 0.0016966660041362047, + 0.001749332994222641, + 0.0016923749935813248, + 0.0016833330009831116, + 0.001748041991959326, + 0.001666458003455773, + 0.0017484999989392236, + 0.0017667500069364905, + 0.0016705419984646142, + 0.0018174170108977705, + 0.0017019170045387, + 0.0016647080046823248, + 0.0017510000034235418, + 0.00176320799801033, + 0.0017771249986253679, + 0.0016540409997105598, + 0.0018542080069892108, + 0.001768957998137921, + 0.0017497920052846894, + 0.0017492500046500936, + 0.001701707995380275, + 0.001811541005736217, + 0.001805666004656814, + 0.001829707995057106, + 0.0017204999894602224, + 0.0018029170023510233, + 0.0018265420076204464, + 0.001815000010537915, + 0.0017921660037245601, + 0.00187404100142885, + 0.0017907079891301692, + 0.0017866249982034788, + 0.0018597080052131787, + 0.0018073340033879504, + 0.0019080829952144995, + 0.0019465419900370762, + 0.0019602919928729534, + 0.001969666001969017, + 0.0019657500088214874, + 0.0019335830002091825, + 0.0019284159934613854, + 0.0018697920022532344, + 0.0019141669909004122, + 0.0018087499920511618, + 0.001898709000670351, + 0.0019303340086480603, + 0.0019806249911198393, + 0.0019355419935891405, + 0.0019317910046083853, + 0.0019461659976514056, + 0.0019311670039314777, + 0.0019005420035682619, + 0.001891207997687161, + 0.0019094169983873144, + 0.0019128749991068617, + 0.0019385420018807054, + 0.0019059580081375316, + 0.001906667006551288, + 0.0018975410057464615, + 0.0018517080025048926, + 0.0018848749896278605, + 0.0019484170043142512, + 0.0018889580096583813, + 0.0018736250058282167, + 0.0019016660080524161, + 0.0019029170071007684, + 0.0019206669967388734, + 0.0019177500071236864, + 0.0018345829885220155, + 0.001909083002828993, + 0.0018413750076433644, + 0.0019190000020898879, + 0.0019031670090043917, + 0.0019840839959215373, + 0.0019196670036762953, + 0.001921958988532424, + 0.0019137500057695433, + 0.0018448329938109964, + 0.0018395410006633028, + 0.0018484579923097044, + 0.0018567090010037646, + 0.0018867090111598372, + 0.0018549999949755147, + 0.0018497080018278211, + 0.0018247920088469982, + 0.001809916997444816, + 0.0018189999973401427, + 0.001970666999113746, + 0.0019433749985182658, + 0.0018201249913545325, + 0.0018742090032901615, + 0.00211312499595806, + 0.0019315419922349975, + 0.0018646249955054373, + 0.0019563750101951882, + 0.001953665996552445, + 0.0019246250012656674, + 0.0019248339958721772, + 0.0018940000009024516, + 0.001986999996006489, + 0.0019413330010138452, + 0.0019187500001862645, + 0.0019242499984102324, + 0.0018769169982988387, + 0.0018572080007288605, + 0.001868791994638741, + 0.0018586249934742227, + 0.0018944160110550001, + 0.0019195420027244836, + 0.0017544169968459755, + 0.0017212079983437434, + 0.0018251659930683672, + 0.001771958006429486, + 0.0019146670092595741, + 0.0017792920116335154, + 0.0020489580056164414, + 0.0018959579902002588, + 0.0017638749995967373, + 0.002635707991430536, + 0.0019480420014588162, + 0.0019105829996988177, + 0.001960083012818359, + 0.0018856670067179948, + 0.0019252079946454614, + 0.001811666996218264, + 0.0019671250047395006, + 0.001978124986635521, + 0.0019323339947732165, + 0.0019047079986194149, + 0.0018117500003427267, + 0.0018890829960582778, + 0.0019164159893989563, + 0.0018366250005783513, + 0.0017982919962378219, + 0.0018243750091642141, + 0.0018777500081341714, + 0.0018561250035418198, + 0.0018543750047683716, + 0.0017959580000024289, + 0.0018797499942593277, + 0.0019141249940730631, + 0.0019485000084387138, + 0.0017448750004405156, + 0.001941875001648441, + 0.001977707986952737, + 0.001969499993720092, + 0.001938291999977082, + 0.001799459001631476, + 0.0019104170060018077, + 0.001927582998177968, + 0.0019175829947926104, + 0.0019391250098124146, + 0.001980750006623566, + 0.0019192080071661621, + 0.0018474579992471263, + 0.0018699590000323951, + 0.001864833990111947, + 0.0018490419897716492, + 0.001861500000813976, + 0.0018617089954204857, + 0.001933541992912069, + 0.0019125419930787757, + 0.0018564579950179905, + 0.0018487910128897056, + 0.0018187499954365194, + 0.001953541999682784, + 0.001924416996189393, + 0.0019186249992344528, + 0.0019285000016679987, + 0.001988749994779937, + 0.001825208993977867, + 0.0018934999970952049, + 0.0019228750024922192, + 0.0019483329961076379, + 0.001894333996460773, + 0.0018094169936375692, + 0.0018722499953582883, + 0.001893583990749903, + 0.0018920830043498427, + 0.001939000008860603, + 0.0018575420108390972, + 0.0018610420083859935, + 0.0018166670051869005, + 0.0018336669891141355, + 0.0018576250004116446, + 0.0018134999991161749, + 0.0018741660023806617, + 0.001975042003323324, + 0.0017508329910924658, + 0.0019122500088997185, + 0.0018229170091217384, + 0.0019077500037383288 + ], + [ + 0.001728291012113914, + 0.0017135829984908924, + 0.0016698749968782067, + 0.0015944579936331138, + 0.0015570830000797287, + 0.0015173340070759878, + 0.001618124995729886, + 0.0015713750035502017, + 0.0015793329948792234, + 0.0015593340067425743, + 0.001699208005447872, + 0.001665791991399601, + 0.0015122500044526532, + 0.0016086249961517751, + 0.001650874997721985, + 0.0016772500093793496, + 0.0015992920089047402, + 0.0017088750028051436, + 0.0017197090055560693, + 0.0016123329987749457, + 0.0017590409988770261, + 0.001578541996423155, + 0.0016294999950332567, + 0.001615415996639058, + 0.0016214589995797724, + 0.001713749996270053, + 0.0016604580014245585, + 0.0016478339966852218, + 0.0016987909912131727, + 0.001719249994494021, + 0.0017740000039339066, + 0.0017258329899050295, + 0.0016321659932145849, + 0.0016601669922238216, + 0.001660541005549021, + 0.0017106669984059408, + 0.0017701250035315752, + 0.0017502080008853227, + 0.0017566669994266704, + 0.0017691249959170818, + 0.0017909159942064434, + 0.0017690410022623837, + 0.0016955000028247014, + 0.0016735829995013773, + 0.0017337909957859665, + 0.0016306250035995618, + 0.0016247909952653572, + 0.00174320800579153, + 0.001715666992822662, + 0.001819541008444503, + 0.001688333009951748, + 0.0016262079880107194, + 0.0016169590089702979, + 0.001648165998631157, + 0.0016949999990174547, + 0.0016742500010877848, + 0.0017017909995047376, + 0.0017127079918282107, + 0.0017224579933099449, + 0.0017149159975815564, + 0.0017632910021347925, + 0.0017877089994726703, + 0.0017872079915832728, + 0.00173049999284558, + 0.0016973339952528477, + 0.0017164170130854473, + 0.001806291998946108, + 0.001850291999289766, + 0.0018109579978045076, + 0.0018545000057201833, + 0.0018206670065410435, + 0.0018213750008726493, + 0.0017677080031717196, + 0.0017880830127978697, + 0.0017827089905040339, + 0.001913875006721355, + 0.0018633330037118867, + 0.0019053750002058223, + 0.0018354579951846972, + 0.002109707987983711, + 0.001966417010407895, + 0.0018964999908348545, + 0.0018853750079870224, + 0.0018774579948512837, + 0.0018871670035878196, + 0.0019412499968893826, + 0.0020187500049360096, + 0.0020442079985514283, + 0.0019983329984825104, + 0.001973291000467725, + 0.001911458995891735, + 0.001939417008543387, + 0.0019266250019427389, + 0.0018980830063810572, + 0.0018782920087687671, + 0.00203237499226816, + 0.0019170000014128163, + 0.0020076660002814606, + 0.0019365000043762848, + 0.001903458993183449, + 0.0018034589884337038, + 0.0018414590012980625, + 0.001853707988630049, + 0.0019017500017071143, + 0.001986832998227328, + 0.0019996250048279762, + 0.0018347090081078932, + 0.0019098339980700985, + 0.0018522919999668375, + 0.0018891249928856269, + 0.0018708340066950768, + 0.0019109999993816018, + 0.0019177500071236864, + 0.0019122079975204542, + 0.002002417008043267, + 0.0019278330000815913, + 0.0019218329980503768, + 0.001860749995103106, + 0.0019232500053476542, + 0.0019090420100837946, + 0.0018345419957768172, + 0.0019405000057304278, + 0.0019972919981228188, + 0.0019874580029863864, + 0.0017667920037638396, + 0.0018425000016577542, + 0.0018795420037349686, + 0.0018378329987172037, + 0.001798250013962388, + 0.002047165995463729, + 0.0019707090104930103, + 0.002161500000511296, + 0.0019497920002322644, + 0.0018928749923361465, + 0.001803624996682629, + 0.001782499995897524, + 0.0019188339938409626, + 0.0019482080097077414, + 0.0018900830036727712, + 0.0020004160032840446, + 0.0019865829963237047, + 0.001929334001033567, + 0.0018654169980436563, + 0.0018307920108782127, + 0.0018526659987401217, + 0.0019340830040164292, + 0.0019365410116733983, + 0.001925290998769924, + 0.0019335420074639842, + 0.0019269580079708248, + 0.001981499997782521, + 0.0019217090011807159, + 0.0018397499952698126, + 0.001900749994092621, + 0.0019307920010760427, + 0.0019079589983448386, + 0.001986083996598609, + 0.0019302499858895317, + 0.001927958001033403, + 0.001862084012827836, + 0.0016947919939411804, + 0.0017188750061905012, + 0.00207274999411311, + 0.0019610419985838234, + 0.001876416994491592, + 0.0018048330093733966, + 0.0019080830097664148, + 0.001898124988656491, + 0.0018873329972848296, + 0.001913707994390279, + 0.0018097499996656552, + 0.0018437079997966066, + 0.001883834003820084, + 0.0018417499959468842, + 0.0018058749992633238, + 0.0021452499931911007, + 0.002005083006224595, + 0.002048750000540167, + 0.0019275840022601187, + 0.0019395419949432835, + 0.0018920839938800782, + 0.0018261670047650114, + 0.0018435830133967102, + 0.001988291012821719, + 0.001991208002436906, + 0.002008750001550652, + 0.0018927089986391366, + 0.0019332920055603608, + 0.001858292001998052, + 0.001881875010440126, + 0.0018479590071365237, + 0.0019726669997908175, + 0.0017773339932318777, + 0.0021873340010643005, + 0.002010458003496751, + 0.0019463339995127171, + 0.0017681250028545037, + 0.0018130829994333908, + 0.001861500000813976, + 0.0019149580039083958, + 0.0018190419941674918, + 0.0020664170006057248, + 0.0019905840017599985, + 0.0019377089920453727, + 0.0018989160016644746, + 0.0018961250025313348, + 0.001847958003054373, + 0.001834458002122119, + 0.0018580000032670796, + 0.0018577500013634562, + 0.002148500003386289, + 0.0019924170046579093, + 0.0019255410006735474, + 0.00191516600898467, + 0.001934667001478374, + 0.001858124989666976, + 0.0019087500113528222, + 0.002008207986364141, + 0.001998874999117106, + 0.0019413330010138452, + 0.0019337919948156923, + 0.0019914160075131804, + 0.001907958008814603, + 0.0018687920091906562 + ], + [ + 0.001832957990700379, + 0.0016248750034719706, + 0.0017112089990405366, + 0.0016577920032432303, + 0.0015885000029811636, + 0.0015677079936722293, + 0.0016989999858196825, + 0.0017426249978598207, + 0.00169779198768083, + 0.001589749997947365, + 0.0017083340062526986, + 0.0017018329963320866, + 0.0016674170037731528, + 0.0016259169933618978, + 0.0016475000011269003, + 0.0017682920006336644, + 0.00180283299414441, + 0.001807249995181337, + 0.0017364160012220964, + 0.001697959000011906, + 0.0017486249998910353, + 0.0017802910006139427, + 0.0017739590111887082, + 0.001763249994837679, + 0.0017184159951284528, + 0.0017973749927477911, + 0.0017168750055134296, + 0.001712750003207475, + 0.0017606250039534643, + 0.0018194589938502759, + 0.001757875012117438, + 0.0016944170056376606, + 0.0017512089980300516, + 0.0018036250112345442, + 0.0017654999974183738, + 0.001823833998059854, + 0.001852791989222169, + 0.001865500002168119, + 0.0018294589972356334, + 0.0017481669929111376, + 0.00173462500970345, + 0.0019022499909624457, + 0.0018458340055076405, + 0.0018766669963952154, + 0.001855624999734573, + 0.0018166670051869005, + 0.00179237499833107, + 0.001800375001039356, + 0.0018532499962020665, + 0.0018533749971538782, + 0.0016675000078976154, + 0.0018462500011082739, + 0.0018471250077709556, + 0.001904374992591329, + 0.0018413340003462508, + 0.0020440829975996166, + 0.0018771659961203113, + 0.0018556669965619221, + 0.0018805839936248958, + 0.0019208750018151477, + 0.0017933339986484498, + 0.0019743330049095675, + 0.0019699580006999895, + 0.002005542002734728, + 0.0019636250071926042, + 0.0020033329928992316, + 0.0019429999956628308, + 0.0019198749942006543, + 0.0019731659995159134, + 0.0019745409954339266, + 0.0019491660059429705, + 0.0020303750061430037, + 0.0020397079933900386, + 0.0019795409898506477, + 0.0019854590063914657, + 0.001991415992961265, + 0.0019168750004610047, + 0.0019210000027669594, + 0.001958624998223968, + 0.0019579999934649095, + 0.0019570419972296804, + 0.0019815420091617852, + 0.002003249988774769, + 0.002031791998888366, + 0.0019278750114608556, + 0.002062667001155205, + 0.0020792499999515712, + 0.0021119589946465567, + 0.0020277080038795248, + 0.002051166011369787, + 0.00198433299374301, + 0.001985124996281229, + 0.0019407079962547868, + 0.002090624999254942, + 0.002057833000435494, + 0.002073875002679415, + 0.002071249997243285, + 0.002073583993478678, + 0.0020063330011907965, + 0.002012750002904795, + 0.001963750008144416, + 0.001985291004530154, + 0.0019831669924315065, + 0.001997166997171007, + 0.0019625419954536483, + 0.0019844589987769723, + 0.0019737500115297735, + 0.0019668750028358772, + 0.001988375006476417, + 0.0020000829972559586, + 0.002061291001155041, + 0.0020425829861778766, + 0.0019608750008046627, + 0.0020440000080270693, + 0.00199070900271181, + 0.0019006249931408092, + 0.0019184170087100938, + 0.0018517079879529774, + 0.002089500005240552, + 0.0020335420122137293, + 0.0020362920040497556, + 0.001828542008297518, + 0.002073292009299621, + 0.002077874989481643, + 0.002029292008955963, + 0.001953665996552445, + 0.0019631670002127066, + 0.001891958003398031, + 0.0019827079959213734, + 0.001988750009331852, + 0.001979042004677467, + 0.0020747500093420967, + 0.002042208012426272, + 0.002019749997998588, + 0.0019072499999310821, + 0.0020231249945936725, + 0.001993875004700385, + 0.0020008749997941777, + 0.0019993749883724377, + 0.0019999170035589486, + 0.0019568330026231706, + 0.0018784589919960126, + 0.0019512500002747402, + 0.002037875005044043, + 0.0020524999999906868, + 0.002036707999650389, + 0.0020355419983388856, + 0.002009667005040683, + 0.0019833330006804317, + 0.0019179579976480454, + 0.001956583990249783, + 0.001939207999384962, + 0.0019596670026658103, + 0.0020405830000527203, + 0.0020465420093387365, + 0.0020500419923337176, + 0.002061208986560814, + 0.0019764580065384507, + 0.0019092919974355027, + 0.0019399169977987185, + 0.0019619999948190525, + 0.0020014170004287735, + 0.0019827079959213734, + 0.0019625420100055635, + 0.0020375410094857216, + 0.002049875009106472, + 0.0018846249877242371, + 0.001953458006028086, + 0.002033833006862551, + 0.0019935840100515634, + 0.002040542007307522, + 0.0019137910130666569, + 0.002077250013826415, + 0.0019623340049292892, + 0.001975958002731204, + 0.0019871249969583005, + 0.0020864169928245246, + 0.001938291999977082, + 0.0019788339996011928, + 0.0020852500019827858, + 0.002071750001050532, + 0.002057916994090192, + 0.002064000000245869, + 0.0020114999933866784, + 0.001943208000739105, + 0.002019624997046776, + 0.001994833000935614, + 0.0019995000038761646, + 0.001971958001377061, + 0.0019849579985020682, + 0.0020560830016620457, + 0.002014208002947271, + 0.0020027499995194376, + 0.0021694159950129688, + 0.0020380000059958547, + 0.002046624998911284, + 0.002064000000245869, + 0.0019197500078007579, + 0.001973292004549876, + 0.001961125002708286, + 0.0019162499957019463, + 0.0019049580005230382, + 0.0020919589878758416, + 0.0020688749937107787, + 0.002064375003101304, + 0.0019920420018024743, + 0.0019313340017106384, + 0.0019804580078925937, + 0.0019322080042911693, + 0.001935042004333809, + 0.0019722500001080334, + 0.0019690000044647604, + 0.0019419579912209883, + 0.0020183330052532256, + 0.0020562499994412065, + 0.0020519170066108927, + 0.002047540998319164, + 0.0018893339874921367, + 0.0020796250028070062, + 0.0020190409995848313, + 0.001935459004016593, + 0.0020004160032840446, + 0.0018475410033715889, + 0.0020100419933442026 + ], + [ + 0.0017629590001888573, + 0.0017628339992370456, + 0.0017210420046467334, + 0.001757291000103578, + 0.0016562090022489429, + 0.001725916998111643, + 0.001757917008944787, + 0.0017032089963322505, + 0.0016853329871082678, + 0.0016952499863691628, + 0.001715249993139878, + 0.00172433300758712, + 0.001657374989008531, + 0.0016696669918019325, + 0.001754708995576948, + 0.0018432910001138225, + 0.0017496670043328777, + 0.001724375004414469, + 0.001707834002445452, + 0.001768041998730041, + 0.0017938749952008948, + 0.0017857080092653632, + 0.0016976250044535846, + 0.001739042010740377, + 0.0017898329970194027, + 0.001678708998952061, + 0.001782874998752959, + 0.0016551250009797513, + 0.001665666000917554, + 0.0017829580028774217, + 0.0018021669966401532, + 0.0017306250083493069, + 0.0019052080024266616, + 0.0018971660028910264, + 0.0017738749884301797, + 0.0018297080096090212, + 0.0017220000008819625, + 0.0018065829935949296, + 0.0017709579988149926, + 0.001771291994373314, + 0.0018144170026062056, + 0.0017884580011013895, + 0.001773541996954009, + 0.0016709159972378984, + 0.001744541004882194, + 0.0018213750008726493, + 0.0018465419998392463, + 0.0017988329927902669, + 0.0018572499975562096, + 0.0018235000025015324, + 0.0018431249918648973, + 0.0018670830031624064, + 0.00184208300197497, + 0.0018836250092135742, + 0.0018977080035256222, + 0.001886666999780573, + 0.0018322910036658868, + 0.001864917008788325, + 0.001833166999858804, + 0.0018315829947823659, + 0.001847207997343503, + 0.0019077079923590645, + 0.0019224580028094351, + 0.0018650409911060706, + 0.0018602079944685102, + 0.0019182079995516688, + 0.001958499997272156, + 0.0019949580018874258, + 0.002178042006562464, + 0.0020401249930728227, + 0.0020020000083604828, + 0.001978832995519042, + 0.0019758750131586567, + 0.0018344999989494681, + 0.0017694160051178187, + 0.0017738750029820949, + 0.0017727910017129034, + 0.0018305830017197877, + 0.0018324999982723966, + 0.0017850409931270406, + 0.0017904159903991967, + 0.001778791003744118, + 0.0018127079965779558, + 0.0020190829964121804, + 0.0021195839944994077, + 0.0020382090006023645, + 0.0019475829903967679, + 0.0021020830026827753, + 0.0020470409945119172, + 0.0020225420012138784, + 0.0019871249969583005, + 0.0019899580074707046, + 0.002006958005949855, + 0.0019533750019036233, + 0.002002458000788465, + 0.0022263340069912374, + 0.0020414159953361377, + 0.002008374998695217, + 0.001998290987103246, + 0.001964915994904004, + 0.0019856250000884756, + 0.0019968749984400347, + 0.0019804589974228293, + 0.001984791990253143, + 0.0019862079934682697, + 0.0019131659937556833, + 0.001876915994216688, + 0.0018840840057237074, + 0.001885500008938834, + 0.001983458991162479, + 0.002003000001423061, + 0.0020039170049130917, + 0.001998874999117106, + 0.0019565830007195473, + 0.0019349169888300821, + 0.001897958994959481, + 0.0019409160013310611, + 0.001970625002286397, + 0.0019777919951593503, + 0.0019578329956857488, + 0.001988666001125239, + 0.002008707990171388, + 0.0019166250131092966, + 0.0019200840033590794, + 0.0018939170113299042, + 0.0018669169949134812, + 0.001965167000889778, + 0.0019887920061592013, + 0.0019470830011414364, + 0.0020577499963110313, + 0.0020517910015769303, + 0.001973499995074235, + 0.001968750002561137, + 0.0019242080015828833, + 0.0019865840004058555, + 0.001979249995201826, + 0.002175834000809118, + 0.001878708993899636, + 0.002040500010480173, + 0.001915624990942888, + 0.0018986659997608513, + 0.0019871250115102157, + 0.002136209004675038, + 0.002023749999352731, + 0.001958624998223968, + 0.0019269589975010604, + 0.002041041007032618, + 0.001995040991459973, + 0.0019328749913256615, + 0.0019111250003334135, + 0.0019172500033164397, + 0.0020126250019529834, + 0.001974583006813191, + 0.002048374997684732, + 0.0020617919944925234, + 0.0020935829961672425, + 0.00205462500161957, + 0.0020399170025484636, + 0.0020159169944236055, + 0.0019993339956272393, + 0.00202379199618008, + 0.0022224160056794062, + 0.0020627079939004034, + 0.0021017079998273402, + 0.0020242919999873266, + 0.0019471250125207007, + 0.0019530419958755374, + 0.00196266699640546, + 0.001929541991557926, + 0.0019140830117976293, + 0.0020524999999906868, + 0.002090249996399507, + 0.0020827080006711185, + 0.0019884590001311153, + 0.001920374998007901, + 0.0020139590051257983, + 0.0019465840014163405, + 0.001976750005269423, + 0.001993541998672299, + 0.0019770420040003955, + 0.0019692499918164685, + 0.002055165998172015, + 0.0020132089994149283, + 0.0020823330123675987, + 0.0020345410011941567, + 0.0020040410017827526, + 0.0020840840006712824, + 0.002093707997119054, + 0.0020707499934360385, + 0.0020304170029703528, + 0.0020032089960295707, + 0.001994457998080179, + 0.0019536249892553315, + 0.0019451660045888275, + 0.0019834999984595925, + 0.002076749995467253, + 0.0020922499970765784, + 0.0020923750125803053, + 0.002037666999967769, + 0.0020116670057177544, + 0.0018929169891634956, + 0.001996666003833525, + 0.0019783749885391444, + 0.00203333399258554, + 0.0019197919900761917, + 0.0022267500025918707, + 0.002084957988699898, + 0.0019400000019231811, + 0.0019442919874563813, + 0.0020158750121481717, + 0.002038791004451923, + 0.00197108399879653, + 0.0019881249900208786, + 0.00208208299591206, + 0.0020444999972824007, + 0.002024332992732525, + 0.002053333999356255, + 0.0020092080085305497, + 0.002013874996919185, + 0.002024750006967224, + 0.002012791999732144, + 0.0020848749991273507, + 0.002086416003294289, + 0.002096416996209882 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "speedup_cached_vs_uncached_sdpa": 0.9382282433920099, + "speedup_cached_vs_uncached_mha": 1.0572977238033754 + }, + "memory": [ + { + "batch": 1, + "prompt_len": 128, + "steps": [ + { + "step": 0, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 1, + "cached_positions": 129, + "cache_bytes": 792576, + "step_peak_minus_start_bytes": 805120 + }, + { + "step": 2, + "cached_positions": 130, + "cache_bytes": 798720, + "step_peak_minus_start_bytes": 811264 + }, + { + "step": 3, + "cached_positions": 131, + "cache_bytes": 804864, + "step_peak_minus_start_bytes": 817408 + }, + { + "step": 4, + "cached_positions": 132, + "cache_bytes": 811008, + "step_peak_minus_start_bytes": 823552 + }, + { + "step": 5, + "cached_positions": 133, + "cache_bytes": 817152, + "step_peak_minus_start_bytes": 829696 + }, + { + "step": 6, + "cached_positions": 134, + "cache_bytes": 823296, + "step_peak_minus_start_bytes": 835840 + }, + { + "step": 7, + "cached_positions": 135, + "cache_bytes": 829440, + "step_peak_minus_start_bytes": 841984 + }, + { + "step": 8, + "cached_positions": 136, + "cache_bytes": 835584, + "step_peak_minus_start_bytes": 848128 + }, + { + "step": 9, + "cached_positions": 137, + "cache_bytes": 841728, + "step_peak_minus_start_bytes": 854272 + }, + { + "step": 10, + "cached_positions": 138, + "cache_bytes": 847872, + "step_peak_minus_start_bytes": 860416 + }, + { + "step": 11, + "cached_positions": 139, + "cache_bytes": 854016, + "step_peak_minus_start_bytes": 866560 + }, + { + "step": 12, + "cached_positions": 140, + "cache_bytes": 860160, + "step_peak_minus_start_bytes": 872704 + }, + { + "step": 13, + "cached_positions": 141, + "cache_bytes": 866304, + "step_peak_minus_start_bytes": 878848 + }, + { + "step": 14, + "cached_positions": 142, + "cache_bytes": 872448, + "step_peak_minus_start_bytes": 884992 + }, + { + "step": 15, + "cached_positions": 143, + "cache_bytes": 878592, + "step_peak_minus_start_bytes": 891136 + }, + { + "step": 16, + "cached_positions": 144, + "cache_bytes": 884736, + "step_peak_minus_start_bytes": 897280 + }, + { + "step": 17, + "cached_positions": 145, + "cache_bytes": 890880, + "step_peak_minus_start_bytes": 903424 + }, + { + "step": 18, + "cached_positions": 146, + "cache_bytes": 897024, + "step_peak_minus_start_bytes": 909568 + }, + { + "step": 19, + "cached_positions": 147, + "cache_bytes": 903168, + "step_peak_minus_start_bytes": 915712 + }, + { + "step": 20, + "cached_positions": 148, + "cache_bytes": 909312, + "step_peak_minus_start_bytes": 921856 + }, + { + "step": 21, + "cached_positions": 149, + "cache_bytes": 915456, + "step_peak_minus_start_bytes": 928000 + }, + { + "step": 22, + "cached_positions": 150, + "cache_bytes": 921600, + "step_peak_minus_start_bytes": 934144 + }, + { + "step": 23, + "cached_positions": 151, + "cache_bytes": 927744, + "step_peak_minus_start_bytes": 940288 + }, + { + "step": 24, + "cached_positions": 152, + "cache_bytes": 933888, + "step_peak_minus_start_bytes": 946432 + }, + { + "step": 25, + "cached_positions": 153, + "cache_bytes": 940032, + "step_peak_minus_start_bytes": 952576 + }, + { + "step": 26, + "cached_positions": 154, + "cache_bytes": 946176, + "step_peak_minus_start_bytes": 958720 + }, + { + "step": 27, + "cached_positions": 155, + "cache_bytes": 952320, + "step_peak_minus_start_bytes": 964864 + }, + { + "step": 28, + "cached_positions": 156, + "cache_bytes": 958464, + "step_peak_minus_start_bytes": 971008 + }, + { + "step": 29, + "cached_positions": 157, + "cache_bytes": 964608, + "step_peak_minus_start_bytes": 977152 + }, + { + "step": 30, + "cached_positions": 158, + "cache_bytes": 970752, + "step_peak_minus_start_bytes": 983296 + }, + { + "step": 31, + "cached_positions": 159, + "cache_bytes": 976896, + "step_peak_minus_start_bytes": 989440 + }, + { + "step": 32, + "cached_positions": 160, + "cache_bytes": 983040, + "step_peak_minus_start_bytes": 995584 + }, + { + "step": 33, + "cached_positions": 161, + "cache_bytes": 989184, + "step_peak_minus_start_bytes": 1001728 + }, + { + "step": 34, + "cached_positions": 162, + "cache_bytes": 995328, + "step_peak_minus_start_bytes": 1007872 + }, + { + "step": 35, + "cached_positions": 163, + "cache_bytes": 1001472, + "step_peak_minus_start_bytes": 1014016 + }, + { + "step": 36, + "cached_positions": 164, + "cache_bytes": 1007616, + "step_peak_minus_start_bytes": 1020160 + }, + { + "step": 37, + "cached_positions": 165, + "cache_bytes": 1013760, + "step_peak_minus_start_bytes": 1026304 + }, + { + "step": 38, + "cached_positions": 166, + "cache_bytes": 1019904, + "step_peak_minus_start_bytes": 1032448 + }, + { + "step": 39, + "cached_positions": 167, + "cache_bytes": 1026048, + "step_peak_minus_start_bytes": 1038592 + }, + { + "step": 40, + "cached_positions": 168, + "cache_bytes": 1032192, + "step_peak_minus_start_bytes": 1044736 + }, + { + "step": 41, + "cached_positions": 169, + "cache_bytes": 1038336, + "step_peak_minus_start_bytes": 1050880 + }, + { + "step": 42, + "cached_positions": 170, + "cache_bytes": 1044480, + "step_peak_minus_start_bytes": 1057024 + }, + { + "step": 43, + "cached_positions": 171, + "cache_bytes": 1050624, + "step_peak_minus_start_bytes": 1063168 + }, + { + "step": 44, + "cached_positions": 172, + "cache_bytes": 1056768, + "step_peak_minus_start_bytes": 1069312 + }, + { + "step": 45, + "cached_positions": 173, + "cache_bytes": 1062912, + "step_peak_minus_start_bytes": 1075456 + }, + { + "step": 46, + "cached_positions": 174, + "cache_bytes": 1069056, + "step_peak_minus_start_bytes": 1081600 + }, + { + "step": 47, + "cached_positions": 175, + "cache_bytes": 1075200, + "step_peak_minus_start_bytes": 1087744 + }, + { + "step": 48, + "cached_positions": 176, + "cache_bytes": 1081344, + "step_peak_minus_start_bytes": 1093888 + }, + { + "step": 49, + "cached_positions": 177, + "cache_bytes": 1087488, + "step_peak_minus_start_bytes": 1100032 + }, + { + "step": 50, + "cached_positions": 178, + "cache_bytes": 1093632, + "step_peak_minus_start_bytes": 1106176 + }, + { + "step": 51, + "cached_positions": 179, + "cache_bytes": 1099776, + "step_peak_minus_start_bytes": 1112320 + }, + { + "step": 52, + "cached_positions": 180, + "cache_bytes": 1105920, + "step_peak_minus_start_bytes": 1118464 + }, + { + "step": 53, + "cached_positions": 181, + "cache_bytes": 1112064, + "step_peak_minus_start_bytes": 1124608 + }, + { + "step": 54, + "cached_positions": 182, + "cache_bytes": 1118208, + "step_peak_minus_start_bytes": 1130752 + }, + { + "step": 55, + "cached_positions": 183, + "cache_bytes": 1124352, + "step_peak_minus_start_bytes": 1136896 + }, + { + "step": 56, + "cached_positions": 184, + "cache_bytes": 1130496, + "step_peak_minus_start_bytes": 1143040 + }, + { + "step": 57, + "cached_positions": 185, + "cache_bytes": 1136640, + "step_peak_minus_start_bytes": 1149184 + }, + { + "step": 58, + "cached_positions": 186, + "cache_bytes": 1142784, + "step_peak_minus_start_bytes": 1155328 + }, + { + "step": 59, + "cached_positions": 187, + "cache_bytes": 1148928, + "step_peak_minus_start_bytes": 1161472 + }, + { + "step": 60, + "cached_positions": 188, + "cache_bytes": 1155072, + "step_peak_minus_start_bytes": 1167616 + }, + { + "step": 61, + "cached_positions": 189, + "cache_bytes": 1161216, + "step_peak_minus_start_bytes": 1173760 + }, + { + "step": 62, + "cached_positions": 190, + "cache_bytes": 1167360, + "step_peak_minus_start_bytes": 1179904 + }, + { + "step": 63, + "cached_positions": 191, + "cache_bytes": 1173504, + "step_peak_minus_start_bytes": 1186048 + } + ] + }, + { + "batch": 8, + "prompt_len": 128, + "steps": [ + { + "step": 0, + "cached_positions": 128, + "cache_bytes": 6291456, + "step_peak_minus_start_bytes": 25953280 + }, + { + "step": 1, + "cached_positions": 129, + "cache_bytes": 6340608, + "step_peak_minus_start_bytes": 6439168 + }, + { + "step": 2, + "cached_positions": 130, + "cache_bytes": 6389760, + "step_peak_minus_start_bytes": 6488320 + }, + { + "step": 3, + "cached_positions": 131, + "cache_bytes": 6438912, + "step_peak_minus_start_bytes": 6537472 + }, + { + "step": 4, + "cached_positions": 132, + "cache_bytes": 6488064, + "step_peak_minus_start_bytes": 6586624 + }, + { + "step": 5, + "cached_positions": 133, + "cache_bytes": 6537216, + "step_peak_minus_start_bytes": 6635776 + }, + { + "step": 6, + "cached_positions": 134, + "cache_bytes": 6586368, + "step_peak_minus_start_bytes": 6684928 + }, + { + "step": 7, + "cached_positions": 135, + "cache_bytes": 6635520, + "step_peak_minus_start_bytes": 6734080 + }, + { + "step": 8, + "cached_positions": 136, + "cache_bytes": 6684672, + "step_peak_minus_start_bytes": 6783232 + }, + { + "step": 9, + "cached_positions": 137, + "cache_bytes": 6733824, + "step_peak_minus_start_bytes": 6832384 + }, + { + "step": 10, + "cached_positions": 138, + "cache_bytes": 6782976, + "step_peak_minus_start_bytes": 6881536 + }, + { + "step": 11, + "cached_positions": 139, + "cache_bytes": 6832128, + "step_peak_minus_start_bytes": 6930688 + }, + { + "step": 12, + "cached_positions": 140, + "cache_bytes": 6881280, + "step_peak_minus_start_bytes": 6979840 + }, + { + "step": 13, + "cached_positions": 141, + "cache_bytes": 6930432, + "step_peak_minus_start_bytes": 7028992 + }, + { + "step": 14, + "cached_positions": 142, + "cache_bytes": 6979584, + "step_peak_minus_start_bytes": 7078144 + }, + { + "step": 15, + "cached_positions": 143, + "cache_bytes": 7028736, + "step_peak_minus_start_bytes": 7127296 + }, + { + "step": 16, + "cached_positions": 144, + "cache_bytes": 7077888, + "step_peak_minus_start_bytes": 7176448 + }, + { + "step": 17, + "cached_positions": 145, + "cache_bytes": 7127040, + "step_peak_minus_start_bytes": 7225600 + }, + { + "step": 18, + "cached_positions": 146, + "cache_bytes": 7176192, + "step_peak_minus_start_bytes": 7274752 + }, + { + "step": 19, + "cached_positions": 147, + "cache_bytes": 7225344, + "step_peak_minus_start_bytes": 7323904 + }, + { + "step": 20, + "cached_positions": 148, + "cache_bytes": 7274496, + "step_peak_minus_start_bytes": 7373056 + }, + { + "step": 21, + "cached_positions": 149, + "cache_bytes": 7323648, + "step_peak_minus_start_bytes": 7422208 + }, + { + "step": 22, + "cached_positions": 150, + "cache_bytes": 7372800, + "step_peak_minus_start_bytes": 7471360 + }, + { + "step": 23, + "cached_positions": 151, + "cache_bytes": 7421952, + "step_peak_minus_start_bytes": 7520512 + }, + { + "step": 24, + "cached_positions": 152, + "cache_bytes": 7471104, + "step_peak_minus_start_bytes": 7569664 + }, + { + "step": 25, + "cached_positions": 153, + "cache_bytes": 7520256, + "step_peak_minus_start_bytes": 7618816 + }, + { + "step": 26, + "cached_positions": 154, + "cache_bytes": 7569408, + "step_peak_minus_start_bytes": 7667968 + }, + { + "step": 27, + "cached_positions": 155, + "cache_bytes": 7618560, + "step_peak_minus_start_bytes": 7717120 + }, + { + "step": 28, + "cached_positions": 156, + "cache_bytes": 7667712, + "step_peak_minus_start_bytes": 7766272 + }, + { + "step": 29, + "cached_positions": 157, + "cache_bytes": 7716864, + "step_peak_minus_start_bytes": 7815424 + }, + { + "step": 30, + "cached_positions": 158, + "cache_bytes": 7766016, + "step_peak_minus_start_bytes": 7864576 + }, + { + "step": 31, + "cached_positions": 159, + "cache_bytes": 7815168, + "step_peak_minus_start_bytes": 7913728 + }, + { + "step": 32, + "cached_positions": 160, + "cache_bytes": 7864320, + "step_peak_minus_start_bytes": 7962880 + }, + { + "step": 33, + "cached_positions": 161, + "cache_bytes": 7913472, + "step_peak_minus_start_bytes": 8012032 + }, + { + "step": 34, + "cached_positions": 162, + "cache_bytes": 7962624, + "step_peak_minus_start_bytes": 8061184 + }, + { + "step": 35, + "cached_positions": 163, + "cache_bytes": 8011776, + "step_peak_minus_start_bytes": 8110336 + }, + { + "step": 36, + "cached_positions": 164, + "cache_bytes": 8060928, + "step_peak_minus_start_bytes": 8159488 + }, + { + "step": 37, + "cached_positions": 165, + "cache_bytes": 8110080, + "step_peak_minus_start_bytes": 8208640 + }, + { + "step": 38, + "cached_positions": 166, + "cache_bytes": 8159232, + "step_peak_minus_start_bytes": 8257792 + }, + { + "step": 39, + "cached_positions": 167, + "cache_bytes": 8208384, + "step_peak_minus_start_bytes": 8306944 + }, + { + "step": 40, + "cached_positions": 168, + "cache_bytes": 8257536, + "step_peak_minus_start_bytes": 8356096 + }, + { + "step": 41, + "cached_positions": 169, + "cache_bytes": 8306688, + "step_peak_minus_start_bytes": 8405248 + }, + { + "step": 42, + "cached_positions": 170, + "cache_bytes": 8355840, + "step_peak_minus_start_bytes": 8454400 + }, + { + "step": 43, + "cached_positions": 171, + "cache_bytes": 8404992, + "step_peak_minus_start_bytes": 11993344 + }, + { + "step": 44, + "cached_positions": 172, + "cache_bytes": 8454144, + "step_peak_minus_start_bytes": 13777152 + }, + { + "step": 45, + "cached_positions": 173, + "cache_bytes": 8503296, + "step_peak_minus_start_bytes": 11993344 + }, + { + "step": 46, + "cached_positions": 174, + "cache_bytes": 8552448, + "step_peak_minus_start_bytes": 13777152 + }, + { + "step": 47, + "cached_positions": 175, + "cache_bytes": 8601600, + "step_peak_minus_start_bytes": 11993344 + }, + { + "step": 48, + "cached_positions": 176, + "cache_bytes": 8650752, + "step_peak_minus_start_bytes": 13777152 + }, + { + "step": 49, + "cached_positions": 177, + "cache_bytes": 8699904, + "step_peak_minus_start_bytes": 11993344 + }, + { + "step": 50, + "cached_positions": 178, + "cache_bytes": 8749056, + "step_peak_minus_start_bytes": 13973760 + }, + { + "step": 51, + "cached_positions": 179, + "cache_bytes": 8798208, + "step_peak_minus_start_bytes": 11993344 + }, + { + "step": 52, + "cached_positions": 180, + "cache_bytes": 8847360, + "step_peak_minus_start_bytes": 13973760 + }, + { + "step": 53, + "cached_positions": 181, + "cache_bytes": 8896512, + "step_peak_minus_start_bytes": 11993344 + }, + { + "step": 54, + "cached_positions": 182, + "cache_bytes": 8945664, + "step_peak_minus_start_bytes": 13973760 + }, + { + "step": 55, + "cached_positions": 183, + "cache_bytes": 8994816, + "step_peak_minus_start_bytes": 11993344 + }, + { + "step": 56, + "cached_positions": 184, + "cache_bytes": 9043968, + "step_peak_minus_start_bytes": 13973760 + }, + { + "step": 57, + "cached_positions": 185, + "cache_bytes": 9093120, + "step_peak_minus_start_bytes": 11993344 + }, + { + "step": 58, + "cached_positions": 186, + "cache_bytes": 9142272, + "step_peak_minus_start_bytes": 13973760 + }, + { + "step": 59, + "cached_positions": 187, + "cache_bytes": 9191424, + "step_peak_minus_start_bytes": 11993344 + }, + { + "step": 60, + "cached_positions": 188, + "cache_bytes": 9240576, + "step_peak_minus_start_bytes": 13973760 + }, + { + "step": 61, + "cached_positions": 189, + "cache_bytes": 9289728, + "step_peak_minus_start_bytes": 11993344 + }, + { + "step": 62, + "cached_positions": 190, + "cache_bytes": 9338880, + "step_peak_minus_start_bytes": 13973760 + }, + { + "step": 63, + "cached_positions": 191, + "cache_bytes": 9388032, + "step_peak_minus_start_bytes": 11993344 + } + ] + }, + { + "batch": 1, + "prompt_len": 32, + "steps": [ + { + "step": 0, + "cached_positions": 32, + "cache_bytes": 196608, + "step_peak_minus_start_bytes": 688384 + }, + { + "step": 1, + "cached_positions": 33, + "cache_bytes": 202752, + "step_peak_minus_start_bytes": 215296 + }, + { + "step": 2, + "cached_positions": 34, + "cache_bytes": 208896, + "step_peak_minus_start_bytes": 221440 + }, + { + "step": 3, + "cached_positions": 35, + "cache_bytes": 215040, + "step_peak_minus_start_bytes": 227584 + }, + { + "step": 4, + "cached_positions": 36, + "cache_bytes": 221184, + "step_peak_minus_start_bytes": 233728 + }, + { + "step": 5, + "cached_positions": 37, + "cache_bytes": 227328, + "step_peak_minus_start_bytes": 239872 + }, + { + "step": 6, + "cached_positions": 38, + "cache_bytes": 233472, + "step_peak_minus_start_bytes": 246016 + }, + { + "step": 7, + "cached_positions": 39, + "cache_bytes": 239616, + "step_peak_minus_start_bytes": 252160 + }, + { + "step": 8, + "cached_positions": 40, + "cache_bytes": 245760, + "step_peak_minus_start_bytes": 258304 + }, + { + "step": 9, + "cached_positions": 41, + "cache_bytes": 251904, + "step_peak_minus_start_bytes": 264448 + }, + { + "step": 10, + "cached_positions": 42, + "cache_bytes": 258048, + "step_peak_minus_start_bytes": 270592 + }, + { + "step": 11, + "cached_positions": 43, + "cache_bytes": 264192, + "step_peak_minus_start_bytes": 276736 + }, + { + "step": 12, + "cached_positions": 44, + "cache_bytes": 270336, + "step_peak_minus_start_bytes": 282880 + }, + { + "step": 13, + "cached_positions": 45, + "cache_bytes": 276480, + "step_peak_minus_start_bytes": 289024 + }, + { + "step": 14, + "cached_positions": 46, + "cache_bytes": 282624, + "step_peak_minus_start_bytes": 295168 + }, + { + "step": 15, + "cached_positions": 47, + "cache_bytes": 288768, + "step_peak_minus_start_bytes": 301312 + }, + { + "step": 16, + "cached_positions": 48, + "cache_bytes": 294912, + "step_peak_minus_start_bytes": 307456 + }, + { + "step": 17, + "cached_positions": 49, + "cache_bytes": 301056, + "step_peak_minus_start_bytes": 313600 + }, + { + "step": 18, + "cached_positions": 50, + "cache_bytes": 307200, + "step_peak_minus_start_bytes": 319744 + }, + { + "step": 19, + "cached_positions": 51, + "cache_bytes": 313344, + "step_peak_minus_start_bytes": 325888 + }, + { + "step": 20, + "cached_positions": 52, + "cache_bytes": 319488, + "step_peak_minus_start_bytes": 332032 + }, + { + "step": 21, + "cached_positions": 53, + "cache_bytes": 325632, + "step_peak_minus_start_bytes": 338176 + }, + { + "step": 22, + "cached_positions": 54, + "cache_bytes": 331776, + "step_peak_minus_start_bytes": 344320 + }, + { + "step": 23, + "cached_positions": 55, + "cache_bytes": 337920, + "step_peak_minus_start_bytes": 350464 + }, + { + "step": 24, + "cached_positions": 56, + "cache_bytes": 344064, + "step_peak_minus_start_bytes": 356608 + }, + { + "step": 25, + "cached_positions": 57, + "cache_bytes": 350208, + "step_peak_minus_start_bytes": 362752 + }, + { + "step": 26, + "cached_positions": 58, + "cache_bytes": 356352, + "step_peak_minus_start_bytes": 368896 + }, + { + "step": 27, + "cached_positions": 59, + "cache_bytes": 362496, + "step_peak_minus_start_bytes": 375040 + }, + { + "step": 28, + "cached_positions": 60, + "cache_bytes": 368640, + "step_peak_minus_start_bytes": 381184 + }, + { + "step": 29, + "cached_positions": 61, + "cache_bytes": 374784, + "step_peak_minus_start_bytes": 387328 + }, + { + "step": 30, + "cached_positions": 62, + "cache_bytes": 380928, + "step_peak_minus_start_bytes": 393472 + }, + { + "step": 31, + "cached_positions": 63, + "cache_bytes": 387072, + "step_peak_minus_start_bytes": 399616 + }, + { + "step": 32, + "cached_positions": 64, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 405760 + }, + { + "step": 33, + "cached_positions": 65, + "cache_bytes": 399360, + "step_peak_minus_start_bytes": 411904 + }, + { + "step": 34, + "cached_positions": 66, + "cache_bytes": 405504, + "step_peak_minus_start_bytes": 418048 + }, + { + "step": 35, + "cached_positions": 67, + "cache_bytes": 411648, + "step_peak_minus_start_bytes": 424192 + }, + { + "step": 36, + "cached_positions": 68, + "cache_bytes": 417792, + "step_peak_minus_start_bytes": 430336 + }, + { + "step": 37, + "cached_positions": 69, + "cache_bytes": 423936, + "step_peak_minus_start_bytes": 436480 + }, + { + "step": 38, + "cached_positions": 70, + "cache_bytes": 430080, + "step_peak_minus_start_bytes": 442624 + }, + { + "step": 39, + "cached_positions": 71, + "cache_bytes": 436224, + "step_peak_minus_start_bytes": 448768 + }, + { + "step": 40, + "cached_positions": 72, + "cache_bytes": 442368, + "step_peak_minus_start_bytes": 454912 + }, + { + "step": 41, + "cached_positions": 73, + "cache_bytes": 448512, + "step_peak_minus_start_bytes": 461056 + }, + { + "step": 42, + "cached_positions": 74, + "cache_bytes": 454656, + "step_peak_minus_start_bytes": 467200 + }, + { + "step": 43, + "cached_positions": 75, + "cache_bytes": 460800, + "step_peak_minus_start_bytes": 473344 + }, + { + "step": 44, + "cached_positions": 76, + "cache_bytes": 466944, + "step_peak_minus_start_bytes": 479488 + }, + { + "step": 45, + "cached_positions": 77, + "cache_bytes": 473088, + "step_peak_minus_start_bytes": 485632 + }, + { + "step": 46, + "cached_positions": 78, + "cache_bytes": 479232, + "step_peak_minus_start_bytes": 491776 + }, + { + "step": 47, + "cached_positions": 79, + "cache_bytes": 485376, + "step_peak_minus_start_bytes": 497920 + }, + { + "step": 48, + "cached_positions": 80, + "cache_bytes": 491520, + "step_peak_minus_start_bytes": 504064 + }, + { + "step": 49, + "cached_positions": 81, + "cache_bytes": 497664, + "step_peak_minus_start_bytes": 510208 + }, + { + "step": 50, + "cached_positions": 82, + "cache_bytes": 503808, + "step_peak_minus_start_bytes": 516352 + }, + { + "step": 51, + "cached_positions": 83, + "cache_bytes": 509952, + "step_peak_minus_start_bytes": 522496 + }, + { + "step": 52, + "cached_positions": 84, + "cache_bytes": 516096, + "step_peak_minus_start_bytes": 528640 + }, + { + "step": 53, + "cached_positions": 85, + "cache_bytes": 522240, + "step_peak_minus_start_bytes": 534784 + }, + { + "step": 54, + "cached_positions": 86, + "cache_bytes": 528384, + "step_peak_minus_start_bytes": 540928 + }, + { + "step": 55, + "cached_positions": 87, + "cache_bytes": 534528, + "step_peak_minus_start_bytes": 547072 + }, + { + "step": 56, + "cached_positions": 88, + "cache_bytes": 540672, + "step_peak_minus_start_bytes": 553216 + }, + { + "step": 57, + "cached_positions": 89, + "cache_bytes": 546816, + "step_peak_minus_start_bytes": 559360 + }, + { + "step": 58, + "cached_positions": 90, + "cache_bytes": 552960, + "step_peak_minus_start_bytes": 565504 + }, + { + "step": 59, + "cached_positions": 91, + "cache_bytes": 559104, + "step_peak_minus_start_bytes": 571648 + }, + { + "step": 60, + "cached_positions": 92, + "cache_bytes": 565248, + "step_peak_minus_start_bytes": 577792 + }, + { + "step": 61, + "cached_positions": 93, + "cache_bytes": 571392, + "step_peak_minus_start_bytes": 583936 + }, + { + "step": 62, + "cached_positions": 94, + "cache_bytes": 577536, + "step_peak_minus_start_bytes": 590080 + }, + { + "step": 63, + "cached_positions": 95, + "cache_bytes": 583680, + "step_peak_minus_start_bytes": 596224 + }, + { + "step": 64, + "cached_positions": 96, + "cache_bytes": 589824, + "step_peak_minus_start_bytes": 602368 + }, + { + "step": 65, + "cached_positions": 97, + "cache_bytes": 595968, + "step_peak_minus_start_bytes": 608512 + }, + { + "step": 66, + "cached_positions": 98, + "cache_bytes": 602112, + "step_peak_minus_start_bytes": 614656 + }, + { + "step": 67, + "cached_positions": 99, + "cache_bytes": 608256, + "step_peak_minus_start_bytes": 620800 + }, + { + "step": 68, + "cached_positions": 100, + "cache_bytes": 614400, + "step_peak_minus_start_bytes": 626944 + }, + { + "step": 69, + "cached_positions": 101, + "cache_bytes": 620544, + "step_peak_minus_start_bytes": 633088 + }, + { + "step": 70, + "cached_positions": 102, + "cache_bytes": 626688, + "step_peak_minus_start_bytes": 639232 + }, + { + "step": 71, + "cached_positions": 103, + "cache_bytes": 632832, + "step_peak_minus_start_bytes": 645376 + }, + { + "step": 72, + "cached_positions": 104, + "cache_bytes": 638976, + "step_peak_minus_start_bytes": 651520 + }, + { + "step": 73, + "cached_positions": 105, + "cache_bytes": 645120, + "step_peak_minus_start_bytes": 657664 + }, + { + "step": 74, + "cached_positions": 106, + "cache_bytes": 651264, + "step_peak_minus_start_bytes": 663808 + }, + { + "step": 75, + "cached_positions": 107, + "cache_bytes": 657408, + "step_peak_minus_start_bytes": 669952 + }, + { + "step": 76, + "cached_positions": 108, + "cache_bytes": 663552, + "step_peak_minus_start_bytes": 676096 + }, + { + "step": 77, + "cached_positions": 109, + "cache_bytes": 669696, + "step_peak_minus_start_bytes": 682240 + }, + { + "step": 78, + "cached_positions": 110, + "cache_bytes": 675840, + "step_peak_minus_start_bytes": 688384 + }, + { + "step": 79, + "cached_positions": 111, + "cache_bytes": 681984, + "step_peak_minus_start_bytes": 694528 + }, + { + "step": 80, + "cached_positions": 112, + "cache_bytes": 688128, + "step_peak_minus_start_bytes": 700672 + }, + { + "step": 81, + "cached_positions": 113, + "cache_bytes": 694272, + "step_peak_minus_start_bytes": 706816 + }, + { + "step": 82, + "cached_positions": 114, + "cache_bytes": 700416, + "step_peak_minus_start_bytes": 712960 + }, + { + "step": 83, + "cached_positions": 115, + "cache_bytes": 706560, + "step_peak_minus_start_bytes": 719104 + }, + { + "step": 84, + "cached_positions": 116, + "cache_bytes": 712704, + "step_peak_minus_start_bytes": 725248 + }, + { + "step": 85, + "cached_positions": 117, + "cache_bytes": 718848, + "step_peak_minus_start_bytes": 731392 + }, + { + "step": 86, + "cached_positions": 118, + "cache_bytes": 724992, + "step_peak_minus_start_bytes": 737536 + }, + { + "step": 87, + "cached_positions": 119, + "cache_bytes": 731136, + "step_peak_minus_start_bytes": 743680 + }, + { + "step": 88, + "cached_positions": 120, + "cache_bytes": 737280, + "step_peak_minus_start_bytes": 749824 + }, + { + "step": 89, + "cached_positions": 121, + "cache_bytes": 743424, + "step_peak_minus_start_bytes": 755968 + }, + { + "step": 90, + "cached_positions": 122, + "cache_bytes": 749568, + "step_peak_minus_start_bytes": 762112 + }, + { + "step": 91, + "cached_positions": 123, + "cache_bytes": 755712, + "step_peak_minus_start_bytes": 768256 + }, + { + "step": 92, + "cached_positions": 124, + "cache_bytes": 761856, + "step_peak_minus_start_bytes": 774400 + }, + { + "step": 93, + "cached_positions": 125, + "cache_bytes": 768000, + "step_peak_minus_start_bytes": 780544 + }, + { + "step": 94, + "cached_positions": 126, + "cache_bytes": 774144, + "step_peak_minus_start_bytes": 786688 + }, + { + "step": 95, + "cached_positions": 127, + "cache_bytes": 780288, + "step_peak_minus_start_bytes": 792832 + }, + { + "step": 96, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 798976 + }, + { + "step": 97, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 98, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 99, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 100, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 101, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 102, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 103, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 104, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 105, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 106, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 107, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 108, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 109, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 110, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 111, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 112, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 113, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 114, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 115, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 116, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 117, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 118, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 119, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 120, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 121, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 122, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 123, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 124, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 125, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 126, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 127, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 128, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 129, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 130, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 131, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 132, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 133, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 134, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 135, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 136, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 137, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 138, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 139, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 140, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 141, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 142, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 143, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 144, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 145, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 146, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 147, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 148, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 149, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 150, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 151, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 152, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 153, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 154, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 155, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 156, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 157, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 158, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 159, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 160, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 161, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 162, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 163, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 164, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 165, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 166, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 167, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 168, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 169, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 170, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 171, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 172, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 173, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 174, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 175, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 176, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 177, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 178, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 179, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 180, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 181, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 182, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 183, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 184, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 185, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 186, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 187, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 188, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 189, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 190, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 191, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 192, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 193, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 194, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 195, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 196, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 197, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 198, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 199, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 200, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 201, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 202, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 203, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 204, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 205, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 206, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 207, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 208, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 209, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 210, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 211, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 212, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 213, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 214, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 215, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 216, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 217, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 218, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 219, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 220, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 221, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 222, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + }, + { + "step": 223, + "cached_positions": 128, + "cache_bytes": 786432, + "step_peak_minus_start_bytes": 2753536 + } + ] + } + ] + } +} diff --git a/experiments/records/kv_cache/20261022T053547Z_kv_mps_float16_57be26d.json b/experiments/records/kv_cache/20261022T053547Z_kv_mps_float16_57be26d.json new file mode 100644 index 0000000..929462b --- /dev/null +++ b/experiments/records/kv_cache/20261022T053547Z_kv_mps_float16_57be26d.json @@ -0,0 +1,35862 @@ +{ + "record_version": 2, + "created_utc": "20261022T053547Z", + "experiment": "kv_cache_scaling", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "mps", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "57be26dc620ddc37c1928c7d7a674e4ff0d3d39f", + "branch": "experiments/sdpa-training-v2", + "dirty": false + }, + "command": "python scripts/benchmark_kv_scaling.py --device mps --dtype float16" + }, + "settings": { + "dtype": "float16", + "position": "learned", + "prompt_lens": [ + 1, + 16, + 128, + 512, + 1024 + ], + "batch_sizes": [ + 1, + 8 + ], + "new_tokens": 64, + "warmup": 1, + "repeats": 5, + "seed": 0, + "modes": { + "cached": { + "attention_backend": "sdpa", + "use_cache": true + }, + "uncached_sdpa": { + "attention_backend": "sdpa", + "use_cache": false + }, + "uncached_mha": { + "attention_backend": "mha", + "use_cache": false + } + }, + "weights": "seeded random initialization (timing does not depend on trained values)", + "grid_model_config": { + "vocab_size": 256, + "context_length": 1088, + "embedding_dim": 192, + "num_layers": 4, + "num_heads": 6, + "dropout": 0.0, + "positional_encoding": "learned", + "norm": "rmsnorm", + "activation": "swiglu", + "rope_theta": 10000.0 + }, + "window_model_config": { + "vocab_size": 256, + "context_length": 128, + "embedding_dim": 192, + "num_layers": 4, + "num_heads": 6, + "dropout": 0.0, + "positional_encoding": "learned", + "norm": "rmsnorm", + "activation": "swiglu", + "rope_theta": 10000.0 + }, + "decoding": "greedy argmax; step 0 is the prompt prefill", + "timer": "time.perf_counter", + "synchronization": "device synchronize before and after every step (no-op on CPU); backend/mode order rotated each repeat", + "units": { + "raw": "seconds per step, one list per measured repeat", + "summaries": "milliseconds (ms fields) or tokens/second" + } + }, + "results": { + "grid": [ + { + "batch": 1, + "prompt_len": 1, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_mha": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 1.4871669991407543, + "p95": 1.5692745975684375, + "min": 1.3592079922091216, + "max": 1.575415997649543, + "iqr": 0.11658399307634681, + "mean": 1.4789249980822206, + "cv": 0.059158586487837034 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.561915996717289, + "p95": 1.7575163001311012, + "min": 1.470458009862341, + "max": 2.6388330006739125, + "iqr": 0.07325050682993606, + "mean": 1.6056320384248264, + "cv": 0.10397342310706954 + }, + "total_ms": { + "count": 5, + "median": 101.33237502304837, + "p95": 107.99091842200141, + "min": 99.73187802825123, + "max": 109.65091800608207, + "iqr": 0.24829413450788707, + "mean": 102.63374341884628, + "cv": 0.0387736131196081 + }, + "tokens_per_second": { + "count": 5, + "median": 631.5849202729434, + "p95": 639.9805062254927, + "min": 583.6704440217279, + "max": 641.7205939095081, + "iqr": 1.5508018230167409, + "mean": 624.2930934720049, + "cv": 0.03700452470831303 + }, + "median_step_ms_by_index": [ + 1.4871669991407543, + 1.6522500081919134, + 1.5595830045640469, + 1.630124999792315, + 1.5447500045411289, + 1.605291006853804, + 1.5454170061275363, + 1.5420840063598007, + 1.5450829960172996, + 1.52395800978411, + 1.5772500046296045, + 1.5248749987222254, + 1.5364999999292195, + 1.5390000044135377, + 1.5287499991245568, + 1.5174169966485351, + 1.5439579874509946, + 1.6365000046789646, + 1.5190839912975207, + 1.5946670027915388, + 1.513374998467043, + 1.5682500088587403, + 1.5324580017477274, + 1.5509999939240515, + 1.560750009957701, + 1.5941669989842921, + 1.535415998660028, + 1.573917004861869, + 1.5597499877912924, + 1.564624995808117, + 1.5565840003546327, + 1.558290998218581, + 1.601208990905434, + 1.5618329925928265, + 1.5449580096174031, + 1.5959159936755896, + 1.5676249895477667, + 1.5432920044986531, + 1.561583994771354, + 1.5370839973911643, + 1.574166992213577, + 1.5506669878959656, + 1.561915996717289, + 1.563249999890104, + 1.5582919877488166, + 1.6350420046364889, + 1.5711250016465783, + 1.568958003190346, + 1.5631669957656413, + 1.565749989822507, + 1.5593749994877726, + 1.56787499145139, + 1.5830420015845448, + 1.6510830027982593, + 1.5698749921284616, + 1.5951249952195212, + 1.5776660002302378, + 1.5792089980095625, + 1.5786250005476177, + 1.643832991248928, + 1.6100000066217035, + 1.5712089953012764, + 1.6474580042995512, + 1.5815000078873709 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0015447089972440153, + 0.0016710420022718608, + 0.001659208006458357, + 0.0015663749945815653, + 0.0015057920099934563, + 0.001605291006853804, + 0.00164154201047495, + 0.0015420840063598007, + 0.001532583002699539, + 0.0015255829930538312, + 0.0016446670051664114, + 0.0015248749987222254, + 0.001527790998807177, + 0.0015286660054698586, + 0.0015400840056827292, + 0.0016504590021213517, + 0.001510833011707291, + 0.0016906249948078766, + 0.001571459011756815, + 0.0017550409975228831, + 0.001470458009862341, + 0.0016768750065239146, + 0.001544792001368478, + 0.0015595000004395843, + 0.0016887090023374185, + 0.0015451250073965639, + 0.001535415998660028, + 0.00166095799067989, + 0.0015597499877912924, + 0.0015686250117141753, + 0.0015586669906042516, + 0.0015793330094311386, + 0.0015656659961678088, + 0.0015618329925928265, + 0.0015735000051790848, + 0.0016737920086598024, + 0.0015676249895477667, + 0.0016115000034915283, + 0.001561583994771354, + 0.0015350419998867437, + 0.001659250003285706, + 0.0015506669878959656, + 0.0016644999996060506, + 0.0015655000024707988, + 0.0015362919948529452, + 0.001560916003654711, + 0.0016108340059872717, + 0.0015594580036122352, + 0.0015631669957656413, + 0.0015627919929102063, + 0.0015595420118188486, + 0.00156695899204351, + 0.0015884999884292483, + 0.0016606250137556344, + 0.0016497920005349442, + 0.001535208008135669, + 0.001570667009218596, + 0.0015521659952355549, + 0.0015562089974991977, + 0.0016098750056698918, + 0.0015687919949414209, + 0.00165125000057742, + 0.0015326249995268881, + 0.0015778750093886629 + ], + [ + 0.0014871669991407543, + 0.0016897910099942237, + 0.0015595830045640469, + 0.001668582990532741, + 0.0015447500045411289, + 0.001589167004567571, + 0.0015430420025950298, + 0.0015273329918272793, + 0.0015450829960172996, + 0.0015239580097841099, + 0.0015385830047307536, + 0.0015933339891489595, + 0.001544041995657608, + 0.0015390000044135377, + 0.001526166990515776, + 0.001513584007625468, + 0.0015439579874509946, + 0.0016365000046789646, + 0.001505957989138551, + 0.0015946670027915388, + 0.001513374998467043, + 0.0015682500088587403, + 0.0015324580017477274, + 0.0015509999939240515, + 0.001560750009957701, + 0.001651417012908496, + 0.0016652500053169206, + 0.001573917004861869, + 0.0015671250002924353, + 0.0016508329863427207, + 0.0015565840003546327, + 0.0016547919949516654, + 0.001601208990905434, + 0.0015723749966127798, + 0.0015328330046031624, + 0.0015959159936755896, + 0.001531583009636961, + 0.00165891699725762, + 0.001574665991938673, + 0.00163162499666214, + 0.0015637080068700016, + 0.001574916997924447, + 0.001561915996717289, + 0.002153666995582171, + 0.0016627080040052533, + 0.0016350420046364889, + 0.0015957090072333813, + 0.001577167000505142, + 0.002526959011447616, + 0.001565749989822507, + 0.001545332997920923, + 0.0015742079995106906, + 0.0017129589978139848, + 0.0016510830027982593, + 0.0015776250074850395, + 0.001556999995955266, + 0.002527167001971975, + 0.0024954590044217184, + 0.002554749997216277, + 0.0025408340006833896, + 0.002597040991531685, + 0.0026388330006739125, + 0.0022785000037401915, + 0.0020234580006217584 + ], + [ + 0.0013592079922091216, + 0.0016522500081919134, + 0.0015433329972438514, + 0.001630124999792315, + 0.0015265830006683245, + 0.0015610000118613243, + 0.0016230830078711733, + 0.0015290000010281801, + 0.0016015419969335198, + 0.0015175419976003468, + 0.0015772500046296045, + 0.0015248330018948764, + 0.0015530829987255856, + 0.0015437080000992864, + 0.0015230419958243147, + 0.0015254169993568212, + 0.0015448750054929405, + 0.0016440830077044666, + 0.0015598750032950193, + 0.001538500000606291, + 0.0015405000012833625, + 0.0015329169982578605, + 0.001523625003756024, + 0.0015410839987453073, + 0.0015274170000338927, + 0.0015941669989842921, + 0.0016534999886061996, + 0.001643958006752655, + 0.001554709000629373, + 0.0015406250022351742, + 0.0015655000024707988, + 0.001558290998218581, + 0.0017759170004865155, + 0.0015527089999523014, + 0.0015449580096174031, + 0.0016534159949515015, + 0.0015599159960402176, + 0.0015432920044986531, + 0.0015419580013258383, + 0.0015370839973911643, + 0.0015919170109555125, + 0.001546875006170012, + 0.0015446670004166663, + 0.0015277079946827143, + 0.0015582919877488166, + 0.0018962920003104955, + 0.0015711250016465783, + 0.001568958003190346, + 0.0015470419893972576, + 0.0016132919990923256, + 0.0015593749994877726, + 0.00156787499145139, + 0.0015447909972863272, + 0.0017632920062169433, + 0.0014991249918239191, + 0.001998249994358048, + 0.0015776660002302378, + 0.0016775419935584068, + 0.0015786250005476177, + 0.0016056669992394745, + 0.0016194580093724653, + 0.0015640830097254366, + 0.0016474580042995512, + 0.0015991249965736642 + ], + [ + 0.001575415997649543, + 0.0016151660092873499, + 0.001627040997846052, + 0.0016390839882660657, + 0.0015596659941365942, + 0.0016604580014245585, + 0.0015454170061275363, + 0.0015617919998476282, + 0.0015507909993175417, + 0.0015583329950459301, + 0.0016314999957103282, + 0.0015562920016236603, + 0.0015364999999292195, + 0.0015394170040963218, + 0.0015287499991245568, + 0.001507666995166801, + 0.0016541670047445223, + 0.0015749159938422963, + 0.0015190839912975207, + 0.001626500001293607, + 0.0015201250062091276, + 0.00189124999451451, + 0.0015834170044399798, + 0.0016705419984646142, + 0.0015965420025167987, + 0.0016869170067366213, + 0.001512458999059163, + 0.0015184579970082268, + 0.0015521250024903566, + 0.001564624995808117, + 0.0015372909983852878, + 0.0015520410088356584, + 0.0015411659987876192, + 0.001715790989692323, + 0.0015637500036973506, + 0.0015364999999292195, + 0.001576208000187762, + 0.0015394999936688691, + 0.0015692919987486675, + 0.0016344589967047796, + 0.001566499995533377, + 0.0016988340066745877, + 0.0016578329959884286, + 0.001563249999890104, + 0.0015523330075666308, + 0.0015397920069517568, + 0.0015440830029547215, + 0.0015306249988498166, + 0.0015374999929917976, + 0.001523291997727938, + 0.001604042001417838, + 0.0015342919941758737, + 0.0015367090090876445, + 0.0015839999978197739, + 0.0015450000064447522, + 0.001600999996298924, + 0.001527874992461875, + 0.0015555419959127903, + 0.0016386249917559326, + 0.001643832991248928, + 0.001524417006294243, + 0.0015712089953012764, + 0.001526499996543862, + 0.0015651249996153638 + ], + [ + 0.0014281250041676685, + 0.0015660420031053945, + 0.0015548749943263829, + 0.0015255000034812838, + 0.0015454579988727346, + 0.0016241249977611005, + 0.0014999999984866008, + 0.0015539580053882673, + 0.0015194169973256066, + 0.0015107920044101775, + 0.0015370000037364662, + 0.0014755000011064112, + 0.001484459004132077, + 0.0014914160128682852, + 0.0015601670020259917, + 0.0015174169966485351, + 0.0015148329985095188, + 0.0015728340076748282, + 0.001505542008089833, + 0.0015233750018524006, + 0.001472459000069648, + 0.0015502499882131815, + 0.0015180420014075935, + 0.0015317090001190081, + 0.0015187920071184635, + 0.0015217910113278776, + 0.001529167013359256, + 0.0015141659969231114, + 0.0016589580045547336, + 0.0015248339914251119, + 0.0015267920098267496, + 0.0015349159948527813, + 0.0016394999984186143, + 0.0015391249908134341, + 0.0015221249923342839, + 0.001537167001515627, + 0.0015747500001452863, + 0.001524207997135818, + 0.0015610830014338717, + 0.001487790999817662, + 0.001574166992213577, + 0.0015383330028271303, + 0.0015491249941987917, + 0.0015350000030593947, + 0.0016047499957494438, + 0.0016972500015981495, + 0.0015574580029351637, + 0.0015934999973978847, + 0.0015942500031087548, + 0.0016844999918248504, + 0.0015168329991865903, + 0.0017825419927248731, + 0.0015830420015845448, + 0.001609917002497241, + 0.0015698749921284616, + 0.0015951249952195212, + 0.0015950830129440874, + 0.0015792089980095625, + 0.0015687919949414209, + 0.0016865420038811862, + 0.0016100000066217035, + 0.0015568749950034544, + 0.001669749995926395, + 0.0015815000078873709 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + ] + } + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 1.4497090014629066, + "p95": 1.7063077946659178, + "min": 1.396291991113685, + "max": 1.760165992891416, + "iqr": 0.06016700353939086, + "mean": 1.5055499970912933, + "cv": 0.09722301160084941 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.4167500048642978, + "p95": 1.5580958002828993, + "min": 1.1702079937094823, + "max": 2.1240000060060993, + "iqr": 0.12747949949698523, + "mean": 1.4206630446659845, + "cv": 0.09030415672725085 + }, + "total_ms": { + "count": 5, + "median": 89.76303400413599, + "p95": 94.58030642708763, + "min": 89.68862300389446, + "max": 95.67120604333468, + "iqr": 0.5196699203224853, + "mean": 91.00732181104831, + "cv": 0.028748680451318887 + }, + "tokens_per_second": { + "count": 5, + "median": 712.9883777886907, + "p95": 713.5665261522283, + "min": 668.9578050370864, + "max": 713.579915227609, + "iqr": 4.110006190285162, + "mean": 703.6884063129024, + "cv": 0.0276995052485275 + }, + "median_step_ms_by_index": [ + 1.4497090014629066, + 1.4868749858578667, + 1.4949579926906154, + 1.4805419923504815, + 1.4715420111315325, + 1.473750002332963, + 1.497542005381547, + 1.5500000008614734, + 1.204667001729831, + 1.2100419990019873, + 1.2180000048829243, + 1.2352500052656978, + 1.2517080031102523, + 1.2525409983936697, + 1.2344999995548278, + 1.281791992369108, + 1.3511660072254017, + 1.391041005263105, + 1.3361250021262094, + 1.4222079917090014, + 1.3924580125603825, + 1.3136659981682897, + 1.3890410045860335, + 1.3370410015340894, + 1.3692920038010925, + 1.3908749970141798, + 1.309249986661598, + 1.346167002338916, + 1.3882909988751635, + 1.320583003689535, + 1.3996249908814207, + 1.3399590097833425, + 1.3909160043112934, + 1.444207999156788, + 1.3455000007525086, + 1.4047089935047552, + 1.405375005560927, + 1.385582989314571, + 1.4355830062413588, + 1.4256249996833503, + 1.4102910063229501, + 1.4127920003375039, + 1.4167500048642978, + 1.3910839916206896, + 1.4566250029020011, + 1.432624994777143, + 1.4777079923078418, + 1.496042008511722, + 1.4510420005535707, + 1.4439579972531646, + 1.5102500037755817, + 1.520958001492545, + 1.5332909970311448, + 1.4799589989706874, + 1.5074999973876402, + 1.4717500016558915, + 1.5113749977899715, + 1.5157500019995496, + 1.445208996301517, + 1.4720410108566284, + 1.5161670016823336, + 1.4668340008938685, + 1.4942910056561232, + 1.5154169959714636 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.001430707998224534, + 0.0015899999998509884, + 0.0015240409993566573, + 0.0014475420030066743, + 0.001461374995415099, + 0.001473750002332963, + 0.0014648750075139105, + 0.001440249994629994, + 0.0011702079937094823, + 0.0012100419990019873, + 0.0012180000048829243, + 0.0013040409976383671, + 0.0012505409977165982, + 0.0012247919949004427, + 0.0012701249943347648, + 0.0012569579994305968, + 0.0013135420012986287, + 0.001439916988601908, + 0.001296791000640951, + 0.0013756249973084778, + 0.0013500420027412474, + 0.0014363749942276627, + 0.0014034160121809691, + 0.0013175420026527718, + 0.0013759580033365637, + 0.0013942499936092645, + 0.001309249986661598, + 0.0013822500040987507, + 0.0012974160054000095, + 0.001384458999382332, + 0.0014130000054137781, + 0.001323499993304722, + 0.0014117499958956614, + 0.0013571249874075875, + 0.0013366659986786544, + 0.001389957993524149, + 0.0013909999979659915, + 0.0014654170081485063, + 0.0013580840022768825, + 0.0013390000094659626, + 0.0014195410039974377, + 0.0014492499904008582, + 0.0013585840060841292, + 0.0013910839916206896, + 0.0015299160004360601, + 0.0014283329946920276, + 0.0014777079923078418, + 0.001496042008511722, + 0.0014510420005535707, + 0.0014439579972531646, + 0.0015762919938424602, + 0.0014875420019961894, + 0.0015567079972242936, + 0.0014799589989706874, + 0.0015997500013327226, + 0.001446499998564832, + 0.0015376670053228736, + 0.0014660419983556494, + 0.0013799160078633577, + 0.0015404170117108151, + 0.0015613340074196458, + 0.0015005000022938475, + 0.0015236249892041087, + 0.0015154169959714636 + ], + [ + 0.001760165992891416, + 0.0018220420024590567, + 0.0020212079980410635, + 0.0020273330010240898, + 0.0020739579922519624, + 0.002052125011687167, + 0.0021240000060060993, + 0.0018407910101814196, + 0.0015358749951701611, + 0.0015161250048549846, + 0.001480375009123236, + 0.001457374994060956, + 0.001284583006054163, + 0.0012851249921368435, + 0.0012870000064140186, + 0.0014426670095417649, + 0.0013511660072254017, + 0.0014698750019306317, + 0.0013906669919379056, + 0.0014430000010179356, + 0.0014119580009719357, + 0.0013634170027216896, + 0.001400249995640479, + 0.0013959580101072788, + 0.0013026249944232404, + 0.0013557499914895743, + 0.001379124994855374, + 0.001346167002338916, + 0.0013995829940540716, + 0.0013102499942760915, + 0.001484916007029824, + 0.0014055420033400878, + 0.0013909160043112934, + 0.001444207999156788, + 0.0014590000064345077, + 0.0014047089935047552, + 0.001395832994603552, + 0.0013591669994639233, + 0.0013517499901354313, + 0.0014491250040009618, + 0.001458833008655347, + 0.0013683750003110617, + 0.0014558749971911311, + 0.0014786250103497878, + 0.0014211250090738758, + 0.0014696669968543574, + 0.0015024589956738055, + 0.001427791008609347, + 0.0015083749894984066, + 0.0014501670084428042, + 0.001412833997164853, + 0.001505209002061747, + 0.0015332909970311448, + 0.0014384579990291968, + 0.0015012920048320666, + 0.0014717500016558915, + 0.0014689170056954026, + 0.0014921670081093907, + 0.0015247919945977628, + 0.0014203330065356568, + 0.0014001249946886674, + 0.0015241249930113554, + 0.001454124998417683, + 0.0015067909989738837 + ], + [ + 0.001396291991113685, + 0.00144749999162741, + 0.0014949579926906154, + 0.0014262500044424087, + 0.001436582999303937, + 0.0014749999972991645, + 0.0014510000037262216, + 0.0015500000008614734, + 0.0012010829959763214, + 0.001200208003865555, + 0.0011994160013273358, + 0.0012205410021124408, + 0.0012976250000065193, + 0.0012525409983936697, + 0.001215124997543171, + 0.0012980000028619543, + 0.0014198749995557591, + 0.001391041005263105, + 0.0012970410025445744, + 0.0013962080120109022, + 0.0013924580125603825, + 0.0012972910044481978, + 0.0013581249950220808, + 0.0013370410015340894, + 0.001480999999330379, + 0.0014238339936127886, + 0.001306542006204836, + 0.0013644169957842678, + 0.0013882909988751635, + 0.0013046660023974255, + 0.0013507500116247684, + 0.0013399590097833425, + 0.0013974580069771037, + 0.0014508329913951457, + 0.0013455000007525086, + 0.0014117499958956614, + 0.0014295419969130307, + 0.001385582989314571, + 0.0014355830062413588, + 0.0014810410066274926, + 0.0013709159975405782, + 0.0014322500064736232, + 0.0014167500048642978, + 0.0013961670047137886, + 0.0013609169982373714, + 0.001432624994777143, + 0.0015067909989738837, + 0.0015073329996084794, + 0.001440415988327004, + 0.0013964999961899593, + 0.0016602500109001994, + 0.001532207999844104, + 0.0014634169929195195, + 0.0013884999934816733, + 0.0015074999973876402, + 0.0014853339962428436, + 0.0015113749977899715, + 0.0015266249974956736, + 0.0014452089963015169, + 0.0013949580024927855, + 0.0013725419994443655, + 0.0014323340001283213, + 0.0015187079989118502, + 0.0015154580032685772 + ], + [ + 0.001490875001763925, + 0.0014868749858578667, + 0.001486916997237131, + 0.0014805419923504815, + 0.0015031249931780621, + 0.0014643340109614655, + 0.0015192080027190968, + 0.0014815829927101731, + 0.001204667001729831, + 0.0012341249966993928, + 0.0012329169985605404, + 0.0012352500052656978, + 0.0012517080031102523, + 0.0012605840020114556, + 0.0012344999995548278, + 0.001281791992369108, + 0.0013925419916631654, + 0.0013527910050470382, + 0.0013979590003145859, + 0.0014222079917090014, + 0.0013308749912539497, + 0.001233957998920232, + 0.0012479160068323836, + 0.0012131249968660995, + 0.0012624590017367154, + 0.0013770830119028687, + 0.001284250000026077, + 0.0013167500001145527, + 0.0013761669979430735, + 0.001320583003689535, + 0.0013953750021755695, + 0.0013091660075588152, + 0.0013306250039022416, + 0.0014115410012891516, + 0.0013449590042000636, + 0.0013364170008571818, + 0.001405375005560927, + 0.0013885419903090224, + 0.0014779999910388142, + 0.0014256249996833503, + 0.0014102910063229501, + 0.001345374999800697, + 0.0014764579973416403, + 0.0013825830101268366, + 0.0014795409952057526, + 0.0015131660038605332, + 0.0013594580086646602, + 0.0014986250025685877, + 0.0015192920109257102, + 0.0014919169916538522, + 0.0015102500037755817, + 0.001520958001492545, + 0.0014488330052699894, + 0.0015462500014109537, + 0.0015350000030593947, + 0.0015336659998865798, + 0.0014515410002786666, + 0.0015157500019995496, + 0.0015412499924423173, + 0.0014720410108566284, + 0.0015161670016823336, + 0.0014668340008938685, + 0.0014942910056561232, + 0.0014642080059275031 + ], + [ + 0.0014497090014629066, + 0.0014136660029180348, + 0.0014524169964715838, + 0.0015220000059343874, + 0.0014715420111315325, + 0.0014722920022904873, + 0.001497542005381547, + 0.001554665999719873, + 0.0012107910006307065, + 0.0011999170092167333, + 0.0012045000039506704, + 0.001210542002809234, + 0.0012255000037839636, + 0.0012080000014975667, + 0.0012169999972684309, + 0.001263874990399927, + 0.0013197500084061176, + 0.0013774590042885393, + 0.0013361250021262094, + 0.001430666001397185, + 0.0014326659875223413, + 0.0013136659981682897, + 0.0013890410045860335, + 0.0013533329911297187, + 0.0013692920038010925, + 0.0013908749970141798, + 0.0013592500035883859, + 0.0012994589924346656, + 0.0014018749934621155, + 0.0013300839927978814, + 0.0013996249908814207, + 0.00141891700332053, + 0.0013356669951463118, + 0.0014641659945482388, + 0.0013878750032745302, + 0.0014093749923631549, + 0.0014303749921964481, + 0.001383250011713244, + 0.0014629999932367355, + 0.0013362919999053702, + 0.0013324999890755862, + 0.0014127920003375039, + 0.0013878750032745302, + 0.001363625007797964, + 0.0014566250029020011, + 0.0013851660041837022, + 0.00134954099485185, + 0.0013735410029767081, + 0.0013860839972039685, + 0.001421374996425584, + 0.0014705409994348884, + 0.0015837079990888014, + 0.0015974580019246787, + 0.0014871670136926696, + 0.0014444999978877604, + 0.0014374580059666187, + 0.0016314159875037149, + 0.0015546670038020238, + 0.0014196669944794849, + 0.001531667003291659, + 0.0015938750002533197, + 0.0014643330068793148, + 0.001377416992909275, + 0.0015195830055745319 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_mha": { + "first_token_ms": { + "count": 5, + "median": 1.5404169971589, + "p95": 1.5737833979073912, + "min": 1.4669999945908785, + "max": 1.5799999964656308, + "iqr": 0.06991700502112508, + "mean": 1.5230667981086299, + "cv": 0.03165188642786238 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.530833003926091, + "p95": 1.6566794060054235, + "min": 1.2607919925358146, + "max": 1.7630419897614047, + "iqr": 0.12400049308780581, + "mean": 1.5154425656729718, + "cv": 0.06421485896470804 + }, + "total_ms": { + "count": 5, + "median": 97.06479198939633, + "p95": 97.28555387118831, + "min": 96.5363710274687, + "max": 97.33307808346581, + "iqr": 0.14541296695824713, + "mean": 96.99594843550585, + "cv": 0.003012907273316014 + }, + "tokens_per_second": { + "count": 5, + "median": 659.3533936279549, + "p95": 662.3968086920462, + "min": 657.5359709174945, + "max": 662.9625634237823, + "iqr": 0.9886354717637005, + "mean": 659.8261744055343, + "cv": 0.0030181826624316194 + }, + "median_step_ms_by_index": [ + 1.5404169971589, + 1.5611249982612208, + 1.571832995978184, + 1.5691250009695068, + 1.6644999996060506, + 1.5817080129636452, + 1.617166999494657, + 1.6314170061377808, + 1.3142079988028854, + 1.321333009400405, + 1.3152080064173788, + 1.3367499923333526, + 1.3303339947015047, + 1.362541996059008, + 1.3485000090440735, + 1.3875419972464442, + 1.3770410005236045, + 1.4675840066047385, + 1.4041250105947256, + 1.4962090062908828, + 1.5032499941298738, + 1.5019999991636723, + 1.4982919965405017, + 1.474541990319267, + 1.50320801185444, + 1.4697909937240183, + 1.4654170081485063, + 1.5376249939436093, + 1.4395420002983883, + 1.4753750001545995, + 1.47312501212582, + 1.4411250012926757, + 1.5454170061275363, + 1.5595829900121316, + 1.4847080019535497, + 1.5100839955266565, + 1.5439169947057962, + 1.4710419927723706, + 1.5334579948103055, + 1.561540993861854, + 1.4964160072850063, + 1.5172919956967235, + 1.4832910092081875, + 1.5666249964851886, + 1.5165420045377687, + 1.5416250098496675, + 1.59087500651367, + 1.55033401097171, + 1.554125003167428, + 1.587791004567407, + 1.5921669983072206, + 1.5843330038478598, + 1.5892079973127693, + 1.600957999471575, + 1.6218340024352074, + 1.6037499881349504, + 1.6222499980358407, + 1.5735000051790848, + 1.584582991199568, + 1.6325839969795197, + 1.575791000504978, + 1.6141670057550073, + 1.64170800417196, + 1.6172920004464686 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0015799999964656308, + 0.0015611249982612208, + 0.0014934580103727058, + 0.0015222920046653599, + 0.0015658749907743186, + 0.0015587910020258278, + 0.0015810829936526716, + 0.0015485830081161112, + 0.0013142079988028854, + 0.0013285839959280565, + 0.0013262909924378619, + 0.0013429999962681904, + 0.001373833991237916, + 0.001448541006539017, + 0.0013485000090440735, + 0.0013780829904135317, + 0.001372040991554968, + 0.0014778750046389177, + 0.00147758299135603, + 0.001515084004495293, + 0.001564916005008854, + 0.0014808749983785674, + 0.0014057499938644469, + 0.0014561249990947545, + 0.0014279169990913942, + 0.001436417005606927, + 0.0015191249985946342, + 0.001502166996942833, + 0.0014395420002983883, + 0.0014327079989016056, + 0.0014662499888800085, + 0.0014016249915584922, + 0.0015506250056205317, + 0.0015131249965634197, + 0.0014847080019535497, + 0.0015345830033766106, + 0.001472750009270385, + 0.0014388749987119809, + 0.0015448329941136762, + 0.001561540993861854, + 0.001477750003687106, + 0.0015172919956967235, + 0.001474833014071919, + 0.001510458008851856, + 0.0015948750078678131, + 0.001596750007593073, + 0.0016107080009533092, + 0.001462125001125969, + 0.001554125003167428, + 0.001587791004567407, + 0.001558290998218581, + 0.0016654589999234304, + 0.0015618339966749772, + 0.001600957999471575, + 0.0015720410010544583, + 0.0015938750002533197, + 0.0016222499980358407, + 0.001582917000632733, + 0.0015858340047998354, + 0.001647417011554353, + 0.001558083007694222, + 0.0016141670057550073, + 0.001658667009905912, + 0.0015785829891683534 + ], + [ + 0.0014789999986533076, + 0.0015295829944079742, + 0.0015949160006130114, + 0.0015830420015845448, + 0.0015498749999096617, + 0.0015467500052182004, + 0.001569708008901216, + 0.0016585000121267512, + 0.001342208997812122, + 0.0012607919925358146, + 0.0012957500002812594, + 0.0013319169956957921, + 0.0013303339947015047, + 0.0013335000112419948, + 0.0013591669994639233, + 0.0014804579986957833, + 0.00148095800250303, + 0.0014291250117821619, + 0.0013896250020479783, + 0.0015304160042433068, + 0.0014313749998109415, + 0.0015019999991636723, + 0.0014982919965405017, + 0.001474541990319267, + 0.0017630419897614047, + 0.0014923330018064007, + 0.001390583987813443, + 0.0014228329964680597, + 0.0014993330114521086, + 0.001487625006120652, + 0.001454124998417683, + 0.0015511660021729767, + 0.0014483749982900918, + 0.0015278339997166768, + 0.0015451669896719977, + 0.0014747910026926547, + 0.001492208000854589, + 0.0015290000010281801, + 0.00153854199743364, + 0.0014608750061597675, + 0.001514083007350564, + 0.001443790999474004, + 0.001457959006074816, + 0.0015732500032754615, + 0.001586124999448657, + 0.0015570830000797287, + 0.00159087500651367, + 0.0015972500114003196, + 0.0016012909909477457, + 0.0015327500004786998, + 0.0015961250028340146, + 0.0016384170012315735, + 0.0015435829991474748, + 0.0015677919873269275, + 0.0016500409983564168, + 0.0016565420082770288, + 0.0015774579951539636, + 0.0015658329939469695, + 0.0016343330062227324, + 0.0016921670030569658, + 0.001606124991667457, + 0.0015542089968221262, + 0.00164170800417196, + 0.0016570000007050112 + ], + [ + 0.0015489170036744326, + 0.00170899998920504, + 0.0015845409943722188, + 0.0015691250009695068, + 0.0016986249975161627, + 0.0015895830001682043, + 0.001617166999494657, + 0.0016314170061377808, + 0.0013436669978545979, + 0.0013860420003766194, + 0.0013287920010043308, + 0.0013135839981259778, + 0.0013264579902170226, + 0.0013228750030975789, + 0.001332084008026868, + 0.0014576660032616928, + 0.0014090830081840977, + 0.0014675840066047385, + 0.0014041250105947256, + 0.001476792007451877, + 0.0015060410078149289, + 0.0015566250076517463, + 0.0014024159900145605, + 0.0014477920049102977, + 0.001427333991159685, + 0.0014697909937240183, + 0.001412999990861863, + 0.0015395000082207844, + 0.0014048749872017652, + 0.0014616249973187223, + 0.00147312501212582, + 0.001423292007530108, + 0.0014602919982280582, + 0.0015703750104876235, + 0.001476834004279226, + 0.0015423750010086223, + 0.0015610840055160224, + 0.0014669170050183311, + 0.0015334579948103055, + 0.0015498749999096617, + 0.0014764160005142912, + 0.0015260410000337288, + 0.0015037919947644696, + 0.001614166991203092, + 0.0015165420045377687, + 0.0015082089958013967, + 0.001611832994967699, + 0.00155033401097171, + 0.0016144579858519137, + 0.001527250002254732, + 0.0015674580063205212, + 0.0015843330038478598, + 0.0015892079973127693, + 0.0015247910050675273, + 0.0016218340024352074, + 0.0015999580064089969, + 0.0016379169974243268, + 0.0015395000082207844, + 0.001584582991199568, + 0.0016325839969795197, + 0.0016396660066675395, + 0.0016551250009797513, + 0.0014905830030329525, + 0.0016297090041916817 + ], + [ + 0.0015404169971589, + 0.001549500011606142, + 0.0015588329988531768, + 0.0015625830128556117, + 0.0016644999996060506, + 0.0015835840022191405, + 0.0016806670028017834, + 0.001698874999419786, + 0.0013080000062473118, + 0.001301209005760029, + 0.0013152080064173788, + 0.0013367499923333526, + 0.001299665993428789, + 0.0013859169994248077, + 0.0013150829909136519, + 0.0013357499992707744, + 0.0013770410005236045, + 0.0013697079994017258, + 0.0013597499928437173, + 0.0013279169943416491, + 0.0013544999965233728, + 0.0014321250055218115, + 0.001601708005182445, + 0.0015025410102680326, + 0.00150320801185444, + 0.001466875008190982, + 0.0014654170081485063, + 0.0015384159923996776, + 0.001530833003926091, + 0.001586124999448657, + 0.0015864579909248278, + 0.0015091660025063902, + 0.0016125420079333708, + 0.0016052080027293414, + 0.0016182079998543486, + 0.0015100839955266565, + 0.0015490410005440935, + 0.0015845000016270205, + 0.0014837500057183206, + 0.0015656250034226105, + 0.0016288750048261136, + 0.001526916996226646, + 0.0014832910092081875, + 0.0015633750008419156, + 0.0014481249963864684, + 0.0014615410036640242, + 0.0015662079968024045, + 0.0016482919891132042, + 0.0015514590049861, + 0.001593374996446073, + 0.0016060000052675605, + 0.0015370830078609288, + 0.001640708011109382, + 0.0016294170054607093, + 0.0016294579982059076, + 0.0016037499881349504, + 0.0015788340097060427, + 0.001613208994967863, + 0.0015467500052182004, + 0.0015304159896913916, + 0.001575791000504978, + 0.001643625000724569, + 0.0016570830048294738, + 0.0015922080056043342 + ], + [ + 0.0014669999945908785, + 0.001562541991006583, + 0.001571832995978184, + 0.0015870829956838861, + 0.0016682919958839193, + 0.0015817080129636452, + 0.0017126250022556633, + 0.0015938750002533197, + 0.0012854579981649294, + 0.001321333009400405, + 0.001302500008023344, + 0.001405708011589013, + 0.001364208001177758, + 0.001362541996059008, + 0.0014824160025455058, + 0.0013875419972464442, + 0.0013536249898606911, + 0.0014835000038146973, + 0.0014892919862177223, + 0.0014962090062908828, + 0.0015032499941298738, + 0.001541416990221478, + 0.0015872079966356978, + 0.0015331670001614839, + 0.0015090000088093802, + 0.001474083008361049, + 0.0015039999998407438, + 0.0015376249939436093, + 0.001373584003886208, + 0.0014753750001545995, + 0.001488958005211316, + 0.0014411250012926757, + 0.0015454170061275363, + 0.0015595829900121316, + 0.0014277079899329692, + 0.0014822500088484958, + 0.0015439169947057962, + 0.0014710419927723706, + 0.0014930000033928081, + 0.0015627910033799708, + 0.0014964160072850063, + 0.0015065829938976094, + 0.0015307919966289774, + 0.0015666249964851886, + 0.0014587910118279979, + 0.0015416250098496675, + 0.0014591669896617532, + 0.0015054590039653704, + 0.0015263749955920503, + 0.001611667001270689, + 0.0015921669983072206, + 0.0015275000041583553, + 0.0015984159981599078, + 0.0016637920052744448, + 0.00159104100021068, + 0.0016435419966001064, + 0.0016587919963058084, + 0.0015735000051790848, + 0.001581749995239079, + 0.0016264999867416918, + 0.0015460419963346794, + 0.0015660839999327436, + 0.0015410830092150718, + 0.0016172920004464686 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "speedup_cached_vs_uncached_sdpa": 0.8858277917962458, + "speedup_cached_vs_uncached_mha": 0.9578852954676985 + }, + { + "batch": 1, + "prompt_len": 16, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_mha": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 1.4136670070001855, + "p95": 1.5124750032555312, + "min": 1.2856670073233545, + "max": 1.5331250033341348, + "iqr": 0.03545801155269146, + "mean": 1.4113502023974434, + "cv": 0.06263745835658797 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.563708996400237, + "p95": 1.691321206453722, + "min": 1.4892079925630242, + "max": 2.3287080111913383, + "iqr": 0.04225000157020986, + "mean": 1.5805043523580515, + "cv": 0.051275442145427584 + }, + "total_ms": { + "count": 5, + "median": 101.07941497699358, + "p95": 102.03324638714548, + "min": 99.22833493328653, + "max": 102.17112096142955, + "iqr": 0.526745046954602, + "mean": 100.9831244009547, + "cv": 0.010790373251469169 + }, + "tokens_per_second": { + "count": 5, + "median": 633.1655165848247, + "p95": 642.7708098117369, + "min": 626.400096208796, + "max": 644.9770626810241, + "iqr": 3.29052086306433, + "mean": 633.8287502561512, + "cv": 0.010873474295981006 + }, + "median_step_ms_by_index": [ + 1.4136670070001855, + 1.8413750076433644, + 1.5756669890834019, + 1.6596249915892258, + 1.630124999792315, + 1.5691669977968559, + 1.5800410037627444, + 1.540458993986249, + 1.5700829972047359, + 1.5591660048812628, + 1.5630830021109432, + 1.5765000134706497, + 1.6058329929364845, + 1.5652080037398264, + 1.5702499949838966, + 1.5640830097254366, + 1.546166997286491, + 1.5717079950263724, + 1.5803749993210658, + 1.5949999942677096, + 1.5670419961679727, + 1.5357079973910004, + 1.564417005283758, + 1.5750410093460232, + 1.5875420067459345, + 1.5507079951930791, + 1.5667910047341138, + 1.548624990391545, + 1.5759579982841387, + 1.5560829924652353, + 1.549708002130501, + 1.555457987706177, + 1.5739579976070672, + 1.5490000077988952, + 1.5552090044366196, + 1.5472090017283335, + 1.5903330058790743, + 1.6347499913536012, + 1.5542500041192397, + 1.5487079945160076, + 1.545209001051262, + 1.5296670026145875, + 1.5543330082437024, + 1.552584013552405, + 1.5371250046882778, + 1.5732079918961972, + 1.5573340060655028, + 1.5450000064447522, + 1.5446660108864307, + 1.5718749928055331, + 1.5594169963151217, + 1.5511249948758632, + 1.5637920005246997, + 1.5605840017087758, + 1.553333000629209, + 1.575415997649543, + 1.5694589965278283, + 1.556540999445133, + 1.5410000050906092, + 1.5669999993406236, + 1.5540840104222298, + 1.5678330091759562, + 1.533374990685843, + 1.577584000187926 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0014298750029411167, + 0.0023287080111913383, + 0.0015961669996613637, + 0.0016596249915892258, + 0.001531457994133234, + 0.0016776249976828694, + 0.0015800410037627444, + 0.0015367089945357293, + 0.0017993750079767779, + 0.0015316659992095083, + 0.0018041250004898757, + 0.0016127079870784655, + 0.0016168750007636845, + 0.0015734589978819713, + 0.0015702499949838966, + 0.0015814579965081066, + 0.0015773339982843027, + 0.0015901250008028, + 0.0015803749993210658, + 0.0015850409981794655, + 0.0015332919865613803, + 0.0015634160081390291, + 0.0015738750080345199, + 0.0015662500081816688, + 0.001741875006700866, + 0.001563708996400237, + 0.0015667910047341138, + 0.0015734159969724715, + 0.0016757080011302605, + 0.0015560829924652353, + 0.0015599170001223683, + 0.001555457987706177, + 0.0015294160111807287, + 0.0015423750010086223, + 0.001548957996419631, + 0.0015472090017283335, + 0.0015903330058790743, + 0.001655667001614347, + 0.0015655829920433462, + 0.0015454590029548854, + 0.001545209001051262, + 0.0015296670026145875, + 0.0015543330082437024, + 0.0015329159941757098, + 0.0015371250046882778, + 0.0015732079918961972, + 0.0015534169942839071, + 0.0015435829991474748, + 0.001544582992210053, + 0.001556417002575472, + 0.001613959000678733, + 0.001542124999104999, + 0.0015644999948563054, + 0.0015844580047996715, + 0.001519249999546446, + 0.0016790830122772604, + 0.0015694589965278283, + 0.0015515409904764965, + 0.0015818749961908907, + 0.0017985419981414452, + 0.0016269999905489385, + 0.0016107080009533092, + 0.0015322909894166514, + 0.0016080830100690946 + ], + [ + 0.0013944169913884252, + 0.001784374995622784, + 0.0015390000044135377, + 0.0015237499901559204, + 0.001630124999792315, + 0.0015675000031478703, + 0.0015654580056434497, + 0.0015501669986406341, + 0.0015430839994223788, + 0.0015657920012017712, + 0.0015627499960828573, + 0.0015765000134706497, + 0.0015477080014534295, + 0.0015652080037398264, + 0.001570667009218596, + 0.0015640830097254366, + 0.001546166997286491, + 0.0015625410014763474, + 0.0015572920092381537, + 0.0016912090068217367, + 0.0015720840019639581, + 0.001585584002896212, + 0.0015635830059181899, + 0.0015750410093460232, + 0.0015758749941596761, + 0.0015507079951930791, + 0.0016614999913144857, + 0.001548624990391545, + 0.001624916010769084, + 0.001596957998117432, + 0.0015855829988140613, + 0.0015806250012246892, + 0.0015773750055814162, + 0.0016779160068836063, + 0.001578875002451241, + 0.001676624990068376, + 0.0015789999888511375, + 0.001672624988714233, + 0.0015538749867118895, + 0.0015524999907938763, + 0.0015486670017708093, + 0.001516541000455618, + 0.0015332500042859465, + 0.001552584013552405, + 0.0015363749989774078, + 0.001549167005578056, + 0.0015573340060655028, + 0.0015479590074392036, + 0.0016460839979117736, + 0.0015915000112727284, + 0.0015603330102749169, + 0.0015511249948758632, + 0.001578124996740371, + 0.0015502089954679832, + 0.0015841249987715855, + 0.001690292003331706, + 0.0015795420040376484, + 0.0015635829913662747, + 0.0015624590014340356, + 0.0015642910002497956, + 0.001567208004416898, + 0.0015907089982647449, + 0.001558290998218581, + 0.001577584000187926 + ], + [ + 0.0012856670073233545, + 0.0017353330040350556, + 0.0015482500020880252, + 0.001665917006903328, + 0.0015402499993797392, + 0.0015691669977968559, + 0.0015509580116486177, + 0.0015105409984244034, + 0.001571499989950098, + 0.0015426660102093592, + 0.0015343749983003363, + 0.0016303339943988249, + 0.0016058329929364845, + 0.0016562499949941412, + 0.0015798749955138192, + 0.0015624590014340356, + 0.001536375013529323, + 0.0018141250038752332, + 0.001618334004888311, + 0.0015949999942677096, + 0.001766667002812028, + 0.0015312079922296107, + 0.001665584000875242, + 0.0015982080076355487, + 0.0015996659931261092, + 0.001575625006807968, + 0.00158770898997318, + 0.0015696659975219518, + 0.0015642920043319464, + 0.0015705410041846335, + 0.001549708002130501, + 0.0015484579926123843, + 0.0015739579976070672, + 0.0015970419917721301, + 0.0015789580065757036, + 0.0015677500050514936, + 0.0016089579876279458, + 0.0015706249978393316, + 0.0015577919984934852, + 0.0015487079945160076, + 0.0015745000127935782, + 0.0015642090002074838, + 0.001572417007992044, + 0.0015715420013293624, + 0.0015717080095782876, + 0.0015941669989842921, + 0.0016626249998807907, + 0.0015450000064447522, + 0.0016047919925767928, + 0.0015718749928055331, + 0.0015140829927986488, + 0.0016210419998969883, + 0.0015637920005246997, + 0.0015823749999981374, + 0.0015978329902281985, + 0.001575415997649543, + 0.0015469579957425594, + 0.0015685419930377975, + 0.0015352909977082163, + 0.0015886250039329752, + 0.0015447079931618646, + 0.0015376669907709584, + 0.001533374990685843, + 0.0015525410126429051 + ], + [ + 0.0015331250033341348, + 0.002250999998068437, + 0.0016880839975783601, + 0.0015567089867545292, + 0.0016844579949975014, + 0.0015326660068240017, + 0.0015856249956414104, + 0.001556083996547386, + 0.0015635000017937273, + 0.0015591660048812628, + 0.0016819580050650984, + 0.0015608329995302483, + 0.0015582500054733828, + 0.001556375005748123, + 0.0015549160016234964, + 0.0015615829906892031, + 0.0016801669989945367, + 0.0015717079950263724, + 0.0016475420125061646, + 0.001649333004024811, + 0.0015507079951930791, + 0.0015207919932436198, + 0.0015194160077953711, + 0.0016353750106645748, + 0.0015875420067459345, + 0.0015270000003511086, + 0.0015247090050252154, + 0.0015270419971784577, + 0.0015759579982841387, + 0.0015327079890994355, + 0.0015339999954449013, + 0.0016671250050421804, + 0.0015674580063205212, + 0.0015490000077988952, + 0.0015552090044366196, + 0.0015069590008351952, + 0.0017004580004140735, + 0.0015771250036777928, + 0.0015542500041192397, + 0.0015615419979440048, + 0.001542500001960434, + 0.0015518750005867332, + 0.0015607499954057857, + 0.0015673750021960586, + 0.0015418749972013757, + 0.0016286250029224902, + 0.001658499997574836, + 0.001561375000164844, + 0.0015446660108864307, + 0.0015738750080345199, + 0.0015594169963151217, + 0.00157066599058453, + 0.0015622080100001767, + 0.0015605840017087758, + 0.001553333000629209, + 0.0015542500041192397, + 0.0015874579985393211, + 0.001556540999445133, + 0.0015410000050906092, + 0.0015669999993406236, + 0.0015540840104222298, + 0.0015678330091759562, + 0.001557125011458993, + 0.0016513750015292317 + ], + [ + 0.0014136670070001855, + 0.0018413750076433644, + 0.0015756669890834019, + 0.001698874999419786, + 0.001767583002219908, + 0.0016034589934861287, + 0.0017271249962504953, + 0.001540458993986249, + 0.0015700829972047359, + 0.001577332994202152, + 0.0015630830021109432, + 0.001574292007717304, + 0.001691583005595021, + 0.0015529170050285757, + 0.0015475419932045043, + 0.001567499988595955, + 0.0015271669981302693, + 0.0015340840036515146, + 0.0015713750035502017, + 0.001593374996446073, + 0.0015670419961679727, + 0.0015357079973910004, + 0.001564417005283758, + 0.0015447500045411289, + 0.0015702090022386983, + 0.0015295829944079742, + 0.0015231249999487773, + 0.0015378329990198836, + 0.0015680840006098151, + 0.001522709004348144, + 0.001534707989776507, + 0.001527958011138253, + 0.0016081670037237927, + 0.00153012499504257, + 0.001523040991742164, + 0.0015318750083679333, + 0.0015253750025294721, + 0.0016347499913536012, + 0.001538458003778942, + 0.0014892079925630242, + 0.0015020419959910214, + 0.0014894579944666475, + 0.0015003330045146868, + 0.0014905830030329525, + 0.0015001669962657616, + 0.0014939579996280372, + 0.0014936249935999513, + 0.0014929579920135438, + 0.0015194589941529557, + 0.0015217920008581132, + 0.0015229160053422675, + 0.0014892080071149394, + 0.0015221249923342839, + 0.0015058749995660037, + 0.001525208994280547, + 0.0015383749996544793, + 0.0015077499992912635, + 0.0015277079946827143, + 0.0015187499957391992, + 0.0015308329893741757, + 0.001527708998764865, + 0.0015456249966518953, + 0.0014996660029282793, + 0.0015165420045377687 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + ] + } + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 1.2907089985674247, + "p95": 1.3701416028197855, + "min": 1.284415993723087, + "max": 1.382125003146939, + "iqr": 0.03154099977109581, + "mean": 1.3140249997377396, + "cv": 0.031077457382575738 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.4570830098818988, + "p95": 1.6753587042330764, + "min": 1.2485420011216775, + "max": 3.8797089946456254, + "iqr": 0.1606659934623167, + "mean": 1.506061133298476, + "cv": 0.18586712362830396 + }, + "total_ms": { + "count": 5, + "median": 92.46196107415017, + "p95": 106.56887562654447, + "min": 92.3135429766262, + "max": 109.67779303609859, + "iqr": 1.7403270758222789, + "mean": 96.19587639754172, + "cv": 0.0787407092335416 + }, + "tokens_per_second": { + "count": 5, + "median": 692.1765367779188, + "p95": 693.1703283192493, + "min": 583.5274236320153, + "max": 693.2893910940544, + "iqr": 12.806471905325566, + "mean": 668.3150068077441, + "cv": 0.07140932244564056 + }, + "median_step_ms_by_index": [ + 1.2907089985674247, + 1.4021670067450032, + 1.3659579999512061, + 1.341959010460414, + 1.3308329944266006, + 1.3800419983454049, + 1.2974999990547076, + 1.3345409970497712, + 1.4094999933149666, + 1.3325000036275014, + 1.3762090093223378, + 1.381708003464155, + 1.4014589978614822, + 1.3675000082002953, + 1.3181249960325658, + 1.4079169923206791, + 1.3573749893112108, + 1.4107909955782816, + 1.4087080053286627, + 1.4275839930633083, + 1.392999998643063, + 1.4503749989671633, + 1.3949590065749362, + 1.4088329917285591, + 1.4380840002559125, + 1.4439170045079663, + 1.4041670074220747, + 1.4816250040894374, + 1.4423329994315282, + 1.4790839923080057, + 1.4525000005960464, + 1.4648750075139105, + 1.426083006663248, + 1.4728749956702814, + 1.4536249946104363, + 1.4497079973807558, + 1.4857079950161278, + 1.476000004913658, + 1.49049999890849, + 1.4949589967727661, + 1.4696669968543574, + 1.4751660055480897, + 1.4931250043446198, + 1.5195839951047674, + 1.5221249923342839, + 1.491082992288284, + 1.4590420032618567, + 1.472624993766658, + 1.4899999951012433, + 1.5833749930607155, + 1.552624991745688, + 1.5482909948332235, + 1.5519169974140823, + 1.5473329985979944, + 1.5387500025099143, + 1.5803749993210658, + 1.6022499912651256, + 1.5953330002957955, + 1.5454160020453855, + 1.6239160031545907, + 1.601583993760869, + 1.5677910123486072, + 1.5694169997004792, + 1.55929199536331 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0012906670017400756, + 0.0014516660012304783, + 0.0013659579999512061, + 0.001341959010460414, + 0.001441959000658244, + 0.001402125009917654, + 0.001295499998377636, + 0.0013841250038240105, + 0.0013548340066336095, + 0.0013325000036275014, + 0.001371082995319739, + 0.001339124995865859, + 0.0014335839950945228, + 0.0013675000082002953, + 0.0013717080000787973, + 0.0014069579920032993, + 0.0013573749893112108, + 0.0014107909955782816, + 0.0014311249979073182, + 0.0014275839930633083, + 0.0013663749996339902, + 0.0014503749989671633, + 0.0013949590065749362, + 0.0014713329874211922, + 0.0014380840002559125, + 0.0013876250013709068, + 0.0014412499876925722, + 0.001511666996520944, + 0.0014423329994315282, + 0.0015094170084921643, + 0.0015144589997362345, + 0.0014517920062644407, + 0.001426083006663248, + 0.0015151660045376047, + 0.0014536249946104363, + 0.0014497079973807558, + 0.001520750010968186, + 0.0015076249983394518, + 0.0014501670084428042, + 0.0015077919961186126, + 0.001471457988373004, + 0.0014889999874867499, + 0.0015138750022742897, + 0.001530499997898005, + 0.0014635410043410957, + 0.001515041003585793, + 0.0014590420032618567, + 0.001472624993766658, + 0.0015035830001579598, + 0.001626665994990617, + 0.001552624991745688, + 0.001561958997626789, + 0.001618458001757972, + 0.0015473329985979944, + 0.0015131660038605332, + 0.0016080829955171794, + 0.0016642080008750781, + 0.0015765419957460836, + 0.0015454160020453855, + 0.0016239160031545907, + 0.0016342919989256188, + 0.0015677910123486072, + 0.001636417000554502, + 0.001648958001169376 + ], + [ + 0.001382125003146939, + 0.0014335419982671738, + 0.0013449590042000636, + 0.0014254170091589913, + 0.001281750010093674, + 0.0013196250074543059, + 0.001310167004703544, + 0.0013864999928046018, + 0.0013276249956106767, + 0.0014739170001121238, + 0.00137779199576471, + 0.001332625004579313, + 0.001385750001645647, + 0.0013519580097636208, + 0.0013181249960325658, + 0.0014079169923206791, + 0.0013157909997971728, + 0.0038797089946456254, + 0.003318457995192148, + 0.003151249999064021, + 0.0035450839932309464, + 0.0025345839967485517, + 0.0029384160006884485, + 0.0028894999995827675, + 0.00167258401052095, + 0.0015612919960403815, + 0.0015167090023169294, + 0.0014816250040894374, + 0.0015437080000992864, + 0.0015816660015843809, + 0.0015251669974531978, + 0.001589125007740222, + 0.0017085420113289729, + 0.0016479159967275336, + 0.0017052079929271713, + 0.001520624995464459, + 0.0015088750078575686, + 0.001596916001290083, + 0.001760540995746851, + 0.0016901250055525452, + 0.0017188750061905012, + 0.0016316670080414042, + 0.001692958001513034, + 0.0016818329895613715, + 0.0015753340121591464, + 0.0016066669923020527, + 0.0016358330030925572, + 0.0016953750018728897, + 0.0016576670022914186, + 0.0015833749930607155, + 0.0015451250073965639, + 0.0015379589895019308, + 0.0015802910056663677, + 0.0015732079918961972, + 0.001576541006215848, + 0.0016085419920273125, + 0.0016477919998578727, + 0.001523291997727938, + 0.0015505830087931827, + 0.00163254099607002, + 0.001557750001666136, + 0.0016201249964069575, + 0.0016178749938262627, + 0.0015833750076126307 + ], + [ + 0.001284415993723087, + 0.0013216670049587265, + 0.001399625005433336, + 0.0013094159949105233, + 0.0013308329944266006, + 0.0013872499985154718, + 0.0013384589983616024, + 0.0013345409970497712, + 0.0014473330084001645, + 0.0012836660025641322, + 0.001418290994479321, + 0.001425209004082717, + 0.0013386250066105276, + 0.0014121670101303607, + 0.001294374989811331, + 0.0015513329999521375, + 0.0014354169979924336, + 0.0014473749906755984, + 0.0013498330081347376, + 0.0014831669977866113, + 0.001392999998643063, + 0.0014520830009132624, + 0.0013848749949829653, + 0.0014088329917285591, + 0.0013314169918885455, + 0.0014681250031571835, + 0.0013730409991694614, + 0.0014538340037688613, + 0.0014042909897398204, + 0.0014570830098818988, + 0.0014525000005960464, + 0.0014648750075139105, + 0.0014217089919839054, + 0.0014707090012961999, + 0.0014340829948196188, + 0.00142920900543686, + 0.0014319589972728863, + 0.0014232090034056455, + 0.0015105410129763186, + 0.0014949589967727661, + 0.0014418750070035458, + 0.0014648329961346462, + 0.0014931250043446198, + 0.0014197499986039475, + 0.0015221249923342839, + 0.001491082992288284, + 0.0014395420002983883, + 0.0013914589944761246, + 0.0014178330020513386, + 0.0015439169947057962, + 0.001471708994358778, + 0.0015482909948332235, + 0.001527958011138253, + 0.001462500003981404, + 0.0015387500025099143, + 0.001556041999720037, + 0.001503457999206148, + 0.0015953330002957955, + 0.001532458991277963, + 0.0015550419921055436, + 0.001574499998241663, + 0.0015180000045802444, + 0.0015432920044986531, + 0.0015073340036906302 + ], + [ + 0.0013222080015111715, + 0.0014021670067450032, + 0.00134170800447464, + 0.0013553750031860545, + 0.0013680830015800893, + 0.0013800419983454049, + 0.0012974999990547076, + 0.0013254999939817935, + 0.0014342079957714304, + 0.0013148750003892928, + 0.0013659999967785552, + 0.001381708003464155, + 0.0014132910000625998, + 0.0014447919966187328, + 0.0013145840057404712, + 0.0013823750050505623, + 0.0013995000044815242, + 0.0013547499984269962, + 0.0014087080053286627, + 0.001404000009642914, + 0.0013891249982407317, + 0.0014112499920884147, + 0.0013812919933116063, + 0.0014074589998926967, + 0.0014803329977439716, + 0.0014221660094335675, + 0.0014041670074220747, + 0.0014986250025685877, + 0.0014820829965174198, + 0.0014790839923080057, + 0.0014027500001247972, + 0.001383458002237603, + 0.001323082993621938, + 0.001359625006443821, + 0.001351291997707449, + 0.0013503339869203046, + 0.0013614159979624674, + 0.0013553749886341393, + 0.0013636249932460487, + 0.0013617090007755905, + 0.0013684580044355243, + 0.0013795829872833565, + 0.0014232919929781929, + 0.0015195839951047674, + 0.0015347089938586578, + 0.0014549579937011003, + 0.0014259169984143227, + 0.001634333995752968, + 0.0014899999951012433, + 0.0016312079969793558, + 0.001658458000747487, + 0.00157816699356772, + 0.0015519169974140823, + 0.0014946669980417937, + 0.0015316249919123948, + 0.0015803749993210658, + 0.0016022499912651256, + 0.0016343330062227324, + 0.0014852500025881454, + 0.0016574170003877953, + 0.0016417089937021956, + 0.001632999992580153, + 0.0015612090064678341, + 0.001540833996841684 + ], + [ + 0.0012907089985674247, + 0.0012963330082129687, + 0.0013712090003537014, + 0.0013007079978706315, + 0.0012652919976972044, + 0.0012524170015240088, + 0.0012485420011216775, + 0.00129854200349655, + 0.0014094999933149666, + 0.001375292005832307, + 0.0013762090093223378, + 0.0014245420024963096, + 0.0014014589978614822, + 0.00132170801225584, + 0.0013812909892294556, + 0.0014264159981394187, + 0.0012929999938933179, + 0.0013952080043964088, + 0.0013853750133421272, + 0.0013870420079911128, + 0.0014162910083541647, + 0.001349792000837624, + 0.0014131250063655898, + 0.0013862920022802427, + 0.0013538330094888806, + 0.0014439170045079663, + 0.001312042004428804, + 0.0014273749984567985, + 0.0014219580043572932, + 0.001453416989534162, + 0.0013687919999938458, + 0.0015004999877419323, + 0.001447292001103051, + 0.0014728749956702814, + 0.0015182080096565187, + 0.0015417500108014792, + 0.0014857079950161278, + 0.001476000004913658, + 0.00149049999890849, + 0.0014278750022640452, + 0.0014696669968543574, + 0.0014751660055480897, + 0.0014433750038733706, + 0.0015002919972175732, + 0.0014432499883696437, + 0.0014199160068528727, + 0.0014634170074714348, + 0.0014687500079162419, + 0.0014688330120407045, + 0.001523457991424948, + 0.0017230829980690032, + 0.0015109580126591027, + 0.0015060829900903627, + 0.0016007499943953007, + 0.00162041699513793, + 0.0015748340083518997, + 0.0015362499980255961, + 0.0016040829941630363, + 0.001634125001146458, + 0.0015866670000832528, + 0.001601583993760869, + 0.0015499580040341243, + 0.0015694169997004792, + 0.00155929199536331 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_mha": { + "first_token_ms": { + "count": 5, + "median": 1.5260000072885305, + "p95": 1.5744745964184403, + "min": 1.4866669953335077, + "max": 1.5810409968253225, + "iqr": 0.02879199746530503, + "mean": 1.5322667983127758, + "cv": 0.022881118207821186 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.5612090064678341, + "p95": 1.7195463020470925, + "min": 1.3576659985119477, + "max": 1.8402919959044084, + "iqr": 0.15168700338108465, + "mean": 1.571210056990533, + "cv": 0.05825478565233986 + }, + "total_ms": { + "count": 5, + "median": 100.85695605084766, + "p95": 101.4549345447449, + "min": 99.05441400769632, + "max": 101.47912494721822, + "iqr": 1.5143389318836853, + "mean": 100.51850038871635, + "cv": 0.010363946690406203 + }, + "tokens_per_second": { + "count": 5, + "median": 634.5620818432593, + "p95": 645.0878220299078, + "min": 630.6715793350403, + "max": 646.1095211267145, + "iqr": 9.576857794506623, + "mean": 636.7536751591738, + "cv": 0.010410497577260026 + }, + "median_step_ms_by_index": [ + 1.5260000072885305, + 1.4584169985027984, + 1.4472079928964376, + 1.4942080015316606, + 1.5315839991671965, + 1.4574579981854185, + 1.4894589985487983, + 1.4985829911893234, + 1.4374580059666187, + 1.4335000014398247, + 1.4971250056987628, + 1.4561250136466697, + 1.4927500014891848, + 1.4816249895375222, + 1.4691669930471107, + 1.5088749933056533, + 1.4900829992257059, + 1.5001249994384125, + 1.516042000730522, + 1.5506250056205317, + 1.4795000024605542, + 1.5286670095520094, + 1.5041250007925555, + 1.5005839959485456, + 1.4934999926481396, + 1.5237079933285713, + 1.539832999696955, + 1.5477499982807785, + 1.528417007648386, + 1.601165989995934, + 1.575832997332327, + 1.5155409928411245, + 1.580416996148415, + 1.5801250119693577, + 1.5833750076126307, + 1.6431669937446713, + 1.6272499924525619, + 1.5935830015223473, + 1.6494579904247075, + 1.6202090046135709, + 1.640750007936731, + 1.5982920012902468, + 1.6545840044273064, + 1.5912499948171899, + 1.6733750089770183, + 1.6001250041881576, + 1.6536249895580113, + 1.6056669992394745, + 1.597540991497226, + 1.6524999955436215, + 1.6976659971987829, + 1.6625830030534416, + 1.6603749973000959, + 1.6612920007901266, + 1.6867080121301115, + 1.6502079961355776, + 1.618124995729886, + 1.7455829947721213, + 1.6682080022292212, + 1.6696250095264986, + 1.652375009143725, + 1.7319590115221217, + 1.6776670090621337, + 1.6613750049145892 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0014866669953335077, + 0.001456458994653076, + 0.0014380409993464127, + 0.0015050420042825863, + 0.00147050000668969, + 0.0014834589965175837, + 0.0014894589985487983, + 0.0014265840000007302, + 0.0014374580059666187, + 0.0015563749911962077, + 0.0015077499992912635, + 0.0015146250079851598, + 0.0014216249983292073, + 0.001435874990420416, + 0.0015296250057872385, + 0.0015237499901559204, + 0.0014598749985452741, + 0.00154899999324698, + 0.001548416999867186, + 0.0015506250056205317, + 0.0014685409987578169, + 0.0015289169969037175, + 0.0014919160021236166, + 0.0014902499970048666, + 0.0015179589972831309, + 0.0015622919891029596, + 0.0014980420091887936, + 0.0015612919960403815, + 0.001528417007648386, + 0.0016033329884521663, + 0.0015000420098658651, + 0.0015742919931653887, + 0.0016028329991968349, + 0.0015801250119693577, + 0.0016748750058468431, + 0.001650541991693899, + 0.001658292007050477, + 0.0015959169977577403, + 0.0016679169930284843, + 0.0016631250036880374, + 0.001660083988099359, + 0.0015982920012902468, + 0.0016590419982094318, + 0.0016554169997107238, + 0.0016809589869808406, + 0.0016001250041881576, + 0.0016536249895580113, + 0.001652042003115639, + 0.001662541995756328, + 0.0016524999955436215, + 0.0016515000024810433, + 0.0017102919955505058, + 0.0017256669962080196, + 0.0016559170035179704, + 0.0016370829980587587, + 0.0017022080137394369, + 0.0017550829943502322, + 0.0017579580016899854, + 0.0016632499900879338, + 0.0016641659894958138, + 0.001652375009143725, + 0.0017769999976735562, + 0.0016850829997565597, + 0.0016358330030925572 + ], + [ + 0.0015482089947909117, + 0.0014387080009328201, + 0.001507832988863811, + 0.0015326249995268881, + 0.001719459003652446, + 0.0014574579981854185, + 0.0014390419964911416, + 0.001560999997309409, + 0.0014394589961739257, + 0.0013871250121155754, + 0.0014907500008121133, + 0.001454124998417683, + 0.0015211249992717057, + 0.001507082997704856, + 0.0014552079956047237, + 0.0015088749933056533, + 0.0014900829992257059, + 0.0015001249994384125, + 0.0015232910081977025, + 0.0015719170041847974, + 0.0014795000024605542, + 0.0015310839953599498, + 0.0015635000017937273, + 0.0015005839959485456, + 0.0014934999926481396, + 0.0015405409940285608, + 0.0015552499971818179, + 0.0015477499982807785, + 0.0015776660002302378, + 0.001601165989995934, + 0.001575832997332327, + 0.001496042008511722, + 0.001580416996148415, + 0.0015597919991705567, + 0.001597333000972867, + 0.0016431669937446713, + 0.0016650409961584955, + 0.001580208001541905, + 0.0016494579904247075, + 0.0016424170025857165, + 0.001640750007936731, + 0.001641958995605819, + 0.0016545840044273064, + 0.0015912499948171899, + 0.0016733750089770183, + 0.0016683339927112684, + 0.0016210419998969883, + 0.0016709999908925965, + 0.0016681669949321076, + 0.0016599170048721135, + 0.0017054999916581437, + 0.001651583006605506, + 0.0017197499983012676, + 0.0017457909998483956, + 0.0016728750051697716, + 0.0016343340103048831, + 0.0017120830016210675, + 0.0017455829947721213, + 0.0017132079956354573, + 0.0016696250095264986, + 0.0017254160047741607, + 0.0016504169907420874, + 0.0016029999969759956, + 0.0016058330074883997 + ], + [ + 0.0015260000072885305, + 0.0014903330011293292, + 0.0013576659985119477, + 0.0014622500020777807, + 0.0014612079976359382, + 0.0013920000055804849, + 0.0014234169939300045, + 0.0014985829911893234, + 0.0013964579993626103, + 0.0014989159972174093, + 0.001444207999156788, + 0.0014533339999616146, + 0.0014927500014891848, + 0.0014449579903157428, + 0.0014691669930471107, + 0.0014319580077426508, + 0.0014826670085312799, + 0.0014567920006811619, + 0.0015025839966256171, + 0.0015199579938780516, + 0.001437209008145146, + 0.0015251250006258488, + 0.0015041250007925555, + 0.0014535000082105398, + 0.0014831250009592623, + 0.0014970000047469512, + 0.0014996669924585149, + 0.001446541995392181, + 0.001423292007530108, + 0.0014322909992188215, + 0.0015426250029122457, + 0.0015155409928411245, + 0.0015357920055976138, + 0.0015715000045020133, + 0.0015833750076126307, + 0.0015465830074390396, + 0.001483625004766509, + 0.0015935830015223473, + 0.001552916000946425, + 0.0015255000034812838, + 0.0015826249873498455, + 0.0015384589933091775, + 0.0017222499882336706, + 0.0015880420105531812, + 0.0018078330031130463, + 0.001592041997355409, + 0.0016895840090001002, + 0.0015966659993864596, + 0.001597540991497226, + 0.0015791659971000627, + 0.0016976659971987829, + 0.0016625830030534416, + 0.0016438750026281923, + 0.0016612920007901266, + 0.001687958007096313, + 0.0016707909962860867, + 0.001618124995729886, + 0.0016035419976105914, + 0.0017084159917430952, + 0.0016839169984450564, + 0.0015689169958932325, + 0.0017319590115221217, + 0.0016997089987853542, + 0.0017633329989621416 + ], + [ + 0.0015810409968253225, + 0.0014584169985027984, + 0.001471333991503343, + 0.0014942080015316606, + 0.0015315839991671965, + 0.0014309160033008084, + 0.0014979169936850667, + 0.0015258750063367188, + 0.0014292499981820583, + 0.0014335000014398247, + 0.0015244579990394413, + 0.0014561250136466697, + 0.0015036249969853088, + 0.0015199589979602024, + 0.0014669160009361804, + 0.0015214160084724426, + 0.0015447499899892136, + 0.0014724579959874973, + 0.0014915840001776814, + 0.0015994999994290993, + 0.0015064170002005994, + 0.0014733340067323297, + 0.0015389580075861886, + 0.001558666001074016, + 0.0014609170029871166, + 0.0015237079933285713, + 0.001539832999696955, + 0.001521750004030764, + 0.0015333750052377582, + 0.0016347499913536012, + 0.0015907080087345093, + 0.0014979999978095293, + 0.0015719999937573448, + 0.001647333992877975, + 0.0015538750012638047, + 0.0016248750034719706, + 0.0016272499924525619, + 0.0015228330012178048, + 0.0015977920120349154, + 0.0015544589987257496, + 0.0015480000001844019, + 0.0015858749975450337, + 0.001579250005306676, + 0.0014467500004684553, + 0.0015915000112727284, + 0.0015875419921940193, + 0.0016432920092483982, + 0.0016056669992394745, + 0.0015121249889489263, + 0.0016878750029718503, + 0.0017178749985760078, + 0.0016637499938951805, + 0.0016603749973000959, + 0.0016603330004727468, + 0.0017010000010486692, + 0.0016077500040410087, + 0.0015612090064678341, + 0.0015435829991474748, + 0.0015294579934561625, + 0.0018402919959044084, + 0.0015801249974174425, + 0.00155329100380186, + 0.0016538749914616346, + 0.0017473750049248338 + ], + [ + 0.0015194169973256066, + 0.0015334579948103055, + 0.0014472079928964376, + 0.0013899170007789508, + 0.001589416991919279, + 0.0014870830054860562, + 0.0015475830005016178, + 0.001491667004302144, + 0.0015254999889293686, + 0.0014334160077851266, + 0.0014971250056987628, + 0.00154208300227765, + 0.001433416997315362, + 0.0014816249895375222, + 0.001503457999206148, + 0.0014047500008018687, + 0.0014924580027582124, + 0.0015373329952126369, + 0.001516042000730522, + 0.001506416010670364, + 0.0015575419965898618, + 0.0015286670095520094, + 0.0014709159877384081, + 0.0015917080017970875, + 0.0015010000061010942, + 0.0014873750042170286, + 0.0015492080128751695, + 0.0015824169968254864, + 0.00150154200673569, + 0.0015676659968448803, + 0.0015783749986439943, + 0.0015417500108014792, + 0.0016143329994520172, + 0.0016442500054836273, + 0.0015567089867545292, + 0.0016456669982289895, + 0.0015737920039100572, + 0.0016862079937709495, + 0.0016924579977057874, + 0.0016202090046135709, + 0.0016551250009797513, + 0.001655459011089988, + 0.0015833329962333664, + 0.001668957993388176, + 0.001524042003438808, + 0.001611250001587905, + 0.001670500001637265, + 0.0015474170068046078, + 0.0015116249996935949, + 0.0016249580075964332, + 0.0015911249938653782, + 0.0016561670054215938, + 0.0016179160011233762, + 0.0017157080001197755, + 0.0016867080121301115, + 0.0016502079961355776, + 0.001617917005205527, + 0.0017701670003589243, + 0.0016682080022292212, + 0.0016615409986115992, + 0.0017010000010486692, + 0.0017574169905856252, + 0.0016776670090621337, + 0.0016613750049145892 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "speedup_cached_vs_uncached_sdpa": 0.9147457085618788, + "speedup_cached_vs_uncached_mha": 0.9977991668610612 + }, + { + "batch": 1, + "prompt_len": 128, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": false, + "uncached_mha": false + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 1.714124999125488, + "p95": 2.2938494046684355, + "min": 1.6200829995796084, + "max": 2.3980410041986033, + "iqr": 0.20474901248235255, + "mean": 1.856333200703375, + "cv": 0.17115185086177095 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.5836670063436031, + "p95": 1.8981578992679717, + "min": 1.5116249996935949, + "max": 2.90175000554882, + "iqr": 0.05829149449709803, + "mean": 1.6292617684008466, + "cv": 0.09633001428866517 + }, + "total_ms": { + "count": 5, + "median": 102.475583073101, + "p95": 110.55003742512781, + "min": 100.98158501205035, + "max": 112.03787005797494, + "iqr": 2.1933288808213547, + "mean": 104.4998246099567, + "cv": 0.04217437724145239 + }, + "tokens_per_second": { + "count": 5, + "median": 624.539017790663, + "p95": 632.016572268347, + "min": 571.2354221557645, + "max": 633.7789211009388, + "iqr": 13.104928344248037, + "mean": 613.2765573158156, + "cv": 0.040382253311325354 + }, + "median_step_ms_by_index": [ + 1.714124999125488, + 1.927624995005317, + 1.6875410074135289, + 1.6016250010579824, + 1.6094169986899942, + 1.6530419961782172, + 1.5616669988958165, + 1.6204170096898451, + 1.5894999960437417, + 1.5825840091565624, + 1.5615419979440048, + 1.6292499931296334, + 1.6007499943953007, + 1.598125003511086, + 1.5785829891683534, + 1.639790993067436, + 1.5875410026637837, + 1.6121670050779358, + 1.5865419991314411, + 1.5692080050939694, + 1.624166005058214, + 1.6626669967081398, + 1.6044160001911223, + 1.5809589967830107, + 1.5801249974174425, + 1.6129159921547398, + 1.5794589999131858, + 1.5864170127315447, + 1.57125000259839, + 1.5776250074850395, + 1.575292000779882, + 1.5737500070827082, + 1.6065420059021562, + 1.5836670063436031, + 1.58229201042559, + 1.5748339937999845, + 1.5705830010119826, + 1.5704170073149726, + 1.676041996688582, + 1.570874999742955, + 1.5692919987486675, + 1.580416996148415, + 1.5782919945195317, + 1.5862500004004687, + 1.5757079963805154, + 1.574124995386228, + 1.5642500075045973, + 1.6799169970909134, + 1.5754170017316937, + 1.5732500032754615, + 1.5696250047767535, + 1.555291994009167, + 1.5700419899076223, + 1.5736670029582456, + 1.5858749975450337, + 1.558874995680526, + 1.619583010324277, + 1.5849170013098046, + 1.6096670005936176, + 1.5759999951114878, + 1.582917000632733, + 1.6033340070862323, + 1.577291011926718, + 1.6325420001521707 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.001877083006547764, + 0.0020181250001769513, + 0.0017200409929500893, + 0.0016202499973587692, + 0.0017435839981772006, + 0.0016530419961782172, + 0.0015581670013489202, + 0.0016204170096898451, + 0.0015730419981991872, + 0.0015734579937998205, + 0.0016429999959655106, + 0.0016292499931296334, + 0.0015865830064285547, + 0.001598125003511086, + 0.0015785829891683534, + 0.0018654579907888547, + 0.0016204580024350435, + 0.001617583999177441, + 0.0015865419991314411, + 0.0015695829933974892, + 0.0015364160062745214, + 0.0021187919919611886, + 0.0016044160001911223, + 0.001576166003360413, + 0.0015730420127511024, + 0.0016277920076390728, + 0.0015659580094506964, + 0.0016517079930054024, + 0.0015806250012246892, + 0.0016214999923249707, + 0.001680375004070811, + 0.001577707997057587, + 0.001567250001244247, + 0.0017155829991679639, + 0.0016155829944182187, + 0.0015817089879419655, + 0.0015947919891914353, + 0.0017067500011762604, + 0.0016837919974932447, + 0.001570874999742955, + 0.0015749159938422963, + 0.0016099169879453257, + 0.0015782919945195317, + 0.0015862500004004687, + 0.0015746669960208237, + 0.001574124995386228, + 0.0015728329890407622, + 0.0016799169970909134, + 0.0015796669904375449, + 0.001586416008649394, + 0.0016122090019052848, + 0.0015531249955529347, + 0.0016176670033019036, + 0.0015601250051986426, + 0.0016026249941205606, + 0.0015547919902019203, + 0.0016954160091700032, + 0.0016088330012280494, + 0.0016096670005936176, + 0.0015574159915558994, + 0.0015211669960990548, + 0.0016094169986899942, + 0.0018618749891174957, + 0.0017141660064226016 + ], + [ + 0.0016200829995796084, + 0.00192229199456051, + 0.0016781250014901161, + 0.0016016250010579824, + 0.0015347500011557713, + 0.0018353750056121498, + 0.0015572909906040877, + 0.0016241669945884496, + 0.0016105830000014976, + 0.001577749993884936, + 0.0015615419979440048, + 0.0016777079872554168, + 0.0016007499943953007, + 0.001576000009663403, + 0.001542500001960434, + 0.001530707988422364, + 0.0015822499990463257, + 0.001571667002281174, + 0.0015666249964851886, + 0.0015212080033961684, + 0.001624166005058214, + 0.0015330000023823231, + 0.0015327080036513507, + 0.0016529589920537546, + 0.001555041002575308, + 0.0016226669977186248, + 0.0015794589999131858, + 0.001567208004416898, + 0.001538917000289075, + 0.001560249991598539, + 0.001546458006487228, + 0.001556750008603558, + 0.0015367079904535785, + 0.0015510840021306649, + 0.0015849170013098046, + 0.0015325840067816898, + 0.001517750002676621, + 0.0015116249996935949, + 0.0016069170087575912, + 0.0015309580048779026, + 0.0015555420104647055, + 0.0015165000077104196, + 0.0015862500004004687, + 0.0015761670074425638, + 0.0015953749971231446, + 0.0015542500041192397, + 0.0015570420073345304, + 0.0015536670107394457, + 0.0015615419979440048, + 0.0015585829969495535, + 0.001556999995955266, + 0.001555291994009167, + 0.001569208994624205, + 0.001545791994431056, + 0.0015198750043055043, + 0.0015501660091103986, + 0.0015619579935446382, + 0.0015283329994417727, + 0.0015709580038674176, + 0.0015450840000994503, + 0.0015901669976301491, + 0.0015782499976921827, + 0.0016057500033639371, + 0.0015537079889327288 + ], + [ + 0.001714124999125488, + 0.0017194169922731817, + 0.0015960830060066655, + 0.001589125007740222, + 0.001575457994476892, + 0.0015764169947942719, + 0.0015616669988958165, + 0.0015422500000568107, + 0.0015894999960437417, + 0.0015825840091565624, + 0.0015585000073770061, + 0.00157200000830926, + 0.0016662919952068478, + 0.0015507920033996925, + 0.001567625004099682, + 0.0015495830011786893, + 0.0015598330064676702, + 0.0016121670050779358, + 0.0015950000088196248, + 0.0015686249971622601, + 0.0016688330069882795, + 0.0016626669967081398, + 0.0016812919930089265, + 0.0015582919877488166, + 0.0015921249869279563, + 0.0015882500010775402, + 0.0015655000024707988, + 0.0016722080035833642, + 0.00157125000259839, + 0.0015776250074850395, + 0.001564999998663552, + 0.0015737500070827082, + 0.0017401670047547668, + 0.001581374992383644, + 0.0015755419881315902, + 0.0015748339937999845, + 0.0015705830010119826, + 0.0015568329981761053, + 0.0016797910066088662, + 0.00158862498938106, + 0.0016120410000439733, + 0.0015845000016270205, + 0.0015660829958505929, + 0.0015421670104842633, + 0.0016246659943135455, + 0.0015879169950494543, + 0.0015642500075045973, + 0.0016814580012578517, + 0.0016287090111291036, + 0.0015907499910099432, + 0.0016587499994784594, + 0.0015775829961057752, + 0.0015784589922986925, + 0.0015736670029582456, + 0.001588999992236495, + 0.001558874995680526, + 0.0015822499990463257, + 0.0015849170013098046, + 0.0015880830032983795, + 0.0016637500084470958, + 0.0016064170049503446, + 0.0016033340070862323, + 0.001575457994476892, + 0.00169070900301449 + ], + [ + 0.0023980410041986033, + 0.00290175000554882, + 0.002515333006158471, + 0.002446375001454726, + 0.0024407500022789463, + 0.002502417002688162, + 0.0023121660051401705, + 0.001974166007130407, + 0.0019137909985147417, + 0.001950040998053737, + 0.0019484999938867986, + 0.0017755409935489297, + 0.0017065409920178354, + 0.0022305829916149378, + 0.002071958006126806, + 0.0018914579995907843, + 0.00166245800210163, + 0.0016229169996222481, + 0.0016489170084241778, + 0.0015907920023892075, + 0.0016861250041984022, + 0.001632957995752804, + 0.0016920829948503524, + 0.0016488750115968287, + 0.0015907909983070567, + 0.0016129159921547398, + 0.0015964580088621005, + 0.0015864170127315447, + 0.0015795829967828467, + 0.0016592920001130551, + 0.0016057920001912862, + 0.0015916660049697384, + 0.001757917008944787, + 0.001624416996492073, + 0.0015802920097485185, + 0.0015936670097289607, + 0.0015667910047341138, + 0.001570916996570304, + 0.001676041996688582, + 0.0015664170059608296, + 0.001562874997034669, + 0.0015620000049239025, + 0.0015539580053882673, + 0.0015883750020293519, + 0.0015415410016430542, + 0.001585125006386079, + 0.0015693339955760166, + 0.0016767919878475368, + 0.0015739579976070672, + 0.0015634169976692647, + 0.0015696250047767535, + 0.001552000001538545, + 0.0015700419899076223, + 0.001656999986153096, + 0.001554500006022863, + 0.0015601250051986426, + 0.0016748329944675788, + 0.0015343749983003363, + 0.0017011250020004809, + 0.0015865419991314411, + 0.001578334005898796, + 0.0016721249994589016, + 0.0015746249991934747, + 0.0015513329999521375 + ], + [ + 0.0016723339940654114, + 0.001927624995005317, + 0.0016875410074135289, + 0.0015842499997233972, + 0.0016094169986899942, + 0.0015774999919813126, + 0.0015745830023661256, + 0.001586833008332178, + 0.0015287499991245568, + 0.0015984579949872568, + 0.0015462500014109537, + 0.0015757920045871288, + 0.001560375007102266, + 0.0016412080003647134, + 0.0016589159931754693, + 0.001639790993067436, + 0.0015875410026637837, + 0.0016068339900812134, + 0.0015844580047996715, + 0.0015692080050939694, + 0.001545749997603707, + 0.0016936669999267906, + 0.0015922500024316832, + 0.0015809589967830107, + 0.0015801249974174425, + 0.0015930000081425533, + 0.001622334006242454, + 0.00157200000830926, + 0.0015635830059181899, + 0.001558290998218581, + 0.001575292000779882, + 0.001522334001492709, + 0.0016065420059021562, + 0.0015836670063436031, + 0.00158229201042559, + 0.0015623330109519884, + 0.0015835829981369898, + 0.0015704170073149726, + 0.0016751249931985512, + 0.0015874170057941228, + 0.0015692919987486675, + 0.001580416996148415, + 0.001684165996266529, + 0.001591416003066115, + 0.0015757079963805154, + 0.001556291987071745, + 0.001550125001813285, + 0.001687084004515782, + 0.0015754170017316937, + 0.0015732500032754615, + 0.0015631250134902075, + 0.0016757499979576096, + 0.0015563329943688586, + 0.0016703330038581043, + 0.0015858749975450337, + 0.001568792009493336, + 0.001619583010324277, + 0.0016136250051204115, + 0.0016651249898131937, + 0.0015759999951114878, + 0.001582917000632733, + 0.0015764999989187345, + 0.001577291011926718, + 0.0016325420001521707 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + ] + } + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 1.7571249918546528, + "p95": 1.7820922017563134, + "min": 1.708416995825246, + "max": 1.7825840041041374, + "iqr": 0.04629198519978672, + "mean": 1.752416798262857, + "cv": 0.018003032529698773 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.9230410107411444, + "p95": 2.059340992127545, + "min": 1.697542000329122, + "max": 2.3065419954946265, + "iqr": 0.1254170056199655, + "mean": 1.9200969553951706, + "cv": 0.04735931745666446 + }, + "total_ms": { + "count": 5, + "median": 123.02028796693776, + "p95": 123.28987416403834, + "min": 121.513128047809, + "max": 123.34337396896444, + "iqr": 0.43591493158601224, + "mean": 122.7185249881586, + "cv": 0.005859653456507307 + }, + "tokens_per_second": { + "count": 5, + "median": 520.2393934990648, + "p95": 525.7241991470152, + "min": 518.8766768785134, + "max": 526.6920622339619, + "iqr": 1.8483184013264236, + "mean": 521.533061561734, + "cv": 0.005893089814685971 + }, + "median_step_ms_by_index": [ + 1.7571249918546528, + 1.8801249971147627, + 1.8900830036727712, + 1.8157500016968697, + 1.8714170000748709, + 1.776833989424631, + 1.915792003273964, + 1.7514999926788732, + 1.8209169938927516, + 1.8185420049121603, + 1.840167009504512, + 1.7878330108942464, + 1.8048329948214814, + 1.8430840136716142, + 1.8179159960709512, + 1.8279579962836578, + 1.8345830030739307, + 1.8462089938111603, + 1.8607079982757568, + 1.8981250032084063, + 1.9101249927189201, + 1.9264169968664646, + 1.8584580102469772, + 1.8225840030936524, + 1.886708996607922, + 1.9114579918095842, + 1.8405419978080317, + 1.8676250037970021, + 1.9158750073984265, + 1.8781660037348047, + 1.8935420084744692, + 1.885500008938834, + 1.934874992002733, + 1.9533750019036233, + 1.890832994831726, + 1.9653750059660524, + 1.9191250030416995, + 1.916708002681844, + 1.9587919960031286, + 1.9378330034669489, + 1.9613330077845603, + 1.9802500028163195, + 1.9299580017104745, + 1.9728750048670918, + 1.9958329939981923, + 1.978667001822032, + 1.960125009645708, + 1.9940420024795458, + 1.9893749995389953, + 1.978291998966597, + 1.9942919898312539, + 1.9946249958593398, + 1.9620840030256659, + 1.9716669921763241, + 1.971207995666191, + 1.9925000087823719, + 2.019417006522417, + 1.9608750008046627, + 1.9694170041475445, + 1.9915830052923411, + 1.9948749977629632, + 1.9825000053970143, + 1.9818340078927577, + 2.0227919885655865 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0017338330071652308, + 0.0018424580048304051, + 0.001854791000369005, + 0.001782499995897524, + 0.0017770410049706697, + 0.001776833989424631, + 0.0018905830074800178, + 0.0017869170114863664, + 0.0018209169938927516, + 0.0018185420049121603, + 0.0017795000021578744, + 0.0017759589973138645, + 0.0017951250047190115, + 0.0018430840136716142, + 0.0018179159960709512, + 0.0017655419942457229, + 0.0017770830017980188, + 0.0018700420041568577, + 0.0019063340005232021, + 0.0018981250032084063, + 0.0018532919930294156, + 0.001851249995524995, + 0.0018251670117024332, + 0.0018225840030936524, + 0.0018339579983148724, + 0.0018433750083204359, + 0.0017942919948836789, + 0.0018507920030970126, + 0.0018551670073065907, + 0.0017990000051213428, + 0.0017080830002669245, + 0.002119999990100041, + 0.0018560829921625555, + 0.001909541999339126, + 0.0018844170117517933, + 0.0019777499983320013, + 0.0019191250030416995, + 0.001802916987799108, + 0.0023065419954946265, + 0.0019762500014621764, + 0.002000209002289921, + 0.0019904170039808378, + 0.002005875008762814, + 0.0019728750048670918, + 0.001957542001036927, + 0.001960125009645708, + 0.001960125009645708, + 0.0019375409901840612, + 0.0019172920001437888, + 0.0019365829939488322, + 0.0019425409991526976, + 0.0019532500009518117, + 0.0020761249907081947, + 0.0019402500038268045, + 0.001971207995666191, + 0.0019711249915417284, + 0.001987790994462557, + 0.0019608750008046627, + 0.0019371670059626922, + 0.002009082993026823, + 0.0019574170000851154, + 0.0019825000053970143, + 0.0020648749923566356, + 0.00201762501092162 + ], + [ + 0.0017571249918546528, + 0.0018094579863827676, + 0.001820541001507081, + 0.0017321250052191317, + 0.0020137920073466375, + 0.0018021660071099177, + 0.0020073339983355254, + 0.0017514999926788732, + 0.0017688749940134585, + 0.0018046659970423207, + 0.001827208005124703, + 0.0018277089984621853, + 0.0018048329948214814, + 0.0019229579920647666, + 0.0018742919928627089, + 0.0018279579962836578, + 0.0018345830030739307, + 0.0018462089938111603, + 0.0019856250000884756, + 0.001924291005707346, + 0.001875624991953373, + 0.0019264169968664646, + 0.0018584580102469772, + 0.0017954999930225313, + 0.0019118750060442835, + 0.0019200829992769286, + 0.001938291999977082, + 0.0018507500062696636, + 0.0019345410109963268, + 0.0018781660037348047, + 0.0019250409968663007, + 0.0018151249969378114, + 0.001934874992002733, + 0.0020350830018287525, + 0.0020222089951857924, + 0.0019269589975010604, + 0.0018784170097205788, + 0.001916708002681844, + 0.0019447500089881942, + 0.001902832998894155, + 0.0018786250002449378, + 0.0019859159947372973, + 0.0019760830036830157, + 0.002005500005907379, + 0.0019958329939981923, + 0.001978667001822032, + 0.001977500010980293, + 0.0019940420024795458, + 0.002000958993448876, + 0.001969124990864657, + 0.001955207990249619, + 0.0020375409949338064, + 0.002046292007435113, + 0.0018943330069305375, + 0.0019734590023290366, + 0.0020842499943682924, + 0.002019417006522417, + 0.0021417499956442043, + 0.002022208005655557, + 0.001991583005292341, + 0.001994874997762963, + 0.0019697919924510643, + 0.0019695829978445545, + 0.0020227919885655865 + ], + [ + 0.0017801249923650175, + 0.0018842920107999817, + 0.0018974169943248853, + 0.0018182909989263862, + 0.0018726670095929876, + 0.0017677919968264177, + 0.001720416999887675, + 0.001706499999272637, + 0.001712584009510465, + 0.001697542000329122, + 0.002208167003118433, + 0.0017789590056054294, + 0.0017141669959528372, + 0.0017141660064226016, + 0.0017794580053305253, + 0.0018219580088043585, + 0.0018933329993160442, + 0.001839999997173436, + 0.001821875004679896, + 0.0020638749992940575, + 0.0019101249927189201, + 0.0019178749935235828, + 0.0018472920055501163, + 0.0018100419983966276, + 0.001856165996287018, + 0.0019566249975468963, + 0.0018749580049188808, + 0.0020189589995425195, + 0.0019493330037221313, + 0.0018671249999897555, + 0.0018935420084744692, + 0.0018573749985080212, + 0.0018192089919466525, + 0.001998624997213483, + 0.0020411670120665804, + 0.001968374999705702, + 0.0020249169901944697, + 0.0019503340008668602, + 0.0019587919960031286, + 0.0019370000081835315, + 0.0019097080075880513, + 0.0019080830097664148, + 0.0019230830075684935, + 0.001927042001625523, + 0.0020329580001998693, + 0.0020327920065028593, + 0.0019393330003367737, + 0.001957707994733937, + 0.001990999997360632, + 0.001978291998966597, + 0.001994291989831254, + 0.0019752920052269474, + 0.001959000001079403, + 0.002017541992245242, + 0.0019646659930003807, + 0.0020617909904103726, + 0.0020396659965626895, + 0.0019509580015437678, + 0.0019694170041475445, + 0.002000374995986931, + 0.0021012499928474426, + 0.0019757080008275807, + 0.0019079579942626879, + 0.002170624997233972 + ], + [ + 0.0017825840041041374, + 0.0018835840019164607, + 0.001890916988486424, + 0.001817875003325753, + 0.0018387090094620362, + 0.0017611660005059093, + 0.001915792003273964, + 0.0019037500023841858, + 0.0018394999933661893, + 0.00192154198884964, + 0.001840167009504512, + 0.0018494579999241978, + 0.0018142499902751297, + 0.0018481250008335337, + 0.0018692500016186386, + 0.0018966670031659305, + 0.0017987079918384552, + 0.001796374999685213, + 0.0018427499890094623, + 0.0018927920027635992, + 0.001934790998348035, + 0.0019347079942235723, + 0.0019218329980503768, + 0.0018567499937489629, + 0.001886708996607922, + 0.0019114579918095842, + 0.0018405419978080317, + 0.0019104170060018077, + 0.0019158750073984265, + 0.001902874995721504, + 0.0018689999997150153, + 0.001885500008938834, + 0.001972582991584204, + 0.0019247079908382148, + 0.001890832994831726, + 0.0019653750059660524, + 0.0020136669918429106, + 0.0019125410035485402, + 0.001988750009331852, + 0.0019378330034669489, + 0.0019613330077845603, + 0.001949707992025651, + 0.0019187500001862645, + 0.0019197090005036443, + 0.002016792001086287, + 0.0019478750036796555, + 0.002015958001720719, + 0.0020665829943027347, + 0.0019021659973077476, + 0.00209854100830853, + 0.0019949169945903122, + 0.00199462499585934, + 0.0019267919997218996, + 0.001971666992176324, + 0.001934541985974647, + 0.001992500008782372, + 0.002093709001201205, + 0.0020582909928634763, + 0.0019090420100837946, + 0.001987540992558934, + 0.0019945829990319908, + 0.0019872909906553105, + 0.0019818340078927577, + 0.0020704170019598678 + ], + [ + 0.001708416995825246, + 0.0018801249971147627, + 0.0018900830036727712, + 0.0018157500016968697, + 0.0018714170000748709, + 0.0018504159961594269, + 0.001968207987374626, + 0.001749500006553717, + 0.00196433300152421, + 0.001941124995937571, + 0.0018565420032246038, + 0.0017878330108942464, + 0.0018667500116862357, + 0.001839249991462566, + 0.0017707499937387183, + 0.0018920829897979274, + 0.0019057500030612573, + 0.001938291999977082, + 0.0018607079982757568, + 0.0018767090077744797, + 0.0019736669928533956, + 0.0020502910047071055, + 0.0019485829980112612, + 0.00193349999608472, + 0.0019104590028291568, + 0.0018079169967677444, + 0.001818250006181188, + 0.0018676250037970021, + 0.001867999992100522, + 0.001915458997245878, + 0.0019410410022828728, + 0.001960209003300406, + 0.0019392919930396602, + 0.0019533750019036233, + 0.0018792500050039962, + 0.0018847920000553131, + 0.0019017909944523126, + 0.0019230410107411444, + 0.001907208003103733, + 0.0020033329928992316, + 0.0019911249983124435, + 0.0019802500028163195, + 0.0019299580017104745, + 0.002002207998884842, + 0.001960749999852851, + 0.0020154579979134724, + 0.001953332990524359, + 0.0020162500004516914, + 0.0019893749995389953, + 0.00207366599352099, + 0.0020263749902369455, + 0.0020143750007264316, + 0.001962084003025666, + 0.0019935830059694126, + 0.0019826249917969108, + 0.0019157089991495013, + 0.001896791989565827, + 0.0019124590035062283, + 0.002048083013505675, + 0.0019393749971641228, + 0.0020290419925004244, + 0.0020520840043900535, + 0.0020390830031828955, + 0.0019982079975306988 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_mha": { + "first_token_ms": { + "count": 5, + "median": 1.8841249984689057, + "p95": 1.9580170075641945, + "min": 1.8784590065479279, + "max": 1.9697920070029795, + "iqr": 0.029251008527353406, + "mean": 1.9049918046221137, + "cv": 0.02019075896696476 + }, + "decode_ms_per_step": { + "count": 315, + "median": 2.0625000033760443, + "p95": 2.806154498830438, + "min": 1.8449999915901572, + "max": 4.424457991262898, + "iqr": 0.15524998889304698, + "mean": 2.1341604479987706, + "cv": 0.14734723174221723 + }, + "total_ms": { + "count": 5, + "median": 132.76182803383563, + "p95": 148.71286177658476, + "min": 131.2207130540628, + "max": 152.48367097228765, + "iqr": 1.9399619050091133, + "mean": 136.35710002854466, + "cv": 0.06646993504231022 + }, + "tokens_per_second": { + "count": 5, + "median": 482.0662757346862, + "p95": 487.3805164282005, + "min": 419.71707260137606, + "max": 487.7278785524665, + "iqr": 7.055352868086231, + "mean": 470.8876019765431, + "cv": 0.061180514766782525 + }, + "median_step_ms_by_index": [ + 1.8841249984689057, + 1.9291670032544062, + 1.9949160050600767, + 1.9347080087754875, + 1.9827499927487224, + 1.9377920107217506, + 2.070249989628792, + 1.9989589927718043, + 1.970666999113746, + 1.9907090027118102, + 2.009874995565042, + 1.9448749953880906, + 2.0025410049129277, + 2.0184580062050372, + 1.9870840042131022, + 1.9787079945672303, + 1.997500003199093, + 2.0406670082593337, + 2.013082994380966, + 2.019000006839633, + 2.0076250075362623, + 2.050833005341701, + 2.0520830003079027, + 1.9923340005334467, + 1.9878750026691705, + 2.014790996327065, + 2.0101659902138636, + 2.001083004870452, + 1.9676670053740963, + 2.0380830101203173, + 1.9917080062441528, + 1.997542000026442, + 2.0232089882483706, + 2.063583000563085, + 2.032167001743801, + 2.060291008092463, + 2.0889580046059564, + 2.071374998195097, + 2.052166993962601, + 2.063375010038726, + 2.047375004622154, + 2.076458011288196, + 2.1065420005470514, + 2.1413750073406845, + 2.1079169964650646, + 2.163375000236556, + 2.174708992242813, + 2.1463330049300566, + 2.225916992756538, + 2.208707999670878, + 2.151957989553921, + 2.209250000305474, + 2.197041001636535, + 2.2047500096959993, + 2.15179099177476, + 2.1611660049529746, + 2.226416996563785, + 2.2082920040702447, + 2.211792001617141, + 2.17762500687968, + 2.223417002824135, + 2.202540999860503, + 2.2183749970281497, + 2.1836250089108944 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0018841249984689057, + 0.0018609170074341819, + 0.0018610000115586445, + 0.0019317499973112717, + 0.0018841669952962548, + 0.001905166995129548, + 0.002070249989628792, + 0.0019857500010402873, + 0.001989333992241882, + 0.0020013750036014244, + 0.001994792008190416, + 0.0019436250004218891, + 0.001958499997272156, + 0.0019322919979458675, + 0.001987084004213102, + 0.0019430829997872934, + 0.001997500003199093, + 0.0020406670082593337, + 0.0018449999915901572, + 0.0021322499960660934, + 0.0020076250075362623, + 0.002050833005341701, + 0.0020520830003079027, + 0.001968374999705702, + 0.0018995840073330328, + 0.0019750830106204376, + 0.0019012920092791319, + 0.0019446250080363825, + 0.0019439590105321258, + 0.002015999998548068, + 0.0019780000002356246, + 0.002023207998718135, + 0.0020232089882483706, + 0.002080542006297037, + 0.0020732919947477058, + 0.002060291008092463, + 0.002063624997390434, + 0.002071374998195097, + 0.0021122920006746426, + 0.002063375010038726, + 0.002029707990004681, + 0.0020143330038990825, + 0.0021065420005470514, + 0.0021413750073406845, + 0.0021079169964650646, + 0.002248290998977609, + 0.002174708992242813, + 0.0021463330049300566, + 0.0021999589953338727, + 0.002237709006294608, + 0.0022019579919287935, + 0.0021244579984340817, + 0.0020872919994872063, + 0.0021823749993927777, + 0.002209291997132823, + 0.002119708005920984, + 0.0022337499976856634, + 0.0022082920040702447, + 0.002211792001617141, + 0.0020220830047037452, + 0.0021089579968247563, + 0.0023310830001719296, + 0.0022367909987224266, + 0.0020787090033991262 + ], + [ + 0.0019697920070029795, + 0.0020134160004090518, + 0.0019949160050600767, + 0.0020932079933118075, + 0.0020812080038012937, + 0.001906208010041155, + 0.002399750010226853, + 0.0020064169948454946, + 0.001966874988283962, + 0.00195516599342227, + 0.0020250839879736304, + 0.002100124998833053, + 0.0020625000033760443, + 0.002107958003762178, + 0.00206583300314378, + 0.0019787079945672303, + 0.002065625012619421, + 0.002008458992349915, + 0.002099417004501447, + 0.002019000006839633, + 0.002001083004870452, + 0.002020582993282005, + 0.0020951250044163316, + 0.0019923340005334467, + 0.0019878750026691705, + 0.0020236670097801834, + 0.002067374996840954, + 0.0020739159954246134, + 0.0019676670053740963, + 0.0020380830101203173, + 0.002002249995712191, + 0.001997542000026442, + 0.0019766670011449605, + 0.002038667007582262, + 0.002032167001743801, + 0.002065500011667609, + 0.0020289999956730753, + 0.0020511249895207584, + 0.0020439590007299557, + 0.0020987499883631244, + 0.0021330000017769635, + 0.002118708987836726, + 0.0020333749998826534, + 0.0020672089885920286, + 0.0021887920011067763, + 0.002149124993593432, + 0.0020908750011585653, + 0.0021357079967856407, + 0.002225916992756538, + 0.002198124988353811, + 0.002151957989553921, + 0.002190291997976601, + 0.002197041001636535, + 0.0022047500096959993, + 0.00215179099177476, + 0.0021892079967074096, + 0.0022264169965637848, + 0.002220124995801598, + 0.0022293750080280006, + 0.002191875013522804, + 0.002223417002824135, + 0.0022025409998605028, + 0.002203374999226071, + 0.0021836250089108944 + ], + [ + 0.0019109170098090544, + 0.0019291670032544062, + 0.002087375003611669, + 0.0019138329953420907, + 0.0019217499939259142, + 0.0019856250000884756, + 0.0021554999984800816, + 0.0020672500104410574, + 0.002005500005907379, + 0.0020369160047266632, + 0.002048541995463893, + 0.0020147920004092157, + 0.001995915998122655, + 0.0020184580062050372, + 0.0019509999983711168, + 0.002009332994930446, + 0.002032290998613462, + 0.002078083998640068, + 0.002057499994407408, + 0.002055624994682148, + 0.002061416002106853, + 0.0020590830099536106, + 0.002028708011494018, + 0.0020248749933671206, + 0.002023084001848474, + 0.002014790996327065, + 0.002059582999208942, + 0.002001083004870452, + 0.001979166001547128, + 0.0019677499949466437, + 0.0020835830073338, + 0.0020944999996572733, + 0.0020854169997619465, + 0.002063583000563085, + 0.0020320420007919893, + 0.0020409999997355044, + 0.0021052499942015857, + 0.0021236659958958626, + 0.002052166993962601, + 0.0020429590076673776, + 0.002047375004622154, + 0.002073916999506764, + 0.002020582993282005, + 0.0020597910042852163, + 0.0020044590055476874, + 0.002163375000236556, + 0.0021705420076614246, + 0.002072917006444186, + 0.0023360420018434525, + 0.0021604579960694537, + 0.0020756659941980615, + 0.002209250000305474, + 0.0022487909882329404, + 0.0021442500001285225, + 0.0020160830026725307, + 0.0021611660049529746, + 0.0022149170108605176, + 0.0022014169953763485, + 0.002181709001888521, + 0.00217762500687968, + 0.0022480409970739856, + 0.0021120409946888685, + 0.0022183749970281497, + 0.00222395799937658 + ], + [ + 0.0018784590065479279, + 0.0018970420060213655, + 0.0019993339956272393, + 0.0019721249991562217, + 0.002009333999012597, + 0.001991375000216067, + 0.0020589170017046854, + 0.0019989589927718043, + 0.001970666999113746, + 0.00199070900271181, + 0.0019560420041671023, + 0.0019400000019231811, + 0.0020025410049129277, + 0.002023958004429005, + 0.001996332997805439, + 0.001927790988702327, + 0.0019440000032773241, + 0.0020457080099731684, + 0.002013082994380966, + 0.001983791997190565, + 0.0020019999938085675, + 0.002029541996307671, + 0.001975959006813355, + 0.0019602919928729534, + 0.0018970419914694503, + 0.0019504169904394075, + 0.002008541996474378, + 0.001971000005141832, + 0.0019706659950315952, + 0.0022844999912194908, + 0.0019519170018611476, + 0.001908833990455605, + 0.0021381670085247606, + 0.0020926670113112777, + 0.0019779160065809265, + 0.001999207990593277, + 0.0023266659991350025, + 0.001957332991878502, + 0.0019650000031106174, + 0.0019680420082295313, + 0.001968374999705702, + 0.0021470829960890114, + 0.0023183330049505457, + 0.0030557499994756654, + 0.0031140409992076457, + 0.0031362909940071404, + 0.002838792002876289, + 0.0029218749987194315, + 0.0029487499996321276, + 0.002490583006874658, + 0.002863500005332753, + 0.002746958998613991, + 0.002558459003921598, + 0.002579916996182874, + 0.0027921669970965013, + 0.0031308340112445876, + 0.0031950839911587536, + 0.003189166003721766, + 0.0031331249920185655, + 0.003241958998842165, + 0.004424457991262898, + 0.0038985829887678847, + 0.0038905000110389665, + 0.003963208000641316 + ], + [ + 0.001881666001281701, + 0.0019918750040233135, + 0.0019151249871356413, + 0.0019347080087754875, + 0.0019827499927487224, + 0.0019377920107217506, + 0.0019345420005265623, + 0.0019498749898048118, + 0.0019387500069569796, + 0.0019895830046152696, + 0.002009874995565042, + 0.0019448749953880906, + 0.002026541988016106, + 0.0020139580010436475, + 0.0019512500002747402, + 0.0019955000025220215, + 0.001931999999214895, + 0.0019775830005528405, + 0.0019889169925590977, + 0.001946791002410464, + 0.0020320839976193383, + 0.0020696669962489977, + 0.0020699580054497346, + 0.002089250003336929, + 0.002088167006149888, + 0.00207162500009872, + 0.0020101659902138636, + 0.002131792003638111, + 0.0019060000049648806, + 0.0020857499912381172, + 0.001991708006244153, + 0.0019606249989010394, + 0.0019248330063419417, + 0.0020409999997355044, + 0.0021025000023655593, + 0.0021256250038277358, + 0.0020889580046059564, + 0.002123375001247041, + 0.0020758750033564866, + 0.002065625012619421, + 0.0020888340077362955, + 0.002076458011288196, + 0.0021385000000009313, + 0.002168167004128918, + 0.0020571659988490865, + 0.002090375011903234, + 0.0021927920024609193, + 0.002177292000851594, + 0.0021699580101994798, + 0.002208707999670878, + 0.002130417007720098, + 0.002212542007328011, + 0.0021799170062877238, + 0.002224083000328392, + 0.0020962080016033724, + 0.002099541001371108, + 0.0020752919954247773, + 0.0021022910077590495, + 0.0021273339953040704, + 0.0021303329995134845, + 0.0021892910008318722, + 0.002140665994375013, + 0.00213350000558421, + 0.002181457995902747 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "speedup_cached_vs_uncached_sdpa": 1.2004839033624348, + "speedup_cached_vs_uncached_mha": 1.2955459637553848 + }, + { + "batch": 1, + "prompt_len": 512, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_mha": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 3.335790999699384, + "p95": 3.76898359390907, + "min": 3.284042002633214, + "max": 3.77204199321568, + "iqr": 0.4657500103348866, + "mean": 3.48792499571573, + "cv": 0.07259908953737113 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.7430000007152557, + "p95": 2.235337602905929, + "min": 1.592375003383495, + "max": 2.628709000418894, + "iqr": 0.1055624961736612, + "mean": 1.8135093721202649, + "cv": 0.1097309449645518 + }, + "total_ms": { + "count": 5, + "median": 118.33212907367852, + "p95": 120.0572738511255, + "min": 114.39383197284769, + "max": 120.07654205081053, + "iqr": 4.067828005645424, + "mean": 117.73901543929242, + "cv": 0.02138809938952101 + }, + "tokens_per_second": { + "count": 5, + "median": 540.8505745734612, + "p95": 558.0048248447088, + "min": 532.9933632908776, + "max": 559.4707240438534, + "iqr": 18.719884871213026, + "mean": 543.775446626648, + "cv": 0.02153162635819572 + }, + "median_step_ms_by_index": [ + 3.335790999699384, + 2.437999995891005, + 1.7979999975068495, + 2.216542008682154, + 1.7429160070605576, + 1.7753749998519197, + 1.8061669979942963, + 2.2175409976625815, + 1.7764590011211112, + 1.634125001146458, + 1.811000009183772, + 1.7947079904843122, + 1.8134580022888258, + 1.7756669985828921, + 1.9613330077845603, + 1.7391669971402735, + 2.103000006172806, + 1.7383329977747053, + 2.0567089959513396, + 1.769583992427215, + 1.7789169942261651, + 1.8088330107275397, + 1.6728750051697716, + 1.7196250119013712, + 1.7471669998485595, + 1.74320800579153, + 1.73049999284558, + 1.789332993212156, + 1.770333998138085, + 1.7580829880898818, + 1.75216599018313, + 1.7193330131703988, + 1.745542002026923, + 1.6135000041685998, + 1.7502080008853227, + 1.7394999886164442, + 1.7403329984517768, + 1.7473749903729185, + 1.6659999964758754, + 1.8064169998979196, + 1.731916010612622, + 1.742250009556301, + 1.708041992969811, + 1.704541005892679, + 1.7729579994920641, + 1.726874994346872, + 1.7237500142073259, + 1.7095829971367493, + 1.691832992946729, + 1.7248329968424514, + 1.6431249969173223, + 1.7335829907096922, + 1.7155410023406148, + 1.6592909960309044, + 1.7269579984713346, + 1.7377920012222603, + 1.746458001434803, + 1.7337080062134191, + 1.662667011260055, + 1.714124999125488, + 1.7302079941146076, + 1.7450000013923272, + 1.7110830085584894, + 1.732042001094669 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.003335790999699384, + 0.002056707991869189, + 0.0022548749984707683, + 0.0018433750083204359, + 0.002220540991402231, + 0.0017654170078458264, + 0.0017036249919328839, + 0.002285666996613145, + 0.002271958001074381, + 0.0016024159995140508, + 0.002014625002630055, + 0.002095207993988879, + 0.0017894170014187694, + 0.002228666009614244, + 0.002296374994330108, + 0.0021008329931646585, + 0.0017481249960837886, + 0.002209333993960172, + 0.0017482080002082512, + 0.0019711250060936436, + 0.0016747080080676824, + 0.0022030830004950985, + 0.001626083001610823, + 0.00218508300895337, + 0.0017313330026809126, + 0.001749917006236501, + 0.002286999995703809, + 0.0017328329995507374, + 0.0016717500111553818, + 0.0017600000137463212, + 0.002112583999405615, + 0.0016294999950332567, + 0.001745542002026923, + 0.0016135000041685998, + 0.0016481249913340434, + 0.002152125001884997, + 0.0017426669946871698, + 0.0017473749903729185, + 0.0016659999964758754, + 0.0016867500089574605, + 0.001731916010612622, + 0.001742833002936095, + 0.001708041992969811, + 0.0016546249971725047, + 0.002113082999130711, + 0.0017685830098344013, + 0.0017237500142073259, + 0.0017193750099977478, + 0.001735583005938679, + 0.0016289159975713119, + 0.00164079200476408, + 0.0017219169967574999, + 0.0017155410023406148, + 0.0016592909960309044, + 0.0017269579984713346, + 0.0017377920012222603, + 0.001746458001434803, + 0.0017482919938629493, + 0.001647042008698918, + 0.0017260409949813038, + 0.001829290995374322, + 0.0017375000024912879, + 0.0017365000094287097, + 0.0021718329953728244 + ], + [ + 0.00377204199321568, + 0.002437999995891005, + 0.001786833003279753, + 0.002216542008682154, + 0.0017429160070605576, + 0.001761083010933362, + 0.001873458008049056, + 0.0022175409976625815, + 0.0016976670012809336, + 0.002175458997953683, + 0.0016262919962173328, + 0.0017240829911315814, + 0.0022004170023137704, + 0.0017265000060433522, + 0.0019613330077845603, + 0.0016234999930020422, + 0.002103000006172806, + 0.001709374992060475, + 0.0022191250027390197, + 0.0017607499903533608, + 0.0019455840083537623, + 0.0017150000057881698, + 0.0016381250025006011, + 0.001618124995729886, + 0.0016972500015981495, + 0.0016942079964792356, + 0.0016655419894959778, + 0.001789332993212156, + 0.001770333998138085, + 0.0017235000123037025, + 0.00168041700089816, + 0.0017193330131703988, + 0.0017240839952137321, + 0.0016081250068964437, + 0.0017655410047154874, + 0.001641500013647601, + 0.0017255840066354722, + 0.0019228330056648701, + 0.0020100839901715517, + 0.0018064169998979196, + 0.0016554999892832711, + 0.0016647499869577587, + 0.0017477499932283536, + 0.0017518750100862235, + 0.001740541003528051, + 0.0016531250003026798, + 0.0017662500031292439, + 0.0016829999949550256, + 0.001719249994494021, + 0.0017248329968424514, + 0.0016404159978264943, + 0.0017060000100173056, + 0.0016136250051204115, + 0.0016918330074986443, + 0.001725333000649698, + 0.0017332080024061725, + 0.0017077920056181028, + 0.0017337080062134191, + 0.0016814999980852008, + 0.0016292499931296334, + 0.0017302079941146076, + 0.0017450000013923272, + 0.001636916000279598, + 0.0016338749992428347 + ], + [ + 0.003284042002633214, + 0.002015791993471794, + 0.0017979999975068495, + 0.0022010420070728287, + 0.0016601670067757368, + 0.0017753749998519197, + 0.00175124999077525, + 0.0020751250121975318, + 0.0017764590011211112, + 0.001634125001146458, + 0.0017371670110151172, + 0.0017404999962309375, + 0.001734124991344288, + 0.0017713749984977767, + 0.0022469170071417466, + 0.0017221250018337741, + 0.001689291006186977, + 0.001748750000842847, + 0.0021213330037426203, + 0.0019244579889345914, + 0.0017789169942261651, + 0.001726874994346872, + 0.0016728750051697716, + 0.0018025000026682392, + 0.0017471669998485595, + 0.00174320800579153, + 0.00173049999284558, + 0.001752374999341555, + 0.0017745000077411532, + 0.0017580829880898818, + 0.00175216599018313, + 0.0017289170064032078, + 0.001752791999024339, + 0.0017922499973792583, + 0.0017502080008853227, + 0.0018945839983643964, + 0.0017403329984517768, + 0.0016353749961126596, + 0.0016506249958183616, + 0.0017761250055627897, + 0.0017594999953871593, + 0.001650874997721985, + 0.001594916990143247, + 0.0017528330063214526, + 0.0017423329991288483, + 0.0017371660069329664, + 0.0016414580022683367, + 0.001634083004319109, + 0.001691832992946729, + 0.0017376659961882979, + 0.0016431249969173223, + 0.0017335829907096922, + 0.0017494579951744527, + 0.001738250008202158, + 0.0017320839979220182, + 0.001635291991988197, + 0.0017217089916812256, + 0.0018144580099033192, + 0.0017785420059226453, + 0.001714124999125488, + 0.0016574589972151443, + 0.0017255409911740571, + 0.0017110830085584894, + 0.0017286660004174337 + ], + [ + 0.003756749996682629, + 0.002503875002730638, + 0.001780541002517566, + 0.002229541991255246, + 0.0016172920004464686, + 0.0020551250054268166, + 0.001937499997438863, + 0.0021508750069187954, + 0.001774499993189238, + 0.001617583999177441, + 0.0022642090043518692, + 0.0017947079904843122, + 0.0021042080043116584, + 0.0017756669985828921, + 0.0016109590069390833, + 0.0017759579932317138, + 0.0022303750010905787, + 0.0017383329977747053, + 0.0020567089959513396, + 0.0016817499999888241, + 0.001635084001463838, + 0.002109292006934993, + 0.001856540999142453, + 0.0017196250119013712, + 0.0017649999936111271, + 0.0017321250052191317, + 0.0016050000122049823, + 0.002116707997629419, + 0.00179049999860581, + 0.001732582997647114, + 0.0022520839993376285, + 0.0016250830085482448, + 0.002150291998987086, + 0.001592375003383495, + 0.0017645420011831447, + 0.0017394999886164442, + 0.0015975410060491413, + 0.001750250012264587, + 0.001726124988636002, + 0.0021605829970212653, + 0.00171237500035204, + 0.001742250009556301, + 0.0016801250021671876, + 0.0016864589997567236, + 0.0021012919896747917, + 0.0017259590094909072, + 0.0017553340003360063, + 0.00172266700246837, + 0.0016846250073285773, + 0.0016467910027131438, + 0.0017294999997830018, + 0.0017461670067859814, + 0.0017342089995509014, + 0.0016467079985886812, + 0.0017237910069525242, + 0.0017398750060237944, + 0.001769125010468997, + 0.0016966249968390912, + 0.001648207995458506, + 0.0016527920088265091, + 0.001780209000571631, + 0.0017522080015623942, + 0.0016455000004498288, + 0.001732042001094669 + ], + [ + 0.0032909999863477424, + 0.002628709000418894, + 0.0023895420017652214, + 0.002325249995919876, + 0.002281709006638266, + 0.002187209000112489, + 0.0018061669979942963, + 0.0023137920070439577, + 0.0021919589926255867, + 0.0022141250083222985, + 0.001811000009183772, + 0.002305791000253521, + 0.0018134580022888258, + 0.0021330000017769635, + 0.0017787079996196553, + 0.0017391669971402735, + 0.002172291002352722, + 0.0016977920022327453, + 0.001768000001902692, + 0.001769583992427215, + 0.0020272920082788914, + 0.0018088330107275397, + 0.0021187500096857548, + 0.0016673339996486902, + 0.0017613749951124191, + 0.0017656249983701855, + 0.0017904589913086966, + 0.0018401249981252477, + 0.001679249995504506, + 0.0018035419925581664, + 0.0016604160045972094, + 0.0017557079991092905, + 0.001712915996904485, + 0.0017205420008394867, + 0.001745290996041149, + 0.0016329160134773701, + 0.0017438750073779374, + 0.0016945829993346706, + 0.001662707989453338, + 0.0022151250013848767, + 0.001778833000571467, + 0.0017818749911384657, + 0.0017657919961493462, + 0.001704541005892679, + 0.0017729579994920641, + 0.001726874994346872, + 0.0016758330020820722, + 0.0017095829971367493, + 0.0016712500073481351, + 0.0017430000007152557, + 0.0017474580090492964, + 0.0017553340003360063, + 0.0016812499961815774, + 0.0016412080003647134, + 0.0017816250037867576, + 0.0018204999942099676, + 0.0017846660048235208, + 0.0017174999957205728, + 0.001662667011260055, + 0.001761500010616146, + 0.0016754999960539863, + 0.0017494170024292544, + 0.0017677090072538704, + 0.0017747500096447766 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + ] + } + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 3.2972920016618446, + "p95": 3.533683199202642, + "min": 3.2201669964706525, + "max": 3.544332997989841, + "iqr": 0.21075100812595338, + "mean": 3.3666417992208153, + "cv": 0.04220924013695092 + }, + "decode_ms_per_step": { + "count": 315, + "median": 3.6199159949319437, + "p95": 4.1804330016020685, + "min": 3.2942499965429306, + "max": 4.810582991922274, + "iqr": 0.34643750404939055, + "mean": 3.6926051495977426, + "cv": 0.07314806176721375 + }, + "total_ms": { + "count": 5, + "median": 229.823710033088, + "p95": 249.73000620375387, + "min": 227.50508108583745, + "max": 251.23795500257984, + "iqr": 15.95933701901231, + "mean": 236.0007662238786, + "cv": 0.045931650200821676 + }, + "tokens_per_second": { + "count": 5, + "median": 278.47431403307274, + "p95": 281.25464024607544, + "min": 254.73858040017407, + "max": 281.31239836288694, + "iqr": 18.40370697955416, + "mean": 271.6337602748477, + "cv": 0.044912480750848964 + }, + "median_step_ms_by_index": [ + 3.2972920016618446, + 3.4100000048056245, + 3.5016670008189976, + 3.4976669994648546, + 3.486292000161484, + 3.7261660036165267, + 3.558541997335851, + 3.4269590105395764, + 3.459041006863117, + 3.5088329896098003, + 3.8992080080788583, + 3.5962499969173223, + 3.5008330014534295, + 3.68362499284558, + 3.562249999959022, + 3.5870830033672974, + 3.5079170047538355, + 3.4825830080080777, + 3.7410410004667938, + 3.6199159949319437, + 3.534333998686634, + 3.509500005748123, + 3.5136250080540776, + 3.483583001070656, + 3.5317919973749667, + 3.692582991789095, + 3.433958991081454, + 3.5130420001223683, + 3.496417004498653, + 3.4792910009855404, + 3.795625001657754, + 3.435416001593694, + 3.3854590001283213, + 3.7857919960515574, + 3.5763329942710698, + 3.7711670011049137, + 3.7554999871645123, + 3.5196670069126412, + 3.7110830016899854, + 3.7223749968688935, + 3.5368750104680657, + 3.5904589894926175, + 3.6412500048754737, + 3.630457998951897, + 3.778667000005953, + 3.6180000024614856, + 3.591792003135197, + 3.815750009380281, + 4.028916999232024, + 3.6532499943859875, + 3.6423749988898635, + 3.699958004290238, + 3.572374989744276, + 3.827417007414624, + 3.7307500024326146, + 3.9518749981652945, + 3.7711250042775646, + 3.8177499955054373, + 3.583832993172109, + 3.6941250000381842, + 3.6776249908143654, + 3.5833750007441267, + 3.936542008887045, + 3.5998749954160303 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0032972920016618446, + 0.003372375009348616, + 0.0038086669956101105, + 0.004263499999069609, + 0.004183583005215041, + 0.004257791995769367, + 0.0041699170105857775, + 0.0047703750024084, + 0.004810582991922274, + 0.004111999995075166, + 0.0041091659950325266, + 0.0037785839958814904, + 0.003748917006305419, + 0.0039366249984595925, + 0.003965791009250097, + 0.00381166698934976, + 0.0037861669989069924, + 0.0037460000021383166, + 0.003766499998164363, + 0.003716000006534159, + 0.0033607079967623577, + 0.003645707998657599, + 0.0035136250080540776, + 0.0037716250080848113, + 0.0037647920107701793, + 0.003796416989644058, + 0.003376042004674673, + 0.003513707997626625, + 0.003485416993498802, + 0.003439457999775186, + 0.003795625001657754, + 0.0033642089983914047, + 0.003412416990613565, + 0.003874250003718771, + 0.0035410410055192187, + 0.0038270829973043874, + 0.0037554999871645123, + 0.0035057920031249523, + 0.003948458994273096, + 0.003865499995299615, + 0.003473416989436373, + 0.0038785840006312355, + 0.003793667012359947, + 0.0035200840065954253, + 0.003906875004759058, + 0.003880666001350619, + 0.003550124994944781, + 0.003815750009380281, + 0.0039012080087559298, + 0.0039002080011414364, + 0.003942541996366344, + 0.00431245900108479, + 0.003567624997231178, + 0.0038586660084547475, + 0.0038875000027474016, + 0.0039518749981652945, + 0.0035532920010155067, + 0.0038177499955054373, + 0.003542040998581797, + 0.003924624994397163, + 0.003911957988748327, + 0.003938041001674719, + 0.003975167011958547, + 0.003925208991859108 + ], + [ + 0.0032803329959278926, + 0.0034100000048056245, + 0.0035016670008189976, + 0.0034137910115532577, + 0.003486292000161484, + 0.003432333003729582, + 0.003434665995882824, + 0.0034269590105395764, + 0.003405209004995413, + 0.003417750005610287, + 0.0034100000048056245, + 0.0034630000009201467, + 0.0033966659975703806, + 0.0033959160064114258, + 0.0033750000002328306, + 0.0035117090010317042, + 0.0034376669937046245, + 0.003455666999798268, + 0.0034841250017052516, + 0.003445791997364722, + 0.003534333998686634, + 0.003488582995487377, + 0.0038316669961204752, + 0.003483583001070656, + 0.003412749996641651, + 0.003692582991789095, + 0.003433958991081454, + 0.0034299999970244244, + 0.003496417004498653, + 0.0034515419974923134, + 0.0038042079977458343, + 0.003529708003043197, + 0.0033584159973543137, + 0.0035387920070206746, + 0.0035763329942710698, + 0.0037711670011049137, + 0.0035348750097909942, + 0.0035196670069126412, + 0.003541290992870927, + 0.0037223749968688935, + 0.0037503329949686304, + 0.0035394169972278178, + 0.0035133750061504543, + 0.003630457998951897, + 0.003778667000005953, + 0.0036180000024614856, + 0.003591792003135197, + 0.003914999993867241, + 0.0036820000095758587, + 0.0036532499943859875, + 0.0035699580039363354, + 0.003563582999049686, + 0.003572374989744276, + 0.0037828330096090212, + 0.0037307500024326146, + 0.003549332992406562, + 0.004059665996464901, + 0.003692582991789095, + 0.003705583992996253, + 0.003558292009984143, + 0.0035714169935090467, + 0.0035802909987978637, + 0.0038189589977264404, + 0.0035741659958148375 + ], + [ + 0.0032201669964706525, + 0.003384541007108055, + 0.0032942499965429306, + 0.003378124994924292, + 0.0034380839933874086, + 0.003412958001717925, + 0.0033955410035559908, + 0.0033210840047104284, + 0.003294250011094846, + 0.0035088329896098003, + 0.0033335830084979534, + 0.003530083005898632, + 0.0034361250000074506, + 0.00368362499284558, + 0.0034401250013615936, + 0.0034382079902570695, + 0.003446292001171969, + 0.0034825830080080777, + 0.0037410410004667938, + 0.0034688330051722005, + 0.0033737920020939782, + 0.003409124998142943, + 0.0034734159999061376, + 0.0036835000064456835, + 0.0033667499956209213, + 0.003336042005685158, + 0.003685874995426275, + 0.0035130420001223683, + 0.003587790997698903, + 0.0034792910009855404, + 0.003445791997364722, + 0.003435416001593694, + 0.003701958994497545, + 0.0037857919960515574, + 0.0035507080028764904, + 0.003548749999026768, + 0.0034558750048745424, + 0.003732957993634045, + 0.0037110830016899854, + 0.0035497500066412613, + 0.0035368750104680657, + 0.0037162500084377825, + 0.0037936669978080317, + 0.003550292007275857, + 0.003516833996400237, + 0.0035311250103404745, + 0.003750583011424169, + 0.003695916006108746, + 0.004028916999232024, + 0.00356645901047159, + 0.0036423749988898635, + 0.003699958004290238, + 0.0035638330009533092, + 0.003827417007414624, + 0.003601874996093102, + 0.004077249992405996, + 0.0036292919976403937, + 0.0034702499979175627, + 0.003583832993172109, + 0.0036941250000381842, + 0.00377541700436268, + 0.0035833750007441267, + 0.003552959009539336, + 0.0036411660112207755 + ], + [ + 0.003544332997989841, + 0.0036594580014934763, + 0.004088625006261282, + 0.004393708994030021, + 0.0037096250016475096, + 0.004208750004181638, + 0.003558541997335851, + 0.0037486670044017956, + 0.004444624995812774, + 0.004041166001115926, + 0.004085917011252604, + 0.003933083004085347, + 0.004074040989507921, + 0.00404549999802839, + 0.003973707993282005, + 0.003952915998524986, + 0.004055208002682775, + 0.0038429169944720343, + 0.003815459000179544, + 0.0037958330067340285, + 0.0038779579917900264, + 0.0038356669974746183, + 0.003863792007905431, + 0.003437125007621944, + 0.0038038329948903993, + 0.0037716670049121603, + 0.0038079589867265895, + 0.0036941250000381842, + 0.003840749996015802, + 0.004006165996543132, + 0.00400416599586606, + 0.0038979160017333925, + 0.0033854590001283213, + 0.004150875000050291, + 0.0035808329994324595, + 0.0038592080090893432, + 0.003905667006620206, + 0.0039669159887125716, + 0.003556208001100458, + 0.0037926670047454536, + 0.0035964159906143323, + 0.0035904589894926175, + 0.0036412500048754737, + 0.003709166994667612, + 0.003841499987174757, + 0.0038977920048637316, + 0.004175667010713369, + 0.004005791008239612, + 0.004042625005240552, + 0.004012541990960017, + 0.004314500009058975, + 0.004676458003814332, + 0.004655915996409021, + 0.0045635830028913915, + 0.004358791004051454, + 0.004295874998206273, + 0.004162666999036446, + 0.004179083000053652, + 0.003967917000409216, + 0.003906957994331606, + 0.0035664589959196746, + 0.0035674999962793663, + 0.003936542008887045, + 0.003561458011972718 + ], + [ + 0.003491084004053846, + 0.0034267500013811514, + 0.0034047499939333647, + 0.0034976669994648546, + 0.003429542004596442, + 0.0037261660036165267, + 0.0035606250021373853, + 0.003382082999451086, + 0.0034590410068631172, + 0.0034684579877648503, + 0.0038992080080788583, + 0.0035962499969173223, + 0.0035008330014534295, + 0.0034808329946827143, + 0.003562249999959022, + 0.0035870830033672974, + 0.0035079170047538355, + 0.003449708005064167, + 0.0034736669913399965, + 0.0036199159949319437, + 0.003665917000034824, + 0.003509500005748123, + 0.003313584005809389, + 0.0033155839919345453, + 0.0035317919973749667, + 0.0035686670016730204, + 0.003426499999477528, + 0.0033279159979429096, + 0.0033839169918792322, + 0.003518124998663552, + 0.0035406670067459345, + 0.0033856250083772466, + 0.003366124990861863, + 0.003522790997521952, + 0.0037970419944031164, + 0.003681416012113914, + 0.0038387079985113814, + 0.0034790419886121526, + 0.003968333010561764, + 0.0036789590085390955, + 0.00343295800848864, + 0.003419999993639067, + 0.0035917089990107343, + 0.003708542004460469, + 0.003733709003427066, + 0.003482708998490125, + 0.0035119170061079785, + 0.0037646669952664524, + 0.004048457994940691, + 0.0036057910037925467, + 0.003522083003190346, + 0.003607042002840899, + 0.00383916700957343, + 0.003718624997418374, + 0.003647792007541284, + 0.003567250008927658, + 0.0037711250042775646, + 0.0042276250023860484, + 0.003510916998493485, + 0.00350287499895785, + 0.0036776249908143654, + 0.0038404999941121787, + 0.004146708000916988, + 0.0035998749954160303 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_mha": { + "first_token_ms": { + "count": 5, + "median": 3.9806669956305996, + "p95": 4.157308395951986, + "min": 3.7408329953905195, + "max": 4.188624996459112, + "iqr": 0.21487599587999284, + "mean": 3.9518665958894417, + "cv": 0.04490951888772772 + }, + "decode_ms_per_step": { + "count": 315, + "median": 4.132917005335912, + "p95": 4.651200091757346, + "min": 3.7604590033879504, + "max": 4.886541995801963, + "iqr": 0.3267290012445301, + "mean": 4.178171688626106, + "cv": 0.05434980592668127 + }, + "total_ms": { + "count": 5, + "median": 265.95800496579614, + "p95": 271.79801456513815, + "min": 262.5318280333886, + "max": 271.95703897450585, + "iqr": 6.887290932354517, + "mean": 267.17668297933415, + "cv": 0.01568153269561584 + }, + "tokens_per_second": { + "count": 5, + "median": 240.63949497677575, + "p95": 243.4584369929979, + "min": 235.33128703463922, + "max": 243.7799655737762, + "iqr": 6.15097894604591, + "mean": 239.58888279578295, + "cv": 0.01564451649482819 + }, + "median_step_ms_by_index": [ + 3.9806669956305996, + 3.965875002904795, + 3.9482499996665865, + 3.9248329994734377, + 4.336583006079309, + 3.9657080051256344, + 3.9472909993492067, + 3.944875003071502, + 4.324624998844229, + 3.9529160130769014, + 3.938000008929521, + 3.9365830016322434, + 3.963041992392391, + 4.029582996736281, + 3.994416998466477, + 3.9720000058878213, + 4.015707992948592, + 3.9543340099044144, + 4.07316698692739, + 4.031917007523589, + 4.135209004743956, + 3.977874992415309, + 4.115125004318543, + 3.964916992117651, + 4.041250009322539, + 3.9766249974491075, + 3.970790989114903, + 3.988542011938989, + 4.182958000455983, + 4.391417009173892, + 4.020584005047567, + 4.0135409944923595, + 4.167082995991223, + 4.309208001359366, + 4.091750000952743, + 4.103917002794333, + 4.350667004473507, + 4.2670840048231184, + 4.075958000612445, + 4.069250004249625, + 4.2672919953474775, + 4.178082992439158, + 4.123749997233972, + 4.136000003200024, + 4.113625007448718, + 4.141957993851975, + 4.167291001067497, + 4.1438329935772344, + 4.140374992857687, + 4.169709005509503, + 4.2081660067196935, + 4.239500005496666, + 4.152666006120853, + 4.173875000560656, + 4.439291005837731, + 4.288083000574261, + 4.219583002850413, + 4.6687499998370185, + 4.222750008921139, + 4.215875000227243, + 4.195000001345761, + 4.6319999964907765, + 4.556874991976656, + 4.578666994348168 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.004032041993923485, + 0.003870250002364628, + 0.004215500011923723, + 0.004297166989999823, + 0.004408875000081025, + 0.003935875007300638, + 0.00399670799379237, + 0.003928999998606741, + 0.004325625006458722, + 0.003926207995391451, + 0.003893832996254787, + 0.003932000006898306, + 0.003937999994377606, + 0.004328333001467399, + 0.004283665999537334, + 0.00391204199695494, + 0.004253541002981365, + 0.003954334009904414, + 0.0043623340025078505, + 0.004031917007523589, + 0.003951666003558785, + 0.004225790995405987, + 0.004390250003780238, + 0.004095290991244838, + 0.003985374991316348, + 0.0039962080045370385, + 0.003976874999352731, + 0.003986083000199869, + 0.003969625002355315, + 0.0043914170091738924, + 0.004005583003163338, + 0.003881500000716187, + 0.0041670829959912226, + 0.0045215419959276915, + 0.00410587499209214, + 0.004103917002794333, + 0.004461624994291924, + 0.0042670840048231184, + 0.004079958991496824, + 0.004067792004207149, + 0.004082291008671746, + 0.004077958001289517, + 0.004071041999850422, + 0.004136000003200024, + 0.0040645830013090745, + 0.003985792005551048, + 0.004653416996006854, + 0.004399666999233887, + 0.004132917005335912, + 0.004106209002202377, + 0.004101166996406391, + 0.004132665999350138, + 0.004127666994463652, + 0.004150416003540158, + 0.004128041997319087, + 0.004133249996812083, + 0.004140874996664934, + 0.004149166998104192, + 0.00413504199241288, + 0.004149542000959627, + 0.004195000001345761, + 0.004217124995193444, + 0.004181291995337233, + 0.004165708000073209 + ], + [ + 0.0038171659980434924, + 0.00394662500184495, + 0.0039482499996665865, + 0.003924332995666191, + 0.003953708001063205, + 0.0039034579967847094, + 0.003978207998443395, + 0.003971707992604934, + 0.00432383400038816, + 0.0039529160130769014, + 0.003916417001164518, + 0.003897457994753495, + 0.003960874993936159, + 0.0039449580071959645, + 0.003923333992133848, + 0.003972000005887821, + 0.0039482499996665865, + 0.003975125000579283, + 0.003976166000938974, + 0.003966000003856607, + 0.0039410000026691705, + 0.003977874992415309, + 0.00401225000678096, + 0.003937208006391302, + 0.004041250009322539, + 0.0039766249974491075, + 0.003989582997746766, + 0.003906708006979898, + 0.003935207991162315, + 0.004509417005465366, + 0.004004166999948211, + 0.003947167002479546, + 0.003917708003427833, + 0.004309208001359366, + 0.004091750000952743, + 0.003932457999326289, + 0.004081457998836413, + 0.00403595800162293, + 0.004075958000612445, + 0.00408083300862927, + 0.0042672919953474775, + 0.004178082992439158, + 0.004123749997233972, + 0.0041002499929163605, + 0.004113625007448718, + 0.004291416989872232, + 0.004167291001067497, + 0.004113749993848614, + 0.004140374992857687, + 0.004297709005186334, + 0.004286250012228265, + 0.003999666994786821, + 0.004152666006120853, + 0.00432570900011342, + 0.004439291005837731, + 0.004058332997374237, + 0.004147083003772423, + 0.004785125012858771, + 0.004222750008921139, + 0.004215875000227243, + 0.004201624993584119, + 0.00485508400015533, + 0.00462345800769981, + 0.004520791990216821 + ], + [ + 0.0039806669956305996, + 0.003972457998315804, + 0.004387416993267834, + 0.003916583998943679, + 0.004458292009076104, + 0.003971375001128763, + 0.003898082999512553, + 0.003944875003071502, + 0.004715832998044789, + 0.00418625000747852, + 0.004269375000149012, + 0.004020749998744577, + 0.004191583997453563, + 0.0042063329892698675, + 0.003994416998466477, + 0.004153665999183431, + 0.004491375002544373, + 0.004285666989744641, + 0.00407316698692739, + 0.004034832993056625, + 0.004237249988364056, + 0.004233250001561828, + 0.004115125004318543, + 0.003949374993680976, + 0.004187041995464824, + 0.004304583999328315, + 0.003970790989114903, + 0.003988542011938989, + 0.0043628330022329465, + 0.004316124992328696, + 0.004062833002535626, + 0.0040135409944923595, + 0.004384625004604459, + 0.004277542000636458, + 0.004072832991369069, + 0.004095041993423365, + 0.004369125002995133, + 0.00431541699799709, + 0.004038416998810135, + 0.004065084009198472, + 0.0044051669974578544, + 0.004241250004270114, + 0.004121042002225295, + 0.004107624990865588, + 0.00439520800136961, + 0.004141957993851975, + 0.004100332997040823, + 0.0041438329935772344, + 0.004476040994632058, + 0.00425037500099279, + 0.0042081660067196935, + 0.004239500005496666, + 0.0044911660079378635, + 0.004681874997913837, + 0.0046106250083539635, + 0.004690333007602021, + 0.0046140829945215955, + 0.0046687499998370185, + 0.0046641670051030815, + 0.004167667008005083, + 0.004143416997976601, + 0.004149541986407712, + 0.004203416989184916, + 0.004734000001917593 + ], + [ + 0.004188624996459112, + 0.00419300000066869, + 0.003925457989680581, + 0.00395783300336916, + 0.004336583006079309, + 0.004210874991258606, + 0.003945415999623947, + 0.004135916009545326, + 0.004324624998844229, + 0.0042455840011825785, + 0.003938000008929521, + 0.003936583001632243, + 0.004345875000581145, + 0.003939959002309479, + 0.00391879199014511, + 0.0039385839918395504, + 0.003953708990593441, + 0.0039290419954340905, + 0.004327167000155896, + 0.0039871669869171456, + 0.004350749994046055, + 0.0039606670034118, + 0.004457667004317045, + 0.003964916992117651, + 0.003970582998590544, + 0.003948749988921918, + 0.0039333750028163195, + 0.004357750003691763, + 0.004182958000455983, + 0.004007291994639672, + 0.004020584005047567, + 0.004178707997198217, + 0.004006333998404443, + 0.004006540999398567, + 0.004062541993334889, + 0.0044694590033032, + 0.004107624990865588, + 0.004132750007556751, + 0.004062042004079558, + 0.004069250004249625, + 0.004042915999889374, + 0.0041109169978881255, + 0.004135583993047476, + 0.004461250005988404, + 0.004075167002156377, + 0.004130625005927868, + 0.004091459006303921, + 0.004076917000929825, + 0.004089249996468425, + 0.004078291996847838, + 0.0041112920007435605, + 0.004433124995557591, + 0.0040572079888079315, + 0.0041257499979110435, + 0.004202957992674783, + 0.004288083000574261, + 0.004336250000051223, + 0.004088959001819603, + 0.004162791999988258, + 0.00439858301251661, + 0.004483416007133201, + 0.0046319999964907765, + 0.004614916004356928, + 0.004800959010026418 + ], + [ + 0.0037408329953905195, + 0.003965875002904795, + 0.0038871249998919666, + 0.003924832999473438, + 0.003943790987250395, + 0.003965708005125634, + 0.003947290999349207, + 0.00393074999738019, + 0.003929250000510365, + 0.003935792003176175, + 0.004090458998689428, + 0.004098625009646639, + 0.003963041992392391, + 0.004029582996736281, + 0.004015083002741449, + 0.004158750001806766, + 0.004015707992948592, + 0.003864167010760866, + 0.0040526669909013435, + 0.0046470420056721196, + 0.004135209004743956, + 0.0037604590033879504, + 0.0037801250000484288, + 0.003979083005106077, + 0.004255709005519748, + 0.0039735409955028445, + 0.0039661250048084185, + 0.004360791004728526, + 0.004441500001121312, + 0.004420665994985029, + 0.004295999999158084, + 0.004297416002373211, + 0.004271916011930443, + 0.0043575839954428375, + 0.00429337500827387, + 0.004332874988904223, + 0.004350667004473507, + 0.0043744160066125914, + 0.004307707989937626, + 0.004656999997678213, + 0.004804750002222136, + 0.004714209004305303, + 0.004214165994198993, + 0.00430512499588076, + 0.0043447920033941045, + 0.004175249996478669, + 0.00430604099528864, + 0.004409874993143603, + 0.004759749994263984, + 0.004169709005509503, + 0.004488040998694487, + 0.004485750003368594, + 0.0045519999985117465, + 0.004173875000560656, + 0.004886541995801963, + 0.0045449579920386896, + 0.004219583002850413, + 0.004774417000589892, + 0.004650249989936128, + 0.004331625008489937, + 0.004153041998506524, + 0.0046392079966608435, + 0.004556874991976656, + 0.004578666994348168 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "speedup_cached_vs_uncached_sdpa": 1.942191962843753, + "speedup_cached_vs_uncached_mha": 2.2475553093462857 + }, + { + "batch": 1, + "prompt_len": 1024, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_mha": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 7.59420900430996, + "p95": 9.481349808629602, + "min": 6.645625006058253, + "max": 9.502708009677008, + "iqr": 2.535167004680261, + "mean": 7.999841804848984, + "cv": 0.1712078051027626 + }, + "decode_ms_per_step": { + "count": 315, + "median": 2.276208993862383, + "p95": 2.8213294004672207, + "min": 1.7253340047318488, + "max": 7.7198329963721335, + "iqr": 0.38306201167870313, + "mean": 2.308872218949661, + "cv": 0.2268236149222591 + }, + "total_ms": { + "count": 5, + "median": 151.43387496937066, + "p95": 165.64615316165145, + "min": 144.63866701407824, + "max": 168.38267096318305, + "iqr": 6.561418864293955, + "mean": 153.45879159867764, + "cv": 0.05958698081080801 + }, + "tokens_per_second": { + "count": 5, + "median": 422.62670761706903, + "p95": 440.3910952153475, + "min": 380.08661837888076, + "max": 442.4819539699618, + "iqr": 18.323936249287897, + "mean": 418.1853328220808, + "cv": 0.05700666297832273 + }, + "median_step_ms_by_index": [ + 7.59420900430996, + 3.0419169925153255, + 2.5453330017626286, + 2.720625008805655, + 2.572707991930656, + 2.3454170004697517, + 2.4735000042710453, + 2.45441599690821, + 2.2714999940944836, + 2.366540997172706, + 2.398249998805113, + 2.327708003576845, + 2.332500007469207, + 2.0622090087272227, + 2.200500006438233, + 2.368125002249144, + 2.388916997006163, + 2.191417006542906, + 2.187042002333328, + 2.095083997119218, + 2.442750002956018, + 2.1276250045048073, + 2.344208987778984, + 2.199583002948202, + 2.3365830129478127, + 2.126000006683171, + 2.125625003827736, + 2.4108750076266006, + 2.2655829961877316, + 1.9062079954892397, + 2.069499998469837, + 2.2407910000765696, + 2.39670900919009, + 2.155042006052099, + 2.1330839954316616, + 2.350624999962747, + 2.3284159979084507, + 1.984749993425794, + 2.139833988621831, + 2.2955830063438043, + 2.4132910039043054, + 1.9690000044647604, + 2.1813749917782843, + 2.0837909978581592, + 2.3072500043781474, + 2.1298339997883886, + 2.3097079974832013, + 2.3508750018663704, + 2.344124994124286, + 2.051957999356091, + 2.2592919995076954, + 2.2889999963808805, + 1.9980419892817736, + 2.1162499906495214, + 2.129625005181879, + 2.326458998140879, + 2.330041999812238, + 2.0924590062350035, + 2.100541998515837, + 1.9975830073235556, + 2.1491670049726963, + 2.1242079965304583, + 2.052875002846122, + 2.356041004532017 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.006860749999759719, + 0.0029885419935453683, + 0.00254491699161008, + 0.0037444580084411427, + 0.0046531249972758815, + 0.0077198329963721335, + 0.0030405839934246615, + 0.0026745000068331137, + 0.0024581250036135316, + 0.0024122499889926985, + 0.002398249998805113, + 0.0023170420026872307, + 0.002305791000253521, + 0.0019804160110652447, + 0.0019324160093674436, + 0.0025901250046445057, + 0.0022587499988730997, + 0.0018674580060178414, + 0.0025118749908870086, + 0.0025267090095439926, + 0.00252754200482741, + 0.0018865839956561103, + 0.0023862909874878824, + 0.002199583002948202, + 0.0023365830129478127, + 0.0020111250050831586, + 0.0023543749994132668, + 0.0024135419953381643, + 0.0018924169999081641, + 0.0018679159984458238, + 0.0018217089964309707, + 0.0024516670091543347, + 0.001895084002171643, + 0.0018743330001598224, + 0.0018631669954629615, + 0.0018207090033683926, + 0.002605250003398396, + 0.0017580409912625328, + 0.0017300000035902485, + 0.0018920419970527291, + 0.002489666992914863, + 0.0019322080042911693, + 0.0019897919992217794, + 0.00197966699488461, + 0.0023950409959070385, + 0.0019677499949466437, + 0.001839291988289915, + 0.0017939160024980083, + 0.0023849579883972183, + 0.00186191700049676, + 0.001813707989640534, + 0.0023257919965544716, + 0.001855208000051789, + 0.0018026670004474, + 0.0019106660038232803, + 0.0018964999908348545, + 0.002439332994981669, + 0.001983459005714394, + 0.0019216249929741025, + 0.0019394169939914718, + 0.0018130000098608434, + 0.001907208003103733, + 0.0018922080053016543, + 0.0019250000041211024 + ], + [ + 0.00939591700443998, + 0.0030419169925153255, + 0.002328125003259629, + 0.002449416002491489, + 0.0028339579876046628, + 0.0023334589932346717, + 0.0021509170037461445, + 0.002217500004917383, + 0.0022714999940944836, + 0.0021827500022482127, + 0.0025059589970624074, + 0.002327708003576845, + 0.002332500007469207, + 0.0023312499979510903, + 0.002200500006438233, + 0.0021425829909276217, + 0.0020139999978709966, + 0.002191417006542906, + 0.002187042002333328, + 0.0020198749989503995, + 0.0023870830045780167, + 0.002068542002234608, + 0.001927291988977231, + 0.002481082992744632, + 0.002208041987614706, + 0.0021250419958960265, + 0.0023238750000018626, + 0.0024108750076266006, + 0.0022655829961877316, + 0.002329334005480632, + 0.002224875002866611, + 0.0022407910000765696, + 0.00239670900919009, + 0.0024571249959990382, + 0.0023899999941932037, + 0.002350624999962747, + 0.0022208339942153543, + 0.001984749993425794, + 0.002156583999749273, + 0.0022955830063438043, + 0.0024132910039043054, + 0.0024409159959759563, + 0.002522167007555254, + 0.0020837909978581592, + 0.0023072500043781474, + 0.0020780000049853697, + 0.0023097079974832013, + 0.0023508750018663704, + 0.002344124994124286, + 0.002421457989839837, + 0.0022592919995076954, + 0.0022179160005180165, + 0.0022546660038642585, + 0.002304124995134771, + 0.002373249997617677, + 0.002381208003498614, + 0.0024055419926298782, + 0.0023072920012054965, + 0.0023040000087348744, + 0.002276208993862383, + 0.0024375409993808717, + 0.0024427079915767536, + 0.002405790990451351, + 0.002356041004532017 + ], + [ + 0.009502708009677008, + 0.003554166993126273, + 0.0026135420048376545, + 0.0028169169963803142, + 0.002572707991930656, + 0.0022021670010872185, + 0.002504707998014055, + 0.0024544159969082102, + 0.002328374990611337, + 0.002366540997172706, + 0.002180541996494867, + 0.0022656249930150807, + 0.0022390840022126213, + 0.0020622090087272227, + 0.0046346669987542555, + 0.004600250002113171, + 0.0047186250012600794, + 0.0036693750007543713, + 0.0049906249914783984, + 0.00281408301088959, + 0.0022089580015745014, + 0.0021276250045048073, + 0.002344208987778984, + 0.002531959005864337, + 0.0023822909861337394, + 0.0021415829978650436, + 0.0020059999951627105, + 0.0023357089958153665, + 0.0020677079883171245, + 0.001901916999486275, + 0.002069499998469837, + 0.0020003340032417327, + 0.0019186660065315664, + 0.001968167009181343, + 0.0021330839954316616, + 0.002434207999613136, + 0.002353415999095887, + 0.0022918750037206337, + 0.002236166998045519, + 0.002336833000299521, + 0.002401499994448386, + 0.0021649999980581924, + 0.0021813749917782843, + 0.00226529200153891, + 0.002466834004735574, + 0.0024699170025996864, + 0.002611584000987932, + 0.002401499994448386, + 0.0023679580044699833, + 0.0023794580047251657, + 0.002525834002881311, + 0.0028948330000275746, + 0.0025395830016350374, + 0.0023567089956486598, + 0.0022196669888217, + 0.002326458998140879, + 0.002330041999812238, + 0.0023130000045057386, + 0.0023312080011237413, + 0.00259129201003816, + 0.0022457500017480925, + 0.0023756659938953817, + 0.0023369580012513325, + 0.0024037090042838827 + ], + [ + 0.006645625006058253, + 0.003060250004637055, + 0.0025453330017626286, + 0.002720625008805655, + 0.002156707996618934, + 0.0024932910018833354, + 0.0024735000042710453, + 0.002406583007541485, + 0.0022375410044332966, + 0.0023166669998317957, + 0.0024150000099325553, + 0.0024030829954426736, + 0.0026140830013900995, + 0.0024828750028973445, + 0.0023307090013986453, + 0.002368125002249144, + 0.002388916997006163, + 0.0023992080095922574, + 0.0019092919974355027, + 0.001810667003155686, + 0.002442750002956018, + 0.0023752499982947484, + 0.002411166002275422, + 0.0019859580061165616, + 0.0023286250070668757, + 0.0024957089917734265, + 0.0017253340047318488, + 0.00267633399926126, + 0.002512709004804492, + 0.0023332080018008128, + 0.0019533750019036233, + 0.002451374995871447, + 0.0024287919950438663, + 0.002155042006052099, + 0.0018982080073328689, + 0.002386375010246411, + 0.0023284159979084507, + 0.0019060000049648806, + 0.0018838339892681688, + 0.0023895409976830706, + 0.0024960000009741634, + 0.0019265000009909272, + 0.0024591249966761097, + 0.0024587910011177883, + 0.0019182499963790178, + 0.002597333994344808, + 0.0023485830024583265, + 0.0024154160055331886, + 0.0018494999967515469, + 0.001928333003888838, + 0.0023830830032238737, + 0.0022889999963808805, + 0.0018765830027405173, + 0.0017822079971665516, + 0.0019001670007128268, + 0.0025688339956104755, + 0.00198041700059548, + 0.0019370410009287298, + 0.0018977080035256222, + 0.0018265410035382956, + 0.00185500000952743, + 0.001880750001873821, + 0.001873707995400764, + 0.002443707999191247 + ], + [ + 0.00759420900430996, + 0.0028316250100033358, + 0.0026441660011187196, + 0.002704124999581836, + 0.0021209999977145344, + 0.0023454170004697517, + 0.001987374998861924, + 0.002590208998299204, + 0.0018434160010656342, + 0.002643083003931679, + 0.00206395800341852, + 0.002503875002730638, + 0.0023545000003650784, + 0.0019267919997218996, + 0.002044416993157938, + 0.0021455000096466392, + 0.002433583009405993, + 0.0018605419900268316, + 0.0021124999911990017, + 0.002095083997119218, + 0.0024784999986877665, + 0.0021497499983524904, + 0.00217500000144355, + 0.0021004170121159405, + 0.002369334004470147, + 0.002126000006683171, + 0.0021256250038277358, + 0.0020850000000791624, + 0.00244845800625626, + 0.0019062079954892397, + 0.0021580000029643998, + 0.002114583010552451, + 0.002569250005763024, + 0.0027237500034971163, + 0.002182999989599921, + 0.002188165992265567, + 0.002219458998297341, + 0.0021949169895378873, + 0.002139833988621831, + 0.002139417003490962, + 0.0020608329941751435, + 0.0019690000044647604, + 0.0021077080018585548, + 0.0020021669915877283, + 0.0019539579952834174, + 0.0021298339997883886, + 0.0020811659924220294, + 0.0019133330060867593, + 0.0019608750008046627, + 0.002051957999356091, + 0.0020114159997319803, + 0.0020558329997584224, + 0.0019980419892817736, + 0.0021162499906495214, + 0.002129625005181879, + 0.0021044159948360175, + 0.001981833003810607, + 0.0020924590062350035, + 0.002100541998515837, + 0.0019975830073235556, + 0.0021491670049726963, + 0.0021242079965304583, + 0.002052875002846122, + 0.002053542004432529 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + ] + } + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 6.990207999479026, + "p95": 8.471833399380557, + "min": 6.088709007599391, + "max": 8.513749999110587, + "iqr": 1.7142089927801862, + "mean": 7.297358402865939, + "cv": 0.14613549469680498 + }, + "decode_ms_per_step": { + "count": 315, + "median": 7.917875002021901, + "p95": 10.243916902982164, + "min": 6.094917000154965, + "max": 10.710832997574471, + "iqr": 2.272541998536326, + "mean": 8.002197483253282, + "cv": 0.1652564513638532 + }, + "total_ms": { + "count": 5, + "median": 512.6850009837653, + "p95": 619.4847002625465, + "min": 409.00683106156066, + "max": 636.2911240721587, + "iqr": 105.32196692656726, + "mean": 511.43579984782264, + "cv": 0.1746359465285142 + }, + "tokens_per_second": { + "count": 5, + "median": 124.83298687730992, + "p95": 153.82066406063393, + "min": 100.5828897791478, + "max": 156.47660415326217, + "iqr": 27.309250582124335, + "mean": 128.1954075215675, + "cv": 0.1722372004515618 + }, + "median_step_ms_by_index": [ + 6.990207999479026, + 7.6322499953676015, + 7.20162499055732, + 7.25550000788644, + 7.32333300402388, + 7.468957992387004, + 7.667041994864121, + 7.259707999764942, + 7.71120899298694, + 7.459250002284534, + 8.480041986331344, + 7.620124990353361, + 8.191917004296556, + 7.4067499954253435, + 8.120000013150275, + 8.758083000429906, + 8.854207990225405, + 9.167833995888941, + 8.355083002243191, + 7.282167003722861, + 8.43400000303518, + 7.439459004672244, + 8.673459000419825, + 7.902999990619719, + 8.015041996259242, + 7.645083009265363, + 8.724333994905464, + 7.504749999498017, + 8.090958988759667, + 7.78579201141838, + 8.01883298845496, + 7.814583004801534, + 7.93570899986662, + 7.944458004203625, + 8.05716699687764, + 8.135249998304062, + 7.8979160025483, + 8.110791997751221, + 8.143291997839697, + 8.548833997338079, + 7.481624998035841, + 8.362541993847117, + 7.622708013514057, + 8.450207998976111, + 7.77191600354854, + 8.545665987185203, + 7.43829199927859, + 7.890583001426421, + 7.827166002243757, + 7.99699999333825, + 7.923375000245869, + 7.911292006610893, + 7.464166003046557, + 8.110374998068437, + 7.870917004765943, + 7.888667008955963, + 8.58337500540074, + 7.83100000990089, + 8.5045000014361, + 8.364459004951641, + 7.952166008180939, + 7.464249996701255, + 8.117666002362967, + 7.3794580093817785 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.006088709007599391, + 0.006338915991364047, + 0.006360875006066635, + 0.00631900000735186, + 0.006400958998710848, + 0.006300832988927141, + 0.006332249991828576, + 0.0064062919991556555, + 0.006163708007079549, + 0.0063077910017455, + 0.0065243330027442425, + 0.006264333002036437, + 0.006416666990844533, + 0.006305375005467795, + 0.006217708010808565, + 0.006341582993627526, + 0.006399958991096355, + 0.00632966699777171, + 0.006291083001997322, + 0.00637529201048892, + 0.006334249992505647, + 0.006362583008012734, + 0.00632070901338011, + 0.006414000003132969, + 0.006109916997957043, + 0.0063119999977061525, + 0.006881417008116841, + 0.006406790998880751, + 0.006409417008399032, + 0.006257582994294353, + 0.006325000009383075, + 0.006362625004840083, + 0.00635470800625626, + 0.006463666999479756, + 0.006518416004837491, + 0.006536625005537644, + 0.0065374999976484105, + 0.00637862499570474, + 0.006481332995463163, + 0.006457458002842031, + 0.00645095799700357, + 0.006463540994445793, + 0.006547709010192193, + 0.006513457992696203, + 0.006437667005229741, + 0.006467750004958361, + 0.006681500002741814, + 0.006452624991652556, + 0.006338624996715225, + 0.006317459003184922, + 0.006361666994052939, + 0.006459790994995274, + 0.0064437920082127675, + 0.006337792001431808, + 0.006540082991705276, + 0.0063411250011995435, + 0.006380082995747216, + 0.006459875003201887, + 0.006546708013047464, + 0.006519625007058494, + 0.006374290998792276, + 0.00628237500495743, + 0.006094917000154965, + 0.006483457997092046 + ], + [ + 0.008304167000460438, + 0.01027370899100788, + 0.008944249988417141, + 0.009317500007455237, + 0.00918783301312942, + 0.009274166994146071, + 0.009353166999062523, + 0.009111082996241748, + 0.009864249994279817, + 0.009093333006603643, + 0.009900500008370727, + 0.009274874988477677, + 0.009827000001678243, + 0.009485209011472762, + 0.00940541599993594, + 0.010442917002364993, + 0.009578291996149346, + 0.00981420800962951, + 0.009271750008338131, + 0.009902500009047799, + 0.00969812499533873, + 0.009993250001571141, + 0.01005966600496322, + 0.009510999996564351, + 0.009997916989959776, + 0.010200125005212612, + 0.010214916008408181, + 0.010192499990807846, + 0.009643625002354383, + 0.010081040993100032, + 0.009741250003571622, + 0.009750958997756243, + 0.010112957999808714, + 0.010042500012787059, + 0.010238041999400593, + 0.010531791995163076, + 0.010321624999050982, + 0.010084957990329713, + 0.009973000007448718, + 0.010272291998262517, + 0.010051500008557923, + 0.010315500010619871, + 0.010312625003280118, + 0.009987834011553787, + 0.010290542006259784, + 0.010410458009573631, + 0.010109124996233732, + 0.010390541006927378, + 0.010237834008876234, + 0.01035333400068339, + 0.010710832997574471, + 0.01016150000214111, + 0.01037929100857582, + 0.010186542000155896, + 0.009972874991944991, + 0.010242917007417418, + 0.010137665987713262, + 0.010238917006063275, + 0.010630791002768092, + 0.010086750000482425, + 0.010282707997248508, + 0.010102582993567921, + 0.010164541003177874, + 0.010246249992633238 + ], + [ + 0.008513749999110587, + 0.008699917001649737, + 0.008915667000110261, + 0.00871491600992158, + 0.008862167000188492, + 0.008846167009323835, + 0.009009000001242384, + 0.00930508399324026, + 0.008696040997165255, + 0.00878670800011605, + 0.008852124999975786, + 0.008679457998368889, + 0.009000041987746954, + 0.008902166999178007, + 0.008827625002595596, + 0.008758083000429906, + 0.008854207990225405, + 0.009167833995888941, + 0.009220250009093434, + 0.009188790994812734, + 0.009008167005958967, + 0.009108291007578373, + 0.00869195799168665, + 0.009247083988157101, + 0.008601082998211496, + 0.009629874999518506, + 0.008724333994905464, + 0.009012084003188647, + 0.008898749991203658, + 0.008658417005790398, + 0.009043124999152496, + 0.00881850000587292, + 0.009090417006518692, + 0.008392750009079464, + 0.009068874991498888, + 0.008551374994567595, + 0.008829458005493507, + 0.008623834000900388, + 0.008523208001861349, + 0.008548833997338079, + 0.008707875007530674, + 0.008393000010983087, + 0.008432500006165355, + 0.008513708991813473, + 0.008557332999771461, + 0.008596958999987692, + 0.008145790998241864, + 0.007890583001426421, + 0.00813795899739489, + 0.00799699999333825, + 0.008082832995569333, + 0.007911292006610893, + 0.008637709004688077, + 0.008110374998068437, + 0.00812204199610278, + 0.007888667008955963, + 0.00858337500540074, + 0.007716291991528124, + 0.008549666992621496, + 0.008466583007248119, + 0.008555417007301003, + 0.0074642499967012554, + 0.008547917008399963, + 0.0073794580093817785 + ], + [ + 0.006589958007680252, + 0.007119624991901219, + 0.006829459001892246, + 0.0066470000019762665, + 0.0071027500089257956, + 0.006732084002578631, + 0.007067541009746492, + 0.007259707999764942, + 0.0070799170061945915, + 0.0070433329965453595, + 0.00662033399567008, + 0.006582417001482099, + 0.00725024999701418, + 0.007071874992107041, + 0.007034042006125674, + 0.007106458011548966, + 0.006872666999697685, + 0.0067280830116942525, + 0.006932332995347679, + 0.007282167003722861, + 0.0069488750013988465, + 0.007043833000352606, + 0.007050333006191067, + 0.006758124989573844, + 0.00707041600253433, + 0.006905834001372568, + 0.007218832994112745, + 0.006731040994054638, + 0.006807541998568922, + 0.006386292006936856, + 0.006837375010945834, + 0.006877665990032256, + 0.006774416004191153, + 0.006924542001797818, + 0.006430333000025712, + 0.006904458001372404, + 0.007154999999329448, + 0.0069844999961787835, + 0.006969292007852346, + 0.007007709005847573, + 0.006977458004257642, + 0.007093709005857818, + 0.006903417001012713, + 0.00693583300744649, + 0.006926040994585492, + 0.006683792002149858, + 0.006359791004797444, + 0.006501374999061227, + 0.006947957997908816, + 0.006521208997583017, + 0.006613625009777024, + 0.006725707993609831, + 0.006493708002381027, + 0.0066978749964619055, + 0.006371625000610948, + 0.006651416013482958, + 0.009231207994162105, + 0.00783100000990089, + 0.0085045000014361, + 0.007073000000673346, + 0.007604915997944772, + 0.007111084007192403, + 0.008117666002362967, + 0.00732070799858775 + ], + [ + 0.006990207999479026, + 0.0076322499953676015, + 0.00720162499055732, + 0.0072555000078864396, + 0.00732333300402388, + 0.007468957992387004, + 0.007667041994864121, + 0.007254707990796305, + 0.00771120899298694, + 0.007459250002284534, + 0.008480041986331344, + 0.007620124990353361, + 0.008191917004296556, + 0.0074067499954253435, + 0.008120000013150275, + 0.009167625001282431, + 0.009141833012108691, + 0.009428167002624832, + 0.008355083002243191, + 0.007102250005118549, + 0.00843400000303518, + 0.007439459004672244, + 0.008673459000419825, + 0.007902999990619719, + 0.008015041996259242, + 0.007645083009265363, + 0.008793000000878237, + 0.007504749999498017, + 0.008090958988759667, + 0.00778579201141838, + 0.00801883298845496, + 0.007814583004801534, + 0.00793570899986662, + 0.007944458004203625, + 0.00805716699687764, + 0.008135249998304062, + 0.0078979160025483, + 0.008110791997751221, + 0.008143291997839697, + 0.00857904199801851, + 0.007481624998035841, + 0.008362541993847117, + 0.007622708013514057, + 0.008450207998976111, + 0.00777191600354854, + 0.008545665987185203, + 0.00743829199927859, + 0.00856299999577459, + 0.007827166002243757, + 0.008553499996196479, + 0.00792337500024587, + 0.008939292005379684, + 0.007464166003046557, + 0.00840950000565499, + 0.007870917004765943, + 0.008538207999663427, + 0.0077889579988550395, + 0.008386749992496334, + 0.007917875002021901, + 0.008364459004951641, + 0.007952166008180939, + 0.00839849999465514, + 0.007905167003627867, + 0.008309583994559944 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_mha": { + "first_token_ms": { + "count": 5, + "median": 8.652040996821597, + "p95": 9.985850399243645, + "min": 7.965500000864267, + "max": 10.1239590003388, + "iqr": 1.4399159990716726, + "mean": 8.833683197735809, + "cv": 0.10615065872374824 + }, + "decode_ms_per_step": { + "count": 315, + "median": 9.270374997868203, + "p95": 11.68218730163062, + "min": 7.517874997574836, + "max": 13.691915999515913, + "iqr": 1.3393545013968833, + "mean": 9.351786770922546, + "cv": 0.11726083724820797 + }, + "total_ms": { + "count": 5, + "median": 596.1065799201606, + "p95": 664.4356519973371, + "min": 538.2239549071528, + "max": 671.8152109970106, + "iqr": 85.99932899232954, + "mean": 597.9962497658562, + "cv": 0.09458712036509956 + }, + "tokens_per_second": { + "count": 5, + "median": 107.36335104466022, + "p95": 118.44628477680483, + "min": 95.2642913592571, + "max": 118.9096089397218, + "iqr": 15.792477086490635, + "mean": 107.78615010148447, + "cv": 0.09359222549690586 + }, + "median_step_ms_by_index": [ + 8.652040996821597, + 8.555416003218852, + 8.556000000680797, + 8.406583001487888, + 8.628083000076003, + 8.135916999890469, + 8.717500008060597, + 8.17808300780598, + 8.587458010879345, + 8.314332997542806, + 8.645458001410589, + 8.468332991469651, + 8.71566699061077, + 8.856541986460797, + 8.716875003301539, + 8.762457990087569, + 9.220833002473228, + 9.261958999559283, + 8.941958993091248, + 9.45929199224338, + 9.033624999574386, + 9.000667007057928, + 9.191208009724505, + 9.367999999085441, + 9.421958005987108, + 9.189999997033738, + 9.449417004361749, + 9.134624997386709, + 9.479792002821341, + 9.085500001674518, + 9.385709010530263, + 8.911166005418636, + 8.969624992460012, + 9.150375000899658, + 9.488333002082072, + 9.782833003555425, + 9.300666992203332, + 9.363334000227042, + 9.199541993439198, + 9.78245800069999, + 9.898124990286306, + 9.623082994949073, + 9.944042001734488, + 9.393540996825323, + 9.669207996921614, + 9.007207991089672, + 9.634542002459057, + 9.694874999695458, + 9.530000010272488, + 9.953959000995383, + 9.5314589998452, + 9.902541991323233, + 9.270374997868203, + 9.691583007224835, + 9.612708003260195, + 9.777375002158806, + 9.369166000396945, + 9.800957996048965, + 9.421708004083484, + 9.11554200865794, + 9.742875001393259, + 9.33324999641627, + 9.675084002083167, + 9.605166997062042 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.007993499995791353, + 0.008026583003811538, + 0.008177417010301724, + 0.00796262500807643, + 0.008253125008195639, + 0.008022833004361019, + 0.008201541000744328, + 0.008049582989769988, + 0.008157417003531009, + 0.00797962500655558, + 0.008121624996419996, + 0.008226999998441897, + 0.008144999999785796, + 0.008569832993089221, + 0.008149834000505507, + 0.008389166003325954, + 0.007614124988322146, + 0.008045792012126185, + 0.008156165989930741, + 0.008644833011203445, + 0.007750832999590784, + 0.008646499991300516, + 0.008568917008233257, + 0.008130124988383614, + 0.008717292002984323, + 0.008148749999236315, + 0.008031666991882958, + 0.008444709004834294, + 0.008170750006684102, + 0.00839849999465514, + 0.008254083993961103, + 0.00867791699420195, + 0.008205999998608604, + 0.008538124995538965, + 0.008330249998834915, + 0.008710291993338615, + 0.008509874998708256, + 0.00868541699310299, + 0.008215458001359366, + 0.008528499995009042, + 0.008486042002914473, + 0.008571457990910858, + 0.009002541992231272, + 0.008943207998527214, + 0.009059208008693531, + 0.008984042011434212, + 0.009465374998399056, + 0.009182666995911859, + 0.009508083006949164, + 0.009161750000203028, + 0.008915957994759083, + 0.008863459006533958, + 0.008970250011770986, + 0.009319874996435829, + 0.009072333996300586, + 0.009643875004258007, + 0.009024165992741473, + 0.009560209000483155, + 0.009024375001899898, + 0.00911554200865794, + 0.008714084004168399, + 0.00927270900865551, + 0.008900125001673587, + 0.009605166997062042 + ], + [ + 0.008652040996821597, + 0.008393500000238419, + 0.009005542000522837, + 0.008458166994387284, + 0.009512625008937903, + 0.008271250000689179, + 0.00951716600684449, + 0.0084921670058975, + 0.009742666996316984, + 0.008767457999056205, + 0.009245125009329058, + 0.009480833992711268, + 0.009653916989918798, + 0.009510250005405396, + 0.009561917002429254, + 0.009148875004029833, + 0.009220833002473228, + 0.009261958999559283, + 0.00986508300411515, + 0.009709042002214119, + 0.009676166999270208, + 0.009000667007057928, + 0.009506542002782226, + 0.009533916992950253, + 0.009421958005987108, + 0.00964133300294634, + 0.009770749995368533, + 0.011136333996546455, + 0.013691915999515913, + 0.012855875000241213, + 0.011421999995945953, + 0.010666500005754642, + 0.009857666998868808, + 0.010167833999730647, + 0.010151750000659376, + 0.010295083993696608, + 0.010133832998690195, + 0.010299332992872223, + 0.010492875007912517, + 0.010792000000947155, + 0.01090824999846518, + 0.01098525000270456, + 0.011064082995289937, + 0.010438375000376254, + 0.011252041003899649, + 0.011069917003624141, + 0.010905916002229787, + 0.01171016700391192, + 0.011204291993635707, + 0.011520959000336006, + 0.012249250008608215, + 0.011330749999615364, + 0.011680374998832121, + 0.011410874998546205, + 0.011642374985967763, + 0.011796457998570986, + 0.01168641600816045, + 0.012117791993659921, + 0.01242308400105685, + 0.012030416997731663, + 0.012207624997245148, + 0.013166374992579222, + 0.01280883299477864, + 0.012220582997542806 + ], + [ + 0.0101239590003388, + 0.009517083002720028, + 0.009705707998364232, + 0.009767374998773448, + 0.009161959009361453, + 0.01011758299137, + 0.009734999999636784, + 0.009551791998092085, + 0.009635750000597909, + 0.009761583001818508, + 0.00995274999877438, + 0.009826334004173987, + 0.010036249994300306, + 0.009437083004740998, + 0.009858625009655952, + 0.009527125002932735, + 0.009795833000680432, + 0.009821458996157162, + 0.009834541997406632, + 0.00945929199224338, + 0.009520417006569915, + 0.00956058299925644, + 0.009425625001313165, + 0.009887332998914644, + 0.010410332994069904, + 0.00993216599454172, + 0.009449417004361749, + 0.009265000000596046, + 0.009479792002821341, + 0.009431667000171728, + 0.009990082995500416, + 0.009459957989747636, + 0.009976415996789001, + 0.010045083006843925, + 0.010424499996588565, + 0.010134291995200329, + 0.0096835410076892, + 0.009363334000227042, + 0.009839542006375268, + 0.009822208012337796, + 0.010306124997441657, + 0.010427917004562914, + 0.010094541998114437, + 0.010433165996801108, + 0.010305333998985589, + 0.01029312500031665, + 0.010820665993378498, + 0.010897165993810631, + 0.01194266699894797, + 0.01196416700258851, + 0.01169520799885504, + 0.010018624991062097, + 0.010431542003061622, + 0.009967958001652732, + 0.009730249992571771, + 0.010038374995929189, + 0.009369166000396945, + 0.009800957996048965, + 0.009421708004083484, + 0.008962917010649107, + 0.009742875001393259, + 0.00933324999641627, + 0.009675084002083167, + 0.00951625000743661 + ], + [ + 0.009433415994863026, + 0.00893158400140237, + 0.008556000000680797, + 0.008406583001487888, + 0.008628083000076003, + 0.008135916999890469, + 0.008717500008060597, + 0.007626292004715651, + 0.008587458010879345, + 0.0075242500024614856, + 0.008645458001410589, + 0.007519666993175633, + 0.008412290990236215, + 0.00803141700453125, + 0.008283874994958751, + 0.007517874997574836, + 0.00827716599451378, + 0.008082291999016888, + 0.008126499989884906, + 0.007972583989612758, + 0.00855737499659881, + 0.008038082989514805, + 0.008072333002928644, + 0.008017665997613221, + 0.00816483300877735, + 0.007925792000605725, + 0.00818295800127089, + 0.008055041995248757, + 0.008287124990602024, + 0.00786258299194742, + 0.008237875008489937, + 0.008003750001080334, + 0.008105624991003424, + 0.008301249996293336, + 0.008331290999194607, + 0.008285957999760285, + 0.008381084000575356, + 0.009129625002969988, + 0.008479916999931447, + 0.008081874999334104, + 0.00836374999198597, + 0.008129249996272847, + 0.007905666003352962, + 0.008326040988322347, + 0.007962000003317371, + 0.008823958996799774, + 0.008429041990893893, + 0.00832504199934192, + 0.008519333001459017, + 0.008426000000326894, + 0.008530874998541549, + 0.008600957997259684, + 0.008530708000762388, + 0.008606707997387275, + 0.008597374995588325, + 0.009127041994361207, + 0.00915979100682307, + 0.009172417005174793, + 0.00938412499090191, + 0.009047374987858348, + 0.009395832996233366, + 0.00868954200996086, + 0.009327249994385056, + 0.00892562500666827 + ], + [ + 0.007965500000864267, + 0.008555416003218852, + 0.008263416006229818, + 0.008127167006023228, + 0.008570332996896468, + 0.008042499990551732, + 0.0085618750017602, + 0.00817808300780598, + 0.00852691700856667, + 0.008314332997542806, + 0.008508625003742054, + 0.008468332991469651, + 0.008715666990610771, + 0.008856541986460797, + 0.008716875003301539, + 0.008762457990087569, + 0.009224624998751096, + 0.009525499990559183, + 0.008941958993091248, + 0.009994124993681908, + 0.009033624999574386, + 0.009533834003377706, + 0.009191208009724505, + 0.009367999999085441, + 0.009450040990486741, + 0.009189999997033738, + 0.009713625011499971, + 0.009134624997386709, + 0.00972820799506735, + 0.009085500001674518, + 0.009385709010530263, + 0.008911166005418636, + 0.008969624992460012, + 0.009150375000899658, + 0.009488333002082072, + 0.009782833003555425, + 0.009300666992203332, + 0.010134458003449254, + 0.009199541993439198, + 0.00978245800069999, + 0.009898124990286306, + 0.009623082994949073, + 0.009944042001734488, + 0.009393540996825323, + 0.009669207996921614, + 0.009007207991089672, + 0.009634542002459057, + 0.009694874999695458, + 0.009530000010272488, + 0.009953959000995383, + 0.0095314589998452, + 0.009902541991323233, + 0.009270374997868203, + 0.009691583007224835, + 0.009612708003260195, + 0.009777375002158806, + 0.010354665995691903, + 0.009927416002028622, + 0.010187749998294748, + 0.010580666988971643, + 0.010346333001507446, + 0.010064541987958364, + 0.010145416992600076, + 0.010011082995333709 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "speedup_cached_vs_uncached_sdpa": 3.385537093912852, + "speedup_cached_vs_uncached_mha": 3.936415019695761 + }, + { + "batch": 8, + "prompt_len": 1, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": false, + "uncached_mha": false + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 1.8345830030739307, + "p95": 1.8497085984563455, + "min": 1.6935000021476299, + "max": 1.8500419973861426, + "iqr": 0.025083005311898887, + "mean": 1.8099584005540237, + "cv": 0.03647041387553342 + }, + "decode_ms_per_step": { + "count": 315, + "median": 2.114874994731508, + "p95": 2.5458831019932404, + "min": 1.748374997987412, + "max": 5.511750001460314, + "iqr": 0.3864999962388538, + "mean": 2.2109660128730217, + "cv": 0.16817961788451874 + }, + "total_ms": { + "count": 5, + "median": 139.32583802670706, + "p95": 151.84115120209754, + "min": 134.1394570627017, + "max": 154.75721000984777, + "iqr": 3.0722509836778045, + "mean": 141.10081721155439, + "cv": 0.0565785456992812 + }, + "tokens_per_second": { + "count": 5, + "median": 3674.8388328506294, + "p95": 3800.413185675259, + "min": 3308.408053927953, + "max": 3816.9231575215967, + "iqr": 81.84608685111107, + "mean": 3637.4141108057765, + "cv": 0.05350341987161359 + }, + "median_step_ms_by_index": [ + 1.8345830030739307, + 2.3827090044505894, + 2.317082995432429, + 2.3494589986512437, + 2.39779100229498, + 2.3350830015260726, + 2.1843330032425, + 2.386624997598119, + 2.0161249994998798, + 2.358084006118588, + 2.2899169998709112, + 1.9802919996436685, + 2.2600420052185655, + 2.0238330034771934, + 2.0050000021001324, + 2.3689999943599105, + 2.041583997197449, + 2.2886670049047098, + 1.984542002901435, + 2.1585000067716464, + 2.286375005496666, + 2.0092089980607852, + 2.3156670067692176, + 2.181832998758182, + 2.391375004663132, + 2.1804170100949705, + 2.3631249932805076, + 2.3747499944875017, + 2.3619159910595044, + 2.3609579948242754, + 2.3351249983534217, + 2.057665988104418, + 2.343792002648115, + 2.319917010026984, + 2.3971659975359216, + 1.97108399879653, + 2.015958001720719, + 2.0132089994149283, + 2.0148339972365648, + 2.427540995995514, + 2.0435839978745207, + 2.492290994268842, + 2.133499991032295, + 2.015459001995623, + 2.278958010720089, + 2.186084006098099, + 2.0283339981688187, + 2.1135840070201084, + 1.9617910002125427, + 2.13741599873174, + 2.027374997851439, + 2.081667000311427, + 2.0115000079385936, + 1.9185420096619055, + 2.0272910041967407, + 2.0389159908518195, + 2.058084006421268, + 2.277499996125698, + 2.140707991202362, + 2.0495829958235845, + 2.0532499911496416, + 2.0176669931970537, + 2.0014170004287735, + 2.0796250028070062 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0016935000021476299, + 0.0023827090044505894, + 0.0024725420080358163, + 0.00237800000468269, + 0.0027032079960918054, + 0.002431500004604459, + 0.0023455420014215633, + 0.0023190840001916513, + 0.0019795000116573647, + 0.0024177500017685816, + 0.0024223329965025187, + 0.0019802919996436685, + 0.0022600420052185655, + 0.0019331249932292849, + 0.0018893749947892502, + 0.0023689999943599105, + 0.0023344579967670143, + 0.0018880830029956996, + 0.001950833000591956, + 0.001962833004654385, + 0.0023212910018628463, + 0.0019231249898439273, + 0.0020021660020574927, + 0.002181832998758182, + 0.001983166002901271, + 0.002552125006332062, + 0.0023631249932805076, + 0.002313832999789156, + 0.001978583008167334, + 0.0024120829912135378, + 0.002297708997502923, + 0.002057665988104418, + 0.002343792002648115, + 0.002039166007307358, + 0.0020709169912151992, + 0.001933916995767504, + 0.0019763339951168746, + 0.0019937920005759224, + 0.00198154199460987, + 0.002427540995995514, + 0.0020653329993365332, + 0.0025288329925388098, + 0.00194283299788367, + 0.002060166996670887, + 0.0023464580008294433, + 0.001988125004572794, + 0.0024849580076988786, + 0.0024466670001856983, + 0.0019113750022370368, + 0.0020461250096559525, + 0.002027374997851439, + 0.00193500000750646, + 0.0018972090038005263, + 0.0018589999963296577, + 0.001922416005982086, + 0.0020389159908518195, + 0.0019820419984171167, + 0.002078499994240701, + 0.0020947500015608966, + 0.0020054170017829165, + 0.002094208000926301, + 0.0020176669931970537, + 0.0020014170004287735, + 0.002062458996078931 + ], + [ + 0.0018345830030739307, + 0.002343375002965331, + 0.0021082079911138862, + 0.0023781659983797, + 0.0023958749952726066, + 0.0023350830015260726, + 0.0019967090047430247, + 0.002386624997598119, + 0.0020161249994998798, + 0.002299582993146032, + 0.001748374997987412, + 0.0021342090039979666, + 0.0023004170070635155, + 0.0020736660080729052, + 0.002114874994731508, + 0.002477250003721565, + 0.0022734170051990077, + 0.00196341700211633, + 0.002493792009772733, + 0.002508334000594914, + 0.0019827499927487224, + 0.0020092089980607852, + 0.002370792004512623, + 0.0019743340089917183, + 0.0024037089897319674, + 0.0019862499902956188, + 0.0023914590128697455, + 0.002035999990766868, + 0.0023880840017227456, + 0.0019960000063292682, + 0.0023351249983534217, + 0.001960167006473057, + 0.0023968750028871, + 0.002035999990766868, + 0.0023971659975359216, + 0.0020772499992744997, + 0.002368500005104579, + 0.0020225829939590767, + 0.0023937920050229877, + 0.002008209004998207, + 0.0024109170044539496, + 0.0020161249994998798, + 0.0023870829900261015, + 0.001976583997020498, + 0.002103790990076959, + 0.0024240409984486178, + 0.0020283339981688187, + 0.0021135840070201084, + 0.0023790830018697307, + 0.00213741599873174, + 0.002352541996515356, + 0.001972542013390921, + 0.0020115000079385936, + 0.0019003750057891011, + 0.002048209003987722, + 0.0025432080001337454, + 0.002058084006421268, + 0.0024474579986417666, + 0.002140707991202362, + 0.0020495829958235845, + 0.0020532499911496416, + 0.002009292002185248, + 0.0019661660044221207, + 0.0020796250028070062 + ], + [ + 0.001848375002737157, + 0.0022539579949807376, + 0.0020217080018483102, + 0.0023494589986512437, + 0.002399707998847589, + 0.0023709170054644346, + 0.002004374997341074, + 0.0024577500007580966, + 0.00244679200113751, + 0.0024089579965220764, + 0.002530624988139607, + 0.0023701669997535646, + 0.0018824170110747218, + 0.0020238330034771934, + 0.0020050000021001324, + 0.002381250000325963, + 0.002041583997197449, + 0.0023527500015916303, + 0.001984542002901435, + 0.0023430829896824434, + 0.001979583001229912, + 0.002382791994023137, + 0.002029667011811398, + 0.002366582994000055, + 0.0019822079921141267, + 0.0021804170100949705, + 0.002021292006247677, + 0.0023747499944875017, + 0.002140124997822568, + 0.0023609579948242754, + 0.002397874995949678, + 0.0019501249917084351, + 0.0020057500078110024, + 0.00238491699565202, + 0.002500167000107467, + 0.001996541002881713, + 0.002015958001720719, + 0.001966666997759603, + 0.0020148339972365648, + 0.0024878340045688674, + 0.0020262079924577847, + 0.002565040995250456, + 0.002133499991032295, + 0.002473958011250943, + 0.0024577500007580966, + 0.0024708749988349155, + 0.0018924169999081641, + 0.0018755420023808256, + 0.0019617910002125427, + 0.00201500000548549, + 0.001983957990887575, + 0.0021085830085212365, + 0.0021810000034747645, + 0.0021252080041449517, + 0.0021204999939072877, + 0.0021754580084234476, + 0.0021446250029839575, + 0.0025102910003624856, + 0.0026343329955125228, + 0.0021372079936554655, + 0.002096374999382533, + 0.002121208992321044, + 0.0020195419929223135, + 0.001930250000441447 + ], + [ + 0.0018500419973861426, + 0.0023829589918022975, + 0.0024213340075220913, + 0.0022846669889986515, + 0.00239779100229498, + 0.0020351249986561015, + 0.002437999995891005, + 0.002425790997222066, + 0.002346375011256896, + 0.0020619999995687976, + 0.0022899169998709112, + 0.001971666992176324, + 0.0023167909967014566, + 0.002351582996197976, + 0.0019955839961767197, + 0.0023486249992856756, + 0.0019368750072317198, + 0.002430084001389332, + 0.001968833996215835, + 0.0021585000067716464, + 0.005511750001460314, + 0.004705832994659431, + 0.004528707999270409, + 0.0028677910013357177, + 0.004648500005714595, + 0.002612541997223161, + 0.0023737919982522726, + 0.0024992500111693516, + 0.0023619159910595044, + 0.002683166996575892, + 0.002263083995785564, + 0.0025577910128049552, + 0.0024537919962313026, + 0.002613292002934031, + 0.0025574169994797558, + 0.00197108399879653, + 0.0021652080031344667, + 0.0020132089994149283, + 0.0019125000108033419, + 0.0026524159911787137, + 0.0020435839978745207, + 0.002492290994268842, + 0.002511333004804328, + 0.002015459001995623, + 0.002278958010720089, + 0.002186084006098099, + 0.0025038749881787226, + 0.0024929579958552495, + 0.0023574580118292943, + 0.0021552919934038073, + 0.0025080420018639416, + 0.002081667000311427, + 0.0021238750050542876, + 0.002449874999001622, + 0.0020272910041967407, + 0.001885332996607758, + 0.0019729169871425256, + 0.002277499996125698, + 0.0019968330016126856, + 0.002006041002459824, + 0.0019715420057764277, + 0.001987625000765547, + 0.0019758330017793924, + 0.0020899580122204497 + ], + [ + 0.0018232919974252582, + 0.0024341249954886734, + 0.002317082995432429, + 0.001901667012134567, + 0.001915957996970974, + 0.0022927910031285137, + 0.0021843330032425, + 0.001916957990033552, + 0.0019604580011218786, + 0.002358084006118588, + 0.0019785420008702204, + 0.0018897500121966004, + 0.0018647920078365132, + 0.0019230000034440309, + 0.0024462909932481125, + 0.002343000000109896, + 0.001946832999237813, + 0.0022886670049047098, + 0.002461583004333079, + 0.0019610829913290218, + 0.002286375005496666, + 0.0019987080013379455, + 0.0023156670067692176, + 0.00197370900423266, + 0.002391375004663132, + 0.0019136250048177317, + 0.0019955419993493706, + 0.002438750001601875, + 0.002401874997303821, + 0.0019708750041900203, + 0.0023941250110510737, + 0.002308166993316263, + 0.001966083000297658, + 0.002319917010026984, + 0.001950625010067597, + 0.0018041669973172247, + 0.0019288749899715185, + 0.002052957992418669, + 0.002048584006843157, + 0.0019790829974226654, + 0.0018469159986125305, + 0.002126916006091051, + 0.0020440829975996166, + 0.00199612500728108, + 0.0021437079994939268, + 0.0019227500015404075, + 0.002026667003519833, + 0.0020269170054234564, + 0.0018910829967353493, + 0.0021496249974006787, + 0.0018368330056546256, + 0.002090415990096517, + 0.0019935410091420636, + 0.0019185420096619055, + 0.0019164999976055697, + 0.0019823750044452026, + 0.0020854589965892956, + 0.0020835420000366867, + 0.0025534169981256127, + 0.0024377499939873815, + 0.001981958994292654, + 0.002099958001053892, + 0.0021853749931324273, + 0.00215162499807775 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + ] + } + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 1.6812499961815774, + "p95": 2.1738253941293806, + "min": 1.5892500086920336, + "max": 2.264583992655389, + "iqr": 0.1660819980315864, + "mean": 1.7981167999096215, + "cv": 0.15194092463568018 + }, + "decode_ms_per_step": { + "count": 315, + "median": 2.4670410057296976, + "p95": 3.322287497576326, + "min": 1.5719999937573448, + "max": 3.558458003681153, + "iqr": 0.8693545023561455, + "mean": 2.500806345848278, + "cv": 0.2024332311570568 + }, + "total_ms": { + "count": 5, + "median": 157.14695902715903, + "p95": 165.0233175372705, + "min": 156.257707028999, + "max": 166.08891892246902, + "iqr": 4.270826029824093, + "mean": 159.34891658835113, + "cv": 0.026252167033724877 + }, + "tokens_per_second": { + "count": 5, + "median": 3258.0967724072425, + "p95": 3275.6652552792675, + "min": 3082.686089606036, + "max": 3276.6383798591182, + "iqr": 86.91896606309274, + "mean": 3214.809557945807, + "cv": 0.025695619680888827 + }, + "median_step_ms_by_index": [ + 1.6812499961815774, + 1.8132080003852025, + 1.808207991416566, + 1.836542011005804, + 1.9558750063879415, + 2.00133299222216, + 2.0933749910909683, + 1.9473749998724088, + 1.719834006507881, + 1.6772079980000854, + 1.7570000054547563, + 1.8289169966010377, + 1.8962920003104955, + 1.8828750035027042, + 1.8393339996691793, + 1.8863340083044022, + 1.9664169958559796, + 1.9702500139828771, + 1.972209007362835, + 1.9873749988619238, + 2.0403340022312477, + 2.178042006562464, + 2.2038340102881193, + 2.1190840052440763, + 2.1804579882882535, + 2.191209001466632, + 2.175540997995995, + 2.1964579937048256, + 2.2907079983269796, + 2.3419579956680536, + 2.3309590033022687, + 2.34529199951794, + 2.4670410057296976, + 2.4415000079898164, + 2.5011670077219605, + 2.5930419942596927, + 2.6469169970368966, + 2.600416002678685, + 2.6204590103589, + 2.562416993896477, + 2.6447499985806644, + 2.7439999976195395, + 2.7353750047041103, + 2.8359999996609986, + 2.84583299071528, + 2.7894169907085598, + 2.763875003438443, + 2.83929098804947, + 2.961416990729049, + 3.016457994817756, + 2.975875002448447, + 2.957375007099472, + 3.050208993954584, + 3.206124994903803, + 3.1996660109143704, + 3.192000003764406, + 3.270083005190827, + 3.1873749976512045, + 3.145208000205457, + 3.295832997537218, + 3.3678750041872263, + 3.3211249974556267, + 3.3166250068461522, + 3.3052499929908663 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.002264583992655389, + 0.0022749589988961816, + 0.0021815420041093603, + 0.0022352500091074035, + 0.0022429169912356883, + 0.0022947499965084717, + 0.002394416995230131, + 0.002298958002938889, + 0.0021669589914381504, + 0.002130082997609861, + 0.0022005829960107803, + 0.0018530419911257923, + 0.0021994579874444753, + 0.0021493750100489706, + 0.0021932499948889017, + 0.002241124995634891, + 0.002230250000138767, + 0.0021636669989675283, + 0.001946415999555029, + 0.001987374998861924, + 0.0020274999988032505, + 0.00251387499156408, + 0.002404875005595386, + 0.0025687919987831265, + 0.002185625009587966, + 0.0026157089887419716, + 0.0021710829896619543, + 0.0021964579937048256, + 0.0026692920073401183, + 0.0026796669990289956, + 0.0023970830079633743, + 0.00234529199951794, + 0.002451749998726882, + 0.002441083997837268, + 0.0026858750061364844, + 0.00288883299799636, + 0.0026499160012463108, + 0.0026341670018155128, + 0.0025752499932423234, + 0.002562416993896477, + 0.0026940410025417805, + 0.002957791992230341, + 0.0027353750047041103, + 0.0029349579999689013, + 0.002779125003144145, + 0.0026977910019923, + 0.002763875003438443, + 0.0031802500016056, + 0.0028689169994322583, + 0.0029818749899277464, + 0.00289837499440182, + 0.002957375007099472, + 0.003050208993954584, + 0.003391874997760169, + 0.0033466250024503097, + 0.0029843749944120646, + 0.0031519170006504282, + 0.003064207994611934, + 0.003145208000205457, + 0.0033458329999120906, + 0.0033705829991959035, + 0.0031349580094683915, + 0.0031580840004608035, + 0.003181791995302774 + ], + [ + 0.0016447090019937605, + 0.001840791999711655, + 0.001808207991416566, + 0.0017957079980988055, + 0.001859208001405932, + 0.0020385410025482997, + 0.0020933749910909683, + 0.001801125006750226, + 0.0016223339916905388, + 0.0016772079980000854, + 0.0017599160055397078, + 0.0018289169966010377, + 0.0019014580029761419, + 0.0018828750035027042, + 0.0018078749999403954, + 0.0018360829999437556, + 0.0018409160111332312, + 0.0018859590054489672, + 0.0019784169999184087, + 0.0019865409994963557, + 0.002129917003912851, + 0.002178042006562464, + 0.0022038340102881193, + 0.0021190840052440763, + 0.002052875002846122, + 0.0021427090105134994, + 0.002175540997995995, + 0.0021912920055910945, + 0.002278041996760294, + 0.0023053329932736233, + 0.0023903340043034405, + 0.0024019580014282838, + 0.0024670410057296976, + 0.0024415000079898164, + 0.0025011670077219605, + 0.0025930419942596927, + 0.0026469169970368966, + 0.002600416002678685, + 0.002666000000317581, + 0.002561415996751748, + 0.002510209000320174, + 0.0027439999976195395, + 0.0027249580016359687, + 0.0028662499971687794, + 0.002832874990417622, + 0.0027894169907085598, + 0.0028176659980090335, + 0.002928874993813224, + 0.003025000012712553, + 0.0029817499889759347, + 0.0029489589942386374, + 0.002883833003579639, + 0.0030644580110674724, + 0.003206124994903803, + 0.003222958999685943, + 0.0032047919958131388, + 0.0033987500064540654, + 0.0031873749976512045, + 0.0032554169883951545, + 0.003295832997537218, + 0.0033678750041872263, + 0.0033310830040136352, + 0.0033166250068461522, + 0.0033052499929908663 + ], + [ + 0.0015892500086920336, + 0.0017235420091310516, + 0.0017401249933755025, + 0.0018732090102275833, + 0.0019558750063879415, + 0.001980000000912696, + 0.0020363750081742182, + 0.0019473749998724088, + 0.0015719999937573448, + 0.0015991249965736642, + 0.0017570000054547563, + 0.0016422089975094423, + 0.0017607499903533608, + 0.0019177909998688847, + 0.0019032919954042882, + 0.0019019580067833886, + 0.001958665990969166, + 0.0019859579915646464, + 0.001972209007362835, + 0.0019944580126320943, + 0.0020082499977434054, + 0.002074082993203774, + 0.0023258339933818206, + 0.0020861669909209013, + 0.002109541994286701, + 0.0021568340016528964, + 0.002202458999818191, + 0.0022494169970741495, + 0.0022907079983269796, + 0.0023419579956680536, + 0.0023309590033022687, + 0.0023379579943139106, + 0.0025135000032605603, + 0.0025287919997936115, + 0.00245687500864733, + 0.002485249991877936, + 0.0026301250036340207, + 0.002666083993972279, + 0.0026204590103589, + 0.002537666994612664, + 0.0026447499985806644, + 0.002667749999091029, + 0.0028079159965272993, + 0.0028359999996609986, + 0.0028525000088848174, + 0.0026978329988196492, + 0.002718624993576668, + 0.0027367080037947744, + 0.002987874991958961, + 0.003084624986513518, + 0.003005750011652708, + 0.002753000007942319, + 0.0028429169906303287, + 0.003104834002442658, + 0.0031996660109143704, + 0.003558458003681153, + 0.0031386249902425334, + 0.002958791999844834, + 0.0029765409999527037, + 0.0035451660078251734, + 0.0034443749900674447, + 0.003397332999156788, + 0.003293791989563033, + 0.0034721669944701716 + ], + [ + 0.0018107910000253469, + 0.0018132080003852025, + 0.002212917010183446, + 0.001836542011005804, + 0.0023503749980591238, + 0.00200133299222216, + 0.0020361249917186797, + 0.001934708998305723, + 0.0017421660013496876, + 0.002336916993954219, + 0.0017257500003324822, + 0.0018095830018864945, + 0.0018392920028418303, + 0.0018275000038556755, + 0.0018204999942099676, + 0.0018863340083044022, + 0.001969332995940931, + 0.001970250013982877, + 0.0020611660002032295, + 0.0020680410088971257, + 0.0021276659972500056, + 0.002192333005950786, + 0.002201291994424537, + 0.002173459011828527, + 0.0021804579882882535, + 0.002191209001466632, + 0.002354375013965182, + 0.002362540995818563, + 0.002404333994491026, + 0.002393834001850337, + 0.0022882090124767274, + 0.0022404999908758327, + 0.0024467079929308966, + 0.0025622910034144297, + 0.002580707994638942, + 0.0026150829944526777, + 0.0026916670030914247, + 0.0025575420004315674, + 0.002523125003790483, + 0.0026441670052008703, + 0.0027322079986333847, + 0.002772166990325786, + 0.002951875008875504, + 0.0026725409989012405, + 0.0029514159978134558, + 0.002879290987038985, + 0.0030420420080190524, + 0.00283929098804947, + 0.002961416990729049, + 0.003016457994817756, + 0.002975875002448447, + 0.0031840410083532333, + 0.0032470000005559996, + 0.0032117499940795824, + 0.003045624995138496, + 0.0030910000059520826, + 0.0032770840043667704, + 0.0033022080024238676, + 0.003341332994750701, + 0.0032506249990547076, + 0.003181291001965292, + 0.003308790997834876, + 0.0033787080028560013, + 0.003362541989190504 + ], + [ + 0.0016812499961815774, + 0.0017985419981414452, + 0.0017985000013140962, + 0.00175966699316632, + 0.0017907080036820844, + 0.0019419580057729036, + 0.0021293749887263402, + 0.002040708001004532, + 0.001719834006507881, + 0.001665500007220544, + 0.0016868330130819231, + 0.0018572080007288605, + 0.0018962920003104955, + 0.0017705420032143593, + 0.0018393339996691793, + 0.001846832994488068, + 0.0019664169958559796, + 0.001938250003149733, + 0.0018782080005621538, + 0.00196433300152421, + 0.0020403340022312477, + 0.0021347500005504116, + 0.0021244170056888834, + 0.0020711249962914735, + 0.002277292005601339, + 0.002236959000583738, + 0.002119208002113737, + 0.0021625839872285724, + 0.002289458003360778, + 0.002284125002915971, + 0.0022997079940978438, + 0.0023600840067956597, + 0.0025610830052755773, + 0.002277457999298349, + 0.0024904580059228465, + 0.002463707991410047, + 0.0025507079990347847, + 0.002599917002953589, + 0.0030664999940199777, + 0.0027072500088252127, + 0.0026072499895235524, + 0.0026670829975046217, + 0.002731125001446344, + 0.002781582996249199, + 0.00284583299071528, + 0.0028694579959847033, + 0.00275445899751503, + 0.002783167001325637, + 0.002880208005080931, + 0.0030279159982455894, + 0.003053708001971245, + 0.0030572500108974054, + 0.002926916000433266, + 0.003040208001038991, + 0.003033041997696273, + 0.003192000003764406, + 0.003270083005190827, + 0.0032448749989271164, + 0.0030970829975558445, + 0.003149542011669837, + 0.0032781659974716604, + 0.0033211249974556267, + 0.003324999997857958, + 0.003233209004974924 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_mha": { + "first_token_ms": { + "count": 5, + "median": 2.368166999076493, + "p95": 2.6830005983356386, + "min": 1.9714999943971634, + "max": 2.7237089962000027, + "iqr": 0.2071670169243589, + "mean": 2.379308597301133, + "cv": 0.116868168150788 + }, + "decode_ms_per_step": { + "count": 315, + "median": 2.971749985590577, + "p95": 3.9330498970230114, + "min": 1.8117500003427267, + "max": 5.855374998645857, + "iqr": 0.9020620054798201, + "mean": 2.9526042286768583, + "cv": 0.2021427799123564 + }, + "total_ms": { + "count": 5, + "median": 180.49216506187804, + "p95": 209.44902292103507, + "min": 177.27987898979336, + "max": 213.57408090261742, + "iqr": 15.276831923983991, + "mean": 188.39337500394322, + "cv": 0.08204778488209695 + }, + "tokens_per_second": { + "count": 5, + "median": 2836.6882286800164, + "p95": 2886.813948000882, + "min": 2397.294642852541, + "max": 2888.0886139902977, + "iqr": 228.1614713424765, + "mean": 2731.468116453364, + "cv": 0.0767766262263178 + }, + "median_step_ms_by_index": [ + 2.368166999076493, + 2.4379170063184574, + 2.5508329999865964, + 2.412458008620888, + 2.6834159943973646, + 2.708958010771312, + 2.9242909949971363, + 2.650417009135708, + 2.063749998342246, + 2.022499989834614, + 1.9949999987147748, + 2.0025000121677294, + 2.488749989424832, + 2.0565830054692924, + 2.078582998365164, + 2.0816249889321625, + 2.158667004550807, + 2.320917003089562, + 2.3054170014802366, + 2.2800409933552146, + 2.2842090111225843, + 2.326624991837889, + 2.4184159992728382, + 2.357999997911975, + 2.4465000024065375, + 2.597499988041818, + 2.529874996980652, + 2.4876669922377914, + 2.532332990085706, + 2.586000002338551, + 2.8417090070433915, + 2.740625001024455, + 2.900082996347919, + 2.8474169957917184, + 2.7919590065721422, + 2.818332999595441, + 2.885541005525738, + 2.9740839963778853, + 2.975166993564926, + 2.955916992505081, + 3.0058749980526045, + 2.993791989865713, + 3.0865830049151555, + 3.145750000840053, + 3.2071250025182962, + 3.0018749966984615, + 3.1814160029171035, + 3.2252920063911006, + 3.3277499896939844, + 3.33016699005384, + 3.3046669996110722, + 3.2397499890066683, + 3.3239170006709173, + 3.479582999716513, + 3.54249999509193, + 3.4951249981531873, + 3.4657910000532866, + 3.5246250045020133, + 3.6322920059319586, + 3.6052920040674508, + 3.6160409945296124, + 3.5001249925699085, + 3.698667002026923, + 3.6659579927800223 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0019714999943971634, + 0.0025127499975496903, + 0.0025508329999865964, + 0.002377292010351084, + 0.002099959005136043, + 0.005855374998645857, + 0.00497337500564754, + 0.004935208009555936, + 0.002578582993010059, + 0.003977290994953364, + 0.002749166000285186, + 0.002231958002084866, + 0.002488749989424832, + 0.0020556250092340633, + 0.0020714999991469085, + 0.002479000002495013, + 0.002591708005638793, + 0.0023229170037666336, + 0.0023054170014802366, + 0.0022800409933552146, + 0.0021888339979341254, + 0.0025499589974060655, + 0.002596874997834675, + 0.002796250002575107, + 0.0024465000024065375, + 0.0024977499997476116, + 0.0024912920052884147, + 0.002442040989990346, + 0.002512500010197982, + 0.0031823750032344833, + 0.0029658330022357404, + 0.002740625001024455, + 0.002820875000907108, + 0.0028323750011622906, + 0.0027919590065721422, + 0.0031787500047357753, + 0.0030535419937223196, + 0.0029404169908957556, + 0.002926167013356462, + 0.0027852500061271712, + 0.003033375003724359, + 0.003018500006874092, + 0.003121333007584326, + 0.003080333990510553, + 0.0030244999943533912, + 0.003089165998972021, + 0.003491708994260989, + 0.0032432089938083664, + 0.0032483329996466637, + 0.003232582996133715, + 0.0032521249959245324, + 0.0033317499910481274, + 0.0034182919916929677, + 0.0033938330016098917, + 0.003373707993887365, + 0.003406874995562248, + 0.003427624993491918, + 0.003632749998359941, + 0.0035575410001911223, + 0.003491417010081932, + 0.003474875004030764, + 0.003920874994946644, + 0.003963166003813967, + 0.0035724999906960875 + ], + [ + 0.0023129999899538234, + 0.002413250011159107, + 0.0024754170008236542, + 0.0024476670078001916, + 0.002437667004414834, + 0.0021680409990949556, + 0.0021169999963603914, + 0.00211387500166893, + 0.0018117500003427267, + 0.002022499989834614, + 0.0018840420088963583, + 0.002000957989366725, + 0.00203650000912603, + 0.0020565830054692924, + 0.002078582998365164, + 0.0020816249889321625, + 0.002158667004550807, + 0.0021940000005997717, + 0.0022889579995535314, + 0.0022372079984052107, + 0.002164584002457559, + 0.002307583999936469, + 0.0024184159992728382, + 0.0022620419913437217, + 0.002456041009281762, + 0.0024357090005651116, + 0.002434374997392297, + 0.0024876669922377914, + 0.0024634580040583387, + 0.0025581249938113615, + 0.0026525420107645914, + 0.0025158329954138026, + 0.003276415998698212, + 0.00284845799615141, + 0.0025830840022535995, + 0.00282754200452473, + 0.0028041659970767796, + 0.003019667012267746, + 0.002975166993564926, + 0.003003874997375533, + 0.0030058749980526045, + 0.002993791989865713, + 0.0030865830049151555, + 0.003209708011127077, + 0.003244417006499134, + 0.002971749985590577, + 0.0030375420028576627, + 0.003151500000967644, + 0.0034119590127374977, + 0.0033399589883629233, + 0.0034052499977406114, + 0.0032312090042978525, + 0.0033239170006709173, + 0.00351429199508857, + 0.00354249999509193, + 0.003519000005326234, + 0.0034921669866889715, + 0.0035157919919583946, + 0.0036660000041592866, + 0.003660208996734582, + 0.003721332992427051, + 0.0034877500002039596, + 0.003698667002026923, + 0.004216666013235226 + ], + [ + 0.0027237089962000027, + 0.0026569579931674525, + 0.002803875002427958, + 0.002412458008620888, + 0.002728874998865649, + 0.003099416004261002, + 0.0031323750008596107, + 0.0031770829955348745, + 0.0025622910034144297, + 0.0025078339967876673, + 0.0025382919993717223, + 0.003296625000075437, + 0.0034164999960921705, + 0.0033040830021491274, + 0.003550916997483, + 0.0034547909890534356, + 0.00424608399043791, + 0.0042605829949025065, + 0.003990625002188608, + 0.00405654200585559, + 0.003972916994825937, + 0.003979582994361408, + 0.003907207996235229, + 0.004100207996089011, + 0.003650124999694526, + 0.0032372920104535297, + 0.003333374988869764, + 0.0031991669966373593, + 0.003096082989941351, + 0.00328529100806918, + 0.0029178749973652884, + 0.0029370840056799352, + 0.0028794170066248626, + 0.0032462090020999312, + 0.003106833013589494, + 0.002818332999595441, + 0.003395124993403442, + 0.0029592499922728166, + 0.0029964169953018427, + 0.0029618330008815974, + 0.0031052499980432913, + 0.003279666998423636, + 0.0031803329911781475, + 0.0031452089897356927, + 0.0032071250025182962, + 0.0035978329979116097, + 0.0033374579943483695, + 0.0032471250015078112, + 0.0033277499896939844, + 0.00333016699005384, + 0.0033046669996110722, + 0.003398832996026613, + 0.003376958004082553, + 0.0034663749975152314, + 0.0036473340005613863, + 0.0035024159878958017, + 0.0037354160012910143, + 0.003961458001867868, + 0.0035681250010384247, + 0.003994582992163487, + 0.003609916995628737, + 0.0040935420111054555, + 0.0035910410078940913, + 0.0036659579927800223 + ], + [ + 0.0025201670068781823, + 0.0023962080013006926, + 0.0025057079910766333, + 0.002358583005843684, + 0.0026834159943973646, + 0.0025960000057239085, + 0.0028321669960860163, + 0.0026396250032121316, + 0.0018202080100309104, + 0.001905875004013069, + 0.001994999998714775, + 0.0020025000121677294, + 0.002546708012232557, + 0.0021018329862272367, + 0.002392791007878259, + 0.0020376249885885045, + 0.0021558330045081675, + 0.002165124999010004, + 0.002270916011184454, + 0.002280875007272698, + 0.0022842090111225843, + 0.002284333997522481, + 0.002314625002327375, + 0.002357999997911975, + 0.002428708001389168, + 0.002597499988041818, + 0.0026059590018121526, + 0.002588625007774681, + 0.002558417007094249, + 0.002458249990013428, + 0.0028417090070433915, + 0.0027439160039648414, + 0.0030847919988445938, + 0.0028474169957917184, + 0.0027660829946398735, + 0.0027830420003738254, + 0.002885541005525738, + 0.0029740839963778853, + 0.003028084000106901, + 0.002955916992505081, + 0.002835874998709187, + 0.002988291991641745, + 0.0030567499925382435, + 0.0032131250045495108, + 0.003223082996555604, + 0.0030018749966984615, + 0.0031814160029171035, + 0.0032252920063911006, + 0.0034441250027157366, + 0.0034081250050803646, + 0.0033250830019824207, + 0.0032397499890066683, + 0.0033233330032089725, + 0.0035029580030823126, + 0.003556374998879619, + 0.0034951249981531873, + 0.0034657910000532866, + 0.0035246250045020133, + 0.0036526250041788444, + 0.0035783329949481413, + 0.0036832920013694093, + 0.003427834002650343, + 0.0038613750075455755, + 0.0036813330079894513 + ], + [ + 0.002368166999076493, + 0.0024379170063184574, + 0.0026400420028949156, + 0.0024243330117315054, + 0.0027391249896027148, + 0.002708958010771312, + 0.0029242909949971363, + 0.002650417009135708, + 0.002063749998342246, + 0.001961125002708286, + 0.0019802920141955838, + 0.001997749990550801, + 0.0020477079960983247, + 0.001860708012827672, + 0.0019641249964479357, + 0.001996917009819299, + 0.0021090420050313696, + 0.002320917003089562, + 0.0023779160110279918, + 0.0021807089942740276, + 0.002292290999321267, + 0.002326624991837889, + 0.0023773749999236315, + 0.0023085830034688115, + 0.002368083005421795, + 0.0026484170084586367, + 0.002529874996980652, + 0.002478958005667664, + 0.002532332990085706, + 0.002586000002338551, + 0.002520499998354353, + 0.002632333998917602, + 0.002900082996347919, + 0.002706915998714976, + 0.0028462920017773286, + 0.0027156249998370185, + 0.0027967499918304384, + 0.0030120000010356307, + 0.002956832991912961, + 0.0028172500024084, + 0.002979874989250675, + 0.0029313749982975423, + 0.0030605000065406784, + 0.003145750000840053, + 0.0031963340006768703, + 0.0029681670130230486, + 0.0030264999950304627, + 0.003132833997369744, + 0.0033250420092372224, + 0.003320417003124021, + 0.003208417008863762, + 0.003208791997167282, + 0.003268999993451871, + 0.003479582999716513, + 0.003525749998516403, + 0.003431834004004486, + 0.0034050410031341016, + 0.003405916999327019, + 0.0036322920059319586, + 0.0036052920040674508, + 0.0036160409945296124, + 0.0035001249925699085, + 0.0035739580052904785, + 0.0036157910071779042 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "speedup_cached_vs_uncached_sdpa": 1.1279096630808414, + "speedup_cached_vs_uncached_mha": 1.2954680023333496 + }, + { + "batch": 8, + "prompt_len": 16, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_mha": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 1.8549170054029673, + "p95": 1.9019664061488584, + "min": 1.7793330043787137, + "max": 1.9042080093640834, + "iqr": 0.09349998435936868, + "mean": 1.8461916042724624, + "cv": 0.030015783009770487 + }, + "decode_ms_per_step": { + "count": 315, + "median": 1.9770000071730465, + "p95": 2.424399902520236, + "min": 1.7120420088758692, + "max": 2.5740830024005845, + "iqr": 0.1672074940870516, + "mean": 2.01462606071598, + "cv": 0.08935831446921118 + }, + "total_ms": { + "count": 5, + "median": 126.53816904639825, + "p95": 134.92390322498977, + "min": 122.98237603681628, + "max": 135.46824303921312, + "iqr": 6.643708911724389, + "mean": 128.7676334293792, + "cv": 0.0400285805507917 + }, + "tokens_per_second": { + "count": 5, + "median": 4046.2099606661995, + "p95": 4142.594186183563, + "min": 3779.4835786848926, + "max": 4163.1981467549995, + "iqr": 203.20410792033272, + "mean": 3981.2088531962786, + "cv": 0.03965716712561269 + }, + "median_step_ms_by_index": [ + 1.8549170054029673, + 2.443707999191247, + 1.9396670104470104, + 2.2550829889951274, + 1.936583998030983, + 2.3202919983305037, + 1.9105419924017042, + 1.9647079898277298, + 1.9012919947272167, + 1.9027920061489567, + 1.8808330059982836, + 1.901334006106481, + 1.8866250029532239, + 1.9819170120172203, + 1.9024170032935217, + 1.85425000381656, + 2.0807910041185096, + 2.305292000528425, + 1.915458997245878, + 1.9087920081801713, + 1.9775410037254915, + 2.006291993893683, + 1.9332500087330118, + 2.0040420058649033, + 2.1011670032748953, + 1.9239589892094955, + 1.9623749976744875, + 1.9237079977756366, + 1.9150830048602074, + 1.8703329988056794, + 1.9509590056259185, + 2.121874989825301, + 1.9343750027474016, + 1.9330420036567375, + 1.9182079995516688, + 1.875417001429014, + 1.8606670055305585, + 1.9319589919177815, + 1.931750011863187, + 1.9777499983320013, + 1.9573749887058511, + 1.9634159980341792, + 1.936000000569038, + 1.9592080061556771, + 2.001875007408671, + 1.9422089972067624, + 1.9846659997710958, + 2.0099999965168536, + 2.0107500022277236, + 2.0246669882908463, + 2.0954579958925024, + 2.0179580023977906, + 2.0453750039450824, + 1.990791002754122, + 1.9824589980999008, + 2.080666003166698, + 2.042459003860131, + 1.9183340045856312, + 2.0244579936843365, + 2.031000010902062, + 2.003540997975506, + 2.0145000016782433, + 2.017874998273328, + 2.0272910041967407 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0017793330043787137, + 0.0018966659990837798, + 0.0018759169906843454, + 0.0018540000019129366, + 0.0023470839951187372, + 0.0017474999913247302, + 0.001856375005445443, + 0.0017707920051179826, + 0.0022118749911896884, + 0.001798041004803963, + 0.0018709169962676242, + 0.001901334006106481, + 0.0018714170000748709, + 0.0018508330103941262, + 0.0017120420088758692, + 0.0017597500118426979, + 0.0018457089900039136, + 0.002305292000528425, + 0.001915458997245878, + 0.001910250008222647, + 0.0017181250004796311, + 0.0018911670049419627, + 0.0018611659907037392, + 0.001782457999070175, + 0.0018057919951388612, + 0.0019239589892094955, + 0.0018669580022105947, + 0.0018244999955641106, + 0.001879166011349298, + 0.0017882500105770305, + 0.0018823330028681085, + 0.001901124996948056, + 0.001924957992741838, + 0.0019330420036567375, + 0.0019182079995516688, + 0.0018230419955216348, + 0.0018094590050168335, + 0.0019324169988976792, + 0.001931750011863187, + 0.0019777499983320013, + 0.0019709579937625676, + 0.0019795419939327985, + 0.001936000000569038, + 0.001890457991976291, + 0.0019384170009288937, + 0.0018752920004772022, + 0.0018022909935098141, + 0.0018171660049119964, + 0.0018760410021059215, + 0.0019944169907830656, + 0.002550792007241398, + 0.0019809999939752743, + 0.0020678330038208514, + 0.0019847090006805956, + 0.0019897500023944303, + 0.0019770000071730465, + 0.0019456660083960742, + 0.0019183340045856312, + 0.0019730000058189034, + 0.0020286250073695555, + 0.0019426249928073958, + 0.002041083003859967, + 0.002017874998273328, + 0.0020272910041967407 + ], + [ + 0.0019042080093640834, + 0.0022812079987488687, + 0.0019821250025415793, + 0.0022550829889951274, + 0.0018801669939421117, + 0.0023202919983305037, + 0.0019105419924017042, + 0.0023596250102855265, + 0.0019012919947272167, + 0.001795708987629041, + 0.0018808330059982836, + 0.0018161670013796538, + 0.0017576659884070978, + 0.0018953329999931157, + 0.0019024170032935217, + 0.00185425000381656, + 0.0021821669943165034, + 0.0018970840028487146, + 0.0018431670032441616, + 0.0017747079982655123, + 0.0019201660034013912, + 0.0020326670055510476, + 0.0019332500087330118, + 0.002252499994938262, + 0.002448208993882872, + 0.001773166994098574, + 0.0019814590050373226, + 0.0019237079977756366, + 0.0019265419978182763, + 0.0018703329988056794, + 0.0017766250093700364, + 0.002144875004887581, + 0.001921833012602292, + 0.0019485000084387138, + 0.0018252080044476315, + 0.0018575410067569464, + 0.0017622500017751008, + 0.0017658330034464598, + 0.001780458987923339, + 0.001845332997618243, + 0.001787624991266057, + 0.0018604159995447844, + 0.0017734589928295463, + 0.001779166006599553, + 0.0024070420040516183, + 0.0019422089972067624, + 0.001984665999771096, + 0.0020099999965168536, + 0.002068875008262694, + 0.0020541249978123233, + 0.002039584011072293, + 0.0020179580023977906, + 0.0019022090127691627, + 0.001990791002754122, + 0.0019824589980999008, + 0.002080666003166698, + 0.002042459003860131, + 0.0019086670072283596, + 0.0020244579936843365, + 0.002031000010902062, + 0.0021200000046519563, + 0.002076707998639904, + 0.0020647089986596256, + 0.002069082998787053 + ], + [ + 0.0017995000089285895, + 0.002529291989048943, + 0.001952791993971914, + 0.0018465000030118972, + 0.001936583998030983, + 0.0019447079976089299, + 0.0019482920033624396, + 0.0018450829957146198, + 0.0018855410016840324, + 0.0019027920061489567, + 0.0018949170043924823, + 0.0018391249905107543, + 0.0020195839897496626, + 0.0020040000090375543, + 0.0018952090031234547, + 0.0017543750000186265, + 0.0022579999931622297, + 0.0024743340036366135, + 0.0019112089939881116, + 0.002339708007639274, + 0.0019775410037254915, + 0.0019264169968664646, + 0.0018971660028910264, + 0.0019637919904198498, + 0.002300833002664149, + 0.0019055410084547475, + 0.0018795409996528178, + 0.0019405000057304278, + 0.001842916986788623, + 0.0018604170036269352, + 0.0019509590056259185, + 0.0020075420034117997, + 0.0019719590054592118, + 0.0019087920081801713, + 0.0018577080045361072, + 0.001875417001429014, + 0.0018606670055305585, + 0.0019319589919177815, + 0.0019490419945213944, + 0.001967333009815775, + 0.001957374988705851, + 0.001963415998034179, + 0.0019072910072281957, + 0.0019952919974457473, + 0.002001875007408671, + 0.0020160000130999833, + 0.0020879170042462647, + 0.002028500006417744, + 0.002019084000494331, + 0.002080666003166698, + 0.0020954579958925024, + 0.0020193329983158037, + 0.0019164580007782206, + 0.0019038329919567332, + 0.0018894999957410619, + 0.001967000003787689, + 0.001928375000716187, + 0.001934499989147298, + 0.002050790993962437, + 0.0019992910092696548, + 0.0020031669992022216, + 0.0020008340070489794, + 0.0020032500033266842, + 0.002011375006986782 + ], + [ + 0.0018549170054029673, + 0.002443707999191247, + 0.0019396670104470104, + 0.002388791006524116, + 0.0019186659919796512, + 0.0023378339974442497, + 0.0018746669957181439, + 0.0023230410006362945, + 0.0018983330082846805, + 0.0023548749886685982, + 0.0018797910015564412, + 0.0023360000050161034, + 0.0018866250029532239, + 0.0022180829982971773, + 0.0019133750029141083, + 0.0020787079993169755, + 0.0020807910041185096, + 0.002535166990128346, + 0.0024453750083921477, + 0.0019087920081801713, + 0.0020156659884378314, + 0.002496832996257581, + 0.0025338749983347952, + 0.002214249994722195, + 0.0019332499941810966, + 0.002082207996863872, + 0.0023620410065632313, + 0.002048959009698592, + 0.002485709002939984, + 0.001917833011248149, + 0.0020890000014333054, + 0.002121874989825301, + 0.0020730000105686486, + 0.0020130420016357675, + 0.002041291998466477, + 0.0024714589962968603, + 0.0020381659996928647, + 0.002041667001321912, + 0.0020948750025127083, + 0.0019859580061165616, + 0.00206433299172204, + 0.002020165993599221, + 0.0025740830024005845, + 0.0020525830041151494, + 0.002088374996674247, + 0.002118040996720083, + 0.0019061250059166923, + 0.0019053330033784732, + 0.0020107500022277236, + 0.0020246669882908463, + 0.002096957992762327, + 0.002363167004659772, + 0.0020453750039450824, + 0.0020664160110754892, + 0.001973625010577962, + 0.0021918330021435395, + 0.0020928339945385233, + 0.0020612499938579276, + 0.002012125012697652, + 0.0020764999935636297, + 0.002003540997975506, + 0.0020114580111112446, + 0.002012332988670096, + 0.002018208004301414 + ], + [ + 0.0018929999932879582, + 0.0025430000096093863, + 0.001932915998622775, + 0.0022927909885765985, + 0.002416125003946945, + 0.002359542006161064, + 0.0024545000051148236, + 0.0019647079898277298, + 0.0021127500076545402, + 0.0021838749962626025, + 0.0020309159881435335, + 0.0019360830046935007, + 0.002302459004567936, + 0.0019819170120172203, + 0.0023567089956486598, + 0.0019047909881919622, + 0.0019612500036600977, + 0.001855042006354779, + 0.0019623749976744875, + 0.0018417499959468842, + 0.0019870410033036023, + 0.002006291993893683, + 0.0020694169943453744, + 0.0020040420058649033, + 0.0021011670032748953, + 0.0020031250023748726, + 0.0019623749976744875, + 0.0018845000013243407, + 0.0019150830048602074, + 0.0023143329890444875, + 0.001995625003473833, + 0.0025137920019915327, + 0.0019343750027474016, + 0.0019311670039314777, + 0.0025134580064332113, + 0.0019813749968307093, + 0.0019700839911820367, + 0.0019176669884473085, + 0.001928375000716187, + 0.0023541659902548417, + 0.0019317499973112717, + 0.0019226250005885959, + 0.0020084580028196797, + 0.001959208006155677, + 0.0019383750041015446, + 0.001917332992888987, + 0.0020010839944006875, + 0.00232641700131353, + 0.001982541987672448, + 0.001966833006008528, + 0.002095249990816228, + 0.0019863750057993457, + 0.0022348750062519684, + 0.002044916996965185, + 0.0021086250053485855, + 0.002169874991523102, + 0.002100167010212317, + 0.0018970840028487146, + 0.0022296249953797087, + 0.002177832997404039, + 0.002071209004498087, + 0.0020145000016782433, + 0.002057958990917541, + 0.0020296669972594827 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + ] + } + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 1.9403750047786161, + "p95": 2.2501998028019443, + "min": 1.7843329987954348, + "max": 2.289458003360778, + "iqr": 0.30233399593271315, + "mean": 1.9796332024270669, + "cv": 0.10840320856314503 + }, + "decode_ms_per_step": { + "count": 315, + "median": 2.8668330051004887, + "p95": 3.8572464996832423, + "min": 1.8166249938076362, + "max": 4.3076250003650784, + "iqr": 0.869791503646411, + "mean": 2.9041378345388535, + "cv": 0.20112700223992283 + }, + "total_ms": { + "count": 5, + "median": 184.62646396073978, + "p95": 190.60498918406665, + "min": 179.09920896636322, + "max": 192.05491397588048, + "iqr": 0.6895830447319895, + "mean": 184.94031677837484, + "cv": 0.02500372001650066 + }, + "tokens_per_second": { + "count": 5, + "median": 2773.1669069330987, + "p95": 2843.1726038133934, + "min": 2665.9041906332077, + "max": 2858.750761407099, + "iqr": 10.37650971615949, + "mean": 2769.8330592268767, + "cv": 0.024777923214475615 + }, + "median_step_ms_by_index": [ + 1.9403750047786161, + 1.9736249960260466, + 1.963624992640689, + 2.0149589981883764, + 2.0063340052729473, + 2.001959001063369, + 2.0893330074613914, + 2.125625003827736, + 2.1454169909702614, + 2.2047080128686503, + 2.1604590001516044, + 2.1876669925404713, + 2.217790999566205, + 2.2580829972866923, + 2.2727909963577986, + 2.3787090030964464, + 2.3462910030502826, + 2.561084009357728, + 2.4556249991292134, + 2.526416996261105, + 2.5070409901672974, + 2.5192499888362363, + 2.6432080048834905, + 2.650209004059434, + 2.630249990033917, + 2.7240420022280887, + 2.6529159949859604, + 2.8146669937996194, + 2.856082996004261, + 2.896624995628372, + 2.8394169930834323, + 2.7607079973677173, + 2.7585419884417206, + 2.9712919931625947, + 2.9968750022817403, + 3.0589999951189384, + 2.993957998114638, + 2.9722079925704747, + 3.0071670043980703, + 3.2224589958786964, + 3.108750010142103, + 3.253792005125433, + 3.1817910057725385, + 3.1284580036299303, + 3.219834004994482, + 3.2898329955060035, + 3.2416670001111925, + 3.2314169948222116, + 3.2014170137699693, + 3.501290993881412, + 3.487249996396713, + 3.4359160054009408, + 3.5064170078840107, + 3.5129589959979057, + 3.6051659990334883, + 3.6871249903924763, + 3.6932500079274178, + 3.7264169950503856, + 3.703791997395456, + 3.8183330034371465, + 3.8370840047718957, + 3.8168749888427556, + 3.839958008029498, + 3.879374999087304 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.002289458003360778, + 0.0023179579875431955, + 0.002344708002055995, + 0.0023421249934472144, + 0.001954334002221003, + 0.002001959001063369, + 0.002409417007584125, + 0.0023858750064391643, + 0.0021454169909702614, + 0.002579791995231062, + 0.0021052079973742366, + 0.0021016249957028776, + 0.002491332998033613, + 0.0021404159924713895, + 0.0024609579995740205, + 0.0026605839957483113, + 0.0021947499917587265, + 0.0024124159972416237, + 0.0024355839996133, + 0.0028222079999977723, + 0.002437709001242183, + 0.002265082992380485, + 0.002498333007679321, + 0.002508040997781791, + 0.002630249990033917, + 0.002464292003423907, + 0.002622833999339491, + 0.003053958003874868, + 0.0026742080081021413, + 0.002896624995628372, + 0.0027685410022968426, + 0.0026687920035328716, + 0.0026844170060940087, + 0.00311145800515078, + 0.002787542005535215, + 0.0027988749934593216, + 0.0026426669937791303, + 0.002997000003233552, + 0.002942749997600913, + 0.0032306250068359077, + 0.0030207079980755225, + 0.004016708000563085, + 0.003042165990336798, + 0.0031284580036299303, + 0.003482292013359256, + 0.0032898329955060035, + 0.0032060410012491047, + 0.0030873329960741103, + 0.0030912090005585924, + 0.003657374996691942, + 0.0034755420056171715, + 0.003308790997834876, + 0.00324045900197234, + 0.003322542004752904, + 0.003368165998836048, + 0.0035776669974438846, + 0.003592542008846067, + 0.003440375003265217, + 0.003519040998071432, + 0.0039375829946948215, + 0.004197417001705617, + 0.003685874995426275, + 0.0035479170037433505, + 0.0035995419893879443 + ], + [ + 0.0017843329987954348, + 0.0018621670024003834, + 0.0018607920064823702, + 0.0018435000092722476, + 0.0018166249938076362, + 0.0019831670069834217, + 0.0020893330074613914, + 0.0021256250038277358, + 0.002071249997243285, + 0.002164332996471785, + 0.0019734169909497723, + 0.0021876669925404713, + 0.0022133749880595133, + 0.0023072080075507984, + 0.0021550840028794482, + 0.0022292089997790754, + 0.0022874999995110556, + 0.0026043749967357144, + 0.0025166669947793707, + 0.0024375420034630224, + 0.002433082991046831, + 0.0024564170016674325, + 0.0024678340123500675, + 0.002666417000000365, + 0.0025915830046869814, + 0.0026340829936088994, + 0.002621082996483892, + 0.0026834169984795153, + 0.0027089590003015473, + 0.002932832998340018, + 0.00277108300360851, + 0.0026977499946951866, + 0.0027201659977436066, + 0.0028066670056432486, + 0.0028127919940743595, + 0.0031582500087097287, + 0.002993957998114638, + 0.0029361250053625554, + 0.0030049579945625737, + 0.0029881659866077825, + 0.0029590420017484576, + 0.003214707990991883, + 0.0031365000031655654, + 0.0030367080034920946, + 0.002895500001613982, + 0.002876166006899439, + 0.0032737079891376197, + 0.0032314169948222116, + 0.0031465829961234704, + 0.0032715419947635382, + 0.003338875001645647, + 0.003396958010853268, + 0.0035381250054342672, + 0.0035129589959979057, + 0.003368541001691483, + 0.0034226660063723102, + 0.0037222089886199683, + 0.0037976669991621748, + 0.003703791997395456, + 0.0035892499872716144, + 0.0036751250008819625, + 0.003808458990533836, + 0.003839958008029498, + 0.003743958004633896 + ], + [ + 0.001790833004633896, + 0.0019600000086938962, + 0.0019158749928465113, + 0.0020149589981883764, + 0.00202358300157357, + 0.002081541999359615, + 0.0020624580065486953, + 0.002087582994136028, + 0.0020756669982802123, + 0.0021713329915655777, + 0.0021604590001516044, + 0.0022375830012606457, + 0.002217790999566205, + 0.0022909589897608384, + 0.0022830420057289302, + 0.0023787090030964464, + 0.0023462910030502826, + 0.002536542000598274, + 0.0024505839974153787, + 0.002526416996261105, + 0.0025232080079149455, + 0.0026108750025741756, + 0.0026432080048834905, + 0.0025262089911848307, + 0.002782416995614767, + 0.0027534169930731878, + 0.0026529159949859604, + 0.00318212500133086, + 0.0028637500072363764, + 0.002832666999893263, + 0.0028643749974435195, + 0.002788500001770444, + 0.002837958003510721, + 0.0029712919931625947, + 0.0029968750022817403, + 0.0030589999951189384, + 0.0030026250024093315, + 0.0028516250022221357, + 0.003077207991736941, + 0.0032224589958786964, + 0.003116666994174011, + 0.003234749994589947, + 0.0031995419994927943, + 0.0032106669968925416, + 0.003219834004994482, + 0.003266500003519468, + 0.0032416670001111925, + 0.0031902090122457594, + 0.003217584002413787, + 0.003501290993881412, + 0.003487249996396713, + 0.0034359160054009408, + 0.0033683749934425578, + 0.003545208994182758, + 0.0036051659990334883, + 0.0036871249903924763, + 0.0036736250040121377, + 0.0035972920013591647, + 0.003966458010836504, + 0.0038113749906187877, + 0.0038370840047718957, + 0.0038168749888427556, + 0.003861709003103897, + 0.003879374999087304 + ], + [ + 0.0019403750047786161, + 0.0019736249960260466, + 0.001979541004402563, + 0.0024720000074012205, + 0.002010457988944836, + 0.002350584007217549, + 0.002110458997776732, + 0.002185040997574106, + 0.0021832080092281103, + 0.0023785419907653704, + 0.0021773749904241413, + 0.002579167005023919, + 0.002213457992183976, + 0.0022580829972866923, + 0.0022727909963577986, + 0.002452416010783054, + 0.0025889579992508516, + 0.002561084009357728, + 0.00258808299258817, + 0.00283837498864159, + 0.002533457998652011, + 0.0025525829987600446, + 0.0028609579894691706, + 0.002650209004059434, + 0.0026209170027868822, + 0.002739749994361773, + 0.0027842910058097914, + 0.0028146669937996194, + 0.003070749997277744, + 0.0029041660018265247, + 0.0028394169930834323, + 0.002810416990541853, + 0.0028668330051004887, + 0.003002333003678359, + 0.0031975419988157228, + 0.0030907500040484592, + 0.0029752499976893887, + 0.0030608749948441982, + 0.003451249998761341, + 0.003567040999769233, + 0.003108750010142103, + 0.003253792005125433, + 0.0032311660033883527, + 0.0032725839992053807, + 0.003442917004576884, + 0.0033035410015145317, + 0.0032597500103292987, + 0.003310083004180342, + 0.0032523749978281558, + 0.0036850409960607067, + 0.0035277919960208237, + 0.003539999990607612, + 0.0035064170078840107, + 0.003581665994715877, + 0.004101624988834374, + 0.004058457998326048, + 0.004021125001600012, + 0.0037417499988805503, + 0.004009417010820471, + 0.0038553339982172474, + 0.004246416996465996, + 0.0043076250003650784, + 0.004040291998535395, + 0.003889917003107257 + ], + [ + 0.002093167000566609, + 0.002012333003222011, + 0.001963624992640689, + 0.0018809169996529818, + 0.0020063340052729473, + 0.0019360419973963872, + 0.00201258399465587, + 0.0020926249999320135, + 0.0021547499927692115, + 0.0022047080128686503, + 0.002255582992802374, + 0.0021596250007860363, + 0.0022479170002043247, + 0.0022537920012837276, + 0.002249124998343177, + 0.0023537500092061237, + 0.0023933749907882884, + 0.0026302080077584833, + 0.0024556249991292134, + 0.0024424580042250454, + 0.0025070409901672974, + 0.0025192499888362363, + 0.0026729579985840246, + 0.0027392499905545264, + 0.002684499995666556, + 0.0027240420022280887, + 0.0026698750007199124, + 0.002781208007945679, + 0.002856082996004261, + 0.002894375007599592, + 0.002897417012718506, + 0.0027607079973677173, + 0.0027585419884417206, + 0.002896791003877297, + 0.003006833008839749, + 0.0030384580022655427, + 0.0030808750016149133, + 0.0029722079925704747, + 0.0030071670043980703, + 0.0031640829984098673, + 0.003187875001458451, + 0.003275040988228284, + 0.0031817910057725385, + 0.0031187919958028942, + 0.003177332997438498, + 0.003367042008903809, + 0.003154874997562729, + 0.0034178329951828346, + 0.0032014170137699693, + 0.003385500007425435, + 0.00378400000045076, + 0.003558917000191286, + 0.0035783330095000565, + 0.003371333994437009, + 0.0036149579973425716, + 0.0037017919967183843, + 0.0036932500079274178, + 0.0037264169950503856, + 0.003540167002938688, + 0.0038183330034371465, + 0.0038162080018082634, + 0.003916500005288981, + 0.0037431249947985634, + 0.004044250003062189 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_mha": { + "first_token_ms": { + "count": 5, + "median": 2.1023330045863986, + "p95": 2.4791502015432343, + "min": 2.0294590067351237, + "max": 2.525042000343092, + "iqr": 0.26383300428278744, + "mean": 2.196833404013887, + "cv": 0.09700939236923299 + }, + "decode_ms_per_step": { + "count": 315, + "median": 3.2470000005559996, + "p95": 4.261662397766486, + "min": 2.053625008556992, + "max": 7.332624998525716, + "iqr": 1.0237284950562753, + "mean": 3.2981670318385734, + "cv": 0.22758691329422429 + }, + "total_ms": { + "count": 5, + "median": 211.0939960111864, + "p95": 217.10855714045465, + "min": 199.54657905327622, + "max": 217.78895692841616, + "iqr": 7.2966659208759665, + "mean": 209.981356409844, + "cv": 0.03357768106341339 + }, + "tokens_per_second": { + "count": 5, + "median": 2425.4597936213586, + "p95": 2547.12388622249, + "min": 2350.899729816358, + "max": 2565.816975811462, + "iqr": 84.14655120377438, + "mean": 2440.5466007557225, + "cv": 0.03410246685357876 + }, + "median_step_ms_by_index": [ + 2.1023330045863986, + 2.1779579983558506, + 2.1927909983787686, + 2.166082995245233, + 2.2520409984281287, + 2.226042008260265, + 2.298207997228019, + 2.442791999783367, + 2.3480420059058815, + 2.38754200108815, + 2.4259170022560284, + 2.4806250003166497, + 2.551541998400353, + 2.565208007581532, + 2.6380830095149577, + 2.606167006888427, + 2.5666250003268942, + 2.791292004985735, + 2.78566699125804, + 2.7890830097021535, + 2.8047079977113754, + 2.8755829989677295, + 2.95145800919272, + 2.9664999892702326, + 2.990542008774355, + 2.9944580019218847, + 3.0165420030243695, + 3.0837499944027513, + 3.0737079941900447, + 3.066584002226591, + 3.096790998824872, + 3.160207998007536, + 3.131833000225015, + 3.297167000710033, + 3.281582990894094, + 3.2601669954601675, + 3.3021660055965185, + 3.3585830096853897, + 3.4416249982314184, + 3.4849580115405843, + 3.4394999966025352, + 3.5442920052446425, + 3.5526249994290993, + 3.620207993662916, + 3.604167010053061, + 3.566500003216788, + 3.5219999990658835, + 3.608792001614347, + 3.704708011355251, + 3.886999998940155, + 3.839583005174063, + 4.071583010954782, + 3.952208993723616, + 3.9752500015310943, + 3.912833009962924, + 4.0494169952580705, + 4.121541001950391, + 4.192624997813255, + 4.095708005479537, + 4.139832992223091, + 4.238458990585059, + 4.232832987327129, + 4.174374989815988, + 4.167667008005083 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0020294590067351237, + 0.0021779579983558506, + 0.002106040992657654, + 0.002053625008556992, + 0.002482999989297241, + 0.002370540998526849, + 0.002246750009362586, + 0.0026264579937560484, + 0.0028051250119460747, + 0.002402792000793852, + 0.0024259170022560284, + 0.0022507079993374646, + 0.002496166998753324, + 0.0024141670000972226, + 0.00239424999745097, + 0.0025482079945504665, + 0.0025405839987797663, + 0.0026882499951170757, + 0.0026927080034511164, + 0.0026910829910775647, + 0.0030343750113388523, + 0.0028755829989677295, + 0.00295145800919272, + 0.0027477500116219744, + 0.002748790997429751, + 0.0028835410048486665, + 0.0029392910073511302, + 0.0030477499967673793, + 0.0030737079941900447, + 0.002911832998506725, + 0.0029679580038646236, + 0.0029976670048199594, + 0.0030730830039829016, + 0.0032065840059658512, + 0.003154500009259209, + 0.003086458003963344, + 0.0031622919923393056, + 0.0033218329917872325, + 0.0033958749991143122, + 0.0034849580115405843, + 0.0032242089946521446, + 0.003351082996232435, + 0.0033972919918596745, + 0.003457958999206312, + 0.003533541996148415, + 0.0033962910092668608, + 0.0034047079971060157, + 0.003621541996835731, + 0.003589542000554502, + 0.0038376669981516898, + 0.0036088329943595454, + 0.004071583010954782, + 0.003853125002933666, + 0.003943750009057112, + 0.003784542001085356, + 0.00408083300862927, + 0.003924042001017369, + 0.0039230830006999895, + 0.0038121660036267713, + 0.0038785409997217357, + 0.0041657499969005585, + 0.00418254200485535, + 0.0039974169922061265, + 0.00399745799950324 + ], + [ + 0.002031750002061017, + 0.0021370000031311065, + 0.00207441599923186, + 0.002067249995889142, + 0.002083540995954536, + 0.002197499998146668, + 0.002298207997228019, + 0.0023166250030044466, + 0.0023082499974407256, + 0.0023424999963026494, + 0.0022302089928416535, + 0.002180499999667518, + 0.007085542005370371, + 0.0048967079928843305, + 0.004621707994374447, + 0.0056168329901993275, + 0.0036321669904282317, + 0.00302025000564754, + 0.0030923750018700957, + 0.0031187919958028942, + 0.003329208993818611, + 0.003264250000938773, + 0.0032147909951163456, + 0.003165333007927984, + 0.0031352079968200997, + 0.003006167011335492, + 0.00301258300896734, + 0.0029436669865390286, + 0.0030596249998779967, + 0.003066584002226591, + 0.0030331250018207356, + 0.0032252499950118363, + 0.003131833000225015, + 0.0034127090038964525, + 0.0033035410015145317, + 0.0032601669954601675, + 0.0032764999923529103, + 0.0033585830096853897, + 0.003324207995319739, + 0.0035388339892961085, + 0.003325583995319903, + 0.0037147080001886934, + 0.003530125002725981, + 0.003620207993662916, + 0.003660959002445452, + 0.003595625006710179, + 0.0035219999990658835, + 0.003589125000871718, + 0.003704708011355251, + 0.0037316660018404946, + 0.003861958990455605, + 0.0038682919985149056, + 0.0037745000008726493, + 0.0038763750053476542, + 0.003924415999790654, + 0.00404945899208542, + 0.003963291994296014, + 0.004122166006709449, + 0.0040526249940739945, + 0.004139832992223091, + 0.004175708003458567, + 0.004228833000524901, + 0.004117457996471785, + 0.004225041993777268 + ], + [ + 0.0022955830063438043, + 0.0022635000059381127, + 0.0022384160110959783, + 0.002166082995245233, + 0.0022520409984281287, + 0.002226042008260265, + 0.002798000001348555, + 0.002442791999783367, + 0.0023480420059058815, + 0.002472082996973768, + 0.002444583995384164, + 0.002520124995498918, + 0.002551541998400353, + 0.002563500005635433, + 0.0026380830095149577, + 0.002606167006888427, + 0.0026047080027638003, + 0.002745040997979231, + 0.00278566699125804, + 0.002755499997874722, + 0.0028047079977113754, + 0.0028113330045016482, + 0.0029104170098435134, + 0.0030051250068936497, + 0.003022749995579943, + 0.0029944580019218847, + 0.003109167009824887, + 0.003124583003227599, + 0.0030375420028576627, + 0.003127625008346513, + 0.0032030839938670397, + 0.0032229999924311414, + 0.0032470000005559996, + 0.003401750000193715, + 0.003330833002110012, + 0.003310290994704701, + 0.0033021660055965185, + 0.003567792009562254, + 0.0034690840111579746, + 0.003459250001469627, + 0.003491291994578205, + 0.0036125000042375177, + 0.0035526249994290993, + 0.00369633300579153, + 0.003604167010053061, + 0.00355370799661614, + 0.0036086670006625354, + 0.0034785830066539347, + 0.003561542005627416, + 0.003886999998940155, + 0.0038156660011736676, + 0.0038864589878357947, + 0.003952208993723616, + 0.003975250001531094, + 0.00443766699754633, + 0.0040494169952580705, + 0.004130749992327765, + 0.004230541002471, + 0.0041992500046035275, + 0.004211167004541494, + 0.0043463339970912784, + 0.004307041992433369, + 0.004174374989815988, + 0.004148291001911275 + ], + [ + 0.002525042000343092, + 0.0022682090057060122, + 0.0021927909983787686, + 0.002552040998125449, + 0.0030925409955671057, + 0.002373625000473112, + 0.002818749999278225, + 0.0023442499950760975, + 0.002310832991497591, + 0.00238754200108815, + 0.002434000009088777, + 0.003136250001261942, + 0.0029326660005608574, + 0.002565208007581532, + 0.002674417002708651, + 0.0026355829904787242, + 0.0025666250003268942, + 0.003179625011398457, + 0.0028092089924030006, + 0.0028152500017313287, + 0.002799333000439219, + 0.0028880830068374053, + 0.002916417011874728, + 0.0029664999892702326, + 0.002990542008774355, + 0.0029726659995503724, + 0.003034540990483947, + 0.003499916012515314, + 0.0031430000090040267, + 0.0031646670104237273, + 0.0031643749971408397, + 0.003160207998007536, + 0.0031148339912761003, + 0.0032190000056289136, + 0.003281582990894094, + 0.003691582998726517, + 0.0033333339961245656, + 0.003376291992026381, + 0.0038504999974975362, + 0.003514665993861854, + 0.0034471669932827353, + 0.0035442920052446425, + 0.003859165997710079, + 0.003585707992897369, + 0.00400062499102205, + 0.003566500003216788, + 0.0034575410099932924, + 0.003608792001614347, + 0.0037416249979287386, + 0.003942459006793797, + 0.003839583005174063, + 0.0041412090067751706, + 0.004147125000599772, + 0.004250666999723762, + 0.003912833009962924, + 0.003975624989834614, + 0.004121541001950391, + 0.004192624997813255, + 0.004095708005479537, + 0.004024290989036672, + 0.004238458990585059, + 0.004253874998539686, + 0.004252375001669861, + 0.004197708010906354 + ], + [ + 0.0021023330045863986, + 0.002135750008164905, + 0.0022052090062061325, + 0.0024120839952956885, + 0.0022176249913172796, + 0.002169665996916592, + 0.002198292000684887, + 0.003215457996702753, + 0.0023689160007052124, + 0.0023411660076817498, + 0.0023632499942323193, + 0.0024806250003166497, + 0.002459540992276743, + 0.002579374995548278, + 0.00256979200639762, + 0.002564334005001001, + 0.0024731250014156103, + 0.002791292004985735, + 0.0026924170088022947, + 0.0027890830097021535, + 0.0027518340066308156, + 0.0028578750061569735, + 0.0029586659948108718, + 0.0028691250045085326, + 0.002817792003042996, + 0.0030072079971432686, + 0.0030165420030243695, + 0.0030837499944027513, + 0.003092208004090935, + 0.0030195829895092174, + 0.003096790998824872, + 0.003156291990308091, + 0.0031571250001434237, + 0.003297167000710033, + 0.0032563330023549497, + 0.0032561249972786754, + 0.003329750004922971, + 0.003335250003146939, + 0.0034416249982314184, + 0.0034474580024834722, + 0.0034394999966025352, + 0.0034924579958897084, + 0.0035698339925147593, + 0.0036441249976633117, + 0.0034353749942965806, + 0.003637791989604011, + 0.003761542000574991, + 0.006178583003929816, + 0.007332624998525716, + 0.005541667007491924, + 0.004109457993763499, + 0.004527124998276122, + 0.004073165997397155, + 0.004052959004184231, + 0.0038250840007094666, + 0.003944666008464992, + 0.004281458997866139, + 0.004609249997884035, + 0.004348208996816538, + 0.0042798329959623516, + 0.004289624994271435, + 0.004232832987327129, + 0.00423429100192152, + 0.004167667008005083 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "speedup_cached_vs_uncached_sdpa": 1.4590574950791493, + "speedup_cached_vs_uncached_mha": 1.668223885346276 + }, + { + "batch": 8, + "prompt_len": 128, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_mha": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 7.938166992971674, + "p95": 14.083391599706372, + "min": 5.476167003507726, + "max": 15.605749998940155, + "iqr": 1.328958009253256, + "mean": 8.73580839834176, + "cv": 0.4553608169595629 + }, + "decode_ms_per_step": { + "count": 315, + "median": 3.0513749952660874, + "p95": 6.826658209320158, + "min": 2.026084010140039, + "max": 8.85345799906645, + "iqr": 0.6502710020868108, + "mean": 3.432291571285561, + "cv": 0.4044398152388575 + }, + "total_ms": { + "count": 5, + "median": 207.45908794924617, + "p95": 337.7003101777518, + "min": 155.5350830167299, + "max": 370.1235019834712, + "iqr": 24.28187191253528, + "mean": 224.9701773893321, + "cv": 0.3731326964636763 + }, + "tokens_per_second": { + "count": 5, + "median": 2467.9564778828017, + "p95": 3190.842154976026, + "min": 1383.322045901491, + "max": 3291.861810656104, + "iqr": 325.31433321837585, + "mean": 2478.2706131466903, + "cv": 0.2821955293085176 + }, + "median_step_ms_by_index": [ + 7.938166992971674, + 3.5315829882165417, + 3.2797079911688343, + 3.222125000320375, + 3.131208010017872, + 2.989915999933146, + 3.1638750078855082, + 3.132000012556091, + 3.1740419944981113, + 3.118499997071922, + 3.0361249955603853, + 3.088833997026086, + 3.1298749963752925, + 2.881249994970858, + 3.127874995698221, + 3.1060420005815104, + 2.8677919908659533, + 3.1085000082384795, + 2.915541990660131, + 3.1057909945957363, + 3.0513749952660874, + 3.009459003806114, + 2.6356669986853376, + 3.1439160084119067, + 3.096458996878937, + 2.8882079932373017, + 3.068916004849598, + 3.083208008320071, + 3.0928330088499933, + 3.1215420021908358, + 3.0980000010458753, + 3.2001670042518526, + 2.8909169923281297, + 3.12024999584537, + 3.139458000077866, + 2.940834005130455, + 2.866583992727101, + 2.822915994329378, + 2.9297080036485568, + 2.990791996126063, + 2.9717920115217566, + 2.8942499920958653, + 3.32212500507012, + 2.873459001420997, + 3.1405409972649068, + 3.0397079972317442, + 2.9933339974377304, + 3.2185830059461296, + 2.9165000014472753, + 3.218500001821667, + 2.803124996717088, + 3.1585410033585504, + 3.3305840042885393, + 2.8931249980814755, + 3.2517499930690974, + 2.889792012865655, + 3.017584007466212, + 2.859917003661394, + 2.813624989357777, + 3.070916995056905, + 2.917417004937306, + 2.7161670004716143, + 2.7468750049592927, + 3.1603340030414984 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.006664999993517995, + 0.003105125011643395, + 0.002774959008092992, + 0.003328041988424957, + 0.002825042000040412, + 0.0032696249982109293, + 0.0028839579899795353, + 0.0029332919948501512, + 0.0028605420084204525, + 0.0030187499942258, + 0.0029005840042373165, + 0.003111833008006215, + 0.0029346250084927306, + 0.0026944999990519136, + 0.00302025000564754, + 0.0029554169886978343, + 0.0027301669906591997, + 0.0029096660000504926, + 0.002736624999670312, + 0.0026442079979460686, + 0.0028917500021634623, + 0.0029179170087445527, + 0.002545208000810817, + 0.0028923329955432564, + 0.0027258750051259995, + 0.002601625004899688, + 0.0028910420078318566, + 0.0028880000027129427, + 0.0025805409968597814, + 0.0028412910032784566, + 0.00256641699525062, + 0.002612500000395812, + 0.0028909169923281297, + 0.0029284580086823553, + 0.0025312499928986654, + 0.0028832079988205805, + 0.0025805830082390457, + 0.00252641701081302, + 0.0029140840051695704, + 0.002990791996126063, + 0.0029717920115217566, + 0.00289408300886862, + 0.0026017500058514997, + 0.0026330830005463213, + 0.0026254999975208193, + 0.002617541002109647, + 0.002772084000753239, + 0.0027142500039190054, + 0.002612666998174973, + 0.0027403749991208315, + 0.002345874992897734, + 0.002661582999280654, + 0.0033305840042885393, + 0.0026197089900961146, + 0.002961042002425529, + 0.002756916990620084, + 0.003017584007466212, + 0.0026335409929743037, + 0.002627875001053326, + 0.002617375008412637, + 0.002917417004937306, + 0.0027161670004716143, + 0.0026741249894257635, + 0.0031603340030414984 + ], + [ + 0.007938166992971674, + 0.0035315829882165417, + 0.003296374998171814, + 0.003215874996385537, + 0.003131208010017872, + 0.002989915999933146, + 0.0031638750078855082, + 0.003132000012556091, + 0.0031740419944981113, + 0.003118499997071922, + 0.003063874988583848, + 0.003024667006684467, + 0.0031298749963752925, + 0.002881249994970858, + 0.003127874995698221, + 0.0031060420005815104, + 0.0030540409934474155, + 0.0031085000082384795, + 0.0031335419917013496, + 0.0031057909945957363, + 0.0030845419969409704, + 0.003223833002266474, + 0.0026356669986853376, + 0.0031439160084119067, + 0.003096458996878937, + 0.003032166001503356, + 0.003252375012380071, + 0.003106749994913116, + 0.0030928330088499933, + 0.0031215420021908358, + 0.0031452919938601553, + 0.0033551670057931915, + 0.0030954590038163587, + 0.00312024999584537, + 0.003139458000077866, + 0.002940834005130455, + 0.002950333000626415, + 0.003270499990321696, + 0.0029297080036485568, + 0.0035354999999981374, + 0.00315541701274924, + 0.00331016699783504, + 0.003471957999863662, + 0.002873459001420997, + 0.0031405409972649068, + 0.0030397079972317442, + 0.003234874995541759, + 0.003237750002881512, + 0.00320008400012739, + 0.0032664999889675528, + 0.002803124996717088, + 0.00325858399446588, + 0.0033372500038240105, + 0.0028931249980814755, + 0.003361458992003463, + 0.0032740419992478564, + 0.0033382080000592396, + 0.003318374991067685, + 0.0031604999967385083, + 0.004521833005128428, + 0.0032187920005526394, + 0.0033495829993626103, + 0.0032723329932196066, + 0.0032702919997973368 + ], + [ + 0.007993958002771251, + 0.0035519590019248426, + 0.0032797079911688343, + 0.003222125000320375, + 0.003213375006453134, + 0.002961624995805323, + 0.0032555420038988814, + 0.0032092920009745285, + 0.003254124996601604, + 0.0031686670117778704, + 0.0030361249955603853, + 0.003088833997026086, + 0.0031975419988157228, + 0.0029997499950695783, + 0.0032584169966867194, + 0.0031456669967155904, + 0.0028677919908659533, + 0.0032608749897917733, + 0.002915541990660131, + 0.0032858750055311248, + 0.0030513749952660874, + 0.003009459003806114, + 0.0033380420063622296, + 0.003208833993994631, + 0.003206542009138502, + 0.0028882079932373017, + 0.003068916004849598, + 0.003083208008320071, + 0.0032871250004973263, + 0.0032645419996697456, + 0.0030980000010458753, + 0.0032001670042518526, + 0.0028870840033050627, + 0.003333875007228926, + 0.0033453340001869947, + 0.0030392499902518466, + 0.002866583992727101, + 0.002822915994329378, + 0.003158291001454927, + 0.002984040998853743, + 0.002894375007599592, + 0.0028942499920958653, + 0.00332212500507012, + 0.003317708004033193, + 0.003460207997704856, + 0.003268249987741001, + 0.0029933339974377304, + 0.0032185830059461296, + 0.0029165000014472753, + 0.003218500001821667, + 0.003295040994998999, + 0.0031585410033585504, + 0.0033139169972855598, + 0.0032803749927552417, + 0.0034447499929228798, + 0.002889792012865655, + 0.0032187089964281768, + 0.003186166999512352, + 0.0032540419924771413, + 0.0035970409953733906, + 0.0032579999970039353, + 0.00328024999180343, + 0.0032002500083763152, + 0.00326979199599009 + ], + [ + 0.015605749998940155, + 0.00885345799906645, + 0.007278290999238379, + 0.006000666995532811, + 0.006274832994677126, + 0.007109999991371296, + 0.006944708002265543, + 0.00775416599935852, + 0.0055337920057354495, + 0.005624749988783151, + 0.006388875000993721, + 0.0061221249925438315, + 0.006667166991974227, + 0.00767958301003091, + 0.006021291992510669, + 0.006715708004776388, + 0.00661604201013688, + 0.0066106250014854595, + 0.006656833997112699, + 0.006829167003161274, + 0.00799158398876898, + 0.007353542008786462, + 0.007754917009151541, + 0.006825583011959679, + 0.006545833995915018, + 0.006784249999327585, + 0.006814791995566338, + 0.008275542000774294, + 0.007075750007061288, + 0.007512083990150131, + 0.006883500012918375, + 0.007150749996071681, + 0.006903208006406203, + 0.006684624997433275, + 0.006216499998117797, + 0.006402083003195003, + 0.006754792004358023, + 0.006498332993942313, + 0.006716292002238333, + 0.006253125000512227, + 0.00593170800129883, + 0.004992832997231744, + 0.004843208007514477, + 0.004740792006487027, + 0.004548166005406529, + 0.004361125000286847, + 0.003991374993347563, + 0.0039114999963203445, + 0.003744208996067755, + 0.003750457995920442, + 0.003629749990068376, + 0.00361466700269375, + 0.003533666007569991, + 0.003316624992294237, + 0.0032517499930690974, + 0.0029816669994033873, + 0.003001582997967489, + 0.002859917003661394, + 0.002813624989357777, + 0.003070916995056905, + 0.0028443749906728044, + 0.002400292010861449, + 0.0027468750049592927, + 0.0025575000036042184 + ], + [ + 0.005476167003507726, + 0.002829249991918914, + 0.002636791003169492, + 0.002737000002525747, + 0.002386375010246411, + 0.002585541005828418, + 0.0025464580103289336, + 0.0023801659990567714, + 0.0022294170048553497, + 0.002141083008609712, + 0.0025163329992210492, + 0.002402415993856266, + 0.0026955829962389544, + 0.002380374993663281, + 0.0023474999907193705, + 0.002446375001454726, + 0.0023606670001754537, + 0.0024665000091772527, + 0.0025357090053148568, + 0.0024264579988084733, + 0.0024802499974612147, + 0.002503916999557987, + 0.002553250000346452, + 0.002107957989210263, + 0.0025120420032180846, + 0.002440291005768813, + 0.0024729590077186003, + 0.0021383330022217706, + 0.0025226669968105853, + 0.002538792003178969, + 0.0020556670060614124, + 0.0025074580044019967, + 0.0025339589919894934, + 0.002026084010140039, + 0.0020268750085961074, + 0.0024137500004144385, + 0.0024895410024328157, + 0.002093292001518421, + 0.002127041996573098, + 0.002542999995057471, + 0.0025213329936377704, + 0.002187749996664934, + 0.002611749994684942, + 0.002126416002283804, + 0.00250508300086949, + 0.0023036670027067885, + 0.002583875000709668, + 0.002111000008881092, + 0.002234999992651865, + 0.0021269589924486354, + 0.002642542007379234, + 0.002065958993625827, + 0.002194624990806915, + 0.0025347909977426752, + 0.0025615829945309088, + 0.002166375008528121, + 0.00225575000513345, + 0.0025152499874820933, + 0.002259791988763027, + 0.0022390829981304705, + 0.0021967920038150623, + 0.0025447499938309193, + 0.0021956670098006725, + 0.0022360000002663583 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + ] + } + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 7.275083000422455, + "p95": 7.845241602626629, + "min": 4.958040997735225, + "max": 7.933375003631227, + "iqr": 1.8282910023117438, + "mean": 6.664724799338728, + "cv": 0.19245630487471593 + }, + "decode_ms_per_step": { + "count": 315, + "median": 7.681666989810765, + "p95": 10.858782997820526, + "min": 5.108249999466352, + "max": 11.957208000239916, + "iqr": 2.7178125019418076, + "mean": 8.067891124029837, + "cv": 0.20554873114311137 + }, + "total_ms": { + "count": 5, + "median": 523.8958270056173, + "p95": 626.0665224312106, + "min": 407.2447110374924, + "max": 637.0839140436146, + "iqr": 157.50903598382138, + "mean": 514.9418656132184, + "cv": 0.19243076162576006 + }, + "tokens_per_second": { + "count": 5, + "median": 977.293525940817, + "p95": 1247.0153087131155, + "min": 803.6617919769806, + "max": 1257.2293417774147, + "iqr": 326.4294893195623, + "mean": 1024.8147046574975, + "cv": 0.19461914598999935 + }, + "median_step_ms_by_index": [ + 7.275083000422455, + 7.6392920018406585, + 7.4293750076321885, + 7.729832999757491, + 7.801416009897366, + 7.769500007270835, + 7.911624998087063, + 8.093374999589287, + 7.945249992189929, + 7.962834002682939, + 8.0666659923736, + 8.127707988023758, + 8.168249987647869, + 8.071500007645227, + 8.296750005683862, + 8.165250008460134, + 8.401749990298413, + 9.38283400319051, + 8.831250001094304, + 8.496875001583248, + 8.935958001529798, + 8.278624998638406, + 8.4220419958001, + 7.663333002710715, + 7.434792001731694, + 7.071499989251606, + 7.721667003352195, + 7.026958002825268, + 7.893750007497147, + 7.249458009027876, + 7.7308750041993335, + 7.161957997595891, + 7.607374995131977, + 7.345082995016128, + 7.478458996047266, + 7.382582989521325, + 7.382499999948777, + 7.44762500107754, + 7.388791011180729, + 7.434040991938673, + 7.409249999909662, + 7.600416007335298, + 7.457667001290247, + 7.497500002500601, + 7.600832992466167, + 7.553374991402961, + 7.520082988776267, + 7.598915995913558, + 8.079917010036297, + 7.572959002573043, + 7.621791010024026, + 7.586541003547609, + 7.657499998458661, + 7.674292006413452, + 7.578624994494021, + 7.667166006285697, + 7.510874987929128, + 7.808624999597669, + 7.6353339973138645, + 7.83904199488461, + 7.567874999949709, + 8.64579199696891, + 7.6202919881325215, + 7.893667003372684 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.005664416996296495, + 0.0058935420092893764, + 0.005545416992390528, + 0.005986791002214886, + 0.005845374995260499, + 0.005677791996276937, + 0.006094249998568557, + 0.0057029589952435344, + 0.0057768340047914535, + 0.006032249992131256, + 0.005800832994282246, + 0.005955708009423688, + 0.006815582994022407, + 0.005791125004179776, + 0.006066916001145728, + 0.006013999998685904, + 0.005956709006568417, + 0.006512292005936615, + 0.006065124995075166, + 0.0061201659991638735, + 0.006301957997493446, + 0.006119083001976833, + 0.006389749993104488, + 0.006270625002798624, + 0.006195374997332692, + 0.006644583001616411, + 0.006422582999221049, + 0.006343540997477248, + 0.00633808400016278, + 0.006293291997280903, + 0.006574667000677437, + 0.006261709000682458, + 0.006358124999678694, + 0.006763708996004425, + 0.006521417002659291, + 0.006840208006906323, + 0.006531667007948272, + 0.006603959001949988, + 0.0068967090046498924, + 0.006601625005714595, + 0.0067850409977836534, + 0.0067306249984540045, + 0.0068242499983171, + 0.007397707988275215, + 0.006914917001267895, + 0.007553374991402961, + 0.006901000000652857, + 0.007220250001410022, + 0.006918208004208282, + 0.007041165998089127, + 0.0074825420015258715, + 0.007097209003404714, + 0.007657499998458661, + 0.007120458991266787, + 0.007446832998539321, + 0.007347582999500446, + 0.007321375000174157, + 0.007669875005376525, + 0.007232332995044999, + 0.00774579199787695, + 0.007469625008525327, + 0.00864579199696891, + 0.0074800420115934685, + 0.007893667003372684 + ], + [ + 0.007275083000422455, + 0.007942458003526554, + 0.007681666989810765, + 0.00786875000630971, + 0.007960999995702878, + 0.008164999992004596, + 0.0082315410109004, + 0.008093374999589287, + 0.008273583007394336, + 0.00796283400268294, + 0.009093000000575557, + 0.008127707988023758, + 0.009065666992682964, + 0.008320291992276907, + 0.009103333999519236, + 0.00852800000575371, + 0.00938208399747964, + 0.00938283400319051, + 0.009239250008249655, + 0.010797374998219311, + 0.008935958001529798, + 0.010245124998618849, + 0.009045207989402115, + 0.009999166999477893, + 0.01003050000872463, + 0.00961125000321772, + 0.010767375002615154, + 0.00954712500970345, + 0.010213583998847753, + 0.010819582996191457, + 0.009378791000926867, + 0.010752083995612338, + 0.009554624994052574, + 0.01054716699582059, + 0.01102754200110212, + 0.01002487500954885, + 0.010916833009105176, + 0.010731125003076158, + 0.01010637498984579, + 0.011013750001438893, + 0.010848582998733036, + 0.010545292010647245, + 0.011461415997473523, + 0.010745499996119179, + 0.010687207992305048, + 0.011957208000239916, + 0.010627249997924082, + 0.010662333006621338, + 0.011603750012000091, + 0.010882582995691337, + 0.01011537500016857, + 0.011600958998315036, + 0.010213708010269329, + 0.010456749994773418, + 0.01123145800374914, + 0.010136333003174514, + 0.011049083012039773, + 0.011005292006302625, + 0.010345582995796576, + 0.01162175000354182, + 0.011582958002691157, + 0.010778915995615534, + 0.01186433300608769, + 0.01129741600016132 + ], + [ + 0.007933375003631227, + 0.0076392920018406585, + 0.00793395799701102, + 0.007729832999757491, + 0.007801416009897366, + 0.007769500007270835, + 0.007911624998087063, + 0.008118249999824911, + 0.007945249992189929, + 0.008265500000561588, + 0.0080666659923736, + 0.008458374999463558, + 0.008168249987647869, + 0.008071500007645227, + 0.008296750005683862, + 0.008165250008460134, + 0.008401749990298413, + 0.00947608299611602, + 0.008831250001094304, + 0.008496875001583248, + 0.009212582997861318, + 0.008278624998638406, + 0.009837667006650008, + 0.010047000003396533, + 0.009241124993423, + 0.009079958006623201, + 0.009148541998001747, + 0.009072624990949407, + 0.009309958011726849, + 0.00935816700803116, + 0.009559250000165775, + 0.009612583991838619, + 0.009009665998746641, + 0.009875208997982554, + 0.009452957994653843, + 0.009717332999571227, + 0.009258417005185038, + 0.009418624991667457, + 0.009527165995677933, + 0.008973292002337985, + 0.009753165999427438, + 0.009184875001665205, + 0.009824749999097548, + 0.009647749990108423, + 0.009395416986080818, + 0.009890582994557917, + 0.009155124993412755, + 0.009865832995274104, + 0.009483208996243775, + 0.009806624992052093, + 0.00952487500035204, + 0.009581042002537288, + 0.009693500003777444, + 0.009617125004297122, + 0.009570666996296495, + 0.009295000010752119, + 0.009477500003413297, + 0.009799583000130951, + 0.009640625008614734, + 0.010118958001839928, + 0.009771583005203865, + 0.009840250000706874, + 0.009774749996722676, + 0.009812667005462572 + ], + [ + 0.007492707998608239, + 0.00782612498733215, + 0.0074293750076321885, + 0.008242000010795891, + 0.00801362500351388, + 0.008414541007368825, + 0.008048583011259325, + 0.009259957994800061, + 0.009452499987673946, + 0.010143832987523638, + 0.010407041991129518, + 0.010230749991023913, + 0.01012170800822787, + 0.01084604200150352, + 0.010272250001435168, + 0.010325750001356937, + 0.010430292008095421, + 0.009542125000734814, + 0.011091166001278907, + 0.009009000001242384, + 0.01021216700610239, + 0.00887587500619702, + 0.0084220419958001, + 0.007663333002710715, + 0.007434792001731694, + 0.007071499989251606, + 0.007721667003352195, + 0.007026958002825268, + 0.007893750007497147, + 0.007249458009027876, + 0.0077308750041993335, + 0.007161957997595891, + 0.007607374995131977, + 0.007345082995016128, + 0.007478458996047266, + 0.007382582989521325, + 0.007382499999948777, + 0.00744762500107754, + 0.007388791011180729, + 0.007434040991938673, + 0.007409249999909662, + 0.007600416007335298, + 0.007457667001290247, + 0.007497500002500601, + 0.007600832992466167, + 0.007392458996037021, + 0.007520082988776267, + 0.007598915995913558, + 0.008079917010036297, + 0.007572959002573043, + 0.007621791010024026, + 0.007586541003547609, + 0.007603165999171324, + 0.007674292006413452, + 0.007578624994494021, + 0.007667166006285697, + 0.007510874987929128, + 0.007808624999597669, + 0.0076353339973138645, + 0.00783904199488461, + 0.007567874999949709, + 0.008249416001490317, + 0.0075394580053398386, + 0.00775341599364765 + ], + [ + 0.004958040997735225, + 0.005108249999466352, + 0.005485375004354864, + 0.005152582991286181, + 0.0051880829996662214, + 0.0052279590017860755, + 0.005324124998878688, + 0.005353458007448353, + 0.00524033400870394, + 0.005321707998518832, + 0.005947334007942118, + 0.0054163339955266565, + 0.00544779199117329, + 0.005829708999954164, + 0.005417541993665509, + 0.005414166997070424, + 0.0058857499971054494, + 0.005884083002456464, + 0.0056213750067399815, + 0.006150333007099107, + 0.006365666995407082, + 0.006254040999920107, + 0.006093457996030338, + 0.006291707992204465, + 0.006520540991914459, + 0.006042000008164905, + 0.005829374989843927, + 0.006270749989198521, + 0.006653375006862916, + 0.006415334006305784, + 0.006767124999896623, + 0.005892166998819448, + 0.006761374999769032, + 0.006598540989216417, + 0.006524500000523403, + 0.006723166996380314, + 0.006480958007159643, + 0.00692650000564754, + 0.006392334005795419, + 0.006689750007353723, + 0.006215541987330653, + 0.006338874998618849, + 0.006951583010959439, + 0.006735167000442743, + 0.006890832999488339, + 0.00686145800864324, + 0.006870209006592631, + 0.006923250010004267, + 0.006760916992789134, + 0.006877917010569945, + 0.007411708997096866, + 0.0067241250071674585, + 0.006977666009333916, + 0.006912791999639012, + 0.007279666009708308, + 0.0073229170084232464, + 0.006694165989756584, + 0.00757341600547079, + 0.00721250000060536, + 0.007591249988763593, + 0.0073520840087439865, + 0.007720834008068778, + 0.0076202919881325215, + 0.0075585419981507584 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_mha": { + "first_token_ms": { + "count": 5, + "median": 6.398374986019917, + "p95": 7.923475006828085, + "min": 6.302666995907202, + "max": 8.212750006350689, + "iqr": 0.4210000188322738, + "mean": 6.805108397384174, + "cv": 0.11875100374966613 + }, + "decode_ms_per_step": { + "count": 315, + "median": 8.002832997590303, + "p95": 11.83839620498475, + "min": 5.975124993710779, + "max": 20.01874998677522, + "iqr": 2.1740619995398447, + "mean": 8.52488864445135, + "cv": 0.26280997523129834 + }, + "total_ms": { + "count": 5, + "median": 566.7709990229923, + "p95": 608.4716832498088, + "min": 446.8485029647127, + "max": 610.2634990675142, + "iqr": 107.12637602409814, + "mean": 543.8730929978192, + "cv": 0.130382318650367 + }, + "tokens_per_second": { + "count": 5, + "median": 903.3630882359765, + "p95": 1123.854531249511, + "min": 838.9818509256062, + "max": 1145.802205004662, + "iqr": 184.5816535137161, + "mean": 955.1386326220685, + "cv": 0.13832197635865762 + }, + "median_step_ms_by_index": [ + 6.398374986019917, + 6.458207994000986, + 6.547457989654504, + 6.591166005819105, + 6.7784170096274465, + 6.656874989857897, + 6.468791005318053, + 6.713624999974854, + 6.604417008929886, + 6.894834004924633, + 6.870540993986651, + 6.7784170096274465, + 7.10800000524614, + 6.6578749974723905, + 7.048917002975941, + 6.894833000842482, + 6.932332995347679, + 7.3477079859003425, + 6.804708988056518, + 7.155290993978269, + 7.121499991626479, + 7.301500008907169, + 7.281666999915615, + 7.142124988604337, + 7.332624998525716, + 7.271084003150463, + 7.530417002271861, + 7.522458996390924, + 7.47883299482055, + 7.463417001417838, + 7.586457999423146, + 7.754917009151541, + 7.387292003841139, + 7.607374995131977, + 7.515916004194878, + 7.699916997808032, + 7.696959000895731, + 7.955791996209882, + 7.985250005731359, + 9.201165987178683, + 8.640250001917593, + 8.02562500757631, + 7.95099999231752, + 8.633999997982755, + 9.120000002440065, + 9.57479199860245, + 9.747292002430186, + 9.652250009821728, + 9.363875011331402, + 9.616416005883366, + 9.814874996664003, + 9.156582993455231, + 9.792625001864508, + 9.257666999474168, + 9.43908400950022, + 9.57854199805297, + 9.032957997987978, + 9.874666997347958, + 8.639666993985884, + 9.77308300207369, + 9.65962499321904, + 9.800917003303766, + 10.334791004424915, + 9.909290994983166 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.006345374989905395, + 0.006458042000303976, + 0.006547457989654504, + 0.006591166005819105, + 0.0067784170096274465, + 0.006656874989857897, + 0.006468791005318053, + 0.006713624999974854, + 0.006604417008929886, + 0.006894834004924633, + 0.006870540993986651, + 0.0067784170096274465, + 0.007065083002089523, + 0.006412290997104719, + 0.007048917002975941, + 0.006894833000842482, + 0.006932332995347679, + 0.007234708013129421, + 0.006804708988056518, + 0.007155290993978269, + 0.007063874989398755, + 0.007301500008907169, + 0.007281666999915615, + 0.007142124988604337, + 0.007332624998525716, + 0.007271084003150463, + 0.007530417002271861, + 0.007522458996390924, + 0.00747883299482055, + 0.007463417001417838, + 0.007586457999423146, + 0.007476750004570931, + 0.007387292003841139, + 0.007518333004554734, + 0.007515916004194878, + 0.007699916997808032, + 0.007696959000895731, + 0.007955791996209882, + 0.007985250005731359, + 0.009355165995657444, + 0.008640250001917593, + 0.00802562500757631, + 0.007527666995883919, + 0.008633999997982755, + 0.007646792000741698, + 0.008279458998003975, + 0.008103959000436589, + 0.008358999999472871, + 0.008185999991837889, + 0.008548625002731569, + 0.008509042003424838, + 0.008266040997114033, + 0.00809024999034591, + 0.008357375001651235, + 0.00810804199136328, + 0.009251291994587518, + 0.008376124998903833, + 0.009165665993350558, + 0.008639666993985884, + 0.00977308300207369, + 0.00965962499321904, + 0.009800917003303766, + 0.010010541998781264, + 0.00939704199845437 + ], + [ + 0.006766375008737668, + 0.007028958003502339, + 0.0077862909965915605, + 0.007576333999168128, + 0.007691167003940791, + 0.0075788750109495595, + 0.007352416010689922, + 0.007164916998590343, + 0.007187291994341649, + 0.008190499996999279, + 0.007930459003546275, + 0.008091458003036678, + 0.007847166998544708, + 0.008095707991742529, + 0.007821624996722676, + 0.008142042002873495, + 0.007975000000442378, + 0.008813499996904284, + 0.008170000000973232, + 0.008603040987509303, + 0.007790584000758827, + 0.009319124990724958, + 0.008015416009584442, + 0.008749583997996524, + 0.008088250004220754, + 0.008458040989353321, + 0.008538833004422486, + 0.00831366700003855, + 0.010506125006941147, + 0.00930020899977535, + 0.009480708002229221, + 0.009307790998718701, + 0.009448374999919906, + 0.009497624996583909, + 0.01046979200327769, + 0.010362125001847744, + 0.010936999999103136, + 0.011486540999612771, + 0.011369999992894009, + 0.010850667007616721, + 0.01063554099528119, + 0.010241666997899301, + 0.010447624998050742, + 0.010715208991314285, + 0.01006916600454133, + 0.010753958005807362, + 0.010797833005199209, + 0.01027875000727363, + 0.011008500005118549, + 0.01061124999250751, + 0.010886415999266319, + 0.011215874998015352, + 0.0106735420122277, + 0.011217749997740611, + 0.011286875000223517, + 0.010899625005549751, + 0.011414000007789582, + 0.011487208001199178, + 0.01124212499416899, + 0.011657917013508268, + 0.01171295800304506, + 0.0115716670115944, + 0.011592625000048429, + 0.011741834008716978 + ], + [ + 0.008212750006350689, + 0.008365707995835692, + 0.008172415997250937, + 0.008207291990402155, + 0.008202375000109896, + 0.008092499992926605, + 0.008234334003645927, + 0.007774166006129235, + 0.008142332997522317, + 0.008171209003194235, + 0.008299166991491802, + 0.007812583004124463, + 0.008368957991478965, + 0.00825416600855533, + 0.00830600000335835, + 0.008184416990843602, + 0.008431291003944352, + 0.008268457997473888, + 0.008539583999663591, + 0.008249125006841496, + 0.008338375002495013, + 0.007887291998486035, + 0.008328124997206032, + 0.008002832997590303, + 0.00869362500088755, + 0.008121207996737212, + 0.008675959004904144, + 0.008261584007414058, + 0.00855479198799003, + 0.008446874999208376, + 0.00860787500278093, + 0.008298125001601875, + 0.008616207996965386, + 0.008516166009940207, + 0.009455999999772757, + 0.009274875003029592, + 0.009522541004116647, + 0.009202041997923516, + 0.009166250005364418, + 0.009201165987178683, + 0.00939104100689292, + 0.010186209008679725, + 0.010617958992952481, + 0.009707750010420568, + 0.009419750000233762, + 0.009751291989232413, + 0.009747292002430186, + 0.009652250009821728, + 0.009363875011331402, + 0.009616416005883366, + 0.009814874996664003, + 0.009156582993455231, + 0.009792625001864508, + 0.009257666999474168, + 0.00943908400950022, + 0.00957854199805297, + 0.009032957997987978, + 0.009874666997347958, + 0.008620667009381577, + 0.009134124993579462, + 0.008721749996766448, + 0.009188791998894885, + 0.010334791004424915, + 0.009909290994983166 + ], + [ + 0.006302666995907202, + 0.006458207994000986, + 0.006470207998063415, + 0.006306416995357722, + 0.006510999999591149, + 0.006323666995740496, + 0.006415292009478435, + 0.006339458006550558, + 0.006396958997356705, + 0.0065089170093415305, + 0.006485042002168484, + 0.006537375011248514, + 0.00710800000524614, + 0.0066578749974723905, + 0.006551375001436099, + 0.006822916999226436, + 0.006322875007754192, + 0.0073477079859003425, + 0.006663500011200085, + 0.006977541008382104, + 0.007121499991626479, + 0.006776333990273997, + 0.00722091700299643, + 0.006934292003279552, + 0.007269250010722317, + 0.007155333005357534, + 0.007208791997982189, + 0.007288458000402898, + 0.007059083000058308, + 0.007430375000694767, + 0.007060375006403774, + 0.007754917009151541, + 0.007095458000549115, + 0.007607374995131977, + 0.0072504169947933406, + 0.007496249992982484, + 0.0073012499924516305, + 0.0074825839983532205, + 0.007336959009990096, + 0.0071841249882709235, + 0.007221209001727402, + 0.007516832993132994, + 0.00795099999231752, + 0.008253374995547347, + 0.009120000002440065, + 0.00957479199860245, + 0.011470832992927171, + 0.012063707996276207, + 0.012114333003410138, + 0.0106263749912614, + 0.013762792004854418, + 0.01382045799982734, + 0.01325070799794048, + 0.014822540993918665, + 0.01517674999195151, + 0.01555587501206901, + 0.015890124996076338, + 0.016865874989889562, + 0.017081042009522207, + 0.017507958997157402, + 0.017180084003484808, + 0.018437999999150634, + 0.019479958005831577, + 0.02001874998677522 + ], + [ + 0.006398374986019917, + 0.006382874998962507, + 0.005975124993710779, + 0.006309374992270023, + 0.006207749989698641, + 0.006156125004054047, + 0.006068791000870988, + 0.006096208991948515, + 0.006201124997460283, + 0.006251958999200724, + 0.006260540991206653, + 0.006270124998991378, + 0.00630795799952466, + 0.0063487500010523945, + 0.006336124992230907, + 0.0063036669889697805, + 0.00637733300391119, + 0.006497416994534433, + 0.006464874997618608, + 0.006665207998594269, + 0.006591833007405512, + 0.006645834000664763, + 0.006616333004785702, + 0.006603667003219016, + 0.006678333986201324, + 0.006760999996913597, + 0.006837833003373817, + 0.006757542010745965, + 0.006645917004789226, + 0.006371499999659136, + 0.006812166990130208, + 0.006828957994002849, + 0.006900666005094536, + 0.007151833997340873, + 0.007367041005636565, + 0.007124208001187071, + 0.007077375004882924, + 0.007136667001759633, + 0.007085707999067381, + 0.007163750007748604, + 0.007193249999545515, + 0.007362499993178062, + 0.007327166997129098, + 0.007404000003589317, + 0.007351624997681938, + 0.007463375004590489, + 0.007510042007197626, + 0.007496542006265372, + 0.007615125010488555, + 0.007547583998530172, + 0.007582541991723701, + 0.0075990829936927184, + 0.007625458005350083, + 0.007751416997052729, + 0.00788204200216569, + 0.00759904200094752, + 0.007686249999096617, + 0.007851416987250559, + 0.008040167012950405, + 0.007910166008514352, + 0.008007791999261826, + 0.008018417007406242, + 0.007974749998538755, + 0.008010874997125939 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "speedup_cached_vs_uncached_sdpa": 2.52529707030133, + "speedup_cached_vs_uncached_mha": 2.731965153349416 + }, + { + "batch": 8, + "prompt_len": 512, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_mha": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 27.43995799391996, + "p95": 28.828741793404333, + "min": 23.11920801002998, + "max": 28.957499991520308, + "iqr": 2.45950001408346, + "mean": 26.73691679665353, + "cv": 0.08727158504999466 + }, + "decode_ms_per_step": { + "count": 315, + "median": 4.580207998515107, + "p95": 6.361666400334798, + "min": 3.1268339953385293, + "max": 9.298792007029988, + "iqr": 0.9813335054786876, + "mean": 4.744406603184337, + "cv": 0.1897953776293421 + }, + "total_ms": { + "count": 5, + "median": 334.8962469754042, + "p95": 352.69065600878093, + "min": 284.1223340219585, + "max": 352.7632880141027, + "iqr": 48.409461000119336, + "mean": 325.63453279726673, + "cv": 0.09377479594421513 + }, + "tokens_per_second": { + "count": 5, + "median": 1528.8317042191363, + "p95": 1778.4849795945331, + "min": 1451.3981964572554, + "max": 1802.040665907066, + "iqr": 231.36832387973118, + "mean": 1583.8853422785057, + "cv": 0.09750854678009813 + }, + "median_step_ms_by_index": [ + 27.43995799391996, + 8.229041995946318, + 6.360790997860022, + 6.343207991449162, + 6.262625000090338, + 5.917125003179535, + 6.000458000926301, + 5.811208000523038, + 5.53750000835862, + 5.933082997216843, + 4.554583007120527, + 4.9425410106778145, + 4.547457996523008, + 5.3148750012042, + 4.580207998515107, + 4.530999998678453, + 4.561249996186234, + 5.222540989052504, + 4.498833994148299, + 4.683874998590909, + 4.514583997661248, + 4.495500004850328, + 4.49187500635162, + 4.650167000363581, + 5.178874998819083, + 4.473667009733617, + 4.500082999584265, + 4.872292003710754, + 4.944792002788745, + 4.586208000546321, + 4.556542000500485, + 4.98345801315736, + 4.5748749980703, + 4.62383299600333, + 5.0839159957831725, + 4.556874991976656, + 4.501583011006005, + 4.5503339933929965, + 5.297832991345786, + 4.689458000939339, + 4.561333000310697, + 4.987583990441635, + 5.002124991733581, + 4.422124999109656, + 4.4472080044215545, + 5.0469580019125715, + 4.564249989925884, + 4.504541997448541, + 4.752709006424993, + 4.72474998969119, + 4.349000009824522, + 4.4975000055273995, + 4.457749993889593, + 4.608250004821457, + 4.261207999661565, + 4.620541003532708, + 4.633665987057611, + 4.217583991703577, + 4.46887499128934, + 4.789999991771765, + 4.457041010027751, + 4.337791993748397, + 4.3165410024812445, + 4.50887500483077 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.02311920801002998, + 0.00564854200638365, + 0.005711125006200746, + 0.005003666010452434, + 0.005178999999770895, + 0.00459450000198558, + 0.004729209002107382, + 0.006388290988979861, + 0.008264582997071557, + 0.006032917008269578, + 0.0036406670114956796, + 0.0049425410106778145, + 0.0035953339975094423, + 0.0033915410022018477, + 0.0035029159917030483, + 0.0034194170002592728, + 0.0034216670028399676, + 0.003510166992782615, + 0.003303874997072853, + 0.0035176669916836545, + 0.0034204580006189644, + 0.003569167005480267, + 0.003683375005493872, + 0.0034405420010443777, + 0.0037162919907132164, + 0.003647999998065643, + 0.003660957998363301, + 0.0034774579980876297, + 0.003622791002271697, + 0.0038631249917671084, + 0.003860334007185884, + 0.0035856250033248216, + 0.0038009169948054478, + 0.0038804590003564954, + 0.003831417008768767, + 0.0037905420031165704, + 0.003529499997966923, + 0.0038877909973962232, + 0.004051833995617926, + 0.004095165990293026, + 0.003599457995733246, + 0.004374666008516215, + 0.00416758400388062, + 0.0038656249962514266, + 0.004406041989568621, + 0.0045697079913225025, + 0.004254915998899378, + 0.003918625006917864, + 0.0038230830105021596, + 0.004136917006690055, + 0.004173499997705221, + 0.0038484159886138514, + 0.0038557079969905317, + 0.003944166994187981, + 0.004171874999883585, + 0.0038093750044936314, + 0.003748959003132768, + 0.00412233300448861, + 0.004108332999749109, + 0.004207999998470768, + 0.004496334004215896, + 0.004337791993748397, + 0.004339459002949297, + 0.00450887500483077 + ], + [ + 0.028957499991520308, + 0.008410916998400353, + 0.007101291994331405, + 0.007799334009177983, + 0.006262625000090338, + 0.005917125003179535, + 0.006096708006225526, + 0.0056459999905200675, + 0.00486516700766515, + 0.005203458000323735, + 0.004554583007120527, + 0.004685040999902412, + 0.004547457996523008, + 0.005331916996510699, + 0.005596332994173281, + 0.004558540997095406, + 0.005361375006032176, + 0.005339458002708852, + 0.004498833994148299, + 0.005555666008149274, + 0.0051437499932944775, + 0.004688416011049412, + 0.004561916008242406, + 0.004575541999656707, + 0.0041249999922001734, + 0.004317250000895001, + 0.00422512499790173, + 0.0042285829986212775, + 0.004113249990041368, + 0.004450041989912279, + 0.004556542000500485, + 0.004500208990066312, + 0.0045748749980703, + 0.00462383299600333, + 0.004711708999820985, + 0.004451667002285831, + 0.0047117500071180984, + 0.0045503339933929965, + 0.0053157079964876175, + 0.004922082996927202, + 0.004720958997495472, + 0.005134708000696264, + 0.005002124991733581, + 0.004381374994409271, + 0.005106208001961932, + 0.0050469580019125715, + 0.0044419170008040965, + 0.0041876670002238825, + 0.00421858299523592, + 0.00422962500306312, + 0.004349000009824522, + 0.0044975000055273995, + 0.004457749993889593, + 0.004314250007155351, + 0.00438383300206624, + 0.004490208011702634, + 0.004211790990666486, + 0.004170582993538119, + 0.00446887499128934, + 0.004512208994128741, + 0.004457041010027751, + 0.00414962500508409, + 0.004125708001083694, + 0.004200833005597815 + ], + [ + 0.028313709000940435, + 0.009298792007029988, + 0.006360790997860022, + 0.006343207991449162, + 0.006742875004420057, + 0.006363709006109275, + 0.006000458000926301, + 0.005811208000523038, + 0.005687374999979511, + 0.005933082997216843, + 0.005636332993162796, + 0.006436415991629474, + 0.005857458992977627, + 0.006482249998953193, + 0.005311041997629218, + 0.005454917001770809, + 0.004561708003166132, + 0.005222540989052504, + 0.004689708992373198, + 0.004683874998590909, + 0.004514583997661248, + 0.004495500004850328, + 0.004539874993497506, + 0.004650167000363581, + 0.005344332996173762, + 0.004581374989356846, + 0.004500082999584265, + 0.0052910830127075315, + 0.005410041005234234, + 0.00473150001198519, + 0.004492124993703328, + 0.005346707999706268, + 0.005378665999160148, + 0.004503165997448377, + 0.00536495799315162, + 0.005366583005525172, + 0.004385624997667037, + 0.005320042007951997, + 0.005476709004142322, + 0.005479458006448112, + 0.005377499997848645, + 0.0050245419988641515, + 0.004645624998374842, + 0.0044752079993486404, + 0.00501916601206176, + 0.00511749999714084, + 0.004566916002659127, + 0.004760833995533176, + 0.004752709006424993, + 0.004898458006209694, + 0.004329540999606252, + 0.004917999991448596, + 0.004382167011499405, + 0.004916709003737196, + 0.004261207999661565, + 0.004620541003532708, + 0.00486366699624341, + 0.004217583991703577, + 0.004608250004821457, + 0.004849625009228475, + 0.004205082994303666, + 0.0046287080040201545, + 0.0043165410024812445, + 0.004643167005269788 + ], + [ + 0.025854208986856975, + 0.003816875003394671, + 0.0031268339953385293, + 0.003453749988693744, + 0.0035502079990692437, + 0.0036758750065928325, + 0.0033817910007201135, + 0.0032555840007262304, + 0.0035528329899534583, + 0.003838874996290542, + 0.003401666006539017, + 0.003610875006415881, + 0.003643207994173281, + 0.0034306250017834827, + 0.003923166994354688, + 0.0037172079901210964, + 0.0036846250004600734, + 0.0038806249940535054, + 0.0038440839998656884, + 0.004371458009700291, + 0.004088792004040442, + 0.004482541000470519, + 0.004299708001781255, + 0.005131958008860238, + 0.005513084004633129, + 0.004657874989788979, + 0.004818542001885362, + 0.004872292003710754, + 0.004944792002788745, + 0.004898500003037043, + 0.004989583001588471, + 0.00498345801315736, + 0.004727667008410208, + 0.0049376250099157915, + 0.0050839159957831725, + 0.004556874991976656, + 0.004501583011006005, + 0.004521583003224805, + 0.004520582995610312, + 0.004655749988160096, + 0.004515708991675638, + 0.004846167008508928, + 0.005020583994337358, + 0.004422124999109656, + 0.0044472080044215545, + 0.004216790999635123, + 0.004564249989925884, + 0.004610208998201415, + 0.005521708008018322, + 0.004975083007593639, + 0.004536333988653496, + 0.005121000009239651, + 0.005114125000545755, + 0.004608250004821457, + 0.004284791997633874, + 0.0076204590004635975, + 0.00585862499428913, + 0.005622957993182354, + 0.0036610419920179993, + 0.005990999998175539, + 0.0038181249983608723, + 0.0035469580034259707, + 0.0033582500036573038, + 0.003507832996547222 + ], + [ + 0.02743995799391996, + 0.008229041995946318, + 0.0066361670033074915, + 0.006445040999096818, + 0.006694000010611489, + 0.006253625004319474, + 0.006479125004261732, + 0.005996458989102393, + 0.00553750000835862, + 0.00626945799740497, + 0.005917249989579432, + 0.005844624989549629, + 0.0057965830055763945, + 0.0053148750012042, + 0.004580207998515107, + 0.004530999998678453, + 0.004561249996186234, + 0.005479416999150999, + 0.00539037499402184, + 0.004750959007651545, + 0.005609290994470939, + 0.005267000000458211, + 0.00449187500635162, + 0.005401625006925315, + 0.005178874998819083, + 0.004473667009733617, + 0.004597082996042445, + 0.005262667007627897, + 0.005239207996055484, + 0.004586208000546321, + 0.005312291003065184, + 0.005350709005142562, + 0.004520124988630414, + 0.00544570799684152, + 0.00512458301091101, + 0.004643791995476931, + 0.005215917000896297, + 0.004971624992322177, + 0.0052978329913457856, + 0.004689458000939339, + 0.004561333000310697, + 0.004987583990441635, + 0.005239292004262097, + 0.004516583998338319, + 0.004393584007630125, + 0.005081916999188252, + 0.005143709000549279, + 0.004504541997448541, + 0.005131417012307793, + 0.00472474998969119, + 0.00447300000814721, + 0.004175791997113265, + 0.004762124997796491, + 0.005284208993543871, + 0.004106000007595867, + 0.005036709000705741, + 0.004633665987057611, + 0.004838082997594029, + 0.004685749998316169, + 0.004789999991771765, + 0.004535875006695278, + 0.004722708006738685, + 0.004545167001197115, + 0.004699875004007481 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + ] + } + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 21.617584003251977, + "p95": 24.79231719044037, + "min": 20.396166990394704, + "max": 25.542958988808095, + "iqr": 0.8371669973712415, + "mean": 22.059808595804498, + "cv": 0.09177346405418939 + }, + "decode_ms_per_step": { + "count": 315, + "median": 26.10899999854155, + "p95": 30.614616900857072, + "min": 19.62512500176672, + "max": 33.44437500345521, + "iqr": 4.585561997373588, + "mean": 25.87487419992353, + "cv": 0.11653008747104769 + }, + "total_ms": { + "count": 5, + "median": 1645.2090829698136, + "p95": 1744.8951408092398, + "min": 1588.0226690060226, + "max": 1762.9907070077024, + "iqr": 80.36379505938385, + "mean": 1652.1768831909867, + "cv": 0.04328883516550412 + }, + "tokens_per_second": { + "count": 5, + "median": 311.2066455868176, + "p95": 322.24641208719163, + "min": 290.4155977481072, + "max": 322.4135335048282, + "iqr": 15.451733104583354, + "mean": 310.3479793136921, + "cv": 0.04224479464044555 + }, + "median_step_ms_by_index": [ + 21.617584003251977, + 22.28254200599622, + 22.593083995161578, + 22.745833004591987, + 23.44479200837668, + 22.427625008276664, + 21.945667002000846, + 22.602249999181367, + 22.82433300570119, + 24.197874998208135, + 22.974416991928592, + 22.07020799687598, + 21.081375001813285, + 20.95125000050757, + 20.902249991195276, + 21.41204099461902, + 21.139834003406577, + 21.74041699618101, + 22.633457992924377, + 22.381959002814256, + 23.68983298947569, + 22.7489160024561, + 23.394957999698818, + 23.476874994230457, + 23.418415992637165, + 24.4023749983171, + 24.558042001444846, + 24.820250007905997, + 24.917000002460554, + 25.255124986870214, + 26.821209001354873, + 25.53908400295768, + 26.078458002302796, + 27.121624996652827, + 28.22766600002069, + 26.50379200349562, + 26.304209008230828, + 26.214500001515262, + 26.68529200309422, + 26.70025000406895, + 26.715958010754548, + 27.187208004761487, + 27.73845801129937, + 27.17566699720919, + 27.881999994860962, + 27.25908299908042, + 28.918040989083238, + 27.841832998092286, + 28.204625006765127, + 28.486499999416992, + 28.7306670070393, + 28.966499987291172, + 29.490291999536566, + 29.472249996615574, + 28.82095800305251, + 29.28095900279004, + 29.183624996221624, + 28.99424999486655, + 28.432833001716062, + 30.664874997455627, + 28.98933300457429, + 28.56241700646933, + 28.200584012665786, + 28.799374995287508 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.021789749996969476, + 0.022976541993557476, + 0.022593083995161578, + 0.02264574999571778, + 0.02344479200837668, + 0.023178624993306585, + 0.023409375004121102, + 0.023716875002719462, + 0.02347800000279676, + 0.024197874998208135, + 0.024657417001435533, + 0.02450783298991155, + 0.02472779199888464, + 0.025221666001016274, + 0.024730999997700565, + 0.02560241699393373, + 0.025026374991284683, + 0.02585154199914541, + 0.025413875002413988, + 0.02669670899922494, + 0.026146416988922283, + 0.026447958007338457, + 0.026390916013042443, + 0.02663079100602772, + 0.02705779099778738, + 0.026493290992220864, + 0.027206625003600493, + 0.027050000004237518, + 0.026640124997356907, + 0.027626291994238272, + 0.027870417005033232, + 0.027508499988471158, + 0.030018290999578312, + 0.030567583002266474, + 0.03344437500345521, + 0.031006917008198798, + 0.03128941700560972, + 0.031037124994327314, + 0.030484875009278767, + 0.03092550000292249, + 0.03090870899904985, + 0.02968866701121442, + 0.030827958005829714, + 0.029592208011308685, + 0.030596291995607316, + 0.030138165995595045, + 0.029202208999777213, + 0.029949666000902653, + 0.02976212500652764, + 0.030657375013106503, + 0.028025666004396044, + 0.028966499987291172, + 0.03024566599924583, + 0.029674749996047467, + 0.02953012500074692, + 0.029692124997382052, + 0.029356625003856607, + 0.02899424999486655, + 0.028432833001716062, + 0.030664874997455627, + 0.02898933300457429, + 0.02856241700646933, + 0.028027041989844292, + 0.026794624995091 + ], + [ + 0.021617584003251977, + 0.02228254200599622, + 0.02340070799982641, + 0.022745833004591987, + 0.023797625006409362, + 0.022226124987355433, + 0.021380125006544404, + 0.022602249999181367, + 0.02282433300570119, + 0.024273957998957485, + 0.022974416991928592, + 0.02207020799687598, + 0.021081375001813285, + 0.02034129099047277, + 0.02054695799597539, + 0.020472999996854924, + 0.020611750005627982, + 0.021331374999135733, + 0.021400249999715015, + 0.02110937499674037, + 0.021368250003433786, + 0.021856499995919876, + 0.02219837499433197, + 0.02183241699822247, + 0.02251191700634081, + 0.022272125002928078, + 0.023000874993158504, + 0.023164124999311753, + 0.023131249996367842, + 0.02353070800018031, + 0.023881959001300856, + 0.023448209001799114, + 0.024637583002913743, + 0.024243541003670543, + 0.02527399999962654, + 0.024739500004216097, + 0.025798624992603436, + 0.025432916008867323, + 0.025704291998408735, + 0.025590332996216603, + 0.026249500006088056, + 0.026231875002849847, + 0.026272332994267344, + 0.026890916997217573, + 0.026865167004871182, + 0.026519125007325783, + 0.028918040989083238, + 0.026747416995931417, + 0.02836783300153911, + 0.028486499999416992, + 0.029686583002330735, + 0.030758499997318722, + 0.03275425000174437, + 0.029472249996615574, + 0.02859370899386704, + 0.027122707993839867, + 0.02741349999269005, + 0.028345916987746023, + 0.027754875001846813, + 0.028049707994796336, + 0.029732625000178814, + 0.029992833005962893, + 0.029289124999195337, + 0.028925207996508107 + ], + [ + 0.020952582999598235, + 0.02181525000196416, + 0.022341375006362796, + 0.022759333995054476, + 0.023165040998719633, + 0.022427625008276664, + 0.021945667002000846, + 0.021180042007472366, + 0.02084024999930989, + 0.02031179200275801, + 0.020533957998850383, + 0.02068962500197813, + 0.020493625008384697, + 0.02095125000050757, + 0.020902249991195276, + 0.02141204099461902, + 0.021139834003406577, + 0.02174041699618101, + 0.022633457992924377, + 0.022381959002814256, + 0.02368983298947569, + 0.0227489160024561, + 0.023394957999698818, + 0.023476874994230457, + 0.023418415992637165, + 0.0244023749983171, + 0.02449729200452566, + 0.024820250007905997, + 0.024917000002460554, + 0.025255124986870214, + 0.02748191700084135, + 0.02553908400295768, + 0.026078458002302796, + 0.027121624996652827, + 0.02879295799357351, + 0.028986666991841048, + 0.02722904199617915, + 0.027376749989343807, + 0.027251833002083004, + 0.02684341699932702, + 0.027502209006343037, + 0.027187208004761487, + 0.027911249999306165, + 0.028517874990939163, + 0.02800754200143274, + 0.028418165995390154, + 0.029365833004703745, + 0.027841832998092286, + 0.028204625006765127, + 0.029778666008496657, + 0.028883708000648767, + 0.029017542008659802, + 0.029490291999536566, + 0.030004791013197973, + 0.02948045899393037, + 0.031166000000666827, + 0.029183624996221624, + 0.029532582993851975, + 0.03104462500778027, + 0.030715250002685934, + 0.03017724999517668, + 0.030340166995301843, + 0.029969041992444545, + 0.029528374987421557 + ], + [ + 0.025542958988808095, + 0.025651666001067497, + 0.025667291003628634, + 0.02605250000488013, + 0.025432000009459443, + 0.02610899999854155, + 0.025145833002170548, + 0.02639304100011941, + 0.025153834008960985, + 0.025988082998082973, + 0.02545704200747423, + 0.02629229100421071, + 0.02565883299394045, + 0.025611083998228423, + 0.025939166996977292, + 0.025406499989912845, + 0.026534667005762458, + 0.025462499994318932, + 0.025927375012543052, + 0.025402250001206994, + 0.026081582996994257, + 0.02569095899525564, + 0.024872791997040622, + 0.025650666997535154, + 0.025255417000153102, + 0.02750154100067448, + 0.025788083003135398, + 0.026575415991828777, + 0.02638716700312216, + 0.027302000002237037, + 0.026821209001354873, + 0.026836374992853962, + 0.027033833001041785, + 0.02742374999797903, + 0.02822766600002069, + 0.02650379200349562, + 0.025740875003975816, + 0.02592958399327472, + 0.024770417003310286, + 0.02516483400540892, + 0.024693749990547076, + 0.025395667005795985, + 0.025465749989962205, + 0.025374125005328096, + 0.025293624988989905, + 0.025410540998564102, + 0.0256541249982547, + 0.025825415999861434, + 0.02584291600214783, + 0.02591716700408142, + 0.02636170800542459, + 0.026441166002769023, + 0.02613825000298675, + 0.026648084007319994, + 0.02618899999652058, + 0.027343541994923726, + 0.026482333007152192, + 0.02686895900114905, + 0.027378208003938198, + 0.026863999999477528, + 0.027417709003202617, + 0.027336958999512717, + 0.02698462500120513, + 0.028799374995287508 + ], + [ + 0.020396166990394704, + 0.021309874995495193, + 0.021321249994798563, + 0.020916082998155616, + 0.02118625000002794, + 0.021116207994055003, + 0.020504959000390954, + 0.020009375002700835, + 0.01962512500176672, + 0.019626791996415704, + 0.020219124999130145, + 0.0200634170032572, + 0.020198375001200475, + 0.02053087500098627, + 0.020729207986732945, + 0.020869499989203177, + 0.020634707994759083, + 0.021095791991683654, + 0.02134791600110475, + 0.02191741699061822, + 0.021503333002328873, + 0.02227033300732728, + 0.02266099999542348, + 0.023232667008414865, + 0.023039125007926486, + 0.023485666999476962, + 0.024558042001444846, + 0.02366325000184588, + 0.02447483300056774, + 0.02448204200481996, + 0.025188624989823438, + 0.024901582990423776, + 0.02514708299713675, + 0.025409541005501524, + 0.026410749997012317, + 0.025925833004293963, + 0.026304209008230828, + 0.026214500001515262, + 0.02668529200309422, + 0.02670025000406895, + 0.026715958010754548, + 0.02726187500229571, + 0.02773845801129937, + 0.02717566699720919, + 0.027881999994860962, + 0.02725908299908042, + 0.02774416700412985, + 0.028056916999048553, + 0.02765216700208839, + 0.02792116699856706, + 0.0287306670070393, + 0.027934333993471228, + 0.028767459007212892, + 0.029201832992839627, + 0.02882095800305251, + 0.02928095900279004, + 0.029677666010684334, + 0.029095916004735045, + 0.030900833997293375, + 0.030790999997407198, + 0.028935624999576248, + 0.028354499998386018, + 0.028200584012665786, + 0.028046499995980412 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_mha": { + "first_token_ms": { + "count": 5, + "median": 27.604542003246024, + "p95": 31.12577520078048, + "min": 24.519874990801327, + "max": 31.720792001578957, + "iqr": 3.8664169987896457, + "mean": 27.494041598401964, + "cv": 0.10778324217937553 + }, + "decode_ms_per_step": { + "count": 315, + "median": 32.18674998788629, + "p95": 35.53644560452085, + "min": 24.38466699095443, + "max": 40.671166992979124, + "iqr": 3.2412915024906397, + "mean": 31.681101059491215, + "cv": 0.08940143084933179 + }, + "total_ms": { + "count": 5, + "median": 2027.2855838848045, + "p95": 2084.618798745214, + "min": 1943.44241800718, + "max": 2088.476164950407, + "iqr": 80.5657929595327, + "mean": 2023.4034083463484, + "cv": 0.02918525565609722 + }, + "tokens_per_second": { + "count": 5, + "median": 252.55445215512032, + "p95": 262.2529425493583, + "min": 245.15482081748246, + "max": 263.4500488699884, + "iqr": 10.024618169283258, + "mean": 253.2127476413967, + "cv": 0.029392676357850096 + }, + "median_step_ms_by_index": [ + 27.604542003246024, + 27.561374998185784, + 29.446625005220994, + 29.506958002457395, + 29.975124998600222, + 30.68741600145586, + 29.132666008081287, + 29.33804200438317, + 29.020999994827434, + 29.039542001555674, + 28.59216601063963, + 29.403583001112565, + 29.416583987767808, + 29.774124996038154, + 29.19420800753869, + 29.441792008583434, + 29.585042007965967, + 29.942665991256945, + 28.962625001440756, + 28.67349999723956, + 28.65091699641198, + 29.822957993019372, + 30.765957999392413, + 31.408583003212698, + 30.724708005436696, + 30.586624998250045, + 31.045042007463053, + 31.481666999752633, + 31.49183400091715, + 31.454500000108965, + 30.392166998353787, + 30.95391699753236, + 30.411207990255207, + 32.648166990838945, + 31.354042002931237, + 31.469250010559335, + 32.07337501225993, + 32.41408399480861, + 32.15979100787081, + 32.59520899155177, + 32.46120900439564, + 33.251665998250246, + 32.627166001475416, + 32.85012498963624, + 33.34266699675936, + 33.37466700759251, + 33.202541002538055, + 33.526083003380336, + 33.15329100587405, + 33.539000010932796, + 33.38716599682812, + 34.048000001348555, + 33.49783399607986, + 33.864583005197346, + 33.75016700010747, + 33.33412499341648, + 33.90579199185595, + 34.446666002622806, + 34.20900000492111, + 33.898334004334174, + 33.603666990529746, + 33.65316700364929, + 33.49091699055862, + 33.53116600192152 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.027604542003246024, + 0.029294292005943134, + 0.030054250004468486, + 0.029506958002457395, + 0.029975124998600222, + 0.03068741600145586, + 0.03160749998642132, + 0.03165520800394006, + 0.03107770900533069, + 0.03246624999155756, + 0.03319212498900015, + 0.03218674998788629, + 0.03035387500131037, + 0.030227625000406988, + 0.03130787500413135, + 0.030606875006924383, + 0.031796333991223946, + 0.03210708299593534, + 0.028962625001440756, + 0.028637916999286972, + 0.02865091699641198, + 0.029822957993019372, + 0.030765957999392413, + 0.031464957995922305, + 0.031411290998221375, + 0.03292295899882447, + 0.03143741699750535, + 0.033244082995224744, + 0.032379417010815814, + 0.03186237499176059, + 0.03214262500114273, + 0.03253466699970886, + 0.03326129200286232, + 0.03060445901064668, + 0.03117295799893327, + 0.03037758299615234, + 0.030968167004175484, + 0.032445916993310675, + 0.03215979100787081, + 0.032656540992320515, + 0.035097042011329904, + 0.03568454099877272, + 0.0325697499938542, + 0.03285012498963624, + 0.03503958300279919, + 0.03592812499846332, + 0.03549833300348837, + 0.035662791997310705, + 0.03500958299264312, + 0.03499616699991748, + 0.03495800000382587, + 0.036581665990524925, + 0.03488195899990387, + 0.03483974999107886, + 0.03475654199428391, + 0.034041832986986265, + 0.03492099999857601, + 0.033421500003896654, + 0.03420900000492111, + 0.033898334004334174, + 0.033603666990529746, + 0.03365316700364929, + 0.030595999996876344, + 0.03089620800164994 + ], + [ + 0.024519874990801327, + 0.02557645800698083, + 0.02760145800129976, + 0.026790292002260685, + 0.02855408299365081, + 0.026163375005126, + 0.0251471249939641, + 0.02438466699095443, + 0.024731207988224924, + 0.024431917001493275, + 0.02529929199954495, + 0.026020042001619004, + 0.02555604100052733, + 0.026768124997033738, + 0.026077624992467463, + 0.027601082998444326, + 0.026809666989720426, + 0.027625292001175694, + 0.02833329199347645, + 0.028255666999029927, + 0.028566542008775286, + 0.02885795899783261, + 0.03017145799822174, + 0.030090000000200234, + 0.029599624991533346, + 0.030586624998250045, + 0.030754959007026628, + 0.03148166699975263, + 0.03149183400091715, + 0.03148670800146647, + 0.03204391698818654, + 0.031879542002570815, + 0.031606249991455115, + 0.03388287499547005, + 0.03359154200006742, + 0.03361795800446998, + 0.03377516599721275, + 0.034540124994236976, + 0.0342288749961881, + 0.034078083001077175, + 0.03434625000227243, + 0.03467229098896496, + 0.03473866599961184, + 0.03505220799706876, + 0.03456404199823737, + 0.03459912499238271, + 0.03425779099052306, + 0.03471500000159722, + 0.03471375000663102, + 0.03500112499750685, + 0.034243542002514005, + 0.03424458400695585, + 0.034657541997148655, + 0.03585495799779892, + 0.033648707991233096, + 0.03562537500692997, + 0.03655175000312738, + 0.037177541002165526, + 0.040576792001957074, + 0.040671166992979124, + 0.03812795899284538, + 0.03894524999486748, + 0.038250333003816195, + 0.03947154099296313 + ], + [ + 0.03172079200157896, + 0.0326526660064701, + 0.03227566598798148, + 0.03240004100371152, + 0.032894374991883524, + 0.03277200000593439, + 0.03244933299720287, + 0.033018292000633664, + 0.03268991700315382, + 0.0323951250029495, + 0.03286150000349153, + 0.03272649999416899, + 0.03232691599987447, + 0.032287790992995724, + 0.03234054100175854, + 0.03265254199504852, + 0.031887000004644506, + 0.03235716599738225, + 0.03280137499677949, + 0.03228433401091024, + 0.032121416996233165, + 0.03272020800795872, + 0.032285165987559594, + 0.034321790997637436, + 0.03275537499575876, + 0.03331695799715817, + 0.0356386250059586, + 0.03412062500137836, + 0.032310374997905456, + 0.031454500000108965, + 0.030392166998353787, + 0.03095391699753236, + 0.030278207996161655, + 0.030918666001525708, + 0.031126875008339994, + 0.03153012500843033, + 0.031959791987901554, + 0.03190066700335592, + 0.03191720899485517, + 0.032557541999267414, + 0.03191195899853483, + 0.0322025420027785, + 0.032627166001475416, + 0.03198195798904635, + 0.03207200000179, + 0.03244283300591633, + 0.03269570799602661, + 0.03244791700853966, + 0.03252054199401755, + 0.032803875001263805, + 0.033097499996074475, + 0.033125375004601665, + 0.03349783399607986, + 0.03340070899866987, + 0.034589584000059403, + 0.03333412499341648, + 0.03390579199185595, + 0.034446666002622806, + 0.03403854199859779, + 0.03314612498797942, + 0.03336229200067464, + 0.033428958005970344, + 0.03349091699055862, + 0.03353116600192152 + ], + [ + 0.028745707997586578, + 0.027561374998185784, + 0.029446625005220994, + 0.030291292001493275, + 0.029978958002175204, + 0.03076116599550005, + 0.029132666008081287, + 0.02933804200438317, + 0.029020999994827434, + 0.029039542001555674, + 0.02859216601063963, + 0.029403583001112565, + 0.029416583987767808, + 0.029774124996038154, + 0.02919420800753869, + 0.029441792008583434, + 0.029585042007965967, + 0.029942665991256945, + 0.029682667009183206, + 0.030012041999725625, + 0.030075207992922515, + 0.030249249990447424, + 0.031103749992325902, + 0.0314085830032127, + 0.030724708005436696, + 0.03049483400536701, + 0.031045042007463053, + 0.03073674999177456, + 0.03044775000307709, + 0.03113654199114535, + 0.030082500001299195, + 0.030531957992934622, + 0.030411207990255207, + 0.032796250001410954, + 0.03135404200293124, + 0.03126908399281092, + 0.03211279198876582, + 0.031558083996060304, + 0.031203000005916692, + 0.031344582996098325, + 0.0315772920002928, + 0.03162637499917764, + 0.031994332995964214, + 0.03205254199565388, + 0.03233591699972749, + 0.03191437500936445, + 0.032343124999897555, + 0.032220875000348315, + 0.03187216600053944, + 0.03255837499455083, + 0.0323683749884367, + 0.03238424999290146, + 0.03240166700561531, + 0.0324724160018377, + 0.03272445799666457, + 0.0327452909987187, + 0.032749125006375834, + 0.03341175000241492, + 0.03286841699446086, + 0.03296691599825863, + 0.03344562501297332, + 0.033338124994770624, + 0.03297687499434687, + 0.03282770900113974 + ], + [ + 0.024879290998796932, + 0.02544125000713393, + 0.026119166999706067, + 0.026274457995896228, + 0.026707625002018176, + 0.02671304099203553, + 0.025282875008997507, + 0.025106624991167337, + 0.024542832994484343, + 0.025451458001043648, + 0.02536800000234507, + 0.025830291997408494, + 0.026296167008695193, + 0.026382000010926276, + 0.027074500001617707, + 0.026166791998548433, + 0.026984124997397885, + 0.027100874998723157, + 0.02725754199491348, + 0.02867349999723956, + 0.027538875001482666, + 0.02815879200352356, + 0.028612291003810242, + 0.028705083997920156, + 0.029096250000293367, + 0.02931795798940584, + 0.03084537500399165, + 0.03016899999056477, + 0.030136291999951936, + 0.0298350419907365, + 0.02985824999632314, + 0.029988458001753315, + 0.030206499999621883, + 0.032648166990838945, + 0.032172999999602325, + 0.031469250010559335, + 0.03207337501225993, + 0.03241408399480861, + 0.032402708995505236, + 0.03259520899155177, + 0.03246120900439564, + 0.033251665998250246, + 0.03292862500529736, + 0.03288833399710711, + 0.03334266699675936, + 0.03337466700759251, + 0.033202541002538055, + 0.033526083003380336, + 0.03315329100587405, + 0.033539000010932796, + 0.033387165996828116, + 0.034048000001348555, + 0.033444582993979566, + 0.033864583005197346, + 0.03375016700010747, + 0.0323980829998618, + 0.033509250002680346, + 0.036226166994310915, + 0.03477412500069477, + 0.03497079201042652, + 0.03488920799281914, + 0.03518587500730064, + 0.034275708996574394, + 0.03515424999932293 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "speedup_cached_vs_uncached_sdpa": 4.912593371315513, + "speedup_cached_vs_uncached_mha": 6.053473582323227 + }, + { + "batch": 8, + "prompt_len": 1024, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_mha": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 52.332332998048514, + "p95": 57.28198340802919, + "min": 50.61191700224299, + "max": 58.36387501040008, + "iqr": 1.0222510027233511, + "mean": 53.2389416010119, + "cv": 0.05617474043555502 + }, + "decode_ms_per_step": { + "count": 315, + "median": 5.376457993406802, + "p95": 7.483978701930028, + "min": 3.8981250108918175, + "max": 10.851708997506648, + "iqr": 1.2036875050398521, + "mean": 5.51560806033815, + "cv": 0.1978521363213112 + }, + "total_ms": { + "count": 5, + "median": 406.0085420496762, + "p95": 407.1576826070668, + "min": 382.47562303149607, + "max": 407.22399900550954, + "iqr": 5.881751101696864, + "mean": 400.72224940231536, + "cv": 0.02621193032882981 + }, + "tokens_per_second": { + "count": 5, + "median": 1261.057211789784, + "p95": 1326.2727274941954, + "min": 1257.293286374001, + "max": 1338.6474043545459, + "iqr": 18.456148812471156, + "mean": 1278.4179587622893, + "cv": 0.027045787639495316 + }, + "median_step_ms_by_index": [ + 52.332332998048514, + 8.821958006592467, + 7.909124993602745, + 7.47191600385122, + 6.494834000477567, + 7.022624995443039, + 6.820374997914769, + 6.353167002089322, + 5.550124988076277, + 5.586500003118999, + 6.6531249904073775, + 6.128249995526858, + 6.430833003832959, + 6.117750002886169, + 5.885874998057261, + 6.057083999621682, + 5.775333993369713, + 5.996499996399507, + 5.5163749930216, + 5.492041003890336, + 5.65004198870156, + 5.427374999271706, + 5.397041008109227, + 5.597208000835963, + 5.648500009556301, + 5.562708989600651, + 5.346292004105635, + 5.384916992625222, + 5.294084010529332, + 5.408707991591655, + 5.191125004785135, + 5.425583993201144, + 4.988833999959752, + 5.607917002635077, + 5.329542007530108, + 5.255750002106652, + 5.348083999706432, + 5.471208001836203, + 5.335833004210144, + 5.498208003700711, + 5.269166998914443, + 5.424540999229066, + 5.376457993406802, + 5.348000006051734, + 5.231375005678274, + 5.056207999587059, + 5.319750009221025, + 4.966499996953644, + 5.124208008055575, + 4.944583008182235, + 5.137792002642527, + 4.9454160034656525, + 5.367208999814466, + 5.020083000999875, + 5.2578340109903365, + 5.09862499893643, + 5.00050000846386, + 5.15974999871105, + 5.148999989614822, + 4.835832995013334, + 5.216749996179715, + 4.894750003586523, + 5.081249997601844, + 5.116957996506244 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.05836387501040008, + 0.010851708997506648, + 0.007909124993602745, + 0.007939375005662441, + 0.008053917001234367, + 0.007965749988215975, + 0.0076725830003852025, + 0.007428333003190346, + 0.00755958299851045, + 0.007311041001230478, + 0.007059415991534479, + 0.006665167005849071, + 0.006846832999144681, + 0.006941208004718646, + 0.006937334008398466, + 0.0070180420007091016, + 0.006784291996154934, + 0.005996791995130479, + 0.0055163749930216, + 0.006167875006212853, + 0.005707375006750226, + 0.005427374999271706, + 0.005397041008109227, + 0.005411082995124161, + 0.005333875000360422, + 0.005091208993690088, + 0.005305874990881421, + 0.00533937499858439, + 0.005368582991650328, + 0.005561915997532196, + 0.004982875005225651, + 0.005536040989682078, + 0.00492925000435207, + 0.005655750006553717, + 0.005296041999827139, + 0.0052604579977924, + 0.005348083999706432, + 0.00516158300160896, + 0.004660916005377658, + 0.00551395901129581, + 0.004595000005792826, + 0.004378166995593347, + 0.004218957998091355, + 0.0041640839917818084, + 0.0041369999962626025, + 0.0038981250108918175, + 0.0041588750027585775, + 0.004266375006409362, + 0.004048500006319955, + 0.003988374999607913, + 0.004126917003304698, + 0.004594833997543901, + 0.0043411250080680475, + 0.004147125000599772, + 0.00442254199879244, + 0.004216583009110764, + 0.004492082996875979, + 0.004184666002402082, + 0.004245084011927247, + 0.004514542000833899, + 0.004332917000283487, + 0.0042114160023629665, + 0.004671750008128583, + 0.004376208991743624 + ], + [ + 0.05193216599582229, + 0.010354375001043081, + 0.008057416998781264, + 0.00747191600385122, + 0.007521457999246195, + 0.0075634999957401305, + 0.0069527079904219136, + 0.0068938749900553375, + 0.00669191699125804, + 0.007459874992491677, + 0.006975917000090703, + 0.006869333010399714, + 0.006986958003835753, + 0.006603166999411769, + 0.0068182079994585365, + 0.006618749990593642, + 0.006390208000084385, + 0.0071865409991005436, + 0.006707292006467469, + 0.00654295900312718, + 0.0063575839885743335, + 0.006814332999056205, + 0.005380084010539576, + 0.006025209004292265, + 0.00670837500365451, + 0.005562708989600651, + 0.005446958995889872, + 0.005384916992625222, + 0.005294084010529332, + 0.005408707991591655, + 0.005191125004785135, + 0.005425583993201144, + 0.005296708011883311, + 0.0053694159869337454, + 0.005329542007530108, + 0.005255750002106652, + 0.004846833006013185, + 0.005136624997248873, + 0.004758083989145234, + 0.00479224999435246, + 0.004196957990643568, + 0.004275665996829048, + 0.0044152080081403255, + 0.003960000001825392, + 0.0043935000139754266, + 0.00419300000066869, + 0.004388125002151355, + 0.00413091600057669, + 0.004138958000112325, + 0.0041550829919287935, + 0.004344165994552895, + 0.004100749996723607, + 0.004265375013346784, + 0.004422416997840628, + 0.004390041998703964, + 0.0039012499910313636, + 0.004860791988903657, + 0.004381749997264706, + 0.004113415998290293, + 0.004342207990703173, + 0.004356040997663513, + 0.004396791991894133, + 0.004277416999684647, + 0.004227416997309774 + ], + [ + 0.05295441699854564, + 0.008421624996117316, + 0.006678290999843739, + 0.0068041250051464885, + 0.005846832995302975, + 0.005665875010890886, + 0.005670166006893851, + 0.005501542007550597, + 0.005550124988076277, + 0.005586500003118999, + 0.0066531249904073775, + 0.006128249995526858, + 0.006430833003832959, + 0.006117750002886169, + 0.005885874998057261, + 0.006057083999621682, + 0.005775333993369713, + 0.005996499996399507, + 0.005893125009606592, + 0.005397375003667548, + 0.00565004198870156, + 0.005286624989821576, + 0.005413041988504119, + 0.006559625006048009, + 0.0061767919978592545, + 0.006483542005298659, + 0.00621641700854525, + 0.005541874998016283, + 0.005729541997425258, + 0.005373541993321851, + 0.005374041997129098, + 0.005638000002363697, + 0.00530179199995473, + 0.005938292000791989, + 0.0055472089879913256, + 0.005513041003723629, + 0.005469084004289471, + 0.005541499995160848, + 0.005420624991529621, + 0.005482082997332327, + 0.005269166998914443, + 0.005424540999229066, + 0.005376457993406802, + 0.0054011660104151815, + 0.005231375005678274, + 0.005056207999587059, + 0.005319750009221025, + 0.004966499996953644, + 0.005124208008055575, + 0.004944583008182235, + 0.005137792002642527, + 0.0049454160034656525, + 0.005573874994297512, + 0.005020083000999875, + 0.0052578340109903365, + 0.00509862499893643, + 0.0050005000084638596, + 0.0051597499987110496, + 0.005148999989614822, + 0.004835832995013334, + 0.005356583002139814, + 0.005031958004110493, + 0.005246500004432164, + 0.005293250011163764 + ], + [ + 0.05061191700224299, + 0.004772582993609831, + 0.004497625006479211, + 0.004154583002673462, + 0.0041813749994616956, + 0.004203584001515992, + 0.004197875008685514, + 0.004137416006415151, + 0.004200875002425164, + 0.004308625007979572, + 0.004236291002598591, + 0.004214250002405606, + 0.004187792001175694, + 0.004151750006712973, + 0.004244666997692548, + 0.004222417002893053, + 0.004207457997836173, + 0.004263833005097695, + 0.004184459001407959, + 0.004095249998499639, + 0.004122292011743411, + 0.004298542000469752, + 0.0041337080037919804, + 0.0042941249994328246, + 0.0043532910058274865, + 0.004441832992597483, + 0.004067665999173187, + 0.004729457999928854, + 0.004494958004215732, + 0.004611832991940901, + 0.004496792011195794, + 0.00491712499933783, + 0.004849374992772937, + 0.0047838339960435405, + 0.005080041999462992, + 0.005139792003319599, + 0.0047115829947870225, + 0.005594500005827285, + 0.005634000001009554, + 0.005498208003700711, + 0.007512124997447245, + 0.006597125000553206, + 0.0073627919919090346, + 0.007268790999660268, + 0.007375542001682334, + 0.0069676660059485584, + 0.007396542001515627, + 0.007110291000572033, + 0.0072424169920850545, + 0.007090583996614441, + 0.007407874989439733, + 0.006565790987224318, + 0.006688542009214871, + 0.00657974999921862, + 0.006361207997542806, + 0.0067700840008910745, + 0.006096415992942639, + 0.006372708012349904, + 0.005941541996435262, + 0.005660541995894164, + 0.0054867500002728775, + 0.004894750003586523, + 0.005081249997601844, + 0.005116957996506244 + ], + [ + 0.052332332998048514, + 0.008821958006592467, + 0.008006457996089011, + 0.007797582991770469, + 0.006494834000477567, + 0.007022624995443039, + 0.006820374997914769, + 0.006353167002089322, + 0.005548959001316689, + 0.005560999998124316, + 0.005510750008397736, + 0.005548417000682093, + 0.00560795800993219, + 0.005447957999422215, + 0.005336249989341013, + 0.005512417003046721, + 0.005456375001813285, + 0.005264459003228694, + 0.005511208000825718, + 0.0054920410038903356, + 0.0051982079894514754, + 0.005802041996503249, + 0.005614458001218736, + 0.005597208000835963, + 0.005648500009556301, + 0.0055755419889464974, + 0.005346292004105635, + 0.005884333004360087, + 0.00523079201229848, + 0.005468624993227422, + 0.005229125003097579, + 0.005319791991496459, + 0.004988833999959752, + 0.005607917002635077, + 0.005336250003892928, + 0.005127833996084519, + 0.00544741700286977, + 0.005471208001836203, + 0.005335833004210144, + 0.0055163749930216, + 0.0053025420056656, + 0.005472624994581565, + 0.0054589159990428016, + 0.005348000006051734, + 0.0054577920091105625, + 0.005357749992981553, + 0.005624375000479631, + 0.005335249996278435, + 0.005561917001614347, + 0.00537149999581743, + 0.00546645799477119, + 0.005283749997033738, + 0.005367208999814466, + 0.005381333001423627, + 0.005470874995808117, + 0.005250707996310666, + 0.00538050000614021, + 0.005197667007450946, + 0.005385041004046798, + 0.0053461659990716726, + 0.005216749996179715, + 0.005161041000974365, + 0.005678415996953845, + 0.005153707999852486 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false + ] + ] + } + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 44.56174999359064, + "p95": 53.51425838889554, + "min": 43.96175000874791, + "max": 53.94437498762272, + "iqr": 7.508791983127594, + "mean": 47.70933339896146, + "cv": 0.10010376460384263 + }, + "decode_ms_per_step": { + "count": 315, + "median": 55.049250004231, + "p95": 59.95262479846133, + "min": 44.43412499676924, + "max": 62.86099999852013, + "iqr": 3.6288334958953783, + "mean": 55.00382247615591, + "cv": 0.06272186244637631 + }, + "total_ms": { + "count": 5, + "median": 3532.6567069714656, + "p95": 3607.261502626352, + "min": 3383.075870005996, + "max": 3622.670253040269, + "iqr": 64.90508497518022, + "mean": 3512.9501493967837, + "cv": 0.02522127758685444 + }, + "tokens_per_second": { + "count": 5, + "median": 144.93341484033863, + "p95": 150.49244144208512, + "min": 141.33221194236822, + "max": 151.34156598122422, + "iqr": 2.6926904725688985, + "mean": 145.82127777248394, + "cv": 0.02544043066350769 + }, + "median_step_ms_by_index": [ + 44.56174999359064, + 47.07729100482538, + 47.9550830059452, + 48.31345898855943, + 52.54887499904726, + 47.97245799272787, + 50.32562499400228, + 49.75295900658239, + 51.97354199481197, + 52.19029099680483, + 52.77875000319909, + 52.6354169996921, + 51.83129099896178, + 53.280958003597334, + 52.76133300503716, + 53.39466700388584, + 53.19733399664983, + 52.716249992954545, + 53.482749994145706, + 53.90425000223331, + 54.75341599958483, + 54.57179200311657, + 54.54608298896346, + 55.00166599813383, + 55.574957994394936, + 54.65804199047852, + 55.04704199847765, + 57.247375007136725, + 56.46195799636189, + 56.60329101374373, + 57.703790997038595, + 54.424124988145195, + 55.2209169982234, + 57.039916995563544, + 56.880832999013364, + 55.78979199344758, + 58.66954200610053, + 56.441583001287654, + 56.24125000031199, + 56.77329200261738, + 57.034249999560416, + 58.68391699914355, + 57.39775000256486, + 57.37412499729544, + 56.354084008489735, + 56.898791997809894, + 55.45229201379698, + 55.32120900170412, + 55.94912500237115, + 57.225666008889675, + 57.37966700689867, + 55.5732920038281, + 54.618125010165386, + 54.5148329983931, + 55.32508400210645, + 56.55500000284519, + 55.76991700218059, + 56.858415991882794, + 56.07158300699666, + 55.824124996433966, + 56.275916009326465, + 57.91408399818465, + 57.067000001552515, + 56.88220899901353 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.051793791993986815, + 0.05289041700598318, + 0.0528537909995066, + 0.05278187501244247, + 0.05260904200258665, + 0.05368462498881854, + 0.05370575000415556, + 0.05356129199208226, + 0.0539750829921104, + 0.05418604200531263, + 0.0546008340024855, + 0.054633291001664475, + 0.053725291989394464, + 0.0548307920107618, + 0.05456087499624118, + 0.05391241700272076, + 0.05319733399664983, + 0.055578208004590124, + 0.055751832987880334, + 0.05692350000026636, + 0.06286099999852013, + 0.05889220899553038, + 0.05846016699797474, + 0.05728525000449736, + 0.05696058299508877, + 0.06096341600641608, + 0.060705583993694745, + 0.05741137498989701, + 0.05820691600092687, + 0.06013500000699423, + 0.06226045799849089, + 0.05754791700746864, + 0.05643679200147744, + 0.05816604199935682, + 0.05765945799066685, + 0.058607167011359707, + 0.06262312500621192, + 0.061048417002893984, + 0.05624125000031199, + 0.06047183299961034, + 0.05704595899442211, + 0.06188533299427945, + 0.05758937500650063, + 0.0627181670133723, + 0.05858108300890308, + 0.058281374993384816, + 0.05385112499061506, + 0.05407783400733024, + 0.05514633400889579, + 0.057225666008889675, + 0.058944958000211045, + 0.05550333300197963, + 0.05363858399505261, + 0.0548689999995986, + 0.05532508400210645, + 0.057602916000178084, + 0.05576991700218059, + 0.056858415991882794, + 0.05519766701036133, + 0.055824124996433966, + 0.056275916009326465, + 0.05791408399818465, + 0.057067000001552515, + 0.052776958007598296 + ], + [ + 0.04456174999359064, + 0.04443412499676924, + 0.04556862500612624, + 0.047503041991149075, + 0.04562554099538829, + 0.04529299998830538, + 0.04597637499682605, + 0.046423416992183775, + 0.04793187500035856, + 0.04853749999892898, + 0.04923495798720978, + 0.04945649999717716, + 0.05064412500360049, + 0.050856791000114754, + 0.05276133300503716, + 0.05193987500388175, + 0.05359462500200607, + 0.052716249992954545, + 0.053482749994145706, + 0.05390425000223331, + 0.05510820799099747, + 0.05457179200311657, + 0.05487370799528435, + 0.055484500000602566, + 0.05590350000420585, + 0.056650375001481734, + 0.05629866599338129, + 0.057247375007136725, + 0.05646195799636189, + 0.05753208399983123, + 0.057703790997038595, + 0.057750208987272345, + 0.057913333002943546, + 0.05904579100024421, + 0.058939333001035266, + 0.058412790996953845, + 0.05866954200610053, + 0.059935249999398366, + 0.05886208300944418, + 0.05844437499763444, + 0.05851629200333264, + 0.058842333004577085, + 0.05878070900507737, + 0.058516958000836894, + 0.05832341700443067, + 0.058490417010034434, + 0.05817108400515281, + 0.0581265840010019, + 0.05770704199676402, + 0.057920874998671934, + 0.05737966700689867, + 0.057797124987700954, + 0.056849583008443005, + 0.0576739579992136, + 0.0575621250027325, + 0.05879170898697339, + 0.05857179200393148, + 0.061080375002347864, + 0.05954162499983795, + 0.059310541008017026, + 0.05807033399469219, + 0.05999316599627491, + 0.06270795800082851, + 0.05767566700524185 + ], + [ + 0.04396175000874791, + 0.04707729100482538, + 0.0479550830059452, + 0.04831345898855943, + 0.05254887499904726, + 0.04797245799272787, + 0.05032562499400228, + 0.04975295900658239, + 0.05197354199481197, + 0.05219029099680483, + 0.05277875000319909, + 0.0526354169996921, + 0.051831290998961776, + 0.053280958003597334, + 0.05089583300286904, + 0.05339466700388584, + 0.05162812498747371, + 0.052669083001092076, + 0.05186570799560286, + 0.05105508299311623, + 0.05068754201056436, + 0.05190395801037084, + 0.05394083399733063, + 0.05500166599813383, + 0.055574957994394936, + 0.05255370799568482, + 0.05383479199372232, + 0.057908375005354173, + 0.05936975000076927, + 0.05660329101374373, + 0.05799966699851211, + 0.05400641600135714, + 0.054035959008615464, + 0.055754415996489115, + 0.056880832999013364, + 0.05547733399725985, + 0.06083075000788085, + 0.05561087500245776, + 0.059909249990596436, + 0.05677329200261738, + 0.057034249999560416, + 0.05868391699914355, + 0.05739775000256486, + 0.05737412499729544, + 0.056354084008489735, + 0.056898791997809894, + 0.05990525000379421, + 0.05961675000435207, + 0.05823487498855684, + 0.05776691700157244, + 0.054623249990982004, + 0.05348179199791048, + 0.05383666699344758, + 0.054259249998722225, + 0.05456154099374544, + 0.05471945799945388, + 0.05492279099416919, + 0.055148667001049034, + 0.05607158300699666, + 0.05540745900361799, + 0.056186959001934156, + 0.05626137500803452, + 0.05632779099687468, + 0.05688220899901353 + ], + [ + 0.05394437498762272, + 0.053611166003975086, + 0.05334270899766125, + 0.05503995799517725, + 0.054019582996261306, + 0.053353833995060995, + 0.052943375005270354, + 0.05290662500192411, + 0.05321529199136421, + 0.05360770800325554, + 0.05318200000328943, + 0.05382425000425428, + 0.05432900000596419, + 0.05452466700808145, + 0.05369595800584648, + 0.054129416996147484, + 0.0541945000004489, + 0.0550135829980718, + 0.05415691599773709, + 0.055682124992017634, + 0.05475341599958483, + 0.0547700419992907, + 0.05454608298896346, + 0.05480158299906179, + 0.05462562499451451, + 0.05463466700166464, + 0.054898291011340916, + 0.05531770800007507, + 0.05478574999142438, + 0.05496612499700859, + 0.054981334003969096, + 0.053966709005180746, + 0.054498291996424086, + 0.057039916995563544, + 0.05495599999267142, + 0.054683042006217875, + 0.0554228749970207, + 0.056441583001287654, + 0.05483958299737424, + 0.05576433299575001, + 0.054993500001728535, + 0.05617537499347236, + 0.05567750000045635, + 0.05515979199844878, + 0.0555902500054799, + 0.056230499991215765, + 0.05545229201379698, + 0.05532120900170412, + 0.05594912500237115, + 0.0555188750004163, + 0.058158165993518196, + 0.05590449999726843, + 0.056323458993574604, + 0.05446195798867848, + 0.05601008400844876, + 0.05655500000284519, + 0.05821195800672285, + 0.0579004590108525, + 0.05816870799753815, + 0.057789957994828, + 0.05940533299872186, + 0.06019595899851993, + 0.0594736670027487, + 0.06158887500350829 + ], + [ + 0.04428500001085922, + 0.04569579199596774, + 0.045396582994726487, + 0.04478154100070242, + 0.04533633300161455, + 0.04689391699503176, + 0.0468702910002321, + 0.045939666000776924, + 0.047076500006369315, + 0.04790908400900662, + 0.047779875007108785, + 0.04893495900614653, + 0.0493169580004178, + 0.04995683299785014, + 0.05064120799943339, + 0.050817541996366344, + 0.05141975000151433, + 0.051880290993722156, + 0.0521359590056818, + 0.05274008300330024, + 0.053422791010234505, + 0.05380908300867304, + 0.054145292000612244, + 0.05418158299289644, + 0.05412645799515303, + 0.05465804199047852, + 0.05504704199847765, + 0.05456445799791254, + 0.05463095799495932, + 0.05512591700244229, + 0.054888208003831096, + 0.054424124988145195, + 0.0552209169982234, + 0.0563010000041686, + 0.055049250004231, + 0.05578979199344758, + 0.05587766600365285, + 0.05514900000707712, + 0.05574133401387371, + 0.05562204100715462, + 0.054953833008767106, + 0.05580554099287838, + 0.055839249995187856, + 0.05520575000264216, + 0.0556910000013886, + 0.054912874998990446, + 0.05523979099234566, + 0.05471108399797231, + 0.0549187919968972, + 0.05488070799037814, + 0.054618374997517094, + 0.0555732920038281, + 0.054618125010165386, + 0.054514832998393103, + 0.054700334003428, + 0.05493679099890869, + 0.05445258299005218, + 0.054690249991836026, + 0.055288458010181785, + 0.054835874994751066, + 0.05462570799863897, + 0.05524933399283327, + 0.054650332996970974, + 0.054579832998570055 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_mha": { + "first_token_ms": { + "count": 5, + "median": 62.67941600526683, + "p95": 69.32827480486594, + "min": 53.95300000964198, + "max": 69.8024580051424, + "iqr": 13.17300001392141, + "mean": 61.624991602730006, + "cv": 0.1189195748919102 + }, + "decode_ms_per_step": { + "count": 315, + "median": 71.33329201315064, + "p95": 80.5454044995713, + "min": 54.66100000194274, + "max": 85.36041699699126, + "iqr": 5.6446460002916865, + "mean": 71.76995225401322, + "cv": 0.07320673073626015 + }, + "total_ms": { + "count": 5, + "median": 4605.654961967957, + "p95": 4786.305448398343, + "min": 4318.453372077784, + "max": 4831.245665001916, + "iqr": 52.78324498794973, + "mean": 4583.131983605563, + "cv": 0.0398659691413727 + }, + "tokens_per_second": { + "count": 5, + "median": 111.16768499332541, + "p95": 117.33567747886681, + "min": 105.97680919208587, + "max": 118.56096520816571, + "iqr": 1.2883103712528765, + "mean": 111.85724042913333, + "cv": 0.04018853079200749 + }, + "median_step_ms_by_index": [ + 62.67941600526683, + 65.75245800195262, + 67.2177090018522, + 67.8062079969095, + 68.78704199334607, + 68.10820900136605, + 68.59708299452905, + 69.37149999430403, + 70.29754199902527, + 70.32220800465439, + 68.21366600343026, + 70.23858299362473, + 66.68337500013877, + 68.52204199822154, + 69.06433300173376, + 69.5952920068521, + 70.65920800960157, + 71.32149999961257, + 71.21562499378342, + 71.70733300154097, + 72.25579099031165, + 70.85020799422637, + 71.338708003168, + 71.29824999719858, + 71.3721250067465, + 72.08791701123118, + 69.7854160098359, + 73.14654099172913, + 73.46216699806973, + 73.5821670095902, + 74.42804200400133, + 72.8856670029927, + 72.9061250021914, + 74.60833300137892, + 73.9823749900097, + 72.66083298600279, + 74.13745899975765, + 72.7497920015594, + 72.32387499243487, + 72.7458339970326, + 74.19604100869037, + 77.23812499898486, + 74.34550000471063, + 76.3351670029806, + 77.17674999730662, + 76.53287499852013, + 74.99824999831617, + 72.59162499394733, + 72.11504100996535, + 76.02074999886099, + 74.48729200405069, + 70.85183399613015, + 74.53675000579096, + 70.88412500161212, + 70.51354199938942, + 70.46879100380465, + 70.1057920086896, + 70.65558299655095, + 70.24008400912862, + 70.6140419933945, + 71.33329201315064, + 73.30916699720547, + 71.92374998703599, + 71.78174999717157 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.06267941600526683, + 0.06575245800195262, + 0.0672177090018522, + 0.0678062079969095, + 0.06914258300093934, + 0.06810820900136605, + 0.06859708299452905, + 0.06937149999430403, + 0.0712926249980228, + 0.07127533300081268, + 0.06821366600343026, + 0.07023858299362473, + 0.0660367920063436, + 0.06579999999667052, + 0.06877345900284126, + 0.06820233300095424, + 0.06888749999052379, + 0.0703278749861056, + 0.06918791700445581, + 0.07115208299364895, + 0.06654791699838825, + 0.06863541700295173, + 0.06852070899913087, + 0.07129824999719858, + 0.0713721250067465, + 0.07080933300312608, + 0.06971199999679811, + 0.07314654099172913, + 0.07502512499922886, + 0.0759539160062559, + 0.07553449999250006, + 0.07498095800110605, + 0.0729061250021914, + 0.07631154199771117, + 0.07624458298960235, + 0.08217262499965727, + 0.07555074999982025, + 0.0727497920015594, + 0.0716268329997547, + 0.07377529100631364, + 0.07672312500653788, + 0.0782474169973284, + 0.07434550000471063, + 0.07907945899933111, + 0.07717674999730662, + 0.07887166700675152, + 0.07499824999831617, + 0.07259162499394733, + 0.07211504100996535, + 0.07668616701266728, + 0.07448729200405069, + 0.07085183399613015, + 0.07453675000579096, + 0.07182374999683816, + 0.07138666599348653, + 0.07282904098974541, + 0.07110604199988302, + 0.07146062499668915, + 0.06756608400610276, + 0.07108012500975747, + 0.07383070800278801, + 0.07592924998607486, + 0.07428850000724196, + 0.07359524999628775 + ], + [ + 0.054258541989838704, + 0.05653629200241994, + 0.058363500007544644, + 0.05840100000204984, + 0.058604624995496124, + 0.05959225000697188, + 0.05961616599233821, + 0.061432833012077026, + 0.0626457920006942, + 0.06271666700195055, + 0.06395554100163281, + 0.06524024999816902, + 0.06668337500013877, + 0.06852204199822154, + 0.06906433300173376, + 0.0695952920068521, + 0.07075195800280198, + 0.07132149999961257, + 0.07121562499378342, + 0.07170733300154097, + 0.07225579099031165, + 0.0721707500051707, + 0.07217391699668951, + 0.07314216700615361, + 0.07269691699184477, + 0.07250016699254047, + 0.07249349998892285, + 0.07347141600621399, + 0.07346216699806973, + 0.0735821670095902, + 0.07442804200400133, + 0.0728856670029927, + 0.07271125000261236, + 0.07460833300137892, + 0.0739823749900097, + 0.07266083298600279, + 0.07413745899975765, + 0.0739752499939641, + 0.07375024999782909, + 0.0727458339970326, + 0.07419604100869037, + 0.07723812499898486, + 0.07910404200083576, + 0.0763351670029806, + 0.07777966599678621, + 0.08381700000609271, + 0.08029729200643487, + 0.08066612499533221, + 0.07652220799354836, + 0.07602074999886099, + 0.07604912499664351, + 0.07795283298764843, + 0.08031804200436454, + 0.07843150000553578, + 0.08053929200104903, + 0.07838054199237376, + 0.07510737499978859, + 0.0787982500041835, + 0.07905429200036451, + 0.08071683399612084, + 0.07696733399643563, + 0.075231166993035, + 0.07132912499946542, + 0.07074366700544488 + ], + [ + 0.06743154200376011, + 0.06819808400177862, + 0.06866841699229553, + 0.06989062500360887, + 0.07503800000995398, + 0.07260320900240913, + 0.07648187500308268, + 0.07273029199859593, + 0.07165283299400471, + 0.07403116700879764, + 0.0739161669916939, + 0.07051599999249447, + 0.07219316699774936, + 0.07478795800125226, + 0.07585495800594799, + 0.07335099999909289, + 0.07065920800960157, + 0.07230962500034366, + 0.07435870799236, + 0.0721127080032602, + 0.07293587499589194, + 0.07085020799422637, + 0.071338708003168, + 0.07078458300384227, + 0.07132079200528096, + 0.07208791701123118, + 0.06923683399509173, + 0.07265804198686965, + 0.07169462500314694, + 0.07300662499619648, + 0.07308191701304168, + 0.06786895800905768, + 0.07935374999942724, + 0.06924258300568908, + 0.06710033399576787, + 0.06711224999162368, + 0.06832824999582954, + 0.07074520899914205, + 0.07232387499243487, + 0.07143629100755788, + 0.07330791698768735, + 0.07513295799435582, + 0.07223266600340139, + 0.07407283299835399, + 0.07358870799362194, + 0.07081641699187458, + 0.06685966700024437, + 0.06669645900547039, + 0.06688475000555627, + 0.06737499999871943, + 0.068111125001451, + 0.06724587500502821, + 0.06722729200555477, + 0.06887304200790823, + 0.06871974999376107, + 0.06894987499981653, + 0.06922283298627008, + 0.07031508399813902, + 0.07078024999646004, + 0.07060029200511053, + 0.07133329201315064, + 0.071738083002856, + 0.07192374998703599, + 0.07246024999767542 + ], + [ + 0.0698024580051424, + 0.06896425000741147, + 0.06810474999656435, + 0.06943324999883771, + 0.06878704199334607, + 0.06994087500788737, + 0.0697123330028262, + 0.07140129199251533, + 0.07029754199902527, + 0.07032220800465439, + 0.0714629580033943, + 0.0709588750032708, + 0.07111670800077263, + 0.07198204199085012, + 0.07294945800094865, + 0.07502758300688583, + 0.07518766699649859, + 0.07590591700864024, + 0.07582729098794516, + 0.0744265420071315, + 0.07454529200913385, + 0.07993570799590088, + 0.07770358299603686, + 0.07794425000611227, + 0.07865029099048115, + 0.07714645800297149, + 0.07747158299025614, + 0.07783233300142456, + 0.0794728339969879, + 0.07868445799977053, + 0.07626470799732488, + 0.07562862501072232, + 0.07841137499781325, + 0.08137604199873749, + 0.08172254099918064, + 0.08170475000224542, + 0.08083637499657925, + 0.07978975000150967, + 0.08369924999715295, + 0.08070270900498144, + 0.07961745800275821, + 0.08076349999464583, + 0.08536041699699126, + 0.07995779099292122, + 0.08055966699612327, + 0.07653287499852013, + 0.07680654199793935, + 0.08465616600005887, + 0.07902641700638924, + 0.0790887499897508, + 0.08208470800309442, + 0.0826600839936873, + 0.07603758299956098, + 0.07061833400803152, + 0.07038800000736956, + 0.06829875000403263, + 0.06849554200016428, + 0.06753362499875948, + 0.06842924999364186, + 0.06963737501064315, + 0.07038470800034702, + 0.07330916699720547, + 0.07808325000223704, + 0.07178174999717157 + ], + [ + 0.05395300000964198, + 0.056840791992726736, + 0.05466100000194274, + 0.05478649999713525, + 0.055494790998636745, + 0.056531875001383014, + 0.05767300000297837, + 0.058994542006985284, + 0.0598829999944428, + 0.06140979200426955, + 0.06190087500726804, + 0.06293870799709111, + 0.06420420900394674, + 0.06493749999208376, + 0.06553949999215547, + 0.06646495799941476, + 0.06711262500903104, + 0.06722504200297408, + 0.06817345900344662, + 0.06833987499703653, + 0.06876095899497159, + 0.06998387500061654, + 0.06922029099951033, + 0.0689289580041077, + 0.07000387500738725, + 0.06999483400431927, + 0.0697854160098359, + 0.06969204101187643, + 0.06974579200323205, + 0.0700900410010945, + 0.06999479200749192, + 0.07011629101180006, + 0.06976816699898336, + 0.0712687079940224, + 0.07162604200129863, + 0.07170920800126623, + 0.07070270800613798, + 0.07054745800269302, + 0.07052787499560509, + 0.07046483299927786, + 0.07009724999079481, + 0.07108887500362471, + 0.07076558300468605, + 0.07021220799651928, + 0.07022712500474881, + 0.07038241600093897, + 0.06961137500184122, + 0.0703804999939166, + 0.06973354100773577, + 0.06997145900095347, + 0.07028629200067371, + 0.07044287500320934, + 0.0705684169952292, + 0.07088412500161212, + 0.07051354199938942, + 0.07046879100380465, + 0.0701057920086896, + 0.07065558299655095, + 0.07024008400912862, + 0.0706140419933945, + 0.0703260419977596, + 0.06923704099608585, + 0.06994229099655058, + 0.07170091599982698 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "speedup_cached_vs_uncached_sdpa": 8.700941830281089, + "speedup_cached_vs_uncached_mha": 11.343739071884954 + } + ], + "window": { + "batch": 1, + "prompt_len": 32, + "context_length": 128, + "new_tokens": 224, + "token_identical_to_cached": { + "cached": true, + "uncached_sdpa": true, + "uncached_mha": true + }, + "cached": { + "first_token_ms": { + "count": 5, + "median": 1.4496669900836423, + "p95": 1.5866913949139416, + "min": 1.3198750093579292, + "max": 1.6020829934859648, + "iqr": 0.1162919943453744, + "mean": 1.461116599966772, + "cv": 0.07401617358182219 + }, + "decode_ms_per_step": { + "count": 1115, + "median": 1.6848749946802855, + "p95": 1.8204963984317146, + "min": 1.494166994234547, + "max": 2.6582499995129183, + "iqr": 0.16291649808408692, + "mean": 1.686159821554513, + "cv": 0.07353366391188304 + }, + "total_ms": { + "count": 5, + "median": 376.6727979382267, + "p95": 385.040429251967, + "min": 372.6677860249765, + "max": 387.06491207994986, + "iqr": 2.9167078901082277, + "mean": 377.4747568066232, + "cv": 0.014980182945568452 + }, + "tokens_per_second": { + "count": 5, + "median": 594.6805854473605, + "p95": 600.6350685773191, + "min": 578.7143009070578, + "max": 601.0715398539635, + "iqr": 4.634088266182857, + "mean": 593.5221409767364, + "cv": 0.014759336552875032 + }, + "median_step_ms_by_index": [ + 1.4496669900836423, + 1.84041699685622, + 1.596458998392336, + 1.6933750011958182, + 1.5866670000832528, + 1.6001669864635915, + 1.5692500019213185, + 1.603249998879619, + 1.5617919998476282, + 1.570709006045945, + 1.6681249981047586, + 1.5658339980291203, + 1.5796249936101958, + 1.5585420042043552, + 1.57891699927859, + 1.5852910000830889, + 1.6259169933618978, + 1.5703329991083592, + 1.5673749876441434, + 1.6364160110242665, + 1.5810830082045868, + 1.6008750098990276, + 1.7399159987689927, + 1.5657500043744221, + 1.5798340027686208, + 1.6717499966034666, + 1.572583001689054, + 1.5812080091563985, + 1.6199579986277968, + 1.5614579897373915, + 1.5770409954711795, + 1.5738340007374063, + 1.5585840010317042, + 1.5930000081425533, + 1.5718340000603348, + 1.567250001244247, + 1.5812079946044832, + 1.5652500005671754, + 1.5702500095358118, + 1.5651249996153638, + 1.572792010847479, + 1.5766249998705462, + 1.5713339962530881, + 1.5656250034226105, + 1.5681249933550134, + 1.5711250016465783, + 1.669917008257471, + 1.5591669944114983, + 1.5915420081000775, + 1.5706249978393316, + 1.5614169969921932, + 1.5630830021109432, + 1.6199579986277968, + 1.5744160045869648, + 1.6031249979278073, + 1.59366600564681, + 1.5915420081000775, + 1.5610829868819565, + 1.5861660067457706, + 1.583040997502394, + 1.5719160001026466, + 1.5759999951114878, + 1.573833986185491, + 1.570874999742955, + 1.5698750066803768, + 1.59108400112018, + 1.5680829965276644, + 1.5782500122440979, + 1.5637500036973506, + 1.5661250072298571, + 1.5657909971196204, + 1.5718750073574483, + 1.56004200107418, + 1.5751669998280704, + 1.5859589911997318, + 1.584959012689069, + 1.7097919917432591, + 1.6274999943561852, + 1.5666250110371038, + 1.5675829927204177, + 1.5748340083518997, + 1.5704159886809066, + 1.593125009094365, + 1.5978340088622645, + 1.5950829983921722, + 1.59029099449981, + 1.6107080009533092, + 1.6046670061768964, + 1.6055829910328612, + 1.589167004567571, + 1.59200000052806, + 1.6313750093104318, + 1.6173750045709312, + 1.6130410076584667, + 1.5984170022420585, + 1.5894579992163926, + 1.5922080056043342, + 1.7313749995082617, + 1.7525000002933666, + 1.7574169905856252, + 1.7198749992530793, + 1.7740830080583692, + 1.7052500043064356, + 1.6992919991025701, + 1.7081249970942736, + 1.7419580108253285, + 1.7160840070573613, + 1.6954580059973523, + 1.7635000112932175, + 1.7126250022556633, + 1.7141670105047524, + 1.7081669939216226, + 1.6952910082181916, + 1.7050419992301613, + 1.7013330070767552, + 1.685749986791052, + 1.690832999884151, + 1.782874998752959, + 1.7583750013727695, + 1.707834002445452, + 1.714416008326225, + 1.724000001559034, + 1.7236659914487973, + 1.744374996633269, + 1.7379999917466193, + 1.6710000054445118, + 1.673332997597754, + 1.7044999985955656, + 1.7247910000151023, + 1.7370419955113903, + 1.6969999996945262, + 1.7056670039892197, + 1.7319170001428574, + 1.7461669922340661, + 1.6989580035442486, + 1.7629999929340556, + 1.6730830102460459, + 1.7306660010945052, + 1.7039169906638563, + 1.7329580005025491, + 1.7570000054547563, + 1.762416010024026, + 1.7436670023016632, + 1.6984169924398884, + 1.7335000011371449, + 1.6971670120256022, + 1.697166997473687, + 1.7237919964827597, + 1.6797919961391017, + 1.7437500064261258, + 1.7505830037407577, + 1.712915996904485, + 1.7415830079698935, + 1.701500004855916, + 1.7594169912626967, + 1.7026669956976548, + 1.748416994814761, + 1.7168750055134296, + 1.7619999998714775, + 1.7665000050328672, + 1.7104579892475158, + 1.7050419992301613, + 1.7116660019382834, + 1.7069160094251856, + 1.6971250006463379, + 1.732207994791679, + 1.7162080039270222, + 1.738833001581952, + 1.720582993584685, + 1.7157499969471246, + 1.7366250103805214, + 1.7690829990897328, + 1.7693749978207052, + 1.7413750028936192, + 1.724000001559034, + 1.7539999971631914, + 1.6816249990370125, + 1.6788750072009861, + 1.6886660014279187, + 1.7202080052811652, + 1.7115000082412735, + 1.723666995530948, + 1.7162919975817204, + 1.7421670054318383, + 1.7535830120323226, + 1.74937499104999, + 1.7070830072043464, + 1.717542007099837, + 1.76470800943207, + 1.7614999960642308, + 1.7572920041857287, + 1.7566659953445196, + 1.723207999020815, + 1.725708003505133, + 1.6882909985724837, + 1.7310419934801757, + 1.7654590046731755, + 1.7930419999174774, + 1.7660000012256205, + 1.780333011993207, + 1.7162909934995696, + 1.6935420135268942, + 1.7563340079504997, + 1.7555000085849315, + 1.7968750034924597, + 1.6900830087251961, + 1.699415995972231, + 1.7661250021774322, + 1.744749999488704, + 1.747250003973022, + 1.7759589973138645, + 1.7526669980725273, + 1.7527089948998764, + 1.7178330017486587, + 1.7529589968034998, + 1.744791996316053, + 1.674083003308624, + 1.7317500023636967, + 1.7518750100862235, + 1.7326670058537275, + 1.762667001457885, + 1.745583998854272, + 1.735459009069018, + 1.7064169951481745 + ], + "refreshed_steps": [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0016020829934859648, + 0.0024061250005615875, + 0.0015903330058790743, + 0.0016933750011958182, + 0.0015752910112496465, + 0.0015799170068930835, + 0.0015510420053033158, + 0.0016559579962631688, + 0.0015502079913858324, + 0.0015642090002074838, + 0.0015530410018982366, + 0.0016488340042997152, + 0.0016330000071320683, + 0.0015994999994290993, + 0.00157891699927859, + 0.0015884580061538145, + 0.0016688750038156286, + 0.001559625001391396, + 0.0015673749876441434, + 0.0015544999914709479, + 0.0015896670083748177, + 0.0015862500004004687, + 0.0018071250087814406, + 0.0016367499920306727, + 0.0015648330008843914, + 0.0016717499966034666, + 0.001572583001689054, + 0.0015812080091563985, + 0.001563874990097247, + 0.0015646669926354662, + 0.0016851669934112579, + 0.0015896249969955534, + 0.0015539579908363521, + 0.0017389999993611127, + 0.0016727079928386956, + 0.001568958003190346, + 0.0015827919996809214, + 0.001563958008773625, + 0.0015442500007338822, + 0.0016482920036651194, + 0.001572792010847479, + 0.0015766249998705462, + 0.0015713339962530881, + 0.0015315409982576966, + 0.0015721250092610717, + 0.0016140840016305447, + 0.00167966699518729, + 0.0015580409963149577, + 0.0015915420081000775, + 0.0015623749932274222, + 0.0015574999997625127, + 0.00154208300227765, + 0.001662334005231969, + 0.0017374169983668253, + 0.0015734170010546222, + 0.0015951249952195212, + 0.0015554999990854412, + 0.001585791993420571, + 0.0015737089997855946, + 0.0015244999958667904, + 0.001665416988544166, + 0.0015854170051170513, + 0.0015922910097287968, + 0.0015786250005476177, + 0.0015547909861197695, + 0.0016071669961092994, + 0.0015680829965276644, + 0.0015787910087965429, + 0.0015670830034650862, + 0.0015661250072298571, + 0.0016288749902741984, + 0.0015875829994911328, + 0.0015552499971818179, + 0.001556582996272482, + 0.0015859589911997318, + 0.0016335419932147488, + 0.0022829170047771186, + 0.0016411249962402508, + 0.0015582920023007318, + 0.001666833006311208, + 0.0015748340083518997, + 0.0015457910049008206, + 0.0016175830096472055, + 0.0021901670115767047, + 0.0016907079989323393, + 0.0016426660004071891, + 0.0015873749944148585, + 0.0016052919963840395, + 0.001765666005667299, + 0.0016256670060101897, + 0.0017023749969666824, + 0.0016313750093104318, + 0.001632999992580153, + 0.0016302919975714758, + 0.0016106249968288466, + 0.001624416996492073, + 0.0016404169873567298, + 0.0017779579939087853, + 0.0017924580024555326, + 0.0017717919981805608, + 0.0017198749992530793, + 0.001766416011378169, + 0.0016230000037467107, + 0.0017259580054087564, + 0.0016384579939767718, + 0.001772791991243139, + 0.0016951249999692664, + 0.0017854579928098246, + 0.0017635000112932175, + 0.0016593329928582534, + 0.0016846249927766621, + 0.0017176670080516487, + 0.0016489170084241778, + 0.0016611250030109659, + 0.0017775409942260012, + 0.0017860000079963356, + 0.0017149999912362546, + 0.0017084999999497086, + 0.0016912500141188502, + 0.0016972909943433478, + 0.0016790829977253452, + 0.0017382499936502427, + 0.001746875001117587, + 0.001768165995599702, + 0.0017379999917466193, + 0.0016605840064585209, + 0.0016526669933227822, + 0.0016723330045351759, + 0.0017419579962734133, + 0.0017626249900786206, + 0.001770041009876877, + 0.0017056670039892197, + 0.0017319170001428574, + 0.0017438750073779374, + 0.0016732079966459423, + 0.0016671670018695295, + 0.0017225829942617565, + 0.0017482919938629493, + 0.0016979999927571043, + 0.0016164160042535514, + 0.0016511249996256083, + 0.0016925840027397498, + 0.0016499999910593033, + 0.0016984169924398884, + 0.0017496670043328777, + 0.0017706249927869067, + 0.0016863749915501103, + 0.001709374992060475, + 0.0016682499990565702, + 0.001673708000453189, + 0.0017417500057490543, + 0.0017570829950273037, + 0.00190320800174959, + 0.0017637499986449257, + 0.0017594169912626967, + 0.0016826250066515058, + 0.0016568340070080012, + 0.0016839159943629056, + 0.0016841670003486797, + 0.0017759170004865155, + 0.0017795829917304218, + 0.001785332991858013, + 0.00196508300723508, + 0.0017722909979056567, + 0.0016675410006428137, + 0.0016720410058042035, + 0.0017162080039270222, + 0.0016957079933490604, + 0.0017457500071031973, + 0.0017697920120554045, + 0.0017673330003162846, + 0.0017510829929960892, + 0.001740375009831041, + 0.0016772499948274344, + 0.0016904590011108667, + 0.0017539999971631914, + 0.0017574159865034744, + 0.0016761669976403937, + 0.001643415991566144, + 0.0016673329955665395, + 0.0016634589992463589, + 0.0017078750097425655, + 0.0017162919975817204, + 0.0017317499878117815, + 0.0017535830120323226, + 0.0017594999953871593, + 0.0017423749959561974, + 0.001758083002641797, + 0.0017060420068446547, + 0.0017008750000968575, + 0.0016749579954193905, + 0.0018981670000357553, + 0.0017669170047156513, + 0.0017522089910926297, + 0.0017528340104036033, + 0.0017505840078229085, + 0.0016628340090392157, + 0.0016642919945297763, + 0.0016781250014901161, + 0.0017449589940952137, + 0.0017567089962540194, + 0.0018341670074732974, + 0.0017964160069823265, + 0.0017546669987495989, + 0.0016757080011302605, + 0.0016732499934732914, + 0.001699415995972231, + 0.0017729170067468658, + 0.001757583988364786, + 0.001747250003973022, + 0.0018220420024590567, + 0.0018193750001955777, + 0.0017502499977126718, + 0.001696791994618252, + 0.0017529589968034998, + 0.0017019999941112474, + 0.0016738749982323498, + 0.0017317500023636967, + 0.00175104099616874, + 0.0017676660063443705, + 0.001829500004532747, + 0.0017267910006921738, + 0.0016629580059088767, + 0.0017064169951481745 + ], + [ + 0.0014496669900836423, + 0.0023292080004466698, + 0.001617208996322006, + 0.0018527499923948199, + 0.001568375009810552, + 0.0016806670028017834, + 0.0015692500019213185, + 0.0015535830025328323, + 0.0015617919998476282, + 0.0015344170096796006, + 0.0016713329969206825, + 0.001564750011311844, + 0.0015175000007729977, + 0.0015585420042043552, + 0.0015469159989152104, + 0.0015536669961875305, + 0.0016259169933618978, + 0.0015703329991083592, + 0.0015633750008419156, + 0.001648207995458506, + 0.0015810830082045868, + 0.0016008750098990276, + 0.0015758749941596761, + 0.0015657500043744221, + 0.0017740000039339066, + 0.0015885000029811636, + 0.0015522500034421682, + 0.001537584001198411, + 0.001677332998951897, + 0.001538083000923507, + 0.001494166994234547, + 0.0015614579897373915, + 0.0015585840010317042, + 0.0015771250036777928, + 0.0015580409963149577, + 0.001567250001244247, + 0.0015812079946044832, + 0.001680000001215376, + 0.0015702500095358118, + 0.0015939169970806688, + 0.0015778749948367476, + 0.0015825409936951473, + 0.0015792909980518743, + 0.0015877080004429445, + 0.0015681249933550134, + 0.001763084001140669, + 0.001669917008257471, + 0.0015670410066377372, + 0.0015593330026604235, + 0.0015835840022191405, + 0.0015614169969921932, + 0.0015462500014109537, + 0.0016092080040834844, + 0.0015633329894626513, + 0.001577959003043361, + 0.0016722080035833642, + 0.0015915420081000775, + 0.0015599159960402176, + 0.0015861660067457706, + 0.001585374993737787, + 0.0015719160001026466, + 0.0015759999951114878, + 0.001573833986185491, + 0.001570874999742955, + 0.0016806659987196326, + 0.00159108400112018, + 0.0015583330095978454, + 0.0015595419972669333, + 0.0015593749994877726, + 0.0015633329894626513, + 0.0015657909971196204, + 0.0015639170014765114, + 0.0015267919952748343, + 0.0016848749946802855, + 0.0015692919987486675, + 0.001573333007399924, + 0.0017470830061938614, + 0.0016274999943561852, + 0.0015666250110371038, + 0.0015574999997625127, + 0.0015637920005246997, + 0.0015728750004200265, + 0.001568042003782466, + 0.001568042003782466, + 0.0015524160116910934, + 0.0015517919964622706, + 0.0016841250035213307, + 0.0015342090046033263, + 0.0016055829910328612, + 0.001556207993417047, + 0.0015533750120084733, + 0.0016604999982519075, + 0.0015547080110991374, + 0.0016130410076584667, + 0.00155237500439398, + 0.001555041002575308, + 0.0015605419903295115, + 0.0017313749995082617, + 0.001735040990752168, + 0.001682667003478855, + 0.0017519579996587709, + 0.0018039590067928657, + 0.0017052500043064356, + 0.001806291998946108, + 0.0017081249970942736, + 0.0017104579892475158, + 0.0017160840070573613, + 0.001665500007220544, + 0.0017405830003554001, + 0.0017828329873736948, + 0.0017141670105047524, + 0.0017081669939216226, + 0.0016952910082181916, + 0.001698124993708916, + 0.0017013330070767552, + 0.001685749986791052, + 0.0016870410036062822, + 0.0017985000013140962, + 0.0017688329971861094, + 0.001707834002445452, + 0.001714416008326225, + 0.0016969580028671771, + 0.0016900000046007335, + 0.0017482089897384867, + 0.0017671669920673594, + 0.001774667005520314, + 0.0017542079876875505, + 0.001765916997101158, + 0.001662499998928979, + 0.0016697920073056594, + 0.0016969999996945262, + 0.0016797910066088662, + 0.001731583004584536, + 0.001777541998308152, + 0.0017213330138474703, + 0.0017629999929340556, + 0.0016707499889889732, + 0.0016794999974081293, + 0.0017725000070640817, + 0.001774625008692965, + 0.001759624996338971, + 0.001684750008280389, + 0.0017880840023281053, + 0.001676375002716668, + 0.0016924160008784384, + 0.0016713750082999468, + 0.0016607080033281818, + 0.0017237919964827597, + 0.0017396660114172846, + 0.0017538749962113798, + 0.0017566250025993213, + 0.0016397080034948885, + 0.0016679169930284843, + 0.001701500004855916, + 0.0017629999929340556, + 0.0017612920055398718, + 0.0017662090103840455, + 0.0017783750081434846, + 0.0017638749995967373, + 0.0017665000050328672, + 0.0017104579892475158, + 0.0017050419992301613, + 0.0017745000077411532, + 0.0016941670037340373, + 0.001697166997473687, + 0.001732207994791679, + 0.0016940839996095747, + 0.0017568330076755956, + 0.0017395409959135577, + 0.0017157499969471246, + 0.0017089169996324927, + 0.001833166999858804, + 0.001773166994098574, + 0.0016709170013200492, + 0.0016879169997991994, + 0.0018905000033555552, + 0.0016816249990370125, + 0.001661583999521099, + 0.0017044159903889522, + 0.0017707910010358319, + 0.0016652909980621189, + 0.0017512079939479008, + 0.0017675419949227944, + 0.0017627500055823475, + 0.0016699999978300184, + 0.0017556660022819415, + 0.0017070830072043464, + 0.0017051249888027087, + 0.0017585000023245811, + 0.0017624169995542616, + 0.0017572920041857287, + 0.0017566659953445196, + 0.0016660840046824887, + 0.0016753749951021746, + 0.0017352089926134795, + 0.0017310419934801757, + 0.0017382079968228936, + 0.0017459999944549054, + 0.0016829999949550256, + 0.001755834004143253, + 0.0017162909934995696, + 0.0016935420135268942, + 0.0017563340079504997, + 0.00175012499676086, + 0.0018155409925384447, + 0.001657917004195042, + 0.0016467919922433794, + 0.001677125008427538, + 0.001668957993388176, + 0.0016823329933686182, + 0.0016785829939180985, + 0.0017526669980725273, + 0.001760207989718765, + 0.0017178330017486587, + 0.0017368749977322295, + 0.001744791996316053, + 0.001674083003308624, + 0.0017278749874094501, + 0.0017518750100862235, + 0.0017326670058537275, + 0.001766291999956593, + 0.0017652920068940148, + 0.001735459009069018, + 0.0017740000039339066 + ], + [ + 0.0014088330062804744, + 0.0017878750077215955, + 0.001596458998392336, + 0.0016102090012282133, + 0.0016957920015556738, + 0.0016001669864635915, + 0.0015673750021960586, + 0.0015578750026179478, + 0.0015637080068700016, + 0.001570709006045945, + 0.0015313750045606866, + 0.0015652080037398264, + 0.0015104170015547425, + 0.0015516670100623742, + 0.0015244999958667904, + 0.0015852910000830889, + 0.0015776670043123886, + 0.0016364579932997003, + 0.0017693749978207052, + 0.0015502500027650967, + 0.0015080000011948869, + 0.0018989579984918237, + 0.0017399159987689927, + 0.0015624160005245358, + 0.0015798340027686208, + 0.001876333000836894, + 0.0015881250001257285, + 0.0015439999988302588, + 0.0016199579986277968, + 0.0015419999981531873, + 0.001612958003534004, + 0.001687625001068227, + 0.0015578750026179478, + 0.0015982500044628978, + 0.0015718340000603348, + 0.001750708994222805, + 0.001602709002327174, + 0.0015550420066574588, + 0.0015726669953437522, + 0.0015627910033799708, + 0.0016199999954551458, + 0.0015801249974174425, + 0.0015662500081816688, + 0.0015599579928675666, + 0.0017833329911809415, + 0.0015615419979440048, + 0.0016118750063469633, + 0.0015590419934596866, + 0.001610916995559819, + 0.0016976670012809336, + 0.0016402920009568334, + 0.0015751659957459196, + 0.0016199579986277968, + 0.0015891669900156558, + 0.0016193749906960875, + 0.0015576669975416735, + 0.0016439589962828904, + 0.0015525000053457916, + 0.0017115829978138208, + 0.0015974999987520278, + 0.0015512909885728732, + 0.0015560829924652353, + 0.0015588329988531768, + 0.0015790420002304018, + 0.0015737500070827082, + 0.001550458007841371, + 0.0017197080014739186, + 0.0017112909990828484, + 0.001582875003805384, + 0.0015490829973714426, + 0.001595249996171333, + 0.0015955840062815696, + 0.0015648750122636557, + 0.0015544170018984005, + 0.001573333007399924, + 0.0016117499908432364, + 0.0017097919917432591, + 0.0015585000073770061, + 0.0016006250079954043, + 0.001600207993760705, + 0.0015887089975876734, + 0.0015704159886809066, + 0.001593125009094365, + 0.001694292004685849, + 0.0015952090034261346, + 0.0015320410020649433, + 0.0015405839949380606, + 0.0016097500047180802, + 0.0015795419894857332, + 0.0015699999930802733, + 0.0015849170013098046, + 0.001575667003635317, + 0.0016310420032823458, + 0.001673708000453189, + 0.001688165997620672, + 0.0015808329917490482, + 0.001563666999572888, + 0.0016447499947389588, + 0.0017525000002933666, + 0.0017796669999370351, + 0.0017637910059420392, + 0.00178958399919793, + 0.0017532079946249723, + 0.0016650420002406463, + 0.0017278750019613653, + 0.0017086669977288693, + 0.0018490420043235645, + 0.0017804999952204525, + 0.0017839579959399998, + 0.0017126250022556633, + 0.0017594579985598102, + 0.001662833004957065, + 0.001687249998212792, + 0.0017538340034661815, + 0.001748957991367206, + 0.0016528750129509717, + 0.0018198339967057109, + 0.001782874998752959, + 0.0017589159979252145, + 0.0016966669936664402, + 0.0016754169919295236, + 0.001724000001559034, + 0.0017275829886784777, + 0.001744374996633269, + 0.0017947919986909255, + 0.0016710000054445118, + 0.001673332997597754, + 0.0017044999985955656, + 0.0017247910000151023, + 0.001749125003698282, + 0.001741959000355564, + 0.0017947500018635765, + 0.001764750006259419, + 0.0017692080000415444, + 0.0017679170123301446, + 0.0017797090113162994, + 0.0017012919997796416, + 0.001741208994644694, + 0.0017089169996324927, + 0.0016981670050881803, + 0.0017319170001428574, + 0.001762416010024026, + 0.0017436670023016632, + 0.0017565419984748587, + 0.0017335000011371449, + 0.0016777090058894828, + 0.001697166997473687, + 0.0017725419893395156, + 0.0016487499960931018, + 0.0017437500064261258, + 0.001768041998730041, + 0.0017038749938365072, + 0.0017446669953642413, + 0.001698250009212643, + 0.0018052919913316146, + 0.0017026669956976548, + 0.0017759579932317138, + 0.0017736669979058206, + 0.001793749994249083, + 0.0016685000009601936, + 0.0016540000069653615, + 0.001679624998359941, + 0.0017116660019382834, + 0.0017069160094251856, + 0.0016817499999888241, + 0.0017102919955505058, + 0.001801291000447236, + 0.0017666249914327636, + 0.001706332986941561, + 0.0017142500000772998, + 0.0017366250103805214, + 0.0017718750023050234, + 0.0017821669898694381, + 0.0017413750028936192, + 0.0017717080045258626, + 0.0017741250048857182, + 0.001707999996142462, + 0.0016788750072009861, + 0.0016886660014279187, + 0.0016979579959297553, + 0.0017529160104459152, + 0.0017735830042511225, + 0.001796083990484476, + 0.001828749998821877, + 0.0017561249987920746, + 0.0016890829865587875, + 0.0016816249990370125, + 0.001717542007099837, + 0.00176470800943207, + 0.0017614999960642308, + 0.001774791002389975, + 0.0017766249948181212, + 0.001750416005961597, + 0.0018071249942295253, + 0.0016859169991221279, + 0.0016510830027982593, + 0.0017654590046731755, + 0.0018368749879300594, + 0.00186358300561551, + 0.00180079200072214, + 0.0017623749881749973, + 0.0017627920024096966, + 0.001839624994318001, + 0.0017737080052029341, + 0.0016889999969862401, + 0.0016900830087251961, + 0.001709332995233126, + 0.0017661250021774322, + 0.001744749999488704, + 0.0017797919863369316, + 0.0017759589973138645, + 0.0017785839881980792, + 0.0017527089948998764, + 0.0016927079996094108, + 0.0016970419965218753, + 0.0016930839919950813, + 0.0017491659964434803, + 0.0017782909999368712, + 0.001782709005055949, + 0.0017765420052455738, + 0.001762667001457885, + 0.001745583998854272, + 0.0017385829996783286, + 0.001699250002275221 + ], + [ + 0.0015251250006258488, + 0.001837125004385598, + 0.0017287919908994809, + 0.0018221250065835193, + 0.0019122919911751524, + 0.001904333010315895, + 0.0022485829977085814, + 0.0018458749982528389, + 0.001923166011692956, + 0.002338834005058743, + 0.001843459001975134, + 0.0026412090082885697, + 0.0026582499995129183, + 0.0025701669947011396, + 0.0026073750050272793, + 0.0026382910000393167, + 0.0025602079986128956, + 0.001926084005390294, + 0.0019188330043107271, + 0.0019247089949203655, + 0.002383332990575582, + 0.0017893330077640712, + 0.0018909999926108867, + 0.0023013330064713955, + 0.0019155420013703406, + 0.0017065419960999861, + 0.001579415999003686, + 0.0016087089898064733, + 0.0015667919942643493, + 0.0016038339963415638, + 0.0015770409954711795, + 0.0015738340007374063, + 0.001599249997525476, + 0.0015911249938653782, + 0.0016285419987980276, + 0.001548209009342827, + 0.0015768330049468204, + 0.0015652500005671754, + 0.0015607499954057857, + 0.0015619170007994398, + 0.0015505409974139184, + 0.0015610419941367581, + 0.001563499987241812, + 0.00167108399909921, + 0.0015635830059181899, + 0.0015645420062355697, + 0.0015693329914938658, + 0.0015591669944114983, + 0.0016064590017776936, + 0.0015706249978393316, + 0.0016312079969793558, + 0.001613542000995949, + 0.0016976250044535846, + 0.0015744160045869648, + 0.0019278330000815913, + 0.00159366600564681, + 0.0016498750046594068, + 0.0016679159889463335, + 0.0016755840042605996, + 0.001583040997502394, + 0.001603832992259413, + 0.0015837499959161505, + 0.0015845419984543696, + 0.0015683329984312877, + 0.0015664160018786788, + 0.0016067919932538643, + 0.0015479590074392036, + 0.0015782500122440979, + 0.0015591249975841492, + 0.0015699579962529242, + 0.0015650830027880147, + 0.0015659999917261302, + 0.00156004200107418, + 0.0017544589936733246, + 0.0015985000063665211, + 0.001584959012689069, + 0.0015633750008419156, + 0.0016790419904282317, + 0.0016507499967701733, + 0.001536916010081768, + 0.0015957920113578439, + 0.0015684169920859858, + 0.0015970829990692437, + 0.0015969169908203185, + 0.0015935830015223473, + 0.00159029099449981, + 0.0016368329961551353, + 0.0015921250014798716, + 0.0016062909999163821, + 0.001589167004567571, + 0.0015930420049699023, + 0.0015919169964035973, + 0.0015945830091368407, + 0.001600207993760705, + 0.0015984170022420585, + 0.0015894579992163926, + 0.0015922080056043342, + 0.0017229579971171916, + 0.0017332080024061725, + 0.0017393750022165477, + 0.001713167002890259, + 0.0017740830080583692, + 0.0017593330121599138, + 0.0016992919991025701, + 0.0016794999974081293, + 0.0017419580108253285, + 0.0016937920008786023, + 0.0016954580059973523, + 0.001816375006455928, + 0.0017490839964011684, + 0.0017881669919006526, + 0.0017732499982230365, + 0.0017093330097850412, + 0.0017050419992301613, + 0.0015778749948367476, + 0.001669875011430122, + 0.0016892499988898635, + 0.0018405839946353808, + 0.0017583750013727695, + 0.0017761249910108745, + 0.00175350000790786, + 0.0016957920015556738, + 0.0017236659914487973, + 0.0017073749913834035, + 0.0016903750074561685, + 0.00177087499469053, + 0.0017633329989621416, + 0.001858124989666976, + 0.0017499579989816993, + 0.0017370419955113903, + 0.001665167001192458, + 0.0016712079959688708, + 0.0016311670042341575, + 0.0016588329890510067, + 0.0016989580035442486, + 0.00177237500611227, + 0.0016730830102460459, + 0.0017206250049639493, + 0.0017039169906638563, + 0.0017332500137854367, + 0.0017570000054547563, + 0.0017735829896992072, + 0.001765125009114854, + 0.00179312500404194, + 0.0017883750115288422, + 0.0017436670023016632, + 0.0017136249953182414, + 0.0016829160013003275, + 0.0016797919961391017, + 0.001615291999769397, + 0.001746667010593228, + 0.0017874580080388114, + 0.0017415830079698935, + 0.0018950410012621433, + 0.0016865829966263846, + 0.0016847079969011247, + 0.0016443339991383255, + 0.0016156249912455678, + 0.0017619999998714775, + 0.0017076249932870269, + 0.0016776249976828694, + 0.0016683330031810328, + 0.0016955409955698997, + 0.0016875420114956796, + 0.0017332919960608706, + 0.0017510000034235418, + 0.001763583000865765, + 0.001738833001581952, + 0.001720582993584685, + 0.0017278750019613653, + 0.001708791998680681, + 0.00163050000264775, + 0.0017536670056870207, + 0.001765500011970289, + 0.001724000001559034, + 0.0016877919988473877, + 0.0016787909989943728, + 0.0017102910060202703, + 0.0016458750033052638, + 0.0017202080052811652, + 0.0017115000082412735, + 0.001723666995530948, + 0.001702250010566786, + 0.0016789580113254488, + 0.0016757080011302605, + 0.0017356669995933771, + 0.0017627909983275458, + 0.0017067089938791469, + 0.0017684169870335609, + 0.0017364999948767945, + 0.0017789580015232787, + 0.0016838750016177073, + 0.0016914169973460957, + 0.001725708003505133, + 0.0016708329931134358, + 0.0017307919915765524, + 0.0017959160031750798, + 0.0017930419999174774, + 0.0017660000012256205, + 0.001780333011993207, + 0.0016577909991610795, + 0.0016720409912522882, + 0.0016882909985724837, + 0.0017567079921718687, + 0.001852000001235865, + 0.0017770840058801696, + 0.0016682080022292212, + 0.0017211669910466298, + 0.0017115420050686225, + 0.0017003329994622618, + 0.00178283300192561, + 0.0016764580068411306, + 0.0017661669990047812, + 0.0017728340026224032, + 0.001934457992319949, + 0.0017614170064916834, + 0.001662082999246195, + 0.001680916000623256, + 0.0017070000030798838, + 0.0016652080084895715, + 0.0017389999993611127, + 0.0017663340113358572, + 0.001762124986271374, + 0.0017536249943077564 + ], + [ + 0.0013198750093579292, + 0.00184041699685622, + 0.0015569590032100677, + 0.001679415989201516, + 0.0015866670000832528, + 0.0015562089974991977, + 0.0018133749981643632, + 0.001603249998879619, + 0.0015576670120935887, + 0.001576208000187762, + 0.0016681249981047586, + 0.0015658339980291203, + 0.0015796249936101958, + 0.0015518339932896197, + 0.0016638329980196431, + 0.0015759579982841387, + 0.001571124987094663, + 0.0015508329961448908, + 0.00155312501010485, + 0.0016364160110242665, + 0.0015637079923180863, + 0.0015686249971622601, + 0.0015544159978162497, + 0.001552834000904113, + 0.0015515830018557608, + 0.0015657920012017712, + 0.0015507499920204282, + 0.0018023330048890784, + 0.0017355000018142164, + 0.0015614579897373915, + 0.0015631249989382923, + 0.0015557500009890646, + 0.001566124992677942, + 0.0015930000081425533, + 0.0015675419999752194, + 0.001564582998980768, + 0.001564207996125333, + 0.0016777920100139454, + 0.0015793330094311386, + 0.0015651249996153638, + 0.0015424590092152357, + 0.0015667499974370003, + 0.0015727499994682148, + 0.0015656250034226105, + 0.0015347500011557713, + 0.0015711250016465783, + 0.0016711669886717573, + 0.0015772080078022555, + 0.0015539999876637012, + 0.0015585829969495535, + 0.0015435419918503612, + 0.0015630830021109432, + 0.0015549579984508455, + 0.001552041998365894, + 0.0016031249979278073, + 0.0015324999985750765, + 0.0015519169974140823, + 0.0015610829868819565, + 0.0015638750046491623, + 0.0015469170029973611, + 0.0015543749905191362, + 0.001551457986352034, + 0.0015524160116910934, + 0.0015532079996773973, + 0.0015698750066803768, + 0.0015438750124303624, + 0.001570124994032085, + 0.0015631659916834906, + 0.0015637500036973506, + 0.0016783750033937395, + 0.001560750009957701, + 0.0015718750073574483, + 0.0016445840010419488, + 0.0015751669998280704, + 0.0015912079979898408, + 0.0015641250065527856, + 0.001557708004838787, + 0.0015774580097058788, + 0.0015619999903719872, + 0.0015675829927204177, + 0.0015707080019637942, + 0.0016602499963482842, + 0.0015387500025099143, + 0.0015978340088622645, + 0.0015950829983921722, + 0.0017029160080710426, + 0.0016107080009533092, + 0.0016046670061768964, + 0.0015998339949874207, + 0.0016340420115739107, + 0.00159200000052806, + 0.001701374989352189, + 0.0016173750045709312, + 0.0015995420108083636, + 0.0015954170085024089, + 0.0015899999998509884, + 0.001596000001882203, + 0.0017935409996425733, + 0.0018157080048695207, + 0.0017574169905856252, + 0.0016821250028442591, + 0.0016914169973460957, + 0.0016909999976633117, + 0.0016797080024844036, + 0.0017456250061513856, + 0.0017509590106783435, + 0.0017793750012060627, + 0.0016638750093989074, + 0.0016426249931100756, + 0.0016265839949483052, + 0.001608375008800067, + 0.0016219579993048683, + 0.001759459002641961, + 0.001775208002072759, + 0.0016993750032270327, + 0.001703791000181809, + 0.001690832999884151, + 0.0017713749984977767, + 0.0016568749997531995, + 0.0017333750001853332, + 0.0017178749985760078, + 0.001778082994860597, + 0.0016175000055227429, + 0.0015905829932307824, + 0.0016102919908007607, + 0.001592417000210844, + 0.0015822499990463257, + 0.0015863750013522804, + 0.0015980419993866235, + 0.0015608749963575974, + 0.0015752909966977313, + 0.001723042005323805, + 0.001758083002641797, + 0.0017461669922340661, + 0.001695625003776513, + 0.0016910419944906607, + 0.0016718749975552782, + 0.0017306660010945052, + 0.001684582995949313, + 0.0017329580005025491, + 0.0017857500060927123, + 0.0017855409969342873, + 0.0016967080009635538, + 0.0016895409935386851, + 0.0016871249972609803, + 0.0016971670120256022, + 0.0019006669899681583, + 0.0018062909948639572, + 0.0017849589930847287, + 0.0017835000035120174, + 0.0017505830037407577, + 0.001712915996904485, + 0.001705583999864757, + 0.0016882089985301718, + 0.001746292007737793, + 0.001767958005075343, + 0.001748416994814761, + 0.0017168750055134296, + 0.0017338330071652308, + 0.001767417008522898, + 0.0017230000084964558, + 0.001749666000250727, + 0.00170116699882783, + 0.0018047499906970188, + 0.0016971250006463379, + 0.0017768329998943955, + 0.0016466250090161338, + 0.001710041004116647, + 0.0016865829966263846, + 0.0016637919907225296, + 0.0017517079977551475, + 0.0017690829990897328, + 0.0017693749978207052, + 0.0017693339905235916, + 0.0017496250075055286, + 0.001662499998928979, + 0.0016575000045122579, + 0.0017028339934768155, + 0.0017815830069594085, + 0.0017566659953445196, + 0.0017642919992795214, + 0.001604750010301359, + 0.0015974170091794804, + 0.0017421670054318383, + 0.001800333004212007, + 0.00174937499104999, + 0.001596957998117432, + 0.0018084999901475385, + 0.001872749999165535, + 0.0019432500121183693, + 0.0017069590103346854, + 0.0017119170079240575, + 0.001723207999020815, + 0.0016482500068377703, + 0.0016882909985724837, + 0.0017749160033417866, + 0.0017829999997047707, + 0.0018051249935524538, + 0.0018793750059558079, + 0.0017811659927247092, + 0.0016677090025041252, + 0.001619415997993201, + 0.001651832993957214, + 0.0017555000085849315, + 0.0017968750034924597, + 0.0017816250037867576, + 0.0017795000021578744, + 0.001858625008026138, + 0.001774541990016587, + 0.001767415989888832, + 0.0017077919910661876, + 0.0016834999987622723, + 0.0016497500037075952, + 0.0017578749975655228, + 0.0017778749897843227, + 0.0017727499944157898, + 0.0018408329924568534, + 0.0018632919964147732, + 0.0017751249979482964, + 0.001710458003799431, + 0.0017312909913016483, + 0.0016060420020949095, + 0.0016227919986704364, + 0.001686667004832998 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + false, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_sdpa": { + "first_token_ms": { + "count": 5, + "median": 1.455499994335696, + "p95": 1.4754663978237659, + "min": 1.3849590031895787, + "max": 1.4803329977439716, + "iqr": 0.0694170012138784, + "mean": 1.4326749980682507, + "cv": 0.03069910635719534 + }, + "decode_ms_per_step": { + "count": 1115, + "median": 1.7031250026775524, + "p95": 1.8106081974110566, + "min": 1.3626669970108196, + "max": 4.13187500089407, + "iqr": 0.1174380086013116, + "mean": 1.6961249202331152, + "cv": 0.10098327331866896 + }, + "total_ms": { + "count": 5, + "median": 376.8033370433841, + "p95": 388.31472189340275, + "min": 376.30600393458735, + "max": 391.0124571120832, + "iqr": 0.8266990771517158, + "mean": 379.66853221005294, + "cv": 0.016742736785142695 + }, + "tokens_per_second": { + "count": 5, + "median": 594.4745653200233, + "p95": 595.1366370912659, + "min": 572.8717740974444, + "max": 595.2602341123888, + "iqr": 1.3021436614213826, + "mean": 590.1177855763966, + "cv": 0.01637932302147312 + }, + "median_step_ms_by_index": [ + 1.455499994335696, + 1.4657910069217905, + 1.4367080002557486, + 1.4416659978451207, + 1.4283329946920276, + 1.4615830004913732, + 1.472042000386864, + 1.4456670032814145, + 1.4372080040629953, + 1.4648749929619953, + 1.4543750003213063, + 1.5051249938551337, + 1.5205840027192608, + 1.4890000020386651, + 1.5185829979600385, + 1.4664170012110844, + 1.5287080022972077, + 1.546500003314577, + 1.5479169960599393, + 1.486665991251357, + 1.468792004743591, + 1.52824999531731, + 1.5479169960599393, + 1.5517079882556573, + 1.4976250095060095, + 1.542458005133085, + 1.5462500014109537, + 1.5292500029318035, + 1.527874992461875, + 1.5104999911272898, + 1.5607079985784367, + 1.5310410090023652, + 1.5346250002039596, + 1.5782499976921827, + 1.6350829973816872, + 1.6435419966001064, + 1.6289170016534626, + 1.6330420039594173, + 1.5727499994682148, + 1.64079200476408, + 1.654667008551769, + 1.6453749994980171, + 1.5828329924261197, + 1.5860419953241944, + 1.5728329890407622, + 1.6353329992853105, + 1.6428340022685006, + 1.644207994104363, + 1.580916010425426, + 1.6362089954782277, + 1.6262910066870973, + 1.6500420024385676, + 1.655040992773138, + 1.652833991101943, + 1.6624589916318655, + 1.6215830110013485, + 1.6727910115150735, + 1.6847499937284738, + 1.6930420097196475, + 1.6359170112991706, + 1.6183330008061603, + 1.6119159990921617, + 1.6701670101610944, + 1.6730420029489323, + 1.6608330042799935, + 1.7683339974610135, + 1.7665420018602163, + 1.7269999952986836, + 1.7059160018106923, + 1.713167002890259, + 1.7172919906442985, + 1.7504579882370308, + 1.7794580053305253, + 1.7765829979907721, + 1.7621250008232892, + 1.7774170119082555, + 1.7718750023050234, + 1.7104169965023175, + 1.738499995553866, + 1.7653750110184774, + 1.771291994373314, + 1.7759170004865155, + 1.773916999809444, + 1.7749580001691356, + 1.7707079969113693, + 1.7329169932054356, + 1.7326670058537275, + 1.7718339950079098, + 1.761125007760711, + 1.766291999956593, + 1.7833750025602058, + 1.7754589935066178, + 1.7886249988805503, + 1.7026660061674193, + 1.733958997647278, + 1.7567089962540194, + 1.7705830105114728, + 1.7371249996358529, + 1.7476249922765419, + 1.7654169932939112, + 1.701749992207624, + 1.763415988534689, + 1.755834004143253, + 1.6978329949779436, + 1.7490000027464703, + 1.7486249998910353, + 1.7738750029820949, + 1.7332080024061725, + 1.6851660038810223, + 1.7488340090494603, + 1.6942090005613863, + 1.742874999763444, + 1.7435420013498515, + 1.7568330076755956, + 1.758083002641797, + 1.7389580025337636, + 1.7521249974379316, + 1.73049999284558, + 1.7359589983243495, + 1.7147500038845465, + 1.731832991936244, + 1.7373339942423627, + 1.7672090034466237, + 1.7488330049673095, + 1.746292007737793, + 1.705500006210059, + 1.6895000007934868, + 1.7313749995082617, + 1.7010839947033674, + 1.7410419968655333, + 1.7659590084804222, + 1.7561249987920746, + 1.7646659980528057, + 1.7043750121956691, + 1.6937500040512532, + 1.6910839913180098, + 1.727417009533383, + 1.7654169932939112, + 1.7770830017980188, + 1.7622080049477518, + 1.7256669962080196, + 1.75012499676086, + 1.759417005814612, + 1.7221669986611232, + 1.7522910056868568, + 1.757374993758276, + 1.7695830028969795, + 1.770292001310736, + 1.6811249952297658, + 1.695667000603862, + 1.7150830099126324, + 1.7039589874912053, + 1.7649999936111271, + 1.7887080030050129, + 1.7709170060697943, + 1.6940000059548765, + 1.7208750068675727, + 1.7290410032728687, + 1.7173340020235628, + 1.7237080028280616, + 1.7342089995509014, + 1.7513749917270616, + 1.770416012732312, + 1.76320799801033, + 1.7475000058766454, + 1.7623750027269125, + 1.6806670028017834, + 1.7048340087058023, + 1.6964160022325814, + 1.7549590120324865, + 1.6887499950826168, + 1.756957994075492, + 1.757291000103578, + 1.7080830002669245, + 1.6916660097194836, + 1.720999993267469, + 1.68395800574217, + 1.7340830090688542, + 1.7671670066192746, + 1.7762500065146014, + 1.7655830015428364, + 1.7304170032730326, + 1.702499997918494, + 1.7130409978562966, + 1.684375005424954, + 1.7033749900292605, + 1.7650829977355897, + 1.7048750014510006, + 1.7693749978207052, + 1.7601250001462176, + 1.7528750031488016, + 1.7081250116461888, + 1.7067080043489113, + 1.7463330004829913, + 1.7414159956388175, + 1.6597919893683866, + 1.7303749918937683, + 1.755999997840263, + 1.7550840129842982, + 1.747999995131977, + 1.728417002595961, + 1.6917090106289834, + 1.7587909969734028, + 1.7811670113587752, + 1.7015829944284633, + 1.7295839934377, + 1.7147920007118955, + 1.7002499953377992, + 1.7625830078031868, + 1.7519579996587709, + 1.7322499916190282, + 1.7187920020660385, + 1.7012079915730283, + 1.7477920046076179, + 1.6824159974930808, + 1.6789580113254488, + 1.7457500071031973, + 1.6972500015981495, + 1.6986249975161627, + 1.77012498897966, + 1.7114580114139244, + 1.695457991445437, + 1.7387920088367537, + 1.721332999295555 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0013849590031895787, + 0.0014117499958956614, + 0.0014544169971486554, + 0.0015007500041974708, + 0.0013907919928897172, + 0.0014644590119132772, + 0.001472042000386864, + 0.001476249992265366, + 0.0013963340024929494, + 0.0014648749929619953, + 0.001461584004573524, + 0.0014759580080863088, + 0.0014895420026732609, + 0.001494500000262633, + 0.0015023339947219938, + 0.0014311249979073182, + 0.0014770419948035851, + 0.001546500003314577, + 0.0015401659911731258, + 0.0015378329990198836, + 0.001468792004743591, + 0.0015401250129798427, + 0.0015429159975610673, + 0.0015517079882556573, + 0.0014591670042136684, + 0.0014227089995983988, + 0.001506707994849421, + 0.001556165996589698, + 0.001453749995562248, + 0.0014382090012077242, + 0.0015195409941952676, + 0.001555041002575308, + 0.0015220000059343874, + 0.0015782499976921827, + 0.001633708001463674, + 0.001627666992135346, + 0.0016771660011727363, + 0.001667040996835567, + 0.0015804159920662642, + 0.001595291992998682, + 0.0016756659897509962, + 0.0016378340078517795, + 0.0016356670093955472, + 0.001626707991817966, + 0.0015728329890407622, + 0.0015550000098301098, + 0.0016561250085942447, + 0.0016823749901959673, + 0.0016497909964527935, + 0.0016555000038351864, + 0.0015706249978393316, + 0.001640750007936731, + 0.001655040992773138, + 0.001654665989917703, + 0.0016412080003647134, + 0.0015723329997854307, + 0.0016904169897316024, + 0.0016742910083848983, + 0.001607666999916546, + 0.0016359170112991706, + 0.0015844580047996715, + 0.0015877909900154918, + 0.0016797499993117526, + 0.0016948340053204447, + 0.0017117079987656325, + 0.0017683339974610135, + 0.0019825829949695617, + 0.0018450829957146198, + 0.0017373750015394762, + 0.001662499998928979, + 0.0016825839993543923, + 0.001738250008202158, + 0.0017701250035315752, + 0.001779958009137772, + 0.0017809580022003502, + 0.0018025830067927018, + 0.0018558750016381964, + 0.0017104169965023175, + 0.001655291998758912, + 0.0016829160013003275, + 0.001685333001660183, + 0.0018105000053765252, + 0.001767666995874606, + 0.0017749580001691356, + 0.0017707079969113693, + 0.0017840000073192641, + 0.0018819590040948242, + 0.0017825409886427224, + 0.0017175410030176863, + 0.0017122500139521435, + 0.0016965420072665438, + 0.0017696659924695268, + 0.0017886249988805503, + 0.0017763749929144979, + 0.0017838330095401034, + 0.0018090829980792478, + 0.0017705830105114728, + 0.001715666992822662, + 0.0017077079974114895, + 0.0016799159930087626, + 0.0017567910108482465, + 0.0017711669934215024, + 0.001774499993189238, + 0.001770082992152311, + 0.0018474999960744753, + 0.0017486249998910353, + 0.001699415995972231, + 0.0017044590058503672, + 0.0016851660038810223, + 0.0017567079921718687, + 0.0017770840058801696, + 0.0017852499877335504, + 0.0017784170049708337, + 0.0017737080052029341, + 0.0017402500088792294, + 0.0016980830114334822, + 0.0017071250040316954, + 0.0016860000032465905, + 0.0016696669918019325, + 0.001761542007443495, + 0.0017653749964665622, + 0.001736416990752332, + 0.0017014579934766516, + 0.0018921250011771917, + 0.0017430000007152557, + 0.0016888750105863437, + 0.0018275420006830245, + 0.0017533339996589348, + 0.0017694580019451678, + 0.001783250001608394, + 0.0017659590084804222, + 0.0016851249965839088, + 0.0017712090048007667, + 0.0017660419980529696, + 0.0016841250035213307, + 0.001756959012709558, + 0.001727417009533383, + 0.0017229580116691068, + 0.0018006659956881776, + 0.0017322499916190282, + 0.001701583998510614, + 0.0017023330001393333, + 0.0019062499923165888, + 0.0016821250028442591, + 0.0017521250119898468, + 0.001757374993758276, + 0.0017695830028969795, + 0.0017620409926166758, + 0.0016811249952297658, + 0.0016839159943629056, + 0.0017150830099126324, + 0.0016827910003485158, + 0.0017984169971896335, + 0.0017887080030050129, + 0.0017823749949457124, + 0.0017787079996196553, + 0.0017552500066813082, + 0.0017051249888027087, + 0.0017173340020235628, + 0.0017237080028280616, + 0.0017703750054351985, + 0.0019595420017139986, + 0.0017758330068318173, + 0.0017670000088401139, + 0.0017475000058766454, + 0.0017736250010784715, + 0.0016806670028017834, + 0.0016860409959917888, + 0.0016964160022325814, + 0.0016852090047905222, + 0.0017447080026613548, + 0.0017362920043524355, + 0.001680000001215376, + 0.0016711659991415218, + 0.0016385830094804987, + 0.001720999993267469, + 0.00168395800574217, + 0.0017395409959135577, + 0.0017671670066192746, + 0.0017817909974837676, + 0.0017860419902717695, + 0.0017304170032730326, + 0.0017000829975586385, + 0.0017050000024028122, + 0.001684375005424954, + 0.0017286660004174337, + 0.0017650829977355897, + 0.001646124990656972, + 0.0017709579988149926, + 0.00158095799270086, + 0.0017574579978827387, + 0.0016199579986277968, + 0.0016092080040834844, + 0.0017463330004829913, + 0.0016131249867612496, + 0.0016257500101346523, + 0.001663291986915283, + 0.001670500001637265, + 0.0016567089915042743, + 0.001747999995131977, + 0.001728417002595961, + 0.001599625000380911, + 0.0018031670042546466, + 0.0018382090056547895, + 0.001709332995233126, + 0.0016028340032789856, + 0.0017746249941410497, + 0.0016717499966034666, + 0.0017774159932741895, + 0.0017992089997278526, + 0.0017322499916190282, + 0.0016539159987587482, + 0.0016285830060951412, + 0.0017014589975588024, + 0.0016824159974930808, + 0.0016154580080183223, + 0.0017778339970391244, + 0.0017314160068053752, + 0.0016986249975161627, + 0.0017736250010784715, + 0.0016759580030338839, + 0.0018330419989069924, + 0.0017581249994691461, + 0.0017721670010359958 + ], + [ + 0.0013865829969290644, + 0.0014782500074943528, + 0.001466166999307461, + 0.0014416659978451207, + 0.0014283329946920276, + 0.001434500009054318, + 0.0014604580064769834, + 0.0013727920013479888, + 0.0014316250017145649, + 0.0014559169940184802, + 0.0014543750003213063, + 0.0014833750028628856, + 0.0015205840027192608, + 0.0013993330067023635, + 0.0015026249893708155, + 0.0014816249895375222, + 0.0015287080022972077, + 0.0015578329912386835, + 0.001552249988890253, + 0.0014615410036640242, + 0.0014418750070035458, + 0.0015254580066539347, + 0.0015994169953046367, + 0.0015529590018559247, + 0.001485333006712608, + 0.001542458005133085, + 0.0015462500014109537, + 0.0015539999876637012, + 0.0015422919968841597, + 0.0014737919991603121, + 0.0015152910054894164, + 0.0015685420075897127, + 0.0015362910053227097, + 0.0015915410040179268, + 0.0016350829973816872, + 0.0016435419966001064, + 0.00164787498943042, + 0.0016466669912915677, + 0.0015509999939240515, + 0.0015949999942677096, + 0.001644042000407353, + 0.0016453749994980171, + 0.0015677079936722293, + 0.0015968329971656203, + 0.00156974999117665, + 0.0016353329992853105, + 0.0016428340022685006, + 0.0016566670092288405, + 0.0016387919895350933, + 0.0015945829945849255, + 0.0015825830050744116, + 0.0016500420024385676, + 0.0016597499925410375, + 0.0016935000021476299, + 0.0016624589916318655, + 0.0016139999934239313, + 0.0016083329974208027, + 0.001681542009464465, + 0.0016930420097196475, + 0.001657749991863966, + 0.001671624995651655, + 0.0016181250102818012, + 0.0016376249986933544, + 0.001633374995435588, + 0.0016637919907225296, + 0.0017729999963194132, + 0.0017665420018602163, + 0.0016718749975552782, + 0.0017059160018106923, + 0.001713167002890259, + 0.0016858749877428636, + 0.0017504579882370308, + 0.0017798749904613942, + 0.0018827500025508925, + 0.001782292005373165, + 0.0017774170119082555, + 0.0017718750023050234, + 0.0017266659997403622, + 0.001698334002867341, + 0.001744957990013063, + 0.001771291994373314, + 0.0017545419977977872, + 0.0017831669974839315, + 0.0017903330008266494, + 0.0017682499892544001, + 0.0017072089976863936, + 0.0017326670058537275, + 0.0017085420113289729, + 0.0016852919943630695, + 0.0017483750125393271, + 0.001775249998900108, + 0.0017893750045914203, + 0.0018368330056546256, + 0.0017026660061674193, + 0.0017629170033615083, + 0.0017226659983862191, + 0.0017130839987657964, + 0.0016892499988898635, + 0.0017444999975850806, + 0.00176208300399594, + 0.0016879579925443977, + 0.0017949169996427372, + 0.0017466249992139637, + 0.001660833993810229, + 0.0016910839913180098, + 0.0016808329964987934, + 0.001738499995553866, + 0.0017714999994495884, + 0.0017743749922374263, + 0.0017488340090494603, + 0.0016777909913798794, + 0.0018786250002449378, + 0.001688541000476107, + 0.0016686659946572036, + 0.0017404590034857392, + 0.0017103329882957041, + 0.0017982499994104728, + 0.0017812909936765209, + 0.0017732500127749518, + 0.0016639999957988039, + 0.001675834006164223, + 0.001774874996044673, + 0.001756416997523047, + 0.0017488330049673095, + 0.0016909170080907643, + 0.001685333001660183, + 0.0016889169928617775, + 0.0017313749995082617, + 0.0017010839947033674, + 0.0016678329993737862, + 0.0017329589900327846, + 0.001810667003155686, + 0.0017610420036362484, + 0.0016623750125290826, + 0.0017229170043719932, + 0.0016675410006428137, + 0.00169162500242237, + 0.0016770410002209246, + 0.001738874998409301, + 0.0017645000043557957, + 0.0019190420134691522, + 0.0017714580026222393, + 0.001759417005814612, + 0.0016960420034592971, + 0.0017056249926099554, + 0.0017258750012842938, + 0.0017676660063443705, + 0.001770292001310736, + 0.0017601660074433312, + 0.0018286670092493296, + 0.0018246670078951865, + 0.001764957996783778, + 0.0016870000108610839, + 0.001683624999714084, + 0.001714124999125488, + 0.0016296249959850684, + 0.0016679159889463335, + 0.0015918749995762482, + 0.0017427919956389815, + 0.0017518329987069592, + 0.0017596250108908862, + 0.0017305420042248443, + 0.0017579999985173345, + 0.00176320799801033, + 0.0017713340057525784, + 0.0016719170089345425, + 0.0016738330014050007, + 0.0017399589996784925, + 0.001695040991762653, + 0.001796540993382223, + 0.0016589170118095353, + 0.0017399170028511435, + 0.0017624159954721108, + 0.001772000003256835, + 0.0016957500047283247, + 0.0017113749927375466, + 0.0016292499931296334, + 0.0016739580023568124, + 0.001877167000202462, + 0.001725167006952688, + 0.0017472920008003712, + 0.0016873330023372546, + 0.0017485839925939217, + 0.0016886250086827204, + 0.001651958009460941, + 0.0016355840052710846, + 0.0017315829900326207, + 0.001754083001287654, + 0.001804540996090509, + 0.0017601250001462176, + 0.0016767499910201877, + 0.001697749990853481, + 0.0017067080043489113, + 0.0016771669907029718, + 0.0017414159956388175, + 0.0016597919893683866, + 0.001744416993460618, + 0.001755999997840263, + 0.0017721670010359958, + 0.0016875000001164153, + 0.001701500004855916, + 0.001754290991812013, + 0.0017587909969734028, + 0.0017820000066421926, + 0.001779583006282337, + 0.001756792000378482, + 0.001743583008646965, + 0.0017002499953377992, + 0.0017625830078031868, + 0.0016684590082149953, + 0.001636999993934296, + 0.0017647499917075038, + 0.0017012079915730283, + 0.0017477920046076179, + 0.0016018329915823415, + 0.0016593749896856025, + 0.001686832998530008, + 0.0016972500015981495, + 0.0017985000013140962, + 0.001772916002664715, + 0.0017671670066192746, + 0.0016955409955698997, + 0.0017387920088367537, + 0.0017027499998221174 + ], + [ + 0.001455499994335696, + 0.0014821670047240332, + 0.0013734999956795946, + 0.001434332996723242, + 0.0013812499964842573, + 0.0015098750009201467, + 0.0013826670037815347, + 0.0014456670032814145, + 0.0014913750055711716, + 0.0014992499927757308, + 0.0014138749975245446, + 0.0015107500075828284, + 0.0015015839890111238, + 0.0015134580025915056, + 0.0015185829979600385, + 0.0014499169919872656, + 0.0014984169974923134, + 0.0015650830027880147, + 0.001462665997678414, + 0.001486665991251357, + 0.0015995839930837974, + 0.0015539580053882673, + 0.001556417002575472, + 0.0014989589981269091, + 0.0015625000087311491, + 0.00155237500439398, + 0.001558458010549657, + 0.0015292500029318035, + 0.001527874992461875, + 0.0015440830029547215, + 0.001589708001120016, + 0.0015085409977473319, + 0.0015346250002039596, + 0.0015579169994452968, + 0.0015969590021995828, + 0.0016337919951183721, + 0.0016259169933618978, + 0.0015739579976070672, + 0.0015646250103600323, + 0.0016534580063307658, + 0.0016658749955240637, + 0.0016521249926881865, + 0.001534707989776507, + 0.0015589999966323376, + 0.0016412920085713267, + 0.0016490830021211877, + 0.0016473330033477396, + 0.001644207994104363, + 0.001569791988003999, + 0.0016362089954782277, + 0.0016919589979806915, + 0.0016522090008947998, + 0.0016572500026086345, + 0.0015890409995336086, + 0.0016395839920733124, + 0.0016215830110013485, + 0.0016471660055685788, + 0.0016847499937284738, + 0.0017050000024028122, + 0.0016943329974310473, + 0.0016771249938756227, + 0.001610916995559819, + 0.0016004580102162436, + 0.0016699160041753203, + 0.0016486250096932054, + 0.0017645839980104938, + 0.0016639589885016903, + 0.001692958001513034, + 0.0016996670019580051, + 0.0016854999994393438, + 0.0017491659964434803, + 0.0017730409890646115, + 0.0017794580053305253, + 0.0017679579905234277, + 0.0016864169883774593, + 0.0016874580032890663, + 0.0017201250011567026, + 0.0016828340012580156, + 0.001752374999341555, + 0.0017653750110184774, + 0.0018355409993091598, + 0.0017759170004865155, + 0.001773916999809444, + 0.001715624995995313, + 0.0017257910076295957, + 0.001740541003528051, + 0.0016928340046433732, + 0.0017617499979678541, + 0.0017876250058179721, + 0.0018171670089941472, + 0.0018312500033061951, + 0.0017754589935066178, + 0.0016791250091046095, + 0.0016901670023798943, + 0.0016392910038121045, + 0.0017733749991748482, + 0.0017785829986678436, + 0.0017565830057719722, + 0.0017655830015428364, + 0.0017783749935915694, + 0.0016969999996945262, + 0.001637542009120807, + 0.001755834004143253, + 0.0017693749978207052, + 0.0016794589901110157, + 0.001749666000250727, + 0.0017738750029820949, + 0.0017612499941606075, + 0.00167966699518729, + 0.0017916250071721151, + 0.0016942090005613863, + 0.0016926660027820617, + 0.0017435420013498515, + 0.0017568330076755956, + 0.001758083002641797, + 0.0017389580025337636, + 0.0016659579996485263, + 0.001665125004365109, + 0.0017359589983243495, + 0.0017803329974412918, + 0.0017576249956618994, + 0.0017373339942423627, + 0.0017672090034466237, + 0.0017356669995933771, + 0.0017657909920671955, + 0.001705500006210059, + 0.0016895000007934868, + 0.0016742089937906712, + 0.0018055420077871531, + 0.0017511249898234382, + 0.0017701659962767735, + 0.0017561249987920746, + 0.0017646659980528057, + 0.0016982079978333786, + 0.001666124997427687, + 0.0016694579971954226, + 0.0017263330082641914, + 0.0018112500110873953, + 0.0018113329861080274, + 0.0017622080049477518, + 0.0017523330025142059, + 0.0016739580023568124, + 0.0016752500087022781, + 0.0017321250052191317, + 0.0017522910056868568, + 0.0017425419937353581, + 0.0017570839991094545, + 0.0018260419892612845, + 0.0017450000013923272, + 0.0016619999951217324, + 0.0016754579992266372, + 0.0016942910006036982, + 0.0016618749941699207, + 0.0018024589953711256, + 0.001739417013595812, + 0.0016722920117899776, + 0.0017208750068675727, + 0.0017292080010520294, + 0.0016672089986968786, + 0.0016639170062262565, + 0.0017280839965678751, + 0.0017513749917270616, + 0.001770416012732312, + 0.001742082997225225, + 0.0016654589999234304, + 0.0016625839925836772, + 0.0016879579925443977, + 0.0017048340087058023, + 0.001730874995701015, + 0.0017549590120324865, + 0.0017709579988149926, + 0.0017574589874129742, + 0.001674333994742483, + 0.0016429999959655106, + 0.0016559170035179704, + 0.001725208989228122, + 0.0017968750034924597, + 0.0017509580065961927, + 0.0017548339965287596, + 0.001672375001362525, + 0.0016663749993313104, + 0.001695625003776513, + 0.0016712090000510216, + 0.0017292499978793785, + 0.001744083987432532, + 0.0017588750051800162, + 0.0018048749916488305, + 0.001665707997744903, + 0.0016711249918444082, + 0.001688582997303456, + 0.0017528750031488016, + 0.0017608749913051724, + 0.001775249998900108, + 0.0017058329976862296, + 0.0017897910001920536, + 0.0016626669967081398, + 0.0017030839953804389, + 0.0016821250028442591, + 0.0017550840129842982, + 0.0017569170013302937, + 0.0017625000036787242, + 0.0017641669983277097, + 0.0017587080074008554, + 0.0016622089897282422, + 0.001669332996243611, + 0.0017512079939479008, + 0.0016602919931756333, + 0.0017323749925708398, + 0.0017570409981999546, + 0.001759958002367057, + 0.0017478329973528162, + 0.001651625003432855, + 0.0016797499993117526, + 0.001682874994003214, + 0.0016775419935584068, + 0.001795249991118908, + 0.0017457500071031973, + 0.001666833006311208, + 0.0016452919953735545, + 0.001695958009804599, + 0.0016995000041788444, + 0.001665707997744903, + 0.0017246669885935262, + 0.0017564580048201606 + ], + [ + 0.0014559999981429428, + 0.0014657910069217905, + 0.0013626669970108196, + 0.0013763750030193478, + 0.00146975000097882, + 0.0014615830004913732, + 0.0014744999934919178, + 0.0013802089961245656, + 0.0014372080040629953, + 0.0014686659997096285, + 0.0014734579890500754, + 0.0015217499894788489, + 0.0015216659958241507, + 0.0014890000020386651, + 0.0015215830062516034, + 0.0014664170012110844, + 0.001538875003461726, + 0.0015281670057447627, + 0.0015479169960599393, + 0.0014794169983360916, + 0.0014361659996211529, + 0.00152824999531731, + 0.0015479169960599393, + 0.0014321670023491606, + 0.0014976250095060095, + 0.0015248749987222254, + 0.0015473750099772587, + 0.001461542007746175, + 0.0014139169943518937, + 0.0015104999911272898, + 0.0015607079985784367, + 0.0015310410090023652, + 0.0015238750056596473, + 0.0015601250051986426, + 0.0016723330045351759, + 0.0016483340004924685, + 0.0016289170016534626, + 0.0016330420039594173, + 0.0015727499994682148, + 0.0016987499984679744, + 0.001654667008551769, + 0.0016781669983174652, + 0.0016440829931525514, + 0.0015860419953241944, + 0.0014887919969623908, + 0.0016300829884130508, + 0.0015834590012673289, + 0.0016214579954976216, + 0.0015718750073574483, + 0.0015074999973876402, + 0.0016262910066870973, + 0.0016666669980622828, + 0.0016355419938918203, + 0.0015316249919123948, + 0.0016730420029489323, + 0.0017161249998025596, + 0.0016727910115150735, + 0.0016909999976633117, + 0.0016934580053202808, + 0.0015946670027915388, + 0.0016183330008061603, + 0.0016294170054607093, + 0.0016701670101610944, + 0.0016936249885475263, + 0.0016608330042799935, + 0.0017827919946284965, + 0.0017293749988311902, + 0.0017269999952986836, + 0.0017031250026775524, + 0.001757125006406568, + 0.0017172919906442985, + 0.001774916992872022, + 0.0017709590028971434, + 0.0017765829979907721, + 0.0017621250008232892, + 0.001703541012830101, + 0.0016925840027397498, + 0.0016867920057848096, + 0.001738499995553866, + 0.0017841250082710758, + 0.0017686249921098351, + 0.0017553750076331198, + 0.0016850419924594462, + 0.001727417009533383, + 0.0017877500067697838, + 0.0016873749991646037, + 0.0017771249986253679, + 0.0017723340133670717, + 0.001761125007760711, + 0.001766291999956593, + 0.0017833750025602058, + 0.0017362500075250864, + 0.0016994170000543818, + 0.001782874998752959, + 0.001688582997303456, + 0.0017567089962540194, + 0.0017791669961297885, + 0.0017708750092424452, + 0.0017740839975886047, + 0.0017654169932939112, + 0.001701749992207624, + 0.0017090000037569553, + 0.0017754999862518162, + 0.0016978329949779436, + 0.0017490000027464703, + 0.0017125839949585497, + 0.0017778340115910396, + 0.001654709005379118, + 0.0016805839986773208, + 0.001657542001339607, + 0.0016675829974701628, + 0.001742874999763444, + 0.0018230830028187484, + 0.0017774579901015386, + 0.0017789999983506277, + 0.0017693330009933561, + 0.0017521249974379316, + 0.00173049999284558, + 0.001708791998680681, + 0.0016894169966690242, + 0.001720000000204891, + 0.0018147079972550273, + 0.0017815830069594085, + 0.001764375003403984, + 0.0017669579974608496, + 0.0017081249970942736, + 0.0016646669973852113, + 0.0016736249963287264, + 0.0016543749952688813, + 0.0017410419968655333, + 0.0017732080013956875, + 0.001816333009628579, + 0.0017534590006107464, + 0.0017043750121956691, + 0.0016937500040512532, + 0.0016910839913180098, + 0.001747583010001108, + 0.0017654169932939112, + 0.001773166994098574, + 0.0017684580088825896, + 0.0017256669962080196, + 0.00175012499676086, + 0.0017090839974116534, + 0.0017221669986611232, + 0.0017697909934213385, + 0.001764207991072908, + 0.0017773330037016422, + 0.0016918750043259934, + 0.0016717079997761175, + 0.0017097910022130236, + 0.0017502499977126718, + 0.0017039589874912053, + 0.0017649999936111271, + 0.001757167003233917, + 0.0017709170060697943, + 0.0016940000059548765, + 0.0017989580082939938, + 0.0017290410032728687, + 0.0017098340031225234, + 0.0018281250086147338, + 0.0017342089995509014, + 0.0016794580005807802, + 0.001753332995576784, + 0.0016731249925214797, + 0.001670458004809916, + 0.0017623750027269125, + 0.0016644580027787015, + 0.001673332997597754, + 0.001676375002716668, + 0.0017693749978207052, + 0.0016570409934502095, + 0.001756957994075492, + 0.001757291000103578, + 0.0017080830002669245, + 0.0016916660097194836, + 0.0017517090018372983, + 0.0016820829914649948, + 0.0016917500033741817, + 0.0018061670125462115, + 0.0018772500043269247, + 0.0017655830015428364, + 0.0017637919954722747, + 0.001702499997918494, + 0.0017130409978562966, + 0.001675249994150363, + 0.0016891670093173161, + 0.0017658329888945445, + 0.001767042005667463, + 0.0017693749978207052, + 0.0017744580109138042, + 0.0017122919962275773, + 0.0017081250116461888, + 0.00172433300758712, + 0.0017772080027498305, + 0.0017809999990276992, + 0.001778250007191673, + 0.0017797499895095825, + 0.0017740000039339066, + 0.0017506250005681068, + 0.0016938750050030649, + 0.0016971250006463379, + 0.0016917090106289834, + 0.0018057499983115122, + 0.0016779999859863892, + 0.0016462500061606988, + 0.0017295839934377, + 0.001684165996266529, + 0.0016785409970907494, + 0.0017865420086309314, + 0.0016767079941928387, + 0.0017433749890187755, + 0.0017631670052651316, + 0.0017707500082906336, + 0.00177179099409841, + 0.0016897079913178459, + 0.0016789580113254488, + 0.0017045419954229146, + 0.0016970410069916397, + 0.0017431250016670674, + 0.00177012498897966, + 0.0017657499993219972, + 0.0016546669939998537, + 0.0016696670063538477, + 0.0016295419918606058 + ], + [ + 0.0014803329977439716, + 0.001389209006447345, + 0.0014367080002557486, + 0.0014731669944012538, + 0.001512792005087249, + 0.0013940420030849054, + 0.0014968750037951395, + 0.0014700000028824434, + 0.0014488330052699894, + 0.001433209006791003, + 0.0013982920063426718, + 0.0015051249938551337, + 0.001534875002107583, + 0.0014115420053713024, + 0.0016012499982025474, + 0.0015424590092152357, + 0.0015359170065494254, + 0.0015099160082172602, + 0.0015730409941170365, + 0.0015758340014144778, + 0.001556000002892688, + 0.0015175419976003468, + 0.0015210419951472431, + 0.0015695000038249418, + 0.0015666669933125377, + 0.0015813750069355592, + 0.0014928330056136474, + 0.0014648750075139105, + 0.0015466250042663887, + 0.001568249994306825, + 0.001564792008139193, + 0.001512999995611608, + 0.0015688749990658835, + 0.001585584002896212, + 0.0016451669944217429, + 0.0016542079974897206, + 0.001571667002281174, + 0.0015707910060882568, + 0.0016399159940192476, + 0.00164079200476408, + 0.0016444590000901371, + 0.0015944999904604629, + 0.0015828329924261197, + 0.0015775409992784262, + 0.0016512920119566843, + 0.0016527499974472448, + 0.0016381250025006011, + 0.0015786250005476177, + 0.001580916010425426, + 0.001653166997130029, + 0.0016788750072009861, + 0.0015387500025099143, + 0.0015692499873694032, + 0.001652833991101943, + 0.0016861670010257512, + 0.0016793329996289685, + 0.0016962910012807697, + 0.0017103339923778549, + 0.0016257910028798506, + 0.0016348750068573281, + 0.0016177910001715645, + 0.0016119159990921617, + 0.0017073330091079697, + 0.0016730420029489323, + 0.0015852919896133244, + 0.0017087080050259829, + 0.0017730830004438758, + 0.0017709580133669078, + 0.0017834169993875548, + 0.0017670410015853122, + 0.0017773330037016422, + 0.0015981660108081996, + 0.001786458000424318, + 0.0017345420055789873, + 0.0017126669990830123, + 0.0018162080086767673, + 0.00413187500089407, + 0.003532291011651978, + 0.0035237499978393316, + 0.003754624995053746, + 0.001846333994762972, + 0.003753708995645866, + 0.0019173749897163361, + 0.001826874999096617, + 0.0018955000123241916, + 0.0017329169932054356, + 0.0017058749945135787, + 0.0017718339950079098, + 0.0018475419929018244, + 0.0018124999915016815, + 0.0018019999988609925, + 0.0017824590031523257, + 0.0018130000098608434, + 0.0016923340008361265, + 0.001733958997647278, + 0.001752915995894, + 0.0017273330013267696, + 0.0017371249996358529, + 0.0017476249922765419, + 0.0017754169966792688, + 0.0017949169996427372, + 0.001763415988534689, + 0.0017233340040547773, + 0.0016919159970711917, + 0.001750833005644381, + 0.0017871669988380745, + 0.0018462909938534722, + 0.0017332080024061725, + 0.0018517919961595908, + 0.0017004170076688752, + 0.001788957990356721, + 0.0016772090020822361, + 0.0016715410019969568, + 0.0017422499950043857, + 0.0017662500031292439, + 0.00176320799801033, + 0.001766832996509038, + 0.001798083001631312, + 0.0018105829949490726, + 0.0017147500038845465, + 0.001731832991936244, + 0.001703250003629364, + 0.0017710829997668043, + 0.0016825409984448925, + 0.001746292007737793, + 0.0017779159970814362, + 0.0017732499982230365, + 0.0018680840003071353, + 0.001700041990261525, + 0.0017198750138049945, + 0.001710124997771345, + 0.0017210419900948182, + 0.0018553330010036007, + 0.0017869579896796495, + 0.0017920420068548992, + 0.001801291000447236, + 0.0018539159937063232, + 0.0018036250112345442, + 0.0017770830017980188, + 0.0017179999995278195, + 0.001698916996247135, + 0.001755249992129393, + 0.0017806249961722642, + 0.0017760830087354407, + 0.0017823750094976276, + 0.0018137919978471473, + 0.001837500007241033, + 0.0017723329947330058, + 0.0016319580026902258, + 0.001695667000603862, + 0.001691249999566935, + 0.0017094170034397393, + 0.0017832079902291298, + 0.0017952920024981722, + 0.0017783330113161355, + 0.0018273750029038638, + 0.0017090420005843043, + 0.001738874998409301, + 0.0017231250094482675, + 0.0017014999903040007, + 0.0017000409861793742, + 0.0017721670010359958, + 0.0017981669952860102, + 0.0018029580096481368, + 0.0018334999913349748, + 0.0018349169986322522, + 0.0017798330081859604, + 0.0017193749954458326, + 0.0017415830079698935, + 0.0016623749979771674, + 0.0016887499950826168, + 0.0018123339978046715, + 0.0017709590028971434, + 0.001716750004561618, + 0.0017751249979482964, + 0.0017121249984484166, + 0.0016899999900488183, + 0.0017340830090688542, + 0.0017061669932445511, + 0.0017762500065146014, + 0.0017970420012716204, + 0.0017675410053925589, + 0.0017952500056708232, + 0.0018056670087389648, + 0.0017833330057328567, + 0.0017033749900292605, + 0.0017209590005222708, + 0.0017048750014510006, + 0.0017627500055823475, + 0.001781707993359305, + 0.0017784580122679472, + 0.001786041000741534, + 0.0016959170025074854, + 0.001786875000107102, + 0.0017073330091079697, + 0.0016498750046594068, + 0.0017303749918937683, + 0.0017959169927053154, + 0.0017820000066421926, + 0.001783250001608394, + 0.0017671669920673594, + 0.0016737090045353398, + 0.0017085409926949069, + 0.0017811670113587752, + 0.0017015829944284633, + 0.0016818330041132867, + 0.0017147920007118955, + 0.0017602500010980293, + 0.0017609170026844367, + 0.0017519579996587709, + 0.0017179999995278195, + 0.0017187920020660385, + 0.0017459999944549054, + 0.0018121250031981617, + 0.0017643340106587857, + 0.0017764590011211112, + 0.001786041000741534, + 0.0017618749989196658, + 0.0016878750029718503, + 0.0016823329933686182, + 0.0017114580114139244, + 0.001695457991445437, + 0.0018132500117644668, + 0.001721332999295555 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "uncached_mha": { + "first_token_ms": { + "count": 5, + "median": 1.5796249936101958, + "p95": 1.6336418018909171, + "min": 1.4967079914640635, + "max": 1.6420000029029325, + "iqr": 0.038416997995227575, + "mean": 1.5760667971335351, + "cv": 0.03393826996407106 + }, + "decode_ms_per_step": { + "count": 1115, + "median": 1.8248330015921965, + "p95": 1.9317463025799952, + "min": 1.413749996572733, + "max": 2.2288749896688387, + "iqr": 0.14268750237533823, + "mean": 1.8005055687317182, + "cv": 0.06372358624968207 + }, + "total_ms": { + "count": 5, + "median": 402.24779889103957, + "p95": 405.33377790707164, + "min": 401.2835500470828, + "max": 405.40557709755376, + "iqr": 3.586045204428956, + "mean": 403.0888086243067, + "cv": 0.004933356958079721 + }, + "tokens_per_second": { + "count": 5, + "median": 556.8706668316087, + "p95": 558.1595599139544, + "min": 552.5330993315326, + "max": 558.208777742616, + "iqr": 4.9398748609232825, + "mean": 555.71960924869, + "cv": 0.004926310322008339 + }, + "median_step_ms_by_index": [ + 1.5796249936101958, + 1.5742090035928413, + 1.556916002300568, + 1.5570830000797287, + 1.5588329988531768, + 1.553582987980917, + 1.568958003190346, + 1.5142920019570738, + 1.5235839964589104, + 1.5597500023432076, + 1.5480839938391, + 1.553666006657295, + 1.5481669979635626, + 1.6017919988371432, + 1.6081250068964437, + 1.6197920049307868, + 1.6060000052675605, + 1.5622090140823275, + 1.5720419905846938, + 1.637374996789731, + 1.649917001486756, + 1.6153750038938597, + 1.5818749961908907, + 1.594290995853953, + 1.6062079957919195, + 1.643666997551918, + 1.6840410098666325, + 1.6154580080183223, + 1.647333992877975, + 1.662334005231969, + 1.6565409896429628, + 1.6372079990105703, + 1.6746670007705688, + 1.7161249998025596, + 1.6745409957366064, + 1.6635000065434724, + 1.716999991913326, + 1.7346249951515347, + 1.7311249976046383, + 1.7394589958712459, + 1.73891699523665, + 1.665416988544166, + 1.6742919979151338, + 1.66079199698288, + 1.7250420060008764, + 1.7187909979838878, + 1.7561249987920746, + 1.748957991367206, + 1.655250001931563, + 1.6786249907454476, + 1.7209590005222708, + 1.740584004437551, + 1.7402920057065785, + 1.7705829959595576, + 1.7825830000219867, + 1.7262080073123798, + 1.742791006108746, + 1.7649590008659288, + 1.7454999906476587, + 1.7612920055398718, + 1.7992920038523152, + 1.82395800948143, + 1.8359160021645948, + 1.8149170064134523, + 1.7981250130105764, + 1.8592920096125454, + 1.865749989519827, + 1.832041991292499, + 1.820333010982722, + 1.852416986366734, + 1.8250830034958199, + 1.85162499838043, + 1.8706669943640009, + 1.9015830039279535, + 1.8864169978769496, + 1.8907079938799143, + 1.9096250034635887, + 1.9086250104010105, + 1.9088749977527186, + 1.8807090091286227, + 1.906292003695853, + 1.9324580061947927, + 1.9382080063223839, + 1.9290829950477928, + 1.9174170010956004, + 1.919417001772672, + 1.9079589983448386, + 1.9045000080950558, + 1.8903750024037436, + 1.8792909977491945, + 1.864540987298824, + 1.9175829947926104, + 1.8959159933729097, + 1.8551670073065907, + 1.8789589958032593, + 1.8571250111563131, + 1.8177910096710548, + 1.846874991315417, + 1.8787919980240986, + 1.8202500068582594, + 1.8737499922281131, + 1.8885830068029463, + 1.8532090034568682, + 1.8636669992702082, + 1.8631670100148767, + 1.8180420011049137, + 1.859084004536271, + 1.8624170043040067, + 1.8855829985113814, + 1.8522499885875732, + 1.885332996607758, + 1.8747079884633422, + 1.8290839943801984, + 1.8292080058017746, + 1.8604159995447844, + 1.8560419994173571, + 1.8526250059949234, + 1.8183749925810844, + 1.884750003227964, + 1.865917001850903, + 1.8928339995909482, + 1.8743750115390867, + 1.8871669890359044, + 1.8524580082157627, + 1.8761249957606196, + 1.8645419913809747, + 1.8864589947042987, + 1.8734170007519424, + 1.876750000519678, + 1.817541997297667, + 1.8629999976838008, + 1.88508300925605, + 1.8910829967353493, + 1.8187499954365194, + 1.865124999312684, + 1.8585409998195246, + 1.8775409989757463, + 1.8750000017462298, + 1.8005830061156303, + 1.863208002760075, + 1.8890420033130795, + 1.864791993284598, + 1.851291992352344, + 1.8091660022037104, + 1.862040997366421, + 1.8773749907268211, + 1.8559169984655455, + 1.800915997591801, + 1.812375005101785, + 1.8833749927580357, + 1.8242920050397515, + 1.8721250089583918, + 1.8874999950639904, + 1.8777499935822561, + 1.8563339981483296, + 1.8598750029923394, + 1.819000011892058, + 1.81387500197161, + 1.8749580049188808, + 1.8097499996656552, + 1.8264170066686347, + 1.800333004212007, + 1.8680000066524372, + 1.8884169985540211, + 1.8712920136749744, + 1.8905000033555552, + 1.9009160023415461, + 1.876209003967233, + 1.8917089910246432, + 1.8947920034406707, + 1.883958000689745, + 1.8019580020336434, + 1.8684999959077686, + 1.8742499960353598, + 1.8105830095009878, + 1.8210410053143278, + 1.8117500003427267, + 1.8342080002184957, + 1.833290996728465, + 1.8905840115621686, + 1.8718329956755042, + 1.883790988358669, + 1.8862080032704398, + 1.866874998086132, + 1.8932920065708458, + 1.89183400652837, + 1.8795000069076195, + 1.8634580046636984, + 1.8107080104527995, + 1.7947080050362274, + 1.8212499999208376, + 1.811417008866556, + 1.829917004215531, + 1.8026659963652492, + 1.8581250042188913, + 1.8639170011738315, + 1.8177919992012903, + 1.8277079943800345, + 1.8917080014944077, + 1.8859999981941655, + 1.8818749958882108, + 1.7910000024130568, + 1.8500830046832561, + 1.8030409992206842, + 1.8241670040879399, + 1.7972079949686304, + 1.8539159937063232, + 1.786292006727308, + 1.8712500022957101, + 1.848375002737157, + 1.8372080085100606, + 1.8742079992080107, + 1.8889999919338152, + 1.8385830044280738, + 1.9136669870931655, + 1.8528340006014332, + 1.882083000964485, + 1.825958999688737, + 1.869290994363837, + 1.780791993951425, + 1.8087920034304261, + 1.8202499923063442, + 1.8248749984195456, + 1.8587500089779496 + ], + "refreshed_steps": [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + "raw": { + "unit": "seconds", + "step_seconds": [ + [ + 0.0016002089978428558, + 0.0016106249968288466, + 0.001487082990934141, + 0.0015570830000797287, + 0.0015838750114198774, + 0.001652375009143725, + 0.0015873340016696602, + 0.0015025829925434664, + 0.0015500419976888224, + 0.0015496249980060384, + 0.0015719169896328822, + 0.0014637090062024072, + 0.0014735419972566888, + 0.0016017919988371432, + 0.0016081250068964437, + 0.0016484999941894785, + 0.0016388340009143576, + 0.001531583009636961, + 0.0016332079976564273, + 0.001637374996789731, + 0.0015697089984314516, + 0.0016153750038938597, + 0.0015940420125843957, + 0.0015468749916180968, + 0.0016062079957919195, + 0.001643666997551918, + 0.0017001249943859875, + 0.001701791989034973, + 0.0016869579994818196, + 0.001677374995779246, + 0.0017848749994300306, + 0.0016784170002210885, + 0.0016595840133959427, + 0.001761708001140505, + 0.0017189589998451993, + 0.001683459006017074, + 0.0016700419946573675, + 0.0016981670050881803, + 0.0017555420054122806, + 0.001789917005226016, + 0.0017624159954721108, + 0.001665416988544166, + 0.0016429170063929632, + 0.0016344579926226288, + 0.0016487910033902153, + 0.0017124160076491535, + 0.0017762079951353371, + 0.001795999996829778, + 0.0017934159986907616, + 0.001803084000130184, + 0.0016783330065663904, + 0.0016307090118061751, + 0.0016575000045122579, + 0.001744749999488704, + 0.0017825830000219867, + 0.001801666003302671, + 0.001840750002884306, + 0.0018637920002220199, + 0.0018345830030739307, + 0.0016966660041362047, + 0.0017970420012716204, + 0.0017648750072112307, + 0.001715792008326389, + 0.001730292002321221, + 0.0017939999961527064, + 0.0019238749955547974, + 0.001983374997507781, + 0.0019003339984919876, + 0.001735208003083244, + 0.0022288749896688387, + 0.0019902909989468753, + 0.0018926250049844384, + 0.0019299999985378236, + 0.001917125002364628, + 0.001761500010616146, + 0.0018269159918418154, + 0.0019115829927613959, + 0.0019016250007553026, + 0.0017870829906314611, + 0.0018702919915085658, + 0.0018608750106068328, + 0.0018475000106263906, + 0.0019118750060442835, + 0.0019290829950477928, + 0.0018449589988449588, + 0.0018591249972814694, + 0.001827458996558562, + 0.0017819159984355792, + 0.0018903750024037436, + 0.0019353340030647814, + 0.0019368329958524555, + 0.0019393330003367737, + 0.001986999996006489, + 0.0018162500055041164, + 0.0019064170046476647, + 0.0018266249971929938, + 0.0018743749969871715, + 0.0018769160087686032, + 0.001764582993928343, + 0.0018013750086538494, + 0.001902832998894155, + 0.00202470799558796, + 0.001814250004827045, + 0.0018282079981872812, + 0.0018174160068156198, + 0.0018180420011049137, + 0.0017975419905269518, + 0.0018624170043040067, + 0.0018834579968824983, + 0.0018355830106884241, + 0.001900584000395611, + 0.001886874990304932, + 0.0019005829963134602, + 0.0019000839965883642, + 0.0018715000041993335, + 0.0018776670040097088, + 0.0018680829962249845, + 0.0017739159957272932, + 0.0019871249969583005, + 0.0018433329969411716, + 0.0018055420077871531, + 0.0018221669888589531, + 0.001826625011744909, + 0.0018524580082157627, + 0.0018014160013990477, + 0.0018596669979160652, + 0.0019075000018347055, + 0.002007582996156998, + 0.0018787500011967495, + 0.0019017500017071143, + 0.0018882080039475113, + 0.00188508300925605, + 0.0019184999982826412, + 0.0018760840030154213, + 0.001887624996015802, + 0.0018617500027175993, + 0.0019408750085858628, + 0.0018750000017462298, + 0.0018005830061156303, + 0.0018059580033877864, + 0.0018277500057592988, + 0.0018157500016968697, + 0.0018350419995840639, + 0.0018030830105999485, + 0.0017924580024555326, + 0.0018704590038396418, + 0.0018559169984655455, + 0.0017812080041039735, + 0.001957542001036927, + 0.002001583998207934, + 0.0018964159971801564, + 0.0019000000029336661, + 0.0019229170138714835, + 0.0018777499935822561, + 0.001909458005684428, + 0.001844375001383014, + 0.001819000011892058, + 0.0018314170010853559, + 0.0018227499967906624, + 0.001839999997173436, + 0.0018264170066686347, + 0.001800333004212007, + 0.001859084004536271, + 0.0019132079905830324, + 0.0018854999943869188, + 0.0018905000033555552, + 0.0018927499913843349, + 0.00183104099414777, + 0.0019050830014748499, + 0.001895624998724088, + 0.0017952909984160215, + 0.0017610829963814467, + 0.0018233329901704565, + 0.0018096249987138435, + 0.0018105830095009878, + 0.0018210410053143278, + 0.0017978749965550378, + 0.0018692919984459877, + 0.0018801660044118762, + 0.00189200000022538, + 0.0018751250026980415, + 0.0019143749959766865, + 0.0018865419988287613, + 0.001894333996460773, + 0.0018932920065708458, + 0.00189183400652837, + 0.0018795000069076195, + 0.001822250007535331, + 0.0017699170130072162, + 0.0017947080050362274, + 0.0018446250032866374, + 0.0018127090006601065, + 0.001829917004215531, + 0.0017945830040844157, + 0.0018638750043464825, + 0.001890874991659075, + 0.0017987920000450686, + 0.0018827079911716282, + 0.0018917080014944077, + 0.0019214579951949418, + 0.0018270000000484288, + 0.0017545000009704381, + 0.0018500830046832561, + 0.0018197919998783618, + 0.0018241670040879399, + 0.001787624991266057, + 0.0017688749940134585, + 0.001782499995897524, + 0.001956166001036763, + 0.0018912500090664253, + 0.0019014580029761419, + 0.0018742079992080107, + 0.0018857499962905422, + 0.0018385830044280738, + 0.001929916994413361, + 0.0018891670042648911, + 0.001901875002658926, + 0.0018992909899679944, + 0.001869290994363837, + 0.0018720829975791276, + 0.0017873330070869997, + 0.001792542010662146, + 0.0018248749984195456, + 0.0018235000025015324 + ], + [ + 0.0015617919998476282, + 0.0015742090035928413, + 0.001585374993737787, + 0.001560417003929615, + 0.0015093749971129, + 0.001553582987980917, + 0.001596957998117432, + 0.001604042001417838, + 0.0014770000125281513, + 0.001457374994060956, + 0.0015450420032721013, + 0.0015102500037755817, + 0.0015541669999947771, + 0.001546125000459142, + 0.0016081249923445284, + 0.0016197920049307868, + 0.0017767079989425838, + 0.001680000001215376, + 0.0015510420053033158, + 0.0016033749998314306, + 0.001654874999076128, + 0.0016499999910593033, + 0.0015818749961908907, + 0.0016523749945918098, + 0.0015790000034030527, + 0.001560083997901529, + 0.0015166659868555143, + 0.001506332991993986, + 0.0014915420033503324, + 0.0014868330035824329, + 0.0014870410086587071, + 0.0015058330027386546, + 0.0015002500003902242, + 0.0015563749911962077, + 0.0015318750083679333, + 0.0015709580038674176, + 0.0017382080113748088, + 0.0018776249926304445, + 0.0016984590038191527, + 0.001715291989967227, + 0.0017571669886820018, + 0.001649541998631321, + 0.0017375000024912879, + 0.0017358330078423023, + 0.0017286249931203201, + 0.0017187909979838878, + 0.0016372920072171837, + 0.001774292002664879, + 0.0016431249969173223, + 0.0017328329995507374, + 0.0017510420002508909, + 0.0017739579925546423, + 0.0017229579971171916, + 0.0018027499900199473, + 0.0017180840077344328, + 0.0017262080073123798, + 0.001742791006108746, + 0.0017967089952435344, + 0.0017454999906476587, + 0.0018229169945698231, + 0.0018453749944455922, + 0.0018416249949950725, + 0.0018397499952698126, + 0.0018297909991815686, + 0.0018434160010656342, + 0.0018114589911419898, + 0.001865749989519827, + 0.001832041991292499, + 0.001820333010982722, + 0.001852416986366734, + 0.0018038749985862523, + 0.0018724590045167133, + 0.0018929580110125244, + 0.0017851660086307675, + 0.001924750002217479, + 0.0019752910011447966, + 0.0019192500039935112, + 0.0019086250104010105, + 0.0019273749931016937, + 0.0019006669899681583, + 0.0018720420048339292, + 0.001954082996235229, + 0.0019382080063223839, + 0.002081083002849482, + 0.0019674169889185578, + 0.001942208007676527, + 0.0019340830040164292, + 0.0019100419885944575, + 0.0018165000074077398, + 0.0019321250001667067, + 0.0018206249951617792, + 0.0018261669902130961, + 0.0018959159933729097, + 0.0018923339957837015, + 0.0018789589958032593, + 0.0018734170007519424, + 0.0018473340023774654, + 0.0018258749914821237, + 0.0018787919980240986, + 0.001889958992251195, + 0.0018945000047096983, + 0.0019264159927843139, + 0.0017167090118164197, + 0.0017271669930778444, + 0.0017433749890187755, + 0.00175183400278911, + 0.0017951670015463606, + 0.0018263749952893704, + 0.0018582079937914386, + 0.0018522499885875732, + 0.001841542005422525, + 0.001789917005226016, + 0.0018290839943801984, + 0.0019003329944098368, + 0.001965667004697025, + 0.0017852919991128147, + 0.0018526250059949234, + 0.00193425000179559, + 0.0018993749981746078, + 0.001865917001850903, + 0.0019147920102113858, + 0.0018467920017428696, + 0.0018758750084089115, + 0.0018221250065835193, + 0.0018122499895980582, + 0.0018286669946974143, + 0.001834041002439335, + 0.0018549579981481656, + 0.001876750000519678, + 0.0018099580047419295, + 0.0018629999976838008, + 0.0018928339995909482, + 0.00191124998673331, + 0.0017697909934213385, + 0.001865124999312684, + 0.001906457997392863, + 0.0018887079932028428, + 0.0018909999926108867, + 0.0018852500070352107, + 0.001863208002760075, + 0.0017370830028085038, + 0.001789665999240242, + 0.0017682920006336644, + 0.0018091660022037104, + 0.0018252500012749806, + 0.0018938340072054416, + 0.0018659579945961013, + 0.001959041997906752, + 0.0017646249907556921, + 0.0018792920018313453, + 0.001908584003103897, + 0.0018157090089516714, + 0.001930832993821241, + 0.0018886250036302954, + 0.0018120829918188974, + 0.0018598750029923394, + 0.001807750013540499, + 0.0018138750019716099, + 0.0018390410114079714, + 0.0018071249942295253, + 0.0018429999909130856, + 0.0017997079994529486, + 0.0018680000066524372, + 0.0018897499976446852, + 0.0018257920019095764, + 0.0019037500023841858, + 0.0018900830036727712, + 0.001901709008961916, + 0.0018915419932454824, + 0.0018947920034406707, + 0.001883958000689745, + 0.0018894160020863637, + 0.0018736250058282167, + 0.0019151249871356413, + 0.001793542003724724, + 0.0018043340096483007, + 0.0018204590014647692, + 0.0018270000000484288, + 0.001833290996728465, + 0.001793917006580159, + 0.0018494999967515469, + 0.001775417011231184, + 0.001921666000271216, + 0.0019089999987045303, + 0.001817875003325753, + 0.0019420000026002526, + 0.0018724590045167133, + 0.0018777499935822561, + 0.0018940000009024516, + 0.0017455410124966875, + 0.0017187090124934912, + 0.001811417008866556, + 0.0018801669939421117, + 0.0017820000066421926, + 0.0017969580076169223, + 0.0018616669985931367, + 0.0017944169958354905, + 0.0016665409930283204, + 0.0019566249975468963, + 0.00191199999244418, + 0.0018966250063385814, + 0.001811666996218264, + 0.001824749997467734, + 0.0017947079904843122, + 0.0018161250045523047, + 0.0017837079940363765, + 0.001848957996116951, + 0.0018052090017590672, + 0.0018712500022957101, + 0.0018343749979976565, + 0.0018372080085100606, + 0.0018382089911028743, + 0.0019734589877771214, + 0.0018904169992310926, + 0.0019136669870931655, + 0.0018739579973043874, + 0.001895624998724088, + 0.0018680420034797862, + 0.0018797499942593277, + 0.0017705000063870102, + 0.0018087920034304261, + 0.0018202499923063442, + 0.001822874997742474, + 0.0017709159874357283 + ], + [ + 0.0015796249936101958, + 0.001619250004296191, + 0.0014866669953335077, + 0.0015602909988956526, + 0.0015588329988531768, + 0.0015570409887004644, + 0.00153179100016132, + 0.0014486660074908286, + 0.0015235839964589104, + 0.0015597500023432076, + 0.0015383749996544793, + 0.001572042005136609, + 0.0015904169995337725, + 0.0016015829896787181, + 0.0016090840072138235, + 0.0016082909860415384, + 0.0015706669946666807, + 0.0015622090140823275, + 0.0015287919959519058, + 0.0016507079999428242, + 0.001649917001486756, + 0.0015927499916870147, + 0.001538124997750856, + 0.0016096670005936176, + 0.0015817090024938807, + 0.0016359170112991706, + 0.001574458001414314, + 0.0016154580080183223, + 0.001647333992877975, + 0.001662334005231969, + 0.0016565409896429628, + 0.001634500004001893, + 0.0017613749951124191, + 0.00174170799436979, + 0.0017430000007152557, + 0.001740375009831041, + 0.001716999991913326, + 0.0017271250108024105, + 0.0017310830007772893, + 0.0017126250022556633, + 0.001654334002523683, + 0.001666458003455773, + 0.0016742919979151338, + 0.001731459007714875, + 0.0017286249931203201, + 0.0017194999963976443, + 0.0017217909917235374, + 0.0016362500027753413, + 0.001655250001931563, + 0.0016720830026315525, + 0.0017209590005222708, + 0.0017497500084573403, + 0.0018038330017589033, + 0.0017873749893624336, + 0.0017578749975655228, + 0.0016873340064194053, + 0.0016859579918673262, + 0.0017429999861633405, + 0.001711040997179225, + 0.0017612920055398718, + 0.0017992920038523152, + 0.0019728329934878275, + 0.0018521669990150258, + 0.0018099580047419295, + 0.0017981250130105764, + 0.0017951250047190115, + 0.0017871669988380745, + 0.001828374995966442, + 0.001825290994020179, + 0.0018360840040259063, + 0.0018235409952467307, + 0.0018515419942559674, + 0.0018706669943640009, + 0.0019015830039279535, + 0.0018917499983217567, + 0.0018907079938799143, + 0.0018945829942822456, + 0.0019062079954892397, + 0.0019084170053247362, + 0.0020022089884150773, + 0.001906292003695853, + 0.0019234159990446642, + 0.0019245000003138557, + 0.0019027500093216076, + 0.0018953329999931157, + 0.0018976249994011596, + 0.001881083007901907, + 0.0018160830077249557, + 0.0018817920063156635, + 0.0017423749959561974, + 0.0017284999921685085, + 0.0018142079934477806, + 0.0017150000057881698, + 0.0017293749988311902, + 0.0017230829980690032, + 0.0017139169940492138, + 0.0017063330014934763, + 0.001846874991315417, + 0.0018941669986816123, + 0.0018202500068582594, + 0.0018660000059753656, + 0.0017929169989656657, + 0.0018532090034568682, + 0.0018922919989563525, + 0.0018631670100148767, + 0.001883042001281865, + 0.0019113750022370368, + 0.0019217909866711125, + 0.0018855829985113814, + 0.00185591600893531, + 0.001885332996607758, + 0.0018747079884633422, + 0.0018054159882012755, + 0.001801332997274585, + 0.0017965419974643737, + 0.0018560419994173571, + 0.0018177500023739412, + 0.0017778329929569736, + 0.0018597080052131787, + 0.001883792006992735, + 0.0018928339995909482, + 0.0018774170021060854, + 0.0018871669890359044, + 0.0018868750048568472, + 0.0019377499993424863, + 0.001880750001873821, + 0.0018864589947042987, + 0.0018734170007519424, + 0.001801583988708444, + 0.0017419159994460642, + 0.001855375012382865, + 0.0017690410022623837, + 0.0018237080075778067, + 0.0018117500003427267, + 0.001811583002563566, + 0.0017826249968493357, + 0.0018534580012783408, + 0.0018739590013865381, + 0.001782499995897524, + 0.0018777919904096052, + 0.0018890420033130795, + 0.0018892499938374385, + 0.0019225420110160485, + 0.0018842080025933683, + 0.0018727909919107333, + 0.0018773749907268211, + 0.0017986669990932569, + 0.0017802079964894801, + 0.001812375005101785, + 0.0018084159964928403, + 0.001786665990948677, + 0.0018721250089583918, + 0.0018776250071823597, + 0.0017132920038420707, + 0.0016867919912328944, + 0.0018579580064397305, + 0.0017913329938892275, + 0.00179479100916069, + 0.0018962089961860329, + 0.001851958004408516, + 0.0017785420059226453, + 0.001757167003233917, + 0.001752374999341555, + 0.0018517499993322417, + 0.0017732499982230365, + 0.0018742910033324733, + 0.0019149580039083958, + 0.0019019999890588224, + 0.0019057919998886064, + 0.0019022909982595593, + 0.001917500005220063, + 0.0017541670094942674, + 0.0017573330114828423, + 0.0017913750052684918, + 0.0017884580011013895, + 0.0018687909905565903, + 0.0018117500003427267, + 0.0018342080002184957, + 0.001745917004882358, + 0.0018966249917866662, + 0.0018718329956755042, + 0.001883790988358669, + 0.0018862080032704398, + 0.0018668749980861321, + 0.0019993339956272393, + 0.0018881669966503978, + 0.001884041994344443, + 0.0018634580046636984, + 0.0018107080104527995, + 0.0017848330026026815, + 0.0018191669951193035, + 0.0018094169936375692, + 0.0017754589935066178, + 0.0018895420071203262, + 0.0018581250042188913, + 0.0018639170011738315, + 0.0018177919992012903, + 0.0018277079943800345, + 0.0018795409996528178, + 0.0018859999981941655, + 0.0018818749958882108, + 0.0017910000024130568, + 0.0019077500037383288, + 0.0017895409982884303, + 0.00179237499833107, + 0.0018068750068778172, + 0.0018539159937063232, + 0.001786292006727308, + 0.001883459000964649, + 0.0018844589940272272, + 0.0018960409943247214, + 0.0018822910060407594, + 0.0018889999919338152, + 0.001876708003692329, + 0.0019712909997906536, + 0.0018528340006014332, + 0.0018820830009644851, + 0.001818791002733633, + 0.0018784999992931262, + 0.001780791993951425, + 0.0018114160047844052, + 0.0018315420020371675, + 0.0018266249971929938, + 0.0019775830005528405 + ], + [ + 0.0014967079914640635, + 0.001548541011288762, + 0.0015734170010546222, + 0.0014399589999811724, + 0.0014743339997949079, + 0.001413749996572733, + 0.0014515840011881664, + 0.0015142920019570738, + 0.001485375003539957, + 0.0015620829944964498, + 0.0015480839938391, + 0.001589500010595657, + 0.0015481669979635626, + 0.0016035420121625066, + 0.0016037500026868656, + 0.001604000004590489, + 0.0016060000052675605, + 0.0015872920048423111, + 0.0015720419905846938, + 0.0016285419987980276, + 0.0016321669972967356, + 0.0016590419982094318, + 0.0015625829983036965, + 0.0015507079951930791, + 0.0016353329992853105, + 0.00171070899523329, + 0.0017390419961884618, + 0.0016379169974243268, + 0.001656207998166792, + 0.0015981249889591709, + 0.0015430000057676807, + 0.0016372079990105703, + 0.0016785829939180985, + 0.0017161249998025596, + 0.0016745409957366064, + 0.0016635000065434724, + 0.0016764580068411306, + 0.0017587079928489402, + 0.0017311249976046383, + 0.0017394589958712459, + 0.0017218330031028017, + 0.0016491669957758859, + 0.001656375010497868, + 0.0016522500081919134, + 0.0017048750014510006, + 0.0017100420081987977, + 0.0018117090075975284, + 0.001748957991367206, + 0.0016524590027984232, + 0.0016765000036684796, + 0.0016675000078976154, + 0.001740584004437551, + 0.0017559589905431494, + 0.0017705829959595576, + 0.0018122920009773225, + 0.001774916992872022, + 0.0017463749973103404, + 0.0017649590008659288, + 0.0017397079936927184, + 0.0017863749962998554, + 0.0017260419990634546, + 0.0017909169982885942, + 0.0018092499958584085, + 0.0018149170064134523, + 0.0017690410022623837, + 0.0018592920096125454, + 0.0018779999954858795, + 0.0018033329979516566, + 0.0018092499958584085, + 0.0018350000027567148, + 0.0018343749979976565, + 0.0017904159903991967, + 0.0017840829968918115, + 0.001920791997690685, + 0.0018864169978769496, + 0.0019041659979848191, + 0.001888041995698586, + 0.0019097920012427494, + 0.0019088749977527186, + 0.0018807090091286227, + 0.0019117080082651228, + 0.00201349999406375, + 0.001955459010787308, + 0.0019253749924246222, + 0.0019174170010956004, + 0.001919417001772672, + 0.0019079589983448386, + 0.0019045000080950558, + 0.0019114159949822351, + 0.0018172080017393455, + 0.001864540987298824, + 0.0019175829947926104, + 0.0018511669914005324, + 0.0018551670073065907, + 0.0018532920075813308, + 0.0018571250111563131, + 0.0018177910096710548, + 0.001861334007116966, + 0.001883666991489008, + 0.0019315840036142617, + 0.0018737499922281131, + 0.0018885830068029463, + 0.001894874993013218, + 0.0018636669992702082, + 0.001896000001579523, + 0.0017887910071294755, + 0.001859084004536271, + 0.0017918749945238233, + 0.0019564170070225373, + 0.001810874993680045, + 0.0018249160057166591, + 0.0017778329929569736, + 0.0017787079996196553, + 0.0018280829972354695, + 0.0018604159995447844, + 0.00189216599392239, + 0.0018960830057039857, + 0.002064125001197681, + 0.001884750003227964, + 0.0018732080061454326, + 0.001896333007607609, + 0.0018943750037578866, + 0.0018967500072903931, + 0.0018149590032408014, + 0.0018761249957606196, + 0.0018645419913809747, + 0.0018427920003887266, + 0.0017986669990932569, + 0.0017933339986484498, + 0.001817541997297667, + 0.0018560409953352064, + 0.0018127920047845691, + 0.0018910829967353493, + 0.001886666999780573, + 0.0019001250038854778, + 0.0018585409998195246, + 0.0018775409989757463, + 0.0018899999995483086, + 0.0017586250032763928, + 0.0018430830095894635, + 0.0018924579926533625, + 0.001864791993284598, + 0.001851291992352344, + 0.0018242080113850534, + 0.001862040997366421, + 0.0017591250070836395, + 0.0017630410002311692, + 0.001800915997591801, + 0.0018477499979780987, + 0.0018833749927580357, + 0.0018242920050397515, + 0.001925166987348348, + 0.0018874999950639904, + 0.001920750000863336, + 0.001888375001726672, + 0.0018902090087067336, + 0.0018732090102275833, + 0.0017381250072503462, + 0.0018749580049188808, + 0.0018009170016739517, + 0.001824791994295083, + 0.0018303749966435134, + 0.001906124991364777, + 0.0018092079990310594, + 0.0018712920136749744, + 0.0018916670087492093, + 0.0019009160023415461, + 0.0018252080044476315, + 0.0018917089910246432, + 0.0018879169947467744, + 0.0018939580040751025, + 0.0019190410093870014, + 0.0018869999912567437, + 0.0018784999992931262, + 0.0018804999999701977, + 0.0017966250015888363, + 0.001797707998775877, + 0.00182583300920669, + 0.0018035420071100816, + 0.0018157089943997562, + 0.0018492500093998387, + 0.0018597090092953295, + 0.0018813749920809641, + 0.0018393749924143776, + 0.0019882920023519546, + 0.0018975410057464615, + 0.0018939580040751025, + 0.0018951249949168414, + 0.0018999589956365526, + 0.0019052080024266616, + 0.0018212499999208376, + 0.001795459000277333, + 0.0018085420015268028, + 0.0018026659963652492, + 0.0018350419995840639, + 0.0018338340014452115, + 0.0018395420047454536, + 0.0017977499956032261, + 0.0018958330038003623, + 0.0018817499949363992, + 0.0018867919861804694, + 0.001785500004189089, + 0.0018650420097401366, + 0.0018952920072479174, + 0.001905208991956897, + 0.0018909999926108867, + 0.0018923329917015508, + 0.0017145829915534705, + 0.0016856250003911555, + 0.001730124989990145, + 0.0017498329980298877, + 0.001717542007099837, + 0.0017398339987266809, + 0.0017022079991875216, + 0.0016798750002635643, + 0.0016805830091470852, + 0.0016924160008784384, + 0.001697166997473687, + 0.0016825000056996942, + 0.0016747920017223805, + 0.0016903750074561685, + 0.0016997500060824677, + 0.0017375000024912879, + 0.0019071249989792705 + ], + [ + 0.0016420000029029325, + 0.0015485420008189976, + 0.001556916002300568, + 0.001417749997926876, + 0.0016714999946998432, + 0.0014789999986533076, + 0.001568958003190346, + 0.001611208004760556, + 0.0015923340106382966, + 0.0015744160045869648, + 0.0017843329987954348, + 0.001553666006657295, + 0.0015315409982576966, + 0.0016035000007832423, + 0.0016368750075343996, + 0.0016500000056112185, + 0.0015499169967370108, + 0.0015204160008579493, + 0.0016375830018660054, + 0.001645708005526103, + 0.001761125007760711, + 0.0016080409986898303, + 0.0015826669987291098, + 0.001594290995853953, + 0.0016362080059479922, + 0.0016618329973425716, + 0.0016840410098666325, + 0.0015626249951310456, + 0.0015917919954517856, + 0.0016877080051926896, + 0.001665125004365109, + 0.0016671250050421804, + 0.0016746670007705688, + 0.0016689170006429777, + 0.0016632089973427355, + 0.0016592920001130551, + 0.0017173750093206763, + 0.0017346249951515347, + 0.0017546249873703346, + 0.0017455409979447722, + 0.0017389169952366501, + 0.001678333996096626, + 0.001690334000159055, + 0.00166079199698288, + 0.0017250420060008764, + 0.0017397920018993318, + 0.0017561249987920746, + 0.0016772080125520006, + 0.001704750000499189, + 0.0016786249907454476, + 0.00180283299414441, + 0.0016808750078780577, + 0.0017402920057065785, + 0.0017661659949226305, + 0.0017975840019062161, + 0.0016855830035638064, + 0.001677500011282973, + 0.0016713750082999468, + 0.0017827919946284965, + 0.0017499999958090484, + 0.001826791005441919, + 0.00182395800948143, + 0.0018359160021645948, + 0.001825917002861388, + 0.0018731249874690548, + 0.0019099999917671084, + 0.0018025840108748525, + 0.0018694589962251484, + 0.0018248330015921965, + 0.00188433300354518, + 0.0018250830034958199, + 0.00185162499838043, + 0.0018293750035809353, + 0.0018193750001955777, + 0.0018141250038752332, + 0.0018826250015990809, + 0.0019096250034635887, + 0.00192041699483525, + 0.0019452080014161766, + 0.0018642909999471158, + 0.0019202499970560893, + 0.0019324580061947927, + 0.0019464999932097271, + 0.002007999995839782, + 0.0019309159979457036, + 0.0019329170027049258, + 0.0019349170033819973, + 0.001958499997272156, + 0.0019494999869493768, + 0.0018792909977491945, + 0.0019701660057762638, + 0.0019258329994045198, + 0.0019374160037841648, + 0.0019301659922348335, + 0.001920957991387695, + 0.001887000005808659, + 0.0018107499927282333, + 0.0018316250061616302, + 0.0018094170081894845, + 0.001817500000470318, + 0.0018088340002577752, + 0.0018617500027175993, + 0.0018789169989759102, + 0.0018684170063352212, + 0.0020205830078339204, + 0.0018709170108195394, + 0.0019044999935431406, + 0.001878082999610342, + 0.0018905409961007535, + 0.0019052080024266616, + 0.0019002499902853742, + 0.0018863750010496005, + 0.0018849580083042383, + 0.0018292080058017746, + 0.0017642080056248233, + 0.0017950830078916624, + 0.001826124993385747, + 0.0018183749925810844, + 0.0018769169982988387, + 0.0017932080081664026, + 0.0018539999873610213, + 0.0018743750115390867, + 0.001955333005753346, + 0.0018901660077972338, + 0.0018885000026784837, + 0.001905541997984983, + 0.0018947500066133216, + 0.001900374991237186, + 0.0018932920065708458, + 0.0018647090037120506, + 0.0018829999899026006, + 0.001941292008268647, + 0.001784209001925774, + 0.0018187499954365194, + 0.001790124995750375, + 0.0018066249904222786, + 0.0018311669991817325, + 0.0018283329991390929, + 0.001860749995103106, + 0.0018878750124713406, + 0.001957500004209578, + 0.0018690420110942796, + 0.0018774579948512837, + 0.0017883330001495779, + 0.0018920830043498427, + 0.0018919159920187667, + 0.0018905829929281026, + 0.0018332079926040024, + 0.0017913329938892275, + 0.0019014169956790283, + 0.0017945830040844157, + 0.001818457996705547, + 0.0018252919981023297, + 0.0017792919970816001, + 0.0018563339981483296, + 0.0019002079934580252, + 0.0018907500052591786, + 0.00185162499838043, + 0.0019090839923592284, + 0.0018097499996656552, + 0.0018940410081995651, + 0.0019081669888691977, + 0.0019050830014748499, + 0.0018884169985540211, + 0.0018865419988287613, + 0.0018019169947365299, + 0.0019509999983711168, + 0.001876209003967233, + 0.001821500001824461, + 0.0018159589963033795, + 0.0018290830048499629, + 0.0018019580020336434, + 0.0018684999959077686, + 0.0018742499960353598, + 0.0018905829929281026, + 0.001956542007974349, + 0.001896416986710392, + 0.001891583000542596, + 0.001867583006969653, + 0.0018905840115621686, + 0.0018930410005850717, + 0.0018881660071201622, + 0.0018056249973597005, + 0.0017875420016935095, + 0.0017440840019844472, + 0.001813334005419165, + 0.0017914170020958409, + 0.0017986669990932569, + 0.001806291998946108, + 0.0017957079980988055, + 0.0018840830016415566, + 0.0018752920004772022, + 0.0018885000026784837, + 0.001995416998397559, + 0.0019125410035485402, + 0.001896583999041468, + 0.0019013749988516793, + 0.0018855829985113814, + 0.0018741669919108972, + 0.001859042007708922, + 0.0017958749958779663, + 0.0017948750028153881, + 0.001839834003476426, + 0.0018030409992206842, + 0.001838833006331697, + 0.0017972079949686304, + 0.0018642499926500022, + 0.0018777920049615204, + 0.0018212909926660359, + 0.001848375002737157, + 0.001826457999413833, + 0.0018950410012621433, + 0.0019156250054948032, + 0.0018061250011669472, + 0.0018412500066915527, + 0.0018342499970458448, + 0.0017939999961527064, + 0.001825958999688737, + 0.0017947079904843122, + 0.00179387500975281, + 0.001911792001919821, + 0.0019032919954042882, + 0.0018921669980045408, + 0.0018587500089779496 + ] + ], + "refreshed_steps_per_repeat": [ + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ], + [ + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true, + true + ] + ] + } + }, + "speedup_cached_vs_uncached_sdpa": 1.000346558354816, + "speedup_cached_vs_uncached_mha": 1.0678971274081943 + }, + "memory": [ + { + "batch": 1, + "prompt_len": 128, + "steps": [ + { + "step": 0, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 1, + "cached_positions": 129, + "cache_bytes": 396288, + "step_peak_minus_start_bytes": 404224 + }, + { + "step": 2, + "cached_positions": 130, + "cache_bytes": 399360, + "step_peak_minus_start_bytes": 406272 + }, + { + "step": 3, + "cached_positions": 131, + "cache_bytes": 402432, + "step_peak_minus_start_bytes": 410368 + }, + { + "step": 4, + "cached_positions": 132, + "cache_bytes": 405504, + "step_peak_minus_start_bytes": 412416 + }, + { + "step": 5, + "cached_positions": 133, + "cache_bytes": 408576, + "step_peak_minus_start_bytes": 416512 + }, + { + "step": 6, + "cached_positions": 134, + "cache_bytes": 411648, + "step_peak_minus_start_bytes": 418560 + }, + { + "step": 7, + "cached_positions": 135, + "cache_bytes": 414720, + "step_peak_minus_start_bytes": 422656 + }, + { + "step": 8, + "cached_positions": 136, + "cache_bytes": 417792, + "step_peak_minus_start_bytes": 424704 + }, + { + "step": 9, + "cached_positions": 137, + "cache_bytes": 420864, + "step_peak_minus_start_bytes": 428800 + }, + { + "step": 10, + "cached_positions": 138, + "cache_bytes": 423936, + "step_peak_minus_start_bytes": 430848 + }, + { + "step": 11, + "cached_positions": 139, + "cache_bytes": 427008, + "step_peak_minus_start_bytes": 434944 + }, + { + "step": 12, + "cached_positions": 140, + "cache_bytes": 430080, + "step_peak_minus_start_bytes": 436992 + }, + { + "step": 13, + "cached_positions": 141, + "cache_bytes": 433152, + "step_peak_minus_start_bytes": 441088 + }, + { + "step": 14, + "cached_positions": 142, + "cache_bytes": 436224, + "step_peak_minus_start_bytes": 443136 + }, + { + "step": 15, + "cached_positions": 143, + "cache_bytes": 439296, + "step_peak_minus_start_bytes": 447232 + }, + { + "step": 16, + "cached_positions": 144, + "cache_bytes": 442368, + "step_peak_minus_start_bytes": 449280 + }, + { + "step": 17, + "cached_positions": 145, + "cache_bytes": 445440, + "step_peak_minus_start_bytes": 453376 + }, + { + "step": 18, + "cached_positions": 146, + "cache_bytes": 448512, + "step_peak_minus_start_bytes": 455424 + }, + { + "step": 19, + "cached_positions": 147, + "cache_bytes": 451584, + "step_peak_minus_start_bytes": 459520 + }, + { + "step": 20, + "cached_positions": 148, + "cache_bytes": 454656, + "step_peak_minus_start_bytes": 461568 + }, + { + "step": 21, + "cached_positions": 149, + "cache_bytes": 457728, + "step_peak_minus_start_bytes": 465664 + }, + { + "step": 22, + "cached_positions": 150, + "cache_bytes": 460800, + "step_peak_minus_start_bytes": 467712 + }, + { + "step": 23, + "cached_positions": 151, + "cache_bytes": 463872, + "step_peak_minus_start_bytes": 471808 + }, + { + "step": 24, + "cached_positions": 152, + "cache_bytes": 466944, + "step_peak_minus_start_bytes": 473856 + }, + { + "step": 25, + "cached_positions": 153, + "cache_bytes": 470016, + "step_peak_minus_start_bytes": 477952 + }, + { + "step": 26, + "cached_positions": 154, + "cache_bytes": 473088, + "step_peak_minus_start_bytes": 480000 + }, + { + "step": 27, + "cached_positions": 155, + "cache_bytes": 476160, + "step_peak_minus_start_bytes": 484096 + }, + { + "step": 28, + "cached_positions": 156, + "cache_bytes": 479232, + "step_peak_minus_start_bytes": 486144 + }, + { + "step": 29, + "cached_positions": 157, + "cache_bytes": 482304, + "step_peak_minus_start_bytes": 490240 + }, + { + "step": 30, + "cached_positions": 158, + "cache_bytes": 485376, + "step_peak_minus_start_bytes": 492288 + }, + { + "step": 31, + "cached_positions": 159, + "cache_bytes": 488448, + "step_peak_minus_start_bytes": 496384 + }, + { + "step": 32, + "cached_positions": 160, + "cache_bytes": 491520, + "step_peak_minus_start_bytes": 498432 + }, + { + "step": 33, + "cached_positions": 161, + "cache_bytes": 494592, + "step_peak_minus_start_bytes": 502528 + }, + { + "step": 34, + "cached_positions": 162, + "cache_bytes": 497664, + "step_peak_minus_start_bytes": 504576 + }, + { + "step": 35, + "cached_positions": 163, + "cache_bytes": 500736, + "step_peak_minus_start_bytes": 508672 + }, + { + "step": 36, + "cached_positions": 164, + "cache_bytes": 503808, + "step_peak_minus_start_bytes": 510720 + }, + { + "step": 37, + "cached_positions": 165, + "cache_bytes": 506880, + "step_peak_minus_start_bytes": 514816 + }, + { + "step": 38, + "cached_positions": 166, + "cache_bytes": 509952, + "step_peak_minus_start_bytes": 516864 + }, + { + "step": 39, + "cached_positions": 167, + "cache_bytes": 513024, + "step_peak_minus_start_bytes": 520960 + }, + { + "step": 40, + "cached_positions": 168, + "cache_bytes": 516096, + "step_peak_minus_start_bytes": 523008 + }, + { + "step": 41, + "cached_positions": 169, + "cache_bytes": 519168, + "step_peak_minus_start_bytes": 527104 + }, + { + "step": 42, + "cached_positions": 170, + "cache_bytes": 522240, + "step_peak_minus_start_bytes": 529152 + }, + { + "step": 43, + "cached_positions": 171, + "cache_bytes": 525312, + "step_peak_minus_start_bytes": 533248 + }, + { + "step": 44, + "cached_positions": 172, + "cache_bytes": 528384, + "step_peak_minus_start_bytes": 535296 + }, + { + "step": 45, + "cached_positions": 173, + "cache_bytes": 531456, + "step_peak_minus_start_bytes": 539392 + }, + { + "step": 46, + "cached_positions": 174, + "cache_bytes": 534528, + "step_peak_minus_start_bytes": 541440 + }, + { + "step": 47, + "cached_positions": 175, + "cache_bytes": 537600, + "step_peak_minus_start_bytes": 545536 + }, + { + "step": 48, + "cached_positions": 176, + "cache_bytes": 540672, + "step_peak_minus_start_bytes": 547584 + }, + { + "step": 49, + "cached_positions": 177, + "cache_bytes": 543744, + "step_peak_minus_start_bytes": 551680 + }, + { + "step": 50, + "cached_positions": 178, + "cache_bytes": 546816, + "step_peak_minus_start_bytes": 553728 + }, + { + "step": 51, + "cached_positions": 179, + "cache_bytes": 549888, + "step_peak_minus_start_bytes": 557824 + }, + { + "step": 52, + "cached_positions": 180, + "cache_bytes": 552960, + "step_peak_minus_start_bytes": 559872 + }, + { + "step": 53, + "cached_positions": 181, + "cache_bytes": 556032, + "step_peak_minus_start_bytes": 563968 + }, + { + "step": 54, + "cached_positions": 182, + "cache_bytes": 559104, + "step_peak_minus_start_bytes": 566016 + }, + { + "step": 55, + "cached_positions": 183, + "cache_bytes": 562176, + "step_peak_minus_start_bytes": 570112 + }, + { + "step": 56, + "cached_positions": 184, + "cache_bytes": 565248, + "step_peak_minus_start_bytes": 572160 + }, + { + "step": 57, + "cached_positions": 185, + "cache_bytes": 568320, + "step_peak_minus_start_bytes": 576256 + }, + { + "step": 58, + "cached_positions": 186, + "cache_bytes": 571392, + "step_peak_minus_start_bytes": 578304 + }, + { + "step": 59, + "cached_positions": 187, + "cache_bytes": 574464, + "step_peak_minus_start_bytes": 582400 + }, + { + "step": 60, + "cached_positions": 188, + "cache_bytes": 577536, + "step_peak_minus_start_bytes": 584448 + }, + { + "step": 61, + "cached_positions": 189, + "cache_bytes": 580608, + "step_peak_minus_start_bytes": 588544 + }, + { + "step": 62, + "cached_positions": 190, + "cache_bytes": 583680, + "step_peak_minus_start_bytes": 590592 + }, + { + "step": 63, + "cached_positions": 191, + "cache_bytes": 586752, + "step_peak_minus_start_bytes": 594688 + } + ] + }, + { + "batch": 8, + "prompt_len": 128, + "steps": [ + { + "step": 0, + "cached_positions": 128, + "cache_bytes": 3145728, + "step_peak_minus_start_bytes": 14189568 + }, + { + "step": 1, + "cached_positions": 129, + "cache_bytes": 3170304, + "step_peak_minus_start_bytes": 3219712 + }, + { + "step": 2, + "cached_positions": 130, + "cache_bytes": 3194880, + "step_peak_minus_start_bytes": 3244288 + }, + { + "step": 3, + "cached_positions": 131, + "cache_bytes": 3219456, + "step_peak_minus_start_bytes": 3268864 + }, + { + "step": 4, + "cached_positions": 132, + "cache_bytes": 3244032, + "step_peak_minus_start_bytes": 3293440 + }, + { + "step": 5, + "cached_positions": 133, + "cache_bytes": 3268608, + "step_peak_minus_start_bytes": 3318016 + }, + { + "step": 6, + "cached_positions": 134, + "cache_bytes": 3293184, + "step_peak_minus_start_bytes": 3342592 + }, + { + "step": 7, + "cached_positions": 135, + "cache_bytes": 3317760, + "step_peak_minus_start_bytes": 3367168 + }, + { + "step": 8, + "cached_positions": 136, + "cache_bytes": 3342336, + "step_peak_minus_start_bytes": 3391744 + }, + { + "step": 9, + "cached_positions": 137, + "cache_bytes": 3366912, + "step_peak_minus_start_bytes": 3416320 + }, + { + "step": 10, + "cached_positions": 138, + "cache_bytes": 3391488, + "step_peak_minus_start_bytes": 3440896 + }, + { + "step": 11, + "cached_positions": 139, + "cache_bytes": 3416064, + "step_peak_minus_start_bytes": 3465472 + }, + { + "step": 12, + "cached_positions": 140, + "cache_bytes": 3440640, + "step_peak_minus_start_bytes": 3490048 + }, + { + "step": 13, + "cached_positions": 141, + "cache_bytes": 3465216, + "step_peak_minus_start_bytes": 3514624 + }, + { + "step": 14, + "cached_positions": 142, + "cache_bytes": 3489792, + "step_peak_minus_start_bytes": 3539200 + }, + { + "step": 15, + "cached_positions": 143, + "cache_bytes": 3514368, + "step_peak_minus_start_bytes": 3563776 + }, + { + "step": 16, + "cached_positions": 144, + "cache_bytes": 3538944, + "step_peak_minus_start_bytes": 3588352 + }, + { + "step": 17, + "cached_positions": 145, + "cache_bytes": 3563520, + "step_peak_minus_start_bytes": 3612928 + }, + { + "step": 18, + "cached_positions": 146, + "cache_bytes": 3588096, + "step_peak_minus_start_bytes": 3637504 + }, + { + "step": 19, + "cached_positions": 147, + "cache_bytes": 3612672, + "step_peak_minus_start_bytes": 3662080 + }, + { + "step": 20, + "cached_positions": 148, + "cache_bytes": 3637248, + "step_peak_minus_start_bytes": 3686656 + }, + { + "step": 21, + "cached_positions": 149, + "cache_bytes": 3661824, + "step_peak_minus_start_bytes": 3711232 + }, + { + "step": 22, + "cached_positions": 150, + "cache_bytes": 3686400, + "step_peak_minus_start_bytes": 3735808 + }, + { + "step": 23, + "cached_positions": 151, + "cache_bytes": 3710976, + "step_peak_minus_start_bytes": 3760384 + }, + { + "step": 24, + "cached_positions": 152, + "cache_bytes": 3735552, + "step_peak_minus_start_bytes": 3784960 + }, + { + "step": 25, + "cached_positions": 153, + "cache_bytes": 3760128, + "step_peak_minus_start_bytes": 3809536 + }, + { + "step": 26, + "cached_positions": 154, + "cache_bytes": 3784704, + "step_peak_minus_start_bytes": 3834112 + }, + { + "step": 27, + "cached_positions": 155, + "cache_bytes": 3809280, + "step_peak_minus_start_bytes": 3858688 + }, + { + "step": 28, + "cached_positions": 156, + "cache_bytes": 3833856, + "step_peak_minus_start_bytes": 3883264 + }, + { + "step": 29, + "cached_positions": 157, + "cache_bytes": 3858432, + "step_peak_minus_start_bytes": 3907840 + }, + { + "step": 30, + "cached_positions": 158, + "cache_bytes": 3883008, + "step_peak_minus_start_bytes": 3932416 + }, + { + "step": 31, + "cached_positions": 159, + "cache_bytes": 3907584, + "step_peak_minus_start_bytes": 3956992 + }, + { + "step": 32, + "cached_positions": 160, + "cache_bytes": 3932160, + "step_peak_minus_start_bytes": 3981568 + }, + { + "step": 33, + "cached_positions": 161, + "cache_bytes": 3956736, + "step_peak_minus_start_bytes": 4006144 + }, + { + "step": 34, + "cached_positions": 162, + "cache_bytes": 3981312, + "step_peak_minus_start_bytes": 4030720 + }, + { + "step": 35, + "cached_positions": 163, + "cache_bytes": 4005888, + "step_peak_minus_start_bytes": 4055296 + }, + { + "step": 36, + "cached_positions": 164, + "cache_bytes": 4030464, + "step_peak_minus_start_bytes": 4079872 + }, + { + "step": 37, + "cached_positions": 165, + "cache_bytes": 4055040, + "step_peak_minus_start_bytes": 4104448 + }, + { + "step": 38, + "cached_positions": 166, + "cache_bytes": 4079616, + "step_peak_minus_start_bytes": 4129024 + }, + { + "step": 39, + "cached_positions": 167, + "cache_bytes": 4104192, + "step_peak_minus_start_bytes": 4153600 + }, + { + "step": 40, + "cached_positions": 168, + "cache_bytes": 4128768, + "step_peak_minus_start_bytes": 4178176 + }, + { + "step": 41, + "cached_positions": 169, + "cache_bytes": 4153344, + "step_peak_minus_start_bytes": 4202752 + }, + { + "step": 42, + "cached_positions": 170, + "cache_bytes": 4177920, + "step_peak_minus_start_bytes": 4227328 + }, + { + "step": 43, + "cached_positions": 171, + "cache_bytes": 4202496, + "step_peak_minus_start_bytes": 4251904 + }, + { + "step": 44, + "cached_positions": 172, + "cache_bytes": 4227072, + "step_peak_minus_start_bytes": 4276480 + }, + { + "step": 45, + "cached_positions": 173, + "cache_bytes": 4251648, + "step_peak_minus_start_bytes": 4301056 + }, + { + "step": 46, + "cached_positions": 174, + "cache_bytes": 4276224, + "step_peak_minus_start_bytes": 4325632 + }, + { + "step": 47, + "cached_positions": 175, + "cache_bytes": 4300800, + "step_peak_minus_start_bytes": 4350208 + }, + { + "step": 48, + "cached_positions": 176, + "cache_bytes": 4325376, + "step_peak_minus_start_bytes": 4374784 + }, + { + "step": 49, + "cached_positions": 177, + "cache_bytes": 4349952, + "step_peak_minus_start_bytes": 4399360 + }, + { + "step": 50, + "cached_positions": 178, + "cache_bytes": 4374528, + "step_peak_minus_start_bytes": 4423936 + }, + { + "step": 51, + "cached_positions": 179, + "cache_bytes": 4399104, + "step_peak_minus_start_bytes": 4448512 + }, + { + "step": 52, + "cached_positions": 180, + "cache_bytes": 4423680, + "step_peak_minus_start_bytes": 4473088 + }, + { + "step": 53, + "cached_positions": 181, + "cache_bytes": 4448256, + "step_peak_minus_start_bytes": 4497664 + }, + { + "step": 54, + "cached_positions": 182, + "cache_bytes": 4472832, + "step_peak_minus_start_bytes": 4522240 + }, + { + "step": 55, + "cached_positions": 183, + "cache_bytes": 4497408, + "step_peak_minus_start_bytes": 4546816 + }, + { + "step": 56, + "cached_positions": 184, + "cache_bytes": 4521984, + "step_peak_minus_start_bytes": 4571392 + }, + { + "step": 57, + "cached_positions": 185, + "cache_bytes": 4546560, + "step_peak_minus_start_bytes": 4595968 + }, + { + "step": 58, + "cached_positions": 186, + "cache_bytes": 4571136, + "step_peak_minus_start_bytes": 4620544 + }, + { + "step": 59, + "cached_positions": 187, + "cache_bytes": 4595712, + "step_peak_minus_start_bytes": 4645120 + }, + { + "step": 60, + "cached_positions": 188, + "cache_bytes": 4620288, + "step_peak_minus_start_bytes": 4669696 + }, + { + "step": 61, + "cached_positions": 189, + "cache_bytes": 4644864, + "step_peak_minus_start_bytes": 4694272 + }, + { + "step": 62, + "cached_positions": 190, + "cache_bytes": 4669440, + "step_peak_minus_start_bytes": 4718848 + }, + { + "step": 63, + "cached_positions": 191, + "cache_bytes": 4694016, + "step_peak_minus_start_bytes": 4743424 + } + ] + }, + { + "batch": 1, + "prompt_len": 32, + "steps": [ + { + "step": 0, + "cached_positions": 32, + "cache_bytes": 98304, + "step_peak_minus_start_bytes": 344320 + }, + { + "step": 1, + "cached_positions": 33, + "cache_bytes": 101376, + "step_peak_minus_start_bytes": 109312 + }, + { + "step": 2, + "cached_positions": 34, + "cache_bytes": 104448, + "step_peak_minus_start_bytes": 111360 + }, + { + "step": 3, + "cached_positions": 35, + "cache_bytes": 107520, + "step_peak_minus_start_bytes": 115456 + }, + { + "step": 4, + "cached_positions": 36, + "cache_bytes": 110592, + "step_peak_minus_start_bytes": 117504 + }, + { + "step": 5, + "cached_positions": 37, + "cache_bytes": 113664, + "step_peak_minus_start_bytes": 121600 + }, + { + "step": 6, + "cached_positions": 38, + "cache_bytes": 116736, + "step_peak_minus_start_bytes": 123648 + }, + { + "step": 7, + "cached_positions": 39, + "cache_bytes": 119808, + "step_peak_minus_start_bytes": 127744 + }, + { + "step": 8, + "cached_positions": 40, + "cache_bytes": 122880, + "step_peak_minus_start_bytes": 129792 + }, + { + "step": 9, + "cached_positions": 41, + "cache_bytes": 125952, + "step_peak_minus_start_bytes": 133888 + }, + { + "step": 10, + "cached_positions": 42, + "cache_bytes": 129024, + "step_peak_minus_start_bytes": 135936 + }, + { + "step": 11, + "cached_positions": 43, + "cache_bytes": 132096, + "step_peak_minus_start_bytes": 140032 + }, + { + "step": 12, + "cached_positions": 44, + "cache_bytes": 135168, + "step_peak_minus_start_bytes": 142080 + }, + { + "step": 13, + "cached_positions": 45, + "cache_bytes": 138240, + "step_peak_minus_start_bytes": 146176 + }, + { + "step": 14, + "cached_positions": 46, + "cache_bytes": 141312, + "step_peak_minus_start_bytes": 148224 + }, + { + "step": 15, + "cached_positions": 47, + "cache_bytes": 144384, + "step_peak_minus_start_bytes": 152320 + }, + { + "step": 16, + "cached_positions": 48, + "cache_bytes": 147456, + "step_peak_minus_start_bytes": 154368 + }, + { + "step": 17, + "cached_positions": 49, + "cache_bytes": 150528, + "step_peak_minus_start_bytes": 158464 + }, + { + "step": 18, + "cached_positions": 50, + "cache_bytes": 153600, + "step_peak_minus_start_bytes": 160512 + }, + { + "step": 19, + "cached_positions": 51, + "cache_bytes": 156672, + "step_peak_minus_start_bytes": 164608 + }, + { + "step": 20, + "cached_positions": 52, + "cache_bytes": 159744, + "step_peak_minus_start_bytes": 166656 + }, + { + "step": 21, + "cached_positions": 53, + "cache_bytes": 162816, + "step_peak_minus_start_bytes": 170752 + }, + { + "step": 22, + "cached_positions": 54, + "cache_bytes": 165888, + "step_peak_minus_start_bytes": 172800 + }, + { + "step": 23, + "cached_positions": 55, + "cache_bytes": 168960, + "step_peak_minus_start_bytes": 176896 + }, + { + "step": 24, + "cached_positions": 56, + "cache_bytes": 172032, + "step_peak_minus_start_bytes": 178944 + }, + { + "step": 25, + "cached_positions": 57, + "cache_bytes": 175104, + "step_peak_minus_start_bytes": 183040 + }, + { + "step": 26, + "cached_positions": 58, + "cache_bytes": 178176, + "step_peak_minus_start_bytes": 185088 + }, + { + "step": 27, + "cached_positions": 59, + "cache_bytes": 181248, + "step_peak_minus_start_bytes": 189184 + }, + { + "step": 28, + "cached_positions": 60, + "cache_bytes": 184320, + "step_peak_minus_start_bytes": 191232 + }, + { + "step": 29, + "cached_positions": 61, + "cache_bytes": 187392, + "step_peak_minus_start_bytes": 195328 + }, + { + "step": 30, + "cached_positions": 62, + "cache_bytes": 190464, + "step_peak_minus_start_bytes": 197376 + }, + { + "step": 31, + "cached_positions": 63, + "cache_bytes": 193536, + "step_peak_minus_start_bytes": 201472 + }, + { + "step": 32, + "cached_positions": 64, + "cache_bytes": 196608, + "step_peak_minus_start_bytes": 203520 + }, + { + "step": 33, + "cached_positions": 65, + "cache_bytes": 199680, + "step_peak_minus_start_bytes": 207616 + }, + { + "step": 34, + "cached_positions": 66, + "cache_bytes": 202752, + "step_peak_minus_start_bytes": 209664 + }, + { + "step": 35, + "cached_positions": 67, + "cache_bytes": 205824, + "step_peak_minus_start_bytes": 213760 + }, + { + "step": 36, + "cached_positions": 68, + "cache_bytes": 208896, + "step_peak_minus_start_bytes": 215808 + }, + { + "step": 37, + "cached_positions": 69, + "cache_bytes": 211968, + "step_peak_minus_start_bytes": 219904 + }, + { + "step": 38, + "cached_positions": 70, + "cache_bytes": 215040, + "step_peak_minus_start_bytes": 221952 + }, + { + "step": 39, + "cached_positions": 71, + "cache_bytes": 218112, + "step_peak_minus_start_bytes": 226048 + }, + { + "step": 40, + "cached_positions": 72, + "cache_bytes": 221184, + "step_peak_minus_start_bytes": 228096 + }, + { + "step": 41, + "cached_positions": 73, + "cache_bytes": 224256, + "step_peak_minus_start_bytes": 232192 + }, + { + "step": 42, + "cached_positions": 74, + "cache_bytes": 227328, + "step_peak_minus_start_bytes": 234240 + }, + { + "step": 43, + "cached_positions": 75, + "cache_bytes": 230400, + "step_peak_minus_start_bytes": 238336 + }, + { + "step": 44, + "cached_positions": 76, + "cache_bytes": 233472, + "step_peak_minus_start_bytes": 240384 + }, + { + "step": 45, + "cached_positions": 77, + "cache_bytes": 236544, + "step_peak_minus_start_bytes": 244480 + }, + { + "step": 46, + "cached_positions": 78, + "cache_bytes": 239616, + "step_peak_minus_start_bytes": 246528 + }, + { + "step": 47, + "cached_positions": 79, + "cache_bytes": 242688, + "step_peak_minus_start_bytes": 250624 + }, + { + "step": 48, + "cached_positions": 80, + "cache_bytes": 245760, + "step_peak_minus_start_bytes": 252672 + }, + { + "step": 49, + "cached_positions": 81, + "cache_bytes": 248832, + "step_peak_minus_start_bytes": 256768 + }, + { + "step": 50, + "cached_positions": 82, + "cache_bytes": 251904, + "step_peak_minus_start_bytes": 258816 + }, + { + "step": 51, + "cached_positions": 83, + "cache_bytes": 254976, + "step_peak_minus_start_bytes": 262912 + }, + { + "step": 52, + "cached_positions": 84, + "cache_bytes": 258048, + "step_peak_minus_start_bytes": 264960 + }, + { + "step": 53, + "cached_positions": 85, + "cache_bytes": 261120, + "step_peak_minus_start_bytes": 269056 + }, + { + "step": 54, + "cached_positions": 86, + "cache_bytes": 264192, + "step_peak_minus_start_bytes": 271104 + }, + { + "step": 55, + "cached_positions": 87, + "cache_bytes": 267264, + "step_peak_minus_start_bytes": 275200 + }, + { + "step": 56, + "cached_positions": 88, + "cache_bytes": 270336, + "step_peak_minus_start_bytes": 277248 + }, + { + "step": 57, + "cached_positions": 89, + "cache_bytes": 273408, + "step_peak_minus_start_bytes": 281344 + }, + { + "step": 58, + "cached_positions": 90, + "cache_bytes": 276480, + "step_peak_minus_start_bytes": 283392 + }, + { + "step": 59, + "cached_positions": 91, + "cache_bytes": 279552, + "step_peak_minus_start_bytes": 287488 + }, + { + "step": 60, + "cached_positions": 92, + "cache_bytes": 282624, + "step_peak_minus_start_bytes": 289536 + }, + { + "step": 61, + "cached_positions": 93, + "cache_bytes": 285696, + "step_peak_minus_start_bytes": 293632 + }, + { + "step": 62, + "cached_positions": 94, + "cache_bytes": 288768, + "step_peak_minus_start_bytes": 295680 + }, + { + "step": 63, + "cached_positions": 95, + "cache_bytes": 291840, + "step_peak_minus_start_bytes": 299776 + }, + { + "step": 64, + "cached_positions": 96, + "cache_bytes": 294912, + "step_peak_minus_start_bytes": 301824 + }, + { + "step": 65, + "cached_positions": 97, + "cache_bytes": 297984, + "step_peak_minus_start_bytes": 305920 + }, + { + "step": 66, + "cached_positions": 98, + "cache_bytes": 301056, + "step_peak_minus_start_bytes": 307968 + }, + { + "step": 67, + "cached_positions": 99, + "cache_bytes": 304128, + "step_peak_minus_start_bytes": 312064 + }, + { + "step": 68, + "cached_positions": 100, + "cache_bytes": 307200, + "step_peak_minus_start_bytes": 314112 + }, + { + "step": 69, + "cached_positions": 101, + "cache_bytes": 310272, + "step_peak_minus_start_bytes": 318208 + }, + { + "step": 70, + "cached_positions": 102, + "cache_bytes": 313344, + "step_peak_minus_start_bytes": 320256 + }, + { + "step": 71, + "cached_positions": 103, + "cache_bytes": 316416, + "step_peak_minus_start_bytes": 324352 + }, + { + "step": 72, + "cached_positions": 104, + "cache_bytes": 319488, + "step_peak_minus_start_bytes": 326400 + }, + { + "step": 73, + "cached_positions": 105, + "cache_bytes": 322560, + "step_peak_minus_start_bytes": 330496 + }, + { + "step": 74, + "cached_positions": 106, + "cache_bytes": 325632, + "step_peak_minus_start_bytes": 332544 + }, + { + "step": 75, + "cached_positions": 107, + "cache_bytes": 328704, + "step_peak_minus_start_bytes": 336640 + }, + { + "step": 76, + "cached_positions": 108, + "cache_bytes": 331776, + "step_peak_minus_start_bytes": 338688 + }, + { + "step": 77, + "cached_positions": 109, + "cache_bytes": 334848, + "step_peak_minus_start_bytes": 342784 + }, + { + "step": 78, + "cached_positions": 110, + "cache_bytes": 337920, + "step_peak_minus_start_bytes": 344832 + }, + { + "step": 79, + "cached_positions": 111, + "cache_bytes": 340992, + "step_peak_minus_start_bytes": 348928 + }, + { + "step": 80, + "cached_positions": 112, + "cache_bytes": 344064, + "step_peak_minus_start_bytes": 350976 + }, + { + "step": 81, + "cached_positions": 113, + "cache_bytes": 347136, + "step_peak_minus_start_bytes": 355072 + }, + { + "step": 82, + "cached_positions": 114, + "cache_bytes": 350208, + "step_peak_minus_start_bytes": 357120 + }, + { + "step": 83, + "cached_positions": 115, + "cache_bytes": 353280, + "step_peak_minus_start_bytes": 361216 + }, + { + "step": 84, + "cached_positions": 116, + "cache_bytes": 356352, + "step_peak_minus_start_bytes": 363264 + }, + { + "step": 85, + "cached_positions": 117, + "cache_bytes": 359424, + "step_peak_minus_start_bytes": 367360 + }, + { + "step": 86, + "cached_positions": 118, + "cache_bytes": 362496, + "step_peak_minus_start_bytes": 369408 + }, + { + "step": 87, + "cached_positions": 119, + "cache_bytes": 365568, + "step_peak_minus_start_bytes": 373504 + }, + { + "step": 88, + "cached_positions": 120, + "cache_bytes": 368640, + "step_peak_minus_start_bytes": 375552 + }, + { + "step": 89, + "cached_positions": 121, + "cache_bytes": 371712, + "step_peak_minus_start_bytes": 379648 + }, + { + "step": 90, + "cached_positions": 122, + "cache_bytes": 374784, + "step_peak_minus_start_bytes": 381696 + }, + { + "step": 91, + "cached_positions": 123, + "cache_bytes": 377856, + "step_peak_minus_start_bytes": 385792 + }, + { + "step": 92, + "cached_positions": 124, + "cache_bytes": 380928, + "step_peak_minus_start_bytes": 387840 + }, + { + "step": 93, + "cached_positions": 125, + "cache_bytes": 384000, + "step_peak_minus_start_bytes": 391936 + }, + { + "step": 94, + "cached_positions": 126, + "cache_bytes": 387072, + "step_peak_minus_start_bytes": 393984 + }, + { + "step": 95, + "cached_positions": 127, + "cache_bytes": 390144, + "step_peak_minus_start_bytes": 398080 + }, + { + "step": 96, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 400128 + }, + { + "step": 97, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 98, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 99, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 100, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 101, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 102, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 103, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 104, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 105, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 106, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 107, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 108, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 109, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 110, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 111, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 112, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 113, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 114, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 115, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 116, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 117, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 118, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 119, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 120, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 121, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 122, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 123, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 124, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 125, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 126, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 127, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 128, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 129, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 130, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 131, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 132, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 133, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 134, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 135, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 136, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 137, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 138, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 139, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 140, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 141, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 142, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 143, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 144, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 145, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 146, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 147, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 148, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 149, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 150, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 151, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 152, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 153, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 154, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 155, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 156, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 157, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 158, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 159, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 160, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 161, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 162, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 163, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 164, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 165, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 166, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 167, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 168, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 169, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 170, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 171, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 172, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 173, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 174, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 175, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 176, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 177, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 178, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 179, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 180, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 181, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 182, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 183, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 184, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 185, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 186, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 187, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 188, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 189, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 190, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 191, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 192, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 193, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 194, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 195, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 196, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 197, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 198, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 199, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 200, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 201, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 202, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 203, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 204, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 205, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 206, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 207, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 208, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 209, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 210, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 211, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 212, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 213, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 214, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 215, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 216, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 217, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 218, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 219, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 220, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 221, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 222, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + }, + { + "step": 223, + "cached_positions": 128, + "cache_bytes": 393216, + "step_peak_minus_start_bytes": 1377280 + } + ] + } + ] + } +} diff --git a/experiments/records/legacy/20261022T045414Z_legacy_kv_cache_cpu_13530f4.json b/experiments/records/legacy/20261022T045414Z_legacy_kv_cache_cpu_13530f4.json new file mode 100644 index 0000000..8f73223 --- /dev/null +++ b/experiments/records/legacy/20261022T045414Z_legacy_kv_cache_cpu_13530f4.json @@ -0,0 +1,65 @@ +{ + "record_version": 1, + "created_utc": "20261022T045414Z", + "experiment": "legacy_kv_cache_benchmark", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "cpu", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "13530f436d5b9b9b1e3480fd33d4dc6891c0cf28", + "branch": "experiments/sdpa-training-v2", + "dirty": true + }, + "command": "python scripts/benchmark_kv_cache.py --checkpoint artifacts/runs/sdpa-v1/auto-s42/best_checkpoint.pt --tokens 96 --device cpu --attention-backend mha --record-dir experiments/records/legacy" + }, + "settings": { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s42/best_checkpoint.pt", + "checkpoint_sha256": "23f0aa813704e0cfc02d0780386fe76d1ee26f70aaa674ce83a849a85fb93546", + "tokens": 96, + "repeats": 7, + "warmup": 1, + "seed": 1234, + "attention_backend": "mha" + }, + "results": { + "device": "cpu", + "attention_backend": "mha", + "generated_tokens": 96, + "repeats": 7, + "warmup_runs_per_mode": 1, + "prompt_tokens": 21, + "uncached_seconds_samples": [ + 0.12532191700302064, + 0.12442887500219513, + 0.12452837500313763, + 0.12378300000273157, + 0.12726129200018477, + 0.12416158399719279, + 0.14033087500138208 + ], + "cached_seconds_samples": [ + 0.03586633400118444, + 0.0355714579927735, + 0.03620683400367852, + 0.03698012499080505, + 0.03613904099620413, + 0.03671716600365471, + 0.03586041700327769 + ], + "uncached_seconds": 0.12452837500313763, + "cached_seconds": 0.03613904099620413, + "uncached_tokens_per_second": 770.9086382728528, + "cached_tokens_per_second": 2656.406959168711, + "speedup": 3.4458129372115183 + } +} diff --git a/experiments/records/legacy/20261022T045419Z_legacy_int8_cpu_13530f4.json b/experiments/records/legacy/20261022T045419Z_legacy_int8_cpu_13530f4.json new file mode 100644 index 0000000..f52a798 --- /dev/null +++ b/experiments/records/legacy/20261022T045419Z_legacy_int8_cpu_13530f4.json @@ -0,0 +1,46 @@ +{ + "record_version": 1, + "created_utc": "20261022T045419Z", + "experiment": "legacy_int8_comparison", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "cpu", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "13530f436d5b9b9b1e3480fd33d4dc6891c0cf28", + "branch": "experiments/sdpa-training-v2", + "dirty": true + }, + "command": "python scripts/quantize.py --checkpoint artifacts/runs/sdpa-v1/auto-s42/best_checkpoint.pt --validation-text data/science/validation.txt --attention-backend mha --output /private/tmp/claude-501/-Users-anvit-Downloads-Projects-MiniLM-Lab/ad12857c-f862-4809-a0b1-016af22e423c/scratchpad/int8.json --record-dir experiments/records/legacy" + }, + "settings": { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s42/best_checkpoint.pt", + "checkpoint_sha256": "23f0aa813704e0cfc02d0780386fe76d1ee26f70aaa674ce83a849a85fb93546", + "validation_text": "data/science/validation.txt", + "validation_sha256": "e25d4e45706bbf4251129dbb079f3487660c453cf8b34407901350538b23e2e7", + "latency": "median of 5 repeats of 32 forward passes after one warmup", + "attention_backend": "mha" + }, + "results": { + "method": "per-output-channel symmetric INT8 linear weights; dequantize on each forward pass", + "scope": "MLP nn.Linear layers; attention projections and tied LM head remain FP32", + "device": "cpu", + "attention_backend": "mha", + "original_validation_loss": 2.866917291863034, + "int8_validation_loss": 2.867066869126566, + "loss_delta": 0.00014957726353204137, + "original_stored_bytes": 9844480, + "int8_stored_bytes": 4563712, + "original_32_forward_seconds": 0.0560352909960784, + "int8_32_forward_seconds": 0.06615462500485592 + } +} diff --git a/experiments/records/legacy/20261022T045428Z_legacy_kv_cache_cpu_13530f4.json b/experiments/records/legacy/20261022T045428Z_legacy_kv_cache_cpu_13530f4.json new file mode 100644 index 0000000..43e55e9 --- /dev/null +++ b/experiments/records/legacy/20261022T045428Z_legacy_kv_cache_cpu_13530f4.json @@ -0,0 +1,65 @@ +{ + "record_version": 1, + "created_utc": "20261022T045428Z", + "experiment": "legacy_kv_cache_benchmark", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "cpu", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "13530f436d5b9b9b1e3480fd33d4dc6891c0cf28", + "branch": "experiments/sdpa-training-v2", + "dirty": false + }, + "command": "python scripts/benchmark_kv_cache.py --checkpoint artifacts/runs/sdpa-v1/auto-s42/best_checkpoint.pt --tokens 96 --device cpu --attention-backend mha --record-dir experiments/records/legacy" + }, + "settings": { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s42/best_checkpoint.pt", + "checkpoint_sha256": "23f0aa813704e0cfc02d0780386fe76d1ee26f70aaa674ce83a849a85fb93546", + "tokens": 96, + "repeats": 7, + "warmup": 1, + "seed": 1234, + "attention_backend": "mha" + }, + "results": { + "device": "cpu", + "attention_backend": "mha", + "generated_tokens": 96, + "repeats": 7, + "warmup_runs_per_mode": 1, + "prompt_tokens": 21, + "uncached_seconds_samples": [ + 0.12584333399718162, + 0.12498416600283235, + 0.12487691699061543, + 0.1234795420023147, + 0.12357833299029153, + 0.123227832998964, + 0.12284166700555943 + ], + "cached_seconds_samples": [ + 0.03629633299715351, + 0.03593787499994505, + 0.03621541601023637, + 0.03652374999364838, + 0.0360561669949675, + 0.03638537500228267, + 0.036312374999397434 + ], + "uncached_seconds": 0.12357833299029153, + "cached_seconds": 0.03629633299715351, + "uncached_tokens_per_second": 776.8352078963703, + "cached_tokens_per_second": 2644.895284808211, + "speedup": 3.4047057315675104 + } +} diff --git a/experiments/records/legacy/20261022T045433Z_legacy_int8_cpu_13530f4.json b/experiments/records/legacy/20261022T045433Z_legacy_int8_cpu_13530f4.json new file mode 100644 index 0000000..fd28046 --- /dev/null +++ b/experiments/records/legacy/20261022T045433Z_legacy_int8_cpu_13530f4.json @@ -0,0 +1,46 @@ +{ + "record_version": 1, + "created_utc": "20261022T045433Z", + "experiment": "legacy_int8_comparison", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "cpu", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "13530f436d5b9b9b1e3480fd33d4dc6891c0cf28", + "branch": "experiments/sdpa-training-v2", + "dirty": false + }, + "command": "python scripts/quantize.py --checkpoint artifacts/runs/sdpa-v1/auto-s42/best_checkpoint.pt --validation-text data/science/validation.txt --attention-backend mha --output /private/tmp/claude-501/-Users-anvit-Downloads-Projects-MiniLM-Lab/ad12857c-f862-4809-a0b1-016af22e423c/scratchpad/int8.json --record-dir experiments/records/legacy" + }, + "settings": { + "checkpoint": "artifacts/runs/sdpa-v1/auto-s42/best_checkpoint.pt", + "checkpoint_sha256": "23f0aa813704e0cfc02d0780386fe76d1ee26f70aaa674ce83a849a85fb93546", + "validation_text": "data/science/validation.txt", + "validation_sha256": "e25d4e45706bbf4251129dbb079f3487660c453cf8b34407901350538b23e2e7", + "latency": "median of 5 repeats of 32 forward passes after one warmup", + "attention_backend": "mha" + }, + "results": { + "method": "per-output-channel symmetric INT8 linear weights; dequantize on each forward pass", + "scope": "MLP nn.Linear layers; attention projections and tied LM head remain FP32", + "device": "cpu", + "attention_backend": "mha", + "original_validation_loss": 2.866917291863034, + "int8_validation_loss": 2.867066869126566, + "loss_delta": 0.00014957726353204137, + "original_stored_bytes": 9844480, + "int8_stored_bytes": 4563712, + "original_32_forward_seconds": 0.055284499991103075, + "int8_32_forward_seconds": 0.06569512499845587 + } +} diff --git a/experiments/records/training/20261022T034420Z_training_seed42_0502118.json b/experiments/records/training/20261022T034420Z_training_seed42_0502118.json new file mode 100644 index 0000000..9bc6aa6 --- /dev/null +++ b/experiments/records/training/20261022T034420Z_training_seed42_0502118.json @@ -0,0 +1,127 @@ +{ + "record_version": 1, + "created_utc": "20261022T034420Z", + "experiment": "training", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "cpu", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "0502118a1a0af31c8480f6b499d9347916330a79", + "branch": "experiments/attention-kv-audit", + "dirty": true + }, + "command": "python scripts/record_training_run.py artifacts/runs/science-v2-s42 --code-commit 1fb8f37 --note 'Trained at commit 1fb8f37 (CPU, float32).' --training-command 'python scripts/train_tiny.py data/science/train.txt --validation-text data/science/validation.txt --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 --device cpu --seed 42 --output-dir artifacts/runs/science-v2-s42'" + }, + "settings": { + "dtype": "float32", + "seed": 42, + "model_config": { + "vocab_size": 256, + "context_length": 128, + "embedding_dim": 192, + "num_layers": 4, + "num_heads": 6, + "dropout": 0.0, + "positional_encoding": "learned", + "norm": "rmsnorm", + "activation": "swiglu", + "rope_theta": 10000.0 + }, + "tokenizer": "byte", + "optimizer": "AdamW lr=3e-4 (scripts/train_tiny.py defaults)", + "steps": 1000, + "training_command": "python scripts/train_tiny.py data/science/train.txt --validation-text data/science/validation.txt --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 --device cpu --seed 42 --output-dir artifacts/runs/science-v2-s42", + "training_code_commit": "1fb8f37", + "note": "Trained at commit 1fb8f37 (CPU, float32).", + "data": { + "train": { + "path": "data/science/train.txt", + "sha256": "19c057053f222c3084fba3693c1c295ea94f6cda1ee5fbf5f917953fcb91ef00" + }, + "validation": { + "path": "data/science/validation.txt", + "sha256": "e25d4e45706bbf4251129dbb079f3487660c453cf8b34407901350538b23e2e7" + }, + "corpus_sha256_in_checkpoint": "f6642de41672b3cac70017cd5f02312975c0bedfa32be8e79056cf36a48e943f", + "manifest": { + "split_method": "first approximately 90% of bytes at a paragraph boundary from each document for training; remainder for validation", + "sources": [ + { + "name": "darwin-1859", + "url": "https://www.gutenberg.org/cache/epub/1228/pg1228.txt", + "raw_sha256": "ededa9c0bf8761efed092c303b46c1c92de956838cba6249a33bedfd6d7363b4", + "paragraphs": 1322, + "train_paragraphs": 701, + "validation_bytes": 93406 + }, + { + "name": "einstein-1924", + "url": "https://www.gutenberg.org/cache/epub/30155/pg30155.txt", + "raw_sha256": "86ed8156239455cbb6ed33e06097707d3ab7c40d46e9aa5ff32d3c3f94ef68fd", + "paragraphs": 564, + "train_paragraphs": 490, + "validation_bytes": 18704 + } + ], + "train_bytes": 1008977, + "validation_bytes": 112112, + "train_sha256": "19c057053f222c3084fba3693c1c295ea94f6cda1ee5fbf5f917953fcb91ef00", + "validation_sha256": "e25d4e45706bbf4251129dbb079f3487660c453cf8b34407901350538b23e2e7" + } + } + }, + "results": { + "metrics": [ + { + "step": 250, + "train_loss": 3.0327484011650085, + "validation_loss": 3.275113352777709, + "perplexity": 26.446223063511567, + "bits_per_token": 4.724989792401627, + "validation_tokens": 112111, + "elapsed_seconds": 27.38209637500404 + }, + { + "step": 500, + "train_loss": 3.0437499284744263, + "validation_loss": 3.279687989213292, + "perplexity": 26.567482065577426, + "bits_per_token": 4.731589597701113, + "validation_tokens": 112111, + "elapsed_seconds": 55.14372462499887 + }, + { + "step": 750, + "train_loss": 3.058131754398346, + "validation_loss": 3.273437816850808, + "perplexity": 26.401948568749216, + "bits_per_token": 4.7225725050290555, + "validation_tokens": 112111, + "elapsed_seconds": 83.4805836250016 + }, + { + "step": 1000, + "train_loss": 2.6655086278915405, + "validation_loss": 2.866917291863034, + "perplexity": 17.582732137354572, + "bits_per_token": 4.136087359609617, + "validation_tokens": 112111, + "elapsed_seconds": 114.25916233300813 + } + ], + "best_step": 1000, + "best_validation_loss": 2.866917291863034, + "best_checkpoint_sha256": "657c3c616443474aa04c407c870c50a653446522eda677b15064a8639772501d", + "final_checkpoint_sha256": "ee6c3d2164c964195399797351c2cff2c301a9f1f948dcc126fefcc89cca9251" + } +} diff --git a/experiments/records/training/20261022T034421Z_training_seed43_0502118.json b/experiments/records/training/20261022T034421Z_training_seed43_0502118.json new file mode 100644 index 0000000..b3d54e5 --- /dev/null +++ b/experiments/records/training/20261022T034421Z_training_seed43_0502118.json @@ -0,0 +1,127 @@ +{ + "record_version": 1, + "created_utc": "20261022T034421Z", + "experiment": "training", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "cpu", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "0502118a1a0af31c8480f6b499d9347916330a79", + "branch": "experiments/attention-kv-audit", + "dirty": true + }, + "command": "python scripts/record_training_run.py artifacts/runs/science-v2-s43 --code-commit 1fb8f37 --note 'Trained with 1fb8f37 training code plus uncommitted model.py edits later committed as 274722f; the default training path was verified bitwise-identical by retraining seed 42 on the new code.' --training-command 'python scripts/train_tiny.py data/science/train.txt --validation-text data/science/validation.txt --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 --device cpu --seed 43 --output-dir artifacts/runs/science-v2-s43'" + }, + "settings": { + "dtype": "float32", + "seed": 43, + "model_config": { + "vocab_size": 256, + "context_length": 128, + "embedding_dim": 192, + "num_layers": 4, + "num_heads": 6, + "dropout": 0.0, + "positional_encoding": "learned", + "norm": "rmsnorm", + "activation": "swiglu", + "rope_theta": 10000.0 + }, + "tokenizer": "byte", + "optimizer": "AdamW lr=3e-4 (scripts/train_tiny.py defaults)", + "steps": 1000, + "training_command": "python scripts/train_tiny.py data/science/train.txt --validation-text data/science/validation.txt --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 --device cpu --seed 43 --output-dir artifacts/runs/science-v2-s43", + "training_code_commit": "1fb8f37", + "note": "Trained with 1fb8f37 training code plus uncommitted model.py edits later committed as 274722f; the default training path was verified bitwise-identical by retraining seed 42 on the new code.", + "data": { + "train": { + "path": "data/science/train.txt", + "sha256": "19c057053f222c3084fba3693c1c295ea94f6cda1ee5fbf5f917953fcb91ef00" + }, + "validation": { + "path": "data/science/validation.txt", + "sha256": "e25d4e45706bbf4251129dbb079f3487660c453cf8b34407901350538b23e2e7" + }, + "corpus_sha256_in_checkpoint": "f6642de41672b3cac70017cd5f02312975c0bedfa32be8e79056cf36a48e943f", + "manifest": { + "split_method": "first approximately 90% of bytes at a paragraph boundary from each document for training; remainder for validation", + "sources": [ + { + "name": "darwin-1859", + "url": "https://www.gutenberg.org/cache/epub/1228/pg1228.txt", + "raw_sha256": "ededa9c0bf8761efed092c303b46c1c92de956838cba6249a33bedfd6d7363b4", + "paragraphs": 1322, + "train_paragraphs": 701, + "validation_bytes": 93406 + }, + { + "name": "einstein-1924", + "url": "https://www.gutenberg.org/cache/epub/30155/pg30155.txt", + "raw_sha256": "86ed8156239455cbb6ed33e06097707d3ab7c40d46e9aa5ff32d3c3f94ef68fd", + "paragraphs": 564, + "train_paragraphs": 490, + "validation_bytes": 18704 + } + ], + "train_bytes": 1008977, + "validation_bytes": 112112, + "train_sha256": "19c057053f222c3084fba3693c1c295ea94f6cda1ee5fbf5f917953fcb91ef00", + "validation_sha256": "e25d4e45706bbf4251129dbb079f3487660c453cf8b34407901350538b23e2e7" + } + } + }, + "results": { + "metrics": [ + { + "step": 250, + "train_loss": 3.125346601009369, + "validation_loss": 3.27685190798933, + "perplexity": 26.49224127345457, + "bits_per_token": 4.727497997383744, + "validation_tokens": 112111, + "elapsed_seconds": 29.149193541990826 + }, + { + "step": 500, + "train_loss": 3.1471182703971863, + "validation_loss": 3.2741675599209956, + "perplexity": 26.42122223931517, + "bits_per_token": 4.723625301737538, + "validation_tokens": 112111, + "elapsed_seconds": 58.77116108300106 + }, + { + "step": 750, + "train_loss": 3.054605782032013, + "validation_loss": 3.271949987806797, + "perplexity": 26.36269619050522, + "bits_per_token": 4.72042602144557, + "validation_tokens": 112111, + "elapsed_seconds": 96.52907974999107 + }, + { + "step": 1000, + "train_loss": 2.851951539516449, + "validation_loss": 2.9438789420007883, + "perplexity": 18.989362272891913, + "bits_per_token": 4.247119550601986, + "validation_tokens": 112111, + "elapsed_seconds": 131.09946783300256 + } + ], + "best_step": 1000, + "best_validation_loss": 2.9438789420007883, + "best_checkpoint_sha256": "5ef10871bf409763d01ab42d66f6c013ced19da345baa3b8f1181dc76830a205", + "final_checkpoint_sha256": "cad6068df97de14c466a2508aca2804d93b0c422028905e4d6c1996fd1c6ea0a" + } +} diff --git a/experiments/records/training/20261022T034421Z_training_seed44_0502118.json b/experiments/records/training/20261022T034421Z_training_seed44_0502118.json new file mode 100644 index 0000000..311c9cc --- /dev/null +++ b/experiments/records/training/20261022T034421Z_training_seed44_0502118.json @@ -0,0 +1,127 @@ +{ + "record_version": 1, + "created_utc": "20261022T034421Z", + "experiment": "training", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "cpu", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "0502118a1a0af31c8480f6b499d9347916330a79", + "branch": "experiments/attention-kv-audit", + "dirty": true + }, + "command": "python scripts/record_training_run.py artifacts/runs/science-v2-s44 --code-commit 1fb8f37 --note 'Trained with 1fb8f37 training code plus uncommitted model.py edits later committed as 274722f; the default training path was verified bitwise-identical by retraining seed 42 on the new code.' --training-command 'python scripts/train_tiny.py data/science/train.txt --validation-text data/science/validation.txt --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 --device cpu --seed 44 --output-dir artifacts/runs/science-v2-s44'" + }, + "settings": { + "dtype": "float32", + "seed": 44, + "model_config": { + "vocab_size": 256, + "context_length": 128, + "embedding_dim": 192, + "num_layers": 4, + "num_heads": 6, + "dropout": 0.0, + "positional_encoding": "learned", + "norm": "rmsnorm", + "activation": "swiglu", + "rope_theta": 10000.0 + }, + "tokenizer": "byte", + "optimizer": "AdamW lr=3e-4 (scripts/train_tiny.py defaults)", + "steps": 1000, + "training_command": "python scripts/train_tiny.py data/science/train.txt --validation-text data/science/validation.txt --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 --device cpu --seed 44 --output-dir artifacts/runs/science-v2-s44", + "training_code_commit": "1fb8f37", + "note": "Trained with 1fb8f37 training code plus uncommitted model.py edits later committed as 274722f; the default training path was verified bitwise-identical by retraining seed 42 on the new code.", + "data": { + "train": { + "path": "data/science/train.txt", + "sha256": "19c057053f222c3084fba3693c1c295ea94f6cda1ee5fbf5f917953fcb91ef00" + }, + "validation": { + "path": "data/science/validation.txt", + "sha256": "e25d4e45706bbf4251129dbb079f3487660c453cf8b34407901350538b23e2e7" + }, + "corpus_sha256_in_checkpoint": "f6642de41672b3cac70017cd5f02312975c0bedfa32be8e79056cf36a48e943f", + "manifest": { + "split_method": "first approximately 90% of bytes at a paragraph boundary from each document for training; remainder for validation", + "sources": [ + { + "name": "darwin-1859", + "url": "https://www.gutenberg.org/cache/epub/1228/pg1228.txt", + "raw_sha256": "ededa9c0bf8761efed092c303b46c1c92de956838cba6249a33bedfd6d7363b4", + "paragraphs": 1322, + "train_paragraphs": 701, + "validation_bytes": 93406 + }, + { + "name": "einstein-1924", + "url": "https://www.gutenberg.org/cache/epub/30155/pg30155.txt", + "raw_sha256": "86ed8156239455cbb6ed33e06097707d3ab7c40d46e9aa5ff32d3c3f94ef68fd", + "paragraphs": 564, + "train_paragraphs": 490, + "validation_bytes": 18704 + } + ], + "train_bytes": 1008977, + "validation_bytes": 112112, + "train_sha256": "19c057053f222c3084fba3693c1c295ea94f6cda1ee5fbf5f917953fcb91ef00", + "validation_sha256": "e25d4e45706bbf4251129dbb079f3487660c453cf8b34407901350538b23e2e7" + } + } + }, + "results": { + "metrics": [ + { + "step": 250, + "train_loss": 3.0683940052986145, + "validation_loss": 3.2747702210461864, + "perplexity": 26.437150081902516, + "bits_per_token": 4.724494757954187, + "validation_tokens": 112111, + "elapsed_seconds": 33.43322775000706 + }, + { + "step": 500, + "train_loss": 3.061283588409424, + "validation_loss": 3.275096681362127, + "perplexity": 26.445782171211466, + "bits_per_token": 4.724965740633043, + "validation_tokens": 112111, + "elapsed_seconds": 66.52756354100711 + }, + { + "step": 750, + "train_loss": 3.0149829387664795, + "validation_loss": 3.270937874591696, + "perplexity": 26.336027655370078, + "bits_per_token": 4.718965850729326, + "validation_tokens": 112111, + "elapsed_seconds": 101.18632249999791 + }, + { + "step": 1000, + "train_loss": 2.787301540374756, + "validation_loss": 3.0477406168084813, + "perplexity": 21.067690622638423, + "bits_per_token": 4.396960273785466, + "validation_tokens": 112111, + "elapsed_seconds": 135.10196504100168 + } + ], + "best_step": 1000, + "best_validation_loss": 3.0477406168084813, + "best_checkpoint_sha256": "2f325db359957aaec71472bd94c9444df65e4ef3638aa70da13fbeb512aa68a4", + "final_checkpoint_sha256": "e8de131c738a698599cf36acedddfc7da3ad7f7df775e5e8c52eb264b8d69b44" + } +} diff --git a/experiments/records/training/20261022T040350Z_training_science-v1_auto_seed42_48e95f8.json b/experiments/records/training/20261022T040350Z_training_science-v1_auto_seed42_48e95f8.json new file mode 100644 index 0000000..b0cf131 --- /dev/null +++ b/experiments/records/training/20261022T040350Z_training_science-v1_auto_seed42_48e95f8.json @@ -0,0 +1,129 @@ +{ + "record_version": 1, + "created_utc": "20261022T040350Z", + "experiment": "training", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "cpu", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "48e95f8ea09889e6875fb6129026662f8a699cdb", + "branch": "experiments/sdpa-training-v2", + "dirty": true + }, + "command": "python scripts/record_training_run.py artifacts/runs/sdpa-v1/auto-s42 --protocol science-v1 --code-commit 48e95f8 --training-command 'python scripts/train_tiny.py data/science/train.txt --validation-text data/science/validation.txt --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 --device cpu --seed 42 --attention-backend auto --output-dir artifacts/runs/sdpa-v1/auto-s42' --note 'SDPA matched experiment (v1 corpus)'" + }, + "settings": { + "dtype": "float32", + "seed": 42, + "protocol": "science-v1", + "attention_backend": "auto", + "model_config": { + "vocab_size": 256, + "context_length": 128, + "embedding_dim": 192, + "num_layers": 4, + "num_heads": 6, + "dropout": 0.0, + "positional_encoding": "learned", + "norm": "rmsnorm", + "activation": "swiglu", + "rope_theta": 10000.0 + }, + "tokenizer": "byte", + "optimizer": "AdamW lr=3e-4 (scripts/train_tiny.py defaults)", + "steps": 1000, + "training_command": "python scripts/train_tiny.py data/science/train.txt --validation-text data/science/validation.txt --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 --device cpu --seed 42 --attention-backend auto --output-dir artifacts/runs/sdpa-v1/auto-s42", + "training_code_commit": "48e95f8", + "note": "SDPA matched experiment (v1 corpus)", + "data": { + "train": { + "path": "data/science/train.txt", + "sha256": "19c057053f222c3084fba3693c1c295ea94f6cda1ee5fbf5f917953fcb91ef00" + }, + "validation": { + "path": "data/science/validation.txt", + "sha256": "e25d4e45706bbf4251129dbb079f3487660c453cf8b34407901350538b23e2e7" + }, + "corpus_sha256_in_checkpoint": "f6642de41672b3cac70017cd5f02312975c0bedfa32be8e79056cf36a48e943f", + "manifest_path": "data/science/manifest.json", + "manifest_sha256": "bc001b3366d192ab3c5de6b7e4e71801b0c3bba70029c5a954dbb6af661f8af9" + } + }, + "results": { + "metrics": [ + { + "step": 250, + "train_loss": 3.0327484011650085, + "validation_loss": 3.275113352777709, + "perplexity": 26.446223063511567, + "bits_per_token": 4.724989792401627, + "validation_tokens": 112111, + "elapsed_seconds": 26.916708292003023, + "train_seconds": 25.374983163928846, + "eval_seconds": 1.5413287499977741 + }, + { + "step": 500, + "train_loss": 3.0437499284744263, + "validation_loss": 3.279687989213292, + "perplexity": 26.567482065577426, + "bits_per_token": 4.731589597701113, + "validation_tokens": 112111, + "elapsed_seconds": 54.152566375007154, + "train_seconds": 51.067428036025376, + "eval_seconds": 1.5386290419992292 + }, + { + "step": 750, + "train_loss": 3.058131754398346, + "validation_loss": 3.273437816850808, + "perplexity": 26.401948568749216, + "bits_per_token": 4.7225725050290555, + "validation_tokens": 112111, + "elapsed_seconds": 83.14721091699903, + "train_seconds": 78.36790991391172, + "eval_seconds": 1.69361316600407 + }, + { + "step": 1000, + "train_loss": 2.6655086278915405, + "validation_loss": 2.866917291863034, + "perplexity": 17.582732137354572, + "bits_per_token": 4.136087359609617, + "validation_tokens": 112111, + "elapsed_seconds": 118.00034379200952, + "train_seconds": 111.17878607107559, + "eval_seconds": 2.0377439169969875 + } + ], + "best_step": 1000, + "best_validation_loss": 2.866917291863034, + "best_checkpoint_sha256": "23f0aa813704e0cfc02d0780386fe76d1ee26f70aaa674ce83a849a85fb93546", + "final_checkpoint_sha256": "b1af615e4dc2908a234f45c3c8c48c481f7c2b15294e76abd588f17da9b8763d", + "run_summary": { + "attention_backend": "auto", + "device": "cpu", + "steps": 1000, + "train_seconds": 111.17878607107559, + "train_tokens": 4096000, + "train_tokens_per_second": 36841.56073966723, + "eval_seconds": [ + 1.5413287499977741, + 1.5386290419992292, + 1.69361316600407, + 2.0377439169969875 + ], + "process_peak_rss_bytes": 465944576 + } + } +} diff --git a/experiments/records/training/20261022T040614Z_training_science-v1_sdpa_seed42_48e95f8.json b/experiments/records/training/20261022T040614Z_training_science-v1_sdpa_seed42_48e95f8.json new file mode 100644 index 0000000..0f58442 --- /dev/null +++ b/experiments/records/training/20261022T040614Z_training_science-v1_sdpa_seed42_48e95f8.json @@ -0,0 +1,129 @@ +{ + "record_version": 1, + "created_utc": "20261022T040614Z", + "experiment": "training", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "cpu", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "48e95f8ea09889e6875fb6129026662f8a699cdb", + "branch": "experiments/sdpa-training-v2", + "dirty": false + }, + "command": "python scripts/record_training_run.py artifacts/runs/sdpa-v1/sdpa-s42 --protocol science-v1 --code-commit 48e95f8 --training-command 'python scripts/train_tiny.py data/science/train.txt --validation-text data/science/validation.txt --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 --device cpu --seed 42 --attention-backend sdpa --output-dir artifacts/runs/sdpa-v1/sdpa-s42' --note 'SDPA matched experiment (v1 corpus)'" + }, + "settings": { + "dtype": "float32", + "seed": 42, + "protocol": "science-v1", + "attention_backend": "sdpa", + "model_config": { + "vocab_size": 256, + "context_length": 128, + "embedding_dim": 192, + "num_layers": 4, + "num_heads": 6, + "dropout": 0.0, + "positional_encoding": "learned", + "norm": "rmsnorm", + "activation": "swiglu", + "rope_theta": 10000.0 + }, + "tokenizer": "byte", + "optimizer": "AdamW lr=3e-4 (scripts/train_tiny.py defaults)", + "steps": 1000, + "training_command": "python scripts/train_tiny.py data/science/train.txt --validation-text data/science/validation.txt --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 --device cpu --seed 42 --attention-backend sdpa --output-dir artifacts/runs/sdpa-v1/sdpa-s42", + "training_code_commit": "48e95f8", + "note": "SDPA matched experiment (v1 corpus)", + "data": { + "train": { + "path": "data/science/train.txt", + "sha256": "19c057053f222c3084fba3693c1c295ea94f6cda1ee5fbf5f917953fcb91ef00" + }, + "validation": { + "path": "data/science/validation.txt", + "sha256": "e25d4e45706bbf4251129dbb079f3487660c453cf8b34407901350538b23e2e7" + }, + "corpus_sha256_in_checkpoint": "f6642de41672b3cac70017cd5f02312975c0bedfa32be8e79056cf36a48e943f", + "manifest_path": "data/science/manifest.json", + "manifest_sha256": "bc001b3366d192ab3c5de6b7e4e71801b0c3bba70029c5a954dbb6af661f8af9" + } + }, + "results": { + "metrics": [ + { + "step": 250, + "train_loss": 3.0327484011650085, + "validation_loss": 3.2751133565886295, + "perplexity": 26.44622316429602, + "bits_per_token": 4.724989797899623, + "validation_tokens": 112111, + "elapsed_seconds": 37.06158749999304, + "train_seconds": 35.27056428875949, + "eval_seconds": 1.7904888749908423 + }, + { + "step": 500, + "train_loss": 3.0437498688697815, + "validation_loss": 3.2796879957462988, + "perplexity": 26.56748223914296, + "bits_per_token": 4.731589607126249, + "validation_tokens": 112111, + "elapsed_seconds": 73.36073616700014, + "train_seconds": 69.86160916974768, + "eval_seconds": 1.7007023339974694 + }, + { + "step": 750, + "train_loss": 3.0581318140029907, + "validation_loss": 3.273437819028477, + "perplexity": 26.40194862624392, + "bits_per_token": 4.722572508170768, + "validation_tokens": 112111, + "elapsed_seconds": 107.89351179199002, + "train_seconds": 102.7427780017315, + "eval_seconds": 1.6509697080036858 + }, + { + "step": 1000, + "train_loss": 2.666551172733307, + "validation_loss": 2.872618086572334, + "perplexity": 17.683253938535884, + "bits_per_token": 4.1443118678658495, + "validation_tokens": 112111, + "elapsed_seconds": 142.36697695899056, + "train_seconds": 135.48510687265662, + "eval_seconds": 1.7257641250034794 + } + ], + "best_step": 1000, + "best_validation_loss": 2.872618086572334, + "best_checkpoint_sha256": "de734545979c6a4b9a5d6a21bb0299c8c91ac69c32effbfcaca23a86b7cd1fa0", + "final_checkpoint_sha256": "4c52fc4c8b32fae5ca82295eb33f4c9331ad699b3b87d3c6d533ff043412b242", + "run_summary": { + "attention_backend": "sdpa", + "device": "cpu", + "steps": 1000, + "train_seconds": 135.48510687265662, + "train_tokens": 4096000, + "train_tokens_per_second": 30232.10517042186, + "eval_seconds": [ + 1.7904888749908423, + 1.7007023339974694, + 1.6509697080036858, + 1.7257641250034794 + ], + "process_peak_rss_bytes": 470220800 + } + } +} diff --git a/experiments/records/training/20261022T040834Z_training_science-v1_sdpa_seed43_48e95f8.json b/experiments/records/training/20261022T040834Z_training_science-v1_sdpa_seed43_48e95f8.json new file mode 100644 index 0000000..8854ba1 --- /dev/null +++ b/experiments/records/training/20261022T040834Z_training_science-v1_sdpa_seed43_48e95f8.json @@ -0,0 +1,129 @@ +{ + "record_version": 1, + "created_utc": "20261022T040834Z", + "experiment": "training", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "cpu", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "48e95f8ea09889e6875fb6129026662f8a699cdb", + "branch": "experiments/sdpa-training-v2", + "dirty": false + }, + "command": "python scripts/record_training_run.py artifacts/runs/sdpa-v1/sdpa-s43 --protocol science-v1 --code-commit 48e95f8 --training-command 'python scripts/train_tiny.py data/science/train.txt --validation-text data/science/validation.txt --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 --device cpu --seed 43 --attention-backend sdpa --output-dir artifacts/runs/sdpa-v1/sdpa-s43' --note 'SDPA matched experiment (v1 corpus)'" + }, + "settings": { + "dtype": "float32", + "seed": 43, + "protocol": "science-v1", + "attention_backend": "sdpa", + "model_config": { + "vocab_size": 256, + "context_length": 128, + "embedding_dim": 192, + "num_layers": 4, + "num_heads": 6, + "dropout": 0.0, + "positional_encoding": "learned", + "norm": "rmsnorm", + "activation": "swiglu", + "rope_theta": 10000.0 + }, + "tokenizer": "byte", + "optimizer": "AdamW lr=3e-4 (scripts/train_tiny.py defaults)", + "steps": 1000, + "training_command": "python scripts/train_tiny.py data/science/train.txt --validation-text data/science/validation.txt --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 --device cpu --seed 43 --attention-backend sdpa --output-dir artifacts/runs/sdpa-v1/sdpa-s43", + "training_code_commit": "48e95f8", + "note": "SDPA matched experiment (v1 corpus)", + "data": { + "train": { + "path": "data/science/train.txt", + "sha256": "19c057053f222c3084fba3693c1c295ea94f6cda1ee5fbf5f917953fcb91ef00" + }, + "validation": { + "path": "data/science/validation.txt", + "sha256": "e25d4e45706bbf4251129dbb079f3487660c453cf8b34407901350538b23e2e7" + }, + "corpus_sha256_in_checkpoint": "f6642de41672b3cac70017cd5f02312975c0bedfa32be8e79056cf36a48e943f", + "manifest_path": "data/science/manifest.json", + "manifest_sha256": "bc001b3366d192ab3c5de6b7e4e71801b0c3bba70029c5a954dbb6af661f8af9" + } + }, + "results": { + "metrics": [ + { + "step": 250, + "train_loss": 3.1253466606140137, + "validation_loss": 3.2768519126168765, + "perplexity": 26.492241396048648, + "bits_per_token": 4.727498004059883, + "validation_tokens": 112111, + "elapsed_seconds": 35.86194645900105, + "train_seconds": 34.16292044884176, + "eval_seconds": 1.6984680419991491 + }, + { + "step": 500, + "train_loss": 3.1471184492111206, + "validation_loss": 3.2741675645485415, + "perplexity": 26.42122236158059, + "bits_per_token": 4.723625308413676, + "validation_tokens": 112111, + "elapsed_seconds": 70.68255825000233, + "train_seconds": 67.27363707474433, + "eval_seconds": 1.7036259589949623 + }, + { + "step": 750, + "train_loss": 3.0546058416366577, + "validation_loss": 3.2719499929787603, + "perplexity": 26.362696326852117, + "bits_per_token": 4.720426028907136, + "validation_tokens": 112111, + "elapsed_seconds": 104.57709999999497, + "train_seconds": 99.47416244271153, + "eval_seconds": 1.6872671670134878 + }, + { + "step": 1000, + "train_loss": 2.8518688082695007, + "validation_loss": 2.9449265964036897, + "perplexity": 19.00926698669534, + "bits_per_token": 4.248630996413617, + "validation_tokens": 112111, + "elapsed_seconds": 137.87663416699797, + "train_seconds": 131.15144002575835, + "eval_seconds": 1.6169645839981968 + } + ], + "best_step": 1000, + "best_validation_loss": 2.9449265964036897, + "best_checkpoint_sha256": "0a1357240562d1851b8db80c85e3ce4b4bfd0b00a4c6d9c41ddead928d90c916", + "final_checkpoint_sha256": "02c866b578364f3fcbddb8b7e0f53d5088c0f540ab5ff006b4f821579544c599", + "run_summary": { + "attention_backend": "sdpa", + "device": "cpu", + "steps": 1000, + "train_seconds": 131.15144002575835, + "train_tokens": 4096000, + "train_tokens_per_second": 31231.071494110467, + "eval_seconds": [ + 1.6984680419991491, + 1.7036259589949623, + 1.6872671670134878, + 1.6169645839981968 + ], + "process_peak_rss_bytes": 466665472 + } + } +} diff --git a/experiments/records/training/20261022T041103Z_training_science-v1_auto_seed43_48e95f8.json b/experiments/records/training/20261022T041103Z_training_science-v1_auto_seed43_48e95f8.json new file mode 100644 index 0000000..7aa07f0 --- /dev/null +++ b/experiments/records/training/20261022T041103Z_training_science-v1_auto_seed43_48e95f8.json @@ -0,0 +1,129 @@ +{ + "record_version": 1, + "created_utc": "20261022T041103Z", + "experiment": "training", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "cpu", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "48e95f8ea09889e6875fb6129026662f8a699cdb", + "branch": "experiments/sdpa-training-v2", + "dirty": false + }, + "command": "python scripts/record_training_run.py artifacts/runs/sdpa-v1/auto-s43 --protocol science-v1 --code-commit 48e95f8 --training-command 'python scripts/train_tiny.py data/science/train.txt --validation-text data/science/validation.txt --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 --device cpu --seed 43 --attention-backend auto --output-dir artifacts/runs/sdpa-v1/auto-s43' --note 'SDPA matched experiment (v1 corpus)'" + }, + "settings": { + "dtype": "float32", + "seed": 43, + "protocol": "science-v1", + "attention_backend": "auto", + "model_config": { + "vocab_size": 256, + "context_length": 128, + "embedding_dim": 192, + "num_layers": 4, + "num_heads": 6, + "dropout": 0.0, + "positional_encoding": "learned", + "norm": "rmsnorm", + "activation": "swiglu", + "rope_theta": 10000.0 + }, + "tokenizer": "byte", + "optimizer": "AdamW lr=3e-4 (scripts/train_tiny.py defaults)", + "steps": 1000, + "training_command": "python scripts/train_tiny.py data/science/train.txt --validation-text data/science/validation.txt --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 --device cpu --seed 43 --attention-backend auto --output-dir artifacts/runs/sdpa-v1/auto-s43", + "training_code_commit": "48e95f8", + "note": "SDPA matched experiment (v1 corpus)", + "data": { + "train": { + "path": "data/science/train.txt", + "sha256": "19c057053f222c3084fba3693c1c295ea94f6cda1ee5fbf5f917953fcb91ef00" + }, + "validation": { + "path": "data/science/validation.txt", + "sha256": "e25d4e45706bbf4251129dbb079f3487660c453cf8b34407901350538b23e2e7" + }, + "corpus_sha256_in_checkpoint": "f6642de41672b3cac70017cd5f02312975c0bedfa32be8e79056cf36a48e943f", + "manifest_path": "data/science/manifest.json", + "manifest_sha256": "bc001b3366d192ab3c5de6b7e4e71801b0c3bba70029c5a954dbb6af661f8af9" + } + }, + "results": { + "metrics": [ + { + "step": 250, + "train_loss": 3.125346601009369, + "validation_loss": 3.27685190798933, + "perplexity": 26.49224127345457, + "bits_per_token": 4.727497997383744, + "validation_tokens": 112111, + "elapsed_seconds": 35.971077292007976, + "train_seconds": 34.02679004802485, + "eval_seconds": 1.9438253749976866 + }, + { + "step": 500, + "train_loss": 3.1471182703971863, + "validation_loss": 3.2741675599209956, + "perplexity": 26.42122223931517, + "bits_per_token": 4.723625301737538, + "validation_tokens": 112111, + "elapsed_seconds": 72.10287125001196, + "train_seconds": 68.14442900904396, + "eval_seconds": 2.0082519999996293 + }, + { + "step": 750, + "train_loss": 3.054605782032013, + "validation_loss": 3.271949987806797, + "perplexity": 26.36269619050522, + "bits_per_token": 4.72042602144557, + "validation_tokens": 112111, + "elapsed_seconds": 110.50238375000481, + "train_seconds": 104.46415100306331, + "eval_seconds": 2.0738366250006948 + }, + { + "step": 1000, + "train_loss": 2.851951539516449, + "validation_loss": 2.9438789420007883, + "perplexity": 18.989362272891913, + "bits_per_token": 4.247119550601986, + "validation_tokens": 112111, + "elapsed_seconds": 147.99201750000066, + "train_seconds": 139.8997420352389, + "eval_seconds": 2.048635250001098 + } + ], + "best_step": 1000, + "best_validation_loss": 2.9438789420007883, + "best_checkpoint_sha256": "e2285dfc79e7aa1d430cac7720bc5997139210fef876055c23da375aab24b04c", + "final_checkpoint_sha256": "c9a217ad9a778171738c115cd23a3b271957fb570f7990e616d5444e7baed976", + "run_summary": { + "attention_backend": "auto", + "device": "cpu", + "steps": 1000, + "train_seconds": 139.8997420352389, + "train_tokens": 4096000, + "train_tokens_per_second": 29278.10974067609, + "eval_seconds": [ + 1.9438253749976866, + 2.0082519999996293, + 2.0738366250006948, + 2.048635250001098 + ], + "process_peak_rss_bytes": 470466560 + } + } +} diff --git a/experiments/records/training/20261022T041335Z_training_science-v1_auto_seed44_48e95f8.json b/experiments/records/training/20261022T041335Z_training_science-v1_auto_seed44_48e95f8.json new file mode 100644 index 0000000..11718ba --- /dev/null +++ b/experiments/records/training/20261022T041335Z_training_science-v1_auto_seed44_48e95f8.json @@ -0,0 +1,129 @@ +{ + "record_version": 1, + "created_utc": "20261022T041335Z", + "experiment": "training", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "cpu", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "48e95f8ea09889e6875fb6129026662f8a699cdb", + "branch": "experiments/sdpa-training-v2", + "dirty": false + }, + "command": "python scripts/record_training_run.py artifacts/runs/sdpa-v1/auto-s44 --protocol science-v1 --code-commit 48e95f8 --training-command 'python scripts/train_tiny.py data/science/train.txt --validation-text data/science/validation.txt --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 --device cpu --seed 44 --attention-backend auto --output-dir artifacts/runs/sdpa-v1/auto-s44' --note 'SDPA matched experiment (v1 corpus)'" + }, + "settings": { + "dtype": "float32", + "seed": 44, + "protocol": "science-v1", + "attention_backend": "auto", + "model_config": { + "vocab_size": 256, + "context_length": 128, + "embedding_dim": 192, + "num_layers": 4, + "num_heads": 6, + "dropout": 0.0, + "positional_encoding": "learned", + "norm": "rmsnorm", + "activation": "swiglu", + "rope_theta": 10000.0 + }, + "tokenizer": "byte", + "optimizer": "AdamW lr=3e-4 (scripts/train_tiny.py defaults)", + "steps": 1000, + "training_command": "python scripts/train_tiny.py data/science/train.txt --validation-text data/science/validation.txt --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 --device cpu --seed 44 --attention-backend auto --output-dir artifacts/runs/sdpa-v1/auto-s44", + "training_code_commit": "48e95f8", + "note": "SDPA matched experiment (v1 corpus)", + "data": { + "train": { + "path": "data/science/train.txt", + "sha256": "19c057053f222c3084fba3693c1c295ea94f6cda1ee5fbf5f917953fcb91ef00" + }, + "validation": { + "path": "data/science/validation.txt", + "sha256": "e25d4e45706bbf4251129dbb079f3487660c453cf8b34407901350538b23e2e7" + }, + "corpus_sha256_in_checkpoint": "f6642de41672b3cac70017cd5f02312975c0bedfa32be8e79056cf36a48e943f", + "manifest_path": "data/science/manifest.json", + "manifest_sha256": "bc001b3366d192ab3c5de6b7e4e71801b0c3bba70029c5a954dbb6af661f8af9" + } + }, + "results": { + "metrics": [ + { + "step": 250, + "train_loss": 3.0683940052986145, + "validation_loss": 3.2747702210461864, + "perplexity": 26.437150081902516, + "bits_per_token": 4.724494757954187, + "validation_tokens": 112111, + "elapsed_seconds": 37.51592812499439, + "train_seconds": 35.48828979601967, + "eval_seconds": 2.0271226249897154 + }, + { + "step": 500, + "train_loss": 3.061283588409424, + "validation_loss": 3.275096681362127, + "perplexity": 26.445782171211466, + "bits_per_token": 4.724965740633043, + "validation_tokens": 112111, + "elapsed_seconds": 74.5914915829926, + "train_seconds": 70.57057427109976, + "eval_seconds": 1.987371292008902 + }, + { + "step": 750, + "train_loss": 3.0149829387664795, + "validation_loss": 3.270937874591696, + "perplexity": 26.336027655370078, + "bits_per_token": 4.718965850729326, + "validation_tokens": 112111, + "elapsed_seconds": 111.55503599999065, + "train_seconds": 105.45970522418793, + "eval_seconds": 2.073784791995422 + }, + { + "step": 1000, + "train_loss": 2.787301540374756, + "validation_loss": 3.0477406168084813, + "perplexity": 21.067690622638423, + "bits_per_token": 4.396960273785466, + "validation_tokens": 112111, + "elapsed_seconds": 150.18698241699894, + "train_seconds": 142.06854248027958, + "eval_seconds": 2.016462874991703 + } + ], + "best_step": 1000, + "best_validation_loss": 3.0477406168084813, + "best_checkpoint_sha256": "d4fe3800f75cd1f3b7ef68bfa463dc48f503585e5cd038da8f22b7f67014f794", + "final_checkpoint_sha256": "becaa1b609e62d9591a2cce4cae60e0d158c501db10a7e2edcc5209a1a4ed5ef", + "run_summary": { + "attention_backend": "auto", + "device": "cpu", + "steps": 1000, + "train_seconds": 142.06854248027958, + "train_tokens": 4096000, + "train_tokens_per_second": 28831.15381132711, + "eval_seconds": [ + 2.0271226249897154, + 1.987371292008902, + 2.073784791995422, + 2.016462874991703 + ], + "process_peak_rss_bytes": 472842240 + } + } +} diff --git a/experiments/records/training/20261022T041553Z_training_science-v1_sdpa_seed44_48e95f8.json b/experiments/records/training/20261022T041553Z_training_science-v1_sdpa_seed44_48e95f8.json new file mode 100644 index 0000000..51d95cb --- /dev/null +++ b/experiments/records/training/20261022T041553Z_training_science-v1_sdpa_seed44_48e95f8.json @@ -0,0 +1,129 @@ +{ + "record_version": 1, + "created_utc": "20261022T041553Z", + "experiment": "training", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "cpu", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "48e95f8ea09889e6875fb6129026662f8a699cdb", + "branch": "experiments/sdpa-training-v2", + "dirty": false + }, + "command": "python scripts/record_training_run.py artifacts/runs/sdpa-v1/sdpa-s44 --protocol science-v1 --code-commit 48e95f8 --training-command 'python scripts/train_tiny.py data/science/train.txt --validation-text data/science/validation.txt --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 --device cpu --seed 44 --attention-backend sdpa --output-dir artifacts/runs/sdpa-v1/sdpa-s44' --note 'SDPA matched experiment (v1 corpus)'" + }, + "settings": { + "dtype": "float32", + "seed": 44, + "protocol": "science-v1", + "attention_backend": "sdpa", + "model_config": { + "vocab_size": 256, + "context_length": 128, + "embedding_dim": 192, + "num_layers": 4, + "num_heads": 6, + "dropout": 0.0, + "positional_encoding": "learned", + "norm": "rmsnorm", + "activation": "swiglu", + "rope_theta": 10000.0 + }, + "tokenizer": "byte", + "optimizer": "AdamW lr=3e-4 (scripts/train_tiny.py defaults)", + "steps": 1000, + "training_command": "python scripts/train_tiny.py data/science/train.txt --validation-text data/science/validation.txt --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 --device cpu --seed 44 --attention-backend sdpa --output-dir artifacts/runs/sdpa-v1/sdpa-s44", + "training_code_commit": "48e95f8", + "note": "SDPA matched experiment (v1 corpus)", + "data": { + "train": { + "path": "data/science/train.txt", + "sha256": "19c057053f222c3084fba3693c1c295ea94f6cda1ee5fbf5f917953fcb91ef00" + }, + "validation": { + "path": "data/science/validation.txt", + "sha256": "e25d4e45706bbf4251129dbb079f3487660c453cf8b34407901350538b23e2e7" + }, + "corpus_sha256_in_checkpoint": "f6642de41672b3cac70017cd5f02312975c0bedfa32be8e79056cf36a48e943f", + "manifest_path": "data/science/manifest.json", + "manifest_sha256": "bc001b3366d192ab3c5de6b7e4e71801b0c3bba70029c5a954dbb6af661f8af9" + } + }, + "results": { + "metrics": [ + { + "step": 250, + "train_loss": 3.0683940052986145, + "validation_loss": 3.2747702109744683, + "perplexity": 26.437149815634996, + "bits_per_token": 4.72449474342377, + "validation_tokens": 112111, + "elapsed_seconds": 34.008635500009404, + "train_seconds": 32.33172358122829, + "eval_seconds": 1.67641724999703 + }, + { + "step": 500, + "train_loss": 3.061283588409424, + "validation_loss": 3.275096674829121, + "perplexity": 26.44578199844101, + "bits_per_token": 4.724965731207907, + "validation_tokens": 112111, + "elapsed_seconds": 67.7609861670062, + "train_seconds": 64.43372592516243, + "eval_seconds": 1.6444599999958882 + }, + { + "step": 750, + "train_loss": 3.0149828791618347, + "validation_loss": 3.2709378822135364, + "perplexity": 26.33602785609908, + "bits_per_token": 4.718965861725318, + "validation_tokens": 112111, + "elapsed_seconds": 102.02934879199893, + "train_seconds": 97.07046017116227, + "eval_seconds": 1.631016291998094 + }, + { + "step": 1000, + "train_loss": 2.7379232048988342, + "validation_loss": 3.016298744861834, + "perplexity": 20.415588370621776, + "bits_per_token": 4.3515992410517725, + "validation_tokens": 112111, + "elapsed_seconds": 136.33061937500315, + "train_seconds": 129.70830175618175, + "eval_seconds": 1.6582876669999678 + } + ], + "best_step": 1000, + "best_validation_loss": 3.016298744861834, + "best_checkpoint_sha256": "e7544665b394924f64facd47c731cd2950caa35d0ced5aaa66c77cdeb4034e47", + "final_checkpoint_sha256": "5de158ff5c205dcf1e06ecce60ad189878d8f0899a7c81e9b4b19145e7628244", + "run_summary": { + "attention_backend": "sdpa", + "device": "cpu", + "steps": 1000, + "train_seconds": 129.70830175618175, + "train_tokens": 4096000, + "train_tokens_per_second": 31578.549287457532, + "eval_seconds": [ + 1.67641724999703, + 1.6444599999958882, + 1.631016291998094, + 1.6582876669999678 + ], + "process_peak_rss_bytes": 463060992 + } + } +} diff --git a/experiments/records/training/20261022T041654Z_training_science-v1_auto_seed42_48e95f8.json b/experiments/records/training/20261022T041654Z_training_science-v1_auto_seed42_48e95f8.json new file mode 100644 index 0000000..9fdd722 --- /dev/null +++ b/experiments/records/training/20261022T041654Z_training_science-v1_auto_seed42_48e95f8.json @@ -0,0 +1,129 @@ +{ + "record_version": 1, + "created_utc": "20261022T041654Z", + "experiment": "training", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "cpu", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "48e95f8ea09889e6875fb6129026662f8a699cdb", + "branch": "experiments/sdpa-training-v2", + "dirty": false + }, + "command": "python scripts/record_training_run.py artifacts/runs/sdpa-v1/auto-s42 --protocol science-v1 --code-commit 48e95f8 --training-command 'python scripts/train_tiny.py data/science/train.txt --validation-text data/science/validation.txt --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 --device cpu --seed 42 --attention-backend auto --output-dir artifacts/runs/sdpa-v1/auto-s42' --note 'SDPA matched experiment (v1 corpus); re-recorded from a clean tree (first record was written while unrelated doc edits were uncommitted)'" + }, + "settings": { + "dtype": "float32", + "seed": 42, + "protocol": "science-v1", + "attention_backend": "auto", + "model_config": { + "vocab_size": 256, + "context_length": 128, + "embedding_dim": 192, + "num_layers": 4, + "num_heads": 6, + "dropout": 0.0, + "positional_encoding": "learned", + "norm": "rmsnorm", + "activation": "swiglu", + "rope_theta": 10000.0 + }, + "tokenizer": "byte", + "optimizer": "AdamW lr=3e-4 (scripts/train_tiny.py defaults)", + "steps": 1000, + "training_command": "python scripts/train_tiny.py data/science/train.txt --validation-text data/science/validation.txt --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 --device cpu --seed 42 --attention-backend auto --output-dir artifacts/runs/sdpa-v1/auto-s42", + "training_code_commit": "48e95f8", + "note": "SDPA matched experiment (v1 corpus); re-recorded from a clean tree (first record was written while unrelated doc edits were uncommitted)", + "data": { + "train": { + "path": "data/science/train.txt", + "sha256": "19c057053f222c3084fba3693c1c295ea94f6cda1ee5fbf5f917953fcb91ef00" + }, + "validation": { + "path": "data/science/validation.txt", + "sha256": "e25d4e45706bbf4251129dbb079f3487660c453cf8b34407901350538b23e2e7" + }, + "corpus_sha256_in_checkpoint": "f6642de41672b3cac70017cd5f02312975c0bedfa32be8e79056cf36a48e943f", + "manifest_path": "data/science/manifest.json", + "manifest_sha256": "bc001b3366d192ab3c5de6b7e4e71801b0c3bba70029c5a954dbb6af661f8af9" + } + }, + "results": { + "metrics": [ + { + "step": 250, + "train_loss": 3.0327484011650085, + "validation_loss": 3.275113352777709, + "perplexity": 26.446223063511567, + "bits_per_token": 4.724989792401627, + "validation_tokens": 112111, + "elapsed_seconds": 26.916708292003023, + "train_seconds": 25.374983163928846, + "eval_seconds": 1.5413287499977741 + }, + { + "step": 500, + "train_loss": 3.0437499284744263, + "validation_loss": 3.279687989213292, + "perplexity": 26.567482065577426, + "bits_per_token": 4.731589597701113, + "validation_tokens": 112111, + "elapsed_seconds": 54.152566375007154, + "train_seconds": 51.067428036025376, + "eval_seconds": 1.5386290419992292 + }, + { + "step": 750, + "train_loss": 3.058131754398346, + "validation_loss": 3.273437816850808, + "perplexity": 26.401948568749216, + "bits_per_token": 4.7225725050290555, + "validation_tokens": 112111, + "elapsed_seconds": 83.14721091699903, + "train_seconds": 78.36790991391172, + "eval_seconds": 1.69361316600407 + }, + { + "step": 1000, + "train_loss": 2.6655086278915405, + "validation_loss": 2.866917291863034, + "perplexity": 17.582732137354572, + "bits_per_token": 4.136087359609617, + "validation_tokens": 112111, + "elapsed_seconds": 118.00034379200952, + "train_seconds": 111.17878607107559, + "eval_seconds": 2.0377439169969875 + } + ], + "best_step": 1000, + "best_validation_loss": 2.866917291863034, + "best_checkpoint_sha256": "23f0aa813704e0cfc02d0780386fe76d1ee26f70aaa674ce83a849a85fb93546", + "final_checkpoint_sha256": "b1af615e4dc2908a234f45c3c8c48c481f7c2b15294e76abd588f17da9b8763d", + "run_summary": { + "attention_backend": "auto", + "device": "cpu", + "steps": 1000, + "train_seconds": 111.17878607107559, + "train_tokens": 4096000, + "train_tokens_per_second": 36841.56073966723, + "eval_seconds": [ + 1.5413287499977741, + 1.5386290419992292, + 1.69361316600407, + 2.0377439169969875 + ], + "process_peak_rss_bytes": 465944576 + } + } +} diff --git a/experiments/records/training/20261022T042403Z_training_science-v2_auto_seed42_897db37.json b/experiments/records/training/20261022T042403Z_training_science-v2_auto_seed42_897db37.json new file mode 100644 index 0000000..433eea9 --- /dev/null +++ b/experiments/records/training/20261022T042403Z_training_science-v2_auto_seed42_897db37.json @@ -0,0 +1,129 @@ +{ + "record_version": 1, + "created_utc": "20261022T042403Z", + "experiment": "training", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "cpu", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "897db37958b0c7595f097a26eaa60549b0f8831e", + "branch": "experiments/sdpa-training-v2", + "dirty": false + }, + "command": "python scripts/record_training_run.py artifacts/runs/v2/auto-s42 --protocol science-v2 --code-commit 897db37 --train-text data/science-v2/train.txt --validation-text data/science-v2/validation.txt --manifest data/science-v2/manifest.json --training-command 'python scripts/train_tiny.py data/science-v2/train.txt --validation-text data/science-v2/validation.txt --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 --device cpu --seed 42 --attention-backend auto --output-dir artifacts/runs/v2/auto-s42' --note 'science-v2 protocol'" + }, + "settings": { + "dtype": "float32", + "seed": 42, + "protocol": "science-v2", + "attention_backend": "auto", + "model_config": { + "vocab_size": 256, + "context_length": 128, + "embedding_dim": 192, + "num_layers": 4, + "num_heads": 6, + "dropout": 0.0, + "positional_encoding": "learned", + "norm": "rmsnorm", + "activation": "swiglu", + "rope_theta": 10000.0 + }, + "tokenizer": "byte", + "optimizer": "AdamW lr=3e-4 (scripts/train_tiny.py defaults)", + "steps": 1000, + "training_command": "python scripts/train_tiny.py data/science-v2/train.txt --validation-text data/science-v2/validation.txt --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 --device cpu --seed 42 --attention-backend auto --output-dir artifacts/runs/v2/auto-s42", + "training_code_commit": "897db37", + "note": "science-v2 protocol", + "data": { + "train": { + "path": "data/science-v2/train.txt", + "sha256": "65182db3ac268ec990f79a32fd9151c0de332fb8c411138af9ef965c9f80265e" + }, + "validation": { + "path": "data/science-v2/validation.txt", + "sha256": "6457fe8eaf9011ef75600da24ed0925cfb5e9464fa1c9e453785eecfc9592b17" + }, + "corpus_sha256_in_checkpoint": "84e98dc220b5b274380238ca9c7b45094b7b62f42ca486e70e65d8ac51449988", + "manifest_path": "data/science-v2/manifest.json", + "manifest_sha256": "9a34a5fca7532fccbe7d6be0e59a99523ebf7796cab09cb3ebd66ab1f26025c0" + } + }, + "results": { + "metrics": [ + { + "step": 250, + "train_loss": 3.039337158203125, + "validation_loss": 3.039427177043993, + "perplexity": 20.89327166099037, + "bits_per_token": 4.3849665154645106, + "validation_tokens": 105665, + "elapsed_seconds": 27.196134875004645, + "train_seconds": 25.72327775596932, + "eval_seconds": 1.472483249992365 + }, + { + "step": 500, + "train_loss": 3.0974019169807434, + "validation_loss": 3.0358155694736904, + "perplexity": 20.817949461876797, + "bits_per_token": 4.379756067133197, + "validation_tokens": 105665, + "elapsed_seconds": 55.14430854099919, + "train_seconds": 52.18055421093595, + "eval_seconds": 1.485997416995815 + }, + { + "step": 750, + "train_loss": 3.0544053316116333, + "validation_loss": 3.034946330276321, + "perplexity": 20.799861546697016, + "bits_per_token": 4.378502020053807, + "validation_tokens": 105665, + "elapsed_seconds": 84.23251524999796, + "train_seconds": 79.6536416309973, + "eval_seconds": 1.610307750001084 + }, + { + "step": 1000, + "train_loss": 2.7223730087280273, + "validation_loss": 2.656100203500967, + "perplexity": 14.240645051253535, + "bits_per_token": 3.831942591695012, + "validation_tokens": 105665, + "elapsed_seconds": 120.700620040996, + "train_seconds": 114.02553255201201, + "eval_seconds": 2.091685250008595 + } + ], + "best_step": 1000, + "best_validation_loss": 2.656100203500967, + "best_checkpoint_sha256": "1cb3fbca57f10b72d33c945325dfc9ed1335cca02fc2fe07150fae5ba40a4b8e", + "final_checkpoint_sha256": "f4e18218a9da9670cd647c4baf264ddada9ca50fdb86234721e59292ecaae880", + "run_summary": { + "attention_backend": "auto", + "device": "cpu", + "steps": 1000, + "train_seconds": 114.02553255201201, + "train_tokens": 4096000, + "train_tokens_per_second": 35921.77916934206, + "eval_seconds": [ + 1.472483249992365, + 1.485997416995815, + 1.610307750001084, + 2.091685250008595 + ], + "process_peak_rss_bytes": 469630976 + } + } +} diff --git a/experiments/records/training/20261022T042630Z_training_science-v2_sdpa_seed42_897db37.json b/experiments/records/training/20261022T042630Z_training_science-v2_sdpa_seed42_897db37.json new file mode 100644 index 0000000..e70c2ed --- /dev/null +++ b/experiments/records/training/20261022T042630Z_training_science-v2_sdpa_seed42_897db37.json @@ -0,0 +1,129 @@ +{ + "record_version": 1, + "created_utc": "20261022T042630Z", + "experiment": "training", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "cpu", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "897db37958b0c7595f097a26eaa60549b0f8831e", + "branch": "experiments/sdpa-training-v2", + "dirty": false + }, + "command": "python scripts/record_training_run.py artifacts/runs/v2/sdpa-s42 --protocol science-v2 --code-commit 897db37 --train-text data/science-v2/train.txt --validation-text data/science-v2/validation.txt --manifest data/science-v2/manifest.json --training-command 'python scripts/train_tiny.py data/science-v2/train.txt --validation-text data/science-v2/validation.txt --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 --device cpu --seed 42 --attention-backend sdpa --output-dir artifacts/runs/v2/sdpa-s42' --note 'science-v2 protocol'" + }, + "settings": { + "dtype": "float32", + "seed": 42, + "protocol": "science-v2", + "attention_backend": "sdpa", + "model_config": { + "vocab_size": 256, + "context_length": 128, + "embedding_dim": 192, + "num_layers": 4, + "num_heads": 6, + "dropout": 0.0, + "positional_encoding": "learned", + "norm": "rmsnorm", + "activation": "swiglu", + "rope_theta": 10000.0 + }, + "tokenizer": "byte", + "optimizer": "AdamW lr=3e-4 (scripts/train_tiny.py defaults)", + "steps": 1000, + "training_command": "python scripts/train_tiny.py data/science-v2/train.txt --validation-text data/science-v2/validation.txt --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 --device cpu --seed 42 --attention-backend sdpa --output-dir artifacts/runs/v2/sdpa-s42", + "training_code_commit": "897db37", + "note": "science-v2 protocol", + "data": { + "train": { + "path": "data/science-v2/train.txt", + "sha256": "65182db3ac268ec990f79a32fd9151c0de332fb8c411138af9ef965c9f80265e" + }, + "validation": { + "path": "data/science-v2/validation.txt", + "sha256": "6457fe8eaf9011ef75600da24ed0925cfb5e9464fa1c9e453785eecfc9592b17" + }, + "corpus_sha256_in_checkpoint": "84e98dc220b5b274380238ca9c7b45094b7b62f42ca486e70e65d8ac51449988", + "manifest_path": "data/science-v2/manifest.json", + "manifest_sha256": "9a34a5fca7532fccbe7d6be0e59a99523ebf7796cab09cb3ebd66ab1f26025c0" + } + }, + "results": { + "metrics": [ + { + "step": 250, + "train_loss": 3.0393370985984802, + "validation_loss": 3.0394271731449978, + "perplexity": 20.8932715795276, + "bits_per_token": 4.384966509839449, + "validation_tokens": 105665, + "elapsed_seconds": 39.141335666005034, + "train_seconds": 37.37626767599431, + "eval_seconds": 1.7645393749990035 + }, + { + "step": 500, + "train_loss": 3.097401797771454, + "validation_loss": 3.0358155642750306, + "perplexity": 20.81794935365136, + "bits_per_token": 4.379756059633117, + "validation_tokens": 105665, + "elapsed_seconds": 76.83559637499275, + "train_seconds": 73.36852867974085, + "eval_seconds": 1.6953102500119712 + }, + { + "step": 750, + "train_loss": 3.0544053316116333, + "validation_loss": 3.0349463374966823, + "perplexity": 20.799861696879532, + "bits_per_token": 4.378502030470586, + "validation_tokens": 105665, + "elapsed_seconds": 111.84500058299454, + "train_seconds": 106.79540838992398, + "eval_seconds": 1.576022000008379 + }, + { + "step": 1000, + "train_loss": 2.7209399938583374, + "validation_loss": 2.6489919601914806, + "perplexity": 14.13977799982672, + "bits_per_token": 3.8216875643229837, + "validation_tokens": 105665, + "elapsed_seconds": 144.8335367499967, + "train_seconds": 138.21999243508617, + "eval_seconds": 1.5587145000026794 + } + ], + "best_step": 1000, + "best_validation_loss": 2.6489919601914806, + "best_checkpoint_sha256": "6e8b0471016093200fa1a7045e41b4dcb1be07bb0bb742444bd2c19eaa9f2cc2", + "final_checkpoint_sha256": "264b42c669549c77acc92d05c122beedd9ec87b942a077b073fd7758fb3abef9", + "run_summary": { + "attention_backend": "sdpa", + "device": "cpu", + "steps": 1000, + "train_seconds": 138.21999243508617, + "train_tokens": 4096000, + "train_tokens_per_second": 29633.91856589524, + "eval_seconds": [ + 1.7645393749990035, + 1.6953102500119712, + 1.576022000008379, + 1.5587145000026794 + ], + "process_peak_rss_bytes": 467435520 + } + } +} diff --git a/experiments/records/training/20261022T042859Z_training_science-v2_auto_seed43_897db37.json b/experiments/records/training/20261022T042859Z_training_science-v2_auto_seed43_897db37.json new file mode 100644 index 0000000..e1f4110 --- /dev/null +++ b/experiments/records/training/20261022T042859Z_training_science-v2_auto_seed43_897db37.json @@ -0,0 +1,129 @@ +{ + "record_version": 1, + "created_utc": "20261022T042859Z", + "experiment": "training", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "cpu", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "897db37958b0c7595f097a26eaa60549b0f8831e", + "branch": "experiments/sdpa-training-v2", + "dirty": false + }, + "command": "python scripts/record_training_run.py artifacts/runs/v2/auto-s43 --protocol science-v2 --code-commit 897db37 --train-text data/science-v2/train.txt --validation-text data/science-v2/validation.txt --manifest data/science-v2/manifest.json --training-command 'python scripts/train_tiny.py data/science-v2/train.txt --validation-text data/science-v2/validation.txt --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 --device cpu --seed 43 --attention-backend auto --output-dir artifacts/runs/v2/auto-s43' --note 'science-v2 protocol'" + }, + "settings": { + "dtype": "float32", + "seed": 43, + "protocol": "science-v2", + "attention_backend": "auto", + "model_config": { + "vocab_size": 256, + "context_length": 128, + "embedding_dim": 192, + "num_layers": 4, + "num_heads": 6, + "dropout": 0.0, + "positional_encoding": "learned", + "norm": "rmsnorm", + "activation": "swiglu", + "rope_theta": 10000.0 + }, + "tokenizer": "byte", + "optimizer": "AdamW lr=3e-4 (scripts/train_tiny.py defaults)", + "steps": 1000, + "training_command": "python scripts/train_tiny.py data/science-v2/train.txt --validation-text data/science-v2/validation.txt --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 --device cpu --seed 43 --attention-backend auto --output-dir artifacts/runs/v2/auto-s43", + "training_code_commit": "897db37", + "note": "science-v2 protocol", + "data": { + "train": { + "path": "data/science-v2/train.txt", + "sha256": "65182db3ac268ec990f79a32fd9151c0de332fb8c411138af9ef965c9f80265e" + }, + "validation": { + "path": "data/science-v2/validation.txt", + "sha256": "6457fe8eaf9011ef75600da24ed0925cfb5e9464fa1c9e453785eecfc9592b17" + }, + "corpus_sha256_in_checkpoint": "84e98dc220b5b274380238ca9c7b45094b7b62f42ca486e70e65d8ac51449988", + "manifest_path": "data/science-v2/manifest.json", + "manifest_sha256": "9a34a5fca7532fccbe7d6be0e59a99523ebf7796cab09cb3ebd66ab1f26025c0" + } + }, + "results": { + "metrics": [ + { + "step": 250, + "train_loss": 3.042278289794922, + "validation_loss": 3.0388119267553173, + "perplexity": 20.88042102315425, + "bits_per_token": 4.384078896924132, + "validation_tokens": 105665, + "elapsed_seconds": 35.84622049999598, + "train_seconds": 34.0019872058474, + "eval_seconds": 1.8437756249913946 + }, + { + "step": 500, + "train_loss": 3.043114483356476, + "validation_loss": 3.0366123332859063, + "perplexity": 20.83454306036411, + "bits_per_token": 4.380905554333841, + "validation_tokens": 105665, + "elapsed_seconds": 71.57403299999714, + "train_seconds": 67.89995957478823, + "eval_seconds": 1.823977208987344 + }, + { + "step": 750, + "train_loss": 3.017816126346588, + "validation_loss": 3.0346339543510057, + "perplexity": 20.793365185406195, + "bits_per_token": 4.378051356855461, + "validation_tokens": 105665, + "elapsed_seconds": 109.29384654099704, + "train_seconds": 103.68952695182816, + "eval_seconds": 1.9238174159982009 + }, + { + "step": 1000, + "train_loss": 2.9055304527282715, + "validation_loss": 2.9176224386664407, + "perplexity": 18.49725677517611, + "bits_per_token": 4.209239423450438, + "validation_tokens": 105665, + "elapsed_seconds": 146.84813966599177, + "train_seconds": 139.30257590580732, + "eval_seconds": 1.9362271669961046 + } + ], + "best_step": 1000, + "best_validation_loss": 2.9176224386664407, + "best_checkpoint_sha256": "ad66af927334d8991901682c91a0c1ca539707681a0ba890d7145101ba3c041c", + "final_checkpoint_sha256": "46117ebc02af5b2fe2a5cf6b6301435955f8a865aeef7f30a992e5c33694543c", + "run_summary": { + "attention_backend": "auto", + "device": "cpu", + "steps": 1000, + "train_seconds": 139.30257590580732, + "train_tokens": 4096000, + "train_tokens_per_second": 29403.61995006902, + "eval_seconds": [ + 1.8437756249913946, + 1.823977208987344, + 1.9238174159982009, + 1.9362271669961046 + ], + "process_peak_rss_bytes": 468566016 + } + } +} diff --git a/experiments/records/training/20261022T043119Z_training_science-v2_sdpa_seed43_897db37.json b/experiments/records/training/20261022T043119Z_training_science-v2_sdpa_seed43_897db37.json new file mode 100644 index 0000000..8141fbe --- /dev/null +++ b/experiments/records/training/20261022T043119Z_training_science-v2_sdpa_seed43_897db37.json @@ -0,0 +1,129 @@ +{ + "record_version": 1, + "created_utc": "20261022T043119Z", + "experiment": "training", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "cpu", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "897db37958b0c7595f097a26eaa60549b0f8831e", + "branch": "experiments/sdpa-training-v2", + "dirty": false + }, + "command": "python scripts/record_training_run.py artifacts/runs/v2/sdpa-s43 --protocol science-v2 --code-commit 897db37 --train-text data/science-v2/train.txt --validation-text data/science-v2/validation.txt --manifest data/science-v2/manifest.json --training-command 'python scripts/train_tiny.py data/science-v2/train.txt --validation-text data/science-v2/validation.txt --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 --device cpu --seed 43 --attention-backend sdpa --output-dir artifacts/runs/v2/sdpa-s43' --note 'science-v2 protocol'" + }, + "settings": { + "dtype": "float32", + "seed": 43, + "protocol": "science-v2", + "attention_backend": "sdpa", + "model_config": { + "vocab_size": 256, + "context_length": 128, + "embedding_dim": 192, + "num_layers": 4, + "num_heads": 6, + "dropout": 0.0, + "positional_encoding": "learned", + "norm": "rmsnorm", + "activation": "swiglu", + "rope_theta": 10000.0 + }, + "tokenizer": "byte", + "optimizer": "AdamW lr=3e-4 (scripts/train_tiny.py defaults)", + "steps": 1000, + "training_command": "python scripts/train_tiny.py data/science-v2/train.txt --validation-text data/science-v2/validation.txt --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 --device cpu --seed 43 --attention-backend sdpa --output-dir artifacts/runs/v2/sdpa-s43", + "training_code_commit": "897db37", + "note": "science-v2 protocol", + "data": { + "train": { + "path": "data/science-v2/train.txt", + "sha256": "65182db3ac268ec990f79a32fd9151c0de332fb8c411138af9ef965c9f80265e" + }, + "validation": { + "path": "data/science-v2/validation.txt", + "sha256": "6457fe8eaf9011ef75600da24ed0925cfb5e9464fa1c9e453785eecfc9592b17" + }, + "corpus_sha256_in_checkpoint": "84e98dc220b5b274380238ca9c7b45094b7b62f42ca486e70e65d8ac51449988", + "manifest_path": "data/science-v2/manifest.json", + "manifest_sha256": "9a34a5fca7532fccbe7d6be0e59a99523ebf7796cab09cb3ebd66ab1f26025c0" + } + }, + "results": { + "metrics": [ + { + "step": 250, + "train_loss": 3.042278230190277, + "validation_loss": 3.03881192921024, + "perplexity": 20.880421074414063, + "bits_per_token": 4.384078900465837, + "validation_tokens": 105665, + "elapsed_seconds": 34.683118832996115, + "train_seconds": 33.07528708598693, + "eval_seconds": 1.6073278330004541 + }, + { + "step": 500, + "train_loss": 3.043114483356476, + "validation_loss": 3.036612326065545, + "perplexity": 20.834542909931184, + "bits_per_token": 4.380905543917062, + "validation_tokens": 105665, + "elapsed_seconds": 69.318063291008, + "train_seconds": 66.10483992692025, + "eval_seconds": 1.5991290420060977 + }, + { + "step": 750, + "train_loss": 3.0178160667419434, + "validation_loss": 3.0346339500187893, + "perplexity": 20.79336509532484, + "bits_per_token": 4.378051350605394, + "validation_tokens": 105665, + "elapsed_seconds": 103.68754408300447, + "train_seconds": 98.87572376785101, + "eval_seconds": 1.5932842919864925 + }, + { + "step": 1000, + "train_loss": 2.9055142402648926, + "validation_loss": 2.917622557513584, + "perplexity": 18.497258973522364, + "bits_per_token": 4.209239594910622, + "validation_tokens": 105665, + "elapsed_seconds": 138.5718514580076, + "train_seconds": 132.18597715096257, + "eval_seconds": 1.5686826249875594 + } + ], + "best_step": 1000, + "best_validation_loss": 2.917622557513584, + "best_checkpoint_sha256": "3451e8577c0a5cd8f5b5cadd7aec3f7fd2d9b72fde24db01e6520f9b083de4a4", + "final_checkpoint_sha256": "59dd70bd83b42f3f4075d7f44fbbe162a908c10d8f07c0a831cc13d080c53340", + "run_summary": { + "attention_backend": "sdpa", + "device": "cpu", + "steps": 1000, + "train_seconds": 132.18597715096257, + "train_tokens": 4096000, + "train_tokens_per_second": 30986.645393725663, + "eval_seconds": [ + 1.6073278330004541, + 1.5991290420060977, + 1.5932842919864925, + 1.5686826249875594 + ], + "process_peak_rss_bytes": 467763200 + } + } +} diff --git a/experiments/records/training/20261022T043349Z_training_science-v2_auto_seed44_897db37.json b/experiments/records/training/20261022T043349Z_training_science-v2_auto_seed44_897db37.json new file mode 100644 index 0000000..df39abe --- /dev/null +++ b/experiments/records/training/20261022T043349Z_training_science-v2_auto_seed44_897db37.json @@ -0,0 +1,129 @@ +{ + "record_version": 1, + "created_utc": "20261022T043349Z", + "experiment": "training", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "cpu", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "897db37958b0c7595f097a26eaa60549b0f8831e", + "branch": "experiments/sdpa-training-v2", + "dirty": false + }, + "command": "python scripts/record_training_run.py artifacts/runs/v2/auto-s44 --protocol science-v2 --code-commit 897db37 --train-text data/science-v2/train.txt --validation-text data/science-v2/validation.txt --manifest data/science-v2/manifest.json --training-command 'python scripts/train_tiny.py data/science-v2/train.txt --validation-text data/science-v2/validation.txt --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 --device cpu --seed 44 --attention-backend auto --output-dir artifacts/runs/v2/auto-s44' --note 'science-v2 protocol'" + }, + "settings": { + "dtype": "float32", + "seed": 44, + "protocol": "science-v2", + "attention_backend": "auto", + "model_config": { + "vocab_size": 256, + "context_length": 128, + "embedding_dim": 192, + "num_layers": 4, + "num_heads": 6, + "dropout": 0.0, + "positional_encoding": "learned", + "norm": "rmsnorm", + "activation": "swiglu", + "rope_theta": 10000.0 + }, + "tokenizer": "byte", + "optimizer": "AdamW lr=3e-4 (scripts/train_tiny.py defaults)", + "steps": 1000, + "training_command": "python scripts/train_tiny.py data/science-v2/train.txt --validation-text data/science-v2/validation.txt --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 --device cpu --seed 44 --attention-backend auto --output-dir artifacts/runs/v2/auto-s44", + "training_code_commit": "897db37", + "note": "science-v2 protocol", + "data": { + "train": { + "path": "data/science-v2/train.txt", + "sha256": "65182db3ac268ec990f79a32fd9151c0de332fb8c411138af9ef965c9f80265e" + }, + "validation": { + "path": "data/science-v2/validation.txt", + "sha256": "6457fe8eaf9011ef75600da24ed0925cfb5e9464fa1c9e453785eecfc9592b17" + }, + "corpus_sha256_in_checkpoint": "84e98dc220b5b274380238ca9c7b45094b7b62f42ca486e70e65d8ac51449988", + "manifest_path": "data/science-v2/manifest.json", + "manifest_sha256": "9a34a5fca7532fccbe7d6be0e59a99523ebf7796cab09cb3ebd66ab1f26025c0" + } + }, + "results": { + "metrics": [ + { + "step": 250, + "train_loss": 3.0367985367774963, + "validation_loss": 3.0385916228527456, + "perplexity": 20.875821491581487, + "bits_per_token": 4.383761065576404, + "validation_tokens": 105665, + "elapsed_seconds": 36.75814312500006, + "train_seconds": 34.86062602799211, + "eval_seconds": 1.8970119580044411 + }, + { + "step": 500, + "train_loss": 3.0201011896133423, + "validation_loss": 3.036149389149366, + "perplexity": 20.824900063073567, + "bits_per_token": 4.380237667123846, + "validation_tokens": 105665, + "elapsed_seconds": 73.90597262499796, + "train_seconds": 70.09189898095792, + "eval_seconds": 1.9106220840039896 + }, + { + "step": 750, + "train_loss": 3.015455663204193, + "validation_loss": 3.034243380471162, + "perplexity": 20.78524542587694, + "bits_per_token": 4.37748787785591, + "validation_tokens": 105665, + "elapsed_seconds": 111.3095515830064, + "train_seconds": 105.57070272295095, + "eval_seconds": 1.9197136669972679 + }, + { + "step": 1000, + "train_loss": 2.643465757369995, + "validation_loss": 2.602737171301438, + "perplexity": 13.50064107429109, + "bits_per_token": 3.7549560097739527, + "validation_tokens": 105665, + "elapsed_seconds": 148.4968925830035, + "train_seconds": 140.91734968296078, + "eval_seconds": 1.8356620419945102 + } + ], + "best_step": 1000, + "best_validation_loss": 2.602737171301438, + "best_checkpoint_sha256": "cb77533a909fb6f40f08c3a50854a28765ab214aa87a25a8bb8bc5566211cf40", + "final_checkpoint_sha256": "eeef0fd313c7d9d023674a38ac94c05423bd02c5490937cf05d3bdabadb1b219", + "run_summary": { + "attention_backend": "auto", + "device": "cpu", + "steps": 1000, + "train_seconds": 140.91734968296078, + "train_tokens": 4096000, + "train_tokens_per_second": 29066.68347946707, + "eval_seconds": [ + 1.8970119580044411, + 1.9106220840039896, + 1.9197136669972679, + 1.8356620419945102 + ], + "process_peak_rss_bytes": 469811200 + } + } +} diff --git a/experiments/records/training/20261022T043559Z_training_science-v2_sdpa_seed44_897db37.json b/experiments/records/training/20261022T043559Z_training_science-v2_sdpa_seed44_897db37.json new file mode 100644 index 0000000..991e971 --- /dev/null +++ b/experiments/records/training/20261022T043559Z_training_science-v2_sdpa_seed44_897db37.json @@ -0,0 +1,129 @@ +{ + "record_version": 1, + "created_utc": "20261022T043559Z", + "experiment": "training", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "cpu", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "897db37958b0c7595f097a26eaa60549b0f8831e", + "branch": "experiments/sdpa-training-v2", + "dirty": false + }, + "command": "python scripts/record_training_run.py artifacts/runs/v2/sdpa-s44 --protocol science-v2 --code-commit 897db37 --train-text data/science-v2/train.txt --validation-text data/science-v2/validation.txt --manifest data/science-v2/manifest.json --training-command 'python scripts/train_tiny.py data/science-v2/train.txt --validation-text data/science-v2/validation.txt --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 --device cpu --seed 44 --attention-backend sdpa --output-dir artifacts/runs/v2/sdpa-s44' --note 'science-v2 protocol'" + }, + "settings": { + "dtype": "float32", + "seed": 44, + "protocol": "science-v2", + "attention_backend": "sdpa", + "model_config": { + "vocab_size": 256, + "context_length": 128, + "embedding_dim": 192, + "num_layers": 4, + "num_heads": 6, + "dropout": 0.0, + "positional_encoding": "learned", + "norm": "rmsnorm", + "activation": "swiglu", + "rope_theta": 10000.0 + }, + "tokenizer": "byte", + "optimizer": "AdamW lr=3e-4 (scripts/train_tiny.py defaults)", + "steps": 1000, + "training_command": "python scripts/train_tiny.py data/science-v2/train.txt --validation-text data/science-v2/validation.txt --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 --device cpu --seed 44 --attention-backend sdpa --output-dir artifacts/runs/v2/sdpa-s44", + "training_code_commit": "897db37", + "note": "science-v2 protocol", + "data": { + "train": { + "path": "data/science-v2/train.txt", + "sha256": "65182db3ac268ec990f79a32fd9151c0de332fb8c411138af9ef965c9f80265e" + }, + "validation": { + "path": "data/science-v2/validation.txt", + "sha256": "6457fe8eaf9011ef75600da24ed0925cfb5e9464fa1c9e453785eecfc9592b17" + }, + "corpus_sha256_in_checkpoint": "84e98dc220b5b274380238ca9c7b45094b7b62f42ca486e70e65d8ac51449988", + "manifest_path": "data/science-v2/manifest.json", + "manifest_sha256": "9a34a5fca7532fccbe7d6be0e59a99523ebf7796cab09cb3ebd66ab1f26025c0" + } + }, + "results": { + "metrics": [ + { + "step": 250, + "train_loss": 3.036798596382141, + "validation_loss": 3.038591621264266, + "perplexity": 20.87582145842067, + "bits_per_token": 4.383761063284712, + "validation_tokens": 105665, + "elapsed_seconds": 32.20392554200953, + "train_seconds": 30.747058229899267, + "eval_seconds": 1.456463333001011 + }, + { + "step": 500, + "train_loss": 3.020101249217987, + "validation_loss": 3.036149409944006, + "perplexity": 20.824900496119866, + "bits_per_token": 4.38023769712417, + "validation_tokens": 105665, + "elapsed_seconds": 63.96220758301206, + "train_seconds": 61.02044177992502, + "eval_seconds": 1.479247834009584 + }, + { + "step": 750, + "train_loss": 3.015455663204193, + "validation_loss": 3.034243382204049, + "perplexity": 20.785245461895418, + "bits_per_token": 4.377487880355937, + "validation_tokens": 105665, + "elapsed_seconds": 95.83539833300165, + "train_seconds": 91.40521457779687, + "eval_seconds": 1.483557874991675 + }, + { + "step": 1000, + "train_loss": 2.640721917152405, + "validation_loss": 2.6013543401671297, + "perplexity": 13.481984869643659, + "bits_per_token": 3.7529610061541, + "validation_tokens": 105665, + "elapsed_seconds": 128.22071308300656, + "train_seconds": 122.27788896088896, + "eval_seconds": 1.5077983329974813 + } + ], + "best_step": 1000, + "best_validation_loss": 2.6013543401671297, + "best_checkpoint_sha256": "2488847e2e18787df5cbd5dd3000f2405527961878cb1c4a0ee510b7f561c038", + "final_checkpoint_sha256": "5f25f43bcdd4a01e7bab23978fa15fdf85e650008b93d8a39b7960437885c687", + "run_summary": { + "attention_backend": "sdpa", + "device": "cpu", + "steps": 1000, + "train_seconds": 122.27788896088896, + "train_tokens": 4096000, + "train_tokens_per_second": 33497.4706777128, + "eval_seconds": [ + 1.456463333001011, + 1.479247834009584, + 1.483557874991675, + 1.5077983329974813 + ], + "process_peak_rss_bytes": 463126528 + } + } +} diff --git a/experiments/records/training_step/20261022T042023Z_training_step_cpu_float32_d3e6f26.json b/experiments/records/training_step/20261022T042023Z_training_step_cpu_float32_d3e6f26.json new file mode 100644 index 0000000..475bf1d --- /dev/null +++ b/experiments/records/training_step/20261022T042023Z_training_step_cpu_float32_d3e6f26.json @@ -0,0 +1,737 @@ +{ + "record_version": 1, + "created_utc": "20261022T042023Z", + "experiment": "training_step", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "cpu", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "d3e6f26a1c59e550fe6992e0c29d4262febcc3d0", + "branch": "experiments/sdpa-training-v2", + "dirty": false + }, + "command": "python scripts/benchmark_training_step.py --device cpu" + }, + "settings": { + "dtype": "float32", + "backends": [ + "auto", + "sdpa", + "reference" + ], + "seq_lens": [ + 128, + 256, + 512, + 1024 + ], + "batch_size": 8, + "warmup": 3, + "repeats": 10, + "seed": 0, + "optimizer": "AdamW lr=3e-4", + "architecture": { + "vocab_size": 256, + "embedding_dim": 192, + "num_layers": 4, + "num_heads": 6 + }, + "weights": "seeded random initialization, identical across backends" + }, + "results": [ + { + "backend": "auto", + "seq_len": 128, + "batch": 8, + "step_ms": { + "count": 10, + "median": 27.775708491390105, + "p95": 28.54577949474333, + "min": 27.095459008705802, + "max": 28.73579198785592, + "iqr": 0.8756674942560494, + "mean": 27.774145996954758, + "cv": 0.019913292247051177 + }, + "tokens_per_second": { + "count": 10, + "median": 36867.17123140424, + "p95": 37744.24864062385, + "min": 35635.00182743369, + "max": 37792.31050010955, + "iqr": 1163.806474063742, + "mean": 36881.90981998572, + "cv": 0.019805322064421163 + }, + "step_seconds_samples": [ + 0.02873579198785592, + 0.027312457998050377, + 0.028261458995984867, + 0.027681416991981678, + 0.027172249989234842, + 0.027095459008705802, + 0.027869999990798533, + 0.028313542003161274, + 0.02732170801027678, + 0.027977374993497506 + ], + "losses": [ + 5.589323997497559, + 5.580076694488525, + 5.584849834442139, + 5.581105709075928, + 5.563730716705322, + 5.571087837219238, + 5.575624465942383, + 5.588212013244629, + 5.590721607208252, + 5.58758544921875, + 5.577513217926025, + 5.581334590911865, + 5.5781755447387695 + ], + "max_abs_loss_difference_vs_first_backend": 0.0, + "memory": { + "peak_extra_bytes": 113082368, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "sdpa", + "seq_len": 128, + "batch": 8, + "step_ms": { + "count": 10, + "median": 25.644354500400368, + "p95": 27.062545903754653, + "min": 25.33620900067035, + "max": 27.271084007225, + "iqr": 0.3730727548827417, + "mean": 25.889208300213795, + "cv": 0.024691425597648222 + }, + "tokens_per_second": { + "count": 10, + "median": 39930.82333562999, + "p95": 40370.75793470344, + "min": 37548.929104860996, + "max": 40416.46482995569, + "iqr": 578.3421542307915, + "mean": 39574.21575771362, + "cv": 0.023945973822892767 + }, + "step_seconds_samples": [ + 0.027271084007225, + 0.025608249998185784, + 0.025633334007579833, + 0.02680766599951312, + 0.025655374993220903, + 0.02533620900067035, + 0.02592479099985212, + 0.025773040993954055, + 0.025482290991931222, + 0.025400042010005563 + ], + "losses": [ + 5.589323997497559, + 5.580076694488525, + 5.584849834442139, + 5.581106185913086, + 5.563730716705322, + 5.571087837219238, + 5.575624465942383, + 5.588212013244629, + 5.590721607208252, + 5.587585926055908, + 5.577513694763184, + 5.581334114074707, + 5.5781755447387695 + ], + "max_abs_loss_difference_vs_first_backend": 4.76837158203125e-07, + "memory": { + "peak_extra_bytes": 110870528, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "reference", + "seq_len": 128, + "batch": 8, + "step_ms": { + "count": 10, + "median": 27.655521502310876, + "p95": 29.172470352932578, + "min": 27.116832992760465, + "max": 29.474833005224355, + "iqr": 1.1461140020401217, + "mean": 27.87212919938611, + "cv": 0.029101085321025722 + }, + "tokens_per_second": { + "count": 10, + "median": 37028.872881420815, + "p95": 37717.651261427054, + "min": 34741.50302458026, + "max": 37762.521909301984, + "iqr": 1517.4566970636297, + "mean": 36766.615117721245, + "cv": 0.028467199010130808 + }, + "step_seconds_samples": [ + 0.029474833005224355, + 0.02785408400814049, + 0.028802916000131518, + 0.027875457992195152, + 0.027205209000385366, + 0.027116832992760465, + 0.02718862499750685, + 0.027456958996481262, + 0.028517750004539266, + 0.027228624996496364 + ], + "losses": [ + 5.589323997497559, + 5.580076694488525, + 5.5848493576049805, + 5.581105709075928, + 5.563730716705322, + 5.571087837219238, + 5.575624465942383, + 5.588212013244629, + 5.590721607208252, + 5.587585926055908, + 5.577513694763184, + 5.581334114074707, + 5.5781755447387695 + ], + "max_abs_loss_difference_vs_first_backend": 4.76837158203125e-07, + "memory": { + "peak_extra_bytes": 125353984, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "auto", + "seq_len": 256, + "batch": 8, + "step_ms": { + "count": 10, + "median": 54.55302099289838, + "p95": 55.765791548765264, + "min": 53.83883399190381, + "max": 55.96266700013075, + "iqr": 0.8114787378872279, + "mean": 54.728708299808204, + "cv": 0.012208162023485942 + }, + "tokens_per_second": { + "count": 10, + "median": 37541.96326099252, + "p95": 37934.95228454058, + "min": 36595.82557055787, + "max": 38039.45680376315, + "iqr": 555.4974830398714, + "mean": 37425.93503381465, + "cv": 0.01213495374416074 + }, + "step_seconds_samples": [ + 0.054169541006558575, + 0.05435308298910968, + 0.05552516599709634, + 0.05596266700013075, + 0.05383883399190381, + 0.055147082995972596, + 0.05429045800701715, + 0.054288042010739446, + 0.054752958996687084, + 0.05495925000286661 + ], + "losses": [ + 5.579926013946533, + 5.579548358917236, + 5.591322898864746, + 5.595408916473389, + 5.58452033996582, + 5.587689399719238, + 5.588576793670654, + 5.575793743133545, + 5.583697319030762, + 5.586455345153809, + 5.580563545227051, + 5.583733081817627, + 5.588283061981201 + ], + "max_abs_loss_difference_vs_first_backend": 0.0, + "memory": { + "peak_extra_bytes": 221364224, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "sdpa", + "seq_len": 256, + "batch": 8, + "step_ms": { + "count": 10, + "median": 51.880333507142495, + "p95": 52.70432065735804, + "min": 50.91016700316686, + "max": 52.840708012809046, + "iqr": 0.39747874689055607, + "mean": 51.90709170419723, + "cv": 0.01086925449980035 + }, + "tokens_per_second": { + "count": 10, + "median": 39475.520592707, + "p95": 40113.62337671595, + "min": 38757.99695007014, + "max": 40227.721112613996, + "iqr": 301.5037974240113, + "mean": 39459.312037085314, + "cv": 0.010887546798654833 + }, + "step_seconds_samples": [ + 0.05181575000460725, + 0.05123308300971985, + 0.0521928329981165, + 0.05204504200082738, + 0.05091016700316686, + 0.05194491700967774, + 0.0525376250006957, + 0.0518093749997206, + 0.052840708012809046, + 0.05174141700263135 + ], + "losses": [ + 5.579926013946533, + 5.579548358917236, + 5.591322898864746, + 5.595408916473389, + 5.58452033996582, + 5.587689399719238, + 5.588576793670654, + 5.575793743133545, + 5.583697319030762, + 5.586455345153809, + 5.580563545227051, + 5.583733081817627, + 5.588283061981201 + ], + "max_abs_loss_difference_vs_first_backend": 0.0, + "memory": { + "peak_extra_bytes": 216907776, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "reference", + "seq_len": 256, + "batch": 8, + "step_ms": { + "count": 10, + "median": 60.92856250324985, + "p95": 61.91457294553402, + "min": 60.40262500755489, + "max": 62.00916699890513, + "iqr": 0.766010500228731, + "mean": 61.11503330030246, + "cv": 0.0087669818765352 + }, + "tokens_per_second": { + "count": 10, + "median": 33613.24557062628, + "p95": 33838.9043294406, + "min": 33027.374807924134, + "max": 33905.81120843416, + "iqr": 419.1719032053079, + "mean": 33512.88555736541, + "cv": 0.008732455252869743 + }, + "step_seconds_samples": [ + 0.06080258300062269, + 0.06200916699890513, + 0.061039042004267685, + 0.06040262500755489, + 0.06120187501073815, + 0.061648791990592144, + 0.06179895799141377, + 0.060818083002232015, + 0.06066866600303911, + 0.06076054199365899 + ], + "losses": [ + 5.579926013946533, + 5.579548358917236, + 5.591322898864746, + 5.595408916473389, + 5.58452033996582, + 5.587689399719238, + 5.588576793670654, + 5.575793743133545, + 5.583697319030762, + 5.586455345153809, + 5.580563545227051, + 5.583733081817627, + 5.588283061981201 + ], + "max_abs_loss_difference_vs_first_backend": 0.0, + "memory": { + "peak_extra_bytes": 297517056, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "auto", + "seq_len": 512, + "batch": 8, + "step_ms": { + "count": 10, + "median": 131.9834579990129, + "p95": 135.08371875504963, + "min": 129.68733299931046, + "max": 135.41250000707805, + "iqr": 2.9821042335242964, + "mean": 132.3860249016434, + "cv": 0.014502969890800074 + }, + "tokens_per_second": { + "count": 10, + "median": 31034.204341619698, + "p95": 31496.12694461055, + "min": 30248.31533119838, + "max": 31583.65512861444, + "iqr": 695.7919552627791, + "mean": 30945.66026443605, + "cv": 0.014455721853178812 + }, + "step_seconds_samples": [ + 0.13049095799215138, + 0.1320660829951521, + 0.1332380829990143, + 0.12968733299931046, + 0.1342149169940967, + 0.1319008330028737, + 0.13089337501151022, + 0.13468187500257045, + 0.13541250000707805, + 0.1312742920126766 + ], + "losses": [ + 5.584229946136475, + 5.580632209777832, + 5.579465389251709, + 5.581432342529297, + 5.57510232925415, + 5.5818657875061035, + 5.5859150886535645, + 5.582075595855713, + 5.586726665496826, + 5.584770679473877, + 5.576126575469971, + 5.583481788635254, + 5.58354377746582 + ], + "max_abs_loss_difference_vs_first_backend": 0.0, + "memory": { + "peak_extra_bytes": 435126272, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "sdpa", + "seq_len": 512, + "batch": 8, + "step_ms": { + "count": 10, + "median": 121.95012549636886, + "p95": 134.04771090717983, + "min": 120.62595800671261, + "max": 141.3840420136694, + "iqr": 2.856927501852624, + "mean": 124.15754600224318, + "cv": 0.05027107268885418 + }, + "tokens_per_second": { + "count": 10, + "median": 33587.7144596691, + "p95": 33952.412888557265, + "min": 28970.737727274678, + "max": 33956.207002907824, + "iqr": 778.8389734880693, + "mean": 33057.92695398867, + "cv": 0.04521323547368842 + }, + "step_seconds_samples": [ + 0.12108354100200813, + 0.12508108399924822, + 0.12164291700173635, + 0.12225733399100136, + 0.1242037500051083, + 0.1413840420136694, + 0.12128691699763294, + 0.12335399999574292, + 0.12065591700957157, + 0.12062595800671261 + ], + "losses": [ + 5.584229946136475, + 5.580632209777832, + 5.579465389251709, + 5.581432342529297, + 5.57510232925415, + 5.5818657875061035, + 5.5859150886535645, + 5.582075595855713, + 5.586726665496826, + 5.584770679473877, + 5.576126575469971, + 5.583481311798096, + 5.58354377746582 + ], + "max_abs_loss_difference_vs_first_backend": 4.76837158203125e-07, + "memory": { + "peak_extra_bytes": 425492480, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "reference", + "seq_len": 512, + "batch": 8, + "step_ms": { + "count": 10, + "median": 168.17429150250973, + "p95": 170.2358977017866, + "min": 165.80945800524205, + "max": 171.2296659970889, + "iqr": 1.8630412487254944, + "mean": 168.015866499627, + "cv": 0.009422910410062938 + }, + "tokens_per_second": { + "count": 10, + "median": 24355.684700721824, + "p95": 24675.153265976885, + "min": 23921.088534212508, + "max": 24703.054031275497, + "iqr": 271.25143765927714, + "mean": 24380.590617908045, + "cv": 0.009386460962869096 + }, + "step_seconds_samples": [ + 0.16754383299848996, + 0.16816295799799263, + 0.16902129200752825, + 0.1712296659970889, + 0.16580945800524205, + 0.16818562500702683, + 0.16864225000608712, + 0.16622666698822286, + 0.16872741599217989, + 0.16660949999641161 + ], + "losses": [ + 5.584229946136475, + 5.580631256103516, + 5.579465389251709, + 5.581432342529297, + 5.57510232925415, + 5.5818657875061035, + 5.585914611816406, + 5.582075595855713, + 5.586726665496826, + 5.584770679473877, + 5.576126575469971, + 5.583481788635254, + 5.58354377746582 + ], + "max_abs_loss_difference_vs_first_backend": 9.5367431640625e-07, + "memory": { + "peak_extra_bytes": 717340672, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "auto", + "seq_len": 1024, + "batch": 8, + "step_ms": { + "count": 10, + "median": 353.10985449905274, + "p95": 367.3335918443627, + "min": 341.2384999974165, + "max": 372.95304199506063, + "iqr": 10.995999502483755, + "mean": 354.0680709993467, + "cv": 0.026126311954377514 + }, + "tokens_per_second": { + "count": 10, + "median": 23199.601785814237, + "p95": 23921.100218699885, + "min": 21965.231751906435, + "max": 24006.669822021904, + "iqr": 720.8150969383969, + "mean": 23150.837085865467, + "cv": 0.025809527931567232 + }, + "step_seconds_samples": [ + 0.3412384999974165, + 0.35080241700052284, + 0.3534590420022141, + 0.3439630000066245, + 0.35276066699589137, + 0.35834716699901037, + 0.3472725419997005, + 0.3594189580035163, + 0.3604653749935096, + 0.37295304199506063 + ], + "losses": [ + 5.579966068267822, + 5.580023765563965, + 5.580759048461914, + 5.580639362335205, + 5.587033271789551, + 5.585662841796875, + 5.579523086547852, + 5.577640533447266, + 5.577197551727295, + 5.582764148712158, + 5.579935073852539, + 5.580066204071045, + 5.584075450897217 + ], + "max_abs_loss_difference_vs_first_backend": 0.0, + "memory": { + "peak_extra_bytes": 839614464, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "sdpa", + "seq_len": 1024, + "batch": 8, + "step_ms": { + "count": 10, + "median": 276.11160399828805, + "p95": 288.41621875035344, + "min": 269.7508330020355, + "max": 292.2988749924116, + "iqr": 8.701791997737018, + "mean": 277.1271791003528, + "cv": 0.02565166166901079 + }, + "tokens_per_second": { + "count": 10, + "median": 29670.40475803303, + "p95": 30329.475562484586, + "min": 28026.108551436177, + "max": 30368.766275276652, + "iqr": 935.3489486948165, + "mean": 29577.580632804857, + "cv": 0.02511722604623429 + }, + "step_seconds_samples": [ + 0.2716239160072291, + 0.28367075001006015, + 0.2705286249984056, + 0.2697508330020355, + 0.2810628750012256, + 0.2718112089933129, + 0.2922988749924116, + 0.2778983329917537, + 0.2743248750048224, + 0.2783015000022715 + ], + "losses": [ + 5.579966068267822, + 5.580024242401123, + 5.580759048461914, + 5.580639362335205, + 5.587033271789551, + 5.585662841796875, + 5.579523086547852, + 5.577640533447266, + 5.577197551727295, + 5.582763671875, + 5.579935073852539, + 5.580065727233887, + 5.584075450897217 + ], + "max_abs_loss_difference_vs_first_backend": 4.76837158203125e-07, + "memory": { + "peak_extra_bytes": 805519360, + "method": "process_peak_rss_delta" + } + }, + { + "backend": "reference", + "seq_len": 1024, + "batch": 8, + "step_ms": { + "count": 10, + "median": 531.3853959960397, + "p95": 543.7834042037139, + "min": 513.9356669969857, + "max": 548.3590420044493, + "iqr": 9.075874746486079, + "mean": 530.0921292000567, + "cv": 0.01894226645639349 + }, + "tokens_per_second": { + "count": 10, + "median": 15416.313143093665, + "p95": 15899.234256018779, + "min": 14939.117206958595, + "max": 15939.738231960553, + "iqr": 264.94085966632156, + "mean": 15458.911743998378, + "cv": 0.01895544356278718 + }, + "step_seconds_samples": [ + 0.538190958002815, + 0.5139356669969857, + 0.5310299999982817, + 0.5483590420044493, + 0.5168542500032345, + 0.5332532909960719, + 0.5317407919937978, + 0.5243712500086986, + 0.5283485419931822, + 0.5348375000030501 + ], + "losses": [ + 5.579966068267822, + 5.580024242401123, + 5.580759048461914, + 5.580639362335205, + 5.587033271789551, + 5.585662841796875, + 5.579523086547852, + 5.577640533447266, + 5.577197551727295, + 5.582763671875, + 5.579935073852539, + 5.580066204071045, + 5.584075450897217 + ], + "max_abs_loss_difference_vs_first_backend": 4.76837158203125e-07, + "memory": { + "peak_extra_bytes": 2011955200, + "method": "process_peak_rss_delta" + } + } + ] +} diff --git a/experiments/records/training_step/20261022T042100Z_training_step_mps_float32_d3e6f26.json b/experiments/records/training_step/20261022T042100Z_training_step_mps_float32_d3e6f26.json new file mode 100644 index 0000000..b7ef2f9 --- /dev/null +++ b/experiments/records/training_step/20261022T042100Z_training_step_mps_float32_d3e6f26.json @@ -0,0 +1,737 @@ +{ + "record_version": 1, + "created_utc": "20261022T042100Z", + "experiment": "training_step", + "environment": { + "python": "3.12.14", + "torch": "2.14.1", + "numpy": "2.5.4", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "machine": "arm64", + "processor": "Apple M4", + "memory_bytes": 17179869184, + "cpu_count": 10, + "torch_threads": 4, + "device": "mps", + "mps_available": true, + "cuda_available": false, + "git": { + "commit": "d3e6f26a1c59e550fe6992e0c29d4262febcc3d0", + "branch": "experiments/sdpa-training-v2", + "dirty": false + }, + "command": "python scripts/benchmark_training_step.py --device mps" + }, + "settings": { + "dtype": "float32", + "backends": [ + "auto", + "sdpa", + "reference" + ], + "seq_lens": [ + 128, + 256, + 512, + 1024 + ], + "batch_size": 8, + "warmup": 3, + "repeats": 10, + "seed": 0, + "optimizer": "AdamW lr=3e-4", + "architecture": { + "vocab_size": 256, + "embedding_dim": 192, + "num_layers": 4, + "num_heads": 6 + }, + "weights": "seeded random initialization, identical across backends" + }, + "results": [ + { + "backend": "auto", + "seq_len": 128, + "batch": 8, + "step_ms": { + "count": 10, + "median": 26.59852099895943, + "p95": 26.96273369510891, + "min": 26.336166993132792, + "max": 27.027459000237286, + "iqr": 0.4544380099105183, + "mean": 26.616183400619775, + "cv": 0.009575914135243727 + }, + "tokens_per_second": { + "count": 10, + "median": 38498.409799569825, + "p95": 38879.096753584796, + "min": 37887.39444544194, + "max": 38881.891972624944, + "iqr": 657.836482036997, + "mean": 38476.00231894681, + "cv": 0.009555543881143883 + }, + "step_seconds_samples": [ + 0.026883624988840893, + 0.026840750011615455, + 0.026775499994982965, + 0.027027459000237286, + 0.026336166993132792, + 0.026401375012937933, + 0.02662145800422877, + 0.02657558399369009, + 0.026340375014115125, + 0.02635954099241644 + ], + "losses": [ + 5.589323997497559, + 5.580077171325684, + 5.584850311279297, + 5.581105709075928, + 5.563730239868164, + 5.571087837219238, + 5.575624465942383, + 5.588212013244629, + 5.59072208404541, + 5.58758544921875, + 5.577513694763184, + 5.581334114074707, + 5.5781755447387695 + ], + "max_abs_loss_difference_vs_first_backend": 0.0, + "memory": { + "peak_extra_bytes": 110136320, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "sdpa", + "seq_len": 128, + "batch": 8, + "step_ms": { + "count": 10, + "median": 25.638353996328078, + "p95": 25.853691999509465, + "min": 25.519416987663135, + "max": 25.855041996692307, + "iqr": 0.11917799201910384, + "mean": 25.653579196659848, + "cv": 0.004674097897615823 + }, + "tokens_per_second": { + "count": 10, + "median": 39940.16190431913, + "p95": 40122.506847769866, + "min": 39605.42783612583, + "max": 40126.30854752806, + "iqr": 185.76393656169967, + "mean": 39917.2397460635, + "cv": 0.004659364053158296 + }, + "step_seconds_samples": [ + 0.025855041996692307, + 0.025524790995405056, + 0.025852042002952658, + 0.0256494999921415, + 0.025635874990257435, + 0.025702583996462636, + 0.025519416987663135, + 0.025593291997211054, + 0.02556241600541398, + 0.025640833002398722 + ], + "losses": [ + 5.589323997497559, + 5.580077171325684, + 5.584849834442139, + 5.5811052322387695, + 5.563730239868164, + 5.571087837219238, + 5.575624465942383, + 5.588212013244629, + 5.590721607208252, + 5.58758544921875, + 5.577513694763184, + 5.581334114074707, + 5.5781755447387695 + ], + "max_abs_loss_difference_vs_first_backend": 4.76837158203125e-07, + "memory": { + "peak_extra_bytes": 103320576, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "reference", + "seq_len": 128, + "batch": 8, + "step_ms": { + "count": 10, + "median": 25.13908300170442, + "p95": 25.424095347261755, + "min": 25.00854100799188, + "max": 25.537458001053892, + "iqr": 0.09820825289352797, + "mean": 25.174116497510113, + "cv": 0.006043885514880337 + }, + "tokens_per_second": { + "count": 10, + "median": 40733.40388905625, + "p95": 40936.989376454905, + "min": 40097.96119714582, + "max": 40946.01119164706, + "iqr": 158.85229804121627, + "mean": 40678.02770380936, + "cv": 0.005997806613255973 + }, + "step_seconds_samples": [ + 0.025155208000796847, + 0.02512091699463781, + 0.025221749994670972, + 0.02510487499239389, + 0.025537458001053892, + 0.02528554099262692, + 0.025020791988936253, + 0.02516312499938067, + 0.025122958002611995, + 0.02500854100799188 + ], + "losses": [ + 5.589323997497559, + 5.580077171325684, + 5.584850311279297, + 5.5811052322387695, + 5.563730239868164, + 5.571087837219238, + 5.575624465942383, + 5.588212013244629, + 5.590721607208252, + 5.58758544921875, + 5.577513694763184, + 5.581333637237549, + 5.5781755447387695 + ], + "max_abs_loss_difference_vs_first_backend": 4.76837158203125e-07, + "memory": { + "peak_extra_bytes": 103386112, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "auto", + "seq_len": 256, + "batch": 8, + "step_ms": { + "count": 10, + "median": 51.85595850343816, + "p95": 57.89272065885597, + "min": 50.650000004679896, + "max": 59.933958007604815, + "iqr": 3.3049692574422807, + "mean": 53.087237403087784, + "cv": 0.05522645327960647 + }, + "tokens_per_second": { + "count": 10, + "median": 39498.15850435749, + "p95": 40273.03136887761, + "min": 34170.9452884813, + "max": 40434.35340198957, + "iqr": 2410.6846214368634, + "mean": 38677.183454007965, + "cv": 0.051649659250290664 + }, + "step_seconds_samples": [ + 0.051324833009857684, + 0.050650000004679896, + 0.055132875000708736, + 0.05118483299156651, + 0.051191040998674, + 0.051103083009365946, + 0.052387083997018635, + 0.05256679200101644, + 0.05539787501038518, + 0.059933958007604815 + ], + "losses": [ + 5.579925537109375, + 5.579547882080078, + 5.591322898864746, + 5.5954084396362305, + 5.584521293640137, + 5.58768892288208, + 5.588576316833496, + 5.575794219970703, + 5.583696365356445, + 5.586455345153809, + 5.580563545227051, + 5.583732604980469, + 5.588282585144043 + ], + "max_abs_loss_difference_vs_first_backend": 0.0, + "memory": { + "peak_extra_bytes": 243601408, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "sdpa", + "seq_len": 256, + "batch": 8, + "step_ms": { + "count": 10, + "median": 49.80631249782164, + "p95": 55.68713590764673, + "min": 48.93983399961144, + "max": 58.9244170114398, + "iqr": 0.601531002757838, + "mean": 50.78010430152062, + "cv": 0.05819585679428874 + }, + "tokens_per_second": { + "count": 10, + "median": 41119.414624212135, + "p95": 41674.5303127263, + "min": 34756.38969160092, + "max": 41847.30173004388, + "iqr": 496.1632044802973, + "mean": 40439.81164854755, + "cv": 0.05152387576071167 + }, + "step_seconds_samples": [ + 0.04971816699253395, + 0.04939300000842195, + 0.05018637499597389, + 0.04949704199680127, + 0.04992770800890867, + 0.04893983399961144, + 0.04989445800310932, + 0.049589582995395176, + 0.0589244170114398, + 0.051730459003010765 + ], + "losses": [ + 5.579925537109375, + 5.579547882080078, + 5.591322898864746, + 5.5954084396362305, + 5.584521293640137, + 5.58768892288208, + 5.588576316833496, + 5.575794219970703, + 5.583696365356445, + 5.586455345153809, + 5.580563545227051, + 5.583732604980469, + 5.588282585144043 + ], + "max_abs_loss_difference_vs_first_backend": 0.0, + "memory": { + "peak_extra_bytes": 231542784, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "reference", + "seq_len": 256, + "batch": 8, + "step_ms": { + "count": 10, + "median": 50.298583511903416, + "p95": 51.758925152535085, + "min": 49.91624999092892, + "max": 52.29637499724049, + "iqr": 0.27471824432723224, + "mean": 50.507654204557184, + "cv": 0.013941248045248003 + }, + "tokens_per_second": { + "count": 10, + "median": 40716.87167876718, + "p95": 40967.31265732687, + "min": 39161.4141536209, + "max": 41028.723118667265, + "iqr": 222.6863698945017, + "mean": 40555.234751943564, + "cv": 0.013610380987517004 + }, + "step_seconds_samples": [ + 0.0503332080115797, + 0.05008283301140182, + 0.051102042009006254, + 0.05037516599986702, + 0.050412875003530644, + 0.05018329199810978, + 0.05011054201168008, + 0.05026395901222713, + 0.05229637499724049, + 0.04991624999092892 + ], + "losses": [ + 5.579925537109375, + 5.579547882080078, + 5.591322898864746, + 5.5954084396362305, + 5.584521293640137, + 5.58768892288208, + 5.588576316833496, + 5.575794219970703, + 5.583696365356445, + 5.586455345153809, + 5.580563545227051, + 5.583732604980469, + 5.588282585144043 + ], + "max_abs_loss_difference_vs_first_backend": 0.0, + "memory": { + "peak_extra_bytes": 231804928, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "auto", + "seq_len": 512, + "batch": 8, + "step_ms": { + "count": 10, + "median": 117.31491600221489, + "p95": 117.68787255059578, + "min": 116.89558300713543, + "max": 117.69779100723099, + "iqr": 0.5275837465887889, + "mean": 117.33825810078997, + "cv": 0.0024780138056008003 + }, + "tokens_per_second": { + "count": 10, + "median": 34914.57394437298, + "p95": 35015.86148558452, + "min": 34800.99299185959, + "max": 35039.81839715856, + "iqr": 156.88317278049362, + "mean": 34907.81868082554, + "cv": 0.0024781762760020263 + }, + "step_seconds_samples": [ + 0.11734716600039974, + 0.11709070799406618, + 0.11753987499105278, + 0.11728266600403003, + 0.11765554199519102, + 0.11689558300713543, + 0.11707345800823532, + 0.11767574999248609, + 0.11712404200807214, + 0.11769779100723099 + ], + "losses": [ + 5.584229469299316, + 5.580632209777832, + 5.579465866088867, + 5.581432342529297, + 5.575102806091309, + 5.581865310668945, + 5.585914611816406, + 5.582075595855713, + 5.586727142333984, + 5.584770202636719, + 5.576127052307129, + 5.583481788635254, + 5.58354377746582 + ], + "max_abs_loss_difference_vs_first_backend": 0.0, + "memory": { + "peak_extra_bytes": 594687488, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "sdpa", + "seq_len": 512, + "batch": 8, + "step_ms": { + "count": 10, + "median": 112.25943799945526, + "p95": 115.68386060389457, + "min": 111.61245900439098, + "max": 115.92204200860579, + "iqr": 1.5608537541993428, + "mean": 112.87930440157652, + "cv": 0.01398118784529441 + }, + "tokens_per_second": { + "count": 10, + "median": 36486.9354739079, + "p95": 36696.593568719225, + "min": 35334.091161851014, + "max": 36698.411956310876, + "iqr": 504.6176439128467, + "mean": 36292.838387302916, + "cv": 0.013780616895886465 + }, + "step_seconds_samples": [ + 0.11161245900439098, + 0.11190425000677351, + 0.11167929199291393, + 0.11352345799969044, + 0.1121654589951504, + 0.11592204200860579, + 0.11261516700324137, + 0.11539274999813642, + 0.11162475000310224, + 0.11235341700376011 + ], + "losses": [ + 5.584229469299316, + 5.580632209777832, + 5.579465866088867, + 5.581432342529297, + 5.575102806091309, + 5.581865310668945, + 5.585914611816406, + 5.582075595855713, + 5.586727142333984, + 5.584770202636719, + 5.576126575469971, + 5.583481788635254, + 5.58354377746582 + ], + "max_abs_loss_difference_vs_first_backend": 4.76837158203125e-07, + "memory": { + "peak_extra_bytes": 575813120, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "reference", + "seq_len": 512, + "batch": 8, + "step_ms": { + "count": 10, + "median": 115.31620849564206, + "p95": 124.39401454903418, + "min": 113.75729100836907, + "max": 125.76945799810346, + "iqr": 3.0847292473481502, + "mean": 117.11153740034206, + "cv": 0.034297661997805824 + }, + "tokens_per_second": { + "count": 10, + "median": 35519.72622751699, + "p95": 35937.44875702665, + "min": 32567.525257696234, + "max": 36006.48330926463, + "iqr": 935.4083165792545, + "mean": 35010.77473823181, + "cv": 0.03292281728094827 + }, + "step_seconds_samples": [ + 0.11533866699028295, + 0.11639912500686478, + 0.11375729100836907, + 0.12271291700017173, + 0.11812658299459144, + 0.11424404200806748, + 0.1149903329933295, + 0.12576945799810346, + 0.114483208002639, + 0.11529375000100117 + ], + "losses": [ + 5.584229469299316, + 5.580632209777832, + 5.579465866088867, + 5.581432342529297, + 5.575102806091309, + 5.5818657875061035, + 5.585914134979248, + 5.582075595855713, + 5.586727142333984, + 5.584770679473877, + 5.576126575469971, + 5.583481788635254, + 5.58354377746582 + ], + "max_abs_loss_difference_vs_first_backend": 4.76837158203125e-07, + "memory": { + "peak_extra_bytes": 576861696, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "auto", + "seq_len": 1024, + "batch": 8, + "step_ms": { + "count": 10, + "median": 306.4529999974184, + "p95": 308.23415840786765, + "min": 305.055832999642, + "max": 308.2997090095887, + "iqr": 1.7397082447132561, + "mean": 306.6938583026058, + "cv": 0.0038775546838900965 + }, + "tokens_per_second": { + "count": 10, + "median": 26731.668657620758, + "p95": 26851.562446504122, + "min": 26571.546325219573, + "max": 26854.100508248972, + "iqr": 151.17335735889355, + "mean": 26711.036390895402, + "cv": 0.0038760105791273945 + }, + "step_seconds_samples": [ + 0.3060260830097832, + 0.30692595899745356, + 0.3063105000037467, + 0.3064781249995576, + 0.3081405410048319, + 0.3051199170004111, + 0.3064278749952791, + 0.3082997090095887, + 0.305055832999642, + 0.30815404100576416 + ], + "losses": [ + 5.5799665451049805, + 5.580023765563965, + 5.580759048461914, + 5.580639839172363, + 5.587033271789551, + 5.585662841796875, + 5.579523086547852, + 5.577640533447266, + 5.577197551727295, + 5.582763671875, + 5.579935550689697, + 5.5800652503967285, + 5.584075450897217 + ], + "max_abs_loss_difference_vs_first_backend": 0.0, + "memory": { + "peak_extra_bytes": 1791231488, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "sdpa", + "seq_len": 1024, + "batch": 8, + "step_ms": { + "count": 10, + "median": 296.7739590021665, + "p95": 298.81787134872866, + "min": 294.9864590045763, + "max": 300.0024590000976, + "iqr": 0.9509692536084913, + "mean": 296.81476679979824, + "cv": 0.004450361260788089 + }, + "tokens_per_second": { + "count": 10, + "median": 27603.500142328678, + "p95": 27734.718087946574, + "min": 27306.442844847934, + "max": 27770.766250232904, + "iqr": 88.59614357445753, + "mean": 27600.194375259005, + "cv": 0.004427006667345218 + }, + "step_seconds_samples": [ + 0.2949864590045763, + 0.2967884590034373, + 0.29708333300368395, + 0.29675945900089573, + 0.2959454160009045, + 0.3000024590000976, + 0.2968811669998104, + 0.2973700419970555, + 0.29583983299380634, + 0.2964910409937147 + ], + "losses": [ + 5.5799665451049805, + 5.580023765563965, + 5.580759048461914, + 5.580639839172363, + 5.587033271789551, + 5.585662841796875, + 5.579523086547852, + 5.577640533447266, + 5.577197551727295, + 5.582763671875, + 5.579935550689697, + 5.5800652503967285, + 5.584075927734375 + ], + "max_abs_loss_difference_vs_first_backend": 4.76837158203125e-07, + "memory": { + "peak_extra_bytes": 1753482752, + "method": "mps_live_allocated_peak_per_op" + } + }, + { + "backend": "reference", + "seq_len": 1024, + "batch": 8, + "step_ms": { + "count": 10, + "median": 309.928854003374, + "p95": 311.8625165487174, + "min": 308.21162500069477, + "max": 312.04566700034775, + "iqr": 1.5434482556884177, + "mean": 309.8472126017441, + "cv": 0.0042399714237764735 + }, + "tokens_per_second": { + "count": 10, + "median": 26431.872657958906, + "p95": 26576.14154564137, + "min": 26252.567705068857, + "max": 26579.14022542639, + "iqr": 132.07753805391985, + "mean": 26439.264469694528, + "cv": 0.004235322655297032 + }, + "step_seconds_samples": [ + 0.30992395800421946, + 0.30993375000252854, + 0.3082889170036651, + 0.3103246670070803, + 0.31204566700034775, + 0.30821162500069477, + 0.310070084000472, + 0.30841774999862537, + 0.31163866599672474, + 0.30961704200308304 + ], + "losses": [ + 5.5799665451049805, + 5.580023765563965, + 5.580759048461914, + 5.580639839172363, + 5.587033271789551, + 5.585662841796875, + 5.579523086547852, + 5.577640533447266, + 5.577197551727295, + 5.582763671875, + 5.579935550689697, + 5.5800652503967285, + 5.584075450897217 + ], + "max_abs_loss_difference_vs_first_backend": 0.0, + "memory": { + "peak_extra_bytes": 1757677056, + "method": "mps_live_allocated_peak_per_op" + } + } + ] +} diff --git a/experiments/sdpa_v1.sh b/experiments/sdpa_v1.sh new file mode 100755 index 0000000..622071a --- /dev/null +++ b/experiments/sdpa_v1.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +# Matched training/evaluation comparison of the legacy attention path (mha, +# nn.MultiheadAttention; recorded as "auto" before the first release) and SDPA +# on the unchanged science-v1 corpus. Backends are always passed explicitly. +# Identical hyperparameters, seeds, data and hardware (CPU, float32); only +# --attention-backend differs. Backend order alternates per seed. +set -euo pipefail +cd "$(dirname "$0")/.." +PY=${PYTHON:-python} +RUNS=artifacts/runs/sdpa-v1 +COMMIT=$(git rev-parse --short HEAD) +train() { # seed backend + local out=$RUNS/$2-s$1 + local cmd=(scripts/train_tiny.py data/science/train.txt --validation-text data/science/validation.txt + --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 --device cpu + --seed "$1" --attention-backend "$2" --output-dir "$out") + "$PY" "${cmd[@]}" > "$out.log" 2>&1 + "$PY" scripts/record_training_run.py "$out" --protocol science-v1 --code-commit "$COMMIT" \ + --training-command "python ${cmd[*]}" --note "SDPA matched experiment (v1 corpus)" +} +mkdir -p "$RUNS" +train 42 mha; train 42 sdpa +train 43 sdpa; train 43 mha +train 44 mha; train 44 sdpa +for device in cpu mps; do + "$PY" scripts/evaluate_runs.py "$RUNS"/{mha,sdpa}-s4{2,3,4} --protocol science-v1 --device "$device" \ + --split validation=data/science/validation.txt \ + --split darwin_validation=data/science/darwin-1859-validation.txt \ + --split einstein_validation=data/science/einstein-1924-validation.txt \ + --name "sdpa_v1_eval_$device" +done diff --git a/experiments/v2_protocol.sh b/experiments/v2_protocol.sh new file mode 100755 index 0000000..d53bd03 --- /dev/null +++ b/experiments/v2_protocol.sh @@ -0,0 +1,36 @@ +#!/usr/bin/env bash +# Corpus protocol science-v2: build the cleaned corpus, train with the v1 +# hyperparameters (unchanged; nothing is tuned on v2), select checkpoints on +# validation, then evaluate each selected checkpoint once on the untouched test book. +set -euo pipefail +cd "$(dirname "$0")/.." +PY=${PYTHON:-python} +# mha is the legacy path (recorded as "auto" in the published v2 records). +BACKENDS=${BACKENDS:-"mha sdpa"} +RUNS=artifacts/runs/v2 +COMMIT=$(git rev-parse --short HEAD) +if [ "${SKIP_PREPARE:-0}" != 1 ]; then "$PY" scripts/prepare_science_corpus_v2.py > /dev/null; fi +git diff --quiet -- data/science-v2 || { echo "data/science-v2 differs from the committed corpus" >&2; exit 1; } +mkdir -p "$RUNS" +runs=() +for seed in 42 43 44; do + for backend in $BACKENDS; do + out=$RUNS/$backend-s$seed + # The test split is deliberately not passed to training. + cmd=(scripts/train_tiny.py data/science-v2/train.txt --validation-text data/science-v2/validation.txt + --steps 1000 --eval-interval 250 --gradient-accumulation-steps 4 --device cpu + --seed "$seed" --attention-backend "$backend" --output-dir "$out") + "$PY" "${cmd[@]}" > "$out.log" 2>&1 + "$PY" scripts/record_training_run.py "$out" --protocol science-v2 --code-commit "$COMMIT" \ + --train-text data/science-v2/train.txt --validation-text data/science-v2/validation.txt \ + --manifest data/science-v2/manifest.json --training-command "python ${cmd[*]}" --note "science-v2 protocol" + runs+=("$out") + done +done +# Final evaluation: once per selected checkpoint, after all training is complete. +"$PY" scripts/evaluate_runs.py "${runs[@]}" --protocol science-v2 --device cpu \ + --split validation=data/science-v2/validation.txt \ + --split darwin_validation=data/science-v2/darwin-1859-validation.txt \ + --split einstein_validation=data/science-v2/einstein-1924-validation.txt \ + --split test=data/science-v2/test.txt \ + --name v2_final_cpu diff --git a/requirements/release-lock.txt b/requirements/release-lock.txt new file mode 100644 index 0000000..70eac44 --- /dev/null +++ b/requirements/release-lock.txt @@ -0,0 +1,19 @@ +# Exact environment used for every record in experiments/records (generated with pip freeze). +# Python 3.12.14, macOS 27.0.1 arm64 (Apple M4). Install with: pip install -r requirements/release-lock.txt && pip install -e . --no-deps +filelock==4.2.4 +fsspec==2026.9.0 +iniconfig==2.3.1 +Jinja2==3.1.6 +MarkupSafe==3.0.4 +mpmath==1.3.0 +networkx==3.7 +numpy==2.5.4 +packaging==26.3 +pluggy==1.7.0 +Pygments==2.21.0 +pytest==9.1.1 +ruff==0.17.1 +setuptools==84.2.1 +sympy==1.14.0 +torch==2.14.1 +typing_extensions==4.16.0 diff --git a/scripts/attest_model_identity.py b/scripts/attest_model_identity.py new file mode 100644 index 0000000..14243b7 --- /dev/null +++ b/scripts/attest_model_identity.py @@ -0,0 +1,75 @@ +"""Attest deterministic model identities for training records that predate them. + +Training records written before schema v2 store only checkpoint *file* hashes. +For every current training record whose checkpoints are still on disk, this +script verifies the files are byte-identical to the recorded SHA-256 values and +then records each checkpoint's model identity (minilm-model-identity/v1) in a new, +append-only record. Nothing is inferred for checkpoints that are not available. + +--regenerated RECORD_KEY=PATH adds a regeneration check: the identity of a +checkpoint retrained from the record's command, compared with the original. +""" + +import argparse +import shlex +from pathlib import Path + +import torch + +from minilm_lab.identity import SCHEME, checkpoint_identity +from minilm_lab.provenance import environment, file_sha256, write_record +from minilm_lab.records import load_store + +ROOT = Path(__file__).resolve().parents[1] + + +def output_dir(record: dict) -> Path: + tokens = shlex.split(record["settings"]["training_command"]) + return ROOT / tokens[tokens.index("--output-dir") + 1] + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("--regenerated", action="append", default=[], help="RECORD_KEY=checkpoint path") + parser.add_argument("--output-dir", type=Path, default=ROOT / "experiments/records/identity") + args = parser.parse_args() + store = load_store(ROOT) + attested, unavailable = [], [] + for key, record in store.current("training"): + if record["results"].get("best_model_identity_sha256"): + continue # schema v2 records carry their own identity + directory = output_dir(record) + for kind, filename in (("best", "best_checkpoint.pt"), ("final", "tiny_checkpoint.pt")): + path = directory / filename + expected = record["results"][f"{kind}_checkpoint_sha256"] + if not path.is_file() or file_sha256(path) != expected: + unavailable.append({"training_record": key, "kind": kind, "reason": "file missing or not byte-identical to the recorded hash"}) + continue + identity = checkpoint_identity(torch.load(path, map_location="cpu", weights_only=True)) + attested.append({"training_record": key, "kind": kind, "checkpoint_path": str(path.relative_to(ROOT)), + "checkpoint_sha256": expected, "model_identity_sha256": identity["sha256"]}) + regenerated = [] + by_record = {(row["training_record"], row["kind"]): row for row in attested} + for item in args.regenerated: + key, path = item.split("=", 1) + original = by_record[(key, "best")] + checkpoint = torch.load(path, map_location="cpu", weights_only=True) + identity = checkpoint_identity(checkpoint)["sha256"] + regenerated.append({"training_record": key, "regenerated_checkpoint_sha256": file_sha256(Path(path)), + "original_checkpoint_sha256": original["checkpoint_sha256"], + "file_hash_equal": file_sha256(Path(path)) == original["checkpoint_sha256"], + "model_identity_sha256": identity, "model_identity_equal": identity == original["model_identity_sha256"]}) + payload = { + "experiment": "model_identity", + "environment": environment("cpu"), + "settings": {"scheme": SCHEME, "verification": "checkpoint file SHA-256 equals the training record before hashing tensors"}, + "results": {"attested": attested, "unavailable": unavailable, "regeneration_checks": regenerated}, + } + for row in regenerated: + print(f"{row['training_record']}: file hash equal={row['file_hash_equal']}, model identity equal={row['model_identity_equal']}") + print(f"attested {len(attested)} checkpoints, {len(unavailable)} unavailable") + print(f"wrote {write_record(args.output_dir, 'model_identity_attestation', payload)}") + + +if __name__ == "__main__": + main() diff --git a/scripts/benchmark_attention.py b/scripts/benchmark_attention.py new file mode 100644 index 0000000..d3d9da6 --- /dev/null +++ b/scripts/benchmark_attention.py @@ -0,0 +1,192 @@ +"""Matched latency, throughput and memory comparison of attention backends. + +Every backend runs on the same model object (identical seeded weights), the +same token inputs, dtype and device. Backend order rotates every repeat so +slow thermal or frequency drift does not favour one backend. + +Phases per (batch, sequence length): + forward full-sequence logits via model(x): the training/evaluation path + prefill model.forward_cached(prompt): logits plus a KV cache + decode greedy single-token steps against the cache, each step timed + +Peak memory is measured in a fresh subprocess per configuration because the +process high-water mark (CPU) and allocator pool (MPS) never shrink. +""" + +import argparse +import json +import subprocess +import sys +import time +from pathlib import Path + +import torch + +from minilm_lab.model import MiniLM, MiniLMConfig +from minilm_lab.provenance import ( + environment, + measure_peak_memory, + summarize, + synchronize, + write_record, +) + +ROOT = Path(__file__).resolve().parents[1] +DTYPES = {"float32": torch.float32, "float16": torch.float16, "bfloat16": torch.bfloat16} +# The architecture of the published science checkpoint. Only the context +# length (the position table) grows so that long sequences can be measured. +ARCHITECTURE = {"vocab_size": 256, "embedding_dim": 192, "num_layers": 4, "num_heads": 6} + + +def csv(kind): + return lambda text: [kind(item) for item in text.split(",")] + + +def build(args, context_length: int) -> MiniLM: + torch.manual_seed(args.seed) + config = MiniLMConfig(context_length=context_length, positional_encoding=args.position, **ARCHITECTURE) + return MiniLM(config).to(device=args.device, dtype=DTYPES[args.dtype]).eval() + + +def inputs(args, batch: int, length: int) -> torch.Tensor: + generator = torch.Generator().manual_seed(args.seed + 1000 * batch + length) + return torch.randint(0, ARCHITECTURE["vocab_size"], (batch, length), generator=generator).to(args.device) + + +def timed(device: str, operation): + synchronize(device) + start = time.perf_counter() + result = operation() + synchronize(device) + return time.perf_counter() - start, result + + +@torch.inference_mode() +def trial(model: MiniLM, prompt: torch.Tensor, decode_tokens: int, device: str, phases: set[str]) -> dict: + sample = {} + if "forward" in phases: + sample["forward"], _ = timed(device, lambda: model(prompt)) + if model.blocks[0].attention.backend == "mha": + # forward_cached uses SDPA under "mha"; there is no distinct decode path. + return sample + sample["prefill"], (logits, cache) = timed(device, lambda: model.forward_cached(prompt)) + steps = [] + for _ in range(decode_tokens): + token = logits[:, -1:].argmax(dim=-1) + elapsed, (logits, cache) = timed(device, lambda token=token, cache=cache: model.forward_cached(token, cache)) + steps.append(elapsed) + sample["decode_steps"] = steps + return sample + + +@torch.inference_mode() +def memory_probe(args) -> None: + """Run one forward pass in this fresh process and print peak extra memory.""" + model = build(args, args.context_length).set_attention_backend(args.backends[0]) + prompt = inputs(args, args.batch_sizes[0], args.seq_lens[0]) + # MPS: the driver grows in ~1 GiB heaps, so live allocator bytes are sampled per op. + extra, method = measure_peak_memory(args.device, lambda: model(prompt)) + print(json.dumps({"peak_extra_bytes": extra, "method": method})) + + +def measure_memory(args, backend: str, batch: int, length: int) -> dict: + command = [ + sys.executable, __file__, "--memory-probe", "--device", args.device, "--dtype", args.dtype, + "--position", args.position, "--seed", str(args.seed), "--backends", backend, + "--batch-sizes", str(batch), "--seq-lens", str(length), "--context-length", str(args.context_length), + ] + output = subprocess.run(command, capture_output=True, text=True, check=True, cwd=ROOT).stdout + return json.loads(output.strip().splitlines()[-1]) + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("--device", choices=("cpu", "mps", "cuda"), default="cpu") + parser.add_argument("--dtype", choices=tuple(DTYPES), default="float32") + parser.add_argument("--position", choices=("learned", "rope"), default="learned") + parser.add_argument("--backends", type=csv(str), default=["mha", "sdpa", "reference"]) + parser.add_argument("--seq-lens", type=csv(int), default=[128, 256, 512, 1024, 2048]) + parser.add_argument("--batch-sizes", type=csv(int), default=[1, 4, 8]) + parser.add_argument("--decode-tokens", type=int, default=32) + parser.add_argument("--warmup", type=int, default=3) + parser.add_argument("--repeats", type=int, default=10) + parser.add_argument("--seed", type=int, default=0) + parser.add_argument("--context-length", type=int, default=None, help="defaults to max seq len + decode tokens") + parser.add_argument("--skip-memory", action="store_true") + parser.add_argument("--memory-probe", action="store_true", help=argparse.SUPPRESS) + parser.add_argument("--output-dir", type=Path, default=ROOT / "experiments/records/attention") + parser.add_argument("--name", default=None) + args = parser.parse_args() + if args.context_length is None: + args.context_length = max(args.seq_lens) + args.decode_tokens + if args.memory_probe: + memory_probe(args) + return + if args.device == "mps" and not torch.backends.mps.is_available(): + parser.error("MPS is unavailable in this environment") + if args.device == "cuda" and not torch.cuda.is_available(): + parser.error("CUDA is unavailable in this environment") + if args.repeats < 3 or args.warmup < 1: + parser.error("use at least one warmup and three repeats") + + model = build(args, args.context_length) + results = [] + for batch in args.batch_sizes: + for length in args.seq_lens: + prompt = inputs(args, batch, length) + phases = {"forward"} + samples = {backend: [] for backend in args.backends} + for repeat in range(args.warmup + args.repeats): + shift = repeat % len(args.backends) + for backend in args.backends[shift:] + args.backends[:shift]: + model.set_attention_backend(backend) + sample = trial(model, prompt, args.decode_tokens, args.device, phases) + if repeat >= args.warmup: + samples[backend].append(sample) + for backend, runs in samples.items(): + row = {"backend": backend, "batch": batch, "seq_len": length, "samples_seconds": runs} + row["forward_ms"] = summarize([1000 * run["forward"] for run in runs]) + if "prefill" in runs[0]: + prefill = [run["prefill"] for run in runs] + decode_totals = [sum(run["decode_steps"]) for run in runs] + row["prefill_ms"] = summarize([1000 * value for value in prefill]) + row["decode_ms_per_step"] = summarize([1000 * step for run in runs for step in run["decode_steps"]]) + row["decode_tokens_per_second"] = summarize([batch * args.decode_tokens / total for total in decode_totals]) + row["generation_tokens_per_second"] = summarize( + [batch * args.decode_tokens / (p + d) for p, d in zip(prefill, decode_totals)] + ) + if not args.skip_memory: + row["memory"] = measure_memory(args, backend, batch, length) + results.append(row) + print( + f"{args.device}/{args.dtype} B={batch} L={length:5d} {backend:9s} " + f"forward {row['forward_ms']['median']:8.2f} ms" + + (f" prefill {row['prefill_ms']['median']:8.2f} ms decode {row['decode_ms_per_step']['median']:6.2f} ms/step" if "prefill_ms" in row else "") + + (f" mem {row['memory']['peak_extra_bytes'] / 2**20:8.1f} MiB" if "memory" in row else ""), + flush=True, + ) + payload = { + "experiment": "attention_backends", + "environment": environment(args.device), + "settings": { + "dtype": args.dtype, + "position": args.position, + "backends": args.backends, + "seq_lens": args.seq_lens, + "batch_sizes": args.batch_sizes, + "decode_tokens": args.decode_tokens, + "warmup": args.warmup, + "repeats": args.repeats, + "seed": args.seed, + "weights": "seeded random initialization (timing does not depend on trained values)", + "model_config": model.config.__dict__, + "inference_mode": True, + }, + "results": results, + } + name = args.name or f"attention_{args.device}_{args.dtype}" + print(f"wrote {write_record(args.output_dir, name, payload)}") + + +if __name__ == "__main__": + main() diff --git a/scripts/benchmark_generation.py b/scripts/benchmark_generation.py index b2e497f..0c5294b 100644 --- a/scripts/benchmark_generation.py +++ b/scripts/benchmark_generation.py @@ -6,7 +6,12 @@ import torch from minilm_lab.generation import generate -from minilm_lab.model import MiniLM, MiniLMConfig +from minilm_lab.model import ( + ATTENTION_BACKENDS, + DEFAULT_ATTENTION_BACKEND, + MiniLM, + MiniLMConfig, +) from minilm_lab.tokenizer import BPETokenizer, ByteTokenizer @@ -15,18 +20,23 @@ def main() -> None: parser.add_argument("--checkpoint", type=Path, default=Path("artifacts/best_checkpoint.pt")) parser.add_argument("--tokens", type=int, default=100) parser.add_argument("--output", type=Path, default=None) + parser.add_argument( + "--attention-backend", choices=(*ATTENTION_BACKENDS, "auto"), default=DEFAULT_ATTENTION_BACKEND, + help="sdpa (default); mha reproduces published science-v1 measurements", + ) args = parser.parse_args() device = "mps" if torch.backends.mps.is_available() else "cpu" - checkpoint = torch.load(args.checkpoint, map_location=device, weights_only=False) + checkpoint = torch.load(args.checkpoint, map_location=device, weights_only=True) model = MiniLM(MiniLMConfig(**checkpoint["config"])).to(device) model.load_state_dict(checkpoint["model"]) + model.set_attention_backend(args.attention_backend) tokenizer = ByteTokenizer() if checkpoint.get("tokenizer", "byte") == "byte" else BPETokenizer.load(checkpoint["tokenizer"]) start = time.perf_counter() generate(model, "To be or not to be", tokenizer, args.tokens) if device == "mps": torch.mps.synchronize() elapsed = time.perf_counter() - start - result = {"device": device, "generated_tokens": args.tokens, "seconds": elapsed, "tokens_per_second": args.tokens / elapsed} + result = {"device": device, "attention_backend": model.blocks[0].attention.backend, "generated_tokens": args.tokens, "seconds": elapsed, "tokens_per_second": args.tokens / elapsed} print(json.dumps(result, indent=2)) if args.output: args.output.parent.mkdir(parents=True, exist_ok=True) diff --git a/scripts/benchmark_kv_cache.py b/scripts/benchmark_kv_cache.py index 19e6747..56bbd10 100644 --- a/scripts/benchmark_kv_cache.py +++ b/scripts/benchmark_kv_cache.py @@ -8,7 +8,13 @@ from minilm_lab.generation import generate from minilm_lab.kv_cache import generate_with_kv_cache -from minilm_lab.model import MiniLM, MiniLMConfig +from minilm_lab.model import ( + ATTENTION_BACKENDS, + DEFAULT_ATTENTION_BACKEND, + MiniLM, + MiniLMConfig, +) +from minilm_lab.provenance import environment, file_sha256, write_record from minilm_lab.tokenizer import BPETokenizer, ByteTokenizer @@ -19,6 +25,11 @@ def main() -> None: parser.add_argument("--repeats", type=int, default=7) parser.add_argument("--device", choices=("auto", "cpu", "mps"), default="auto") parser.add_argument("--output", type=Path, default=None) + parser.add_argument("--record-dir", type=Path, default=None, help="also write an experiment record (experiments/records/legacy)") + parser.add_argument( + "--attention-backend", choices=(*ATTENTION_BACKENDS, "auto"), default=DEFAULT_ATTENTION_BACKEND, + help="sdpa (default); mha reproduces published science-v1 measurements", + ) args = parser.parse_args() if args.tokens < 1 or args.repeats < 3: parser.error("tokens must be positive and repeats must be at least three") @@ -26,6 +37,7 @@ def main() -> None: checkpoint = torch.load(args.checkpoint, map_location=device, weights_only=True) model = MiniLM(MiniLMConfig(**checkpoint["config"])).to(device) model.load_state_dict(checkpoint["model"]) + model.set_attention_backend(args.attention_backend) tokenizer = ByteTokenizer() if checkpoint.get("tokenizer", "byte") == "byte" else BPETokenizer.load(checkpoint["tokenizer"]) prompt = "The experiment shows " modes = { @@ -48,6 +60,7 @@ def main() -> None: cached = statistics.median(timings["cached"]) result = { "device": device, + "attention_backend": model.blocks[0].attention.backend, "generated_tokens": args.tokens, "repeats": args.repeats, "warmup_runs_per_mode": 1, @@ -64,6 +77,11 @@ def main() -> None: if args.output: args.output.parent.mkdir(parents=True, exist_ok=True) args.output.write_text(json.dumps(result, indent=2) + "\n", encoding="utf-8") + if args.record_dir: + settings = {"checkpoint": str(args.checkpoint), "checkpoint_sha256": file_sha256(args.checkpoint), "tokens": args.tokens, + "repeats": args.repeats, "warmup": 1, "seed": 1234, "attention_backend": result["attention_backend"]} + payload = {"experiment": "legacy_kv_cache_benchmark", "environment": environment(device), "settings": settings, "results": result} + print(f"wrote {write_record(args.record_dir, 'legacy_kv_cache_' + device, payload)}") if __name__ == "__main__": diff --git a/scripts/benchmark_kv_scaling.py b/scripts/benchmark_kv_scaling.py new file mode 100644 index 0000000..ad8c28b --- /dev/null +++ b/scripts/benchmark_kv_scaling.py @@ -0,0 +1,214 @@ +"""Cached vs uncached autoregressive decoding: latency, equivalence and memory growth. + +Modes (identical weights, inputs, dtype and device): + cached KV cache, SDPA attention (what forward_cached always uses) + uncached_sdpa full recompute each step, SDPA attention + uncached_mha full recompute each step, legacy MHA attention (records before + the first release call this mode uncached_default) +The two uncached modes separate the cache effect from the attention kernel. + +Every step (step 0 = prompt prefill) is synchronized and timed. The timed loop +is checked to emit exactly the tokens of minilm_lab.kv_cache.greedy_decode. + +Scenarios: + grid prompt lengths x batch sizes, all inside the context window + window generation far past the context window (sliding refresh) + memory cache bytes per step and, on MPS, live allocator peak per step +""" + +import argparse +import time +from pathlib import Path + +import torch +from torch.utils._python_dispatch import TorchDispatchMode + +from minilm_lab.kv_cache import greedy_decode +from minilm_lab.model import MiniLM, MiniLMConfig +from minilm_lab.provenance import environment, synchronize, write_record +from minilm_lab.records import kv_summary_from_raw + +ROOT = Path(__file__).resolve().parents[1] +DTYPES = {"float32": torch.float32, "float16": torch.float16, "bfloat16": torch.bfloat16} +ARCHITECTURE = {"vocab_size": 256, "embedding_dim": 192, "num_layers": 4, "num_heads": 6} +MODES = {"cached": ("sdpa", True), "uncached_sdpa": ("sdpa", False), "uncached_mha": ("mha", False)} + + +def csv(kind): + return lambda text: [kind(item) for item in text.split(",")] + + +def build(args, context_length: int) -> MiniLM: + torch.manual_seed(args.seed) + config = MiniLMConfig(context_length=context_length, positional_encoding=args.position, **ARCHITECTURE) + return MiniLM(config).to(device=args.device, dtype=DTYPES[args.dtype]).eval() + + +def prompt_for(args, batch: int, length: int) -> torch.Tensor: + generator = torch.Generator().manual_seed(args.seed + 1000 * batch + length) + return torch.randint(0, ARCHITECTURE["vocab_size"], (batch, length), generator=generator).to(args.device) + + +@torch.inference_mode() +def timed_decode(model: MiniLM, prompt: torch.Tensor, new_tokens: int, use_cache: bool, device: str, on_step=None): + """greedy_decode with per-step timing; on_step(index, cache) may inspect state.""" + window = model.config.context_length + tokens, cache, times, refreshed = prompt, None, [], [] + for index in range(new_tokens): + synchronize(device) + start = time.perf_counter() + if not use_cache: + logits, _ = model(tokens[:, -window:]) + refresh = True + elif cache is None or tokens.size(1) > window: + logits, cache = model.forward_cached(tokens[:, -window:]) + refresh = True + else: + logits, cache = model.forward_cached(tokens[:, -1:], cache) + refresh = False + tokens = torch.cat((tokens, logits[:, -1].argmax(dim=-1, keepdim=True)), dim=1) + synchronize(device) + times.append(time.perf_counter() - start) + refreshed.append(refresh) + if on_step: + on_step(index, cache) + return tokens, times, refreshed + + +def run_modes(args, model: MiniLM, prompt: torch.Tensor, new_tokens: int) -> dict: + samples = {mode: [] for mode in MODES} + outputs = {} + names = list(MODES) + for repeat in range(args.warmup + args.repeats): + shift = repeat % len(names) + for mode in names[shift:] + names[:shift]: + backend, use_cache = MODES[mode] + model.set_attention_backend(backend) + tokens, times, refreshed = timed_decode(model, prompt, new_tokens, use_cache, args.device) + outputs[mode] = tokens + if repeat >= args.warmup: + samples[mode].append({"step_seconds": times, "refreshed": refreshed}) + model.set_attention_backend("sdpa") + reference = greedy_decode(model, prompt, new_tokens, use_cache=True) + if not torch.equal(reference, outputs["cached"]): + raise AssertionError("timed loop diverged from greedy_decode") + batch = prompt.size(0) + result = {"token_identical_to_cached": {mode: bool(torch.equal(outputs[mode], outputs["cached"])) for mode in MODES}} + for mode, runs in samples.items(): + # Raw per-repeat, per-step timings are kept; every summary is derived from them. + raw = {"unit": "seconds", "step_seconds": [run["step_seconds"] for run in runs], + "refreshed_steps_per_repeat": [run["refreshed"] for run in runs]} + result[mode] = {**kv_summary_from_raw(raw, batch, new_tokens), "refreshed_steps": runs[0]["refreshed"], "raw": raw} + result["speedup_cached_vs_uncached_sdpa"] = result["uncached_sdpa"]["total_ms"]["median"] / result["cached"]["total_ms"]["median"] + result["speedup_cached_vs_uncached_mha"] = result["uncached_mha"]["total_ms"]["median"] / result["cached"]["total_ms"]["median"] + return result + + +class PeakAllocated(TorchDispatchMode): + def __init__(self) -> None: + super().__init__() + self.peak = torch.mps.current_allocated_memory() + + def __torch_dispatch__(self, func, types, args=(), kwargs=None): + output = func(*args, **(kwargs or {})) + self.peak = max(self.peak, torch.mps.current_allocated_memory()) + return output + + +@torch.inference_mode() +def memory_growth(args, model: MiniLM, prompt: torch.Tensor, new_tokens: int) -> dict: + model.set_attention_backend("sdpa") + window = model.config.context_length + tokens, cache, rows = prompt, None, [] + for index in range(new_tokens): + tracker = PeakAllocated() if args.device == "mps" else None + before = torch.mps.current_allocated_memory() if tracker else None + if tracker: + tracker.__enter__() + if cache is None or tokens.size(1) > window: + logits, cache = model.forward_cached(tokens[:, -window:]) + else: + logits, cache = model.forward_cached(tokens[:, -1:], cache) + if tracker: + tracker.__exit__(None, None, None) + tokens = torch.cat((tokens, logits[:, -1].argmax(dim=-1, keepdim=True)), dim=1) + row = { + "step": index, + "cached_positions": cache[0][0].size(2), + "cache_bytes": sum(t.numel() * t.element_size() for layer in cache for t in layer), + } + if tracker: + row["step_peak_minus_start_bytes"] = tracker.peak - before + rows.append(row) + return {"batch": prompt.size(0), "prompt_len": prompt.size(1), "steps": rows} + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("--device", choices=("cpu", "mps", "cuda"), default="cpu") + parser.add_argument("--dtype", choices=tuple(DTYPES), default="float32") + parser.add_argument("--position", choices=("learned", "rope"), default="learned") + parser.add_argument("--prompt-lens", type=csv(int), default=[1, 16, 128, 512, 1024]) + parser.add_argument("--batch-sizes", type=csv(int), default=[1, 8]) + parser.add_argument("--new-tokens", type=int, default=64) + parser.add_argument("--window-context", type=int, default=128, help="context for the sliding-window scenario") + parser.add_argument("--window-prompt", type=int, default=32) + parser.add_argument("--window-new-tokens", type=int, default=224) + parser.add_argument("--warmup", type=int, default=1) + parser.add_argument("--repeats", type=int, default=5) + parser.add_argument("--seed", type=int, default=0) + parser.add_argument("--output-dir", type=Path, default=ROOT / "experiments/records/kv_cache") + args = parser.parse_args() + if args.device == "mps" and not torch.backends.mps.is_available(): + parser.error("MPS is unavailable in this environment") + if args.repeats < 3 or args.warmup < 1: + parser.error("use at least one warmup and three repeats") + + context = max(args.prompt_lens) + args.new_tokens + model = build(args, context) + grid = [] + for batch in args.batch_sizes: + for length in args.prompt_lens: + row = {"batch": batch, "prompt_len": length, **run_modes(args, model, prompt_for(args, batch, length), args.new_tokens)} + grid.append(row) + print( + f"{args.device}/{args.dtype} B={batch} L={length:5d} cached {row['cached']['total_ms']['median']:9.1f} ms " + f"uncached_sdpa {row['uncached_sdpa']['total_ms']['median']:9.1f} ms uncached_mha {row['uncached_mha']['total_ms']['median']:9.1f} ms " + f"identical={row['token_identical_to_cached']}", + flush=True, + ) + window_model = build(args, args.window_context) + window = {"batch": 1, "prompt_len": args.window_prompt, "context_length": args.window_context, "new_tokens": args.window_new_tokens} + window.update(run_modes(args, window_model, prompt_for(args, 1, args.window_prompt), args.window_new_tokens)) + print(f"window: cached {window['cached']['total_ms']['median']:.1f} ms vs uncached_sdpa {window['uncached_sdpa']['total_ms']['median']:.1f} ms", flush=True) + memory_prompt = min(128, context - args.new_tokens) + memory = [memory_growth(args, model, prompt_for(args, batch, memory_prompt), args.new_tokens) for batch in args.batch_sizes] + memory.append(memory_growth(args, window_model, prompt_for(args, 1, args.window_prompt), args.window_new_tokens)) + payload = { + "experiment": "kv_cache_scaling", + "environment": environment(args.device), + "settings": { + "dtype": args.dtype, + "position": args.position, + "prompt_lens": args.prompt_lens, + "batch_sizes": args.batch_sizes, + "new_tokens": args.new_tokens, + "warmup": args.warmup, + "repeats": args.repeats, + "seed": args.seed, + "modes": {mode: {"attention_backend": b, "use_cache": c} for mode, (b, c) in MODES.items()}, + "weights": "seeded random initialization (timing does not depend on trained values)", + "grid_model_config": model.config.__dict__, + "window_model_config": window_model.config.__dict__, + "decoding": "greedy argmax; step 0 is the prompt prefill", + "timer": "time.perf_counter", + "synchronization": "device synchronize before and after every step (no-op on CPU); backend/mode order rotated each repeat", + "units": {"raw": "seconds per step, one list per measured repeat", "summaries": "milliseconds (ms fields) or tokens/second"}, + }, + "results": {"grid": grid, "window": window, "memory": memory}, + } + print(f"wrote {write_record(args.output_dir, f'kv_{args.device}_{args.dtype}', payload)}") + + +if __name__ == "__main__": + main() diff --git a/scripts/benchmark_training_step.py b/scripts/benchmark_training_step.py new file mode 100644 index 0000000..bb570ce --- /dev/null +++ b/scripts/benchmark_training_step.py @@ -0,0 +1,136 @@ +"""Matched training-step throughput and memory for each attention backend. + +One step = forward + cross-entropy + backward + AdamW update on a batch of +random tokens. All backends share the same seeded initial weights and the +same batches; each backend has its own model copy and optimizer so the +weights stay comparable. Backend order rotates every repeat. Peak memory is +measured on the first step in a fresh process per configuration with +provenance.measure_peak_memory (CPU: host peak-RSS delta; MPS: live allocator +bytes sampled after every op, including backward; CUDA: device allocator peak); +all include AdamW state allocation, which is identical across backends. +""" + +import argparse +import copy +import json +import subprocess +import sys +import time +from pathlib import Path + +import torch + +from minilm_lab.model import MiniLM, MiniLMConfig +from minilm_lab.provenance import ( + environment, + measure_peak_memory, + summarize, + synchronize, + write_record, +) + +ROOT = Path(__file__).resolve().parents[1] +ARCHITECTURE = {"vocab_size": 256, "embedding_dim": 192, "num_layers": 4, "num_heads": 6} + + +def csv(kind): + return lambda text: [kind(item) for item in text.split(",")] + + +def build(args, length: int) -> MiniLM: + torch.manual_seed(args.seed) + return MiniLM(MiniLMConfig(context_length=length, **ARCHITECTURE)).to(args.device).train() + + +def batch_for(args, length: int, index: int) -> tuple[torch.Tensor, torch.Tensor]: + generator = torch.Generator().manual_seed(args.seed + 7919 * index + length) + tokens = torch.randint(0, ARCHITECTURE["vocab_size"], (args.batch_size, length + 1), generator=generator).to(args.device) + return tokens[:, :-1], tokens[:, 1:] + + +def step(model: MiniLM, optimizer: torch.optim.Optimizer, inputs: torch.Tensor, targets: torch.Tensor) -> float: + _, loss = model(inputs, targets) + loss.backward() + optimizer.step() + optimizer.zero_grad(set_to_none=True) + return loss.item() + + +def memory_probe(args) -> None: + length = args.seq_lens[0] + model = build(args, length).set_attention_backend(args.backends[0]) + optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4) + # The first step is measured: a warm-up step would already set the process + # RSS high-water mark. It includes AdamW state allocation (same for all backends). + inputs, targets = batch_for(args, length, 0) + extra, method = measure_peak_memory(args.device, lambda: step(model, optimizer, inputs, targets)) + print(json.dumps({"peak_extra_bytes": extra, "method": method})) + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("--device", choices=("cpu", "mps", "cuda"), default="cpu") + parser.add_argument("--backends", type=csv(str), default=["mha", "sdpa", "reference"]) + parser.add_argument("--seq-lens", type=csv(int), default=[128, 256, 512, 1024]) + parser.add_argument("--batch-size", type=int, default=8) + parser.add_argument("--warmup", type=int, default=3) + parser.add_argument("--repeats", type=int, default=10) + parser.add_argument("--seed", type=int, default=0) + parser.add_argument("--memory-probe", action="store_true", help=argparse.SUPPRESS) + parser.add_argument("--output-dir", type=Path, default=ROOT / "experiments/records/training_step") + args = parser.parse_args() + if args.memory_probe: + memory_probe(args) + return + if args.device == "mps" and not torch.backends.mps.is_available(): + parser.error("MPS is unavailable in this environment") + results = [] + for length in args.seq_lens: + base = build(args, length) + models = {backend: copy.deepcopy(base).set_attention_backend(backend) for backend in args.backends} + optimizers = {backend: torch.optim.AdamW(model.parameters(), lr=3e-4) for backend, model in models.items()} + samples = {backend: [] for backend in args.backends} + losses = {backend: [] for backend in args.backends} + for repeat in range(args.warmup + args.repeats): + inputs, targets = batch_for(args, length, repeat) + shift = repeat % len(args.backends) + for backend in args.backends[shift:] + args.backends[:shift]: + synchronize(args.device) + start = time.perf_counter() + loss = step(models[backend], optimizers[backend], inputs, targets) + synchronize(args.device) + elapsed = time.perf_counter() - start + losses[backend].append(loss) + if repeat >= args.warmup: + samples[backend].append(elapsed) + for backend in args.backends: + command = [sys.executable, __file__, "--memory-probe", "--device", args.device, "--backends", backend, + "--seq-lens", str(length), "--batch-size", str(args.batch_size), "--seed", str(args.seed)] + memory = json.loads(subprocess.run(command, capture_output=True, text=True, check=True, cwd=ROOT).stdout.strip().splitlines()[-1]) + tokens = args.batch_size * length + row = { + "backend": backend, "seq_len": length, "batch": args.batch_size, + "step_ms": summarize([1000 * s for s in samples[backend]]), + "tokens_per_second": summarize([tokens / s for s in samples[backend]]), + "step_seconds_samples": samples[backend], + "losses": losses[backend], + "max_abs_loss_difference_vs_first_backend": max(abs(a - b) for a, b in zip(losses[backend], losses[args.backends[0]])), + "memory": memory, + } + results.append(row) + print(f"{args.device} L={length:5d} {backend:9s} step {row['step_ms']['median']:8.1f} ms " + f"{row['tokens_per_second']['median']:8.0f} tok/s mem {memory['peak_extra_bytes'] / 2**20:8.1f} MiB " + f"max|Δloss| {row['max_abs_loss_difference_vs_first_backend']:.2e}", flush=True) + payload = { + "experiment": "training_step", + "environment": environment(args.device), + "settings": {"dtype": "float32", "backends": args.backends, "seq_lens": args.seq_lens, "batch_size": args.batch_size, + "warmup": args.warmup, "repeats": args.repeats, "seed": args.seed, "optimizer": "AdamW lr=3e-4", + "architecture": ARCHITECTURE, "weights": "seeded random initialization, identical across backends"}, + "results": results, + } + print(f"wrote {write_record(args.output_dir, f'training_step_{args.device}_float32', payload)}") + + +if __name__ == "__main__": + main() diff --git a/scripts/build_experiment_report.py b/scripts/build_experiment_report.py new file mode 100644 index 0000000..005c9f6 --- /dev/null +++ b/scripts/build_experiment_report.py @@ -0,0 +1,535 @@ +"""Build attention-experiment tables and SVG figures from recorded runs only. + +Tables are written between and + markers in docs/EXPERIMENT_RESULTS.md, so the +hand-written analysis stays put while every number comes from a record. +""" + +import html +import json +import math +import re +from pathlib import Path + +from minilm_lab.model import canonical_backend +from minilm_lab.records import ( + NAME, + kv_speedups, + load_store, + protocol_rows, + validate_store, +) + +ROOT = Path(__file__).resolve().parents[1] +RECORDS = ROOT / "experiments/records" +REPORT = ROOT / "docs/EXPERIMENT_RESULTS.md" +FIGURES = ROOT / "docs/figures" +BG, PANEL, EDGE = "#0b1018", "#151d29", "#2c394b" +INK, MUTED = "#f4f7fb", "#a9b8c9" +# Validated categorical slots 1-3 (dark steps) against the panel surface. +SERIES = ("#3987e5", "#d95926", "#199e70") +BACKEND_LABEL = {"mha": "MHA (legacy v1)", "sdpa": "SDPA", "reference": "Reference"} + + +def canon(backend: str | None) -> str: + """Records written before the first release name the legacy MHA path 'auto'.""" + return canonical_backend(backend) +# Drawn dashed (and last) so a series that coincides with another stays visible. +DASHED = {"MHA (legacy v1)"} +TARGET_ORDER = ("cpu/float32", "mps/float32", "mps/float16", "mps/bfloat16") + + +def current_records(directory: Path) -> list[dict]: + """Records in time order, excluding superseded ones (which stay on disk).""" + return [record for _, record in load_store(ROOT, directory.parent).current(directory.name)] + + +def latest(directory: Path) -> dict[str, dict]: + """Latest current benchmark record per device/dtype (benchmarks are re-measurements).""" + chosen = {} + for record in current_records(directory): + key = f"{record['environment']['device']}/{record['settings'].get('dtype', 'mixed')}" + chosen[key] = record + return dict(sorted(chosen.items(), key=lambda item: TARGET_ORDER.index(item[0]) if item[0] in TARGET_ORDER else 99)) + + +def rows_by(record: dict) -> dict[tuple, dict]: + return {(canon(row["backend"]), row["batch"], row["seq_len"]): row for row in record["results"]} + + +def mib(row: dict) -> str: + return f"{row['memory']['peak_extra_bytes'] / 2**20:.1f}" if "memory" in row else "–" + + +def environment_table(records: list[dict]) -> str: + lines = ["| Record | Device | Dtype | Processor | torch | Python | Threads | Commit | Dirty |", "| --- | --- | --- | --- | --- | --- | ---: | --- | --- |"] + for record in records: + env = record["environment"] + lines.append( + f"| `{Path(record['_path']).name}` | {env['device']} | {record['settings'].get('dtype', 'mixed')} | {env['processor']} " + f"| {env['torch']} | {env['python']} | {env['torch_threads']} | `{(env['git']['commit'] or '')[:7]}` | {env['git']['dirty']} |" + ) + return "\n".join(lines) + + +def forward_table(key: str, record: dict) -> str: + rows = rows_by(record) + settings = record["settings"] + lines = [ + f"**{key}** · warmup {settings['warmup']}, {settings['repeats']} repeats · median ms (p95) · peak extra MiB per forward", + "", + "| B | L | MHA (legacy v1) | SDPA | Reference | Ref / SDPA | MHA MiB | SDPA MiB | Ref MiB |", + "| ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |", + ] + for batch in settings["batch_sizes"]: + for length in settings["seq_lens"]: + cells = [rows.get((backend, batch, length)) for backend in ("mha", "sdpa", "reference")] + if not all(cells): + continue + timing = [f"{c['forward_ms']['median']:.2f} ({c['forward_ms']['p95']:.2f})" for c in cells] + ratio = cells[2]["forward_ms"]["median"] / cells[1]["forward_ms"]["median"] + lines.append(f"| {batch} | {length} | {' | '.join(timing)} | {ratio:.2f}× | {' | '.join(mib(c) for c in cells)} |") + return "\n".join(lines) + + +def decode_table(key: str, record: dict) -> str: + rows = rows_by(record) + settings = record["settings"] + lines = [ + f"**{key}** · {settings['decode_tokens']} greedy decode steps after an L-token prefill · median (p95)", + "", + "| B | L | SDPA prefill ms | Ref prefill ms | SDPA decode ms/step | Ref decode ms/step | SDPA gen tok/s | Ref gen tok/s |", + "| ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |", + ] + for batch in settings["batch_sizes"]: + for length in settings["seq_lens"]: + s, r = rows.get(("sdpa", batch, length)), rows.get(("reference", batch, length)) + if not (s and r): + continue + fmt = lambda stat, digits=2: f"{stat['median']:.{digits}f} ({stat['p95']:.{digits}f})" + lines.append( + f"| {batch} | {length} | {fmt(s['prefill_ms'])} | {fmt(r['prefill_ms'])} | {fmt(s['decode_ms_per_step'])} " + f"| {fmt(r['decode_ms_per_step'])} | {s['generation_tokens_per_second']['median']:.0f} | {r['generation_tokens_per_second']['median']:.0f} |" + ) + return "\n".join(lines) + + +def variability_table(records: dict[str, dict]) -> str: + lines = ["| Target | Phase | Median CV | Worst CV | Worst config |", "| --- | --- | ---: | ---: | --- |"] + for key, record in records.items(): + for phase in ("forward_ms", "prefill_ms", "decode_ms_per_step"): + values = [(row[phase]["cv"], f"{canon(row['backend'])} B={row['batch']} L={row['seq_len']}") for row in record["results"] if phase in row] + if values: + ordered = sorted(values) + lines.append(f"| {key} | {phase} | {ordered[len(ordered) // 2][0]:.3f} | {ordered[-1][0]:.3f} | {ordered[-1][1]} |") + return "\n".join(lines) + + +def quality_table(record: dict) -> str: + settings = record["settings"] + lines = [ + ( + f"Oracle: {settings['oracle']} (RMSNorm variance and RoPE angles still run in float32); " + f"{settings['validation_targets']:,} held-out targets; " + f"greedy: {settings['greedy']['prompts']} prompts × {settings['greedy']['generated_tokens']} tokens." + ), + "", + "| Seed | Target | Backend | max abs Δlogit | mean abs Δlogit | top-1 agree | Perplexity | Greedy identical |", + "| ---: | --- | --- | ---: | ---: | ---: | ---: | ---: |", + ] + for row in record["results"]: + lines.append( + f"| {row['seed']} | {row['device']}/{row['dtype']} | {BACKEND_LABEL[canon(row['backend'])]} | {row['max_abs_logit_error']:.2e} " + f"| {row['mean_abs_logit_error']:.2e} | {100 * row['top1_agreement']:.3f}% | {row['perplexity']:.4f} " + f"| {row['greedy_identical_sequences']}/{row['greedy_prompts']} |" + ) + if record["settings"].get("skipped_targets"): + lines.append(f"\nSkipped (unavailable): {', '.join(record['settings']['skipped_targets'])}") + return "\n".join(lines) + + +def seed_table(record: dict) -> str: + rows = [r for r in record["results"] if r["device"] == "cpu" and r["dtype"] == "float32" and canon(r["backend"]) == "mha"] + if not rows: + return "_No CPU float32 legacy-MHA rows recorded._" + values = [r["perplexity"] for r in rows] + mean = sum(values) / len(values) + std = math.sqrt(sum((v - mean) ** 2 for v in values) / (len(values) - 1)) if len(values) > 1 else 0.0 + lines = ["| Seed | Held-out perplexity | Bits per byte |", "| ---: | ---: | ---: |"] + lines += [f"| {r['seed']} | {r['perplexity']:.3f} | {r['bits_per_byte']:.4f} |" for r in rows] + lines.append(f"| **mean ± sd (n={len(values)})** | **{mean:.3f} ± {std:.3f}** | |") + return "\n".join(lines) + + +def svg(title: str, description: str, width: int, height: int, body: str) -> str: + return f""" +{html.escape(title)}{html.escape(description)} +{body} +""" + + +def log_panel(x0: float, y0: float, w: float, h: float, heading: str, series: list[tuple], y_label: str, y_fmt, reference_line: float | None = None, x_label: str = "sequence length (tokens)") -> str: + """Line panel with log2 x (sequence length) and log10 y. series: (label, color, [(x, y, tip)]).""" + points = [p for _, _, pts in series for p in pts] + if not points: + return "" + xs = sorted({p[0] for p in points}) + low = min(p[1] for p in points + ([(0, reference_line)] if reference_line else [])) + high = max(p[1] for p in points + ([(0, reference_line)] if reference_line else [])) + y_low, y_high = math.floor(math.log10(low) * 4) / 4, math.ceil(math.log10(high) * 4) / 4 + if y_high - y_low < 0.5: + y_low, y_high = y_low - 0.25, y_high + 0.25 + left, right, top, bottom = x0 + 58, x0 + w - 14, y0 + 40, y0 + h - 34 + + def px(x: float) -> float: + return left + (math.log2(x) - math.log2(xs[0])) / max(math.log2(xs[-1]) - math.log2(xs[0]), 1) * (right - left) + + def py(y: float) -> float: + return bottom - (math.log10(y) - y_low) / (y_high - y_low) * (bottom - top) + + body = f'{html.escape(heading)}' + ticks = [] + exponent = math.floor(y_low) + while exponent <= y_high: + for mantissa in (1, 2, 3, 5) if y_high - y_low <= 1 else (1, 2, 5): + value = mantissa * 10**exponent + if y_low <= math.log10(value) <= y_high: + ticks.append(value) + exponent += 1 + for value in ticks: + y = py(value) + body += f'{y_fmt(value)}' + if reference_line: + y = py(reference_line) + body += f'' + for x in xs: + body += f'{x}' + body += f'{x_label}' + body += f'{html.escape(y_label)}' + for label, color, pts in series: + coords = " ".join(f"{px(x):.1f},{py(y):.1f}" for x, y, _ in pts) + dash = ' stroke-dasharray="6 5"' if label in DASHED else "" + body += f'' + for x, y, tip in pts: + body += f'{html.escape(tip)}' + return body + + +def legend(x: float, y: float, entries: list[tuple[str, str]]) -> str: + body = "" + for label, color in entries: + dash = ' stroke-dasharray="6 5"' if label in DASHED else "" + body += f'{html.escape(label)}' + x += 56 + 7.2 * len(label) + return body + + +def grid_figure(name: str, eyebrow: str, title: str, subtitle: str, footer: str, panels: list[tuple], legend_entries: list[tuple[str, str]]) -> tuple[str, str]: + columns = 2 if len(panels) > 1 else 1 + width, panel_w, panel_h = 944, (944 - 42 * 2 - 20 * (columns - 1)) / columns, 300 + rows = math.ceil(len(panels) / columns) + height = 170 + rows * (panel_h + 20) + 40 + body = f'{html.escape(eyebrow)}{html.escape(title)}{html.escape(subtitle)}' + body += legend(42, 146, legend_entries) + for index, panel in enumerate(panels): + x = 42 + (index % columns) * (panel_w + 20) + y = 166 + (index // columns) * (panel_h + 20) + body += log_panel(x, y, panel_w, panel_h, *panel) + body += f'{html.escape(footer)}' + return name, svg(title, f"{subtitle} {footer}", width, height, body) + + +def figures(records: dict[str, dict]) -> list[tuple[str, str]]: + ratio_panels, memory_panels, decode_panels = [], [], [] + for key, record in records.items(): + rows, settings = rows_by(record), record["settings"] + batches = settings["batch_sizes"] + ratio_series = [] + for color, batch in zip(SERIES, batches): + pts = [] + for length in settings["seq_lens"]: + s, r = rows.get(("sdpa", batch, length)), rows.get(("reference", batch, length)) + if s and r: + value = r["forward_ms"]["median"] / s["forward_ms"]["median"] + pts.append((length, value, f"{key} B={batch} L={length}: reference/SDPA = {value:.2f}x")) + ratio_series.append((f"batch {batch}", color, pts)) + ratio_panels.append((key, ratio_series, "reference ÷ SDPA forward time", lambda v: f"{v:g}×", 1.0)) + big = max(batches) + memory_series = [] + for color, backend in sorted(zip(SERIES, ("mha", "sdpa", "reference")), key=lambda item: item[1] == "mha"): + pts = [ + (length, rows[(backend, big, length)]["memory"]["peak_extra_bytes"] / 2**20, + f"{key} {BACKEND_LABEL[backend]} B={big} L={length}: {rows[(backend, big, length)]['memory']['peak_extra_bytes'] / 2**20:.1f} MiB") + for length in settings["seq_lens"] + if (backend, big, length) in rows and "memory" in rows[(backend, big, length)] and rows[(backend, big, length)]["memory"]["peak_extra_bytes"] > 0 + ] + memory_series.append((BACKEND_LABEL[backend], color, pts)) + memory_panels.append((f"{key} · batch {big}", memory_series, "peak extra memory (MiB)", lambda v: f"{v:g}")) + decode_series = [] + for color, backend in zip(SERIES[1:], ("sdpa", "reference")): + pts = [ + (length, rows[(backend, 1, length)]["decode_ms_per_step"]["median"], + f"{key} {BACKEND_LABEL[backend]} B=1 L={length}: {rows[(backend, 1, length)]['decode_ms_per_step']['median']:.2f} ms/step") + for length in settings["seq_lens"] if (backend, 1, length) in rows + ] + decode_series.append((BACKEND_LABEL[backend], color, pts)) + decode_panels.append((f"{key} · batch 1", decode_series, "decode ms per token (median)", lambda v: f"{v:g}")) + backends = [(BACKEND_LABEL[b], c) for c, b in zip(SERIES, ("mha", "sdpa", "reference"))] + some = next(iter(records.values()))["settings"] + return [ + grid_figure( + "attention_speedup.svg", "EXPERIMENT 02 / ATTENTION", "How much faster is fused SDPA than explicit attention?", + "Full-sequence forward time, reference divided by SDPA (median). Above the dashed line, SDPA is faster.", + f"Same seeded weights, inputs, dtype and device per panel · {some['repeats']} repeats after {some['warmup']} warmups · source: experiments/records/attention", + ratio_panels, [(f"batch {b}", c) for c, b in zip(SERIES, some["batch_sizes"])], + ), + grid_figure( + "attention_memory.svg", "EXPERIMENT 02 / MEMORY", "Explicit attention pays for the full score matrix.", + "Peak extra memory during one forward pass, fresh process per point. Log scales on both axes.", + "CPU: process peak-RSS delta. MPS: live allocator bytes sampled after every op. Dashed legacy MHA coincides with SDPA on MPS.", + memory_panels, backends, + ), + grid_figure( + "attention_decode.svg", "EXPERIMENT 02 / DECODE", "Single-token decode against a KV cache.", + "Median time per generated token at batch 1 as the cached context grows.", + f"{some['decode_tokens']} greedy steps per trial, each step synchronized and timed · source: experiments/records/attention", + decode_panels, backends[1:], + ), + ] + + +def latest_by_name(directory: Path) -> dict[str, dict]: + """Latest current record per record name (the part between timestamp and commit).""" + return {NAME.match(Path(record["_key"]).name)["name"]: record for record in current_records(directory)} + + +def mean_sd(values: list[float]) -> str: + mean = sum(values) / len(values) + sd = math.sqrt(sum((v - mean) ** 2 for v in values) / (len(values) - 1)) if len(values) > 1 else 0.0 + return f"{mean:.3f} ± {sd:.3f}" + + +def mha_mode(row: dict) -> dict: + """Uncached legacy-MHA mode; named uncached_default in records before the first release.""" + return row.get("uncached_mha") or row["uncached_default"] + + +def sdpa_speedup(row: dict) -> float: + """Derived from the timings (records.kv_speedups), never read from the stored field.""" + return kv_speedups(row)["speedup_cached_vs_uncached_sdpa"] + + +def mha_speedup(row: dict) -> float: + ratios = kv_speedups(row) + return ratios.get("speedup_cached_vs_uncached_mha", ratios.get("speedup_cached_vs_uncached_default")) + + +def kv_table(key: str, record: dict) -> str: + results, settings = record["results"], record["settings"] + lines = [ + f"**{key}** · {settings['new_tokens']} greedy tokens · median total ms over {settings['repeats']} repeats · step 0 is the prefill", + "", + "| B | Prompt | Cached | Uncached (SDPA) | Uncached (MHA) | Cache speedup vs SDPA | vs MHA | Cached decode ms/step | Tokens identical |", + "| ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | :---: |", + ] + for row in results["grid"]: + same = "yes" if all(row["token_identical_to_cached"].values()) else "**no**" + lines.append( + f"| {row['batch']} | {row['prompt_len']} | {row['cached']['total_ms']['median']:.1f} | {row['uncached_sdpa']['total_ms']['median']:.1f} " + f"| {mha_mode(row)['total_ms']['median']:.1f} | {sdpa_speedup(row):.2f}× | {mha_speedup(row):.2f}× " + f"| {row['cached']['decode_ms_per_step']['median']:.2f} | {same} |" + ) + window = results["window"] + refreshed = sum(window["cached"]["refreshed_steps"]) + lines += [ + "", + (f"Past the window (context {window['context_length']}, prompt {window['prompt_len']}, {window['new_tokens']} tokens): " + f"{refreshed} of {window['new_tokens']} cached steps rebuild the cache; cached {window['cached']['total_ms']['median']:.1f} ms vs " + f"uncached SDPA {window['uncached_sdpa']['total_ms']['median']:.1f} ms ({sdpa_speedup(window):.2f}×); " + f"tokens identical: {'yes' if all(window['token_identical_to_cached'].values()) else 'no'}."), + ] + return "\n".join(lines) + + +def kv_memory_table(records: dict[str, dict]) -> str: + lines = ["| Target | B | Cached positions | Cache bytes | Bytes per position per sequence | Live-allocator peak during step (MPS) |", "| --- | ---: | ---: | ---: | ---: | ---: |"] + for key, record in records.items(): + for growth in record["results"]["memory"][:-1]: + for step in (growth["steps"][1], growth["steps"][-1]): + per = step["cache_bytes"] / step["cached_positions"] / growth["batch"] + peak = f"{step['step_peak_minus_start_bytes'] / 2**20:.2f} MiB" if "step_peak_minus_start_bytes" in step else "–" + lines.append(f"| {key} | {growth['batch']} | {step['cached_positions']} | {step['cache_bytes'] / 2**20:.2f} MiB | {per:.0f} | {peak} |") + return "\n".join(lines) + + +def sdpa_training_table(rows: list[dict]) -> str: + lines = [ + "| Seed | Backend | Selected step | Val loss | Val perplexity | Train tok/s | Train s | Peak RSS MiB | Eval s (CPU) | Eval s (MPS) |", + "| ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |", + ] + for row in rows: + cpu, mps = row.get("sdpa_v1_eval_cpu", {}), row.get("sdpa_v1_eval_mps", {}) + validation = cpu.get("validation") or {} + lines.append( + f"| {row['seed']} | {BACKEND_LABEL[row['backend']]} | {row['best_step']} | {validation.get('validation_loss', float('nan')):.4f} " + f"| {validation.get('perplexity', float('nan')):.3f} | {row['tokens_per_second']:.0f} | {row['train_seconds']:.1f} | {row['peak_rss_mib']:.0f} " + f"| {cpu.get('validation', {}).get('eval_seconds', float('nan')):.2f} | {mps.get('validation', {}).get('eval_seconds', float('nan')):.2f} |" + ) + lines += ["", "| Backend | n | Val perplexity mean ± sd | Train tok/s mean ± sd | Peak RSS MiB mean ± sd | CPU eval s mean ± sd |", "| --- | ---: | ---: | ---: | ---: | ---: |"] + for backend in ("mha", "sdpa"): + group = [r for r in rows if r["backend"] == backend and "sdpa_v1_eval_cpu" in r] + if group: + lines.append( + f"| {BACKEND_LABEL[backend]} | {len(group)} | {mean_sd([r['sdpa_v1_eval_cpu']['validation']['perplexity'] for r in group])} " + f"| {mean_sd([r['tokens_per_second'] for r in group])} | {mean_sd([r['peak_rss_mib'] for r in group])} " + f"| {mean_sd([r['sdpa_v1_eval_cpu']['validation']['eval_seconds'] for r in group])} |" + ) + paired = [] + for seed in sorted({r["seed"] for r in rows}): + pair = {r["backend"]: r for r in rows if r["seed"] == seed and "sdpa_v1_eval_cpu" in r} + if {"mha", "sdpa"} <= pair.keys(): + paired.append((seed, pair["sdpa"]["sdpa_v1_eval_cpu"]["validation"]["perplexity"] - pair["mha"]["sdpa_v1_eval_cpu"]["validation"]["perplexity"], + pair["sdpa"]["tokens_per_second"] / pair["mha"]["tokens_per_second"])) + if paired: + lines += ["", "| Seed | SDPA − MHA val perplexity | SDPA / MHA train tok/s |", "| ---: | ---: | ---: |"] + lines += [f"| {seed} | {delta:+.3f} | {ratio:.3f}× |" for seed, delta, ratio in paired] + return "\n".join(lines) + + +def v2_table(rows: list[dict]) -> str: + lines = [ + "| Seed | Backend | Selected step (on validation) | Val perplexity | Val bits/byte | **Test perplexity** | **Test bits/byte** | Darwin val ppl | Einstein val ppl |", + "| ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |", + ] + for row in rows: + s = row.get("v2_final_cpu") + if not s: + continue + lines.append( + f"| {row['seed']} | {BACKEND_LABEL[row['backend']]} | {row['best_step']} | {s['validation']['perplexity']:.3f} | {s['validation']['bits_per_byte']:.4f} " + f"| **{s['test']['perplexity']:.3f}** | **{s['test']['bits_per_byte']:.4f}** | {s['darwin_validation']['perplexity']:.3f} | {s['einstein_validation']['perplexity']:.3f} |" + ) + lines += ["", "| Backend | n | Val perplexity mean ± sd | Test perplexity mean ± sd |", "| --- | ---: | ---: | ---: |"] + for backend in ("mha", "sdpa"): + group = [r for r in rows if r["backend"] == backend and "v2_final_cpu" in r] + if group: + lines.append(f"| {BACKEND_LABEL[backend]} | {len(group)} | {mean_sd([r['v2_final_cpu']['validation']['perplexity'] for r in group])} | {mean_sd([r['v2_final_cpu']['test']['perplexity'] for r in group])} |") + return "\n".join(lines) + + +def v2_corpus_table(manifest: dict) -> str: + lines = ["| Source | Role | Body SHA-256 | Main-text paragraphs | Dropped (rule: count) | Kept | Train paragraphs |", "| --- | --- | --- | ---: | --- | ---: | ---: |"] + for s in manifest["sources"]: + dropped = ", ".join(f"{k}: {v}" for k, v in s["dropped_paragraphs"].items()) or "–" + lines.append(f"| {s['title']} ({s['author']}) | {s['role']} | `{s['body_sha256'][:12]}…` | {s['main_text_paragraphs']} | {dropped} | {s['kept_paragraphs']} | {s.get('train_paragraphs', '–')} |") + lines += ["", "| File | Bytes | SHA-256 |", "| --- | ---: | --- |"] + lines += [f"| `{name}` | {info['bytes']:,} | `{info['sha256'][:16]}…` |" for name, info in manifest["files"].items()] + lines += ["", "| Check | Paragraphs | Exact duplicates | Near duplicates (≥50% word 8-grams) | Mean 8-gram overlap |", "| --- | ---: | ---: | ---: | ---: |"] + lines += [f"| {name} | {c['paragraphs']} | {c['exact_duplicates']} | {c['near_duplicates']} | {c['mean_ngram_overlap']:.5f} |" for name, c in manifest["contamination_checks"].items()] + return "\n".join(lines) + + +def training_step_table(records: dict[str, dict]) -> str: + lines = [] + for key, record in records.items(): + s = record["settings"] + lines += [ + f"**{key}** · batch {s['batch_size']} · forward + backward + AdamW · {s['repeats']} repeats after {s['warmup']} warmups · median (p95)", + "", + "| L | Backend | Step ms | Tokens/s | First-step extra memory MiB | max abs loss difference vs MHA |", + "| ---: | --- | ---: | ---: | ---: | ---: |", + ] + for row in record["results"]: + lines.append( + f"| {row['seq_len']} | {BACKEND_LABEL[canon(row['backend'])]} | {row['step_ms']['median']:.1f} ({row['step_ms']['p95']:.1f}) " + f"| {row['tokens_per_second']['median']:.0f} | {row['memory']['peak_extra_bytes'] / 2**20:.1f} | {row['max_abs_loss_difference_vs_first_backend']:.1e} |" + ) + lines.append("") + return "\n".join(lines).rstrip() + + +def kv_figure(records: dict[str, dict]) -> tuple[str, str]: + panels = [] + for key, record in records.items(): + series = [] + for color, batch in zip(SERIES, record["settings"]["batch_sizes"]): + pts = [(max(row["prompt_len"], 1), sdpa_speedup(row), + f"{key} B={batch} prompt={row['prompt_len']}: cached is {sdpa_speedup(row):.2f}x uncached SDPA") + for row in record["results"]["grid"] if row["batch"] == batch] + series.append((f"batch {batch}", color, pts)) + panels.append((key, series, "uncached ÷ cached total time", lambda v: f"{v:g}×", 1.0, "prompt length (tokens)")) + batches = next(iter(records.values()))["settings"]["batch_sizes"] + some = next(iter(records.values()))["settings"] + return grid_figure( + "kv_cache_speedup.svg", "EXPERIMENT 03 / KV CACHE", "When does the KV cache pay off?", + f"Time to generate {some['new_tokens']} tokens, uncached (SDPA) divided by cached. Below the dashed line, caching is slower.", + f"Same weights, prompts, dtype and attention kernel · {some['repeats']} repeats · source: experiments/records/kv_cache", + panels, [(f"batch {b}", c) for c, b in zip(SERIES, batches)], + ) + + +def replace_section(text: str, name: str, content: str) -> str: + pattern = re.compile(rf"(\n).*?()", re.DOTALL) + if not pattern.search(text): + raise ValueError(f"missing generated section markers for {name}") + return pattern.sub(lambda m: m.group(1) + content + "\n" + m.group(2), text) + + +def main() -> None: + problems = validate_store(load_store(ROOT, RECORDS), strict=True) + if problems: + # Fail closed: never report from records with unresolved provenance problems. + raise SystemExit("records failed validation (run scripts/validate_records.py):\n" + "\n".join(problems)) + store = load_store(ROOT, RECORDS) + speed = latest(RECORDS / "attention") + quality = latest(RECORDS / "attention_quality") + if not speed: + raise SystemExit("no attention records found; run scripts/benchmark_attention.py first") + sections = { + "environment": environment_table(list(speed.values()) + list(quality.values())), + "forward": "\n\n".join(forward_table(key, record) for key, record in speed.items()), + "decode": "\n\n".join(decode_table(key, record) for key, record in speed.items()), + "variability": variability_table(speed), + } + if quality: + record = list(quality.values())[-1] + sections["quality"] = quality_table(record) + sections["seeds"] = seed_table(record) + kv = latest(RECORDS / "kv_cache") + extra_figures = [] + if kv: + sections["kv"] = "\n\n".join(kv_table(key, record) for key, record in kv.items()) + sections["kv_memory"] = kv_memory_table(kv) + sections["environment"] = environment_table(list(speed.values()) + list(quality.values()) + list(kv.values())) + extra_figures.append(kv_figure(kv)) + step_records = latest(RECORDS / "training_step") if (RECORDS / "training_step").exists() else {} + if step_records: + sections["training_step"] = training_step_table(step_records) + v1_rows = protocol_rows(store, "science-v1", ["sdpa_v1_eval_cpu", "sdpa_v1_eval_mps"]) + if v1_rows: + sections["sdpa_training"] = sdpa_training_table(v1_rows) + v2_manifest = ROOT / "data/science-v2/manifest.json" + if v2_manifest.exists(): + sections["v2_corpus"] = v2_corpus_table(json.loads(v2_manifest.read_text(encoding="utf-8"))) + v2_rows = protocol_rows(store, "science-v2", ["v2_final_cpu"]) + if any("v2_final_cpu" in row for row in v2_rows): + sections["v2_results"] = v2_table(v2_rows) + text = REPORT.read_text(encoding="utf-8") + for name, content in sections.items(): + text = replace_section(text, name, content) + REPORT.write_text(text, encoding="utf-8") + FIGURES.mkdir(parents=True, exist_ok=True) + for name, content in figures(speed) + extra_figures: + (FIGURES / name).write_text(content, encoding="utf-8") + print(f"updated {REPORT.relative_to(ROOT)} with sections: {', '.join(sections)}") + + +if __name__ == "__main__": + main() diff --git a/scripts/build_identity_baseline.py b/scripts/build_identity_baseline.py new file mode 100644 index 0000000..0ef5365 --- /dev/null +++ b/scripts/build_identity_baseline.py @@ -0,0 +1,78 @@ +"""Build or check the audited model-identity baseline for historical checkpoints. + +For every current training record whose checkpoint files are present, this script +hashes the checkpoint bytes, requires them to equal the SHA-256 recorded in the +training record, and recomputes the model identity from the checkpoint tensors. It +never reads attestation records or declared fingerprints. The output is deterministic +(no timestamps), so anyone holding the original artifacts can regenerate it +byte-for-byte with --check. + +After review, the file's SHA-256 is pinned as BASELINE_SHA256 in +src/minilm_lab/records.py; validation trusts the baseline only while that pin matches. +""" + +import argparse +import hashlib +import json +import shlex +import sys +from pathlib import Path + +import torch + +from minilm_lab.identity import SCHEME, checkpoint_identity +from minilm_lab.records import BASELINE_PATH, load_store + +ROOT = Path(__file__).resolve().parents[1] + + +def build(artifact_root: Path) -> tuple[bytes, list[str]]: + store = load_store(ROOT, baseline_path=None) + entries, missing = {}, [] + for key, record in store.current("training"): + tokens = shlex.split(record["settings"]["training_command"]) + directory = artifact_root / tokens[tokens.index("--output-dir") + 1] + for kind, filename in (("best", "best_checkpoint.pt"), ("final", "tiny_checkpoint.pt")): + expected = record["results"][f"{kind}_checkpoint_sha256"] + path = directory / filename + if not path.is_file(): + missing.append(f"{key} {kind}: {path} not found") + continue + actual = hashlib.sha256(path.read_bytes()).hexdigest() + if actual != expected: + missing.append(f"{key} {kind}: bytes {actual[:12]} do not match recorded {expected[:12]}") + continue + identity = checkpoint_identity(torch.load(path, map_location="cpu", weights_only=True)) + entries[expected] = {"kind": kind, "model_identity_sha256": identity["sha256"], "training_record": key} + baseline = { + "scheme": SCHEME, + "method": "checkpoint bytes hashed and required to equal the training record's SHA-256; " + "model identity recomputed from the checkpoint tensors and configuration", + "entries": dict(sorted(entries.items())), + } + return (json.dumps(baseline, indent=2, sort_keys=True) + "\n").encode(), missing + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("--artifact-root", type=Path, default=ROOT) + parser.add_argument("--check", action="store_true", help="regenerate in memory and compare with the committed baseline") + args = parser.parse_args() + data, missing = build(args.artifact_root) + for line in missing: + print(f"MISSING {line}") + if missing: + sys.exit("baseline incomplete: every current training checkpoint must be present and byte-identical") + digest = hashlib.sha256(data).hexdigest() + target = ROOT / BASELINE_PATH + if args.check: + same = target.is_file() and target.read_bytes() == data + print(f"regenerated baseline {digest} {'matches' if same else 'DIFFERS FROM'} {BASELINE_PATH}") + sys.exit(0 if same else 1) + target.parent.mkdir(parents=True, exist_ok=True) + target.write_bytes(data) + print(f"wrote {BASELINE_PATH} ({len(json.loads(data)['entries'])} checkpoints)\nSHA-256 {digest} (pin as BASELINE_SHA256)") + + +if __name__ == "__main__": + main() diff --git a/scripts/compare_attention_quality.py b/scripts/compare_attention_quality.py new file mode 100644 index 0000000..e0972a9 --- /dev/null +++ b/scripts/compare_attention_quality.py @@ -0,0 +1,136 @@ +"""Numerical and output-quality comparison of attention backends on trained weights. + +For each checkpoint, device/dtype and backend: + * logit error against a float64 CPU oracle (reference backend; RMSNorm variance + and RoPE angles are still computed in float32) + * next-token top-1 agreement with the oracle + * held-out loss / perplexity over every validation target (fp32 loss) + * greedy cached generation agreement with the oracle's tokens +""" + +import argparse +import math +from pathlib import Path + +import torch +import torch.nn.functional as F + +from minilm_lab.identity import checkpoint_identity +from minilm_lab.kv_cache import greedy_decode +from minilm_lab.model import MiniLM, MiniLMConfig +from minilm_lab.provenance import environment, file_sha256, write_record +from minilm_lab.tokenizer import ByteTokenizer + +ROOT = Path(__file__).resolve().parents[1] +DTYPES = {"float32": torch.float32, "float16": torch.float16, "bfloat16": torch.bfloat16} + + +def load(path: Path, device: str, dtype: torch.dtype) -> MiniLM: + checkpoint = torch.load(path, map_location="cpu", weights_only=True) + model = MiniLM(MiniLMConfig(**checkpoint["config"])) + model.load_state_dict(checkpoint["model"]) + return model.to(device=device, dtype=dtype).eval() + + +def windows(tokens: torch.Tensor, context: int) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + """Same chunking as metrics.evaluate_tokens, padded into one batch.""" + count = tokens.numel() - 1 + starts = list(range(1, tokens.numel(), context)) + inputs = torch.zeros(len(starts), context, dtype=torch.long) + targets = torch.zeros_like(inputs) + valid = torch.zeros_like(inputs, dtype=torch.bool) + for row, end in enumerate(starts): + stop = min(end + context, tokens.numel()) + inputs[row, : stop - end] = tokens[end - 1 : stop - 1] + targets[row, : stop - end] = tokens[end:stop] + valid[row, : stop - end] = True + assert int(valid.sum()) == count + return inputs, targets, valid + + +@torch.inference_mode() +def logits_for(model: MiniLM, inputs: torch.Tensor, device: str, batch_size: int = 64) -> torch.Tensor: + return torch.cat([model(inputs[i : i + batch_size].to(device))[0].cpu().double() for i in range(0, len(inputs), batch_size)]) + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("checkpoints", type=Path, nargs="+") + parser.add_argument("--validation-text", type=Path, default=ROOT / "data/science/validation.txt") + parser.add_argument("--targets", default="cpu:float32,mps:float32,mps:float16,mps:bfloat16") + parser.add_argument("--backends", default="mha,sdpa,reference") + parser.add_argument("--prompts", type=int, default=8) + parser.add_argument("--generate-tokens", type=int, default=64) + parser.add_argument("--output-dir", type=Path, default=ROOT / "experiments/records/attention_quality") + args = parser.parse_args() + targets = [tuple(item.split(":")) for item in args.targets.split(",")] + skipped = [f"{d}:{t}" for d, t in targets if d == "mps" and not torch.backends.mps.is_available()] + targets = [(d, t) for d, t in targets if f"{d}:{t}" not in skipped] + tokens = torch.tensor(ByteTokenizer().encode(args.validation_text.read_text(encoding="utf-8")), dtype=torch.long) + results = [] + for path in args.checkpoints: + oracle_model = load(path, "cpu", torch.float64).set_attention_backend("reference") + context = oracle_model.config.context_length + inputs, labels, valid = windows(tokens, context) + oracle = logits_for(oracle_model, inputs, "cpu") + oracle_top1 = oracle.argmax(-1) + prompt_starts = torch.linspace(0, tokens.numel() - context - 1, args.prompts).long() + prompts = torch.stack([tokens[s : s + 32] for s in prompt_starts]) + oracle_generation = greedy_decode(oracle_model, prompts, args.generate_tokens)[:, 32:] + raw_checkpoint = torch.load(path, map_location="cpu", weights_only=True) + seed, identity = raw_checkpoint.get("seed"), checkpoint_identity(raw_checkpoint)["sha256"] + for device, dtype_name in targets: + model = load(path, device, DTYPES[dtype_name]) + for backend in args.backends.split(","): + model.set_attention_backend(backend) + logits = logits_for(model, inputs, device) + error = (logits - oracle).abs()[valid] + nll = F.cross_entropy(logits[valid].float(), labels[valid], reduction="sum").item() + loss = nll / int(valid.sum()) + generated = greedy_decode(model, prompts.to(device), args.generate_tokens)[:, 32:].cpu() + mismatch = (generated != oracle_generation).int() + first_divergence = [int(row.argmax()) if row.any() else args.generate_tokens for row in mismatch] + row = { + "checkpoint": str(path), + "checkpoint_sha256": file_sha256(path), + "model_identity_sha256": identity, + "seed": seed, + "device": device, + "dtype": dtype_name, + "backend": backend, + "max_abs_logit_error": error.max().item(), + "mean_abs_logit_error": error.mean().item(), + "top1_agreement": (logits.argmax(-1) == oracle_top1)[valid].float().mean().item(), + "validation_loss": loss, + "perplexity": math.exp(loss), + "bits_per_byte": loss / math.log(2), + "greedy_identical_sequences": int((mismatch.sum(1) == 0).sum()), + "greedy_prompts": args.prompts, + "greedy_first_divergence": first_divergence, + } + results.append(row) + print( + f"seed={seed} {device}/{dtype_name:8s} {backend:9s} max|dlogit|={row['max_abs_logit_error']:.2e} " + f"top1={row['top1_agreement']:.5f} ppl={row['perplexity']:.4f} " + f"greedy_same={row['greedy_identical_sequences']}/{args.prompts}", + flush=True, + ) + payload = { + "experiment": "attention_quality", + "environment": environment(",".join(sorted({d for d, _ in targets}))), + "settings": { + "validation_text": str(args.validation_text.resolve().relative_to(ROOT)) if args.validation_text.resolve().is_relative_to(ROOT) else str(args.validation_text), + "validation_sha256": file_sha256(args.validation_text), + "validation_targets": tokens.numel() - 1, + "oracle": "cpu float64, reference backend (RMSNorm variance and RoPE angles computed in float32)", + "loss_precision": "logits upcast to float32 before cross-entropy", + "greedy": {"prompts": args.prompts, "prompt_tokens": 32, "generated_tokens": args.generate_tokens, "cached": True}, + "skipped_targets": skipped, + }, + "results": results, + } + print(f"wrote {write_record(args.output_dir, 'attention_quality', payload)}") + + +if __name__ == "__main__": + main() diff --git a/scripts/evaluate.py b/scripts/evaluate.py index 04cef28..e6cc139 100644 --- a/scripts/evaluate.py +++ b/scripts/evaluate.py @@ -9,7 +9,13 @@ from minilm_lab.data import load_text, split_tokens from minilm_lab.metrics import evaluate_tokens -from minilm_lab.model import MiniLM, MiniLMConfig +from minilm_lab.model import ( + ATTENTION_BACKENDS, + DEFAULT_ATTENTION_BACKEND, + MiniLM, + MiniLMConfig, + canonical_backend, +) from minilm_lab.tokenizer import BPETokenizer, ByteTokenizer @@ -20,6 +26,10 @@ def main() -> None: parser.add_argument("--checkpoint", type=Path, default=Path("artifacts/best_checkpoint.pt")) parser.add_argument("--device", choices=("auto", "cpu", "mps"), default="auto") parser.add_argument("--output", type=Path, default=Path("artifacts/evaluation.json")) + parser.add_argument( + "--attention-backend", choices=(*ATTENTION_BACKENDS, "auto"), default=DEFAULT_ATTENTION_BACKEND, + help="sdpa (default); use mha to reproduce science-v1 numbers exactly", + ) args = parser.parse_args() device = ("mps" if torch.backends.mps.is_available() else "cpu") if args.device == "auto" else args.device checkpoint = torch.load(args.checkpoint, map_location=device, weights_only=True) @@ -31,7 +41,7 @@ def main() -> None: ids = torch.tensor(tokenizer.encode(load_text(args.text)), dtype=torch.long) _, validation_ids = split_tokens(ids) config = MiniLMConfig(**checkpoint["config"]) - model = MiniLM(config).to(device) + model = MiniLM(config).to(device).set_attention_backend(args.attention_backend) model.load_state_dict(checkpoint["model"]) evaluation = evaluate_tokens(model, validation_ids, device) # The first token is context; count bytes represented by scored targets. @@ -40,6 +50,8 @@ def main() -> None: "checkpoint": str(args.checkpoint), "tokenizer": tokenizer_name, "device": device, + "attention_backend": canonical_backend(args.attention_backend), + "checkpoint_attention_backend": canonical_backend(checkpoint.get("attention_backend")), **evaluation, "bits_per_byte": evaluation["validation_loss"] * evaluation["validation_tokens"] / scored_bytes / math.log(2), "validation_bytes": scored_bytes, diff --git a/scripts/evaluate_runs.py b/scripts/evaluate_runs.py new file mode 100644 index 0000000..af567a2 --- /dev/null +++ b/scripts/evaluate_runs.py @@ -0,0 +1,91 @@ +"""Evaluate selected checkpoints on named splits and write one experiment record. + +Each run's best_checkpoint.pt (chosen on validation during training) is scored +with the attention backend it was trained with. On protocol v2 the test split +is evaluated here and nowhere else. + + python scripts/evaluate_runs.py artifacts/runs/v2-*-s4? \ + --split validation=data/science-v2/validation.txt --split test=data/science-v2/test.txt \ + --name v2_final +""" + +import argparse +import json +import math +import time +from pathlib import Path + +import torch + +from minilm_lab.identity import checkpoint_identity +from minilm_lab.metrics import evaluate_tokens +from minilm_lab.model import MiniLM, MiniLMConfig, canonical_backend +from minilm_lab.provenance import environment, file_sha256, synchronize, write_record +from minilm_lab.tokenizer import ByteTokenizer + +ROOT = Path(__file__).resolve().parents[1] + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("runs", type=Path, nargs="+") + parser.add_argument("--split", action="append", required=True, help="name=path, repeatable") + parser.add_argument("--device", choices=("cpu", "mps"), default="cpu") + parser.add_argument("--name", required=True) + parser.add_argument("--protocol", required=True, help="corpus protocol, e.g. science-v1 or science-v2") + parser.add_argument("--output-dir", type=Path, default=ROOT / "experiments/records/evaluation") + args = parser.parse_args() + splits = {} + for item in args.split: + name, path = item.split("=", 1) + splits[name] = Path(path) + tokenizer = ByteTokenizer() + tokens = {name: torch.tensor(tokenizer.encode(path.read_text(encoding="utf-8")), dtype=torch.long) for name, path in splits.items()} + results = [] + for run in args.runs: + path = run / "best_checkpoint.pt" + checkpoint = torch.load(path, map_location="cpu", weights_only=True) + if checkpoint.get("tokenizer", "byte") != "byte": + parser.error("only byte-tokenizer checkpoints are supported") + backend = canonical_backend(checkpoint.get("attention_backend")) + model = MiniLM(MiniLMConfig(**checkpoint["config"])) + model.load_state_dict(checkpoint["model"]) + model.to(args.device).set_attention_backend(backend) + summary_path = run / "run_summary.json" + row = { + "run": str(run), + "seed": checkpoint["seed"], + "attention_backend": backend, + "selected_step": checkpoint["step"], + "selected_on": "validation loss (during training)", + "checkpoint_sha256": file_sha256(path), + "model_identity_sha256": checkpoint_identity(checkpoint)["sha256"], + "corpus_sha256": checkpoint.get("corpus_sha256"), + "training": json.loads(summary_path.read_text()) if summary_path.exists() else None, + "splits": {}, + } + for name, ids in tokens.items(): + synchronize(args.device) + start = time.perf_counter() + evaluation = evaluate_tokens(model, ids, args.device) + synchronize(args.device) + seconds = time.perf_counter() - start + row["splits"][name] = {**evaluation, "bits_per_byte": evaluation["validation_loss"] / math.log(2), "eval_seconds": seconds} + print(f"{run.name} seed={row['seed']} backend={backend} {name}: ppl={evaluation['perplexity']:.3f} ({seconds:.2f}s)", flush=True) + results.append(row) + payload = { + "experiment": "evaluation", + "environment": environment(args.device), + "settings": { + "protocol": args.protocol, + "splits": {name: {"path": str(path), "sha256": file_sha256(path), "targets": tokens[name].numel() - 1} for name, path in splits.items()}, + "metric": "minilm_lab.metrics.evaluate_tokens (non-overlapping context-length chunks, every target once)", + "dtype": "float32", + }, + "results": results, + } + print(f"wrote {write_record(args.output_dir, args.name, payload)}") + + +if __name__ == "__main__": + main() diff --git a/scripts/generate.py b/scripts/generate.py index 8398edb..0145cac 100644 --- a/scripts/generate.py +++ b/scripts/generate.py @@ -4,7 +4,12 @@ import torch from minilm_lab.generation import generate -from minilm_lab.model import MiniLM, MiniLMConfig +from minilm_lab.model import ( + ATTENTION_BACKENDS, + DEFAULT_ATTENTION_BACKEND, + MiniLM, + MiniLMConfig, +) from minilm_lab.tokenizer import BPETokenizer, ByteTokenizer @@ -16,11 +21,16 @@ def main() -> None: parser.add_argument("--temperature", type=float, default=0.8) parser.add_argument("--top-k", type=int, default=40) parser.add_argument("--top-p", type=float, default=0.95) + parser.add_argument( + "--attention-backend", choices=(*ATTENTION_BACKENDS, "auto"), default=DEFAULT_ATTENTION_BACKEND, + help="sdpa (default); mha reproduces published science-v1 measurements", + ) args = parser.parse_args() device = "mps" if torch.backends.mps.is_available() else "cpu" - checkpoint = torch.load(args.checkpoint, map_location=device, weights_only=False) + checkpoint = torch.load(args.checkpoint, map_location=device, weights_only=True) model = MiniLM(MiniLMConfig(**checkpoint["config"])).to(device) model.load_state_dict(checkpoint["model"]) + model.set_attention_backend(args.attention_backend) tokenizer = ByteTokenizer() if checkpoint.get("tokenizer", "byte") == "byte" else BPETokenizer.load(checkpoint["tokenizer"]) print(generate(model, args.prompt, tokenizer, args.tokens, args.temperature, args.top_k, args.top_p)) diff --git a/scripts/prepare_science_corpus.py b/scripts/prepare_science_corpus.py index fff8330..4bdd301 100644 --- a/scripts/prepare_science_corpus.py +++ b/scripts/prepare_science_corpus.py @@ -1,8 +1,10 @@ """Download two public-domain science books and create document-aware splits.""" +import argparse import hashlib import json import re +import sys from pathlib import Path from urllib.request import Request, urlopen @@ -26,15 +28,30 @@ def book_body(text: str) -> str: return text[text.find("\n", start) + 1 : end].replace("\r\n", "\n").strip() + "\n" +# Published science-v1 split hashes (data/science/manifest.json). Regeneration +# must reproduce these bytes exactly; otherwise nothing is written. +PUBLISHED_TRAIN_SHA256 = "19c057053f222c3084fba3693c1c295ea94f6cda1ee5fbf5f917953fcb91ef00" +PUBLISHED_VALIDATION_SHA256 = "e25d4e45706bbf4251129dbb079f3487660c453cf8b34407901350538b23e2e7" + + def main() -> None: - OUTPUT.mkdir(parents=True, exist_ok=True) - train_parts, validation_parts, sources = [], [], [] + parser = argparse.ArgumentParser(description="Regenerate and verify the legacy science-v1 corpus.") + parser.add_argument("--raw-dir", type=Path, default=None, help="use previously downloaded pg.txt files") + parser.add_argument("--output-dir", type=Path, default=OUTPUT) + args = parser.parse_args() + files, sources = {}, [] + train_parts, validation_parts = [], [] for name, url, expected_hash in SOURCES: - request = Request(url, headers={"User-Agent": "MiniLM-Lab/1.0 (research corpus preparation)"}) - with urlopen(request, timeout=30) as response: - raw = response.read() + if args.raw_dir: + raw = (args.raw_dir / url.rsplit("/", 1)[1]).read_bytes() + else: + request = Request(url, headers={"User-Agent": "MiniLM-Lab/1.0 (research corpus preparation)"}) + with urlopen(request, timeout=30) as response: + raw = response.read() if sha256(raw) != expected_hash: - raise ValueError(f"upstream text changed for {name}; inspect before running a comparable experiment") + # Project Gutenberg edits headers and licences; the split hashes below + # decide whether the published corpus is reproduced. + print(f"note: raw file for {name} differs from the 2026-09 download ({sha256(raw)})", file=sys.stderr) text = book_body(raw.decode("utf-8-sig")) paragraphs = re.split(r"\n\s*\n", text) total_bytes = sum(len(paragraph.encode("utf-8")) for paragraph in paragraphs) @@ -48,15 +65,21 @@ def main() -> None: validation_part = "\n\n".join(paragraphs[split:]).strip() + "\n" train_parts.append(training_part) validation_parts.append(validation_part) - (OUTPUT / f"{name}-validation.txt").write_text(validation_part, encoding="utf-8") + files[f"{name}-validation.txt"] = validation_part.encode("utf-8") sources.append({"name": name, "url": url, "raw_sha256": sha256(raw), "paragraphs": len(paragraphs), "train_paragraphs": split, "validation_bytes": len(validation_part.encode("utf-8"))}) - train = "\n\n".join(train_parts).encode("utf-8") - validation = "\n\n".join(validation_parts).encode("utf-8") - (OUTPUT / "train.txt").write_bytes(train) - (OUTPUT / "validation.txt").write_bytes(validation) - manifest = {"split_method": "first approximately 90% of bytes at a paragraph boundary from each document for training; remainder for validation", "sources": sources, "train_bytes": len(train), "validation_bytes": len(validation), "train_sha256": sha256(train), "validation_sha256": sha256(validation)} - (OUTPUT / "manifest.json").write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8") - print(json.dumps(manifest, indent=2)) + files["train.txt"] = "\n\n".join(train_parts).encode("utf-8") + files["validation.txt"] = "\n\n".join(validation_parts).encode("utf-8") + if sha256(files["train.txt"]) != PUBLISHED_TRAIN_SHA256 or sha256(files["validation.txt"]) != PUBLISHED_VALIDATION_SHA256: + raise ValueError("regenerated splits differ from the published science-v1 corpus; nothing was written") + args.output_dir.mkdir(parents=True, exist_ok=True) + for filename, data in files.items(): + (args.output_dir / filename).write_bytes(data) + manifest_path = args.output_dir / "manifest.json" + if not manifest_path.exists(): + # The committed manifest is a historical record and is never rewritten. + manifest = {"split_method": "first approximately 90% of bytes at a paragraph boundary from each document for training; remainder for validation", "sources": sources, "train_bytes": len(files["train.txt"]), "validation_bytes": len(files["validation.txt"]), "train_sha256": sha256(files["train.txt"]), "validation_sha256": sha256(files["validation.txt"])} + manifest_path.write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8") + print(f"verified science-v1 splits: train {PUBLISHED_TRAIN_SHA256[:12]}, validation {PUBLISHED_VALIDATION_SHA256[:12]}") if __name__ == "__main__": diff --git a/scripts/prepare_science_corpus_v2.py b/scripts/prepare_science_corpus_v2.py new file mode 100644 index 0000000..644a576 --- /dev/null +++ b/scripts/prepare_science_corpus_v2.py @@ -0,0 +1,159 @@ +"""Build corpus protocol v2: cleaned prose, train/validation splits and an untouched test book. + +v1 (data/science/) is left untouched as a legacy experiment. v2 differs in: + * non-prose front/back matter removed with exact heading markers per book + * paragraph rules drop headings, bracketed editorial notes and numeric + lists/tables (see minilm_lab.corpus); displayed equations are kept + * a third book by a different author is held out as a test set that is never + used for training, checkpoint selection or tuning + * held-out paragraphs that duplicate training text (exact or >= 50% word + 8-gram overlap) are removed and reported + +Book bodies are verified by SHA-256 of the extracted Gutenberg body. Raw file +hashes are recorded but may change when Gutenberg edits headers or licences. +Pass --raw-dir to build from previously downloaded pg.txt files. +""" + +import argparse +import hashlib +import json +from pathlib import Path +from urllib.request import Request, urlopen + +from minilm_lab.corpus import ( + clean, + contamination, + gutenberg_body, + main_text, + paragraphs, + remove_indices, + split_by_bytes, +) + +ROOT = Path(__file__).resolve().parents[1] +PROTOCOL = "science-v2" +SOURCES = ( + { + "name": "darwin-1859", "gutenberg_id": 1228, "role": "train_validation", + "title": "On the Origin of Species (1859)", "author": "Charles Darwin", + "start": ("ON THE ORIGIN OF SPECIES.", 1), "end": ("INDEX.", 1), + "body_sha256": "9ff49617f2a0b52b18a2a3b6f09c8a323358160eb98e4e336969b1100768d73f", + }, + { + "name": "einstein-1924", "gutenberg_id": 30155, "role": "train_validation", + "title": "Relativity: The Special and General Theory (Lawson translation)", "author": "Albert Einstein", + "start": ("PREFACE", 1), "end": ("_K_ = co-ordinate system", 1), + "body_sha256": "384e707c77e521bec3cceee2b2b6a0e88f271e0f27752a3ec3790ba06554a36e", + }, + { + "name": "faraday-1861", "gutenberg_id": 14474, "role": "test", + "title": "The Chemical History of a Candle", "author": "Michael Faraday", + "start": ("THE CHEMICAL HISTORY OF A CANDLE", 1), "end": ("NOTES.", 2), + "body_sha256": "5580d9bfa409ee7f2f7d79b3f27921470a692d6e549a55052adf959547d7ade5", + }, +) + + +def sha256(data: bytes) -> str: + return hashlib.sha256(data).hexdigest() + + +def fetch(source: dict, raw_dir: Path | None) -> bytes: + if raw_dir: + return (raw_dir / f"pg{source['gutenberg_id']}.txt").read_bytes() + url = f"https://www.gutenberg.org/cache/epub/{source['gutenberg_id']}/pg{source['gutenberg_id']}.txt" + request = Request(url, headers={"User-Agent": "MiniLM-Lab/1.0 (research corpus preparation)"}) + with urlopen(request, timeout=30) as response: + return response.read() + + +def join(parts: list[str]) -> str: + return "\n\n".join(parts).strip() + "\n" + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("--raw-dir", type=Path, default=None) + parser.add_argument("--output-dir", type=Path, default=ROOT / "data/science-v2") + parser.add_argument("--allow-new-bodies", action="store_true", help="record body hashes instead of verifying them") + args = parser.parse_args() + train_parts, validation_parts, test_parts, sources = [], [], [], [] + for source in SOURCES: + raw = fetch(source, args.raw_dir) + body = gutenberg_body(raw.decode("utf-8-sig")) + body_hash = sha256(body.encode("utf-8")) + if source["body_sha256"] and body_hash != source["body_sha256"] and not args.allow_new_bodies: + raise ValueError(f"{source['name']} body changed upstream ({body_hash}); inspect before comparing runs") + text = main_text(body, *source["start"], *source["end"]) + kept, dropped = clean(paragraphs(text)) + entry = { + "name": source["name"], "title": source["title"], "author": source["author"], "role": source["role"], + "url": f"https://www.gutenberg.org/cache/epub/{source['gutenberg_id']}/pg{source['gutenberg_id']}.txt", + "raw_sha256_observed": sha256(raw), "body_sha256": body_hash, + "start_marker": {"line": source["start"][0], "occurrences": source["start"][1]}, + "end_marker": {"line": source["end"][0], "occurrences": source["end"][1]}, + "main_text_paragraphs": len(kept) + sum(dropped.values()), + "dropped_paragraphs": dict(sorted(dropped.items())), + "kept_paragraphs": len(kept), + } + if source["role"] == "test": + test_parts.append(kept) + else: + split = split_by_bytes(kept) + train_parts.append(kept[:split]) + validation_parts.append((source["name"], kept[split:])) + entry["train_paragraphs"] = split + sources.append(entry) + + train = [p for part in train_parts for p in part] + checks = {} + cleaned_validation = [] + for name, held_out in validation_parts: + report = contamination(train, held_out) + cleaned_validation.append((name, remove_indices(held_out, report["exact_duplicates"] + report["near_duplicates"]))) + checks[f"{name}_validation_vs_train"] = report + test = test_parts[0] + validation = [p for _, part in cleaned_validation for p in part] + for label, reference in (("test_vs_train", train), ("test_vs_validation", validation)): + report = contamination(reference, test) + checks[label] = report + test = remove_indices(test, checks["test_vs_train"]["exact_duplicates"] + checks["test_vs_train"]["near_duplicates"] + + checks["test_vs_validation"]["exact_duplicates"] + checks["test_vs_validation"]["near_duplicates"]) + + args.output_dir.mkdir(parents=True, exist_ok=True) + files = {"train.txt": join(train), "validation.txt": join([join(part) for _, part in cleaned_validation]), "test.txt": join(test)} + for name, part in cleaned_validation: + files[f"{name}-validation.txt"] = join(part) + hashes = {} + for filename, content in files.items(): + data = content.encode("utf-8") + (args.output_dir / filename).write_bytes(data) + hashes[filename] = {"sha256": sha256(data), "bytes": len(data)} + manifest = { + "protocol": PROTOCOL, + "legacy_protocol": "science-v1 in data/science/ (unchanged)", + "rules": [ + "extract the Project Gutenberg body; verify its SHA-256", + "keep text from the start marker line (last of the expected occurrences) up to the end marker line", + "split paragraphs on blank lines", + "drop paragraphs with no lowercase letter (headings, numerals)", + "drop paragraphs entirely enclosed in square brackets (illustrations, editorial notes)", + "drop paragraphs of two or more lines where at least half of the lines end in a number (index, contents, tables)", + "train/validation books: first ~90% of kept bytes at a paragraph boundary to train, rest to validation", + "test book: all kept paragraphs; never used for training, checkpoint selection or tuning", + "remove held-out paragraphs that are exact (normalized) or near (>=50% word 8-gram) duplicates of training or validation text", + ], + "usage": {"train": "optimization only", "validation": "checkpoint selection", "test": "final evaluation only, once per selected checkpoint"}, + "sources": sources, + "files": hashes, + "contamination_checks": { + key: {**value, "exact_duplicates": len(value["exact_duplicates"]), "near_duplicates": len(value["near_duplicates"])} + for key, value in checks.items() + }, + } + (args.output_dir / "manifest.json").write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8") + print(json.dumps(manifest, indent=2)) + + +if __name__ == "__main__": + main() diff --git a/scripts/publish_science_results.py b/scripts/publish_science_results.py index 8948ece..b69aa45 100644 --- a/scripts/publish_science_results.py +++ b/scripts/publish_science_results.py @@ -27,7 +27,7 @@ def main() -> None: int8 = read(INT8) per_source = {"darwin-1859": read(DARWIN), "einstein-1924": read(EINSTEIN)} manifest = read(MANIFEST) - checkpoint = torch.load(RUN / "best_checkpoint.pt", map_location="cpu", weights_only=False) + checkpoint = torch.load(RUN / "best_checkpoint.pt", map_location="cpu", weights_only=True) if not metrics or metrics[-1]["step"] != 1000: raise ValueError("expected the completed 1,000-step science run") if evaluation["validation_tokens"] < 100000 or benchmark["repeats"] < 3: diff --git a/scripts/quantize.py b/scripts/quantize.py index 9f29038..70afdcb 100644 --- a/scripts/quantize.py +++ b/scripts/quantize.py @@ -10,7 +10,13 @@ import torch from minilm_lab.metrics import evaluate_tokens -from minilm_lab.model import MiniLM, MiniLMConfig +from minilm_lab.model import ( + ATTENTION_BACKENDS, + DEFAULT_ATTENTION_BACKEND, + MiniLM, + MiniLMConfig, +) +from minilm_lab.provenance import environment, file_sha256, write_record from minilm_lab.quantization import quantize_linear_weights, stored_parameter_bytes from minilm_lab.tokenizer import ByteTokenizer @@ -33,12 +39,18 @@ def main() -> None: parser.add_argument("--checkpoint", type=Path, required=True) parser.add_argument("--validation-text", type=Path, required=True) parser.add_argument("--output", type=Path, default=Path("artifacts/int8-comparison.json")) + parser.add_argument("--record-dir", type=Path, default=None, help="also write an experiment record (experiments/records/legacy)") + parser.add_argument( + "--attention-backend", choices=(*ATTENTION_BACKENDS, "auto"), default=DEFAULT_ATTENTION_BACKEND, + help="sdpa (default); mha reproduces published science-v1 measurements", + ) args = parser.parse_args() checkpoint = torch.load(args.checkpoint, map_location="cpu", weights_only=True) if checkpoint.get("tokenizer") != "byte": parser.error("this reference experiment currently supports byte checkpoints") model = MiniLM(MiniLMConfig(**checkpoint["config"])) model.load_state_dict(checkpoint["model"]) + model.set_attention_backend(args.attention_backend) model.eval() compact = quantize_linear_weights(copy.deepcopy(model)).eval() tokens = torch.tensor(ByteTokenizer().encode(args.validation_text.read_text(encoding="utf-8")), dtype=torch.long) @@ -49,6 +61,7 @@ def main() -> None: "method": "per-output-channel symmetric INT8 linear weights; dequantize on each forward pass", "scope": "MLP nn.Linear layers; attention projections and tied LM head remain FP32", "device": "cpu", + "attention_backend": model.blocks[0].attention.backend, "original_validation_loss": original["validation_loss"], "int8_validation_loss": quantized["validation_loss"], "loss_delta": quantized["validation_loss"] - original["validation_loss"], @@ -59,6 +72,12 @@ def main() -> None: } args.output.parent.mkdir(parents=True, exist_ok=True) args.output.write_text(json.dumps(result, indent=2) + "\n", encoding="utf-8") + if args.record_dir: + settings = {"checkpoint": str(args.checkpoint), "checkpoint_sha256": file_sha256(args.checkpoint), + "validation_text": str(args.validation_text), "validation_sha256": file_sha256(args.validation_text), + "latency": "median of 5 repeats of 32 forward passes after one warmup", "attention_backend": result["attention_backend"]} + payload = {"experiment": "legacy_int8_comparison", "environment": environment("cpu"), "settings": settings, "results": result} + print(f"wrote {write_record(args.record_dir, 'legacy_int8_cpu', payload)}") print(json.dumps(result, indent=2)) diff --git a/scripts/record_training_run.py b/scripts/record_training_run.py new file mode 100644 index 0000000..91cb415 --- /dev/null +++ b/scripts/record_training_run.py @@ -0,0 +1,79 @@ +"""Write an append-only experiment record for a finished train_tiny.py run.""" + +import argparse +import json +from pathlib import Path + +import torch + +from minilm_lab.identity import SCHEME, checkpoint_identity +from minilm_lab.model import canonical_backend +from minilm_lab.provenance import environment, file_sha256, write_record + +ROOT = Path(__file__).resolve().parents[1] + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("run_dir", type=Path) + parser.add_argument("--train-text", type=Path, default=ROOT / "data/science/train.txt") + parser.add_argument("--validation-text", type=Path, default=ROOT / "data/science/validation.txt") + parser.add_argument("--training-command", required=True, help="the exact command that produced the run") + parser.add_argument("--code-commit", required=True, help="commit whose training code produced the run") + parser.add_argument("--manifest", type=Path, default=ROOT / "data/science/manifest.json") + parser.add_argument("--protocol", default="science-v1") + parser.add_argument("--note", default="") + parser.add_argument("--output-dir", type=Path, default=ROOT / "experiments/records/training") + args = parser.parse_args() + final = torch.load(args.run_dir / "tiny_checkpoint.pt", map_location="cpu", weights_only=True) + summary = args.run_dir / "run_summary.json" + best = torch.load(args.run_dir / "best_checkpoint.pt", map_location="cpu", weights_only=True) + identities = {} + for label, checkpoint in (("best", best), ("final", final)): + identity = checkpoint_identity(checkpoint) + stored = checkpoint.get("model_identity") + if stored and stored != identity: + raise ValueError(f"{label} checkpoint's stored model identity does not match its tensors") + identities[label] = identity["sha256"] + payload = { + "experiment": "training", + "environment": environment("cpu"), + "settings": { + "dtype": "float32", + "seed": final["seed"], + "protocol": args.protocol, + "attention_backend": canonical_backend(final.get("attention_backend")), + "model_config": final["config"], + "tokenizer": final["tokenizer"], + "optimizer": "AdamW lr=3e-4 (scripts/train_tiny.py defaults)", + "steps": final["step"], + "training_command": args.training_command, + "training_code_commit": args.code_commit, + "note": args.note, + "data": { + "train": {"path": str(args.train_text.resolve().relative_to(ROOT)), "sha256": file_sha256(args.train_text)}, + "validation": {"path": str(args.validation_text.resolve().relative_to(ROOT)), "sha256": file_sha256(args.validation_text)}, + "corpus_sha256_in_checkpoint": final["corpus_sha256"], + "manifest_path": str(args.manifest.resolve().relative_to(ROOT)), + "manifest_sha256": file_sha256(args.manifest), + }, + }, + "results": { + "metrics": final["metrics"], + "best_step": best["step"], + "best_validation_loss": best["best_validation_loss"], + "best_checkpoint_sha256": file_sha256(args.run_dir / "best_checkpoint.pt"), + "final_checkpoint_sha256": file_sha256(args.run_dir / "tiny_checkpoint.pt"), + "run_summary": json.loads(summary.read_text(encoding="utf-8")) if summary.exists() else None, + "model_identity_scheme": SCHEME, + "best_model_identity_sha256": identities["best"], + "final_model_identity_sha256": identities["final"], + }, + } + name = f"training_{args.protocol}_{canonical_backend(final.get('attention_backend'))}_seed{final['seed']}" + path = write_record(args.output_dir, name, payload) + print(f"wrote {path}") + + +if __name__ == "__main__": + main() diff --git a/scripts/train_tiny.py b/scripts/train_tiny.py index b500140..70b40c7 100644 --- a/scripts/train_tiny.py +++ b/scripts/train_tiny.py @@ -9,8 +9,16 @@ import torch from minilm_lab.data import load_text, make_batch, split_tokens +from minilm_lab.identity import checkpoint_identity from minilm_lab.metrics import evaluate_tokens -from minilm_lab.model import MiniLM, MiniLMConfig +from minilm_lab.model import ( + ATTENTION_BACKENDS, + DEFAULT_ATTENTION_BACKEND, + MiniLM, + MiniLMConfig, + canonical_backend, +) +from minilm_lab.provenance import peak_rss_bytes, synchronize from minilm_lab.tokenizer import BPETokenizer, ByteTokenizer @@ -18,7 +26,7 @@ def choose_device() -> str: return "mps" if torch.backends.mps.is_available() else "cpu" -def main() -> None: +def build_parser() -> argparse.ArgumentParser: parser = argparse.ArgumentParser() parser.add_argument("text", type=Path) parser.add_argument("--steps", type=int, default=100) @@ -35,8 +43,18 @@ def main() -> None: parser.add_argument("--tokenizer", type=Path, default=None, help="BPE JSON artifact; omit for byte tokenizer") parser.add_argument("--gradient-accumulation-steps", type=int, default=1) parser.add_argument("--seed", type=int, default=42) + parser.add_argument( + "--attention-backend", choices=(*ATTENTION_BACKENDS, "auto"), default=DEFAULT_ATTENTION_BACKEND, + help="sdpa (default); mha reproduces science-v1 results ('auto' is its legacy name); reference is the explicit implementation", + ) parser.add_argument("--output-dir", type=Path, default=Path("artifacts")) + return parser + + +def main() -> None: + parser = build_parser() args = parser.parse_args() + args.attention_backend = canonical_backend(args.attention_backend) if args.steps < 1 or args.eval_interval < 1 or args.gradient_accumulation_steps < 1: parser.error("steps, eval interval and accumulation must be positive") output = args.output_dir / "tiny_checkpoint.pt" @@ -56,7 +74,7 @@ def main() -> None: if train_ids.numel() <= args.context_length or validation_ids.numel() <= args.context_length: raise ValueError("both splits must exceed context_length") config = MiniLMConfig(context_length=args.context_length, vocab_size=tokenizer.vocab_size, embedding_dim=args.embedding_dim, num_layers=args.num_layers, num_heads=args.num_heads, positional_encoding=args.position) - model = MiniLM(config).to(device) + model = MiniLM(config).to(device).set_attention_backend(args.attention_backend) optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4) metrics = [] best_validation_loss = float("inf") @@ -66,6 +84,8 @@ def main() -> None: checkpoint = torch.load(output, map_location=device, weights_only=True) if checkpoint["config"] != config.__dict__ or checkpoint.get("corpus_sha256") != corpus_sha256: raise ValueError("resume config or corpus differs from checkpoint") + if canonical_backend(checkpoint.get("attention_backend")) != args.attention_backend: + raise ValueError("resume attention backend differs from checkpoint") model.load_state_dict(checkpoint["model"]) if "optimizer" in checkpoint: optimizer.load_state_dict(checkpoint["optimizer"]) @@ -77,7 +97,10 @@ def main() -> None: print(f"resumed {output} at step={start_step}") optimizer.zero_grad(set_to_none=True) started = time.perf_counter() + train_seconds, eval_seconds, train_tokens = 0.0, [], 0 for step in range(start_step, start_step + args.steps): + synchronize(device) + step_started = time.perf_counter() train_loss = 0.0 for _ in range(args.gradient_accumulation_steps): inputs, targets = make_batch(train_ids, args.batch_size, config.context_length, device) @@ -86,17 +109,47 @@ def main() -> None: train_loss += loss.item() / args.gradient_accumulation_steps optimizer.step() optimizer.zero_grad(set_to_none=True) + synchronize(device) + train_seconds += time.perf_counter() - step_started + train_tokens += args.gradient_accumulation_steps * args.batch_size * config.context_length if (step + 1) % args.eval_interval == 0 or step == start_step + args.steps - 1: + eval_started = time.perf_counter() evaluation = evaluate_tokens(model, validation_ids, device) + synchronize(device) + eval_seconds.append(time.perf_counter() - eval_started) validation_loss = evaluation["validation_loss"] - record = {"step": step + 1, "train_loss": train_loss, **evaluation, "elapsed_seconds": time.perf_counter() - started} + record = {"step": step + 1, "train_loss": train_loss, **evaluation, "elapsed_seconds": time.perf_counter() - started, "train_seconds": train_seconds, "eval_seconds": eval_seconds[-1]} metrics.append(record) print(f"step={step + 1:04d} train_loss={train_loss:.4f} val_loss={validation_loss:.4f} val_ppl={evaluation['perplexity']:.2f} device={device}", flush=True) if validation_loss < best_validation_loss: best_validation_loss = validation_loss - torch.save({"model": model.state_dict(), "config": config.__dict__, "metrics": metrics, "best_validation_loss": best_validation_loss, "step": step + 1, "seed": args.seed, "corpus_sha256": corpus_sha256, "tokenizer": str(args.tokenizer) if args.tokenizer else "byte"}, args.output_dir / "best_checkpoint.pt") - torch.save({"model": model.state_dict(), "config": config.__dict__, "optimizer": optimizer.state_dict(), "rng_state": torch.get_rng_state(), "metrics": metrics, "best_validation_loss": best_validation_loss, "step": start_step + args.steps, "seed": args.seed, "corpus_sha256": corpus_sha256, "tokenizer": str(args.tokenizer) if args.tokenizer else "byte"}, output) + best = {"model": model.state_dict(), "config": config.__dict__, "metrics": metrics, "best_validation_loss": best_validation_loss, "step": step + 1, "seed": args.seed, "corpus_sha256": corpus_sha256, "tokenizer": str(args.tokenizer) if args.tokenizer else "byte", "attention_backend": args.attention_backend} + # File bytes vary with timing metadata; model_identity does not. + if not args.tokenizer: + best["model_identity"] = checkpoint_identity(best) + torch.save(best, args.output_dir / "best_checkpoint.pt") + final = {"model": model.state_dict(), "config": config.__dict__, "optimizer": optimizer.state_dict(), "rng_state": torch.get_rng_state(), "metrics": metrics, "best_validation_loss": best_validation_loss, "step": start_step + args.steps, "seed": args.seed, "corpus_sha256": corpus_sha256, "tokenizer": str(args.tokenizer) if args.tokenizer else "byte", "attention_backend": args.attention_backend} + if not args.tokenizer: + final["model_identity"] = checkpoint_identity(final) + torch.save(final, output) (args.output_dir / "metrics.json").write_text(json.dumps(metrics, indent=2), encoding="utf-8") + # Timing and memory of this process only (a resumed run reports its own segment). + summary = { + "attention_backend": args.attention_backend, + "device": device, + "steps": args.steps, + "train_seconds": train_seconds, + "train_tokens": train_tokens, + "train_tokens_per_second": train_tokens / train_seconds, + "eval_seconds": eval_seconds, + "process_peak_rss_bytes": peak_rss_bytes(), + # Identity-relevant training hyperparameters (record schema v2). + "hyperparameters": { + "steps": args.steps, "batch_size": args.batch_size, "gradient_accumulation_steps": args.gradient_accumulation_steps, + "eval_interval": args.eval_interval, "learning_rate": 3e-4, "optimizer": "AdamW", "device": device, + }, + } + (args.output_dir / "run_summary.json").write_text(json.dumps(summary, indent=2), encoding="utf-8") print(f"saved {output}") diff --git a/scripts/validate_records.py b/scripts/validate_records.py new file mode 100644 index 0000000..7807b1e --- /dev/null +++ b/scripts/validate_records.py @@ -0,0 +1,49 @@ +"""Validate every experiment record (rules: docs/RECORD_SCHEMA.md). + +Checks naming and provenance fields, recorded commits, dataset identities against +the protocol manifests, training duplicates and conflicts, training-to-evaluation +provenance, model-identity evidence, benchmark summaries and derived ratios against +raw samples, and the supersession index. Exits with status 1 if any check fails. + +--strict requires every current training checkpoint's model identity to be verified +from the checkpoint artifact or the pinned audited baseline (release gate). +--require-artifacts requires verification from the checkpoint artifacts themselves. +""" + +import argparse +import sys +from collections import Counter +from pathlib import Path + +from minilm_lab.records import identity_evidence, load_store, validate_store + +ROOT = Path(__file__).resolve().parents[1] +RECORDS = ROOT / "experiments/records" + + +def validate(records_root: Path | None = None, strict: bool = False, require_artifacts: bool = False) -> tuple[list[str], int]: + store = load_store(ROOT, records_root or RECORDS) + return validate_store(store, strict=strict, require_artifacts=require_artifacts), len(store.records) + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("--records-dir", type=Path, default=RECORDS) + parser.add_argument("--strict", action="store_true") + parser.add_argument("--require-artifacts", action="store_true") + parser.add_argument("--artifact-root", type=Path, default=ROOT, help="where checkpoint artifacts are looked up") + parser.add_argument("--no-baseline", action="store_true", help="ignore the pinned audited baseline") + args = parser.parse_args() + store = load_store(ROOT, args.records_dir, artifact_root=args.artifact_root, baseline_path=None if args.no_baseline else True) + errors = validate_store(store, strict=args.strict, require_artifacts=args.require_artifacts) + levels = Counter(found.status for found in identity_evidence(store, []).values()) + for error in errors: + print(f"FAIL {error}") + print("model identity evidence (checkpoints): " + ", ".join(f"{level}={count}" for level, count in sorted(levels.items()))) + mode = "require-artifacts" if args.require_artifacts else "strict" if args.strict else "standard" + print(f"validated {len(store.records)} records ({mode}): {'OK' if not errors else f'{len(errors)} problems'}") + sys.exit(1 if errors else 0) + + +if __name__ == "__main__": + main() diff --git a/scripts/verify_release.sh b/scripts/verify_release.sh new file mode 100755 index 0000000..954376d --- /dev/null +++ b/scripts/verify_release.sh @@ -0,0 +1,15 @@ +#!/usr/bin/env bash +# Release verification. Any failing step (including inside a pipeline) stops the +# script with a non-zero exit status. Run from a clean checkout. +set -euo pipefail +cd "$(dirname "$0")/.." +PY=${PYTHON:-python} +step() { echo "== $*"; } +step "lint"; "$PY" -m ruff check . +step "tests"; "$PY" -m pytest -q -rs # -rs prints every skip reason (hardware, artifacts) +# --strict: model identities must be verified from checkpoint artifacts or the pinned +# audited baseline; metadata-only consistency fails the release. +step "record validation"; "$PY" scripts/validate_records.py --strict | tail -n 2 +step "report regeneration"; "$PY" scripts/build_experiment_report.py +step "documentation drift"; git diff --exit-code -- docs/EXPERIMENT_RESULTS.md docs/figures +echo "release verification passed" diff --git a/src/minilm_lab/corpus.py b/src/minilm_lab/corpus.py new file mode 100644 index 0000000..63423b6 --- /dev/null +++ b/src/minilm_lab/corpus.py @@ -0,0 +1,130 @@ +"""Deterministic corpus cleaning, splitting and contamination checks (corpus protocol v2). + +Order of operations for every source book: + 1. extract the Project Gutenberg body (between the START and END markers) + 2. keep only the author's main text between two exact heading lines + 3. split into paragraphs on blank lines + 4. drop non-prose paragraphs by the rules in ``paragraph_drop_reason`` + 5. split train/validation at a paragraph boundary (train/validation books only) +Then held-out paragraphs that duplicate training text are removed and reported. +""" + +import re +from collections import Counter + +START = "*** START OF THE PROJECT GUTENBERG EBOOK" +END = "*** END OF THE PROJECT GUTENBERG EBOOK" +# An index, contents or table row: a short line ending in a number. +NUMBERED_LINE = re.compile(r"^.{0,80}?[\s,.]\d{1,4}\.?$") + + +def gutenberg_body(text: str) -> str: + start, end = text.find(START), text.find(END) + if start < 0 or end < 0 or end <= start: + raise ValueError("Project Gutenberg body markers not found") + return text[text.find("\n", start) + 1 : end].replace("\r\n", "\n") + + +def _line_offsets(text: str, line: str) -> list[int]: + return [match.start() for match in re.finditer(rf"(?m)^{re.escape(line)}[ \t]*$", text)] + + +def main_text(body: str, start_line: str, start_count: int, end_line: str | None, end_count: int) -> str: + """Text from the last of ``start_count`` start lines up to the last of ``end_count`` end lines. + + The expected counts are asserted so that upstream edits fail loudly. + """ + starts = _line_offsets(body, start_line) + if len(starts) != start_count: + raise ValueError(f"expected {start_count} '{start_line}' lines, found {len(starts)}") + end = len(body) + if end_line is not None: + ends = _line_offsets(body, end_line) + if len(ends) != end_count: + raise ValueError(f"expected {end_count} '{end_line}' lines, found {len(ends)}") + end = ends[-1] + if end <= starts[-1]: + raise ValueError("end marker precedes start marker") + return body[starts[-1] : end] + + +def paragraphs(text: str) -> list[str]: + return [part.strip("\n") for part in re.split(r"\n\s*\n", text) if part.strip()] + + +def paragraph_drop_reason(paragraph: str) -> str | None: + stripped = paragraph.strip() + if not re.search(r"[a-z]", stripped): + return "no_lowercase_heading" + if stripped.startswith("[") and stripped.endswith("]"): + return "bracketed_editorial" + lines = [line.strip() for line in stripped.splitlines() if line.strip()] + # Multi-line only: single lines ending in a number are often equations or dates. + if len(lines) >= 2 and sum(bool(NUMBERED_LINE.match(line)) for line in lines) * 2 >= len(lines): + return "numeric_list_or_table" + return None + + +def clean(paragraph_list: list[str]) -> tuple[list[str], Counter]: + kept, dropped = [], Counter() + for paragraph in paragraph_list: + reason = paragraph_drop_reason(paragraph) + if reason: + dropped[reason] += 1 + else: + kept.append(paragraph) + return kept, dropped + + +def split_by_bytes(paragraph_list: list[str], train_fraction: float = 0.9) -> int: + """Index of the first held-out paragraph (same rule as corpus protocol v1).""" + total = sum(len(p.encode("utf-8")) for p in paragraph_list) + split, used = 0, 0 + while split < len(paragraph_list) - 1 and used < total * train_fraction: + used += len(paragraph_list[split].encode("utf-8")) + split += 1 + return split + + +def normalize(text: str) -> str: + return " ".join(re.findall(r"[a-z0-9]+", text.lower())) + + +def word_ngrams(text: str, n: int) -> set[tuple[str, ...]]: + words = normalize(text).split() + return {tuple(words[i : i + n]) for i in range(len(words) - n + 1)} + + +def contamination(reference: list[str], candidates: list[str], n: int = 8, threshold: float = 0.5) -> dict: + """Exact and near-duplicate overlap of candidate paragraphs with reference text. + + Exact: identical after lowercasing and keeping only letters and digits. + Near: at least ``threshold`` of a paragraph's word n-grams occur in the reference. + Paragraphs with fewer than n words can only be exact duplicates. + """ + exact_reference = {normalize(p) for p in reference} + reference_ngrams = set().union(*(word_ngrams(p, n) for p in reference)) if reference else set() + exact, near, overlaps = [], [], [] + for index, paragraph in enumerate(candidates): + if normalize(paragraph) in exact_reference: + exact.append(index) + continue + grams = word_ngrams(paragraph, n) + if grams: + share = len(grams & reference_ngrams) / len(grams) + overlaps.append(share) + if share >= threshold: + near.append(index) + return { + "paragraphs": len(candidates), + "exact_duplicates": exact, + "near_duplicates": near, + "ngram_n": n, + "near_threshold": threshold, + "mean_ngram_overlap": sum(overlaps) / len(overlaps) if overlaps else 0.0, + } + + +def remove_indices(paragraph_list: list[str], indices: list[int]) -> list[str]: + drop = set(indices) + return [p for i, p in enumerate(paragraph_list) if i not in drop] diff --git a/src/minilm_lab/identity.py b/src/minilm_lab/identity.py new file mode 100644 index 0000000..f957ef0 --- /dev/null +++ b/src/minilm_lab/identity.py @@ -0,0 +1,74 @@ +"""Deterministic model identity, independent of checkpoint file bytes. + +A checkpoint file's SHA-256 identifies an *artifact*: it changes whenever +runtime metadata such as elapsed seconds changes. The model identity digest +identifies the *model*: architecture configuration, tokenizer and every +tensor's name, dtype, shape and raw values. + +Scheme ``minilm-model-identity/v1`` hashes this byte stream with SHA-256: + +1. ``b"minilm-model-identity/v1\\n"`` +2. canonical JSON (sorted keys, ``separators=(",", ":")``, ASCII, no NaN) of + ``{"config": , "tokenizer": }`` + ``b"\\n"`` +3. for each state-dict entry in ascending name order: + canonical JSON ``{"dtype": "torch.float32", "name": ..., "nbytes": n, "shape": [...]}`` + + ``b"\\n"`` + the tensor's elements in C (row-major) order, each element's + bytes in little-endian order, in the tensor's own dtype (no casting). + +Excluded: elapsed time, metrics, optimizer and RNG state, file paths, timestamps, +seeds and the attention backend (the same weights run on every backend). +""" + +import hashlib +import json +import sys +from dataclasses import fields + +import torch +from torch import Tensor + +from .model import MiniLMConfig + +SCHEME = "minilm-model-identity/v1" + + +def _canonical_json(value) -> bytes: + return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=True, allow_nan=False).encode("ascii") + + +def identity_config(config: dict | MiniLMConfig) -> dict: + """Every architecture field; legacy configs missing a field take its default.""" + values = config if isinstance(config, MiniLMConfig) else MiniLMConfig(**config) + return {field.name: getattr(values, field.name) for field in fields(MiniLMConfig)} + + +def tensor_bytes(tensor: Tensor) -> bytes: + """Raw element bytes, little-endian, row-major, without changing dtype or the tensor.""" + view = tensor.detach() + if view.device.type != "cpu": + view = view.to("cpu") # a copy; the original tensor is untouched + flat = view.contiguous().reshape(-1) + raw = flat.view(torch.uint8) if flat.numel() else torch.empty(0, dtype=torch.uint8) + if sys.byteorder == "big" and flat.element_size() > 1: + raw = raw.reshape(-1, flat.element_size()).flip(-1).reshape(-1) + return raw.numpy().tobytes() + + +def model_identity(state_dict: dict[str, Tensor], config: dict | MiniLMConfig, tokenizer: str = "byte") -> str: + digest = hashlib.sha256() + digest.update(SCHEME.encode("ascii") + b"\n") + digest.update(_canonical_json({"config": identity_config(config), "tokenizer": tokenizer}) + b"\n") + for name in sorted(state_dict): + tensor = state_dict[name] + header = {"dtype": str(tensor.dtype), "name": name, "nbytes": tensor.numel() * tensor.element_size(), "shape": list(tensor.shape)} + digest.update(_canonical_json(header) + b"\n") + digest.update(tensor_bytes(tensor)) + return digest.hexdigest() + + +def checkpoint_identity(checkpoint: dict) -> dict: + """Identity of a loaded checkpoint dictionary (current or legacy format).""" + tokenizer = checkpoint.get("tokenizer", "byte") + if tokenizer != "byte": + raise ValueError("model identity is defined for byte-tokenizer checkpoints only") + return {"scheme": SCHEME, "sha256": model_identity(checkpoint["model"], checkpoint["config"], tokenizer)} diff --git a/src/minilm_lab/kv_cache.py b/src/minilm_lab/kv_cache.py index ad7488b..75bb2d6 100644 --- a/src/minilm_lab/kv_cache.py +++ b/src/minilm_lab/kv_cache.py @@ -28,3 +28,27 @@ def generate_with_kv_cache(model, prompt: str, tokenizer: ByteTokenizer, max_new else: logits, cache = model.forward_cached(next_token, cache) return tokenizer.decode(tokens[0].tolist()) + + +@torch.no_grad() +def greedy_decode(model, input_ids: torch.Tensor, max_new_tokens: int, use_cache: bool = True) -> torch.Tensor: + """Deterministic batched argmax decoding, with or without the KV cache. + + Both modes see the same sliding window of the last context_length tokens, + so their outputs are directly comparable. + """ + if input_ids.dim() != 2 or input_ids.size(1) < 1: + raise ValueError("input_ids must have shape (batch, length >= 1)") + model.eval() + window = model.config.context_length + tokens = input_ids + cache = None + for index in range(max_new_tokens): + if not use_cache: + logits, _ = model(tokens[:, -window:]) + elif cache is None or tokens.size(1) > window: + logits, cache = model.forward_cached(tokens[:, -window:]) + else: + logits, cache = model.forward_cached(tokens[:, -1:], cache) + tokens = torch.cat((tokens, logits[:, -1].argmax(dim=-1, keepdim=True)), dim=1) + return tokens diff --git a/src/minilm_lab/model.py b/src/minilm_lab/model.py index 836ad22..5479bad 100644 --- a/src/minilm_lab/model.py +++ b/src/minilm_lab/model.py @@ -34,10 +34,54 @@ def forward(self, x: Tensor) -> Tensor: return x * torch.rsqrt(variance + self.eps).to(dtype=x.dtype) * self.weight +# sdpa: torch scaled_dot_product_attention (default since the first release). +# mha: the original path that produced every science-v1 result: +# nn.MultiheadAttention for learned positions in forward(), SDPA for RoPE +# and for forward_cached(). +# reference: explicit softmax(QK^T / sqrt(d)) V for numerical comparison. +ATTENTION_BACKENDS = ("sdpa", "mha", "reference") +DEFAULT_ATTENTION_BACKEND = "sdpa" +# Records and checkpoints written before the first release call the mha path "auto". +LEGACY_BACKEND_ALIASES = {"auto": "mha"} + + +def canonical_backend(name: str | None) -> str: + """Canonical backend name; checkpoints without a backend predate the field and used mha.""" + if name is None: + return "mha" + name = LEGACY_BACKEND_ALIASES.get(name, name) + if name not in ATTENTION_BACKENDS: + raise ValueError(f"attention backend must be one of {ATTENTION_BACKENDS} (or legacy 'auto')") + return name + + +def causal_mask(query_length: int, key_length: int, device: torch.device) -> Tensor: + """Boolean mask, True where attention is allowed, with queries occupying the last positions.""" + past_length = key_length - query_length + key_positions = torch.arange(key_length, device=device) + query_positions = past_length + torch.arange(query_length, device=device) + return key_positions[None, :] <= query_positions[:, None] + + +def reference_causal_attention(q: Tensor, k: Tensor, v: Tensor, dropout_p: float = 0.0) -> Tensor: + """Explicit softmax(QK^T / sqrt(d)) V that materializes the full score matrix. + + Matmuls run in the input dtype; the softmax runs in at least float32 (as in + common eager implementations), never below the input precision, so float64 + inputs stay float64. Queries are aligned to the last key positions. + """ + scores = q @ k.transpose(-2, -1) + scores = scores.to(torch.promote_types(scores.dtype, torch.float32)) * q.size(-1) ** -0.5 + scores = scores.masked_fill(~causal_mask(q.size(-2), k.size(-2), q.device), float("-inf")) + weights = scores.softmax(dim=-1).to(v.dtype) + return F.dropout(weights, dropout_p) @ v if dropout_p else weights @ v + + class CausalSelfAttention(nn.Module): def __init__(self, config: MiniLMConfig) -> None: super().__init__() self.rope_theta = config.rope_theta + self.backend = DEFAULT_ATTENTION_BACKEND self.attention = nn.MultiheadAttention( config.embedding_dim, config.num_heads, config.dropout, batch_first=True ) @@ -45,13 +89,7 @@ def __init__(self, config: MiniLMConfig) -> None: "mask", torch.triu(torch.ones(config.context_length, config.context_length), diagonal=1).bool() ) - def forward_cached( - self, - x: Tensor, - past: tuple[Tensor, Tensor] | None = None, - use_rope: bool = False, - position_offset: int = 0, - ) -> tuple[Tensor, tuple[Tensor, Tensor]]: + def _project(self, x: Tensor, use_rope: bool, position_offset: int) -> tuple[Tensor, Tensor, Tensor]: batch, length, _ = x.shape heads = self.attention.num_heads width = x.size(-1) // heads @@ -63,49 +101,56 @@ def forward_cached( if use_rope: q = self._apply_rope(q, position_offset) k = self._apply_rope(k, position_offset) - if past is not None: - k = torch.cat((past[0], k), dim=2) - v = torch.cat((past[1], v), dim=2) - if past is None: - output = F.scaled_dot_product_attention(q, k, v, is_causal=True) + return q, k, v + + def _attend(self, q: Tensor, k: Tensor, v: Tensor) -> Tensor: + # Same attention-weight dropout as nn.MultiheadAttention in training. + dropout = self.attention.dropout if self.training else 0.0 + if self.backend == "reference": + output = reference_causal_attention(q, k, v, dropout) + elif q.size(2) == k.size(2): + output = F.scaled_dot_product_attention(q, k, v, dropout_p=dropout, is_causal=True) else: # Non-square is_causal is upper-left aligned in PyTorch. Align # each query with its absolute position in the concatenated keys. - past_length = past[0].size(2) - key_positions = torch.arange(k.size(2), device=x.device) - query_positions = past_length + torch.arange(length, device=x.device) - mask = key_positions[None, :] <= query_positions[:, None] - output = F.scaled_dot_product_attention(q, k, v, attn_mask=mask) - output = output.transpose(1, 2).reshape(batch, length, -1) - return self.attention.out_proj(output), (k, v) + mask = causal_mask(q.size(2), k.size(2), q.device) + output = F.scaled_dot_product_attention(q, k, v, attn_mask=mask, dropout_p=dropout) + return self.attention.out_proj(output.transpose(1, 2).reshape(q.size(0), q.size(2), -1)) + + def forward_cached( + self, + x: Tensor, + past: tuple[Tensor, Tensor] | None = None, + use_rope: bool = False, + position_offset: int = 0, + ) -> tuple[Tensor, tuple[Tensor, Tensor]]: + q, k, v = self._project(x, use_rope, position_offset) + if past is not None: + k = torch.cat((past[0], k), dim=2) + v = torch.cat((past[1], v), dim=2) + return self._attend(q, k, v), (k, v) + def _apply_rope(self, x: Tensor, position_offset: int = 0) -> Tensor: length = x.size(2) half = x.size(-1) // 2 - positions = torch.arange(position_offset, position_offset + length, device=x.device, dtype=x.dtype) - frequencies = 1.0 / (self.rope_theta ** (torch.arange(half, device=x.device, dtype=x.dtype) / half)) + # Angles are computed in float32: half-precision positions and products + # lose integer resolution and produce large rotation errors. + positions = torch.arange(position_offset, position_offset + length, device=x.device, dtype=torch.float32) + frequencies = 1.0 / (self.rope_theta ** (torch.arange(half, device=x.device, dtype=torch.float32) / half)) angles = positions[:, None] * frequencies[None, :] - cos, sin = angles.cos()[None, None, :, :], angles.sin()[None, None, :, :] + cos = angles.cos()[None, None, :, :].to(x.dtype) + sin = angles.sin()[None, None, :, :].to(x.dtype) first, second = x[..., :half], x[..., half : 2 * half] rotated = torch.cat((first * cos - second * sin, first * sin + second * cos), dim=-1) return torch.cat((rotated, x[..., 2 * half :]), dim=-1) def forward(self, x: Tensor, use_rope: bool = False) -> Tensor: - length = x.size(1) - if use_rope: - # This path is kept explicit for the ablation; PyTorch's attention - # still handles masking and optimized kernels. - qkv = F.linear(x, self.attention.in_proj_weight, self.attention.in_proj_bias) - q, k, v = qkv.chunk(3, dim=-1) - heads = self.attention.num_heads - width = x.size(-1) // heads - q = self._apply_rope(q.reshape(x.size(0), length, heads, width).transpose(1, 2)) - k = self._apply_rope(k.reshape(x.size(0), length, heads, width).transpose(1, 2)) - v = v.reshape(x.size(0), length, heads, width).transpose(1, 2) - output = torch.nn.functional.scaled_dot_product_attention(q, k, v, is_causal=True).transpose(1, 2).reshape(x.size(0), length, -1) - output = self.attention.out_proj(output) - else: + if self.backend == "mha" and not use_rope: + length = x.size(1) output, _ = self.attention(x, x, x, attn_mask=self.mask[:length, :length], need_weights=False) - return output + return output + # RoPE needs explicit q/k access, so it always uses _attend. + return self._attend(*self._project(x, use_rope, 0)) class TransformerBlock(nn.Module): @@ -166,6 +211,15 @@ def __init__(self, config: MiniLMConfig | None = None) -> None: self.lm_head.weight = self.token_embedding.weight nn.init.normal_(self.token_embedding.weight, mean=0.0, std=0.02) + def set_attention_backend(self, backend: str) -> "MiniLM": + """Select how attention is computed without changing any weights.""" + if backend is None: + raise ValueError("attention backend must be named explicitly") + backend = canonical_backend(backend) + for block in self.blocks: + block.attention.backend = backend + return self + def forward(self, input_ids: Tensor, targets: Tensor | None = None) -> tuple[Tensor, Tensor | None]: _, length = input_ids.shape if length > self.config.context_length: diff --git a/src/minilm_lab/provenance.py b/src/minilm_lab/provenance.py new file mode 100644 index 0000000..ec6c460 --- /dev/null +++ b/src/minilm_lab/provenance.py @@ -0,0 +1,156 @@ +"""Environment capture and append-only experiment records.""" + +import hashlib +import json +import os +import platform +import resource +import shlex +import statistics +import subprocess +import sys +from datetime import datetime, timezone +from pathlib import Path + +import numpy +import torch +from torch.utils._python_dispatch import TorchDispatchMode + +ROOT = Path(__file__).resolve().parents[2] +# 1: original schema. 2: training/evaluation records carry deterministic model +# identity digests and explicit hyperparameters; benchmarks keep raw samples. +RECORD_VERSION = 2 + + +def _run(command: list[str]) -> str | None: + try: + return subprocess.run(command, capture_output=True, text=True, check=True, cwd=ROOT).stdout.strip() + except (OSError, subprocess.CalledProcessError): + return None + + +def git_state() -> dict: + status = _run(["git", "status", "--porcelain", "--untracked-files=no"]) + return { + "commit": _run(["git", "rev-parse", "HEAD"]), + "branch": _run(["git", "rev-parse", "--abbrev-ref", "HEAD"]), + # Uncommitted tracked changes mean the commit does not fully describe the code. + "dirty": bool(status) if status is not None else None, + } + + +def environment(device: str) -> dict: + info = { + "python": sys.version.split()[0], + "torch": torch.__version__, + "numpy": numpy.__version__, + "platform": platform.platform(), + "machine": platform.machine(), + "processor": _run(["sysctl", "-n", "machdep.cpu.brand_string"]) or platform.processor(), + "memory_bytes": int(_run(["sysctl", "-n", "hw.memsize"]) or 0) or None, + "cpu_count": os.cpu_count(), + "torch_threads": torch.get_num_threads(), + "device": device, + "mps_available": torch.backends.mps.is_available(), + "cuda_available": torch.cuda.is_available(), + "git": git_state(), + "command": shlex.join([Path(sys.executable).name, *sys.argv]), + } + if device == "cuda": + info["cuda_device"] = torch.cuda.get_device_name() + return info + + +def file_sha256(path: str | Path) -> str: + return hashlib.sha256(Path(path).read_bytes()).hexdigest() + + +def synchronize(device: str) -> None: + if device == "mps": + torch.mps.synchronize() + elif device == "cuda": + torch.cuda.synchronize() + + +def peak_rss_bytes() -> int: + """Process lifetime peak resident set size (macOS reports bytes, Linux KiB).""" + peak = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss + return peak if sys.platform == "darwin" else peak * 1024 + + +class PeakAllocated(TorchDispatchMode): + """Sample the MPS allocator's live bytes after every aten op (forward and backward).""" + + def __init__(self) -> None: + super().__init__() + self.peak = torch.mps.current_allocated_memory() + + def __torch_dispatch__(self, func, types, args=(), kwargs=None): + output = func(*args, **(kwargs or {})) + self.peak = max(self.peak, torch.mps.current_allocated_memory()) + return output + + +def measure_peak_memory(device: str, operation) -> tuple[int, str]: + """Extra memory used by operation() and how it was measured. + + cuda: device allocator peak (torch.cuda.max_memory_allocated) minus allocation before. + mps: live allocator bytes sampled after every op minus allocation before. + cpu: process peak-RSS delta (host memory; includes allocator caching). + Host RSS is never reported for an accelerator. + """ + synchronize(device) + if device == "cuda": + torch.cuda.reset_peak_memory_stats() + before = torch.cuda.memory_allocated() + operation() + synchronize(device) + return torch.cuda.max_memory_allocated() - before, "cuda_max_memory_allocated_delta" + if device == "mps": + before = torch.mps.current_allocated_memory() + with PeakAllocated() as tracker: + operation() + synchronize(device) + return tracker.peak - before, "mps_live_allocated_peak_per_op" + if device != "cpu": + raise ValueError(f"no peak-memory method for device {device!r}") + before = peak_rss_bytes() + operation() + return peak_rss_bytes() - before, "process_peak_rss_delta" + + +def summarize(samples: list[float]) -> dict: + ordered = sorted(samples) + count = len(ordered) + + def quantile(fraction: float) -> float: + position = fraction * (count - 1) + low = int(position) + high = min(low + 1, count - 1) + return ordered[low] + (ordered[high] - ordered[low]) * (position - low) + + mean = statistics.fmean(ordered) + return { + "count": count, + "median": quantile(0.5), + "p95": quantile(0.95), + "min": ordered[0], + "max": ordered[-1], + "iqr": quantile(0.75) - quantile(0.25), + "mean": mean, + "cv": statistics.stdev(ordered) / mean if count > 1 and mean else 0.0, + } + + +def write_record(directory: str | Path, name: str, payload: dict) -> Path: + """Write a new JSON record. Existing records are never overwritten.""" + directory = Path(directory) + directory.mkdir(parents=True, exist_ok=True) + stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ") + commit = (payload.get("environment", {}).get("git", {}).get("commit") or "nogit")[:7] + path = directory / f"{stamp}_{name}_{commit}.json" + record = {"record_version": RECORD_VERSION, "created_utc": stamp, **payload} + with path.open("x", encoding="utf-8") as handle: + json.dump(record, handle, indent=2) + handle.write("\n") + return path diff --git a/src/minilm_lab/records.py b/src/minilm_lab/records.py new file mode 100644 index 0000000..b5e313b --- /dev/null +++ b/src/minilm_lab/records.py @@ -0,0 +1,768 @@ +"""Experiment-record loading, identity, selection and provenance validation. + +The rules are documented in docs/RECORD_SCHEMA.md. Both scripts/validate_records.py +and scripts/build_experiment_report.py use this module, so a record set that fails +validation cannot be reported. +""" + +import hashlib +import json +import math +import re +import shlex +import subprocess +from dataclasses import dataclass, field +from pathlib import Path +from typing import NamedTuple + +from .identity import identity_config +from .model import canonical_backend +from .provenance import summarize + +NAME = re.compile(r"^(?P\d{8}T\d{6}Z)_(?P.+)_(?P[0-9a-f]{7})\.json$") +ENVIRONMENT_FIELDS = ("python", "torch", "platform", "processor", "device", "torch_threads", "git", "command") +PROTOCOLS = { + "science-v1": {"manifest": "data/science/manifest.json", "data_dir": "data/science"}, + "science-v2": {"manifest": "data/science-v2/manifest.json", "data_dir": "data/science-v2"}, +} +# train_tiny.py defaults (unchanged since 1fb8f37); used only to read records that +# predate explicit hyperparameters (record_version 1). Guarded by a test. +LEGACY_TRAIN_DEFAULTS = {"steps": 100, "batch_size": 8, "gradient_accumulation_steps": 1, "eval_interval": 25, "device": "auto"} +# Model-identity evidence levels, strongest first. Only the first two count as verification. +ARTIFACT_VERIFIED = "artifact_verified" +BASELINE_VERIFIED = "trusted_baseline_verified" +METADATA_ONLY = "metadata_consistent_only" +UNVERIFIED = "unverified" +VERIFIED = (ARTIFACT_VERIFIED, BASELINE_VERIFIED) +# Audited baseline of model identities for historical checkpoints, built by +# scripts/build_identity_baseline.py from the original checkpoint files after checking +# their bytes against the recorded SHA-256. It is trusted only if its SHA-256 equals the +# value pinned here, so any change to it is a reviewable code change. This is an +# integrity anchor inside this repository, not a cryptographic signature. +BASELINE_PATH = "experiments/baselines/model_identity_baseline.json" +BASELINE_SHA256 = "02992d75cde737c37953cc7156cdbf08d43b910ad4347fae9d4bab178fdd2439" +LEGACY_FLAGS = {"--steps": "steps", "--batch-size": "batch_size", "--gradient-accumulation-steps": "gradient_accumulation_steps", + "--eval-interval": "eval_interval", "--device": "device"} + + +class RecordError(ValueError): + """Raised when records cannot be reported without an unresolved problem.""" + + +def sha256_bytes(data: bytes) -> str: + return hashlib.sha256(data).hexdigest() + + +def canonical_json_digest(value) -> str: + return sha256_bytes(json.dumps(value, sort_keys=True, separators=(",", ":"), allow_nan=False).encode()) + + +_ARTIFACT_CACHE: dict = {} + + +@dataclass +class Store: + repo_root: Path + records_root: Path + records: dict[str, dict] + superseded: dict[str, dict] + artifact_root: Path | None = None # where checkpoint artifacts are looked up (default: repo_root) + baseline_path: Path | None = None # None: no audited baseline available + _file_hashes: dict = field(default_factory=dict) + _commits: dict = field(default_factory=dict) + _evidence: dict = field(default_factory=dict) + _baseline: tuple | None = None + + def current(self, directory: str) -> list[tuple[str, dict]]: + """(key, record) in time order, excluding superseded records.""" + return [(k, r) for k, r in sorted(self.records.items()) if k.startswith(directory + "/") and k not in self.superseded] + + def file_sha256(self, path: str) -> str | None: + target = Path(path) if Path(path).is_absolute() else self.repo_root / path + if target not in self._file_hashes: + self._file_hashes[target] = sha256_bytes(target.read_bytes()) if target.is_file() else None + return self._file_hashes[target] + + def commit_exists(self, ref: str) -> bool: + if ref not in self._commits: + result = subprocess.run(["git", "cat-file", "-e", f"{ref}^{{commit}}"], check=False, cwd=self.repo_root, capture_output=True) + self._commits[ref] = result.returncode == 0 and bool(re.fullmatch(r"[0-9a-f]{7,40}", ref)) + return self._commits[ref] + + +def load_store(repo_root: Path, records_root: Path | None = None, artifact_root: Path | None = None, + baseline_path: Path | None | bool = True) -> Store: + """baseline_path=True uses the committed baseline; None/False disables it (for tests).""" + records_root = records_root or repo_root / "experiments/records" + records = {} + for path in sorted(records_root.glob("*/*.json")): + record = json.loads(path.read_text(encoding="utf-8")) + key = str(path.relative_to(records_root)) + record["_key"] = key + record["_path"] = str(path.relative_to(repo_root)) if path.is_relative_to(repo_root) else str(path) + records[key] = record + index = records_root / "SUPERSEDED.json" + superseded = json.loads(index.read_text())["records"] if index.exists() else {} + baseline = repo_root / BASELINE_PATH if baseline_path is True else (baseline_path or None) + return Store(repo_root, records_root, records, superseded, artifact_root or repo_root, baseline) + + +# ---------------------------------------------------------------- datasets + + +def protocol_files(store: Store, protocol: str) -> dict[str, str]: + """Published file hashes for a protocol, read from its committed manifest.""" + spec = PROTOCOLS[protocol] + manifest = json.loads((store.repo_root / spec["manifest"]).read_text()) + if protocol == "science-v1": + return {f"{spec['data_dir']}/train.txt": manifest["train_sha256"], f"{spec['data_dir']}/validation.txt": manifest["validation_sha256"]} + return {f"{spec['data_dir']}/{name}": info["sha256"] for name, info in manifest["files"].items()} + + +def relative_data_path(store: Store, path: str) -> str: + candidate = Path(path) + if candidate.is_absolute(): + # Older records stored absolute paths; resolve them against this repository. + parts = candidate.parts + for index in range(len(parts)): + if parts[index] == "data": + return str(Path(*parts[index:])) + return str(candidate) + + +def check_split(store: Store, protocol: str, label: str, info: dict, problems: list[str], where: str) -> None: + path = relative_data_path(store, info["path"]) + if not path.startswith(PROTOCOLS[protocol]["data_dir"] + "/"): + problems.append(f"{where}: {label} file {path} is outside the {protocol} corpus") + return + actual = store.file_sha256(path) + if actual is None: + problems.append(f"{where}: {label} file {path} does not exist") + elif actual != info["sha256"]: + problems.append(f"{where}: {label} hash {info['sha256'][:12]} differs from {path} ({actual[:12]})") + published = protocol_files(store, protocol).get(path) + if published is not None and published != info["sha256"]: + problems.append(f"{where}: {label} hash differs from the {protocol} manifest") + + +# ---------------------------------------------------------------- training identity + + +class TrainingIdentity(NamedTuple): + protocol: str + backend: str + seed: int + config_sha256: str + dataset_sha256: str + + +def training_hyperparameters(record: dict) -> dict: + summary = record["results"].get("run_summary") or {} + if "hyperparameters" in summary: + return summary["hyperparameters"] + # Legacy rule (record_version 1): parse the recorded command with train_tiny defaults. + values = dict(LEGACY_TRAIN_DEFAULTS) + tokens = shlex.split(record["settings"]["training_command"]) + for flag, name in LEGACY_FLAGS.items(): + if flag in tokens: + raw = tokens[tokens.index(flag) + 1] + values[name] = raw if name == "device" else int(raw) + if values["device"] == "auto" and summary.get("device"): + values["device"] = summary["device"] + return {**values, "learning_rate": 3e-4, "optimizer": "AdamW"} + + +def training_identity(record: dict) -> TrainingIdentity: + settings = record["settings"] + config = { + "model": identity_config(settings["model_config"]), + "tokenizer": settings.get("tokenizer", "byte"), + "hyperparameters": training_hyperparameters(record), + } + dataset = {split: settings["data"][split]["sha256"] for split in ("train", "validation")} + return TrainingIdentity(settings["protocol"], canonical_backend(settings.get("attention_backend")), settings["seed"], + canonical_json_digest(config), canonical_json_digest(dataset)) + + +def training_outcome(record: dict) -> tuple: + results = record["results"] + metrics = tuple((m["step"], m["validation_loss"], m["train_loss"]) for m in results["metrics"]) + return metrics, results["best_step"] + + +def check_training(store: Store, key: str, record: dict, problems: list[str]) -> None: + settings = record["settings"] + protocol = settings.get("protocol") + if protocol not in PROTOCOLS: + problems.append(f"{key}: unknown or missing protocol {protocol!r}") + return + if not isinstance(settings.get("seed"), int): + problems.append(f"{key}: training record without integer seed") + published = protocol_files(store, protocol) + for split in ("train", "validation"): + info = settings["data"][split] + check_split(store, protocol, split, info, problems, key) + if info["sha256"] not in published.values(): + problems.append(f"{key}: {split} data is not part of the {protocol} corpus") + if "manifest_path" in settings["data"]: + manifest = settings["data"]["manifest_path"] + if manifest != PROTOCOLS[protocol]["manifest"]: + problems.append(f"{key}: manifest {manifest} does not belong to {protocol}") + elif store.file_sha256(manifest) != settings["data"].get("manifest_sha256"): + problems.append(f"{key}: manifest hash does not match {manifest}") + commit = settings.get("training_code_commit") + if not commit or not store.commit_exists(commit): + problems.append(f"{key}: training_code_commit {commit!r} is not in the repository") + if record.get("record_version", 1) >= 2 and not record["results"].get("best_model_identity_sha256"): + problems.append(f"{key}: schema v2 training record without model identity") + if "best_checkpoint_sha256" not in record["results"]: + problems.append(f"{key}: no checkpoint hash") + + +def duplicate_conflicts(store: Store, first: tuple[str, dict], other: tuple[str, dict]) -> list[str]: + """Reasons two records in the same experiment slot cannot be the same experiment.""" + (key_a, a), (key_b, b) = first, other + reasons = [] + if training_identity(a).config_sha256 != training_identity(b).config_sha256: + reasons.append("configuration differs") + if training_outcome(a) != training_outcome(b): + reasons.append("recorded losses or selected step differ") + identity_a, identity_b = resolve_identity(store, key_a)[0], resolve_identity(store, key_b)[0] + if identity_a.fingerprint is None or identity_b.fingerprint is None: + if a["results"].get("best_checkpoint_sha256") != b["results"].get("best_checkpoint_sha256"): + reasons.append("different checkpoint files without established model identities") + if (identity_a.sources and identity_a.fingerprint is None) or (identity_b.sources and identity_b.fingerprint is None): + reasons.append("model identity sources disagree") + elif identity_a.fingerprint != identity_b.fingerprint: + reasons.append("model identities differ") + return reasons + + +def select_training(store: Store, problems: list[str]) -> dict[TrainingIdentity, dict]: + """One record per experiment (rules in docs/RECORD_SCHEMA.md). + + Records are grouped by experiment slot (protocol, canonical backend, seed, dataset) + after backend canonicalization. Members of a slot are duplicates of one experiment + only if configuration, outcomes and resolved checkpoint identity all agree; then the + earliest record represents them (they are identical, so the choice changes nothing). + Any disagreement is a conflict: no record is selected and validation fails until an + explicit supersession resolves it. + """ + slots: dict[tuple, list[tuple[str, dict]]] = {} + for key, record in store.current("training"): + before = len(problems) + check_training(store, key, record, problems) + if len(problems) == before: + identity = training_identity(record) + slots.setdefault((identity.protocol, identity.backend, identity.seed, identity.dataset_sha256), []).append((key, record)) + selected = {} + for (protocol, backend, seed, _), members in slots.items(): + members.sort(key=lambda item: (item[1]["created_utc"], item[0])) + reasons = {key: duplicate_conflicts(store, members[0], (key, record)) for key, record in members[1:]} + conflicts = {key: why for key, why in reasons.items() if why} + if conflicts: + detail = "; ".join(f"{key}: {', '.join(why)}" for key, why in conflicts.items()) + problems.append(f"conflicting training records for {protocol}/{backend}/seed {seed} ({members[0][0]} vs {detail}); " + "supersede one explicitly") + continue + canonical = members[0][1] + canonical["_replicates"] = [key for key, _ in members[1:]] + canonical["_identity"] = training_identity(canonical) + selected[canonical["_identity"]] = canonical + return selected + + +# ---------------------------------------------------------------- evaluation provenance + + +class IdentityEvidence(NamedTuple): + fingerprint: str | None # None when unknown or when sources disagree + status: str # one of ARTIFACT_VERIFIED, BASELINE_VERIFIED, METADATA_ONLY, UNVERIFIED + sources: dict # source name -> fingerprint, as found + + +def load_baseline(store: Store, problems: list[str]) -> dict: + """Audited baseline entries keyed by checkpoint SHA-256, only if the file matches its pin.""" + if store._baseline is None: + issues: list[str] = [] + entries: dict = {} + path = store.baseline_path + if path is not None and path.is_file(): + data = path.read_bytes() + if sha256_bytes(data) != BASELINE_SHA256: + issues.append(f"{BASELINE_PATH}: SHA-256 {sha256_bytes(data)[:12]} does not match the pinned audited baseline; not trusted") + else: + entries = json.loads(data)["entries"] + store._baseline = (entries, issues) + entries, issues = store._baseline + problems.extend(issue for issue in issues if issue not in problems) + return entries + + +def checkpoint_artifact(store: Store, record: dict, kind: str) -> Path | None: + tokens = shlex.split(record["settings"].get("training_command", "")) + if "--output-dir" not in tokens: + return None + path = store.artifact_root / tokens[tokens.index("--output-dir") + 1] / ("best_checkpoint.pt" if kind == "best" else "tiny_checkpoint.pt") + return path if path.is_file() else None + + +def artifact_fingerprint(path: Path, expected_sha256: str) -> tuple[str, str | None]: + """(file SHA-256, model identity recomputed from tensors if the bytes match the expectation).""" + stat = path.stat() + key = (str(path.resolve()), stat.st_size, stat.st_mtime_ns) + if key not in _ARTIFACT_CACHE: + _ARTIFACT_CACHE[key] = [sha256_bytes(path.read_bytes()), None] + entry = _ARTIFACT_CACHE[key] + if entry[0] == expected_sha256 and entry[1] is None: + import torch # deferred: only needed when artifacts are present + + from .identity import checkpoint_identity + + entry[1] = checkpoint_identity(torch.load(path, map_location="cpu", weights_only=True))["sha256"] + return entry[0], entry[1] if entry[0] == expected_sha256 else None + + +def resolve_identity(store: Store, key: str, kind: str = "best") -> tuple[IdentityEvidence, list[str]]: + """Effective model identity of a training record's checkpoint and the evidence behind it. + + Sources: the record's declared fingerprint, attestation rows for the same checkpoint + hash, the pinned audited baseline, and the checkpoint artifact itself (bytes checked + against the recorded SHA-256, fingerprint recomputed from its tensors). All available + sources must agree; a disagreement yields no fingerprint and a problem. + """ + if (key, kind) in store._evidence: + return store._evidence[(key, kind)] + problems: list[str] = [] + record = store.records[key] + expected = record["results"].get(f"{kind}_checkpoint_sha256") + sources: dict[str, str] = {} + declared = record["results"].get(f"{kind}_model_identity_sha256") + if declared: + sources["declared"] = declared + attested = set() + for attestation_key, attestation in store.current("identity"): + for row in attestation["results"]["attested"]: + if expected and row["checkpoint_sha256"] == expected: + attested.add(row["model_identity_sha256"]) + if len(attested) > 1: + problems.append(f"{key}: {kind} checkpoint attested with {len(attested)} different model identities") + elif attested: + sources["attested"] = attested.pop() + baseline = load_baseline(store, problems).get(expected or "") + if baseline: + sources["baseline"] = baseline["model_identity_sha256"] + artifact = checkpoint_artifact(store, record, kind) if expected else None + if artifact is not None: + file_sha, fingerprint = artifact_fingerprint(artifact, expected) + if fingerprint is None: + problems.append(f"{key}: {kind} checkpoint {artifact.name} bytes ({file_sha[:12]}) do not match the recorded SHA-256 ({expected[:12]})") + else: + sources["artifact"] = fingerprint + values = set(sources.values()) + if len(values) > 1: + detail = ", ".join(f"{name}={value[:12]}" for name, value in sorted(sources.items())) + problems.append(f"{key}: {kind} model identity sources disagree ({detail})") + evidence = IdentityEvidence(None, UNVERIFIED, sources) + elif not values: + evidence = IdentityEvidence(None, UNVERIFIED, sources) + elif "artifact" in sources and not any("artifact" in p and "do not match" in p for p in problems): + evidence = IdentityEvidence(values.pop(), ARTIFACT_VERIFIED, sources) + elif "baseline" in sources: + evidence = IdentityEvidence(values.pop(), BASELINE_VERIFIED, sources) + else: + evidence = IdentityEvidence(values.pop(), METADATA_ONLY, sources) + if any("do not match the recorded" in p for p in problems): + evidence = IdentityEvidence(None, UNVERIFIED, sources) + store._evidence[(key, kind)] = (evidence, problems) + return evidence, problems + + +def check_attestations(store: Store, problems: list[str]) -> None: + """Attestation rows must point at current training records and their recorded hashes.""" + for key, record in store.current("identity"): + for row in record["results"]["attested"]: + training = store.records.get(row["training_record"]) + if training is None or row["training_record"] in store.superseded: + problems.append(f"{key}: attests {row['training_record']}, which is not a current training record") + elif training["results"].get(f"{row['kind']}_checkpoint_sha256") != row["checkpoint_sha256"]: + problems.append(f"{key}: {row['kind']} checkpoint hash differs from {row['training_record']}") + for row in record["results"].get("regeneration_checks", []): + training = row["training_record"] + if training not in store.records or training in store.superseded: + problems.append(f"{key}: regeneration check for unknown training record {training}") + continue + evidence, _ = resolve_identity(store, training) + if evidence.fingerprint is None or row["model_identity_equal"] != (evidence.fingerprint == row["model_identity_sha256"]): + problems.append(f"{key}: regeneration check for {training} is inconsistent with its resolved identity") + + +def model_identity_of(store: Store, training: dict) -> str | None: + return resolve_identity(store, training["_key"])[0].fingerprint + + +def training_by_checkpoint(store: Store) -> dict[str, list[tuple[str, dict]]]: + index: dict[str, list[tuple[str, dict]]] = {} + for key, record in store.current("training"): + index.setdefault(record["results"].get("best_checkpoint_sha256"), []).append((key, record)) + return index + + +def resolve_row(store: Store, where: str, row: dict, protocol: str | None, problems: list[str], index: dict) -> tuple[str, dict] | None: + matches = index.get(row.get("checkpoint_sha256"), []) + identities = set() + for _, candidate in matches: + try: + identities.add(training_identity(candidate)) + except (KeyError, TypeError): + identities.add(None) + # Several records may describe one checkpoint only as replicates of a single + # experiment identity (e.g. a legacy "auto" record and its "mha" re-record). + if not matches or len(identities) != 1 or None in identities: + problems.append(f"{where}: checkpoint {str(row.get('checkpoint_sha256'))[:12]} matches {len(matches)} current training records " + f"with {len(identities)} experiment identities (expected exactly one)") + return None + key, training = min(matches, key=lambda item: (item[1]["created_utc"], item[0])) + settings = training["settings"] + if protocol is not None and settings.get("protocol") != protocol: + problems.append(f"{where}: protocol {protocol} but training record {key} is {settings.get('protocol')}") + if "seed" in row and row["seed"] != settings.get("seed"): + problems.append(f"{where}: seed {row.get('seed')} but training record {key} has seed {settings.get('seed')}") + backend = row.get("attention_backend", row.get("backend")) + if "attention_backend" in row and canonical_backend(backend) != canonical_backend(settings.get("attention_backend")): + problems.append(f"{where}: backend {backend} but training record {key} used {settings.get('attention_backend')}") + if "selected_step" in row and row["selected_step"] != training["results"]["best_step"]: + problems.append(f"{where}: selected step {row['selected_step']} but training record {key} selected {training['results']['best_step']}") + if "corpus_sha256" in row and row["corpus_sha256"] != settings["data"].get("corpus_sha256_in_checkpoint"): + problems.append(f"{where}: corpus identity differs from training record {key}") + a, b = row.get("model_identity_sha256"), model_identity_of(store, training) + if a and b and a != b: + problems.append(f"{where}: model identity differs from training record {key}") + return key, training + + +def check_evaluation(store: Store, key: str, record: dict, problems: list[str], index: dict) -> list[tuple[dict, str]]: + settings = record["settings"] + protocol = settings.get("protocol") + if protocol not in PROTOCOLS: + problems.append(f"{key}: unknown or missing protocol {protocol!r}") + return [] + for name, info in settings["splits"].items(): + check_split(store, protocol, f"split {name}", info, problems, key) + resolved, seen = [], {} + for position, row in enumerate(record["results"]): + where = f"{key} row {position}" + if record.get("record_version", 1) >= 2 and not row.get("model_identity_sha256"): + problems.append(f"{where}: schema v2 evaluation row without model identity") + match = resolve_row(store, where, row, protocol, problems, index) + if match is None: + continue + training_key, training = match + validation = settings["splits"].get("validation") + if validation and validation["sha256"] != training["settings"]["data"]["validation"]["sha256"]: + problems.append(f"{where}: validation split differs from the one training record {training_key} selected on") + identity = training_identity(training) + if identity in seen: + problems.append(f"{where}: duplicates row {seen[identity]} for the same experiment identity") + seen[identity] = position + resolved.append((row, training_key)) + return resolved + + +# ---------------------------------------------------------------- benchmark raw samples + + +# Stored KV-cache ratios and how they are derived. Each is +# median over repeats of the uncached mode's total generation time (ms) +# ÷ median over repeats of the cached mode's total generation time (ms), +# where a repeat's total is the sum of its per-step seconds × 1000 (prefill included). +# Unitless; > 1 means the cache is faster. Records before the first release name the +# legacy MHA mode "uncached_default". +KV_RATIOS = { + "speedup_cached_vs_uncached_sdpa": "uncached_sdpa", + "speedup_cached_vs_uncached_mha": "uncached_mha", + "speedup_cached_vs_uncached_default": "uncached_default", +} +RATIO_TOLERANCE = 1e-12 # relative; derived and stored values come from identical arithmetic + + +def positive_finite(value) -> bool: + return isinstance(value, (int, float)) and not isinstance(value, bool) and math.isfinite(value) and value > 0 + + +def kv_total_ms_median(mode: dict) -> float: + """Median total generation time of a mode: from raw samples if present, else the stored summary.""" + if "raw" in mode: + return summarize([1000 * sum(steps) for steps in mode["raw"]["step_seconds"]])["median"] + return mode["total_ms"]["median"] + + +def kv_speedups(row: dict) -> dict[str, float]: + """Derived cache speedups for one KV scenario; raises ValueError on invalid timings.""" + cached = kv_total_ms_median(row["cached"]) + if not positive_finite(cached): + raise ValueError(f"cached median total time {cached!r} is not a positive finite number") + ratios = {} + for name, mode in KV_RATIOS.items(): + if mode in row: + uncached = kv_total_ms_median(row[mode]) + if not positive_finite(uncached): + raise ValueError(f"{mode} median total time {uncached!r} is not a positive finite number") + ratios[name] = uncached / cached + return ratios + + +def kv_summary_from_raw(raw: dict, batch: int, new_tokens: int) -> dict: + """KV-cache mode summaries recomputed from per-repeat, per-step seconds.""" + runs = raw["step_seconds"] + totals = [sum(steps) for steps in runs] + return { + "first_token_ms": summarize([1000 * steps[0] for steps in runs]), + "decode_ms_per_step": summarize([1000 * t for steps in runs for t in steps[1:]]), + "total_ms": summarize([1000 * t for t in totals]), + "tokens_per_second": summarize([batch * new_tokens / t for t in totals]), + "median_step_ms_by_index": [1000 * sorted(steps[i] for steps in runs)[len(runs) // 2] for i in range(new_tokens)], + } + + +def _close(a, b) -> bool: + if isinstance(a, dict): + return a.keys() == b.keys() and all(_close(a[k], b[k]) for k in a) + if isinstance(a, list): + return len(a) == len(b) and all(_close(x, y) for x, y in zip(a, b)) + return math.isclose(a, b, rel_tol=1e-12, abs_tol=0.0) + + +def check_kv(key: str, record: dict, problems: list[str], require_raw: bool) -> None: + """KV-cache records: raw samples (current) or summaries (documented legacy rule), and derived ratios.""" + settings = record["settings"] + window = record["results"]["window"] + scenarios = [(row, row["batch"], settings["new_tokens"]) for row in record["results"]["grid"]] + scenarios.append((window, window["batch"], window["new_tokens"])) + for row, batch, new_tokens in scenarios: + where = f"{key} B={batch} prompt={row.get('prompt_len')}" + valid = True + for mode in settings["modes"]: + entry = row[mode] + if "raw" not in entry: + if require_raw: + problems.append(f"{key}: no raw per-repeat timing samples (historical records must be superseded)") + return + # Legacy rule: summary-only records are checked for internal consistency only. + values = [entry["total_ms"][stat] for stat in ("median", "p95", "min", "max", "mean")] + if not all(positive_finite(v) for v in values): + problems.append(f"{where}: {mode} summary contains non-positive or non-finite times") + valid = False + continue + runs = entry["raw"]["step_seconds"] + if len(runs) != settings["repeats"] or any(len(steps) != new_tokens for steps in runs): + problems.append(f"{where}: raw sample count does not match repeats x tokens ({mode})") + valid = False + continue + if not all(positive_finite(step) for steps in runs for step in steps): + problems.append(f"{where}: {mode} raw samples contain non-positive or non-finite times") + valid = False + continue + recomputed = kv_summary_from_raw(entry["raw"], batch, new_tokens) + if not all(_close(recomputed[name], entry[name]) for name in recomputed): + problems.append(f"{key}: {mode} B={batch} summaries differ from raw samples") + if not valid: + continue + derived = kv_speedups(row) + for name, value in derived.items(): + stored = row.get(name) + if stored is None: + continue + if not positive_finite(stored) or not math.isclose(stored, value, rel_tol=RATIO_TOLERANCE, abs_tol=0.0): + problems.append(f"{where}: stored {name} {stored!r} differs from derived {value:.6g}") + + +def check_legacy_kv(key: str, record: dict, problems: list[str]) -> None: + """Legacy single-prompt KV benchmark: medians, speedup and tokens/s derived from its samples.""" + import statistics + + results = record["results"] + medians = {} + for mode in ("uncached", "cached"): + samples = results[f"{mode}_seconds_samples"] + if not samples or not all(positive_finite(s) for s in samples): + problems.append(f"{key}: {mode} samples contain non-positive or non-finite times") + return + medians[mode] = statistics.median(samples) + if not math.isclose(results[f"{mode}_seconds"], medians[mode], rel_tol=RATIO_TOLERANCE): + problems.append(f"{key}: stored {mode}_seconds differs from the median of its samples") + if not math.isclose(results[f"{mode}_tokens_per_second"], results["generated_tokens"] / medians[mode], rel_tol=RATIO_TOLERANCE): + problems.append(f"{key}: stored {mode}_tokens_per_second differs from derived value") + derived = medians["uncached"] / medians["cached"] + if not positive_finite(results["speedup"]) or not math.isclose(results["speedup"], derived, rel_tol=RATIO_TOLERANCE): + problems.append(f"{key}: stored speedup {results['speedup']!r} differs from derived {derived:.6g}") + + +def check_benchmark(key: str, record: dict, problems: list[str], legacy: bool = False) -> None: + experiment, settings = record["experiment"], record["settings"] + for name in ("warmup", "repeats", "seed"): + if name not in settings: + problems.append(f"{key}: benchmark settings lack {name}") + if experiment == "kv_cache_scaling": + check_kv(key, record, problems, require_raw=not legacy) + elif experiment == "attention_backends": + for row in record["results"]: + runs = row["samples_seconds"] + if len(runs) != settings["repeats"] or not _close(summarize([1000 * r["forward"] for r in runs]), row["forward_ms"]): + problems.append(f"{key}: {row['backend']} B={row['batch']} L={row['seq_len']} forward summary differs from raw samples") + if "prefill_ms" in row: + decode = summarize([1000 * s for r in runs for s in r["decode_steps"]]) + if not (_close(summarize([1000 * r["prefill"] for r in runs]), row["prefill_ms"]) and _close(decode, row["decode_ms_per_step"])): + problems.append(f"{key}: {row['backend']} B={row['batch']} L={row['seq_len']} prefill/decode summary differs from raw samples") + elif experiment == "training_step": + for row in record["results"]: + samples = row["step_seconds_samples"] + tokens = row["batch"] * row["seq_len"] + if len(samples) != settings["repeats"] or not (_close(summarize([1000 * s for s in samples]), row["step_ms"]) + and _close(summarize([tokens / s for s in samples]), row["tokens_per_second"])): + problems.append(f"{key}: {row['backend']} L={row['seq_len']} summary differs from raw samples") + + +# ---------------------------------------------------------------- whole-store validation + + +def check_supersession(store: Store, problems: list[str]) -> None: + """Each entry: both records exist, same kind and experiment, no cycle, chain ends at a current record.""" + for key, entry in store.superseded.items(): + target = entry.get("superseded_by") + if key not in store.records or target not in store.records: + problems.append(f"SUPERSEDED.json: {key} -> {target}: both records must exist") + continue + if key == target: + problems.append(f"SUPERSEDED.json: {key} supersedes itself") + continue + old, new = store.records[key], store.records[target] + if key.split("/")[0] != target.split("/")[0] or old["experiment"] != new["experiment"]: + problems.append(f"SUPERSEDED.json: {key} and {target} are different kinds of record") + if old["experiment"] == "training": + if old["settings"].get("seed") != new["settings"].get("seed") or ( + canonical_backend(old["settings"].get("attention_backend")) != canonical_backend(new["settings"].get("attention_backend"))): + problems.append(f"SUPERSEDED.json: {target} is not the same seed and backend as {key}") + elif old["experiment"] in {"attention_backends", "kv_cache_scaling", "training_step"}: + if (old["environment"]["device"], old["settings"].get("dtype")) != (new["environment"]["device"], new["settings"].get("dtype")): + problems.append(f"SUPERSEDED.json: {target} is not the same device and dtype as {key}") + elif old["experiment"] == "evaluation" and NAME.match(Path(key).name)["name"] != NAME.match(Path(target).name)["name"]: + problems.append(f"SUPERSEDED.json: {target} is not the same evaluation as {key}") + seen, current = {key}, target + while current in store.superseded: + if current in seen: + problems.append(f"SUPERSEDED.json: supersession cycle through {key}") + break + seen.add(current) + current = store.superseded[current].get("superseded_by") + if current not in store.records: + problems.append(f"SUPERSEDED.json: chain from {key} ends at a missing record") + break + + +def identity_evidence(store: Store, problems: list[str]) -> dict[tuple[str, str], IdentityEvidence]: + """Resolve best and final checkpoint identities of every current training record once.""" + evidence = {} + for key, _ in store.current("training"): + for kind in ("best", "final"): + found, issues = resolve_identity(store, key, kind) + evidence[(key, kind)] = found + problems.extend(issue for issue in issues if issue not in problems) + return evidence + + +def validate_store(store: Store, strict: bool = False, require_artifacts: bool = False) -> list[str]: + """All problems in a record store. + + strict: every current training checkpoint must be artifact_verified or + trusted_baseline_verified. require_artifacts: every one must be artifact_verified. + """ + problems: list[str] = [] + check_supersession(store, problems) + for key, record in sorted(store.records.items()): + match = NAME.match(Path(key).name) + if not match: + problems.append(f"{key}: unexpected file name") + continue + for name in ("record_version", "created_utc", "experiment", "environment", "settings", "results"): + if name not in record: + problems.append(f"{key}: missing {name}") + environment = record.get("environment", {}) + missing = [name for name in ENVIRONMENT_FIELDS if name not in environment] + if missing: + problems.append(f"{key}: environment lacks {missing}") + continue + git = environment["git"] + if record["created_utc"] != match["stamp"] or not str(git["commit"]).startswith(match["commit"]): + problems.append(f"{key}: file name does not match created_utc/commit") + if not store.commit_exists(str(git["commit"])): + problems.append(f"{key}: commit {git['commit']} not in repository") + if git["dirty"] and key not in store.superseded: + problems.append(f"{key}: dirty working tree but not listed in SUPERSEDED.json") + if record["experiment"] in {"attention_backends", "kv_cache_scaling", "training_step"}: + if key not in store.superseded: + check_benchmark(key, record, problems) + elif record["experiment"] == "kv_cache_scaling": + check_benchmark(key, record, problems, legacy=True) + if record["experiment"] == "legacy_kv_cache_benchmark": + check_legacy_kv(key, record, problems) + select_training(store, problems) + evidence = identity_evidence(store, problems) + check_attestations(store, problems) + for (key, kind), found in sorted(evidence.items()): + if require_artifacts and found.status != ARTIFACT_VERIFIED: + problems.append(f"{key}: {kind} checkpoint identity is {found.status}; artifact verification required") + elif strict and found.status not in VERIFIED: + problems.append(f"{key}: {kind} checkpoint identity is {found.status}; strict verification needs the artifact or the pinned audited baseline") + index = training_by_checkpoint(store) + for key, record in store.current("evaluation"): + check_evaluation(store, key, record, problems, index) + for key, record in store.current("attention_quality"): + for position, row in enumerate(record["results"]): + resolve_row(store, f"{key} row {position}", row, None, problems, index) + for key, record in store.current("legacy"): + settings = record["settings"] + resolve_row(store, key, {"checkpoint_sha256": settings.get("checkpoint_sha256")}, None, problems, index) + if "validation_text" in settings: + path = relative_data_path(store, settings["validation_text"]) + if store.file_sha256(path) != settings["validation_sha256"]: + problems.append(f"{key}: validation text hash differs from {path}") + return problems + + +def protocol_rows(store: Store, protocol: str, evaluation_names: list[str], strict: bool = True) -> list[dict]: + """Training identities of a protocol joined to named evaluations by checkpoint identity. + + Fails closed: raises RecordError on any validation problem, on an evaluation + row that does not resolve, or on a training identity missing from an evaluation. + """ + problems = validate_store(store, strict=strict) + if problems: + raise RecordError("records failed validation:\n" + "\n".join(problems)) + selected = {identity: record for identity, record in select_training(store, []).items() if identity.protocol == protocol} + index = training_by_checkpoint(store) + evaluations = {} + for key, record in store.current("evaluation"): + name = NAME.match(Path(key).name)["name"] + if name in evaluation_names: + evaluations[name] = (key, record) # latest current record with this name + rows = {} + for identity, record in selected.items(): + summary = record["results"]["run_summary"] + rows[identity] = {"seed": identity.seed, "backend": identity.backend, "best_step": record["results"]["best_step"], + "tokens_per_second": summary["train_tokens_per_second"], "train_seconds": summary["train_seconds"], + "peak_rss_mib": summary["process_peak_rss_bytes"] / 2**20, "record": record["_path"], + "replicates": record["_replicates"]} + for name in evaluation_names: + if name not in evaluations: + continue + key, record = evaluations[name] + for row, training_key in check_evaluation(store, key, record, [], index): + identity = training_identity(store.records[training_key]) + if identity not in rows: + raise RecordError(f"{key}: evaluates {training_key}, which is not a selected {protocol} training identity") + rows[identity][name] = row["splits"] + missing = [identity for identity in rows if name not in rows[identity]] + if missing: + raise RecordError(f"{key}: no evaluation row for " + ", ".join(f"{i.backend}/seed {i.seed}" for i in missing)) + return sorted(rows.values(), key=lambda r: (r["seed"], r["backend"])) diff --git a/tests/conftest.py b/tests/conftest.py new file mode 100644 index 0000000..6f86523 --- /dev/null +++ b/tests/conftest.py @@ -0,0 +1,29 @@ +"""Shared test fixtures. + +MPS tests use the ``mps_device`` fixture. Without a usable MPS device they are +skipped with an explicit reason (see tests/hardware.py). Set MINILM_REQUIRE_MPS=1 +on Apple Silicon to turn such skips into failures, so the complete MPS suite must run: + + MINILM_REQUIRE_MPS=1 python -m pytest -q -rs +""" + +import os +import sys +from pathlib import Path + +import pytest + +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from hardware import MPS_REASON, MPS_USABLE + +REQUIRE_MPS = os.environ.get("MINILM_REQUIRE_MPS") == "1" + + +@pytest.fixture +def mps_device() -> str: + if not MPS_USABLE: + if REQUIRE_MPS: + pytest.fail(f"MINILM_REQUIRE_MPS=1 but {MPS_REASON}") + pytest.skip(f"hardware: {MPS_REASON}") + return "mps" diff --git a/tests/hardware.py b/tests/hardware.py new file mode 100644 index 0000000..3ece63a --- /dev/null +++ b/tests/hardware.py @@ -0,0 +1,42 @@ +"""Hardware capability checks for tests that need an Apple MPS device. + +torch.backends.mps.is_available() only reports that the backend exists. GitHub's +hosted macOS arm64 runners report MPS as available, yet the first allocation fails +("MPS backend out of memory ... MPS allocated: 0 bytes"). The probe below therefore +allocates, computes and synchronizes. Only a RuntimeError marks the device unusable; +a wrong result is not treated as unavailability, so genuine numerical defects still +fail the MPS tests instead of being skipped. + +Run as a script to print a hardware report (used by CI). +""" + +import platform + +import torch + +SMALL = 1024 # elements; far below any real allocation limit + + +def mps_probe() -> tuple[bool, str]: + """(usable, reason).""" + if not torch.backends.mps.is_available(): + return False, "MPS backend not available on this machine" + try: + x = torch.ones(SMALL, device="mps") + (x * 2).sum().item() + torch.mps.synchronize() + except RuntimeError as error: + first_line = str(error).splitlines()[0] + return False, f"MPS reported available but cannot allocate/compute: {first_line}" + return True, "MPS allocation and computation succeeded" + + +MPS_USABLE, MPS_REASON = mps_probe() + + +if __name__ == "__main__": + print(f"platform: {platform.platform()} ({platform.machine()})") + print(f"python: {platform.python_version()} torch: {torch.__version__}") + print(f"torch.backends.mps.is_built(): {torch.backends.mps.is_built()}") + print(f"torch.backends.mps.is_available(): {torch.backends.mps.is_available()}") + print(f"MPS probe: {'usable' if MPS_USABLE else 'unusable'} ({MPS_REASON})") diff --git a/tests/test_attention_backends.py b/tests/test_attention_backends.py new file mode 100644 index 0000000..82d8929 --- /dev/null +++ b/tests/test_attention_backends.py @@ -0,0 +1,188 @@ +import copy + +import pytest +import torch +import torch.nn.functional as F + +from minilm_lab import MiniLM, MiniLMConfig +from minilm_lab.model import ( + CausalSelfAttention, + causal_mask, + reference_causal_attention, +) + +BACKENDS = ("mha", "sdpa", "reference") + + +def small_model(positional_encoding: str = "learned", context_length: int = 32) -> MiniLM: + torch.manual_seed(11) + config = MiniLMConfig( + vocab_size=64, + context_length=context_length, + embedding_dim=48, + num_layers=2, + num_heads=4, + positional_encoding=positional_encoding, + ) + return MiniLM(config).eval() + + +def test_causal_mask_is_bottom_right_aligned() -> None: + assert torch.equal(causal_mask(3, 3, torch.device("cpu")), torch.ones(3, 3).tril().bool()) + # Two cached keys, two new queries: query 0 is absolute position 2. + expected = torch.tensor([[True, True, True, False], [True, True, True, True]]) + assert torch.equal(causal_mask(2, 4, torch.device("cpu")), expected) + + +def test_reference_attention_keeps_float64_precision() -> None: + torch.manual_seed(0) + q, k, v = (torch.randn(1, 2, 6, 8, dtype=torch.float64) for _ in range(3)) + output = reference_causal_attention(q, k, v) + assert output.dtype == torch.float64 + expected = F.scaled_dot_product_attention(q, k, v, is_causal=True) + assert (output - expected).abs().max() < 1e-12 + + +def test_reference_attention_matches_hand_computed_softmax() -> None: + torch.manual_seed(0) + q, k, v = (torch.randn(2, 3, 5, 8, dtype=torch.float64) for _ in range(3)) + output = reference_causal_attention(q, k, v) + for row in range(5): + weights = torch.softmax(q[:, :, row : row + 1] @ k[:, :, : row + 1].transpose(-2, -1) / 8**0.5, dim=-1) + assert torch.allclose(output[:, :, row : row + 1], weights @ v[:, :, : row + 1]) + + +@pytest.mark.parametrize("positional_encoding", ["learned", "rope"]) +def test_backends_match_float64_oracle(positional_encoding: str) -> None: + model = small_model(positional_encoding) + tokens = torch.randint(0, 64, (3, 32)) + oracle = model.double().set_attention_backend("reference")(tokens)[0] + model.float() + for backend in BACKENDS: + logits, _ = model.set_attention_backend(backend)(tokens) + assert logits.shape == (3, 32, 64) + assert (logits.double() - oracle).abs().max() < 1e-4, backend + + +@pytest.mark.parametrize("positional_encoding", ["learned", "rope"]) +@pytest.mark.parametrize("backend", BACKENDS) +def test_future_tokens_do_not_change_past_logits(positional_encoding: str, backend: str) -> None: + model = small_model(positional_encoding).set_attention_backend(backend) + tokens = torch.randint(0, 64, (2, 20)) + changed = tokens.clone() + changed[:, 12:] = torch.randint(0, 64, (2, 8)) + original, _ = model(tokens) + perturbed, _ = model(changed) + assert torch.allclose(original[:, :12], perturbed[:, :12], atol=1e-6) + assert not torch.allclose(original[:, 12:], perturbed[:, 12:]) + + +@pytest.mark.parametrize("backend", ["sdpa", "reference"]) +def test_cached_backends_match_full_forward_in_chunks(backend: str) -> None: + model = small_model("rope").set_attention_backend(backend) + tokens = torch.randint(0, 64, (3, 32)) + full, _ = model(tokens) + pieces, cache = [], None + for start, stop in ((0, 1), (1, 9), (9, 10), (10, 32)): + logits, cache = model.forward_cached(tokens[:, start:stop], cache) + pieces.append(logits) + assert torch.allclose(torch.cat(pieces, dim=1), full, atol=1e-5) + assert cache[0][0].shape == (3, 4, 32, 12) + + +def test_backend_selection_rejects_unknown_names() -> None: + with pytest.raises(ValueError): + small_model().set_attention_backend("flash") + + +def test_backend_does_not_change_weights() -> None: + model = small_model() + before = {name: tensor.clone() for name, tensor in model.state_dict().items()} + model.set_attention_backend("reference") + assert all(torch.equal(before[name], tensor) for name, tensor in model.state_dict().items()) + + +def rope_layer(width: int = 16) -> CausalSelfAttention: + return CausalSelfAttention(MiniLMConfig(context_length=64, embedding_dim=width * 2, num_heads=2)) + + +def test_rope_matches_complex_rotation() -> None: + torch.manual_seed(1) + x = torch.randn(1, 1, 10, 16, dtype=torch.float64) + rotated = rope_layer()._apply_rope(x, position_offset=3) + frequencies = 1.0 / (10000.0 ** (torch.arange(8, dtype=torch.float64) / 8)) + angles = torch.arange(3, 13, dtype=torch.float64)[:, None] * frequencies[None, :] + expected = torch.complex(x[..., :8], x[..., 8:]) * torch.polar(torch.ones_like(angles), angles) + assert torch.allclose(rotated, torch.cat((expected.real, expected.imag), dim=-1)) + + +def test_rope_scores_depend_only_on_relative_position() -> None: + torch.manual_seed(2) + layer = rope_layer() + q = torch.randn(1, 1, 1, 16, dtype=torch.float64).expand(1, 1, 40, 16) + k = torch.randn(1, 1, 1, 16, dtype=torch.float64).expand(1, 1, 40, 16) + scores = layer._apply_rope(q) @ layer._apply_rope(k).transpose(-2, -1) + # Same offset m - n gives the same score anywhere in the sequence. + assert torch.allclose(scores[0, 0, 7, 3], scores[0, 0, 30, 26]) + assert torch.allclose(layer._apply_rope(q).norm(dim=-1), q.norm(dim=-1)) + + +@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16]) +def test_rope_half_precision_error_is_at_rounding_level(dtype: torch.dtype) -> None: + torch.manual_seed(3) + layer = CausalSelfAttention(MiniLMConfig(context_length=2048, embedding_dim=192, num_heads=6)) + x = torch.randn(1, 6, 2048, 32) + exact = layer._apply_rope(x.double()) + error = (layer._apply_rope(x.to(dtype)).double() - exact).abs().max() + rounding = (exact.to(dtype).double() - exact).abs().max() + # Before computing angles in float32, bf16 error at 2048 positions was ~7.3. + assert error < 4 * rounding + + +@pytest.mark.parametrize("positional_encoding", ["learned", "rope"]) +def test_backend_gradients_match_float64_oracle(positional_encoding: str) -> None: + model = small_model(positional_encoding).train() + tokens = torch.randint(0, 64, (3, 32)) + targets = torch.randint(0, 64, (3, 32)) + + def gradients(backend: str, dtype: torch.dtype) -> dict[str, torch.Tensor]: + model.to(dtype).set_attention_backend(backend).zero_grad(set_to_none=True) + model(tokens, targets)[1].backward() + return {name: p.grad.double().clone() for name, p in model.named_parameters() if p.grad is not None} + + oracle = gradients("reference", torch.float64) + for backend in BACKENDS: + grads = gradients(backend, torch.float32) + assert grads.keys() == oracle.keys() + for name, grad in grads.items(): + scale = oracle[name].abs().max().clamp_min(1e-8) + assert (grad - oracle[name]).abs().max() / scale < 1e-4, (backend, name) + + +@pytest.mark.parametrize("backend", BACKENDS) +@pytest.mark.parametrize("use_rope", [False, True]) +def test_attention_layer_passes_gradcheck(backend: str, use_rope: bool) -> None: + torch.manual_seed(6) + layer = CausalSelfAttention(MiniLMConfig(context_length=8, embedding_dim=8, num_heads=2)).double() + layer.backend = backend + x = torch.randn(2, 5, 8, dtype=torch.float64, requires_grad=True) + assert torch.autograd.gradcheck(lambda value: layer(value, use_rope=use_rope), (x,)) + if backend == "mha" and use_rope: + # nn.MultiheadAttention cannot apply RoPE, so mha + RoPE runs the SDPA path. + sdpa = copy.deepcopy(layer) + sdpa.backend = "sdpa" + assert torch.equal(layer(x, use_rope=True), sdpa(x, use_rope=True)) + + +@pytest.mark.parametrize("backend", BACKENDS) +def test_attention_dropout_only_in_training(backend: str) -> None: + torch.manual_seed(8) + config = MiniLMConfig(vocab_size=64, context_length=16, embedding_dim=32, num_layers=1, num_heads=4, dropout=0.5) + model = MiniLM(config).set_attention_backend(backend) + tokens = torch.randint(0, 64, (2, 16)) + model.eval() + assert torch.equal(model(tokens)[0], model(tokens)[0]) + model.train() + attention = model.blocks[0].attention + x = torch.randn(2, 16, 32) + assert not torch.allclose(attention(x), attention(x)) diff --git a/tests/test_corpus.py b/tests/test_corpus.py new file mode 100644 index 0000000..a1db721 --- /dev/null +++ b/tests/test_corpus.py @@ -0,0 +1,81 @@ +import pytest + +from minilm_lab.corpus import ( + clean, + contamination, + gutenberg_body, + main_text, + paragraph_drop_reason, + paragraphs, + split_by_bytes, +) + +BOOK = """Header +*** START OF THE PROJECT GUTENBERG EBOOK DEMO *** +CONTENTS +CHAPTER I. 3 + +CHAPTER I. + +The first paragraph of prose, which we keep. + +[Illustration: Fig. 1.] + +Oxygen, . . . 20 +Nitrogen, . . 80 + +The value _t_ = 0 is kept. + +INDEX. + +Aberrant groups, 429. +*** END OF THE PROJECT GUTENBERG EBOOK DEMO *** +licence text +""" + + +def test_main_text_cuts_front_and_back_matter() -> None: + text = main_text(gutenberg_body(BOOK), "CHAPTER I.", 1, "INDEX.", 1) + assert text.startswith("CHAPTER I.\n") and "Aberrant" not in text and "CONTENTS" not in text + + +def test_marker_counts_are_enforced() -> None: + with pytest.raises(ValueError, match="expected 2"): + main_text(gutenberg_body(BOOK), "CHAPTER I.", 2, "INDEX.", 1) + + +def test_cleaning_rules_keep_prose_and_equations() -> None: + kept, dropped = clean(paragraphs(main_text(gutenberg_body(BOOK), "CHAPTER I.", 1, "INDEX.", 1))) + assert kept == ["The first paragraph of prose, which we keep.", "The value _t_ = 0 is kept."] + assert dropped == {"no_lowercase_heading": 1, "bracketed_editorial": 1, "numeric_list_or_table": 1} + + +@pytest.mark.parametrize( + ("paragraph", "reason"), + [ + ("CHAPTER XIV.\nRECAPITULATION.", "no_lowercase_heading"), + ("[The Lecturer then lit the candle.]", "bracketed_editorial"), + ("Agassiz, 139.\nAmblyopsis, 140.", "numeric_list_or_table"), + ("It was printed in December, 1916", None), + ("The flame [pointing to it] is bright.", None), + ], +) +def test_paragraph_rules(paragraph: str, reason: str | None) -> None: + assert paragraph_drop_reason(paragraph) == reason + + +def test_split_by_bytes_matches_ninety_percent_rule() -> None: + parts = ["x" * 10] * 10 + assert split_by_bytes(parts) == 9 + + +def test_contamination_detects_exact_and_near_duplicates() -> None: + train = ["The quick brown fox jumps over the lazy dog near the old river bank today."] + held_out = [ + "the QUICK brown fox, jumps over the lazy dog near the old river bank today", # exact after normalizing + "The quick brown fox jumps over the lazy dog near the old river bank tomorrow.", # near + "Completely different words about candles and flames burning in the lecture hall.", + ] + report = contamination(train, held_out) + assert report["exact_duplicates"] == [0] + assert report["near_duplicates"] == [1] diff --git a/tests/test_corpus_data.py b/tests/test_corpus_data.py new file mode 100644 index 0000000..57a1ec7 --- /dev/null +++ b/tests/test_corpus_data.py @@ -0,0 +1,69 @@ +"""Integrity and contamination checks on the committed corpora.""" + +import hashlib +import json +from pathlib import Path + +import pytest + +from minilm_lab.corpus import contamination, paragraphs + +ROOT = Path(__file__).resolve().parents[1] +V1 = ROOT / "data/science" +V2 = ROOT / "data/science-v2" + + +def sha256(path: Path) -> str: + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def test_legacy_v1_corpus_is_unchanged() -> None: + manifest = json.loads((V1 / "manifest.json").read_text()) + assert sha256(V1 / "train.txt") == manifest["train_sha256"] + assert sha256(V1 / "validation.txt") == manifest["validation_sha256"] + + +@pytest.fixture(scope="module") +def v2() -> dict: + return json.loads((V2 / "manifest.json").read_text()) + + +def test_v2_files_match_manifest(v2: dict) -> None: + for name, info in v2["files"].items(): + assert sha256(V2 / name) == info["sha256"], name + assert (V2 / name).stat().st_size == info["bytes"], name + + +def test_v2_test_book_is_a_separate_author(v2: dict) -> None: + roles = {s["role"]: s for s in v2["sources"] if s["role"] == "test"} + train_authors = {s["author"] for s in v2["sources"] if s["role"] == "train_validation"} + assert roles["test"]["author"] not in train_authors + + +def test_v2_held_out_text_has_no_exact_or_near_duplicates() -> None: + train = paragraphs((V2 / "train.txt").read_text()) + validation = paragraphs((V2 / "validation.txt").read_text()) + test = paragraphs((V2 / "test.txt").read_text()) + for reference, candidates in ((train, validation), (train, test), (validation, test)): + report = contamination(reference, candidates) + assert report["exact_duplicates"] == [] and report["near_duplicates"] == [] + + +def test_v2_splits_contain_no_index_or_backmatter() -> None: + for name in ("train.txt", "validation.txt", "test.txt"): + text = (V2 / name).read_text() + assert "\nINDEX.\n" not in text and "Aberrant groups, 429." not in text + assert "[Illustration" not in text and "_K_ = co-ordinate system" not in text + assert "PROJECT GUTENBERG" not in text.upper() + + +def test_v2_per_book_validation_files_concatenate_to_validation() -> None: + parts = [(V2 / f"{s['name']}-validation.txt").read_text() for s in json.loads((V2 / "manifest.json").read_text())["sources"] if s["role"] == "train_validation"] + # Same joining as v1 and prepare_science_corpus_v2.py. + assert "\n\n".join(parts).strip() + "\n" == (V2 / "validation.txt").read_text() + + +def test_legacy_v1_per_book_validation_files_concatenate_to_validation() -> None: + parts = [(V1 / f"{name}-validation.txt").read_text() for name in ("darwin-1859", "einstein-1924")] + # Same joining as scripts/prepare_science_corpus.py. + assert "\n\n".join(parts) == (V1 / "validation.txt").read_text() diff --git a/tests/test_decoding_and_training.py b/tests/test_decoding_and_training.py new file mode 100644 index 0000000..a386e27 --- /dev/null +++ b/tests/test_decoding_and_training.py @@ -0,0 +1,128 @@ +import json +import subprocess +import sys +from pathlib import Path + +import pytest +import torch + +from minilm_lab import MiniLM, MiniLMConfig +from minilm_lab.kv_cache import greedy_decode + +ROOT = Path(__file__).resolve().parents[1] + + +def model_for(positional_encoding: str, context_length: int = 16) -> MiniLM: + torch.manual_seed(5) + config = MiniLMConfig( + vocab_size=48, + context_length=context_length, + embedding_dim=32, + num_layers=2, + num_heads=4, + positional_encoding=positional_encoding, + ) + return MiniLM(config).eval() + + +@pytest.mark.parametrize("positional_encoding", ["learned", "rope"]) +@pytest.mark.parametrize( + ("batch", "prompt_length", "new_tokens"), + [ + (1, 1, 1), # single-token prompt, single generated token + (1, 1, 15), # fill the context exactly + (4, 5, 8), # batched decode + (2, 16, 6), # prompt already at the context limit + (3, 3, 40), # generation well past the window (sliding refresh) + ], +) +def test_cached_greedy_decode_matches_uncached(positional_encoding: str, batch: int, prompt_length: int, new_tokens: int) -> None: + model = model_for(positional_encoding) + prompt = torch.randint(0, 48, (batch, prompt_length)) + uncached = greedy_decode(model, prompt, new_tokens, use_cache=False) + cached = greedy_decode(model, prompt, new_tokens, use_cache=True) + assert cached.shape == (batch, prompt_length + new_tokens) + assert torch.equal(cached, uncached) + + +def test_cached_batch_rows_match_individual_rows() -> None: + model = model_for("rope") + prompt = torch.randint(0, 48, (3, 4)) + batched = greedy_decode(model, prompt, 10) + for row in range(3): + assert torch.equal(batched[row : row + 1], greedy_decode(model, prompt[row : row + 1], 10)) + + +def test_cache_grows_by_one_position_per_decoded_token() -> None: + model = model_for("learned") + logits, cache = model.forward_cached(torch.randint(0, 48, (2, 3))) + sizes = [cache[0][0].size(2)] + for _ in range(5): + logits, cache = model.forward_cached(logits[:, -1:].argmax(-1), cache) + sizes.append(cache[0][0].size(2)) + assert sizes == [3, 4, 5, 6, 7, 8] + # keys and values, every layer, (batch, heads, positions, head_width) + assert all(k.shape == v.shape == (2, 4, 8, 8) for k, v in cache) + + +def test_gradient_accumulation_matches_full_batch() -> None: + model = model_for("learned").train() + inputs = torch.randint(0, 48, (8, 16)) + targets = torch.randint(0, 48, (8, 16)) + _, loss = model(inputs, targets) + loss.backward() + full = {name: p.grad.clone() for name, p in model.named_parameters() if p.grad is not None} + model.zero_grad(set_to_none=True) + # Same arithmetic as scripts/train_tiny.py: each microbatch loss / steps. + for chunk in range(2): + _, loss = model(inputs[chunk * 4 : (chunk + 1) * 4], targets[chunk * 4 : (chunk + 1) * 4]) + (loss / 2).backward() + for name, parameter in model.named_parameters(): + if name in full: + assert torch.allclose(parameter.grad, full[name], atol=1e-6, rtol=1e-4), name + + +def run_training(text: Path, output: Path, steps: int, resume: bool = False) -> dict: + command = [ + sys.executable, str(ROOT / "scripts/train_tiny.py"), str(text), "--steps", str(steps), + "--context-length", "16", "--embedding-dim", "16", "--num-layers", "1", "--num-heads", "2", + "--batch-size", "2", "--eval-interval", "2", "--gradient-accumulation-steps", "2", + "--device", "cpu", "--seed", "3", "--output-dir", str(output), + ] + subprocess.run(command + (["--resume"] if resume else []), check=True, capture_output=True, cwd=ROOT) + return torch.load(output / "tiny_checkpoint.pt", weights_only=True) + + +def test_resume_reproduces_uninterrupted_training(tmp_path: Path) -> None: + text = tmp_path / "corpus.txt" + text.write_text("The quick brown fox jumps over the lazy dog. " * 40, encoding="utf-8") + straight = run_training(text, tmp_path / "straight", steps=6) + run_training(text, tmp_path / "resumed", steps=3) + resumed = run_training(text, tmp_path / "resumed", steps=3, resume=True) + assert resumed["step"] == straight["step"] == 6 + for name, tensor in straight["model"].items(): + assert torch.equal(tensor, resumed["model"][name]), name + # A segment always ends with an evaluation, so the resumed history has an + # extra step-3 record. Shared steps must agree exactly. + resumed_by_step = {m["step"]: m["validation_loss"] for m in resumed["metrics"]} + assert sorted(resumed_by_step) == [2, 3, 4, 6] + assert all(resumed_by_step[m["step"]] == m["validation_loss"] for m in straight["metrics"]) + + +def test_training_records_backend_and_rejects_backend_change_on_resume(tmp_path: Path) -> None: + text = tmp_path / "corpus.txt" + text.write_text("Natural selection acts slowly on small variations. " * 40, encoding="utf-8") + output = tmp_path / "reference" + command = [ + sys.executable, str(ROOT / "scripts/train_tiny.py"), str(text), "--steps", "2", "--context-length", "16", + "--embedding-dim", "16", "--num-layers", "1", "--num-heads", "2", "--batch-size", "2", "--eval-interval", "2", + "--device", "cpu", "--output-dir", str(output), + ] + subprocess.run(command + ["--attention-backend", "reference"], check=True, capture_output=True, cwd=ROOT) + checkpoint = torch.load(output / "tiny_checkpoint.pt", weights_only=True) + assert checkpoint["attention_backend"] == "reference" + summary = json.loads((output / "run_summary.json").read_text()) + assert summary["attention_backend"] == "reference" and summary["train_tokens"] == 2 * 2 * 16 + # Resuming without the flag selects the sdpa default, which must be refused. + resumed = subprocess.run(command + ["--resume"], check=False, capture_output=True, text=True, cwd=ROOT) + assert resumed.returncode != 0 and "attention backend" in resumed.stderr diff --git a/tests/test_memory_measurement.py b/tests/test_memory_measurement.py new file mode 100644 index 0000000..d2664ff --- /dev/null +++ b/tests/test_memory_measurement.py @@ -0,0 +1,36 @@ +"""Peak-memory methods are device-specific; host RSS is never reported for an accelerator.""" + +import pytest +import torch + +from minilm_lab import provenance + + +def test_cpu_uses_host_peak_rss() -> None: + extra, method = provenance.measure_peak_memory("cpu", lambda: torch.ones(4 * 2**20).sum()) + assert method == "process_peak_rss_delta" and extra >= 0 + + +def test_cuda_uses_device_allocator_peak_not_host_rss(monkeypatch: pytest.MonkeyPatch) -> None: + counters = {"allocated": 1000, "peak": 1000} + + def allocate() -> None: + counters["peak"] = 1000 + 5_000_000 + + monkeypatch.setattr(torch.cuda, "synchronize", lambda: None) + monkeypatch.setattr(torch.cuda, "reset_peak_memory_stats", lambda: counters.update(peak=counters["allocated"])) + monkeypatch.setattr(torch.cuda, "memory_allocated", lambda: counters["allocated"]) + monkeypatch.setattr(torch.cuda, "max_memory_allocated", lambda: counters["peak"]) + monkeypatch.setattr(provenance, "peak_rss_bytes", lambda: pytest.fail("host RSS used for CUDA")) + assert provenance.measure_peak_memory("cuda", allocate) == (5_000_000, "cuda_max_memory_allocated_delta") + + +def test_mps_uses_live_allocator_peak(monkeypatch: pytest.MonkeyPatch, mps_device: str) -> None: + monkeypatch.setattr(provenance, "peak_rss_bytes", lambda: pytest.fail("host RSS used for MPS")) + extra, method = provenance.measure_peak_memory("mps", lambda: torch.ones(2**20, device="mps") * 2) + assert method == "mps_live_allocated_peak_per_op" and extra >= 2 * 4 * 2**20 + + +def test_unknown_device_is_rejected() -> None: + with pytest.raises(ValueError): + provenance.measure_peak_memory("xpu", lambda: None) diff --git a/tests/test_model_identity.py b/tests/test_model_identity.py new file mode 100644 index 0000000..861d451 --- /dev/null +++ b/tests/test_model_identity.py @@ -0,0 +1,133 @@ +"""Finding 1: deterministic model identity is separate from checkpoint file bytes.""" + +import copy +import hashlib +import json +import struct +import subprocess +import sys +from pathlib import Path + +import pytest +import torch + +from minilm_lab import MiniLM, MiniLMConfig +from minilm_lab.identity import ( + SCHEME, + checkpoint_identity, + identity_config, + model_identity, +) + +ROOT = Path(__file__).resolve().parents[1] +CONFIG = {"vocab_size": 32, "context_length": 8, "embedding_dim": 16, "num_layers": 1, "num_heads": 2} + + +def state(seed: int = 0, **overrides) -> tuple[dict, dict]: + torch.manual_seed(seed) + config = MiniLMConfig(**{**CONFIG, **overrides}) + return MiniLM(config).state_dict(), config.__dict__ + + +def test_known_answer_for_the_canonical_byte_stream() -> None: + tensors = {"b": torch.tensor([1.5, -2.0], dtype=torch.float32), "a": torch.tensor([[1, 2]], dtype=torch.int16)} + config = identity_config(CONFIG) + expected = hashlib.sha256() + expected.update(b"minilm-model-identity/v1\n") + expected.update(json.dumps({"config": config, "tokenizer": "byte"}, sort_keys=True, separators=(",", ":")).encode() + b"\n") + expected.update(b'{"dtype":"torch.int16","name":"a","nbytes":4,"shape":[1,2]}\n' + struct.pack("<2h", 1, 2)) + expected.update(b'{"dtype":"torch.float32","name":"b","nbytes":8,"shape":[2]}\n' + struct.pack("<2f", 1.5, -2.0)) + assert model_identity(tensors, CONFIG) == expected.hexdigest() + + +def test_identical_model_and_config_give_identical_digest() -> None: + first, config = state(0) + second, _ = state(0) + assert model_identity(first, config) == model_identity(second, config) + assert model_identity(first, config) == model_identity(dict(reversed(list(first.items()))), config) + + +def test_weight_change_changes_digest() -> None: + weights, config = state(0) + changed = {name: tensor.clone() for name, tensor in weights.items()} + changed["blocks.0.up.weight"][0, 0] += 1e-6 + assert model_identity(weights, config) != model_identity(changed, config) + assert model_identity(weights, config) != model_identity(state(1)[0], config) + + +@pytest.mark.parametrize("override", [{"dropout": 0.1}, {"rope_theta": 500.0}, {"positional_encoding": "rope"}, {"norm": "layernorm"}]) +def test_identity_relevant_config_changes_digest(override: dict) -> None: + weights, config = state(0) + assert model_identity(weights, config) != model_identity(weights, {**config, **override}) + + +def test_tokenizer_is_part_of_identity() -> None: + weights, config = state(0) + assert model_identity(weights, config, "byte") != model_identity(weights, config, "bpe-v1") + + +def test_runtime_metadata_does_not_change_digest() -> None: + weights, config = state(0) + base = {"model": weights, "config": config, "tokenizer": "byte", "seed": 1, + "metrics": [{"step": 1, "validation_loss": 2.0, "elapsed_seconds": 1.0, "train_seconds": 0.5, "eval_seconds": 0.1}]} + later = copy.deepcopy(base) + later["metrics"][0].update(elapsed_seconds=99.0, train_seconds=42.0, eval_seconds=7.0) + later.update(seed=2, attention_backend="sdpa", output_dir="/elsewhere", created="2026-10-22T05:00:00Z") + assert checkpoint_identity(base) == checkpoint_identity(later) + assert checkpoint_identity(base)["scheme"] == SCHEME + + +def test_precision_is_preserved_not_cast() -> None: + weights, config = state(0) + half = {name: tensor.to(torch.float16) if tensor.is_floating_point() else tensor for name, tensor in weights.items()} + bf16 = {name: tensor.to(torch.bfloat16) if tensor.is_floating_point() else tensor for name, tensor in weights.items()} + digests = {model_identity(weights, config), model_identity(half, config), model_identity(bf16, config)} + assert len(digests) == 3 + + +def test_hashing_does_not_mutate_model_state() -> None: + torch.manual_seed(0) + model = MiniLM(MiniLMConfig(**CONFIG)).train() + before = {name: (tensor.clone(), tensor._version, tensor.requires_grad, tensor.dtype, tensor.device) for name, tensor in model.state_dict(keep_vars=True).items()} + model_identity(model.state_dict(keep_vars=True), model.config) + after = model.state_dict(keep_vars=True) + for name, (value, version, requires_grad, dtype, device) in before.items(): + assert torch.equal(after[name], value) and after[name]._version == version + assert (after[name].requires_grad, after[name].dtype, after[name].device) == (requires_grad, dtype, device) + assert model.training + + +def test_device_does_not_change_digest_or_move_tensors(mps_device: str) -> None: + weights, config = state(0) + on_mps = {name: tensor.to("mps") for name, tensor in weights.items()} + assert model_identity(on_mps, config) == model_identity(weights, config) + assert all(tensor.device.type == "mps" for tensor in on_mps.values()) + + +def test_legacy_checkpoint_without_identity_or_new_config_fields_loads_and_hashes(tmp_path: Path) -> None: + weights, config = state(0) + legacy_config = {key: value for key, value in config.items() if key not in ("rope_theta", "norm", "activation")} + path = tmp_path / "legacy.pt" + torch.save({"model": weights, "config": legacy_config, "metrics": [], "seed": 42}, path) + loaded = torch.load(path, weights_only=True) + model = MiniLM(MiniLMConfig(**loaded["config"])) + model.load_state_dict(loaded["model"]) + assert checkpoint_identity(loaded)["sha256"] == model_identity(weights, config) + + +def test_training_stores_identity_that_matches_tensors_across_reruns(tmp_path: Path) -> None: + text = tmp_path / "corpus.txt" + text.write_text("Every candle burns with a flame of its own shape. " * 30, encoding="utf-8") + identities = [] + for run in ("a", "b"): + command = [sys.executable, str(ROOT / "scripts/train_tiny.py"), str(text), "--steps", "3", "--context-length", "16", + "--embedding-dim", "16", "--num-layers", "1", "--num-heads", "2", "--batch-size", "2", "--eval-interval", "1", + "--device", "cpu", "--seed", "5", "--output-dir", str(tmp_path / run)] + subprocess.run(command, check=True, capture_output=True, cwd=ROOT) + for name in ("best_checkpoint.pt", "tiny_checkpoint.pt"): + checkpoint = torch.load(tmp_path / run / name, weights_only=True) + assert checkpoint["model_identity"] == checkpoint_identity(checkpoint) + identities.append(torch.load(tmp_path / run / "tiny_checkpoint.pt", weights_only=True)["model_identity"]) + # Same seed and data: the model identity is reproduced. File bytes are not compared: + # they include elapsed seconds and are not expected to match. + assert identities[0] == identities[1] diff --git a/tests/test_provenance.py b/tests/test_provenance.py new file mode 100644 index 0000000..0f4292a --- /dev/null +++ b/tests/test_provenance.py @@ -0,0 +1,38 @@ +import json +from pathlib import Path + +import pytest + +from minilm_lab.provenance import environment, summarize, write_record + + +def test_summary_statistics() -> None: + stats = summarize([5.0, 1.0, 3.0, 2.0, 4.0]) + assert stats["median"] == 3.0 + assert stats["min"] == 1.0 and stats["max"] == 5.0 + assert stats["iqr"] == 2.0 + assert stats["p95"] == pytest.approx(4.8) + + +def test_records_are_never_overwritten(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + payload = {"environment": {"git": {"commit": "abcdef0123"}}, "results": [1]} + first = write_record(tmp_path, "demo", payload) + assert first.name.endswith("_demo_abcdef0.json") + assert json.loads(first.read_text())["results"] == [1] + # Same second, same name: the second write must fail rather than replace. + monkeypatch.setattr("minilm_lab.provenance.datetime", type("Frozen", (), {"now": staticmethod(lambda tz: _stamp(first))})) + with pytest.raises(FileExistsError): + write_record(tmp_path, "demo", {**payload, "results": [2]}) + assert json.loads(first.read_text())["results"] == [1] + + +def _stamp(path: Path): + from datetime import datetime, timezone + + return datetime.strptime(path.name.split("_")[0], "%Y%m%dT%H%M%SZ").replace(tzinfo=timezone.utc) + + +def test_environment_records_versions_and_git() -> None: + info = environment("cpu") + assert info["torch"] and info["python"] and info["device"] == "cpu" + assert set(info["git"]) == {"commit", "branch", "dirty"} diff --git a/tests/test_published_claims.py b/tests/test_published_claims.py new file mode 100644 index 0000000..387d893 --- /dev/null +++ b/tests/test_published_claims.py @@ -0,0 +1,101 @@ +"""Headline numbers in the README must be recomputable from experiment records.""" + +import importlib.util +import json +import statistics +from pathlib import Path + +import pytest + +from minilm_lab.records import kv_speedups + +ROOT = Path(__file__).resolve().parents[1] +README = (ROOT / "README.md").read_text() +spec = importlib.util.spec_from_file_location("report", ROOT / "scripts/build_experiment_report.py") +report = importlib.util.module_from_spec(spec) +spec.loader.exec_module(report) +RECORDS = report.RECORDS + + +def attention_ratio_range(keys: list[str]) -> tuple[float, float]: + records = report.latest(RECORDS / "attention") + ratios = [] + for key in keys: + rows = report.rows_by(records[key]) + ratios += [rows[("reference", b, 2048)]["forward_ms"]["median"] / rows[("sdpa", b, 2048)]["forward_ms"]["median"] for b in (1, 4, 8)] + return min(ratios), max(ratios) + + +def evaluation(name: str, backend: str, split: str) -> list[float]: + rows = [r for r in report.latest_by_name(RECORDS / "evaluation")[name]["results"] if report.canon(r["attention_backend"]) == backend] + return [r["splits"][split]["perplexity"] for r in sorted(rows, key=lambda r: r["seed"])] + + +def mean_sd(values: list[float]) -> str: + return f"{statistics.mean(values):.2f} ± {statistics.stdev(values):.2f}" + + +def test_attention_speedups_and_memory() -> None: + low, high = attention_ratio_range(["cpu/float32"]) + assert f"{low:.1f}–{high:.1f}× (CPU)" in README + low, high = attention_ratio_range(["mps/float32", "mps/float16"]) + assert f"{low:.1f}–{high:.1f}× (MPS)" in README + memory = [] + for record in report.latest(RECORDS / "attention").values(): + rows = report.rows_by(record) + memory.append(rows[("reference", 8, 2048)]["memory"]["peak_extra_bytes"] / rows[("sdpa", 8, 2048)]["memory"]["peak_extra_bytes"]) + assert f"{round(min(memory))}–{round(max(memory))}× less extra memory" in README + + +def test_training_quality_and_step_speed() -> None: + assert f"{mean_sd(evaluation('sdpa_v1_eval_cpu', 'mha', 'validation'))} (MHA) vs {mean_sd(evaluation('sdpa_v1_eval_cpu', 'sdpa', 'validation'))} (SDPA)" in README + steps = report.latest(RECORDS / "training_step") + + def gains(key: str) -> list[float]: + rows = {(report.canon(r["backend"]), r["seq_len"]): r["step_ms"]["median"] for r in steps[key]["results"]} + return [100 * (rows[("mha", length)] / rows[("sdpa", length)] - 1) for length in (128, 256, 512, 1024)] + + cpu = gains("cpu/float32") + assert f"{round(min(cpu))}–{round(max(cpu))}% faster with SDPA on CPU" in README + assert max(abs(g) for g in gains("mps/float32")) < 5 and "within 5% on MPS" in README + + +def test_kv_cache_claims() -> None: + records = report.latest(RECORDS / "kv_cache") + # Ratios are derived from the timings, never read from stored fields. + cpu = [kv_speedups(row)["speedup_cached_vs_uncached_sdpa"] for row in records["cpu/float32"]["results"]["grid"]] + assert f"{min(cpu):.1f}–{round(max(cpu))}× faster than recomputation on CPU" in README + mps = [kv_speedups(row)["speedup_cached_vs_uncached_sdpa"] for row in records["mps/float32"]["results"]["grid"]] + assert f"({min(mps):.2f}×)" in README + windows = [r["results"]["window"] for r in records.values()] + assert all(sum(w["cached"]["refreshed_steps"]) == len(w["cached"]["refreshed_steps"]) - w["context_length"] + w["prompt_len"] for w in windows) + + +def test_seed_and_v2_numbers() -> None: + seeds = " / ".join(f"{p:.2f}" for p in evaluation("sdpa_v1_eval_cpu", "mha", "validation")) + assert f"seeds 42/43/44 give held-out perplexity {seeds}" in README + assert f"test perplexity is {mean_sd(evaluation('v2_final_cpu', 'mha', 'test'))} (MHA) and {mean_sd(evaluation('v2_final_cpu', 'sdpa', 'test'))} (SDPA)" in README + + +def test_legacy_v1_values() -> None: + published = json.loads((ROOT / "docs/data/science_run.json").read_text()) + records = {p.name: json.loads(p.read_text()) for p in sorted((RECORDS / "legacy").glob("*.json"))} + current = [r for name, r in records.items() if f"legacy/{name}" not in json.loads((RECORDS / "SUPERSEDED.json").read_text())["records"]] + int8 = next(r["results"] for r in current if r["experiment"] == "legacy_int8_comparison") + for key in ("original_validation_loss", "int8_validation_loss", "original_stored_bytes", "int8_stored_bytes"): + assert int8[key] == published["quantization"][key] # deterministic values reproduce exactly + kv = next(r["results"] for r in current if r["experiment"] == "legacy_kv_cache_benchmark") + assert f"{published['benchmark']['speedup']:.2f}×) re-measures at {kv['speedup']:.2f}×" in README + # The September published speedup is consistent with its own stored samples. + bench = published["benchmark"] + medians = {mode: statistics.median(bench[f"{mode}_seconds_samples"]) for mode in ("uncached", "cached")} + assert medians["uncached"] == bench["uncached_seconds"] and medians["cached"] == bench["cached_seconds"] + assert bench["speedup"] == medians["uncached"] / medians["cached"] + assert f"{published['evaluation']['perplexity']:.2f}" in README + + +@pytest.mark.parametrize("path", ["docs/EXPERIMENT_RESULTS.md", "docs/research_report.md"]) +def test_detailed_documents_carry_the_same_headline_values(path: str) -> None: + text = (ROOT / path).read_text() + assert mean_sd(evaluation("v2_final_cpu", "mha", "test")) in text + assert "5–28%" in text diff --git a/tests/test_record_provenance.py b/tests/test_record_provenance.py new file mode 100644 index 0000000..f06e81a --- /dev/null +++ b/tests/test_record_provenance.py @@ -0,0 +1,271 @@ +"""Findings 2, 3 and 4: experiment identity, evaluation provenance and raw benchmark samples. + +Every test works on a copy of experiments/records so the committed records stay untouched. +""" + +import importlib.util +import json +import shutil +import subprocess +import sys +from pathlib import Path + +import pytest + +from minilm_lab.records import ( + LEGACY_TRAIN_DEFAULTS, + RecordError, + load_store, + protocol_rows, + validate_store, +) + +ROOT = Path(__file__).resolve().parents[1] +RECORDS = ROOT / "experiments/records" +V1_MHA_42 = "training/20261022T041654Z_training_science-v1_auto_seed42_48e95f8.json" +V2_FINAL = "evaluation/20261022T043652Z_v2_final_cpu_897db37.json" +V1_EVAL = "evaluation/20261022T041615Z_sdpa_v1_eval_cpu_48e95f8.json" + + +@pytest.fixture +def records(tmp_path: Path) -> Path: + copy = tmp_path / "records" + shutil.copytree(RECORDS, copy) + return copy + + +def read(root: Path, key: str) -> dict: + return json.loads((root / key).read_text()) + + +def write(root: Path, key: str, record: dict) -> None: + (root / key).write_text(json.dumps(record, indent=2)) + + +def problems(root: Path) -> list[str]: + return validate_store(load_store(ROOT, root)) + + +def add_mha_duplicate_of_auto_seed42(root: Path, stamp: str = "20261022T060000Z", **metric_change) -> str: + record = read(root, V1_MHA_42) + record["settings"]["attention_backend"] = "mha" # canonical name for the legacy "auto" + record["created_utc"] = stamp + if metric_change: + record["results"]["metrics"][-1].update(metric_change) + key = f"training/{stamp}_training_science-v1_mha_seed42_48e95f8.json" + write(root, key, record) + return key + + +def test_committed_records_validate() -> None: + assert problems(RECORDS) == [] + + +# ---------------------------------------------------------------- finding 2 + + +def test_legacy_auto_and_canonical_mha_seed42_count_once(records: Path) -> None: + duplicate = add_mha_duplicate_of_auto_seed42(records) + assert problems(records) == [] + rows = protocol_rows(load_store(ROOT, records), "science-v1", ["sdpa_v1_eval_cpu"]) + mha = [row for row in rows if row["backend"] == "mha"] + assert sorted(row["seed"] for row in mha) == [42, 43, 44] # not inflated to four + assert next(row for row in mha if row["seed"] == 42)["replicates"] == [duplicate] + assert sorted(row["seed"] for row in rows if row["backend"] == "sdpa") == [42, 43, 44] # all valid seeds kept + + +def test_conflicting_duplicate_identity_fails_validation_and_reporting(records: Path) -> None: + add_mha_duplicate_of_auto_seed42(records, validation_loss=2.5) + assert any("conflicting training records" in p and "seed 42" in p for p in problems(records)) + with pytest.raises(RecordError, match="conflicting training records"): + protocol_rows(load_store(ROOT, records), "science-v1", ["sdpa_v1_eval_cpu"]) + + +def test_explicit_supersession_resolves_a_conflict_and_keeps_the_audit_trail(records: Path) -> None: + duplicate = add_mha_duplicate_of_auto_seed42(records, validation_loss=2.5) + index = json.loads((records / "SUPERSEDED.json").read_text()) + index["records"][duplicate] = {"reason": "test: conflicting re-record", "superseded_by": V1_MHA_42} + (records / "SUPERSEDED.json").write_text(json.dumps(index)) + assert problems(records) == [] + assert duplicate in load_store(ROOT, records).records # kept on disk, not discarded + + +def test_historical_report_rows_are_reproduced() -> None: + store = load_store(ROOT, RECORDS) + v1 = protocol_rows(store, "science-v1", ["sdpa_v1_eval_cpu"]) + perplexity = {(r["backend"], r["seed"]): round(r["sdpa_v1_eval_cpu"]["validation"]["perplexity"], 3) for r in v1} + assert perplexity == {("mha", 42): 17.583, ("mha", 43): 18.989, ("mha", 44): 21.068, + ("sdpa", 42): 17.683, ("sdpa", 43): 19.009, ("sdpa", 44): 20.416} + v2 = protocol_rows(store, "science-v2", ["v2_final_cpu"]) + assert len(v2) == 6 and all("test" in row["v2_final_cpu"] for row in v2) + + +# ---------------------------------------------------------------- finding 3 + + +def mutate_evaluation(root: Path, key: str, change) -> None: + record = read(root, key) + change(record) + write(root, key, record) + + +def test_zeroed_checkpoint_hash_is_rejected(records: Path) -> None: + mutate_evaluation(records, V2_FINAL, lambda r: r["results"][0].update(checkpoint_sha256="0" * 64)) + assert any("matches 0 current training records" in p for p in problems(records)) + with pytest.raises(RecordError): + protocol_rows(load_store(ROOT, records), "science-v2", ["v2_final_cpu"]) + + +def test_evaluation_associated_with_wrong_training_run_is_rejected(records: Path) -> None: + def swap(record: dict) -> None: + a, b = record["results"][0], record["results"][2] # seeds 42 and 43 of the same backend + a["checkpoint_sha256"], b["checkpoint_sha256"] = b["checkpoint_sha256"], a["checkpoint_sha256"] + mutate_evaluation(records, V2_FINAL, swap) + found = problems(records) + assert any("seed 42 but training record" in p for p in found) and any("seed 43 but training record" in p for p in found) + + +def test_changed_evaluation_corpus_identity_is_rejected(records: Path) -> None: + mutate_evaluation(records, V2_FINAL, lambda r: r["settings"]["splits"]["test"].update(sha256="f" * 64)) + assert any("split test hash" in p for p in problems(records)) + mutate_evaluation(records, V2_FINAL, lambda r: r["settings"]["splits"]["test"].update(sha256=read(RECORDS, V2_FINAL)["settings"]["splits"]["test"]["sha256"])) + mutate_evaluation(records, V2_FINAL, lambda r: r["results"][1].update(corpus_sha256="e" * 64)) + assert any("corpus identity differs" in p for p in problems(records)) + + +def test_evaluation_split_from_another_protocol_is_rejected(records: Path) -> None: + v1_validation = read(RECORDS, V1_EVAL)["settings"]["splits"]["validation"] + mutate_evaluation(records, V2_FINAL, lambda r: r["settings"]["splits"].update(validation=v1_validation)) + found = problems(records) + assert any("outside the science-v2 corpus" in p for p in found) + + +def test_changed_protocol_is_rejected(records: Path) -> None: + mutate_evaluation(records, V2_FINAL, lambda r: r["settings"].update(protocol="science-v1")) + found = problems(records) + assert any("protocol science-v1 but training record" in p for p in found) + mutate_evaluation(records, V2_FINAL, lambda r: r["settings"].update(protocol="science-v9")) + assert any("unknown or missing protocol 'science-v9'" in p for p in problems(records)) + + +def test_contradictory_duplicate_training_records_are_rejected(records: Path) -> None: + add_mha_duplicate_of_auto_seed42(records, stamp="20261022T070000Z", validation_loss=3.0) + assert any("conflicting training records" in p for p in problems(records)) + + +def test_invalid_commit_and_manifest_references_are_rejected(records: Path) -> None: + record = read(records, V1_MHA_42) + record["settings"]["training_code_commit"] = "deadbee" + write(records, V1_MHA_42, record) + assert any("training_code_commit 'deadbee' is not in the repository" in p for p in problems(records)) + record["settings"]["training_code_commit"] = "48e95f8" + record["settings"]["data"]["manifest_sha256"] = "0" * 64 + write(records, V1_MHA_42, record) + assert any("manifest hash does not match" in p for p in problems(records)) + record["settings"]["data"]["manifest_path"] = "data/science-v2/manifest.json" + write(records, V1_MHA_42, record) + assert any("does not belong to science-v1" in p for p in problems(records)) + + +def test_invalid_environment_commit_is_rejected(records: Path) -> None: + record = read(records, V2_FINAL) + record["environment"]["git"]["commit"] = "897db37" + "0" * 33 + write(records, V2_FINAL, record) + assert any("not in repository" in p for p in problems(records)) + + +def test_model_identity_mismatch_is_rejected(records: Path) -> None: + training = read(records, V1_MHA_42) + training["results"]["best_model_identity_sha256"] = "a" * 64 + write(records, V1_MHA_42, training) + mutate_evaluation(records, V1_EVAL, lambda r: [row.update(model_identity_sha256="b" * 64) for row in r["results"] if row["seed"] == 42 and row["attention_backend"] == "auto"]) + # The forged declared value already contradicts the artifact/baseline identity. + found = problems(records) + assert any("model identity sources disagree" in p for p in found) + + +def test_training_data_outside_protocol_is_rejected(records: Path) -> None: + record = read(records, V1_MHA_42) + record["settings"]["data"]["train"] = {"path": "data/science-v2/train.txt", "sha256": read(RECORDS, V1_MHA_42)["settings"]["data"]["train"]["sha256"]} + write(records, V1_MHA_42, record) + assert any("outside the science-v1 corpus" in p for p in problems(records)) + + +def test_legacy_hyperparameter_defaults_match_train_tiny() -> None: + spec = importlib.util.spec_from_file_location("train_tiny", ROOT / "scripts/train_tiny.py") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + defaults = vars(module.build_parser().parse_args(["corpus.txt"])) + assert {name: defaults[name] for name in LEGACY_TRAIN_DEFAULTS} == LEGACY_TRAIN_DEFAULTS + + +# ---------------------------------------------------------------- finding 4 + + +def current_kv(root: Path) -> list[str]: + store = load_store(ROOT, root) + return [key for key, _ in store.current("kv_cache")] + + +def test_current_kv_records_have_raw_samples_and_historical_ones_are_marked() -> None: + store = load_store(ROOT, RECORDS) + for key, record in store.current("kv_cache"): + assert "raw" in record["results"]["grid"][0]["cached"], key + for key, record in store.records.items(): + if key.startswith("kv_cache/") and "raw" not in record["results"]["grid"][0]["cached"]: + assert "raw" in store.superseded[key]["reason"] + + +def test_kv_summary_that_disagrees_with_raw_samples_is_rejected(records: Path) -> None: + key = current_kv(records)[0] + record = read(records, key) + record["results"]["grid"][2]["uncached_sdpa"]["total_ms"]["p95"] *= 1.001 + write(records, key, record) + assert any("summaries differ from raw samples" in p for p in problems(records)) + + +def test_kv_record_without_raw_samples_is_rejected(records: Path) -> None: + key = current_kv(records)[0] + record = read(records, key) + for row in record["results"]["grid"]: + for mode in record["settings"]["modes"]: + row[mode].pop("raw") + write(records, key, record) + assert any("no raw per-repeat timing samples" in p for p in problems(records)) + + +# ---------------------------------------------------------------- exit codes + + +def test_validator_failure_propagates_through_exit_codes_and_pipelines(records: Path) -> None: + mutate_evaluation(records, V2_FINAL, lambda r: r["results"][0].update(checkpoint_sha256="0" * 64)) + command = [sys.executable, str(ROOT / "scripts/validate_records.py"), "--records-dir", str(records)] + assert subprocess.run(command, check=False, capture_output=True, cwd=ROOT).returncode == 1 + piped = subprocess.run(["bash", "-o", "pipefail", "-c", " ".join(command) + " | tail -n 1"], check=False, capture_output=True, cwd=ROOT) + assert piped.returncode == 1 + clean = [sys.executable, str(ROOT / "scripts/validate_records.py")] + assert subprocess.run(clean, check=False, capture_output=True, cwd=ROOT).returncode == 0 + + +# ---------------------------------------------------------------- attested identities (finding 1 for historical runs) + +ATTESTATION = "identity/20261022T053914Z_model_identity_attestation_935a229.json" + + +def test_attestation_with_wrong_checkpoint_hash_is_rejected(records: Path) -> None: + record = read(records, ATTESTATION) + record["results"]["attested"][0]["checkpoint_sha256"] = "0" * 64 + write(records, ATTESTATION, record) + assert any("checkpoint hash differs" in p for p in problems(records)) + + +def test_inconsistent_regeneration_check_is_rejected(records: Path) -> None: + record = read(records, ATTESTATION) + record["results"]["regeneration_checks"][0]["model_identity_sha256"] = "1" * 64 + write(records, ATTESTATION, record) + assert any("regeneration check" in p for p in problems(records)) + + +def test_attested_identity_is_enforced_for_historical_evaluations(records: Path) -> None: + mutate_evaluation(records, V1_EVAL, lambda r: r["results"][0].update(model_identity_sha256="2" * 64)) + assert any("model identity differs" in p for p in problems(records)) diff --git a/tests/test_records.py b/tests/test_records.py new file mode 100644 index 0000000..9347b29 --- /dev/null +++ b/tests/test_records.py @@ -0,0 +1,25 @@ +import importlib.util +import subprocess +import sys +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] + + +def load(name: str): + spec = importlib.util.spec_from_file_location(name, ROOT / f"scripts/{name}.py") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def test_all_experiment_records_are_valid() -> None: + errors, count = load("validate_records").validate() + assert count > 0 and errors == [] + + +def test_generated_report_matches_records() -> None: + # Rebuilding from records must not change the committed document or figures. + subprocess.run([sys.executable, str(ROOT / "scripts/build_experiment_report.py")], check=True, capture_output=True, cwd=ROOT) + diff = subprocess.run(["git", "diff", "--stat", "--", "docs/EXPERIMENT_RESULTS.md", "docs/figures"], check=True, capture_output=True, text=True, cwd=ROOT) + assert diff.stdout == "" diff --git a/tests/test_release_backends.py b/tests/test_release_backends.py new file mode 100644 index 0000000..9756969 --- /dev/null +++ b/tests/test_release_backends.py @@ -0,0 +1,192 @@ +"""Release checks for attention backend defaults, compatibility and tolerances.""" + +import json +import subprocess +import sys +from pathlib import Path + +import pytest +import torch +import torch.nn.functional as F + +from minilm_lab import MiniLM, MiniLMConfig +from minilm_lab.metrics import evaluate_tokens +from minilm_lab.model import ( + ATTENTION_BACKENDS, + DEFAULT_ATTENTION_BACKEND, + canonical_backend, +) + +ROOT = Path(__file__).resolve().parents[1] +CONFIG = {"vocab_size": 64, "context_length": 32, "embedding_dim": 48, "num_layers": 2, "num_heads": 4} +# Maximum absolute logit error against the float64 reference-backend oracle. +# float32 kernels differ in reduction order; float16 is limited by its 10-bit mantissa. +TOLERANCES = { + ("cpu", torch.float64, "sdpa"): 1e-10, + ("cpu", torch.float64, "mha"): 1e-10, + ("cpu", torch.float64, "reference"): 1e-10, + ("cpu", torch.float32, "sdpa"): 1e-4, + ("cpu", torch.float32, "mha"): 1e-4, + ("cpu", torch.float32, "reference"): 1e-4, + ("mps", torch.float32, "sdpa"): 1e-3, + ("mps", torch.float32, "mha"): 1e-3, + ("mps", torch.float32, "reference"): 1e-3, + ("mps", torch.float16, "sdpa"): 5e-2, + ("mps", torch.float16, "mha"): 5e-2, + ("mps", torch.float16, "reference"): 5e-2, +} + + +def model(positional_encoding: str = "learned", **overrides) -> MiniLM: + torch.manual_seed(21) + return MiniLM(MiniLMConfig(**{**CONFIG, **overrides}, positional_encoding=positional_encoding)).eval() + + +def backends_in_use(m: MiniLM) -> set[str]: + return {block.attention.backend for block in m.blocks} + + +def test_sdpa_is_the_default_backend() -> None: + assert DEFAULT_ATTENTION_BACKEND == "sdpa" + assert backends_in_use(model()) == {"sdpa"} + + +@pytest.mark.parametrize(("name", "canonical"), [("reference", "reference"), ("mha", "mha"), ("sdpa", "sdpa"), ("auto", "mha")]) +def test_explicit_backend_selection_and_legacy_alias(name: str, canonical: str) -> None: + assert backends_in_use(model().set_attention_backend(name)) == {canonical} + + +def test_unknown_or_missing_backend_names_are_rejected() -> None: + with pytest.raises(ValueError): + model().set_attention_backend("flash") + with pytest.raises(ValueError): + model().set_attention_backend(None) + # A checkpoint without the field predates it and was trained with mha. + assert canonical_backend(None) == "mha" + + +def test_state_dict_is_identical_across_backends() -> None: + reference = model().state_dict() + for backend in ATTENTION_BACKENDS: + state = model().set_attention_backend(backend).state_dict() + assert state.keys() == reference.keys() + assert all(torch.equal(state[key], reference[key]) for key in state) + + +@pytest.mark.parametrize("device", ["cpu", "mps"]) +@pytest.mark.parametrize("positional_encoding", ["learned", "rope"]) +def test_backend_specific_tolerances(positional_encoding: str, device: str, request: pytest.FixtureRequest) -> None: + if device == "mps": + request.getfixturevalue("mps_device") # skipped with a reason, or failed under MINILM_REQUIRE_MPS=1 + tokens = torch.randint(0, 64, (2, 32), generator=torch.Generator().manual_seed(4)) + base = model(positional_encoding) + oracle = base.double().set_attention_backend("reference")(tokens)[0] + for (target, dtype, backend), tolerance in TOLERANCES.items(): + if target != device: + continue + candidate = model(positional_encoding).to(device=device, dtype=dtype).set_attention_backend(backend) + error = (candidate(tokens.to(device))[0].cpu().double() - oracle).abs().max().item() + assert error < tolerance, (device, dtype, backend, error) + + +@pytest.mark.parametrize("backend", ATTENTION_BACKENDS) +def test_causal_masking_per_backend_with_batch_and_rope(backend: str) -> None: + m = model("rope").set_attention_backend(backend) + tokens = torch.randint(0, 64, (3, 32), generator=torch.Generator().manual_seed(5)) + changed = tokens.clone() + changed[:, 20:] = (changed[:, 20:] + 1) % 64 + assert torch.allclose(m(tokens)[0][:, :20], m(changed)[0][:, :20], atol=1e-6) + + +def test_evaluation_matches_manual_loss_for_every_backend() -> None: + ids = torch.randint(0, 64, (200,), generator=torch.Generator().manual_seed(6)) + results = {} + for backend in ATTENTION_BACKENDS: + m = model().set_attention_backend(backend) + total, count = 0.0, 0 + with torch.no_grad(): + for end in range(1, ids.numel(), 32): + stop = min(end + 32, ids.numel()) + logits, _ = m(ids[end - 1 : stop - 1].unsqueeze(0)) + total += F.cross_entropy(logits[0], ids[end:stop], reduction="sum").item() + count += stop - end + evaluation = evaluate_tokens(m, ids, "cpu") + assert evaluation["validation_tokens"] == count == ids.numel() - 1 + assert abs(evaluation["validation_loss"] - total / count) < 1e-9 + results[backend] = evaluation["validation_loss"] + assert max(results.values()) - min(results.values()) < 1e-5 + + +@pytest.mark.parametrize("backend", ATTENTION_BACKENDS) +def test_sdpa_dropout_is_disabled_in_evaluation(backend: str, monkeypatch: pytest.MonkeyPatch) -> None: + seen = [] + real = F.scaled_dot_product_attention + + def spy(*args, **kwargs): + seen.append(kwargs.get("dropout_p", 0.0)) + return real(*args, **kwargs) + + monkeypatch.setattr(F, "scaled_dot_product_attention", spy) + m = model("rope", dropout=0.3).set_attention_backend(backend) + tokens = torch.randint(0, 64, (2, 16), generator=torch.Generator().manual_seed(7)) + m.eval() + first, second = m(tokens)[0], m(tokens)[0] + m.forward_cached(tokens) + assert torch.equal(first, second) + if backend != "reference": # sdpa, and mha with RoPE, call SDPA + assert seen + assert all(p == 0.0 for p in seen) + seen.clear() + m.train() + m(tokens) + if backend != "reference": # the reference path applies dropout itself + assert seen and all(p == 0.3 for p in seen) + + +def train(tmp_path: Path, name: str, extra: list[str]) -> subprocess.CompletedProcess: + text = tmp_path / "corpus.txt" + if not text.exists(): + text.write_text("Light is propagated in vacuo with a constant velocity. " * 40, encoding="utf-8") + command = [ + sys.executable, str(ROOT / "scripts/train_tiny.py"), str(text), "--steps", "2", "--context-length", "16", + "--embedding-dim", "16", "--num-layers", "1", "--num-heads", "2", "--batch-size", "2", "--eval-interval", "2", + "--device", "cpu", "--output-dir", str(tmp_path / name), *extra, + ] + return subprocess.run(command, check=False, capture_output=True, text=True, cwd=ROOT) + + +def test_training_defaults_to_sdpa_and_stores_canonical_names(tmp_path: Path) -> None: + assert train(tmp_path, "default", []).returncode == 0 + assert torch.load(tmp_path / "default/tiny_checkpoint.pt", weights_only=True)["attention_backend"] == "sdpa" + assert train(tmp_path, "legacy", ["--attention-backend", "auto"]).returncode == 0 + assert torch.load(tmp_path / "legacy/tiny_checkpoint.pt", weights_only=True)["attention_backend"] == "mha" + + +def test_checkpoints_without_backend_field_resume_only_as_mha(tmp_path: Path) -> None: + assert train(tmp_path, "old", ["--attention-backend", "mha"]).returncode == 0 + path = tmp_path / "old/tiny_checkpoint.pt" + checkpoint = torch.load(path, weights_only=True) + del checkpoint["attention_backend"] # as written before the field existed + torch.save(checkpoint, path) + assert train(tmp_path, "old", ["--resume"]).returncode != 0 # sdpa default refused + resumed = train(tmp_path, "old", ["--resume", "--attention-backend", "mha"]) + assert resumed.returncode == 0, resumed.stderr + assert torch.load(path, weights_only=True)["step"] == 4 + + +def test_evaluate_script_default_and_explicit_backend(tmp_path: Path) -> None: + assert train(tmp_path, "run", ["--attention-backend", "mha"]).returncode == 0 + outputs = {} + for backend in (None, "mha", "reference"): + out = tmp_path / f"eval-{backend}.json" + command = [ + sys.executable, str(ROOT / "scripts/evaluate.py"), str(tmp_path / "corpus.txt"), "--device", "cpu", + "--checkpoint", str(tmp_path / "run/best_checkpoint.pt"), "--output", str(out), + ] + (["--attention-backend", backend] if backend else []) + subprocess.run(command, check=True, capture_output=True, cwd=ROOT) + outputs[backend] = json.loads(out.read_text()) + assert outputs[None]["attention_backend"] == "sdpa" + assert outputs[None]["checkpoint_attention_backend"] == "mha" + assert outputs["reference"]["attention_backend"] == "reference" + losses = [o["validation_loss"] for o in outputs.values()] + assert max(losses) - min(losses) < 1e-5 diff --git a/tests/test_release_blockers.py b/tests/test_release_blockers.py new file mode 100644 index 0000000..6fd6289 --- /dev/null +++ b/tests/test_release_blockers.py @@ -0,0 +1,339 @@ +"""Regression tests for the three release-blocking defects found in ad71757. + +Fix 1: model identities must be bound to checkpoint artifacts or a pinned audited baseline. +Fix 2: duplicate experiments are detected after backend canonicalization and identity resolution. +Fix 3: KV-cache speedups are derived from timings; stored ratios are checked, never trusted. + +All tests use copies of experiments/records. Tests that need the original checkpoint +files skip when they are absent (fresh clones); every other case runs everywhere. +""" + +import json +import shutil +import subprocess +import sys +from pathlib import Path + +import pytest +import torch + +from minilm_lab import MiniLM, MiniLMConfig +from minilm_lab.identity import checkpoint_identity +from minilm_lab.records import ( + ARTIFACT_VERIFIED, + BASELINE_PATH, + BASELINE_VERIFIED, + METADATA_ONLY, + UNVERIFIED, + RecordError, + identity_evidence, + kv_speedups, + load_store, + protocol_rows, + resolve_identity, + validate_store, +) + +ROOT = Path(__file__).resolve().parents[1] +RECORDS = ROOT / "experiments/records" +ATTESTATION = "identity/20261022T053914Z_model_identity_attestation_935a229.json" +V1_42 = "training/20261022T041654Z_training_science-v1_auto_seed42_48e95f8.json" +V1_43 = "training/20261022T041103Z_training_science-v1_auto_seed43_48e95f8.json" +V1_44 = "training/20261022T041335Z_training_science-v1_auto_seed44_48e95f8.json" +KV = "kv_cache/20261022T053124Z_kv_cpu_float32_57be26d.json" +HISTORICAL_KV = "kv_cache/20261022T035538Z_kv_cpu_float32_93507c4.json" +LEGACY_KV = "legacy/20261022T045428Z_legacy_kv_cache_cpu_13530f4.json" +ARTIFACTS_PRESENT = (ROOT / "artifacts/runs/sdpa-v1/auto-s42/best_checkpoint.pt").is_file() + + +@pytest.fixture +def records(tmp_path: Path) -> Path: + copy = tmp_path / "records" + shutil.copytree(RECORDS, copy) + return copy + + +def read(root: Path, key: str) -> dict: + return json.loads((root / key).read_text()) + + +def write(root: Path, key: str, record: dict) -> None: + (root / key).write_text(json.dumps(record, indent=2)) + + +def edit(root: Path, key: str, change) -> None: + record = read(root, key) + change(record) + write(root, key, record) + + +def problems(root: Path, **options) -> list[str]: + strict = options.pop("strict", False) + return validate_store(load_store(ROOT, root, **options), strict=strict) + + +def attested_row(record: dict, training_key: str, kind: str = "best") -> dict: + return next(r for r in record["results"]["attested"] if r["training_record"] == training_key and r["kind"] == kind) + + +# ================================================================ Fix 1: artifact / attestation integrity + + +@pytest.mark.parametrize("artifacts", ["present", "absent"]) +def test_zeroed_attestation_fingerprint_fails(records: Path, tmp_path: Path, artifacts: str) -> None: + edit(records, ATTESTATION, lambda r: attested_row(r, V1_43).update(model_identity_sha256="0" * 64)) + options = {} if artifacts == "present" else {"artifact_root": tmp_path / "empty"} + assert any("model identity sources disagree" in p and V1_43 in p for p in problems(records, **options)) + + +def test_declared_fingerprint_contradicting_attestation_fails(records: Path, tmp_path: Path) -> None: + edit(records, V1_43, lambda r: r["results"].update(best_model_identity_sha256="a" * 64)) + # Even with no artifact and no baseline, declared vs attested must agree. + found = problems(records, artifact_root=tmp_path / "empty", baseline_path=None) + assert any("model identity sources disagree" in p and "declared=aaaaaaaaaaaa" in p for p in found) + + +def test_attestation_with_another_checkpoints_fingerprint_fails(records: Path) -> None: + other = attested_row(read(RECORDS, ATTESTATION), V1_44)["model_identity_sha256"] + edit(records, ATTESTATION, lambda r: attested_row(r, V1_43).update(model_identity_sha256=other)) + assert any("model identity sources disagree" in p and V1_43 in p for p in problems(records)) + + +def test_changed_checkpoint_bytes_without_updated_hash_fail(records: Path, tmp_path: Path) -> None: + artifact_root = tmp_path / "artifacts-root" + checkpoint = artifact_root / "artifacts/runs/sdpa-v1/auto-s43/best_checkpoint.pt" + checkpoint.parent.mkdir(parents=True) + checkpoint.write_bytes(b"tampered checkpoint bytes") + found = problems(records, artifact_root=artifact_root) + assert any("do not match the recorded SHA-256" in p and V1_43 in p for p in found) + evidence, _ = resolve_identity(load_store(ROOT, records, artifact_root=artifact_root), V1_43) + assert evidence.status == UNVERIFIED and evidence.fingerprint is None + + +@pytest.mark.skipif(not ARTIFACTS_PRESENT, reason="original checkpoint artifacts not in this checkout") +def test_flipped_byte_in_real_checkpoint_fails(records: Path, tmp_path: Path) -> None: + source = ROOT / "artifacts/runs/sdpa-v1/auto-s43/best_checkpoint.pt" + target = tmp_path / "root/artifacts/runs/sdpa-v1/auto-s43/best_checkpoint.pt" + target.parent.mkdir(parents=True) + data = bytearray(source.read_bytes()) + data[len(data) // 2] ^= 0x01 + target.write_bytes(bytes(data)) + assert any("do not match the recorded SHA-256" in p for p in problems(records, artifact_root=tmp_path / "root")) + + +@pytest.mark.skipif(not ARTIFACTS_PRESENT, reason="original checkpoint artifacts not in this checkout") +def test_recomputing_the_historical_baseline_from_artifacts_reproduces_it() -> None: + run = subprocess.run([sys.executable, str(ROOT / "scripts/build_identity_baseline.py"), "--check"], check=False, capture_output=True, text=True, cwd=ROOT) + assert run.returncode == 0, run.stdout + run.stderr + levels = {e.status for e in identity_evidence(load_store(ROOT), []).values()} + assert levels == {ARTIFACT_VERIFIED} + + +def test_legitimate_historical_attestation_agrees_with_pinned_baseline(tmp_path: Path) -> None: + baseline = json.loads((ROOT / BASELINE_PATH).read_text())["entries"] + for row in read(RECORDS, ATTESTATION)["results"]["attested"]: + assert baseline[row["checkpoint_sha256"]]["model_identity_sha256"] == row["model_identity_sha256"] + store = load_store(ROOT, artifact_root=tmp_path / "empty") + assert validate_store(store, strict=True) == [] + assert {e.status for e in identity_evidence(store, []).values()} == {BASELINE_VERIFIED} + + +def test_unavailable_artifacts_and_baseline_are_never_reported_as_verified(records: Path, tmp_path: Path) -> None: + store = load_store(ROOT, records, artifact_root=tmp_path / "empty", baseline_path=None) + assert validate_store(store) == [] # internally consistent metadata... + assert {e.status for e in identity_evidence(store, []).values()} == {METADATA_ONLY} # ...is only that + strict = validate_store(load_store(ROOT, records, artifact_root=tmp_path / "empty", baseline_path=None), strict=True) + assert strict and all("metadata_consistent_only" in p for p in strict) + # A forged attestation cannot be detected without independent evidence, so strict must fail. + edit(records, ATTESTATION, lambda r: attested_row(r, V1_43).update(model_identity_sha256="0" * 64)) + assert problems(records, artifact_root=tmp_path / "empty", baseline_path=None, strict=True) + + +def test_validator_cli_reports_evidence_levels_and_fails_closed(tmp_path: Path) -> None: + command = [sys.executable, str(ROOT / "scripts/validate_records.py"), "--artifact-root", str(tmp_path)] + baseline = subprocess.run(command + ["--strict"], check=False, capture_output=True, text=True, cwd=ROOT) + assert baseline.returncode == 0 and f"{BASELINE_VERIFIED}=24" in baseline.stdout and ARTIFACT_VERIFIED not in baseline.stdout + metadata = subprocess.run(command + ["--strict", "--no-baseline"], check=False, capture_output=True, text=True, cwd=ROOT) + assert metadata.returncode == 1 and f"{METADATA_ONLY}=24" in metadata.stdout + artifacts = subprocess.run(command + ["--require-artifacts"], check=False, capture_output=True, text=True, cwd=ROOT) + assert artifacts.returncode == 1 + + +def test_tampered_baseline_is_not_trusted(tmp_path: Path) -> None: + baseline = json.loads((ROOT / BASELINE_PATH).read_text()) + first = next(iter(baseline["entries"])) + baseline["entries"][first]["model_identity_sha256"] = "0" * 64 + forged = tmp_path / "baseline.json" + forged.write_text(json.dumps(baseline)) + found = validate_store(load_store(ROOT, artifact_root=tmp_path / "empty", baseline_path=forged), strict=True) + assert any("does not match the pinned audited baseline" in p for p in found) + + +def synthetic_run(tmp_path: Path, records: Path, declared: str | None) -> tuple[Path, str, str]: + """A modern record pointing at a real, freshly written checkpoint artifact.""" + torch.manual_seed(0) + model = MiniLM(MiniLMConfig(vocab_size=32, context_length=8, embedding_dim=16, num_layers=1, num_heads=2)) + checkpoint = {"model": model.state_dict(), "config": model.config.__dict__, "tokenizer": "byte", "metrics": []} + output = "artifacts/runs/synthetic" + root = tmp_path / "root" + (root / output).mkdir(parents=True) + for name in ("best_checkpoint.pt", "tiny_checkpoint.pt"): + torch.save(checkpoint, root / output / name) + import hashlib + + sha = hashlib.sha256((root / output / "best_checkpoint.pt").read_bytes()).hexdigest() + record = read(records, V1_42) + record["created_utc"] = "20261022T090000Z" + record["record_version"] = 2 + record["settings"]["training_command"] = f"python scripts/train_tiny.py data/science/train.txt --output-dir {output}" + record["results"].update(best_checkpoint_sha256=sha, final_checkpoint_sha256=sha, best_model_identity_sha256=declared, + final_model_identity_sha256=declared) + key = "training/20261022T090000Z_training_science-v1_mha_seed99_48e95f8.json" + record["settings"]["seed"] = 99 + write(records, key, record) + return root, key, checkpoint_identity(checkpoint)["sha256"] + + +def test_correct_modern_record_is_artifact_verified(records: Path, tmp_path: Path) -> None: + root, key, identity = synthetic_run(tmp_path, records, declared=None) + edit(records, key, lambda r: r["results"].update(best_model_identity_sha256=identity, final_model_identity_sha256=identity)) + evidence, issues = resolve_identity(load_store(ROOT, records, artifact_root=root), key) + assert issues == [] and evidence.status == ARTIFACT_VERIFIED and evidence.fingerprint == identity + + +def test_modern_record_whose_declared_identity_differs_from_artifact_fails(records: Path, tmp_path: Path) -> None: + root, key, _ = synthetic_run(tmp_path, records, declared="d" * 64) + evidence, issues = resolve_identity(load_store(ROOT, records, artifact_root=root), key) + assert evidence.status == UNVERIFIED and any("sources disagree" in p for p in issues) + + +# ================================================================ Fix 2: canonical duplicate integrity + + +def add_duplicate(root: Path, base: str = V1_42, stamp: str = "20261022T060000Z", change=None) -> str: + record = read(root, base) + record["settings"]["attention_backend"] = "mha" + record["created_utc"] = stamp + if change: + change(record) + seed = record["settings"]["seed"] + key = f"training/{stamp}_training_science-v1_mha_seed{seed}_48e95f8.json" + write(root, key, record) + return key + + +def test_legitimate_auto_mha_duplicate_counts_once(records: Path) -> None: + duplicate = add_duplicate(records) + assert problems(records, strict=True) == [] + rows = protocol_rows(load_store(ROOT, records), "science-v1", ["sdpa_v1_eval_cpu"]) + for backend in ("mha", "sdpa"): + assert sorted(r["seed"] for r in rows if r["backend"] == backend) == [42, 43, 44] + assert next(r for r in rows if r["backend"] == "mha" and r["seed"] == 42)["replicates"] == [duplicate] + + +def test_auto_mha_duplicate_with_contradictory_fingerprint_fails(records: Path) -> None: + add_duplicate(records, change=lambda r: r["results"].update(best_model_identity_sha256="b" * 64)) + found = problems(records) + assert any("sources disagree" in p for p in found) and any("conflicting training records" in p for p in found) + with pytest.raises(RecordError): + protocol_rows(load_store(ROOT, records), "science-v1", ["sdpa_v1_eval_cpu"]) + + +def test_duplicate_with_conflicting_checkpoint_identity_fails(records: Path) -> None: + add_duplicate(records, change=lambda r: r["results"].update(best_checkpoint_sha256="c" * 64)) + assert any("conflicting training records" in p and "different checkpoint files" in p for p in problems(records)) + + +def test_duplicate_with_contradictory_losses_fails(records: Path) -> None: + add_duplicate(records, change=lambda r: r["results"]["metrics"][-1].update(validation_loss=2.5)) + assert any("conflicting training records" in p and "losses" in p for p in problems(records)) + + +def test_duplicate_with_contradictory_configuration_fails(records: Path) -> None: + add_duplicate(records, change=lambda r: r["settings"].update(training_command=r["settings"]["training_command"].replace("--steps 1000", "--steps 2000"))) + assert any("conflicting training records" in p and "configuration differs" in p for p in problems(records)) + + +def test_explicit_valid_supersession_resolves_a_conflict(records: Path) -> None: + duplicate = add_duplicate(records, change=lambda r: r["results"]["metrics"][-1].update(validation_loss=2.5)) + index = json.loads((records / "SUPERSEDED.json").read_text()) + index["records"][duplicate] = {"reason": "test: conflicting re-record", "superseded_by": V1_42} + (records / "SUPERSEDED.json").write_text(json.dumps(index)) + assert problems(records, strict=True) == [] + + +@pytest.mark.parametrize("target", ["missing", "wrong_seed", "cycle", "other_kind"]) +def test_invalid_supersession_fails(records: Path, target: str) -> None: + index = json.loads((records / "SUPERSEDED.json").read_text()) + duplicate = add_duplicate(records, change=lambda r: r["results"]["metrics"][-1].update(validation_loss=2.5)) + destination = {"missing": "training/does_not_exist.json", "wrong_seed": V1_43, "cycle": duplicate, "other_kind": KV}[target] + index["records"][duplicate] = {"reason": "test", "superseded_by": destination} + if target == "cycle": + second = add_duplicate(records, stamp="20261022T070000Z") + index["records"][duplicate]["superseded_by"] = second + index["records"][second] = {"reason": "test", "superseded_by": duplicate} + (records / "SUPERSEDED.json").write_text(json.dumps(index)) + assert any("SUPERSEDED.json" in p for p in problems(records)) + + +# ================================================================ Fix 3: KV speedup derivation + + +def kv_problems(root: Path) -> list[str]: + return [p for p in problems(root) if p.startswith(("kv_cache/", "legacy/"))] + + +def test_stored_speedup_of_999_fails_and_is_never_rendered(records: Path) -> None: + edit(records, KV, lambda r: r["results"]["grid"][0].update(speedup_cached_vs_uncached_sdpa=999.0)) + assert any("stored speedup_cached_vs_uncached_sdpa 999.0 differs from derived" in p for p in kv_problems(records)) + import importlib.util + + spec = importlib.util.spec_from_file_location("report", ROOT / "scripts/build_experiment_report.py") + report = importlib.util.module_from_spec(spec) + spec.loader.exec_module(report) + table = report.kv_table("cpu/float32", read(records, KV)) + assert "999.00×" not in table + + +@pytest.mark.parametrize("value", [float("nan"), float("inf"), 0.0, -2.0]) +def test_non_finite_or_impossible_stored_ratio_fails(records: Path, value: float) -> None: + edit(records, KV, lambda r: r["results"]["grid"][1].update(speedup_cached_vs_uncached_mha=value)) + assert any("speedup_cached_vs_uncached_mha" in p for p in kv_problems(records)) + + +def test_altered_median_with_original_raw_samples_fails(records: Path) -> None: + edit(records, KV, lambda r: r["results"]["grid"][3]["uncached_sdpa"]["total_ms"].update(median=1.0)) + assert any("summaries differ from raw samples" in p for p in kv_problems(records)) + + +def test_altered_raw_sample_with_original_summary_fails(records: Path) -> None: + edit(records, KV, lambda r: r["results"]["grid"][3]["cached"]["raw"]["step_seconds"][2].__setitem__(5, 0.5)) + assert any("summaries differ from raw samples" in p for p in kv_problems(records)) + + +@pytest.mark.parametrize("value", [0.0, -0.001, float("nan")]) +def test_zero_or_invalid_timing_denominator_fails_without_crashing(records: Path, value: float) -> None: + def zero(record: dict) -> None: + for steps in record["results"]["grid"][0]["cached"]["raw"]["step_seconds"]: + for index in range(len(steps)): + steps[index] = value + edit(records, KV, zero) + assert any("non-positive or non-finite" in p for p in kv_problems(records)) + + +def test_valid_raw_samples_and_derived_ratios_pass() -> None: + assert kv_problems(RECORDS) == [] + row = read(RECORDS, KV)["results"]["grid"][4] + assert kv_speedups(row)["speedup_cached_vs_uncached_sdpa"] == row["speedup_cached_vs_uncached_sdpa"] + + +def test_historical_summary_only_record_passes_under_legacy_rules(records: Path) -> None: + assert "raw" not in read(RECORDS, HISTORICAL_KV)["results"]["grid"][0]["cached"] + assert kv_problems(records) == [] + edit(records, HISTORICAL_KV, lambda r: r["results"]["grid"][0].update(speedup_cached_vs_uncached_sdpa=999.0)) + assert any(HISTORICAL_KV in p and "differs from derived" in p for p in kv_problems(records)) + + +def test_legacy_single_prompt_kv_benchmark_is_derived_from_samples(records: Path) -> None: + edit(records, LEGACY_KV, lambda r: r["results"].update(speedup=9.99)) + assert any("stored speedup 9.99 differs from derived" in p for p in kv_problems(records))