Each model sets up Llama3.1-8B by itself — how fast does it run?

rows = the model acting as the agent, columns = the hardware it was given; every cell sets up Llama3.1-8B except Jetson† · the last two columns are physical devices on a bench, the rest are cloud instances · cell = speed of the server the model set up (tok/s), color = mean, log scale, normalized within each column · Qwen3.5-9B ran with thinking off · click a cell, or a row/column label, for boxplots · data as of 30 July 2026
Display tokens per second by:
Failed setups:
Chart:
Cell = mean tok/s over runs, taking each run's best attempt; color = log scale, normalized within each column slowest → fastest model on that hardware. ■ no successful setup · ■ not attempted. Click a cell, or a row / column label, for boxplots.
Click a cell, or a row/column label, to see boxplots.
The hardware roofline (dashed) is the highest tok/s achievable on this hardware with regular decode-stage inference — without speculative decoding or EAGLE, which can push measured throughput above it. For the two physical columns the line is a single-stream ceiling (that is how those runs were measured), whereas the cloud columns are aggregates at concurrency 8; and on the Jetson Nano, prefill is compute-bound (no Armv8.2 SDOT), so this decode-bandwidth line is optimistic there.
Throughput vs. capability (ECI)
Each point = one model on one kind of hardware — x = ECI, y = tok/s. ECI scores are Epoch's, as of 1 October 2026. ◇ = our own estimate, for the two models Epoch does not score. Pick one kind of hardware (linear y), or "All" to overlay them (log y, where a model with no successful setup has no point). Llama3.1-8B (ECI 116.58, no successful setup) is left out, because one far-off zero would steer the fit.