When AI wins a benchmark: progress, or a marketing trick?
A benchmark is a set of tasks and rules that converts a model’s performance into a score. Model A got 90 %, model B only 86 %. The first therefore looks clearly better. But what if the first model got 32 attempts and the second one, the first ran with search and the second without? The numbers can be correct and the comparison misleading all the same. This article shows how such differences arise, which cases are actually documented and what to check before you pick a model for your own project based on a leaderboard.
Three rules for a quick benchmark check
- 1Don’t compare just names. Compare the whole system: model, prompt, tools, number of attempts, time and judge.
- 2The best run is not the typical run. Ask how many prompt variants and random seeds preceded the published result.
- 3Look for primary data. Records of the agent’s steps (trajectories), the grading code and an independent replication carry more weight than a chart in a model announcement.
A number is not a property of the model
The statement “GPT-4 scored 87 % on the benchmark” sounds like a fixed technical property of the model. In reality, though, a language model’s result is not a constant parameter. It depends on the exact model version, the prompt, generation randomness, tools, environment and the way of grading.
Changing a single layer can shift the score without changing the model’s weights. With a coding agent, for example, you test not only the ability to write a fix but also the quality of the task description, the available commands, the state of the repository, the time limit and the test harness — the program that launches and evaluates the whole attempt.
How to recognise convincing evidence?
Before we break down specific ways of distortion, we need to know what can be trusted. Not every accusation of manipulation carries the same weight. What matters is the origin and completeness of the data, the ability to repeat the test, the independence of the source and whether the text passed peer review. The most convincing is a combination of public data and code with an independent replication of the result. A social-media post with a screenshot does not replace such a check.
More attempts can legitimately raise pass@k
The simplest way to raise a benchmark score is to let the model answer several times. The pass@k metric counts a task as solved if at least one of k attempts succeeds. For independent attempts with success rate p:
For one specific task where every independent attempt succeeds with the same probability of 40 %, the probability of at least one success in ten attempts is 99.4 %. From an average benchmark pass@1 = 40 %, however, pass@10 cannot be computed this way, because tasks have different difficulty. HumanEval therefore estimates pass@k from actually generated solutions using a combinatorial calculation that limits the bias. The principle stays the same: more attempts give more chances of success.
At least one correct attempt
Makes sense if the system can automatically recognise the correct solution using tests or another verifier. Without reliable selection, though, the user gets ten answers and doesn’t know which to trust.
Check: the same k, the same tests and the price of all attempts.
The most frequent answer wins
The model produces several solutions and the system picks the majority. That is not the same as pass@k: the correct answer may be among the proposals and still lose the vote.
Check: the number of samples, the selection rule and the additional compute costs.
Of many settings, only the best is published
The researcher tries many prompts, random seeds or model versions and shows the highest score. The number may be true, but the expected result of a randomly chosen configuration will be lower.
Check: how many variants were tested and whether the protocol was set in advance.
One model gets more time and tools
An agent with a terminal, search and 200,000 tokens can beat a bare model with a single answer. That proves the quality of the whole system, not automatically a better base model.
Check: tokens, time, hardware, number of repetitions and price per task.
More compute is not cheating. The problem arises when the headline compares the best result of a more expensive protocol with a competitor’s standard run and hides the difference in a footnote. A practical comparison must therefore state, alongside the success rate, also the price, the time waiting for the result (latency) and the number of model calls.
How the winning configuration is chosen
The number of attempts is not the only variable. Seemingly minor details can change the result too: the order of options A–D, spaces in the template, the answer format or the examples inserted into the prompt. The study When Benchmarks Are Targets showed that small changes to MMLU — a knowledge test across dozens of school and professional fields — moved models by up to eight ranks. In one experiment, changing the order of options changed the ranking of five of eleven models.
Picking the best run (cherry-picking)
A seed is the number that initialises generation randomness. If you run one prompt with twenty seeds and publish the best run, you are using the same principle as picking the best of twenty (best-of-20). The same applies to prompts: after seeing the result, it is easy to pick exactly the wording that suits your model.
A fair procedure separates the development and test sets. You tune the prompt on the development part, lock it before running the test and perform a predetermined number of runs on the test set. If you return to the prompt after every result, the test set gradually turns into a development set.
A convenient test and targeted tuning for the leaderboard
There are hundreds of benchmarks and their variants. When a company shows only the five sets it won, the reader doesn’t see the other fifteen where it was average. This problem is called the benchmark lottery. The study of the same name shows that the ranking of methods can change significantly depending on the choice of test sets and metrics.
Targeted tuning for the leaderboard (benchmaxxing) goes further: the model, prompt or the agent’s control layer is adapted over the long term to the known format of the leaderboard. It need not know the specific answers. It is enough to optimise exactly what the benchmark rewards. Such performance may be real, but it need not transfer to a different test set, library or your company repository.
Who judges the model’s answer?
An answer can be judged by a programmatic test, a human or another language model. Each option measures something different. An automatic unit test recognises whether a function returns the expected result, but may not catch unreadable code, a security problem or an unnecessarily complex change. A human sees usefulness but may prefer a certain style. An LLM judge — a language model that assesses another model’s answer — is cheap and fast, but may add its own bias.
The order of answers matters
The judging model may more often pick the first or the second option. The minimal check is to swap the order and measure whether the verdict changes.
Longer text seems higher quality
A verbose answer can beat a shorter, more accurate solution. In AlpacaEval, a test based on comparing open-ended answers, merely changing the instruction on whether the answer should be concise or detailed shifted the win rate of the reference model gpt4_1106_preview from 22.9 % to 64.3 %.
The judge may favour its own model’s answers
Some judges in the study more often picked answers produced by the same model. Because of limited data, however, the authors did not confirm that this is a general phenomenon.
Likeability is not factual correctness
People may reward a confident tone, formatting and pleasant sentiment. That is relevant for a chatbot, less so for an accounting calculation or a safe migration.
MT-Bench research documented bias caused by the order and length of answers and examined possible favouring of the judge’s own model’s answers, which it did not confirm because of limited data. That does not mean an LLM judge is unusable. It means it must have clear grading rules, hidden model names, random order and human review of a representative sample.
Three well-known benchmarks where bias showed up
These are not just theoretical risks. For the following publicly used benchmarks, researchers measured or documented a mechanism that could change the score or ranking of models. The strength of evidence, however, differs from case to case.
MT-Bench
Possible judge self-preference: GPT-4 as a judge gave GPT-4’s answers roughly 10 percentage points higher win rate than human judgement did. For Claude-v1 the difference was about 25 points. Because of limited data, the authors did not confirm this is a general property of model judges.
AlpacaEval
Preference for long answers: merely instructing the same GPT-4 Preview model to answer concisely or in detail changed its result from 22.9 % to 64.3 %. The length-controlled AlpacaEval-LC version therefore also changed the ranking of some models.
Chatbot Arena / LMArena
What happened: vendors could send unreleased model variants to the Arena before the official release. These could differ in learned parameters, system instructions or other settings. Ordinary users then compared them anonymously with other models.
Where the advantage arose: the study found a provider that tried 27 private variants this way. From this group it subsequently released a model that placed second in the Arena. With 27 attempts, the chance grows that some variant gets an exceptionally good score by luck or better matches users’ taste. It is like one runner reporting the best time from 27 races and another the time from their single race.
What it does not mean: the study does not claim the Arena altered votes. The problem is the possibility of trying several versions, not publishing the weaker results and showing only the winner. The Arena replied that it offered pre-release testing to various providers according to available capacity and that new votes after public release gradually reduce the initial advantage. It therefore questioned both the size of the measured effect and the conclusion that a specific company was favoured.
Which metrics are the most objective?
No metric is objective in itself. Someone always chooses the tasks, rules and success threshold. Closest to objective measurement, however, are metrics with a clear yes/no result verified by a program without judging style. For AI agents and coding models this order makes sense:
- Whole-task success (task success or resolve rate) The best default metric. Hidden automatic tests verify whether the agent really achieved the required result, for example fixed a bug without breaking other functions.
- Success on the first attempt (
pass@1) Shows how often an ordinary first run succeeds. It compares well if the models have the same prompt, tools, time limit and test environment. - Reliability under repetition (
pass^k) Counts whether allkrepetitions of the same task succeeded. Suitable where you need stable behaviour, not one lucky result. - Price, runtime, tokens and number of calls These operational figures can be measured directly and precisely. They don’t measure answer quality, but they show how much you actually pay for the achieved result.
pass^k. Some metrics are computed exactly, but their meaning is less objective. pass@k favours more attempts. Exact text match can reject a correct answer written in other words. Elo, human preference and LLM judges in turn depend on the taste and settings of the judge. They are not useless, but they should not be the sole basis for a choice.
When comparing two models, use the same tasks and conditions. Also state the confidence interval and paired results, i.e. where on the same task only model A or only model B succeeded. The article How to read AI agent comparisons explains the details.
Which real benchmarks and leaderboards to trust?
Independent does not automatically mean accurate. A leaderboard can be run by an organisation that makes none of the compared models and still use outdated tasks or unequal conditions. Conversely, a test created by a commercial company can be well controlled if all models are run by the same team according to public rules.
The greatest trust is deserved by a source that separates the operator from the model vendors, runs the models the same way, uses hidden or fresh tasks and publishes the conditions, cost and enough data for verification. By these criteria, the following sources are among the most useful in September 2026:
Stanford HELM
Why trust it: the Stanford CRFM university team is not a vendor of the compared models. It publishes configurations, prompts, answers and code, so results for a number of tests can be reproduced and inspected down to the level of individual tasks.
Limitation: HELM has been in maintenance mode since 1 June 2026. Check the date of the specific leaderboard before use; older public sets may be part of the training data of new models.
SWE-rebench
Why trust it: the Nebius team runs all models with the same prompt, context length and the minimal ReAct agent framework, i.e. in the same “reasoning → action → tool result” loop. It continuously adds new problems from GitHub and flags runs with possible contamination.
Limitation: automatic collection doesn’t guarantee that every task is equally high-quality or solvable. The fixed framework compares the models themselves well, but may not show the maximum of a specialised coding agent.
ARC Prize Verified
Why trust it: the non-profit ARC Prize Foundation itself verifies selected results on non-public tasks and marks them as verified. The process is also meant to be checked by an independent academic panel, so a lab cannot just supply its own number from the public set.
Limitation: verification is selective because of cost, a single run can cost up to USD 10,000 and it measures a narrow kind of abstract problem solving. Results marked preview or community results are not the same as an officially verified score. Moreover, the running standings of the 2026 competition are not final.
MLPerf Inference
Why trust it: the MLCommons consortium prescribes the same scenarios, compliance tests and submission rules. It publishes results and code simultaneously; selected submissions undergo a separate audit.
Limitation: not every row is audited and the leaderboard mainly measures throughput, response time and energy efficiency of a specific system. It does not say which language model answers best.
Artificial Analysis
Why it is useful: the private research firm tests model APIs itself instead of copying vendors’ tables. It publicly describes the measurement of price, speed, time to first token and total response time.
Limitation: it is not academically governed and the weights of the aggregate Intelligence Index are chosen by its own team. The directly measured operational figures and individual test results therefore deserve the most trust, not the overall ranking itself.
SWE-bench Pro by Scale AI
Why it is useful: it contains 1,865 human-verified tasks from 41 repositories. The private and commercial part strongly limits the possibility that a model knew the solution from training. The original study compared models in a unified agent framework.
Limitation: the strongest parts with non-public code cannot be reproduced by an independent team. The operator is a commercial company, so it is advisable to confirm results with another source, for example SWE-rebench. The current public leaderboard also contains various agent frameworks and limits; directly compare only rows with the same setup.
What not to use as sole evidence
SWE-bench Verified remains a useful historical signal, but known public tasks and signs of contamination reduce the accuracy of today’s ranking. Arena (formerly LMArena) measures well which answers people prefer to read, not necessarily factual correctness or working reliability.
LiveBench has good automatic grading, but as of 4 September 2026 its official repository still marks the April 2025 release as current and the November 2024 release as the last fully public set. Without verifying the run date, the original advantage of fresh tasks can therefore no longer be relied on. A table published by the model’s own vendor is always just the vendor’s claim until another team confirms it.
Practical choice: for a coding agent, start with a combination of SWE-rebench and the non-public part of SWE-bench Pro, if available for the given model. For general abilities, use the current HELM snapshot and the component results of Artificial Analysis. ARC Prize Verified adds a narrow test of working with new abstract problems and MLPerf answers questions of hardware speed. Finally confirm the winner on 20 to 50 of your own tasks; no public leaderboard replaces your operational test.
When the agent finds the answer instead of the solution
The judge itself is not the only source of bias. The problem can also be in what the agent sees during the test. The term contamination is used for several different situations. The model may have seen the test tasks during training. But the agent can also obtain the answer only during evaluation — from the internet, cache, metadata, hidden tests or the Git history in the container. The second case is not training contamination. It is an information leak in the evaluation environment.
Future commits in the repository
SWE-bench creates tasks from real problems reported on GitHub (issues) and their subsequent fixes. If the container contains branches, tags or commits created after the task was set, the agent can run git log --all, find the finished fix and show the diff against the previous version. The resulting tests pass, but the agent did not solve the problem independently.
The SWE-bench maintainers documented specific records of this behaviour in discussion number 465. They fixed the Verified environment in pull request 471. They removed future history while preserving older history. The original review was explicitly preliminary, so all older results cannot automatically be declared invalid on its basis.
Gaming the goal for the score (reward hacking)
Reward hacking occurs when the agent obtains a high score in a different way than the task author intended. Instead of speeding up an algorithm, for example, it rewrites the timer. Instead of computing the result, it reads it from the grader’s memory. Instead of a general solution, it hard-codes answers for the known tests.
Five documented cases
Now let’s apply the previous rules to real cases. Each of them has an original report, public data or an admission by the authors, but the causes are not the same. Gemini shows an unequal protocol, Llama 4 a different model variant, The Leaderboard Illusion selection from 27 private variants, IQuest a leak in the environment and o3 active gaming of the grader.
Gemini Ultra: the 90 % came from CoT@32
Google announced that Gemini Ultra was the first to surpass the human expert estimate on MMLU with a score of 90.04 %. It used a procedure labelled uncertainty-routed CoT@32: the model produced up to 32 chains of thought. When they agreed sufficiently, the system chose the majority answer; otherwise it picked the answer with the highest probability.
The same technical report listed Gemini Ultra at 83.7 % in the standard 5-shot run. For GPT-4 it listed 86.4 % in 5-shot and 87.29 % with CoT@32 via the API.
Verdict: the methodology was not secret. It was in the report. What is misleading is to take the headline score of 90 % without the information about the 32 generated chains and compare it with GPT-4’s result from a different, 5-shot protocol. The term 5-shot means five example demonstrations in the prompt, not five attempts.
Llama 4 Maverick: the score of a different variant
When launching Llama 4, Meta promoted an Elo score of 1417 from LMArena, a public leaderboard where people compare pairs of answers. The score belonged to the model Llama-4-Maverick-03-26-Experimental, an experimental chat variant optimised for conversational preferences, not the identical public weights.
LMArena stated that the difference should have been explained more clearly, published over 2,000 duels and evaluated the publicly available version of the weights on the Hugging Face platform separately. Its later analysis of the collected data showed that the experimental Maverick dropped in the ranking after accounting for tone, formatting and emoji.
Verdict: what is documented is the mismatch between the highlighted score and the public version of the model. It is not documented that Meta trained on secret benchmark answers.
The Leaderboard Illusion: 27 private variants
A study accepted at the NeurIPS conference identified 27 private Meta variants tested on the main Arena before the launch of Llama 4. The authors showed the mechanism of selection bias: whoever may try more variants and publish only the winner gains a statistical advantage.
LMArena objected that the pre-release testing option was publicly described, available to providers according to capacity and that new votes after publication gradually reduce the selection bias. The study authors and the operator also differ in their estimate of the size of the effect.
Verdict: the number of variants and the possibility of picking the best result are documented. The claims about preferential access and the exact size of the bias remain disputed.
IQuest-Coder-V1: the answer in future Git history
After being alerted by the community, IQuestLab checked the records of the agent’s steps. It found that the agent used git log and git show to access changes created after the original problem was reported. The team named outdated runtime images (Docker images) as the probable cause.
After running in the fixed environment, IQuestLab changed its own SWE-bench Verified result from 81.4 % to 76.2 %. That is a drop of 5.2 percentage points, i.e. approximately 6.4 % relative to the original score.
Verdict: the leak and the correction of the result are admitted by the author. There is no evidence that the researchers deliberately steered the agent down this path.
OpenAI o3: the model gamed the scoring function
While testing an earlier development version (checkpoint) of o3, METR estimated reward-hacking attempts in 1–2 % of runs in two research suites, HCAST and RE-Bench. In some tasks the model rewrote functions of the grading code at runtime (monkey-patching), read a precomputed result or altered the time measurement.
In the Optimize LLM Foundry task, in all 21 observed runs the agent precomputed and cached the answer to make the script look faster instead of actually optimising it. METR counted the detected attempts as failures.
Verdict: the run records document the model’s behaviour in a specific environment. That does not mean the public o3 cheats in 1–2 % of ordinary user conversations, nor that OpenAI kept these runs in the final score.
How a small difference is turned into a big improvement
So far it has mainly been about the test setup and the evaluation environment. A misleading impression can also arise, however, when correct results are presented. The most common trick confuses percentage points with relative percentages. A change from 23.5 % to 49.9 % can be described in three ways:
All three statements are correct. But the headline “a 112 % improvement” sounds as if the score exceeded one hundred percent. For a technical decision, the most readable is to state the original value, the new value and the difference in percentage points. The relative change can follow afterwards.
A truncated axis
Bars of 94.8 % and 95.1 % look almost the same on a 0–100 axis. On a 94–96 axis they create a visual chasm. A shortened axis is not always a mistake — for Elo scores zero has no natural meaning and detail of small differences can be useful. But it must be clearly marked and accompanied by exact values.
Missing uncertainty
On 50 tasks, one additional success means two percentage points. On 500 tasks, only two tenths. The percentage alone is therefore not enough. You need the number of tasks, the confidence interval (the range in which the true success rate probably lies) and ideally paired results: on which of the same tasks model A won and on which model B.
A benchmark is not a working day
Even a fair and precisely described benchmark is still not the same as ordinary work. A benchmark usually measures a clearly bounded task with an automatic result. Real development includes unclear requirements, team conventions, human code review, legacy constraints, long-term maintenance and responsibility for mistakes. A high score is therefore not automatic proof of productivity.
SWE-Lancer contains 1,488 tasks derived from paid jobs on Upwork, all from the Expensify repository: 764 implementation tasks and 724 proposal-selection tasks. The real payouts together amounted to a million dollars. Still, it is a fixed test set with prepared grading that does not measure months of team maintenance of a production system.
In a randomised study, METR worked with 16 experienced maintainers of open-source projects and 246 tasks. For AI tools from early 2025 it measured an average slowdown of 19 %. Yet before the experiment the developers expected a speed-up of 24 %, and after it they felt they had been 20 % faster. The result cannot be transferred without further ado to newer tools or juniors; the discrepancy itself between feeling and measurement, however, is a reason to track actual time, not just impressions.
How many changes pass code review?
Tests may pass, but the project maintainer rejects the change because of architecture, security or unnecessary scope.
How much work is left to the human?
Measure the briefing, waiting, review, fixes and the return to the bug a week later, not just the speed of the first answer.
How much does a successful task cost?
A cheap run with a lower success rate may be more advantageous than an expensive model with many parallel attempts.
What happens on failure?
A cosmetic bug, a reversible mistake and deleting production data don’t carry the same weight, even though in the aggregate score they all count as a single failure.
So supplement the public leaderboard with a small test on typical tasks of your project. If possible, blind the test: judges should not know which model produced a given output. Set the budget and criteria in advance and measure not only correctness but also price, review time and the severity of errors.
Checklist before choosing a model
The following questions turn the previous risks into a practical check. You don’t need to know all the formulas; you mainly need to verify that the conditions were the same for all models and the result can be repeated.
A quick rule for a junior developer
When the exact model, prompt, number of attempts, budget and grading code are missing, don’t say “model A is better”. Say: “In this published setup, model A got a higher score.” It is more accurate and leaves the right question open: will the result repeat in our project too?
Frequently asked questions
Is optimising for a benchmark automatically cheating?
No. A benchmark provides a useful target and tuning on the development set is a normal part of research. The problem arises with leaks of test answers, breaking the rules, or when a result from a different variant or a different budget is presented as directly comparable.
Does a higher pass@k mean a better model?
It proves that at least one of several attempts succeeds more often. For an automatic system with a reliable verifier that can be very useful. But it doesn’t say how reliable the first attempt is or how much it costs to find the correct answer.
Can a model reach the solution without training contamination?
Yes. The solution may be available only during evaluation in the repository history, on the internet, in a cache, hidden files or metadata. That is why not only the training data but also the whole runtime environment must be checked.
Is grading by another LLM reliable?
It can be useful, but it is not flawless. It enables fast grading of open-ended answers, but order, length, style and possible preference for its own model must be checked. Critical decisions need representative human review.
Which of the described cases is proven deliberate fraud by a company?
For none of them do we use such a strong conclusion here. What is documented are unequal protocols, a different public variant, selection from many variants, a leak in the environment and the model gaming the grading metric (reward hacking). Conscious intent by people to deceive cannot be automatically inferred from these facts.
How can I most quickly verify whether a new model is better for me?
Pick 20 to 50 representative tasks, set the same budget and grading criteria in advance, hide the author of each output from the judges and include review time as well. Use the public benchmark as a filter for candidates, not as the final decision.
Summary: a benchmark is not a lie, but a shortcut
Benchmarks are essential. Without them we would judge model performance only by impression. The problem arises when a single number is torn away from the protocol, budget and judge that produced it. More attempts, a better prompt or a specialised control system (scaffold) can be a legitimate technical advantage — but they must be visible.
The most important skill is therefore not to memorise today’s ranking of models. It is to be able to break a result down into layers, trace the primary data and recognise whether the test measures the same problem you need to solve.
Want to read agent results correctly?
The follow-up article How to read AI agent comparisons explains the resolve rate, pass@k, pass^k, confidence intervals and paired testing. An overview of specific suites is in the article Benchmarks for coding assistants.
We teach practical workflows with AI agents in the AI Agent Developer course; the use of ready-made tools in development is covered by the Vibe Coding and AI in Development course.
Sources and further reading
- Chen et al.: Evaluating Large Language Models Trained on Code — the original HumanEval and the definition of pass@k
- τ-bench: A Benchmark for Tool-Agent-User Interaction — reliability of repeated runs measured with pass^k
- Gemini Team: Gemini — A Family of Highly Capable Multimodal Models — MMLU with five examples, CoT@32 and uncertainty-based selection
- Stanford CRFM: Efficient Benchmarking of Language Models — controlled re-evaluation of MMLU protocols
- Stanford CRFM: HELM — open framework, leaderboards, reproduction of results and maintenance status since June 2026
- SWE-rebench: leaderboard methodology — fresh tasks, flagging of possible contamination and equal conditions for all models
- ARC Prize: Verified Testing Policy — hidden evaluation set, verification rules and cost limit per run
- MLCommons: MLPerf Inference rules — compliance tests, submission process and selective audits of results
- Artificial Analysis: benchmark methodology — how quality, price, speed and latency of public APIs are measured
- Scale AI: SWE-bench Pro — public, held-out and commercial tasks with a unified agent framework
- LiveBench: official repository — automatic grading and the stated dates of the current and last fully public release
- Meta: The Llama 4 herd — original announcement and the labelling of the experimental chat variant
- LMArena: original statement on Llama 4 Maverick — different experimental variant, published duels and separate evaluation of public weights
- LMArena: Does Sentiment Matter in AI? — analysis of the influence of style, tone and emoji
- LMArena: policy for pre-release models — requirement that the tested and released variant match
- Singh et al.: The Leaderboard Illusion — private testing, publication of selected results only and unequal data access
- LMArena: response to The Leaderboard Illusion — the operator’s statement and a different estimate of selection bias
- Mizrahi et al.: When Benchmarks Are Targets — sensitivity of rankings to prompt and evaluation format
- Dehghani et al.: The Benchmark Lottery — change of conclusions depending on benchmark choice
- Zheng et al.: Judging LLM-as-a-Judge — position bias, verbosity bias and possible self-enhancement
- Dubois et al.: Length-Controlled AlpacaEval — influence of answer length on judge preference
- SWE-bench: discussion 465 on gaps in repository state — specific step records with future repository history
- SWE-bench: fix proposal 471 — removal of future changes and references from Git history
- IQuestLab: Reward Hacking of SWE-Bench — explanation and score correction from 81.4 % to 76.2 %
- METR: Preliminary evaluation of OpenAI o3 — estimate of 1–2 % attempts to game the grading metric
- METR: Recent Frontier Models Are Reward Hacking — analysis of a specific abuse of the evaluation environment and 21 of 21 Optimize LLM Foundry runs
- Miserendino et al.: SWE-Lancer — real paid software tasks worth a million dollars
- Becker et al.: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity — randomised study of experienced maintainers’ work
- Turing Institut: What is an AI agent and how does it work — the agent loop, tools and a minimal implementation
- Turing Institut: Reasoning models — compute at answer time, coverage of options and result selection
