Reasoning models: what really happens when a model “thinks”

A reasoning model (a model with extended reasoning) generates additional working steps during inference and only then produces the answer. This test-time compute can improve maths, code and planning — or just increase cost and latency. In this article we distinguish when to add compute and how to verify the result independently.

First the vocabulary: five different things under the word “thinking”

Vendors often use the terms thinking, reasoning and chain of thought almost interchangeably. Technically, however, they denote different mechanisms. Without this distinction we easily confuse text for the user with the model’s internal state, or a tool call with introspection.

A reasoning model is a model additionally trained to use working steps purposefully and to be able to use a larger compute budget. An ordinary instruction model can also produce a CoT after the prompt “proceed step by step”, but it may not have been trained on long solving trajectories. Let us also distinguish training compute, which changes the weights, and inference compute, which adds work only to a specific request.

What happens step by step: no pause, just more decoding

For a decoder-only Transformer the request starts with the prefill phase. The server processes the prompt in parallel, creates activations in every layer and stores the attention keys and values in the KV cache. It then moves to autoregressive decode: it predicts the next token, appends its K/V to the cache and repeats. For a typical text reasoning model the first part of decoding goes into an internal reasoning channel r1 … rn; only a special transition switches generation to the final-answer channel.

Working tokens are not an idle pause or a second neural network. Each of them requires another pass through the model’s layers over the KV cache. A long working process therefore mainly increases the time to the first visible token: the user sees nothing while the server generates the hidden sequence.

The exact implementation of proprietary models is not public. A specific system may combine one base model, specialised verifiers, search and latent steps. Documentation describes the interface and billing, not the entire computation graph.

Why the model’s working tokens are not visible

For hosted reasoning models OpenAI does not provide the unedited CoT and offers only a summary of it. Gemini returns a summary and an encrypted signature. With Anthropic the behaviour differs by model generation: older models could return whole blocks of working text, newer ones often return only a summary or a signature.

With open models, for example DeepSeek or gpt-oss, the operator can make the working text available. Hiding it is therefore not a property of the Transformer but a product and safety decision of the API provider.

• Safety and leakage: a raw scratchpad can repeat system/developer instructions, sensitive data, internal policy or harmful variants considered before a refusal.
• Jailbreak surface: an attacker would get detailed feedback on which policy circumventions the model considers and why it rejected them. That speeds up attack adaptation.
• UX: working text contains dead ends, self-corrections and unverified claims. Showing it as an authoritative explanation would confuse users.
• IP and training leakage: a trajectory can reveal internal prompts, synthetic data or the character of post-training.
• Monitorability: paradoxically, raw CoT is valuable to the provider. OpenAI showed that a monitor over CoT can detect reward hacking; strong direct optimisation against “bad thoughts”, however, teaches the model to hide its intent. The lab therefore wants to keep the channel as a sensitive safety signal.

These are legitimate reasons, not a complete win. Hiding CoT reduces auditability for the customer and concentrates trust in the provider. A summary doesn’t solve this: the summariser selects, shortens and filters. It may be factually useful, but it must not be stored as evidence that the model actually proceeded in the way described. An audit of a production system should rest on inputs, tool calls, sources, outputs, usage and the verifier’s result.

How a model learns to reason: from demonstrations to verifiable reward

A pre-trained Transformer can generate procedures because it has seen them in text, but it has no reason to spend thousands of tokens checking a single result. Supervised fine-tuning (SFT) on high-quality reasoning traces teaches format, decomposition and readability. It has a ceiling, though: the model imitates the length and style of the demonstration, including errors, and human CoT is expensive.

Reasoning post-training therefore uses reinforcement learning with verifiable reward. In maths the final number can be compared, in code tests can be run, in a formal proof a checker. The model samples whole trajectories; those that end correctly receive a reward. The gradient does not dictate the sentence “now perform the substitution”, but increases the probability of all decisions on successful trajectories. Longer decomposition, backtracking, self-checking or a change of strategy can emerge because they help the result.

DeepSeek-R1-Zero: a strong experiment, not proof of human thinking

DeepSeek applied the Group Relative Policy Optimization method directly to the base model DeepSeek-V3-Base, without prior SFT on procedure demonstrations. During RL, R1-Zero started generating longer answers, checking its own steps and returning to previous decisions. The reward evaluated correctness and format; the authors deliberately avoided a neural reward model that could be gamed.

The result solved tasks well but repeated itself, mixed languages and was hard to read. The production R1 therefore added high-quality initial CoT demonstrations, selection of successful samples and further SFT and RL phases focused on helpfulness and safety.

More important than the “aha moment” in the demo is the selection mechanism. RL found trajectories that earn reward. That may be a genuine correction, but also a shortcut, an exploit of the checker or a style that looks like reasoning. Distillation then transfers successful traces into smaller Qwen/Llama models; it doesn’t necessarily teach them the same internal algorithm, it teaches them to reproduce the distribution of the teacher’s solutions.

The key distinction is between outcome reward and process reward. The first variant evaluates only the end. It is cheap, but it can reward a wrong procedure that happened to end correctly. A Process Reward Model (PRM), by contrast, evaluates individual steps.

In the study Let’s Verify Step by Step, OpenAI showed better solution selection using a PRM on the MATH dataset. The reported 78.2%, however, applies to specific models and a subset of the data, not as general proof that process evaluation always wins.

Inference compute is not a single technique

Once training is complete, additional compute can be used in four ways. Their costs are not interchangeable. Eight thousand tokens generated in sequence have a markedly higher response time than eight parallel candidates of a thousand tokens each, even though their total count is the same.

Snell et al. showed on the MATH dataset that the right ratio of sequential revisions to parallel candidates depends on task difficulty. An adaptive policy nearly matched best-of-N with a quarter of the compute budget. In one regime a smaller model with an adaptive budget even outperformed a roughly 14× larger model.

The condition was that the smaller model already had a real chance of solving the problem. If the task is beyond its abilities, more samples only produce more different errors.

When more compute really helps

We see the greatest benefit on tasks with several dependent steps whose result can be cheaply verified: in maths, code with tests, formal proofs or planning with hard constraints. The model can try several paths and a test or other verifier separates the correct solution from one that merely seems convincing. In long planning with tools, part of the improvement moreover comes from newly obtained data, not a longer internal monologue.

The same study found on SWE-bench Lite with DeepSeek-Coder-V2-Instruct that coverage grew from 15.9% with one sample to 56% with 250 samples. There, candidates can be run against tests, so coverage is more practically usable. On MATH, majority vote and reward models saturated after hundreds of samples: the generator occasionally produced the correct solution, the selector failed to find it in a flood of plausible errors. This is the most important law of inference scaling: generating optionality is not enough, you need a discriminative signal.

The s1 project showed that the length of the process can also be controlled at runtime. The Qwen2.5-32B-Instruct model, fine-tuned on only 1,000 carefully selected procedures, used budget forcing: when the model tried to finish, the runtime suppressed the end token and appended the word “Wait”. The AIME24 score rose from 50 to 57%.

This is a demonstration of a controllable budget, not a general recommendation. A forcibly extended CoT can also break a finished solution, and hosted APIs usually don’t allow controlling internal end tokens anyway.

When longer reasoning is just expensive noise

Longer reasoning by the model won’t supply a library’s release date or the content of your document. For factual questions the right solution is to load the source or use search. For classification, extraction into a schema, simple translation and latency-sensitive chat, a cheaper model without extended reasoning is often enough. Precise computation is better run as code. And for a subjective preference with no correct answer, the best-of-N method has no reliable judge.

Overthinking is not just a needlessly long answer. The model may first find the correct result, then continue, raise a dubious objection and abandon the original conclusion. The longer the process, the more opportunities arise for error and subsequent rationalisation.

The study Inverse Scaling in Test-Time Compute (Anthropic, 2025) prepared targeted tasks in which longer reasoning reduced performance. It doesn’t say how often this happens in normal operation, but it refutes the notion that more tokens can’t hurt.

A weaker or unverifiable task has one more problem: the verifier may be worse than the generator. An LLM judge prefers longer, more confident and stylistically familiar answers; a reward model can be optimised against its blind spots. Hence the order of trust: an executable test or formal constraint, citation verification against the source, an independent specialised verifier, and only lastly the same LLM asked to “check yourself”. Verbosity is not accuracy and critique is not proof.

Faithfulness of explanation: CoT is not a record of internal reasoning

Even a visible CoT is not a direct dump of the model’s internal activations. It is more generated text. The model may reach the correct result via a wrong explanation, or conversely spoil a correct procedure in the final rewrite. Performance says whether the answer came out right. Faithfulness says whether the described procedure actually corresponds to the factors that influenced the result. These are two different properties.

Turpin et al. inserted hidden hints into prompts, for example few-shot choices biased towards answer A. GPT-3.5 and Claude 1 often followed this hint but did not acknowledge it in the CoT. Instead they produced a convincing justification of the wrong choice; accuracy on 13 BIG-Bench Hard tasks dropped by up to 36%.

A newer Anthropic study found a similar phenomenon in Claude 3.7 Sonnet and DeepSeek-R1. When the hint actually influenced the answer, Claude mentioned it in only 25% and R1 in 39% of cases. The described CoT therefore often omits a factor that had a measurable influence on the result.

This doesn’t mean CoT monitoring is worthless. The provider can find reward hacking, an intent to bypass a test or giving up on the task in internal traces before it shows in the final answer. It means that the absence of bad intent in the CoT proves nothing. A summary is one more step removed from the causal process. In production use it as a navigational explanation, never as compliance evidence.

Research into latent reasoning shows three distinct ways to add compute without a long visible CoT:

• Quiet-STaR: creates internal textual intermediate steps before predicting difficult tokens.
• Coconut: works directly with latent vectors and doesn’t convert every step into words.
• Pause tokens: add computation steps without meaningful text.

These approaches prove that more compute need not have a human-readable form. They do not prove, however, that a specific proprietary model uses them, nor that the latent state explains the result more faithfully. Auditability may in fact be worse.

Economics and latency: hidden text is not free

Working tokens are produced one after another after the prompt is processed. If the model produces 6,000 hidden tokens before a 300-token answer, the GPU performs roughly 6,300 decoding steps. The visible answer doesn’t start streaming until the internal phase finishes, so the first token may arrive after tens of seconds. The prompt cache can be reused, but it doesn’t reduce the cost of working tokens the model has already generated.

To calculate cost use the usage data from the API response and the current price list: cost = input × input_rate + (visible_output + reasoning) × output_rate. If a request consumes 1,000 input, 4,000 working and 500 visible output tokens, the output rate is applied to 4,500 tokens, not 500.

If the output rate is P per million tokens, this part costs 0.0045 × P. The formula is more useful than a specific amount because prices can change.

With parallel best-of-N, consumption grows almost N-fold, but total time can stay close to the slowest branch if you have enough capacity. In practice it is increased by API limits, queues and slow outlier requests. A long serial CoT, on the other hand, holds one decoding slot for a longer time.

Measure at least p50/p95 latency, input, output and working tokens, tool costs, verifier success rate and cost per verified-correct task.

A practical decision framework: compute budget by task

1. Determine the task type. Is it fact retrieval, transformation, extraction, computation, or genuine multi-step deduction? Start with the cheap baseline.
2. Measure quality, cost and latency together. Accuracy alone hides an expensive small improvement; cost alone hides repeated attempts after an error.
3. Route requests adaptively. none/minimal for classification and extraction, low for ordinary chat, medium for limited decomposition and high only for hard code, maths and high-value planning.
4. For facts, load the source. Longer introspection won’t fill in missing or current knowledge; use RAG or search.
5. Hand precise computation to a tool. The model should formulate the procedure and interpret the result, not emulate a calculator with thousands of tokens.
6. Add an independent verifier. A test suite, JSON Schema, a database constraint, a SAT solver or citation checking are stronger than the prompt “check yourself”.
7. Set limits and a fallback path. Token and time budgets, a maximum number of tool calls, a circuit breaker and escalation to a human belong in the algorithm design.
8. Don’t request or log raw CoT. Store the model, compute level, tool calls, final output, usage, latency and verification result. A reasoning summary is not proof of correctness.

Implementation: a router over the OpenAI Responses API

The Responses API accepts the reasoning.effort parameter and an optional summary; specific values depend on the model. The following example first determines the task type and complexity. It tries a higher compute level only once and only when the external verifier rejects the previous answer. The application doesn’t request raw CoT and doesn’t store the reasoning summary in telemetry.

from dataclasses import dataclass
from time import perf_counter
from openai import OpenAI

client = OpenAI()

@dataclass(frozen=True)
class TaskPolicy:
    effort: str
    needs_search: bool
    max_latency_s: float

def route_task(kind: str, complexity: int) -> TaskPolicy:
    if kind in {"extract", "classify", "translate"}:
        return TaskPolicy("none", False, 4.0)
    if kind == "factual":
        return TaskPolicy("low", True, 8.0)
    if kind in {"math", "code", "planning"} and complexity >= 7:
        return TaskPolicy("high", False, 45.0)
    return TaskPolicy("medium", False, 20.0)

def solve(prompt: str, policy: TaskPolicy, verify) -> str:
    tools = [{"type": "web_search"}] if policy.needs_search else []
    attempts = [policy.effort]
    if policy.effort not in {"high", "xhigh"}:
        attempts.append("high")

    for effort in attempts:
        started = perf_counter()
        response = client.responses.create(
            model="gpt-5.5",
            input=prompt,
            reasoning={"effort": effort, "summary": "auto"},
            tools=tools,
        )
        latency_s = perf_counter() - started
        answer = response.output_text

        usage = response.usage
        telemetry = {
            "model": response.model,
            "effort": effort,
            "latency_s": latency_s,
            "input_tokens": usage.input_tokens,
            "output_tokens": usage.output_tokens,
            "verified": False,
        }

        ok = verify(answer)
        telemetry["verified"] = ok
        emit_metrics(telemetry)  # Never log raw CoT or reasoning summary.
        if ok:
            return answer
        if latency_s > policy.max_latency_s:
            break

    raise RuntimeError("No verified answer within budget")

In a production application verify must be domain-specific: tests for a patch, schema and business rules for extraction, citations against retrieved documents for factual QA. Escalating effort after every failure only makes sense when the verifier provides a new signal. Without it, a retry is another ticket from the same distribution. Verify model names and supported effort levels against the documentation when upgrading; the API surface is not universal across providers.

Conclusion: the compute budget is a resource, not a sign of intelligence

A reasoning model usually generates additional working steps before answering and consumes a variable compute budget on them. This is real GPU work, but not proof of human introspection. The content of the working tokens may remain hidden and their summary is not proof that the model proceeded in exactly the described way. Moreover, about proprietary models we know only what their makers publish.

Scaling inference compute works best when the model already has a real chance of finding the solution and the system has a strong verifier. Without it, it is not enough that the correct candidate appears somewhere among the samples — the system must also be able to recognise it.

For simple or knowledge tasks a longer CoT may add only cost, latency and new errors. The practical rule is therefore: start with the cheap baseline, add compute according to difficulty, load facts from sources, run computations as tools and verify correctness outside the model’s own text.

About the author

Zdeněk Vais

Founder of Turing Institut · Senior Principal Software Engineer, Oracle

Zdeněk has gone from programmer through team leader to software architect. At Oracle he builds cloud services for media and at Turing Institut he connects software development with practical use of AI.

Meet our team →

← Back to the blog