The U-curve in LLMs: why models get lost in the middle of the context
The context windows of today’s models measure hundreds of thousands to millions of tokens, so it is tempting to “dump everything” into the prompt. But models do not read context evenly. If you plot the model’s accuracy against the position of the key information in the prompt, you get a characteristic U-curve: the beginning and the end work great, the middle slumps. This phenomenon is called lost in the middle and it has direct consequences for anyone building RAG, writing longer prompts or running agents with long runs.
The short version
A long context window is not a long working memory. The model does receive all the tokens, but when answering it uses the beginning and the end the most; that is why you must order critical information in the prompt and in RAG results by position, not just by relevance.
What exactly the experiments showed
The phenomenon was first measured systematically by Liu et al. in the study Lost in the Middle: How Language Models Use Long Contexts (2023). They used two tasks. In the first — multi-document QA — the model received a question and 10 to 30 Wikipedia documents, exactly one of which contained the answer; the rest were topically similar but irrelevant “distractors”. The researchers then simply shifted the position of the relevant document and measured accuracy. The second task — key-value retrieval — was even rawer: the model received JSON with hundreds of random key–value pairs and had to return the value for a given key. No semantics, pure lookup in the context.
The most important results can be summarised in four points:
- Accuracy was highest when the relevant document stood at the very beginning (the so-called primacy bias) or at the very end of the context (recency bias).
- With the document in the middle, GPT-3.5-Turbo’s accuracy dropped by more than 20 percentage points — from ~75% to ~54%.
- In the worst case, a model with 20–30 documents in the context was worse than if it had received none and answered only from trained knowledge (closed-book: 56%).
- The drop also affected models explicitly trained for long context — a longer window does not mean an even window.
A longer context window therefore does not mean an evenly used window. The model “sees” the information but does not give it enough weight when generating the answer.
The phenomenon has since been confirmed repeatedly across models, tasks and languages. Thanks to targeted long-context training, newer frontier models have a much milder drop and often shine on a simple “needle in a haystack” test. But a synthetic needle is the easiest variant of the task: benchmarks such as RULER show that as soon as information needs not only to be found but also aggregated or reasoned over in several steps, the effective context length tends to be much shorter than the advertised one. And with smaller and open-source models, which you typically deploy when self-hosting, you still have to count on the U-curve.
Why the U-curve arises
An intuition from psychology suggests itself: people also remember the first and last items of a list best. This is called the serial position effect and was described by Hermann Ebbinghaus back in the 19th century. The similarity, however, is more of an amusing coincidence — in transformers the U-curve has concrete architectural causes, and once you understand them it stops being a mystery and becomes a design constraint you can work with.
1. The causal mask favours the beginning
In an autoregressive decoder, each token “sees” only the tokens before it. The first tokens of the context are therefore within the attention reach of all subsequent tokens: every layer, every generation step can refer to them, and their influence thus accumulates across the whole depth of the network. The later a token stands in the context, the fewer “paths” lead to it.
2. Attention sink: the first token as an anchor
A related and well-measured phenomenon is the attention sink (Xiao et al., 2023): trained models direct a non-negligible part of their attention to the very first tokens of the sequence regardless of their content. The softmax in attention must distribute weights so that they sum to 1 — and when an attention head “has nothing to say” at the moment, it parks the surplus at the beginning. The first positions thus act as an anchor that the model simply keeps watching. In practice this means that content at the beginning of the prompt has a systematically higher chance of influencing the output.
3. Positional encoding favours the end
Rotary position embeddings (RoPE), used by most of today’s models, dampen the attention contribution as the distance between query and key grows. Tokens at the end of the context are closest to the generated text, so the model “reaches” them most strongly — hence the recency bias. Training adds to this: when predicting the next token, the nearest surroundings are statistically the most useful, so the model learns to trust the end of the context.
4. The middle gets the worst of both worlds
A token in the middle of a long context is neither an anchor from the beginning nor close to the end: it has the maximum average distance from both edges and the least privileged position in the causal graph. The more recent theoretical work Lost in the Middle at Birth (2026) even shows that the U-curve is a geometric property of a causal decoder with residual connections already at random weight initialisation — that is, before any training. The causal mask guarantees primacy, the residual connections recency; training and positional encoding only modulate the shape of the curve. So it is not a “bug” that the next generation of models will simply fix, but a structural property of the architecture.
Measure it yourself
You can write a simplified version of the experiment (needle in a haystack) in an evening. Just insert the “needle” at different depths of a long text and measure when the model finds it:
import random
from openai import OpenAI
client = OpenAI()
NEEDLE = "Tajny kod projektu Vcela je 7412."
FILLER = open("dlouhy_text.txt", encoding="utf-8").read()
def test_pozice(hloubka): # hloubka: 0.0 = zacatek, 1.0 = konec
pozice = int(len(FILLER) * hloubka)
kontext = FILLER[:pozice] + "\n" + NEEDLE + "\n" + FILLER[pozice:]
odpoved = client.chat.completions.create(
model="gpt-4.1-mini",
messages=[{"role": "user",
"content": kontext + "\n\nJaky je tajny kod projektu Vcela?"}],
).choices[0].message.content
return "7412" in odpoved
for hloubka in [0.0, 0.25, 0.5, 0.75, 1.0]:
uspechy = sum(test_pozice(hloubka) for pokus in range(10))
print(f"hloubka {hloubka:.2f}: {uspechy}/10")
With large models on a short context, everything will probably score 10/10 — it gets interesting with tens of thousands of tokens, with smaller models and with harder variants of the task. If you want to go further, try three variations that make the test considerably tougher:
- Paraphrase instead of exact match — phrase the needle differently from the question so that it is not just string matching.
- Several needles at once — the model has to find and combine two or three pieces of information from different depths.
- Distractors — add similar but incorrect “project codes” to the text; that is exactly how a more realistic picture of RAG behaviour emerges.
What to do about it in practice
Several rules follow from the U-curve that are worth following even when you are not working with an extremely long context:
Key things at the edges
System instructions and essential facts belong at the beginning of the prompt, the current question and the final instruction at the end. The middle is the weakest spot — that is where supporting material belongs. For long prompts, the “sandwich” works: state the critical instruction at the beginning and repeat it at the end. It sounds cheap, but it relies precisely on the primacy and recency bias.
In RAG, order by position, not just by score
When the retriever returns ten chunks, don’t assemble them into the prompt in the order they arrived — the best chunk would end up first, the second best right after it, and the rest would fill the rest of the context, so the second half of the results would drown exactly in the dead middle. The proven approach is the so-called reordering into a U shape: the most relevant documents at the beginning and the end, the weakest in the middle. LangChain has a ready-made LongContextReorder for this, but the logic is trivial even by hand:
def preskladat_do_u(dokumenty):
"""Dokumenty serazene podle relevance preusporada tak,
aby nejlepsi byly na krajich a nejslabsi uprostred."""
liche = dokumenty[::2] # 1., 3., 5., ...
sude = dokumenty[1::2] # 2., 4., 6., ...
return liche + sude[::-1]
# skore: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
# poradi: [1, 3, 5, 7, 9, 10, 8, 6, 4, 2]
Less is more
Every irrelevant document in the context enlarges the “dead middle” and dilutes attention. Incidentally, the Liu et al. study also showed that adding more documents improves the result only up to a certain point and then starts to hurt. More aggressive filtering and reranking (e.g. with a cross-encoder or an LLM-based reranker) usually helps more than a bigger context. This also applies to agents: tool results that are no longer needed should be summarised or moved out of the conversation.
Test, don’t trust
Behaviour differs from model to model and from version to version. If you are building a product on long context, measure your own U-curve on your own data and repeat the measurement every time you change the model. An hour invested in a test script will save you weeks of mysterious quality debugging.
Model-side mitigations
If you run the model yourself, there are also interventions in inference: the positional attention calibration described in the paper Found in the Middle (2024) separates attention caused by position from attention caused by relevance and demonstrably flattens the U-curve; the related approach of attention sorting (Peysakhovich & Lerer, 2023) iteratively reorders documents by how much attention they actually receive. But these are techniques for operators of their own inference — an API user can make do with the rules above.
Want to go deeper?
We cover attention, positional encoding and working with context in the AI for Developers, Science and Data Analytics course. And if you are building RAG or agents, take a look at AI Agent Developer.
Sources and further reading
- Liu et al.: Lost in the Middle: How Language Models Use Long Contexts (2023)
- Xiao et al.: Efficient Streaming Language Models with Attention Sinks (2023)
- Su et al.: RoFormer: Enhanced Transformer with Rotary Position Embedding (2021)
- Peysakhovich, Lerer: Attention Sorting Combats Recency Bias In Long Context Language Models (2023)
- Hsieh et al.: Found in the Middle: Calibrating Positional Attention Bias (2024)
- Hsieh et al.: RULER: What’s the Real Context Size of Your Long-Context Language Models? (2024)
- Lost in the Middle at Birth: An Exact Theory of Transformer Position Bias (2026)
