Turkish Needle-in-a-Haystack

1. gemma-4-31B-it shows no needle-count limit. From N = 1 to N = 419 it binds 21,227 of 21,252 needles (99.88%), with no downward trend. Excluding the two needle keys that are ordinary Turkish words, 99.98%. The 419-city key pool ran out before the model did.
2. Every Gemma-4 model beats every Gemma-3 model on city-bound retrieval. Gemma-3 models find the right numbers but attach them to the wrong cities — gemma-3-4b-it loses 106 needles that value-only scoring would have credited, 96 of them written against another requested city; no Gemma-4 model does that more than 2 times.
3. Qwen matches or beats the Gemma-4 model of similar size. Qwen3.8-27B binds all 420 needles, tying gemma-4-31B-it as the only perfect models. The Qwen3.6-35B-A3B mixture of experts (3B active) binds 99.8% against gemma-4-26B-A4B-it's 90.0% (4B active); Qwen3.5-9B 97.9% against gemma-4-12B-it's 92.4%; Qwen3.5-4B 95.2% against gemma-4-E4B-it's 83.6%; Qwen3.5-2B 96.9% against gemma-4-E2B-it's 97.1%. Of Qwen3.5-4B's 20 misses, 9 are the right number next to a Turkish or misspelled city name (BEYRUT for Beirut).
4. Needle spacing, not needle count, predicts failure. gemma-4-12B-it misses 1.61% of needles packed within 1,024 tokens of each other and 9.68% when they sit further apart (6.0×). Needle count alone has no significant effect (ρ = -0.109, p = 0.10). Raising N at a fixed context length packs needles closer and makes the task easier.
5. Needle keys must be screened against the haystack's language. Kabul is the everyday Turkish word for “acceptance” and occurs 86 times in the corpus; it caused 20 of the 31B's 25 misses (28.6% miss rate vs 0.014% for every other city). Asking gemma-4-31B-it about all 419 keys flagged 6 as Turkish words: Amman, Bari, Havana, Kabul, Kazan, Nice.

Serving: vLLM 0.29.0 on a single NVIDIA A100-SXM4-80GB, temperature 0, bfloat16 (gemma-4-31B-it at 256K uses fp8 weights — the only way it fits). Qwen models were served text-only (--language-model-only) with thinking disabled (chat_template_kwargs: {"enable_thinking": false}): the Gemma models answer without reasoning, and the answer budget has no room for it. Haystack: 218 Turkish Wikipedia articles (trwiki-67), 200,210 Gemma tokens, extended to 238 articles for the 256K runs with the first 218 byte-identical. Grid: 15 context lengths, evenly spaced from 1,000 to 127,000 tokens (or 261,000). Cells too small to hold their needles plus 500 tokens of haystack are skipped (·). Context length is counted in each model's own tokens: Qwen's tokenizer averages 3.24 characters per token on this corpus against Gemma's 3.42, so a Qwen prompt of the same length holds about 5% less text.

Scoring: every multi-needle answer is re-scored from the stored response. A needle is retrieved only if its number appears on a line naming its own city; the parser accepts any reasonable line format and folds Turkish diacritics.

Needle and question (Turkish, as sent)

{city} şehrinin özel sihirli sayısı: {rnd_number}
Aşağıdaki şehirlerin özel sihirli sayıları nedir: {cities}? Her şehir için bulduğun sayıyı listele.

“The special magic number of the city of {city}: {number}” — “What are the special magic numbers of the following cities: {cities}? List the number you found for each city.”

Full prompt

Kullanıcıların sorularını yanıtlayan yardımcı bir yapay zeka botusunuz.
Aşağıda bir bağlam ve bağlamla ilgili bir soru bulunmaktadır.
#BAĞLAM
{context}
#BAĞLAM SONU

#SORU
{question} Belge dışından bilgi vermeyin. Yanıtını tam olarak şu biçimde ver:
SEHIR: <şehir adı> = <sayı>
(her şehir için bir satır, bulamadığın şehir için <sayı> yerine YOK yaz)
Başka hiçbir şey yazma.

“You are a helpful AI bot that answers users' questions. Below is a context and a question about it. … Do not give information from outside the document. Answer exactly in this format: CITY: <city> = <number> (one line per city; write YOK instead of the number for a city you cannot find). Write nothing else.”

Benchmark version

All results shown use 3-digit values, a single fixed needle sentence, and city keys that were not screened against Turkish vocabulary (see Known limitations).

• Needle keys were not screened. The key pool includes Kabul, the everyday Turkish word for “acceptance”; its misses are counted, not excluded, in every table.
• Qwen models answered with thinking disabled. That matches how the Gemma models answer, so the comparison is like for like, but it is not Qwen's default mode; with reasoning on they may score differently. Their context lengths are in Qwen tokens, which cover about 5% less text than Gemma tokens.
• One run per cell, no repeated seeds. Repeating one condition (gemma-4-12B-it, N = 40, span 100%) gave 6/14 vs 4/14 perfect cells, so treat single-cell differences as noise.
• gemma-4-31B-it at 256K is fp8, not bf16.