Skip to main content

The whole machine

How it works

Ask most AI models a question in Igala and they answer in Yoruba or English instead. So Igala speakers are teaching the models their own language: they write answers, compare model answers without knowing which model wrote what, and fix mistakes by hand. This page shows the whole thing, in plain words, including the parts we got wrong.

Every number on this page is fetched from the project database while you read. Nothing is typed in by hand, so nothing can quietly go stale. And when the live feed is down, the page says so instead of showing old numbers as current.

loading
answers written by speakers
loading
blind comparisons judged
loading
Igala-English sentence pairs
loading
dictionary entries served
loading
community annotators
loading
questions in the frozen exam

Counts exclude demo sessions; the annotator count also excludes seed test accounts. Computed from the live project database, loading.

One picture

The whole system

Four layers, read top to bottom. The community produces the knowledge. The knowledge is packed around each question. A model answers. And every answer goes back to the community for judgment, which becomes new knowledge. The loop on the left is the whole idea.

The Igala community

Answers by speakers

some written cold, most written after rejecting a model's attempt

Blind comparisons

which of two answers is better

Corrections + reasons

what was wrong, fixed by a speaker

The knowledge, growing every session

Community answers

question-answer pairs

how the community actually writes

Dictionary

word-meaning entries

four sources: words lined up from the Bible, a community wordlist, an 1854 wordlist, and Wiktionary

Example sentences

Igala-English pairs

how sentences are built

Grammar rules

read out of all the evidence

four rules stand on two sources; the rest are the community's own spelling choices

the rules travel as instructions, not as pasted text

Answering one question, packed fresh every time

Question

from a person

Retrieval

best-matching pieces

leak guard

The packed prompt

  • 1 the rules: how to use everything below
  • 2 real answers by speakers: what good sounds like
  • 3 corrections speakers made to earlier answers
  • 4 example sentences: how Igala is built
  • 5 dictionary lines for this question's words
  • 6 the question, plus one closing rule

Model

any of them

Answer

in Igala

the leak guard checks every retrieved piece, so no exam question is ever handed its own answer

Judgment, the only measure that finally counts

Native judgment

blind pairs + corrections on the models in the pool

collecting now

what speakers fix becomes tomorrow's rules

Frozen exam

questions no model retrieves on

scored leak-free only

Agreement Score

100 = two native speakers' agreement with each other

bars past 100 are partly built in; see below

Every judgment and every correction re-enters the knowledge, the grammar, and the next round of models. The community is not labeling data for a system. The community IS the system.

Counts in the picture are fetched live, from the same computation as the numbers above. The red funnel is the leak guard: every retrieved piece is checked, so no exam question is ever handed its own answer. That check is the reason the scores below can be believed.

The story so far

Six versions of the same idea

None of these models learns Igala the way a person does. Each version changes what real Igala the model gets to see at the moment it answers, and how it is told to use it. Each fix exposed the next problem.

  1. v0 - plain models

    Ask a top model, nothing added

    What it fixed

    Nothing yet. This is the baseline.

    What it did not

    Asked for Igala, the models answer in Yoruba or English, with invented words in between.

  2. v1 - show it real answers

    Community answers pasted into the prompt

    What it fixed

    Real Igala words start appearing.

    What it did not

    Words without sentence structure. A community reviewer put it plainly: the first sentence is saying three different things.

  3. v2 - give it a method

    A dictionary, example sentences, and a step-by-step procedure

    What it fixed

    Correct spellings, and sentence shapes copied from real ones.

    What it did not

    Still copying, not speaking.

  4. v3 - teach it the grammar

    Everything in v2, plus grammar rules read out of the evidence

    What it fixed

    Pronouns, negation and word order arrive as rules. Speakers prefer this version to the plain model when judging blind.

    What it did not

    The same rules made Claude worse, not better. And on the exam, v3 is not measurably ahead of the plain model.

  5. v4 - translate the meaning

    The instructions rewritten around one rule from the community's review

    What it fixed

    Whole-sentence meaning instead of word by word. Speakers' own corrections are now packed into the prompt too.

    What it did not

    Long, cultural, open-ended questions still collapse. No speaker has judged this version yet.

  6. v4.1 - the rules the failures taught us

    v4 plus rules mined from every judged failure, and a repair round

    What it fixed

    Undoes the damage v3 did to Claude. Beats v3 on the exam in a paired test, the only step between versions that does.

    What it did not

    The gain over v4 is mostly fewer tone marks, which the community rarely writes. No speaker has judged this version yet.

Zooming in

How one answer is built today

When someone asks a question, the system packs a bundle around it, in this order, and sends the whole bundle to the model. Today's version is v4.1. It uses the same bundle as v4 and changes only the rules on top, so any difference between the two comes from the rules and from the repair round described below.

One more thing happens first. On exam questions, every piece passes the leak guard: if a piece contains that question's own community answer, it is dropped and the drop is recorded. Otherwise the test would hand the model its answer key.

  1. The rules (system prompt)

    A numbered procedure telling the model how to use everything below: the dictionary for word forms, the examples for sentence shape, and, since v4, one rule above all others: translate the meaning of the whole sentence, never word by word.

  2. Real answers by speakers

    passes the leak guard

    Question-and-answer pairs written by Igala speakers, shown as example exchanges. The model sees what a good answer looks and sounds like: short, in Igala, spelled the community's way.

  3. Corrections speakers made

    passes the leak guard

    Since v4: a few cases where a model wrote something, a speaker fixed it, and the speaker said why. The model sees the mistake and the fix side by side, which teaches more than the fix alone.

  4. Example sentences (Igala-English pairs)

    passes the leak guard

    Sentence pairs that show how Igala sentences are built. Served only when the question asks the model to build something, a sentence or a story or a greeting, because word-lookup questions were measurably hurt by them.

  5. Dictionary lines for this question's words

    passes the leak guard

    One line per meaningful word of the question, with the exact attested Igala form. Placed right above the question because in Igala spelling is meaning: a changed letter is a different word, not a typo.

  6. The question

    The person's actual question, unchanged.

  7. One closing rule, then a repair round

    A single line under the question restating the output rule: answer in Igala only, nothing else. Then, on v4.1 only, a check of the finished answer. If it uses letters Igala does not have, or is crowded with tone marks the community would not write, the model is asked once to rewrite it. So a v4.1 answer can be the model's second attempt, and its exam score is scored that way.

No secrets

The exact instructions the models receive

These are the real system prompts, word for word, not a summary. Publishing them is deliberate: a community should be able to read every instruction issued in its language's name. Each one opens with a click.

This site is a static page and cannot run the annotation app's code, so the prompts below are byte-for-byte copies taken from the serving code on 2026-09-01, each labelled with its source file and commit. If a serving prompt changes after that date, the copy here lags until the site is rebuilt; the app itself always shows the version actually in service.

System prompt, v2 (the method)web/src/lib/generation-prompt-v2.ts at commit d1cec50cb417 (2026-08-12) - copied 2026-09-01click to read the full text
You are a fluent native speaker of Igala, the Yoruboid language of Kogi State, Nigeria, and you answer entirely in Igala.

THE METHOD
1. Find each content word of the question in the DICTIONARY provided with it, and use those exact Igala forms.
2. Copy sentence SHAPE from the EXAMPLES: build your sentences the way theirs are built. One thought per sentence. Short clauses. Do not pack three ideas into one sentence.
3. If a word is missing from the dictionary, choose the closest attested form from the examples. Never invent a spelling. Never substitute a Yoruba word.
4. Spelling is meaning-bearing in Igala. A nasal ending, a dotted vowel (ẹ, ọ), or a prefix changes the word into a different word. Copy attested spellings exactly, character for character.

ORTHOGRAPHY
Igala has seven vowels: a e ẹ i o ọ u. The dotted vowels ẹ and ọ are separate letters and are required wherever the attested form has them. Tone marks are optional, but be consistent: mark tone the way the dictionary and examples mark it. Never write ṣ, ị, ụ or ṅ - those letters do not exist in Igala. The digraphs ch, gb, gw, kp, kw and the nasals ñ, ñm, ñw are real Igala; write them as attested.

NEVER WRITE
These are Yoruba, not Igala: ati; ṣe or se for 'do'; nitori; okpa, eje or igbe as numerals; wọn; aya; egbon; aburo; alaafia; ma binu; ejoo; o dabọ.

OUTPUT
Give the answer only. No preamble, no meta-commentary, no translation unless the question explicitly asks for one.

The v3 prompt is the same skeleton plus the grammar. Four of its rules stand on two independent sources; the others describe how the community spells, which only the community's own writing can confirm.

System prompt, v3 (the method + the grammar)web/src/lib/generation-prompt-v3.ts at commit 77866f2e4e7a (2026-08-13) - copied 2026-09-01click to read the full text
You are a fluent native speaker of Igala, the Yoruboid language of Kogi State, Nigeria, and you answer entirely in Igala.

THE METHOD
1. Find each content word of the question in the DICTIONARY provided with it; use those exact Igala forms.
2. Copy sentence SHAPE from the EXAMPLES: one thought per sentence, short clauses, built the way theirs are.
3. If a word is missing from the dictionary, use the closest attested form from the examples. Never invent a spelling or substitute a Yoruba word.
4. Spelling is meaning: a nasal ending, a dotted vowel (ẹ, ọ) or a prefix makes a different word. Copy attested spellings character for character.

CLOSED-CLASS GRAMMAR
Order: Subject-Verb-Object; after the noun: possessor + numeral + determiner.
Pronouns subj|obj|poss: I u|mi|mi; you ẹ|ẹ|wẹ; he/she i|u|-wn; we a|wa|wa; you-pl mẹ|mẹ|mẹ; they ma|ma|ma. Preverbal u = I, postverbal u = him/her.
Tense: bare verb = completed; preverbal á = not yet complete (is-doing AND will-do). Copulas: chi/chẹ = is (equals), de = is at / here is.
Negation: ONLY a clause-final nasal (ǹ/-n); prohibition: subject + kì + verb ... ǹ. A nasal added for any other reason makes a different word.
The = lẹ AFTER the noun; lẹ also closes relative clauses (head + kì ... lẹ); never yí. Relativizer kì (singular), ku before plural ma/me.
Elision: vowel meets vowel across a word break -> drop the FIRST vowel, apostrophe at the joint (w'ọla, k'ọla, aj'ẹñwu). Never add or strip a word-initial vowel.
Numerals follow the noun with mẹ- (mẹji two, mẹta three); one is bare ka; ordinals take ẹkẹ-. Plural àmì/abọ ONLY for people and animals; landscape nouns repeat; others unmarked. du = take one, kó = take many.
Joining: kpai links nouns; oñ or a new sentence links clauses; tọdu = because; ichẹñwu = if.

REGISTER
Write like the community, not scripture: ~7-word sentences, dotted vowels, apostrophized elision, sparse or no tone marks, first/second person, negative nasal attached (-n). Never Bible forms like Jihofa or taku.

ORTHOGRAPHY
Seven vowels: a e ẹ i o ọ u; ẹ and ọ are separate letters, required where attested. Mark tone as the dictionary and examples do. Never write ṣ, ị, ụ or ṅ - not Igala letters. Digraphs ch, gb, gw, kp, kw and nasals ñ, ñm, ñw are real Igala; write as attested.

NEVER WRITE
These are Yoruba, not Igala: ati; ṣe or se for 'do'; nitori; okpa, eje or igbe as numerals; wọn; aya; egbon; aburo; alaafia; ma binu; ejoo; o dabọ.

OUTPUT
Give the answer only. No preamble, no meta-commentary, no translation unless the question explicitly asks for one.

The v4 prompt rewrote the method around one instruction from the community's review: translate what the sentence means, never one word at a time, and leave out what the examples leave out.

System prompt, v4 (meaning first)web/src/lib/generation-prompt-v4.ts at commit 12dd0866dce1 (2026-08-29) - copied 2026-09-01click to read the full text
You are a fluent native speaker of Igala, the Yoruboid language of Kogi State, Nigeria, and you answer entirely in Igala.

THE METHOD
1. Understand what the question MEANS before you write. Translate the thought, never word by word - word-for-word Igala is not Igala.
2. Build your sentences the way the EXAMPLES build theirs: one thought per sentence, short clauses, their word order.
3. Use the DICTIONARY for the words your ANSWER needs, in those exact Igala forms.
4. Not every English word has an Igala word: leave out what the examples leave out. Where no Igala word exists, describe the thing in plain attested words or keep the everyday loanword the community uses - never coin an Igala-looking form.
5. If a word is missing from the dictionary, use the closest attested form from the examples. Never invent a spelling or substitute a Yoruba word.
6. Spelling is meaning: a nasal ending, a dotted vowel (ẹ, ọ) or a prefix makes a different word. Copy attested spellings character for character.

CLOSED-CLASS GRAMMAR
Order: Subject-Verb-Object; after the noun: possessor + numeral + determiner.
Pronouns subj|obj|poss: I u|mi|mi; you ẹ|ẹ|wẹ; he/she i|u|-wn; we a|wa|wa; you-pl mẹ|mẹ|mẹ; they ma|ma|ma. Preverbal u = I, postverbal u = him/her.
Tense: bare verb = completed; preverbal á = not yet complete (is-doing AND will-do). Copulas: chi/chẹ = is (equals), de = is at / here is.
Negation: ONLY a clause-final nasal (ǹ/-n); prohibition: subject + kì + verb ... ǹ. A nasal added for any other reason makes a different word.
The = lẹ AFTER the noun; lẹ also closes relative clauses (head + kì ... lẹ); never yí. Relativizer kì (singular), ku before plural ma/me.
Elision: vowel meets vowel across a word break -> drop the FIRST vowel, apostrophe at the joint (w'ọla, k'ọla, aj'ẹñwu). Never add or strip a word-initial vowel.
Numerals follow the noun with mẹ- (mẹji two, mẹta three); one is bare ka; ordinals take ẹkẹ-. Plural àmì/abọ ONLY for people and animals; landscape nouns repeat; others unmarked. du = take one, kó = take many.
Dates: day and month are ordinals after the noun, ẹkẹ- + numeral (the month: ọchu + ẹkẹ-numeral); write the year in digits - Igala has no attested year-name, never compose one.
Joining: kpai links nouns; oñ or a new sentence links clauses; tọdu = because; ichẹñwu = if.
Every small word must have a job. If you cannot say what a particle or nasal (lẹ, á, kì, ku, kpai, oñ, ǹ) is doing in your sentence, remove it - an idle one changes the meaning.

REGISTER
Write like the community, not scripture: ~7-word sentences, dotted vowels, apostrophized elision, sparse or no tone marks, first/second person, negative nasal attached (-n). Never Bible forms like Jihofa or taku.

ORTHOGRAPHY
Seven vowels: a e ẹ i o ọ u; ẹ and ọ are separate letters, required where attested. Mark tone as the dictionary and examples do. Never write ṣ, ị, ụ or ṅ - not Igala letters. Digraphs ch, gb, gw, kp, kw and nasals ñ, ñm, ñw are real Igala; write as attested.

NEVER WRITE
These are Yoruba, not Igala: ati; ṣe or se for 'do'; nitori; okpa, eje or igbe as numerals; wọn; aya; egbon; aburo; alaafia; ma binu; ejoo; o dabọ.
These are Igbo, not Igala: the market-day names Orie and Nkwọ.

OUTPUT
Give the answer only. No preamble, no meta-commentary, no translation unless the question explicitly asks for one.

The v4.1 prompt is v4 plus what the judged failures taught us: rules for serial verbs, wishes, the dative, and a few small words; an instruction to perform a greeting rather than describe one; a rule against inventing dialect facts; and a list of the invented words this model kept producing. This is the version serving today.

System prompt, v4.1 (the method + the mined rules)web/src/lib/generation-prompt-v4-1.ts at commit a01755a9dc55 (2026-08-31) - copied 2026-09-01click to read the full text
You are a fluent native speaker of Igala, the Yoruboid language of Kogi State, Nigeria, and you answer entirely in Igala.

THE METHOD
1. Understand what the question MEANS before you write. Translate the thought, never word by word - word-for-word Igala is not Igala.
2. Build your sentences the way the EXAMPLES build theirs: one thought per sentence, short clauses, their word order.
3. Use the DICTIONARY for the words your ANSWER needs, in those exact Igala forms.
4. Not every English word has an Igala word: leave out what the examples leave out. Where no Igala word exists, describe the thing in plain attested words or keep the everyday loanword the community uses - never coin an Igala-looking form.
5. If a word is missing from the dictionary, use the closest attested form from the examples. Never invent a spelling or substitute a Yoruba word.
6. Spelling is meaning: a nasal ending, a dotted vowel (ẹ, ọ) or a prefix makes a different word. Copy attested spellings character for character.
7. When the question asks how someone would say something, write the words they would SPEAK, in their voice - never a description of them speaking.
8. Never assert which town or area uses a form unless your reference material says so - saying you do not know is correct.

CLOSED-CLASS GRAMMAR
Order: Subject-Verb-Object; after the noun: possessor + numeral + determiner.
Pronouns subj|obj|poss: I u|mi|mi; you ẹ|ẹ|wẹ; he/she i|u|-wn; we a|wa|wa; you-pl mẹ|mẹ|mẹ; they ma|ma|ma. Preverbal u = I, postverbal u = him/her.
Tense: bare verb = completed; preverbal á = not yet complete (is-doing AND will-do). Copulas: chi/chẹ = is (equals), de = is at / here is.
Negation: ONLY a clause-final nasal (ǹ/-n); prohibition: subject + kì + verb ... ǹ. A nasal added for any other reason makes a different word. Subject + kì + verb WITHOUT the nasal is a wish or blessing - the 'may God ...' frame.
The = lẹ AFTER the noun; lẹ also closes relative clauses (head + kì ... lẹ); never yí. Relativizer kì (singular), ku before plural ma/me.
Elision: vowel meets vowel across a word break -> drop the FIRST vowel, apostrophe at the joint (w'ọla, k'ọla, aj'ẹñwu). Never add or strip a word-initial vowel. 'to/for' is ñwu before a consonant, ñw' before a vowel - never nwi or plain nw.
Numerals follow the noun with mẹ- (mẹji two, mẹta three); one is bare ka; ordinals take ẹkẹ-. Plural àmì/abọ ONLY for people and animals; landscape nouns repeat; others unmarked. du = take one, kó = take many.
Dates: day and month are ordinals after the noun, ẹkẹ- + numeral (the month: ọchu + ẹkẹ-numeral); write the year in digits - Igala has no attested year-name, never compose one.
Joining: kpai links nouns; oñ or a new sentence links clauses; tọdu = because; ichẹñwu = if. muda = but (rather) - contrast only, never 'must'.
Verbs chain with kẹ: one action then another is V kẹ V. kẹ links verbs; kì/ki starts a new clause - never swap them.
Igala has no hyphenated prefixes - never é- or any vowel + hyphen fused to a word. The incompletive is the standalone word á; a noun keeps its own first vowel inside the word.
Every small word must have a job. If you cannot say what a particle or nasal (lẹ, á, kì, ku, kpai, oñ, ǹ) is doing in your sentence, remove it - an idle one changes the meaning.

REGISTER
Write like the community, not scripture: ~7-word sentences, dotted vowels, apostrophized elision, no tone marks unless the question asks for them, first/second person, negative nasal attached (-n). Never Bible forms like Jihofa or taku; never end a word in -wñ.

ORTHOGRAPHY
Seven vowels: a e ẹ i o ọ u; ẹ and ọ are separate letters, required where attested. Mark tone as the dictionary and examples do. Igala words use ONLY these letters: a b ch d e ẹ f g gb gw i j k kp kw l m n ñ ñm ñw nw ny o ọ p r t u w y, plus the apostrophe and tone accents - any other letter (ṣ, č, ị, ụ, x, q, v, z...) is not Igala: if a word seems to need one, the word is wrong. Digraphs ch, gb, gw, kp, kw and nasals ñ, ñm, ñw are real Igala; write as attested.

NEVER WRITE
These are Yoruba, not Igala: ati; ṣe or se for 'do'; nitori; okpa, eje or igbe as numerals; wọn; aya; egbon; aburo; alaafia; ma binu; ejoo; o dabọ. Also Yoruba: ra for 'buy'; Yoruba's 'money' word; fún or f'; iyawo; ẹgbẹ; tutu; any alaafia shape (lafia, ọlafia).
These are Igbo, not Igala: the market-day names Orie and Nkwọ.
Nobody's words - never write them again: ádṣa, kpùkẹ̀, ojoji, teketeke, akeli, gbede, abẹki, mímí, efí, kpegwa.

OUTPUT
Give the answer only. No preamble, no meta-commentary, no translation unless the question explicitly asks for one.

And the one line appended below every question:

The closing ruleweb/src/lib/generation-prompt-v2.ts at commit d1cec50cb417 (2026-08-12) - copied 2026-09-01click to read the full text
Answer in Igala only. Give the answer itself, nothing else.

The human verdict

Out of every ten questions

This is the result we trust. Igala speakers read two answers to the same question without knowing which model wrote which, and pick the better one, call it a draw, or reject both. Below is what they decided, ten questions at a time. Each panel is one pair they compared: our package against the plain model, or two versions of our package head to head. Each row is one round of judging, split where the questions or the versions under test changed; inside a panel the same two versions sit in every row, so the change from row to row is the questions and the judges, not the model.

loading…

Why not a bar for every version: speakers can only judge the versions in the blind test. That was the v3 package alone until Oct 8, when version 4.4 joined it, and Oct 10, when version 4.5 joined against version 4.4 and the plain model; a version gets its own panel here once speakers have judged it. Versions 4 through 4.3 have only sat the exam below. Version to version, the exam is the ladder we have; this section is the verdict we trust. We also thought about charting how often each answer was left uncorrected. That would have misled: speakers can only correct the answer they chose, so the answer that wins more gets corrected more (every one of the 180 corrections so far sits on a chosen answer or a draw, none on a loser). A draw also covers the judgments where a speaker recorded no verdict.

Honest measurement

The scoreboard: Community Agreement Score

Every model takes the same exam: a set of frozen questions that no model retrieves community answers for. Each answer is compared with what Igala speakers wrote for the same question. A longer bar means closer to how the community actually writes. The yardstick is this one community's writing, on Igala questions only. A high bar here claims nothing beyond Igala, and a bar past the 100 line is partly built in, which the notes below explain.

Fetching the scoreboard from the project database. Values on this chart are only ever shown live.

What the score means. Underneath is chrF, the standard overlap score used to grade machine translation: 0 to 100 for how much an answer's letters and letter-pairs overlap with the community's answers, measured on the answer itself so an English preamble cannot inflate it. We then rescale it so that the agreement between two native speakers reads exactly 100. A score of 85 means: this model's answers are 85% as close to the community's writing as one speaker's answers are to another's.

A worked example. Say a question asks for one word, and two speakers wrote the same five letters with one accent mark different. Their overlap is high but not perfect, and that speaker-to-speaker overlap is what the 100 line is anchored to. A model that shares four of those five letters lands near the line. A model that answers in English shares almost nothing and lands near zero.

Why a bar could pass 100, what we fixed, and what it revealed. We first wrote that a bar past 100 meant the model matched the community more closely than one speaker matches another. That was mostly built in, and it has now been fixed: a model is scored against the same answers a held-out speaker is scored against, so the two numbers are finally comparable. The old figure is kept beside the new one so you can see what moved. But fixing it exposed something larger. We built a control that takes a plain model's answers and simply deletes the tone marks, with no method involved, and it scores higher than every system we have built. Roughly three quarters of the community's exam answers carry no tone marks, so a measure built on letter overlap rewards a model for leaving them off. The number below is therefore a measure of resemblance, and part of that resemblance is punctuation habits rather than language. That is why the blind judgments of native speakers, not this chart, are what the project treats as the result.

Why it is measured on the leak-free subset, and who wrote the answers. During an early audit, some frozen questions were found to have had one of their own community answers slip into the material served to the models. On those questions a high score measures copying, not competence. So the score uses only the questions where that never happened, and its 100 line is computed with one answer per speaker: someone re-submitting their own answer does not count as two people agreeing. One more thing we got wrong: we called the exam answers cold, written before seeing any model. More than half were written by a speaker after seeing and rejecting a model's attempt. That is still a native speaker's answer, but it is not untouched by the models, and a version of the score using only the cold answers is on the way.

Why we do not call it "% fluent". The score measures resemblance to how the community writes. Only native judgment measures fluency. In blind comparisons to date, speakers have most often found both answers inadequate. That rate has fallen since August, but mostly because the models being compared changed, not because the method improved: the early comparisons were between weaker models. So these bars chart progress from does-not-speak-Igala toward speaks-it-badly, and the speakers judge the rest.

Open questions

What is being tested now

The blind test now holds four systems.

Every blind judgment on the strong systems to date is drawn in the panels above. It began as one pairing: Gemini with our v3 package against the same Gemini with nothing added. Then v4.4 joined against both, and v4.5 joined against v4.4 and the plain model, starting with a new set of questions aimed at what the model still gets wrong.

Two grammar questions only speakers can settle.

Lining up the Bible sentence pairs surfaced number words with a me- prefix (meji, meta) beside the dictionary's plain forms (eji, eta). And three separate corrections closed a yes-or-no question with a small final word our deduced grammar says does not exist. Both are plausible. Neither is confirmed, so neither goes into the rules until the linguists and the speakers have looked.

Pending unlocks.

A score that cannot pass 100 by construction, and a version of it that ignores tone marks, so that a gain in tone-mark habits cannot pass for a gain in grammar. Written permission on file for every text source we use: today only one is, and the Bible corpus, our largest, is not. A published Igala dictionary of roughly 5,000 headwords, several times what the system serves from today. And a collaborating linguist's write-up of Igala sentence structure, to turn the speakers' corrections into rules with a second source behind them.

The record

What changed, when

The dates are fixed history: what each day added and what it corrected. Every live number they produced is recomputed above, not repeated here.

  1. Aug 9, 2026

    The automatic eval harness, the honest human ceiling, and the leak guard. The audit that day found the benchmark had served 15+ of its 43 frozen questions their own community answers - those scores measured copying, so every number since is reported on the leak-free subset.

  2. Aug 12, 2026

    The Bible parallel corpus - 30,907 Igala-English sentence pairs from the Bible Society of Nigeria's Igala Bible - plus a 2,104-entry lexicon, powering retrieval v2 and THE METHOD. Corrected Sep 1: this entry said the pairs were ingested under BSN permission. Our records hold two written requests to the Society and no reply, so no permission is on file. What that means for the corpus is an open item recorded in the Sep 1 entry.

  3. Aug 13, 2026

    The frontier arms joined the board. Gemini 3.1 Pro topped it untouched; Claude Opus 5 gained +22 from community retrieval - a clean read on knowledge versus skill. This page was made public and the cost ledger rebuilt.

  4. Aug 14, 2026

    A working grammar deduced from all the evidence (tasks/igala-grammar-deduced.md) and METHOD v3, which enshrines only its A- and B-grade rules in the system prompt.

  5. Aug 17, 2026

    The benchmark visual and the Community Agreement Score: leak-free stripped chrF rescaled so the deduplicated native-speaker ceiling reads 100, drawn LLM-benchmark style with confidence whiskers. The raw chrF table moved under the chart; nothing was removed and no score is capped.

  6. Aug 29, 2026

    Global Recordings Network signed a copyright agreement (Aug 27) covering their “Words of Life” Igala recording, and the audio (45:38, the only usable Igala speech asset) was acquired, along with six Bible-for-Children booklets as raw assets; the booklets' fonts silently strip the ẹ/ọ subdots on extraction, so nothing from them may enter the corpus until that is solved. Outreach to other rights holders (the JWAL papers, Egbunu's proverbs study, PanLex) is in progress, with a call with the JWAL author scheduled; none of their text enters the corpus before written permission is on file, so the corpus counters above are unchanged.

  7. Aug 31, 2026

    METHOD v4 and v4.1. v4 rewrote the instructions around one rule from the community's review: translate the meaning of the whole sentence, never word by word. v4.1 added eight grammar rules mined from 132 judged failures, a step that tells the model to perform a greeting rather than describe one, a rule against inventing dialect facts, and a repair round: when an answer uses letters Igala does not have or is saturated with tone marks, the model is asked once to rewrite it. On the frozen exam Gemini v4.1 scored 120 and v4 102; Claude v4.1 scored 93 against 55 for Claude v3, so the rules that had hurt Claude at v3 no longer do. Nine grammar entries were added to the knowledge store, but the v4 retrieval path does not read them, so they contribute nothing to these scores.

  8. Sep 1, 2026

    An adversarial audit of every public number, run against the live database. What it corrected. A bar past the 100 line is mostly built in: a model is scored against every community answer for a question while a speaker is scored against the other speakers only. Scored like-for-like the best system sits at about 103, level with the speakers, and the score is being replaced by one that cannot pass 100 by construction. The v4 to v4.1 gain is mostly fewer tone marks, which the community rarely writes; with tone marks ignored the two versions are level. The sentence "the grammar lifts Gemini measurably" was not supported and has been removed. The blind preference belongs to one pairing only, Gemini with the v3 package against the same Gemini with nothing added: 54 to 14, with ties and double rejections counted separately, or 25 to 7 when each question is counted once. No speaker has yet judged v4 or v4.1. The fall in "both answers inadequate" from 99% to about half came with a change of models, not from the method. 131 of the 238 exam answers were written after a speaker saw and rejected a model's attempt, so "written before seeing any model" was wrong for more than half of them. And the Bible corpus had been described as used under a BSN permission that our records do not contain. What held: speakers prefer the v3 package to nothing, every one of the six annotators; v4.1 undid the regression v3 caused for Claude; and v4.1 beats v3 on the exam in a paired test, the only step between Gemini versions that does.

  9. Sep 3, 2026

    The score was rebuilt so that a model is judged the way a speaker is judged: against the same answers, with the same one left out. The old construction stays beside it, marked deprecated, so the change is visible rather than silent. Then a control settled what the score has actually been measuring. Taking bare Gemini's answers and deleting the tone marks, with no model and no method involved at all, scores higher than every real system we have built. The community writes tone marks on about a quarter of its answers, so a measure built on letter overlap rewards leaving them off. Read plainly: the scoreboard has been ranking tone-mark habits as much as Igala. A tone-insensitive column now sits beside the main one, and it presses every system into a narrow band with the order scrambled. The repair round, which asks a model to rewrite an answer that breaks the spelling rules, turns out to be worth little to Gemini and a great deal to Claude. None of this touches the human result: speakers judging blind still prefer the v3 package to the plain model.

  10. Sep 13, 2026

    Names are not translated. Reviewing a live Wikipedia translation on Sep 1, Igala speakers judged the grammar good and found the failures were elsewhere: the model was respelling people, places and schools because Igala has no s, and it dropped a fact it had no Igala word for. The cause was partly ours - a serving check meant to catch invented words was flagging the copied names as illegal letters and asking the model to rewrite them. Serving v4.2 copies names letter for letter and never drops a fact for want of a word. 124 new questions went to the annotators, aimed at what is still open: the direction words the speakers themselves disagreed about, year expressions, borrowed words, and how Igala reads in an encyclopedia.

  11. Sep 23, 2026

    The speakers' verdict is now on this page, ten questions at a time: for each round of blind judging, how many of every ten questions went to our package, to the plain model, to a draw, or to neither. Two rounds so far, on different question batches, with the same v3 package in both, so the rows show the questions changing, not the model. A chart of how often an answer was left uncorrected was considered and dropped, because speakers can only correct the answer they chose, so the answer that wins more is corrected more. Salem's copy of Ejeba's 2023 paper on Igala concord was read line by line: 178 rules, 43 already in the prompt, 24 in abstract notes, the rest nowhere we served. Version 4.3 adds a grammar block to the question, up to three rules retrieved for the question at hand, carrying the forms the paper gives (du and kó with wa for bring, tinyo and rinyo for throw away, abọ for people only) and what the annotators taught us this month (efẹwọ before a town, lo ti before a destination, dates with the day as a plain number). Version 4.4 keeps the block and amends twelve prompt lines from the same corrections. Versions 4.3 and 4.4 sat the frozen exam the same day; their rows are in the table above, and neither has been judged by speakers yet.

  12. Oct 8, 2026

    Version 4.4 joined the blind test: from today speakers judge it against the plain model and against the v3 package, on the questions still in their queues, so the next round measures one version against the next directly. The human verdict above opens a third round at that moment, and now draws every pair the speakers have judged, computing its words (ahead, level, behind) from the counts. Three corrections: the scoreboard had labelled versions 4.2 to 4.4 as retrieval v1 since they were registered; the fill script would have written a pooled 4.x version's answers with the wrong prompt; and the repair round's checker, which re-asks the model when an answer breaks a writing rule, had been misreading quoted questions (a quote mark stuck to a name, and a request for English switched the letters check off for the whole answer). All three are fixed and tested, and every stored answer the old checker had wrongly re-asked was replayed through the new one.

  13. Oct 9, 2026

    The speakers' queues had run dry, so 108 new questions went in, built from what the speakers have told us: every judgment, every word they corrected, their own answers, and Salem and Lydia's write-up on Igala grammar. Each question aims at one thing the model still gets wrong (the linker ki, set phrases such as good night, number agreement, money, loan words, one form of each word in an answer), with control questions where the rule must not fire. Version 4.5 sat the frozen exam: 122.7 against 105.3 for version 4.4, but 94.8 against 94.7 once tone marks are set aside, so the gain is that it now leaves tone marks off, as the speakers do; its words are no better than version 4.4's on this exam. It is not in the blind test. On the chat page, a version 4.5 answer can now show a second rendering under it, every word toned and written out in full, as Salem and Lydia asked; that rendering is not stored and not scored.

  14. Oct 10, 2026

    Version 4.5 joined the blind test: speakers now judge it against version 4.4 and against the plain model, starting with the 108 new questions, in the round the verdict opened on Oct 9. It was meant to join on Oct 9; the job that makes the switch stopped with the machine it ran on, and the switch was made the next morning. The platform where speakers work moved to app.wikitongues.org, and the old address still works. This page is now the only telling of how the project works: the platform's own copy of it redirects here.