How Well Does AI Speak Georgian?

Abstract

Georgian, a Kartvelian language with 3.76 million native speakers ^[16], is served by the AI industry in its second tier: excluded from the leading multilingual knowledge benchmarks ^[6][7], supported by translation services only in their reduced-feature classes ^[17], and never before measured systematically across the skills a learner or user actually needs. We introduce the Georgian AI Benchmark: 230 original test items across nine dimensions, from translation and idiom through verb conjugation, case morphology, official romanization, and factual grammar honesty. Eight large language models and Google Translate were evaluated under a two-layer design: a deterministic layer scored by exact match against grammar-book-verified answer keys, and a translation layer scored by a three-judge model panel under the MQM error taxonomy ^[9][10] with published bias controls. Three statistical tiers emerged. Claude Fable 5 and Gemini 3.1 Pro tie at the top (99.4 of 100, with Gemini 3.5 Flash close behind), a frontier pack follows (Claude Opus 4.8, GPT-5.4, Grok 4.3, GPT-5.5, at 94-96), and the budget tier trails far behind (Claude Haiku 4.5, 81.9). Reading Georgian is close to solved at the frontier, and on the core skill set the top systems approach ceiling. So we added a frontier set: fifty production tasks drawn from rare, corpus-attested morphology in real sentence contexts. There the field spreads across 54 points, from 90 down to 36, no system approaches ceiling, and the ordering shifts: GPT-5.5 climbs past Gemini 3.1 Pro, and the mid-tier flagships drop to between 48 and 60. Alongside that, most models still cannot render Georgia's official romanization and three systems fabricated the same grammar fact with full confidence. The complete dataset, methodology, outputs, judgments, and scoring code are published under CC-BY on this page, and the benchmark re-runs as models change.

1. Introduction

Roughly four million people speak Georgian, and the census of 2024 counted 133,857 foreign citizens permanently resident in a country where daily life runs in a language no mainstream learning app teaches ^[19]. When those residents turn to AI for help, they have no way to know whether the model in front of them handles Georgian like it handles German, or like a confident beginner.

The measurement gap is real. Global-MMLU covers 42 languages without Georgian ^[6]; MMLU-ProX covers 29, also without ^[7]. The standard machine-translation test set, FLORES, does include Georgian, but its test sentences are fully public and peer-reviewed audits have found them in the training data of most open models tested ^[4][5], which makes scores on it unreliable as evidence. Georgian-specific evaluation exists but is narrow: GeoLogicQA measures multiple-choice reasoning over 100 items ^[1], and Gallagher and Heyer probe case alignment with treebank-derived syntactic tests, finding the ergative case to be a systematic weak point ^[2]. No public evaluation covers the range of skills that decide whether AI is actually useful in and around this language.

We built that evaluation. Three design goals governed everything: items no model could have memorized, scoring that does not depend on any competitor's opinion for the majority of the grade, and total reproducibility. Section 2 describes how; Section 3 reports what we found; Section 4 discusses what it means for anyone using AI with Georgian; Section 5 states the limitations plainly, including the ones specific to who built this.

2. Methodology

2.1 Test set

The benchmark contains 230 original items across nine dimensions, all authored in July 2026 specifically for this benchmark and never published before the run. No sentence comes from FLORES, Tatoeba, textbooks, Wikipedia, or our own published course content. Topics rotate across daily life, bureaucracy, food, travel, work, and family, and proper nouns are invented, so no item can be answered from memorized text.

A ninth block, the frontier set, contains 50 additional production items built from the Georgian National Corpus ^[20]: real sentences containing rare, corpus-attested morphology (pluperfects and perfect subjunctives, Series III inversion, polypersonal object chains, syncope genitives, old plurals, preverb alternations), with the target form blanked and the sentence perturbed (all names and places swapped for invented same-declension substitutes, plus a synonym hedge in recognizable passages) for both contamination control and source-licensing reasons. The frontier set reports as its own score and never enters the composite, so composites stay comparable across editions while the frontier layer preserves headroom above the current state of the art. Because its contexts derive from a licensed corpus, the frontier set publishes a smaller public slice (15 of 50 items) with the remainder held privately under published hashes.

147 core items are published in full in the dataset download. 83 core items are held private, with their SHA-256 hashes published, so future editions can detect models that memorize the public set; the hashes guarantee integrity from publication forward, and public-versus-private score divergence is reported every edition as a memorization check. Every published file carries canary strings, following the convention major benchmarks use to keep test data out of training corpora ^[8].

2.2 Ground truth

Every deterministic answer key was authored against printed reference grammars: Hewitt's learner and reference grammars ^[13] and Makharoblidze's The Georgian Verb ^[14], with each verb and noun form carrying a source citation in the dataset, cross-checked against the lingua.ge conjugation dictionary ^[15]. An independent model pass (Gemini 3.1 Pro) then reviewed every Georgian string for correctness and naturalness, with disputes resolved against the printed grammars, never by any model's opinion. Each item carries a list of accepted variant forms, built during verification, so legitimate alternatives are never marked wrong. Finally, every morphology answer key was triangulated against the Georgian National Corpus, the 200-million-token academically annotated corpus of Georgian ^[20]: 69 of 90 forms are attested in real Georgian text with a corpus analysis matching our claimed grammatical reading (verified through constrained corpus queries, published per item), the 20 cloze keys are attestation-verified with their grammatical rules corpus-checkable by construction, and the single inconclusive case is documented rather than hidden. No answer key required correction. Rare correct forms with thin corpus attestation are flagged, never dropped, because dropping them would quietly bias the benchmark toward frequent forms. Because that checker model is also a contestant, we quantify its influence: the verification pass corrected 2 references and added 39 accepted variants across 17 items, and re-scoring without every one of those variants changes no ranking.

2.3 Systems and collection

Eight language models were tested through their production APIs at temperature 0 where the API permits it (three current Anthropic and OpenAI flagships accept only their default temperature, recorded per response in the logs): Claude Fable 5, Claude Opus 4.8, Claude Haiku 4.5, Gemini 3.1 Pro, Gemini 3.5 Flash, GPT-5.5, GPT-5.4, and Grok 4.3, plus Google Cloud Translation's LLM tier on the translation dimensions. Model identifiers, timestamps, parameters, and raw responses are all in the dataset. Every item was asked once, with retries only on transport errors, under fixed prompts published verbatim.

2.4 Scoring

Deterministic layer (130 items, 55 percent of the composite). Responses are normalized and matched exactly against the reference and accepted variants. Because models sometimes ignore "output only the form" and add commentary, a published extraction ladder recovers the answered form, and instruction compliance is tracked as its own statistic rather than mixed into the grammar score. Every extraction-ladder credit granted from a verbose response was individually audited, and the audit artifact is published. Refusals score zero and are counted separately.

Judged layer (100 items, 45 percent). Translation quality is scored by a three-judge panel: GPT-5.5, Gemini 3.1 Pro, and Claude Fable 5, each marking error spans under the MQM taxonomy ^[9] in the reference-anchored style of the GEMBA-MQM line of work ^[10][11]. Judges see the source, the verified reference with accepted variants, and an anonymized candidate. Severity penalties follow the MQM Core weights (minor 1, major 5, critical 10), an item's score is the median across judges, and judge disagreement is published: Krippendorff's alpha runs 0.86 to 0.99 by dimension, alongside the share of unanimous perfect scores, and the judges' system rankings correlate at 0.92 to 0.98.

Composite. The Georgian AI Score is 0.55 times the mean of the six deterministic dimension scores plus 0.45 times the mean of the three judged dimension scores. Every composite carries a 95 percent confidence interval from a 10,000-resample item-paired bootstrap (seed published), every adjacent pair a p-value, and tie groups are maximal runs in which every within-group pair is statistically indistinguishable. Rankings are unchanged under every published sensitivity: layer weights shifted ten points either way, item-weighted aggregation, and exclusion of each judge's scores of its own maker's models.

2.5 Integrity process

Every bug found during the run is in the versioned changelog on this page. The consequential one: our first collection ran Claude through a local CLI tool that leaked context into the model and degraded its outputs, which we detected in a scheduled audit, disclosed, and repaired by re-collecting every Claude response through the clean API path. The contaminated data is quarantined, not deleted. We flag this prominently because the correction moved our own operating model's score up, and the only honest defense against that appearance is the full trail.

3. Results

3.1 Overall

Three statistical tiers emerged, and within each tier every pairwise difference sits inside the error bars. Claude Fable 5 (99.4, CI 98.6-99.9) and Gemini 3.1 Pro (99.4, CI 98.7-99.9) tie at the top with Gemini 3.5 Flash (98.7) statistically alongside them. The frontier pack follows: Claude Opus 4.8 (96.2), GPT-5.4 (95.1), Grok 4.3 (94.7), GPT-5.5 (94.3), all mutually tied. Claude Haiku 4.5 stands alone at the bottom (81.9). The one near-boundary pair is Gemini Flash versus Claude Opus at p equals 0.067. Google Translate's LLM tier, competing only on translation, scored 99.7 there.

3.2 Per-dimension results

The composite compresses; the dimensions discriminate. Three patterns stand out.

Verb morphology separates the field. Claude Opus 4.8 leads at 97.5, four systems cluster at 92.5 to 95, GPT-5.4 and Grok trail slightly, and Haiku collapses to 57.5. The failures are the classic learner traps: nominative subjects where the aorist demands the ergative, მაქვს for animate possession where Georgian requires მყავს, dative objects after Series II verbs where the language demands the nominative, and placement verbs that ignore whether the object stands or lies. Our results corroborate, with a production-side method, the case-alignment weakness Gallagher and Heyer found with syntactic probes ^[2].

Romanization is where most AI still fails Georgian. The official National 2002 system marks ejective consonants with an apostrophe, a real phonemic contrast ^[12]. Two systems render it perfectly (Gemini 3.1 Pro and Claude Fable 5 at 100), Gemini Flash near-perfectly (95), Grok well (85). Then the floor drops: Claude Opus 75, GPT-5.4 60, GPT-5.5 55, Haiku 35, with systematic apostrophe-dropping as the dominant failure. AI-generated learning content that romanizes Georgian in ways no textbook does is not a coincidence; it is the default behavior of most current models.

Translation is near ceiling for every system. Every model's overall translation score cleared 96, and every model cleared 95 in each translation category. Counting only items where all three judges agreed the output was error-free, the hardest colloquial tier still separates systems, from above 85 percent error-free at the top to the low 40s for the budget tier. Reading Georgian is a solved problem at the top of the market and an unsolved one at the bottom.

3.3 Grammar honesty

After a published adjudication pass in which every borderline response was individually reviewed, five systems scored a clean 100. The failures were instructive. Claude Haiku asserted that Georgian has five grammatical cases in both of two independent runs, once inventing a case called the locative; Georgian has had seven cases in every reference grammar for decades ^[13]. And three different systems, GPT-5.5, Claude Opus, and Claude Haiku, gave the same wrong answer to the same question, writing the romanization of წ as ts rather than ts'. Errors are rare, but they arrive with full confidence, in exactly the places a beginner cannot check.

3.4 The frontier set: where the ceiling breaks

On the fifty rare-morphology production items, no system approaches its core-set level, and the field finally spreads. Claude Fable 5 leads at 90, with GPT-5.5 at 88, Gemini 3.1 Pro at 86 and Gemini 3.5 Flash at 84 in one statistical band. Then the cliff: Claude Opus 4.8 at 60, GPT-5.4 at 56, Grok 4.3 at 48, Claude Haiku 4.5 at 36. Two things stand out. First, the composite's top tier survives contact with genuinely hard Georgian, but at 84 to 90 rather than 99, which is exactly the headroom a recurring tracker needs. Second, GPT-5.5 moves ahead of Gemini 3.1 Pro here, the reverse of small differences elsewhere, suggesting the frontier set measures something the core set saturates: sustained morphological reasoning under rare paradigms rather than recall of common forms. Confidence intervals at fifty items are wide and the per-item data is published; we present these as bands, not a ranking.

4. Discussion

4.1 How to read scores this high

A composite of 99.4 does not mean Georgian is solved, and Georgians who use these models daily are right when they sense imperfection. The composite measures short, well-posed core tasks, which frontier models are genuinely excellent at, and the frontier set now shows what happens beyond them: the best system on Earth scores 90, not 100, on rare Georgian morphology in real contexts, and most flagships score between 36 and 60. What no part of this edition measures is long-form production, discourse-level naturalness, sustained conversation, and speech, which require native-speaker evaluation at scale and remain the next layer of this project. The right reading: the top tier handles curated core Georgian at near-human accuracy, holds up impressively but imperfectly on rare morphology, and below the top tier the hard tail of the language is still unmistakably unsolved.

4.2 What this means in practice

For choosing a model: Gemini 3.1 Pro and Claude Fable 5 are the safest choices for Georgian, and Gemini 3.5 Flash delivers top-tier Georgian at budget price, outscoring every OpenAI and xAI flagship here (against GPT-5.4, p equals 0.006). Which family you pick matters more than how much you pay. For using any model: trust AI for comprehension, verify anything it produces, and treat romanized output with suspicion unless you are using a system that scored well on that dimension. For builders of language products: the budget models most apps route to by default scored 35 to 57.5 on the dimensions Georgian learners depend on, and that routing decision is invisible to users.

4.3 The industry picture

Every mainstream translation service now covers Georgian, and none covers it fully. DeepL added Georgian in November 2025 in its reduced-feature tier ^[17]; Azure excludes it from custom translation and its newest LLM tier ^[18]; Google's own classic and LLM translation tiers both cover it, and the LLM tier's 99.7 here shows the coverage is real. Coverage arrived. Parity did not.

5. Limitations

This benchmark was designed and operated by Claude Fable 5, and that model tied for first place. The deterministic 55 percent of the score is immune to authorship familiarity, since answers are verified against printed grammars and accepted-variant lists; removing Claude's judge from Claude's translation scores shifts them by less than 0.1 points; and the full correction trail, including the episode where our own tooling hurt Claude's scores, is in the changelog. But a residual familiarity advantage on the judged translation items cannot be fully excluded, and future editions rotate item authorship. Further limitations: 230 items is modest and the confidence intervals say so; no native speaker validated judgments before this release (the validation layer is built and scheduled for the next edition); text only, single-turn only; and scores are a July 2026 snapshot of systems that change monthly, which is why the benchmark is versioned and re-run.

6. Conclusion

Georgian finally has a measurement. The 2026 edition establishes three facts with published evidence: frontier AI reads Georgian reliably, producing correct Georgian still separates the market sharply, and the official romanization plus confident grammar fabrication remain live failure modes in systems millions of people trust. The next edition adds native-speaker validation, speech, Georgian-tuned open models, and a refreshed frontier block, with this edition's frontier scores as the baseline future models get measured against. The dataset, code, and every judgment are downloadable below under CC-BY; corrections from Georgian speakers reach us at [email protected] and are versioned publicly. Native speakers are the authority we optimize for.

Authorship and roles

Published by EasyGeorgian (Lasse N., Tamar N.). Benchmark design, item authoring, operation, and analysis by Claude Fable 5 under the constraints described in Sections 2 and 5; Georgian reference verification by Gemini 3.1 Pro against printed grammars; native-speaker review layer scheduled for the next edition. We state the division of labor because a benchmark's trustworthiness begins with who did what.

References

1. Koberidze, Elizbarashvili, Tsintsadze (2025). GeoLogicQA. 1st Workshop on Advancing NLP for Low-Resource Languages, RANLP.
2. Gallagher, Heyer (2026). Targeted Syntactic Evaluation of Language Models on Georgian Case Alignment. arXiv:2602.10661. arxiv.org
3. NLLB Team. FLORES-200 / FLORES+. huggingface.co
4. Contamination Report for Multilingual Benchmarks (2024). arXiv:2410.16186. arxiv.org
5. When Flores Blooms Wrong: target-side contamination in MT benchmarks (2026). EACL. arXiv:2601.20858. arxiv.org
6. Singh et al. Global-MMLU. huggingface.co
7. MMLU-ProX (2025). EMNLP.
8. BIG-bench canary convention. github.com
9. Lommel et al. (2014). Multidimensional Quality Metrics; current typology at themqm.org
10. Kocmi, Federmann (2023). GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4. WMT.
11. Fernandes et al. (2023). The Devil is in the Errors (AutoMQM). WMT.
12. BGN/PCGN (2009). Romanization of Georgian: 2011 national system. gov.uk
13. Hewitt, B. G. (1995). Georgian: A Learner's Grammar (Routledge); Georgian: A Structural Reference Grammar (Benjamins).
14. Makharoblidze, T. The Georgian Verb.
15. lingua.ge. Georgian verb conjugation dictionary. lingua.ge
16. Ethnologue (2024). Georgian, 27th edition.
17. DeepL. Supported languages. developers.deepl.com
18. Microsoft. Azure AI Translator language support. learn.microsoft.com
19. Geostat (2026). 2024 Population Census final results. geostat.ge
20. Georgian National Corpus (GNC). University of Bergen CLARINO. clarino.uib.no