The 10,000 most common words in spoken Georgian

Abstract

Every published Georgian word-frequency list is built from written text: news reporting, encyclopedic prose, official documents. That is the Georgian that exists in bulk, so that is the Georgian that gets counted. The consequence is a systematic distortion of what a learner is told to study first. We introduce the first openly licensed frequency list of colloquial Georgian, built from two registers that sit close to speech: film and television subtitle dialogue, and conversational forum writing. After decontamination the corpus contains 68 subtitle documents (268,785 tokens) and 184,209 forum posts (5,454,689 tokens). Word forms were lemmatized with a large-language-model pipeline whose accuracy we measure and report (96.5% on blind re-annotation), and ranked by the mean of each register's Zipf value. The resulting divergence from written Georgian is large and systematic: shemidzlia ("I can") ranks 30th in speech and 9,599th in writing; albat ("probably") ranks 130th against 9,603rd; while op'ozitsia ("opposition") and saarchevno ("electoral") sit inside the written top 150 and fall to 6,798th and 6,114th in speech. Only 54.3% of our spoken top-1,000 appears in the written top-1,000, though 90.4% appears somewhere in the written 10,000 — a disagreement about rank, not about vocabulary. Separately, we document that the de-facto standard Georgian frequency list in circulation is not Georgian: it is the Bulgarian frequency list, byte-mangled into Georgian glyphs. We solve the substitution from the data and show that its top nine entries decode to the top nine Bulgarian words in rank order. The complete dataset, the methodology including our own errors, and the code are published under CC-BY on this page.

1. Why this list did not exist

A frequency list is the backbone of vocabulary instruction. It decides what a learner meets in week one and what they meet in year two. For most languages this is uncontroversial, because large corpora of many registers exist and can be balanced against each other.

Georgian has no such luxury. The text available in bulk is written, formal, and topical: newspapers, government publications, Wikipedia. A frequency list computed over that material is a perfectly accurate description of written Georgian. The problem is that it is then handed to learners as a description of Georgian.

The distortion is not subtle. In the standard written list, the vocabulary of parliamentary reporting occupies the positions a learner reaches in their first month, while the words that carry ordinary conversation — probably, only, still, I can, I need — sit near the bottom of a ten-thousand-word list, if they appear at all. Function words and discourse particles are frequently discarded on principle as "non-content", which for a spoken register removes precisely the vocabulary that makes speech sound like speech.

There is no corpus of transcribed spoken Georgian at any meaningful scale. That resource does not exist and building it was out of scope. What we could do is choose the two registers that lie closest to speech and are available at volume.

2. Corpus

2.1 Sources

Dialogue (subtitles). Film and television subtitles are written to be spoken aloud and to be read at conversational pace. They are dense in the interactional vocabulary that written prose omits.

Conversational writing (forum). Public discussion threads: argument, advice, storytelling, banter. It is typed rather than spoken, but it is talk.

2.2 The subtitle archive is 80% Bulgarian

The only openly available Georgian subtitle corpus is the OpenSubtitles collection distributed via OPUS. We downloaded the document-structured v2024 Georgian archive (ka.zip, SHA256 ee5acede…): 351 documents, 359,753 lines.

282 of those 351 documents (80.3%) are not Georgian. They are Bulgarian subtitle text that was read through the wrong character encoding at some point upstream and re-emitted as sequences of Georgian glyphs. The text looks Georgian and means nothing.

The detection is clean because the corruption has a signature. Two archaic Georgian letters, ჲ (U+10F2) and ჱ (U+10F1), were dropped from the modern alphabet and are near-absent from contemporary Georgian, but the mis-mapping produces them constantly — ჲ in particular stands in for Cyrillic о, one of the commonest letters in Bulgarian. Measuring the share of lines in each document containing an archaic letter separates the corpus into two populations with nothing in between: 68 documents contain zero, 277 contain them in more than 20% of lines, and no document falls in the 1-10% band.

We drop every document above a 1% threshold, which removes all 282 and retains all 68. Surviving documents are then filtered line-by-line (at least 80% modern Mkhedruli among alphabetic characters, removing bilingual and English subtitle lines), near-duplicate documents are collapsed (the same film released twice), and repeated consecutive lines are merged. Identical short lines across different films are retained: "კარგი." spoken in two hundred films is two hundred dialogue events.

Quality gate. 500 retained lines and 500 discarded lines were sampled and read. Approximately 0.2% of retained lines were not genuine Georgian (target: under 1%), and approximately 0.15% of discarded lines were genuine Georgian (target: under 2%).

2.3 The forum crawl

The forum corpus was collected from public, guest-visible discussion pages across 27 top-level sections, under a crawl that honored the site's stated Crawl-delay with margin (5.5 seconds between requests, one request at a time, no authentication, no registration-gated content). Per-section quotas were assigned proportionally to each section's post count, with a floor and a ceiling so that the largest sections could not dominate the sample. 25 of 27 sections reached at least 85% of quota; the two that did not are smaller than their quota floor.

Quoted text is stripped from each post before counting (quoting another user would double-count their words), along with code blocks, signatures, URLs, and markup. Posts are then language-filtered at 80% Mkhedruli.

Latin-transliterated Georgian is excluded by design. Georgian typed in Latin letters ("gamarjoba rogora xar") cannot be back-transliterated reliably, and guessing would corrupt the counts. We measured the cost rather than assuming it: a classified 300-post sample of the discarded Latin-script material puts 34.3% at genuinely transliterated Georgian, which is approximately 1,337 posts, or 0.63% of the crawl.

We publish aggregate statistics only. No post text, no verbatim subtitle lines, no usernames, and no personal data appear in any published artifact. The raw crawl is retained locally and deleted after release.

3. Lemmatization

Georgian is heavily inflected. A single lexeme surfaces as dozens of forms (ts'igni / ts'ignis / ts'ignebshi; vnakhe / nakhavs / minakhavs). Counting forms rather than lemmas produces a list that is not about words at all.

3.1 Why not a standard NLP pipeline

We evaluated Stanza's Georgian model and rejected it. On colloquial input it tagged modi ("come") as a proper noun and emitted non-words as lemmas for common verbs. Its training data is the written register; the failure is the same distortion this project exists to correct, one level down.

Lemmatization was performed with Gemini 3.1 Pro, with Gemini Flash as an independent second annotator. A pilot on 200 stratified forms measured 90.5% lemma agreement between the two models and surfaced a systematic Flash bias worth recording: it translates loanwords and slang into their standard-Georgian synonyms (spichk'a → asanti, "matches"). For a spoken-register list this is precisely backwards, because the colloquial word is the finding. Pro was therefore used for the primary annotation, Flash as cross-check, and all 4,112 disagreements (5.1% of forms) were adjudicated by a third call that saw both candidates.

3.2 Conventions

Nouns are cited in the nominative singular. Regular verbs take the standard dictionary masdar including its preverb. Inversion verbs — the class where the logical subject is dative (me minda, literally "to-me it-wants") — are cited by their first-person singular present form (minda, makvs, mesmis, meshinia), as dictionaries and teaching materials do, because these verbs have no masdar in ordinary use. All persons still collapse into the single entry.

Loanwords, slang, and colloquial lexemes retain their own citation form and are never normalized to a standard synonym. Colloquial spellings of standard words do fold in (ro → rom, mara → magram).

Particles and interjections are retained. Written-language lists routinely discard them. In spoken Georgian they are among the most common and most useful words in the language, and 230 of them appear in our top 10,000. They carry a part-of-speech column, so any consumer who wants a content-word-only list can filter in one line.

4. Accuracy, and the limits of measuring it

Blind re-annotation: 96.5% lemma accuracy, 97.3% part-of-speech accuracy. Four hundred forms, stratified across frequency bands, were re-annotated by a fresh model instance that never saw the pipeline's output. Nouns scored 99.4%, verbs 94.9%, and accuracy was stable across bands.

That number is real, and it is not sufficient. We want to be explicit about why, because it is the most transferable finding in this project.

A blind re-annotation that shares the pipeline's conventions measures consistency, not correctness. If the convention itself is wrong, both passes agree, and the error is scored as accurate. Ours did exactly that. Three systematic defects sat inside a 96.5% score:

1. Invented masdars. An early convention instructed the model to derive a preverbless masdar for present-series verb forms. For inversion verbs this produces words that do not exist: meshinia ("I am afraid") became shineba. 454 verb lemmas were affected.
2. Cross-verb merges. 116 lemmas had fused two genuinely distinct verbs under one headword — mikvirs ("I am surprised") with vakvirdebi ("I observe"); mjera ("I believe") with dajereba ("to come to believe"); mesmis ("I hear") buried inside smena ("hearing").
3. Masdar/noun splits. A Georgian masdar is simultaneously a verbal noun, so 1,247 lexemes were split across a verb row and a noun row, each holding half the true count and each therefore mis-ranked.

None of these were caught by the accuracy gate. All three were caught by an independent check: comparing the output against a separately built, human-verified written frequency list and asking why the disagreements looked the way they did. Every affected form (4,701 in total) was re-annotated under corrected conventions, and the true homographs (da: "and" / "sister") were preserved rather than merged.

The lesson generalizes beyond Georgian: a single-model pipeline that grades itself will overestimate itself. The check that finds your convention errors is the one that does not share your conventions.

5. Ranking

Frequency per million is computed per register with Laplace smoothing, converted to a Zipf value, and combined as the unweighted mean of the two registers. Neither register is real speech, so privileging one over the other would be a claim we cannot support. Both per-register columns are published so that anyone can re-weight.

Dispersion is reported per register and used as the anti-noise filter. An entry must appear in at least 5 documents, and a word found in only one register qualifies on wide dispersion within it (20 or more forum topics, or 5 or more films) rather than on raw frequency alone. This retains ordinary vocabulary that our small subtitle sample happens to miss, while excluding single-thread artifacts. Every single-register entry is flagged.

Tokenization artifacts (single letters produced by users spacing words out for emphasis, detached case endings, abbreviations that lost their periods) and personal names were removed after review. Place names, country names, and brands were retained.

6. Results

6.1 The divergence

The two tables above are the finding. Words that carry ordinary conversation are ranked near the bottom of the written list, and the vocabulary of political reporting is ranked near the top.

Overlap between our spoken top-1,000 and the written top-1,000 is 54.3%. But 90.4% of our spoken top-1,000 appears somewhere in the written 10,000. The two lists do not disagree about which words are Georgian. They disagree about which words matter, and by an average of thousands of rank positions.

6.2 What the written register discards

The single most common spoken form of rom ("that") is ro. On the forum it is the fourteenth most frequent token in the language. It appears in no textbook we are aware of, and no written frequency list, because in edited prose it is corrected away.

The same is true of the discourse particles that hold spoken Georgian together: kho (tag question), nu (prohibitive, and a hesitation marker), aba, ho, kholme (habitual), metki (quotative), khoda ("and so"). These are not marginal. They are how a sentence sounds like a person said it.

6.3 The standard list is Bulgarian

The most widely copied Georgian frequency list — reused in flashcard decks, applications, and academic work — is derived from the same contaminated OpenSubtitles archive described in §2.2, without document-level filtering.

Because the corruption is a one-to-one character substitution, it can be solved from the data rather than assumed. We seeded a substitution map by character-frequency alignment against the Bulgarian frequency list and hill-climbed it to maximize the share of entries decoding to real Bulgarian words. No key was assumed.

The list's top nine entries decode to the top nine Bulgarian words, in rank order. Across its top 200 entries, 184 decode to real Bulgarian words and 166 fall inside Bulgarian's own top 1,000. It is not a Georgian list with contamination in it. It is the Bulgarian frequency list wearing Georgian letters, and it has been circulating unchallenged for years.

7. Limitations

Neither register is speech. Subtitles are scripted and forum posts are typed. This list is a better proxy for spoken Georgian than anything previously available, and it is still a proxy. We name the registers explicitly rather than claiming to have measured speech.

Some subtitles are machine-translated. A portion of the 68 surviving documents is visibly machine-translated Georgian rather than native subtitling. This affects idiom and lexical choice more than function-word statistics, but it is a real limitation of the only openly available Georgian subtitle corpus.

The subtitle register is small. 268,785 tokens across 68 films, against 5.45M forum tokens. Equal Zipf weighting compensates in the ranking, but the dialogue register's dispersion statistics are computed over few documents.

Forum Georgian skews male, online, and argumentative, and over-represents the topics people go to forums to discuss. It under-represents the domestic and intimate vocabulary of ordinary speech.

One register, one era. The forum crawl skews toward contemporary usage by design, with a deliberate deep slice into older threads. It is not a diachronic corpus.

We built this, and we sell a Georgian course. The dataset, code, and methodology are published in full precisely so that this does not have to be taken on trust.

8. Availability

The top 10,000 lemmas are published with English glosses, transliteration, part of speech, per-register frequency and Zipf values, per-register dispersion, and the inflected forms collapsed into each entry. A full file covers every lemma attested at least five times.

Everything on this page is released under CC BY 4.0. Use it, build on it, cite it.

Subtitle data: OpenSubtitles via the OPUS collection. Please cite P. Lison and J. Tiedemann (2016), OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles, LREC 2016. Forum data: aggregate frequency statistics derived from public discussions on forum.ge; no post text or user data is reproduced.