# Changelog All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ## [Unreleased] ### Added - **UID→TR resolver integrated into production** (`uid_resolver.py`, wired into `reranker_integration.apply_reranker()` as a third internal pass, strictly after the frozen Phase 5F reranker pass and the residual verbal MIXED detector pass, before Matrix/Embedded Language recomputation): may promote a token still labeled `UID` after both prior stages to `TR` — `UID→TR` only, no `UID→EN` path exists. Uses an additive, explainable, multi-signal evidence model (trusted Turkish-lexicon match on the whole token or a recovered nominal stem, a valid Turkish suffix-chain analysis via the existing unmodified `Annotator._has_valid_turkish_nominal_analysis`, strong fastText Turkish support, Turkish-specific orthographic evidence, plus a capped sentence-level `MatrixLang` bonus that alone can never be sufficient) behind a conservative threshold requiring at least two of the four primary signals. Hard exclusion gates (checked before any scoring) reject URLs/mentions/hashtags/emails/numbers/punctuation/emoji/alphanumeric codes (reusing `cs_pipeline.is_other_token` unmodified), apostrophe-bearing tokens, all-caps acronyms, alphanumeric identifiers, probable proper names (capitalized tokens), direct whole-token English-lexicon matches, and — critically — any token with an English-root-plus-Turkish-suffix analysis (reusing `mixed_reranker.enumerate_candidate_analyses`/`is_non_turkish_stem_evidence` read-only), so it can never pre-empt a genuine MIXED promotion the way an earlier, reverted Turkish-stem fallback (see 1.3.0 below) once did — this was the specific failure mode this resolver was designed from scratch to avoid, by running strictly after both existing MIXED-detection stages instead of before them. Gated behind a new named production constant, `reranker_integration.UID_TR_RESOLVER_ENABLED` (default `True`); setting it `False` restores the exact pre-integration output byte-for-byte. Never touches `TR`/`EN`/`MIXED`/`NE`/`OTHER`/`LANG3`/`SentenceID`/`MatrixLang`/`EmbedLang` otherwise, and a per-token resolver exception is caught and the token's label left unchanged without interrupting the rest of the block (mirrors the residual verbal detector's own fail-safe contract). Offline-validated, then re-validated end-to-end through the actual integrated code path, on the real corpus and both synthetic benchmarks with **zero harmful `UID→TR` changes and zero regression on `NE`/`EN`/`OTHER`/`LANG3`/`MIXED` (all byte-identical) on every source** — see Validation below. `uid_resolver.py` remains independently testable (46 tests, no dependency on `reranker_integration` at import time) and does not duplicate any logic into `cs_pipeline.py`; `cs_annotator_app.py` required no changes, exactly mirroring how the residual verbal detector was integrated. 15 new tests added to `tests/test_reranker_integration.py` covering placement, safe promotion, MIXED/English-root-candidate/NE/EN/OTHER/LANG3 non-interference, Matrix/Embed recomputation, fail-safety, determinism, and the disabled-flag rollback path. - Added the **Confidence Review Tool** (`Tools → Confidence Review Tool`; originally shipped as "UID Review Tool", since renamed — see Changed below) for sequential review and correction of uncertain/low-confidence tokens, filterable by label and confidence band, not limited to tokens labeled `UID`. - Added **Search** and **Find All Occurrences** to the Confidence Review Tool. - Added immediate synchronization between the Confidence Review Tool and the main annotation table. - Added **Merge Cells** and **Undo Merge Cells** to the main annotation interface (Annotation menu and grid context menu). - Added `MatrixLang`/`EmbedLang` recomputation after Confidence Review Tool edits and after Merge Cells/Undo Merge Cells. - Added model and permanent GUI regression tests (`tests/test_confidence_review_tool_gui.py`). - **Added the TDK Checker** (`Tools → TDK Checker`; a separate tool from the Confidence Review Tool — does not replace or rename it), for looking up a token, a hierarchically-parsed root/lemma, and each proposed suffix segment against the Turkish Language Association's (TDK) online dictionary. Opening it from a selected main-table row (`Open in TDK Checker` in the grid's right-click menu, resolved through the existing row-index mapping — never assumes the visual row equals the token index, and disables/warns for `MatrixLang`/`EmbedLang`/separator/empty-token/no/multi-selection cases) populates the token, sentence ID, token index, and sentence context, runs the morphological parser automatically, and — because the click is an explicit request — immediately starts the TDK lookup; opening it standalone from the Tools menu never auto-runs a lookup. **TDK membership is evidence of Turkish lexicalization, not a language-ID verdict**: this tool never reads or writes a token's `label` or `gloss` at all (Gloss is handled entirely by the main table and the Auto-Glossing Tool) — only a new `tdk_segmentation` correction metadata dict (an ordinary extra row key, round-tripped through `.trenproj` exactly like `confidence`/`reviewed` already are). Provides Re-parse, Check TDK, a freely-editable root/lemma + segments field (accepts a manually-corrected segmentation, e.g. `cloud + umuz + a` for `cloudumuza`; typing alone never triggers a re-parse or a lookup), a read-only Explanation panel (Token/Root/Suffix/Analysis per segment/Status), per-term (full token / root / each segment) `FOUND`/`NOT_FOUND`/`UNAVAILABLE`/`NETWORK_ERROR`/`STALE_RESULT` results, a Dictionary Detail panel showing the full TDK entry (headword, part of speech, every sense's definition/usage labels/examples, origin, pronunciation, compounds, idioms, proverbs) for whichever result is selected, Apply Correction (active dataset only, immediate main-table sync, marks the project dirty, never touches another dataset), Undo, and Find All Occurrences (active dataset only, apply to selected occurrences only). The TDK lookup is the only network-dependent part of TREN and is strictly opt-in per action: never at startup, never during normal annotation, never automatic for every token — only on explicit `Check TDK`/`Open in TDK Checker` clicks, off the Tk main thread (a `queue.Queue` drained by a main-thread `after()` poller, since calling Tk APIs directly from a background thread is not reliably safe), with a short timeout, in-memory caching keyed by normalized query + source, stale-response dropping via a generation counter, and only the specific token/root/segment ever sent (never the sentence). Built on a `DictionaryProvider` abstraction (`dictionary_provider.py`: `TDKProvider` for the real, best-effort/undocumented `sozluk.gov.tr` endpoint, with defensive rich-entry extraction and a NOT_FOUND message that is always TREN's own fixed English text, never TDK's raw Turkish backend string; `MockDictionaryProvider`, deterministic, used by every test; `UnavailableProvider` for offline use) and `tdk_parser.py` (its own structured, atomic verb tense/aspect/mood suffix table plus a lexicon-rank-aware selection policy over `mixed_reranker.enumerate_candidate_analyses`'s nominal/verbal candidates — see "Fixed" below for why a purely rank-agnostic reuse of `mixed_reranker`'s own selection strategy was not sufficient). 146 permanent tests across `tests/test_dictionary_provider.py`, `tests/test_tdk_parser.py`, and `tests/test_tdk_checker_gui.py` — all using the mock/offline providers; none call the real TDK service. - **Multiple annotation datasets per project**: a `.trenproj` project can now hold several independent datasets (own name, source text, annotation blocks, labels/glosses, and Matrix/Embedded Language values), switchable via a compact tab bar above the annotation table. Each dataset also keeps its own Merge Cells/Confidence Review Tool undo history for the current session only (see below) — it is not saved to disk. A `+` button (**Add New Data**) creates a new dataset either from freshly entered/pasted text (**Enter New Text**) or by re-running the production pipeline on the active dataset's own source text (**Re-run Current Text**); either way the active dataset is left untouched, and no tab is created if the name is blank, the text is empty, or annotation fails. Switching datasets never re-runs the pipeline or reloads any NLP model, and always closes dataset-scoped auxiliary windows (Confidence Review Tool, Auto-Glossing Tool, Concordance, Word Frequency List, Full Edit Window, Show Sentence) so none of them can be left editing the wrong dataset via a stale row reference. - **Open New File in Add New Data**: the `+` dialog now offers a third choice, **Open New File**, alongside Enter New Text and Re-run Current Text — pick a UTF-8 `.txt` file via the native file chooser and its complete contents become the new dataset's source text, with the annotation pipeline running only after Create; the file's **name only** is kept as optional, display-only dataset metadata (`source_filename`, new and optional in schema version 2 — omitted from older saves, never required to reopen a project), **never its full local path**, since a shared `.trenproj` containing an absolute path could expose the user's account name and local directory structure. The active dataset is never replaced. Selecting a file never marks the project dirty by itself; an empty, unreadable, or non-UTF-8 file, or a failed annotation run, shows a clear message and leaves the active dataset, its table, and its dirty/undo state completely unchanged. - **`.trenproj` schema version 2**: new saves add `"version": 2`, a top-level `"datasets"` list (`{id, name, source_text, blocks, extra_headers}` per dataset — undo history is intentionally excluded), and `"active_dataset_index"`. Version 1 (the previous single-dataset format, including files with no `"version"` key at all) still opens and appears as a single dataset named `Data 1`; it is not rewritten on disk until explicitly saved again, at which point it becomes version 2. Dispatch is strictly version-driven, not shape-driven: a version 1 payload that also contains a `"datasets"` key is rejected as malformed rather than silently accepted, and a version 2 payload missing `"datasets"` is likewise rejected. A malformed `"datasets"` structure, an invalid `"version"` value, or an unsupported future version now shows a clear error dialog instead of crashing or being silently misinterpreted. Opening a project never rewrites the file on disk. - **Real project dirty-state tracking**: closing the application, opening another project, or starting a new project now only prompts to save when something has actually changed since the last save (or since the project was opened/created) — not merely because the project contains data. Typing/pasting in the input editor, running annotation, manual table edits, Confidence Review Tool Apply/Undo, Merge Cells/Undo Merge Cells, adding a dataset, and changing a saved configuration toggle (e.g. NER, Matrix/Embedded Language) all mark the project dirty; switching dataset tabs, exporting, and configuration changes applied programmatically while loading a project do not. Saving successfully, or successfully opening/restoring a project, marks it clean again. - **Shared unsaved-changes guard for New Project / Open Project / Close**: all three now go through the same Save / Discard / Cancel prompt (skipped entirely when the project is already clean). Save only proceeds if the save actually completed; Discard proceeds without saving; Cancel leaves the current project completely untouched. Opening a project specifically defers this prompt until *after* the selected file has been fully read and successfully validated — cancelling the file chooser, or picking an unreadable/malformed file, changes nothing and never even asks. - **Safer close-on-Save**: closing the application while choosing "Save" no longer destroys the window if the save didn't actually happen — cancelling the project-name prompt, declining an overwrite confirmation, or a save failing to write to disk all now keep the application open with the unsaved work intact. Choosing "Discard" still closes without saving, and "Cancel" always keeps the application open. - **Transactional pipeline table replacement**: `_populate_table` now parses, renumbers, and builds the grid entirely in local variables before replacing `self.blocks`/the row-index map/the visible sheet in one step. If parsing or grid construction fails, the previous dataset, the visible table, and the dirty flag are all left exactly as they were — never a partial or emptied table. - **Export Table dialog**: `File → Export Table` (renamed from `Save Output As`) and the toolbar's Export button now open a dialog to choose which dataset to export and in which format (**TXT**, **CSV**, **TREN CoNLL-style**, or **JSONL**) — exactly one dataset per export, never merged, with a filesystem-safe suggested filename derived from the dataset's name. TXT/CSV output content is unchanged from previous versions. Exporting never mutates annotation data, marks the project dirty, or calls the annotation pipeline; cancelling creates no file. - **TREN CoNLL-style export (`.conll`)**: a deterministic, TREN-specific export format (explicitly not CoNLL-U or any external shared-task format) — a file header, then per-sentence `# sent_id` / `# matrix_lang` / `# embedded_lang` comments followed by tab-separated `TokenIndex Token Label Gloss` lines (index restarting at 1 per sentence, empty gloss as `_`, blank line between sentences). Implemented as pure functions in `annotation_model.py` (`blocks_to_conll`). - **JSONL export (`.jsonl`)**: one JSON object per sentence, one line per object, written with Python's standard `json` module (Unicode preserved, not ASCII-escaped) — `sentence_id`, `dataset`, `source_text`, `matrix_lang`, `embedded_lang`, and a `tokens` array of `{index, token, label, gloss}`. Implemented as pure functions in `annotation_model.py` (`blocks_to_jsonl`/`blocks_to_jsonl_records`). ### Fixed - **TDK Checker parser: "filmin" no longer mis-splits as "fil" + "min"; "sürdü" no longer mis-splits as "sürd" + "ü"**. `tdk_parser.py`'s selection policy previously reused `mixed_reranker.select_best_analysis`'s "most suffix segments" strategy, which picked the split with the most pieces regardless of whether the resulting root was a genuine Turkish root — wrong for this tool's purpose even though it is correct for `mixed_reranker`'s own MIXED-evidence use (left unmodified). Replaced with a purpose-built, lexicon-frequency-rank-aware additive scorer plus `tdk_parser.py`'s own structured, atomic verb tense/aspect/mood suffix table (so "dü" is scored as one past-tense suffix, never decomposed into "d" + "ü", and `mixed_reranker`'s default verbal level, which omits tense/aspect/mood entirely, is no longer relied on for this). A merely-frequent inflected surface form (e.g. "kitaplar", "geldi" as whole-token entries in the corpus-frequency word list) is no longer confused with a genuine base root or preferred over a fuller, correct decomposition. Genuinely close competing readings (e.g. "kalem" as the base noun "pen" vs. "kale" [fortress] + 1st-singular possessive) are now reported as `ambiguous_candidate` instead of confidently picking one. The parser still never assigns a language label. See `tdk_parser.py`'s module docstring for the full policy and `tests/test_tdk_parser.py` for the permanent regression coverage (noun and verb suffix-hierarchy examples, longest-root preference, rejection of arbitrary single-character "suffixes", ambiguous-candidate detection). - **Gloss removed from the TDK Checker window entirely** (label, input field, and all gloss-specific read/write paths in Apply Correction, Undo, and Find All Occurrences → Apply to Selected) — Gloss continues to work exactly as before everywhere else in TREN (the main table, the Auto-Glossing Tool). Apply Correction and Undo now only ever touch a row's `tdk_segmentation`. - **Root/Lemma and Segments are now freely editable with explicit query-snapshot/staleness tracking**: editing either field after a Check TDK run immediately marks every currently-displayed result `STALE_RESULT` (kept visible, clearly labeled, rather than silently discarded) until Check TDK is run again; Check TDK always queries the CURRENT field values, never a value left over from the automatic parse, and an async response is discarded both when a newer lookup has since started (existing per-request generation counter) and when the fields have since changed even for the still-current request. Typing into these fields never itself triggers a re-parse or a lookup — only the explicit Re-parse button or Check TDK do. - **Added an Explanation panel** (Token / Root / one Suffix+Analysis line per segment / Status) showing exactly how the current root/segments were derived, and a **Dictionary Detail panel** showing the FULL TDK entry for whichever result row is selected — headword, part of speech, every sense's definition/usage labels/examples, origin/etymology, pronunciation, compounds, idioms, proverbs, source, and the exact query sent — instead of only a FOUND/NOT_FOUND status. Any field TDK did not provide reads `"Not provided"`, never a guessed value; unrecognized-but-present TDK response fields are preserved structurally (never dumped as raw HTML/JSON into the UI). - **`dictionary_provider.py`'s NOT_FOUND result no longer leaks TDK's raw Turkish backend text** (e.g. `"Sonuç bulunamadı"`) into `LookupResult.message` — always TREN's own fixed English message now. ### Changed - **Renamed the UID Review Tool to the Confidence Review Tool**, reflecting that it now filters and displays uncertain/low-confidence tokens for *any* of the 7 schema labels (TR/EN/MIXED/UID/NE/OTHER/LANG3), not just `UID` — new label and confidence-band (HIGH/MEDIUM/LOW) filter checkboxes, an "Evidence" panel showing the current label, confidence score/band, uncertainty reasons, and relevant pipeline evidence for the selected token, and per-row reviewed-status tracking. All prior behavior (Search, Find All Occurrences, First/Previous/Next/Last, Apply, Undo, immediate main-table synchronization) is preserved. - **The Confidence Review Tool now opens with "All Uncertain" by default, not UID-only**: a new **View** combobox (**All Uncertain** / **UID Only** / **Custom**) selects the filtering mode. **All Uncertain** lists every token, across all 7 labels, whose confidence record is flagged for review (`confidence.is_review_required`, reading the record's own `review_recommended` flag — never a label-based rule); a confidently-labeled token is excluded regardless of label, and an uncertain one is included regardless of label. **UID Only** reproduces the tool's original default exactly (UID-labeled tokens only, no confidence-band restriction). **Custom** is selected automatically whenever the Labels/Confidence checkboxes are touched directly, and those checkboxes continue to work exactly as before. Switching datasets still always closes the tool; reopening it resets to the **All Uncertain** default for whichever dataset is now active. ### Validation - **UID→TR resolver**, re-measured through the actual integrated `reranker_integration.apply_reranker()` code path (flag forced `False` for "without" vs. the real `True` default for "with", both through the same code, not two implementations), on all three evaluation sources, with zero harmful changes and zero regression on any other label on every source: | Source | Accuracy (without → with) | Weighted F1 (without → with) | MIXED F1 | |---|---|---|---| | Real corpus (201 sent.) | 0.8751 → 0.8846 | 0.9070 → 0.9121 | 0.8793 (unchanged) | | First synthetic benchmark (100 sent.) | 0.8655 → 0.8828 | 0.8894 → 0.8987 | 0.7312 (unchanged) | | Synthetic v2, adjudicated gold (100 sent.) | 0.9212 → 0.9341 | 0.9406 → 0.9476 | 0.8571 (unchanged) | Real corpus: 30 `UID→TR` changes (28 beneficial, 0 harmful, 2 neutral — already-wrong `avrupada`/`avrupaya`, lowercase-country-name misses the capitalization-based proper-name gate doesn't catch, still wrong either way, not a new error). First synthetic: 10 changes, all beneficial, 0 harmful. Benchmark v2: 8 changes, all beneficial, 0 harmful. `NE`/`EN`/`OTHER`/`LANG3` per-label precision/recall/F1 are byte-identical between the two conditions on every source (verified programmatically, not just by construction). - **Baseline discrepancy reconciled, not silently replaced**: the real-corpus accuracy documented for 1.3.0 below (0.8725) differs from the freshly re-measured "without UID→TR resolver" baseline above (0.8751, +0.26pp). Investigated and explained rather than papered over: `~/Desktop/corpus_test/output.csv` (the only persisted prediction artifact for this corpus) predates the 1.3.0 commits entirely (scores only 0.8398/0.5545 MIXED F1 against gold today) and is *not* the source of the 0.8725 figure — that original prediction file no longer exists on disk, so an exact byte-for-byte replay isn't possible. The raw corpus text and gold file are unchanged since January 2026, and `cs_pipeline.py`/`reranker_integration.py`/`mixed_reranker.py` have had zero commits since `5d644ae` (2026-08-02, the commit 0.8725 was almost certainly measured at) — ruling out corpus-version or code-version drift as the cause. The residual +0.26pp is attributable to the evaluation harness's alignment-exclusion policy on a small number of the real corpus's own pre-existing, already-documented segmentation-echo rows plus minor raw-text/tokenizer-vintage drift on hyphenated number ranges (e.g. `17-25` vs. two separate tokens) — a handful of tokens out of ~3086, affecting both evaluation conditions identically. Along the way, a real, separate bug was found and fixed in the harness itself: the first synthetic benchmark's gold file keeps sentence-final punctuation glued to the last word in 97/100 sentences (unique to that one file — 0/201 on the real corpus, 0/100 on benchmark v2), which was silently excluding roughly a third of that benchmark's gold-`MIXED` tokens from scoring; fixing it moved that benchmark's baseline from a 1.5-point gap against the previously-documented figures to an exact 4-decimal match on MIXED P/R/F1. ## [1.3.0] - 2026-08-02 ### Added - **NE Policy C** (`cs_pipeline.Annotator._build_ne_map`): a token backed only by a `TIME`-subtype Stanza entity match (`POLICY_C_EXCLUDED_NE_SUBTYPES = {"TIME"}`) is no longer classified `NE`, falling through to the existing non-NE labeling logic instead. Scoped to `TIME` only after evaluation against the real corpus found genuine gold-`NE` `MONEY` collisions that ruled out excluding that subtype too. - **NE Policy D** (`cs_pipeline.Annotator._qualifies_for_policy_d`): a token backed only by an NE match is no longer kept `NE` when it is a direct English-lexicon match, absent from the Turkish lexicon, not the anonymization placeholder literal, not an acronym/all-caps/alphanumeric token, and has no valid Turkish nominal-suffix analysis — with a further guard requiring the other piece of a multi-word `ORGANIZATION` span to be a bare, alphabetic, capitalized word before the override applies (distinguishing genuine compound entity names from NER span-boundary noise). Any piece backed by `PERSON`/`LOCATION` evidence is unconditionally protected from Policy D; a single-piece `ORGANIZATION` span receives no such protection and may still lose `NE` status if it independently satisfies every other Policy D condition — only genuine multi-word `ORGANIZATION` compounds are guarded. - **Strict residual verbal MIXED detector** (`mixed_reranker.evaluate_residual_verbal_promotion`, wired into `reranker_integration.apply_reranker()` as a second internal pass, strictly after the frozen Phase 5F reranker pass): may promote a token whose label is still `UID`/`TR` after the reranker pass to `MIXED`, using strict direct-English-lexicon evidence only (no fastText fallback), gated by 11 conditions including an explicit Turkish verbalizer/passive-inchoative marker, Turkish-lexicon absence of the stem, no lexicon-confirmed competing nominal analysis, uniqueness of the selected analysis, and exclusion of proper names/acronyms/codes/URLs/mentions/hashtags/apostrophe-bearing tokens. Adds one new closed-class suffix table (1st/3rd-person-plural verbal agreement, e.g. `-dık`/`-ık`/`-ız`) on top of the existing, unmodified `PHASE_4E` verbal-suffix level; the frozen model's own default level (`PHASE_4C1`) is never referenced or changed. Never touches `NE`, `EN`, `OTHER`, `LANG3`, or a token the reranker pass already promoted. A per-token exception here is caught and the token's label left unchanged, without interrupting the rest of the block. - **Matrix/Embedded recomputation after promotions**: `reranker_integration.apply_reranker()` recomputes `MatrixLang`/`EmbedLang` (via the existing, unmodified `Annotator._decide_matrix_embed`) whenever either the reranker pass or the residual verbal pass changes a label in a sentence; blocks with no label change are returned byte-for-byte identical to the rule-based annotator's own output. - **Expanded regression coverage**: 707 tests now pass (up from 560 at 1.2.0), including full coverage for NE Policies C/D (`tests/test_cs_pipeline.py`) and the residual verbal detector (`tests/test_mixed_reranker.py`, `tests/test_reranker_integration.py`) — parsing/candidate-generation, all 8 named regression targets, 26 native-Turkish negative controls, placement-after-reranker, fail-safe behavior, and Matrix/Embedded recomputation. ### Fixed - **Tokenizer** (`cs_pipeline.tokenize`): `@mentions`, `#hashtags`, emoji, and hyphen/underscore-joined alphanumeric codes (e.g. `A7K-204`, `TR-9081-ZX`) are now preserved as single tokens instead of being silently dropped (`@`/`#`/emoji, since `\w` never matches them) or fragmented on the separator (codes). `is_other_token` gained a `CODE_RE` check so these codes are correctly classified `OTHER`. This changes token boundaries (and therefore TXT-export row counts) for input containing any of the above — call this out when reviewing/consuming exports of text that includes them. Also fixes a related regression found during evaluation of this change: the separator-joined-number alternative previously matched on a bare leading digit run (`*`), so it wrongly won over the general word pattern and split digit-prefixed words like `20li`/`3d`/`6da` into two tokens; it now requires at least one actual separator group (`+`). - **`resources/frequent_tr_words.txt`**: removed three English function-word entries (`the`, `a`, `i`) that were present in the top-1000 Turkish frequency tier, causing them to resolve `TR` ahead of the English lexicon check regardless of context. `in` was also tried and reverted — a controlled evaluation against the real annotated corpus found one concrete regression (a genuine Turkish-context use of `in` that flipped from correctly `TR` to incorrectly `EN`), so it stays in the Turkish list pending a more targeted fix. `her` (a genuinely common Turkish word, "every/each") was intentionally left in place — its collision with the English possessive pronoun is a real cross-lingual ambiguity with no context-free fix, not a data error. Two further changes — a Turkish-stem morphological fallback for `Annotator._choose_label` (UID over-generation), and matching an apostrophe-split base against recognized NER entities in `Annotator._build_ne_map` (NE priority for `Ankara'da`-style tokens) — were implemented and evaluated against both the real annotated corpus and a synthetic benchmark, but reverted after the evaluation showed real regressions (the Turkish-stem fallback pre-empts MIXED detection and converts genuine `MIXED`/`UID` tokens to `TR`; the NE-priority change suppresses at least one genuine apostrophe-bearing `MIXED` form). Automatic `LANG3` production was also implemented and reverted: it conflicts with the previously-documented manual-only `LANG3` behavior, and real-corpus evaluation found real false positives with no real corpus examples to confirm a true positive. All three remain candidates for future work, not shipped here — the UID→TR resolver evaluated alongside them is likewise **not integrated** into this release. ### Validation - Production validated end-to-end on the real, manually-annotated corpus (primary evidence; accuracy 0.8725, MIXED F1 0.8778) and two synthetic, LLM-authored benchmarks used as secondary diagnostic evidence only, not external validation (first synthetic: accuracy 0.8660, MIXED F1 0.7312; benchmark v2, adjudicated gold: accuracy 0.9212, MIXED F1 0.8571). - The frozen Phase 5F reranker (integrated in 1.2.0) is **unchanged** in this release — no retraining, no threshold change (remains `0.85`), no feature-configuration change. - `LANG3` remains manual-only; no automatic-LANG3 behavior was added. - No `.trenproj`/TXT/CSV schema changes. ### Known limitations - UID remains weak and heterogeneous (low support across every evaluation source). - NE precision remains limited by third-party Stanza NER behavior. - Rare Turkish lexicon coverage causes UID predictions on otherwise-ordinary Turkish words. - TR/EN dual-lexicon collision words are resolved the same way regardless of sentence context. - Proper-name vs. common-word ambiguity (e.g. a Turkish common noun used as a given name) is not resolved. - A pre-existing rule-based nominal-MIXED false-positive pattern can occur on bare English loanwords whose ending coincidentally matches a Turkish case suffix (observed on `office`/`remote`). - Complex nominal MIXED chains (English stem + Turkish possessive + case suffix, e.g. `cloudumuza`) are outside the residual verbal detector's scope and may still be missed. ## [1.2.0] - 2026-07-29 ### Added - Integrated the validated MIXED-token reranker into the production annotation pipeline (`reranker_integration.py`, wired into `cs_annotator_app.py`'s `run_pipeline()`): it now runs automatically, as a post-processing stage after `Annotator.annotate()`, on every annotation request, promoting eligible `UID`/`NE`/`TR` labels to `MIXED` when the frozen Phase 5F model's probability clears its validated 0.85 threshold. The model (`resources/models/model.joblib`, `vectorizer.joblib`, `metadata.json`) loads lazily on the first annotation request and is cached for the session; if dependencies or model resources are missing, corrupted, or fail metadata validation, annotation falls back to the original rule-based output with no crash and no interruption. `Annotator.annotate()`'s internals, the parser, and the `.trenproj`/TXT/CSV schemas are unchanged. ### Documentation - Synchronized README.md, CHANGELOG.md, and CLAUDE.md with the now-production-integrated reranker: architecture, the exact pipeline order, loading/fallback behavior, the frozen feature-batch configuration, and the current (560-test) test count. ## [1.1.0] - 2026-07-29 ### Added - Extracted GUI-independent annotation-state logic from `cs_annotator_app.py` into a new `annotation_model.py` module. - Added an automated unit test suite (`tests/`) covering `annotation_model.py` and `cs_pipeline.py`. - Added a GitHub Actions CI workflow that runs a syntax check, the quickstart example, and the test suite on every push and pull request. - Added a runnable quickstart example (`examples/quickstart.py`) demonstrating the annotation pipeline. - Developed an isolated MIXED-token reranker research module (`mixed_reranker.py`, `tools/build_reranker_dataset.py`, `tools/train_mixed_reranker.py`) with its own test suite (`tests/test_mixed_reranker.py`), kept out of the production annotation pipeline during its development and evaluation. The frozen baseline (Batch A + Batch C + a pruned Batch G) reaches MIXED precision 0.893 / recall 0.781 / F1 0.8333 on a held-out test split; two additional feature groups (Batch B, Batch D) were evaluated and experimentally rejected, with rationale documented in the README's "MIXED-Token Reranker" section. This module was developed with substantial AI (Claude) assistance across an iterative, benchmarked experimentation process. ### Documentation - Added `CONTRIBUTING.md` with development setup, testing, and pull request guidelines. - Added `CODE_OF_CONDUCT.md`. ## [1.0.0] ### Added - Initial public release of TREN.