Журнал изменений

Поддержка итальянского языка и нативное приложение для macOS. Инструмент теперь находит и маскирует итальянские персональные данные — имена, организации, адреса, текстовые даты, телефоны и идентификаторы с метками: codice fiscale (включая C.F./cod. fisc.), Partita IVA, кадастровые ссылки. Модель it_core_news_lg устанавливается по явному запросу. Новое нативное приложение для macOS (Apple Silicon) распространяется подписанным и нотаризованным DMG, с загрузкой языковых моделей по требованию, автообновлениями и той же полностью локальной обработкой: документы не покидают ваш компьютер.

Новые форматы файлов и детекция криптоактивов. Теперь можно анонимизировать обычный текст (.txt), Markdown (.md), CSV-таблицы, сохранённые веб-страницы и HTML-файлы, а также RTF-документы (анонимизированная копия RTF выдаётся как .docx). Текстовые файлы сохраняют исходную кодировку — включая устаревшую Windows-1251. Инструмент также находит и маскирует криптоданные: адреса кошельков (Bitcoin, Ethereum, Tron и другие), хэши транзакций и приватные ключи (WIF, xprv) токенами [CRYPTO_N]. Существующие форматы и языки не изменились.

Исправлен сбой в Settings → Languages: после переустановки приложения ранее скачанный язык мог навсегда показывать «перезапустите для активации», хотя его модель отсутствовала. Список теперь отражает фактически установленные модели и снова предлагает «Скачать».

Поддержка немецких документов. Установите немецкую модель при установке (или скачайте её в Settings → Languages) — и немецкие договоры анонимизируются полностью локально: имена (включая частицы von/zu и академические титулы), компании с немецкими правовыми формами (GmbH, AG, GmbH & Co. KG…), адреса с PLZ, немецкие форматы телефонов, словесные даты («15. Januar 2024») — вместе с существующим распознаванием email, IBAN и дат. Немецкий полностью опционален: без установленной модели ничего не меняется. Русский, английский и испанский не затронуты.

Релиз внутренней архитектуры. Распознавание имён и компаний по каждому языку теперь полностью вынесено в соответствующий языковой пакет, поэтому новые языки можно добавлять, не трогая ядро. Поведение распознавания и результат обработки документов не изменились.

Релиз усиления защиты и качества. Локальный веб-интерфейс теперь отклоняет запросы с не-локальным заголовком Host (защита от DNS-rebinding), а большие таблицы быстрее разрешают пересекающиеся срабатывания. Поведение распознавания и результат обработки документов не изменились.

Точность распознавания дат в русских документах повышена: десятичные числа с запятой (14,7), отдельное слово «год»/«года» без года рядом и лишний предлог в хвосте годовой фразы («2022 год в …») больше не маскируются как даты. Настоящие даты — включая голые годы вроде «2022» — затронуты не были.

Этот релиз повышает качество распознавания в испанских документах. Мексиканские идентификаторы (RFC, CURP, банковские счета CLABE) теперь распознаются и маскируются. Испанские даты словами («15 de julio de 2025») маскируются целиком, проценты оплаты больше не принимаются за даты, а улицы, названные датами («Calle 5 de Mayo»), остаются частью адреса. Названия компаний, которые модель раньше пропускала (например «GreenLeaf Organics Corp., S.A. de C.V.»), теперь маскируются одним фрагментом. Русские и английские документы не затронуты.

В этом релизе добавлен испанский (es) как опциональный язык распознавания. Установите испанскую модель при установке (или скачайте позже в разделе Настройки → Языки), перезапустите — и в испанских документах будут распознаваться имена (включая двойные фамилии), компании и адреса, вместе с уже работающим распознаванием email, телефонов, IBAN и дат. Русский и английский не меняются, а испанский полностью опционален: без установки испанской модели для вас ничего не меняется.

Небольшая чистка интерфейса: убрана разовая подсказка при первом запуске, а меню выбора языка в верхней панели теперь закрывается по клику в стороне или по Esc.

В этом релизе повышена точность маскирования: теперь надёжнее распознаются повторяющиеся даты, номера договоров со строчными буквами, двойные русские фамилии через дефис и номера банковских карт/IBAN с необычными пробелами или тире.

Changed

• Webapp: documents with risky/unsafe structures (e.g. DOCX custom XML parts or bookmark names containing PII) are no longer blocked from download by default. Instead, the download button carries an amber warning indicator and a tooltip explaining that some structures were not anonymized, leaving the final decision to the reviewing lawyer. A second download button is now available at the top of the preview (status row). Hard blocking still applies when the app is started in strict mode (anonymize serve --strict); in that case /download/{doc_id} returns a readable HTML page instead of a JSON error that the browser would otherwise save as a cryptic file.
• Webapp: the warning banner now lists human-readable descriptions instead of raw warning codes (e.g. "custom XML parts are not anonymized" instead of DOCX_CUSTOMXML_UNSUPPORTED), with localized ru/en strings for every DOCX/XLSX/PDF coverage warning and a graceful fallback for unknown codes.
• Webapp: toggling strict mode in Settings now updates the download button and block notice instantly via an HX-Trigger re-render, without re-processing the document, and the choice is persisted to ~/.anonymizer/settings.json so it survives restarts (overridable via ANONYMIZER_SETTINGS_PATH). A CLI --strict flag still forces strict on for that run.

Fixed

• Installer UX: declining optional Tesseract OCR no longer leaves a confusing OCR warning in the post-install doctor check. The installer now labels the slow install/first-run verification steps and ends with the command to start the app. The POSIX installer is also explicitly sh-compatible for the published curl ... | sh entrypoint.

Сканированные и гибридные PDF теперь можно обезличивать локально, если установлен Tesseract OCR с английским и русским языковыми пакетами. Приложение предупреждает о низкой уверенности OCR и встроенных изображениях, которые нужно проверить вручную, а установщик и команда doctor помогают проверить или настроить OCR. Детекторы также усилены для юридических идентификаторов, фрагментов адресов Гонконга, названий компаний и международных телефонных разделителей.

Fixed

• Preview token-sources tooltip no longer disappears when the pointer crosses the gap from the token to the popup; it stays open for 220 ms so the user can reach it and click "remove mask". Crossing into a different token closes any stale popups. The host element is also taken out of the paragraph flow so its injected fragment never affects the spacing of surrounding text.

Сравнивать документы в предпросмотре стало удобнее: исходную и обезличенную панели можно менять по ширине, сворачивать и прокручивать по горизонтали для широких таблиц. В документах KPI, суммы, цели и периодичность теперь остаются видимыми, если это не настоящие даты.

Русские отчетные документы обрабатываются точнее, когда в них есть диапазоны дат или строки с суммами, похожие на номера телефонов. Значения вроде 24-28.02.2022, 13.10.2022 и 2022 - 360 000 остаются видимыми, если это не телефонные номера, а настоящие номера телефонов по-прежнему маскируются.

Внутренние проверки качества покрытия анонимизации стали строже. Релиз сохраняет критические категории сущностей на нужном уровне приоритета и проверяет исправленные регрессии по точной ожидаемой синтетической сущности.

• Русские имена: детектор больше не захватывает префикс К(Ф)Х в персону. В документах вида «Глава К(Ф)Х Иванов И.И.» маскируется только имя, а К(Ф)Х остаётся в тексте как обозначение крестьянского (фермерского) хозяйства.
• Обработка документов: редкие нестандартные случаи теперь обрабатываются безопаснее и точнее, снижая риск лишней маскировки или нестабильного результата в документах со сложной структурой.

Исправлен Windows-only краш при анонимизации документов с именами компаний — словарь правовых форм не загружался на русской Windows (системная кодировка cp1251), падая с UnicodeDecodeError. Теперь файл явно читается как UTF-8 независимо от системной локали.

Английский детектор имён компаний теперь корректно пропускает «Google Looker Studio» в marketing/adtech контексте. Раньше форма с префиксом просачивалась и маскировалась как имя контрагента; теперь, как и краткое «Looker Studio», остаётся видимой.

Таблицы в документах (блоки подписей, реестры, графики) теперь отображаются в preview как настоящие таблицы, а не плоский список абзацев — удобнее проверять анонимизацию построчно. Серии актов гражданского состояния вида «II-МО» / «I-МК» в реестрах теперь маскируются как NUMBER; раньше они утекали в выходной DOCX, хотя номер свидетельства маскировался.

Файлы DOCX с таблицами (блоки подписей, реестры, графики) теперь анонимизируются корректно. Раньше из-за бага дедупликации парсера большинство ячеек больших таблиц могли молча пропадать, и персональные данные попадали в выходной файл нетронутыми. Веб-приложение также теперь автоматически завершает работу при закрытии вкладки браузера, а прогресс загрузки запускается сразу после перетаскивания файла и проходит видимые стадии вместо ложного состояния ошибки.

Индикатор прогресса загрузки теперь видимо анимируется по стадиям, пока документ обрабатывается. В предыдущих версиях все стадии прилетали одним обновлением в конце, поэтому маленькие документы проскакивали прямо к финальному результату без видимой прогрессии.

Индикатор прогресса загрузки теперь остаётся видимым на всех стадиях обработки, включая маленькие документы, которые обрабатываются быстро. В прошлых версиях иногда показывался обманчивый баннер «failed» при медленной загрузке страницы, хотя обработка на самом деле успешно шла дальше.

PDF-документы с заранее размеченными областями редактирования теперь обрабатываются безопасно: закрытый контент полностью редактируется, а встроенный заменяющий текст в этих областях отбрасывается до анонимизации.

Российские банковские счета, СНИЛС и ИНН теперь маскируются в пилотных документах, при этом публичные регуляторы и санкционные ведомства остаются читаемыми, чтобы санкционные оговорки сохраняли смысл. Добавлена новая офлайн-команда диагностики, показывающая состояние установки. Носители документов и диагностические логи больше не сохраняют сырой текст сущностей после анонимизации.

Added

• passport_division detector — context-anchored Russian regex that recognises NNN-NNN passport division codes (код подразделения) as NUMBER tokens. Anchored on код подразделения / подразделение / выдан so bare NNN-NNN sequences (product SKUs, tariff codes) are not hijacked. Registered with conflict precedence 95 — same tier as passport_ru — so the code wins over company_ru even before the post-filter below drops the ORG span. Closes Fix #20 of the post-0.2.3 detector-improvement plan.
• AnonymizerConfig.mask_countries (default False) — opt-in flag for masking country names. In legal documents the country of citizenship is typically NOT what lawyers want masked (it provides useful jurisdiction context, never identifies anyone uniquely), so country GPE spans like Россия / Российская Федерация / Англия / Уэльс / United Kingdom / USA are dropped from address_ner output by default. Set the flag to True for paranoia workflows that want everything geographic anonymised. Closes Fix #19. PR #19 review pass: the country opt-in now runs before the noisy-location filter, so single-word inflected country forms ending in "е" (Уэльсе / Королевстве) are not eaten by the bare-prepositional RU LOC filter when mask_countries=True.

Fixed

• DATE (date) detector now drops Russian <spelled-number> (digit) legal phrasing as a false positive. The _DATEPARSER_SPELLED_NUMBER pattern previously listed only English cardinals (one..ninety), so the equivalent Russian forms used in contracts — в течение трёх (3) лет, в течение пяти (5) рабочих дней, в двух (2) экземплярах, десяти (10) дней, тридцать (30) календарных дней — were misclassified as DATE by dateparser.search. The extended regex enumerates the main inflected forms of one .. ninety (nominative + most common oblique cases — RU nominative один added in PR #20 review pass) so the FP guard fires for both languages. The guard also no longer over-fires: a real long-form RU/EN date that sits next to a duration phrase (14 октября 2025 года ... в течение трёх (3) лет) survives because the spelled-number filter now only triggers when raw lacks a month name. Surfaced on Рус_ДКП-2025.docx, where 5 such phrases were tagged as DATE; pipeline now emits 6 dates (the real ones) on that document. Closes Fix #17.
• ADDRESS (address_ner) detector now drops Natasha LOC/GPE/FAC spans that consist only of Russian contract / boilerplate vocabulary. Adds a _RU_LOC_STOP_TOKENS set covering contract roles (покупатель, продавец, сторона), structural terms (договор, запись, приложение) and Natasha noise tokens from ALL-CAPS section headers (что, включено, указанные, кроме, соответствующая, незамедлительно, исключается). Prepositions / conjunctions are stripped before the check so multi-word noise spans like "соответствующая Запись незамедлительно исключается из" are caught. The stop-list now runs against ANY language pack's NER output (not just ru), because spaCy-EN occasionally emits Cyrillic noise on mixed-script blocks too. On the pilot Рус_ДКП-2025.docx this cut address_ner output from 47 → 10 entries (only legitimate regions / raions remain). Closes Fix #15.
• PERSON (person_ru) detector now drops Natasha PER spans whose tokens are all contract-role labels (Продавца, Покупателю, Стороны). The set is symmetric with the address / company stop-lists. Closes Fix #16.
• COMPANY (company_ru) detector now drops Natasha ORG spans that consist exclusively of generic Russian contract / boilerplate vocabulary (СТОРОНЫ, СТОРОН, СПОРОВ, ГАРАНТИИ, ПОДПИСИ СТОРОН, СО СМЕРТЬЮ, ЕДИНОГО ГОСУДАРСТВЕННОГО, standalone Банк, Стороны предпринимают, …). All inflected forms of each base word are enumerated in _RU_CONTRACT_TERMS; short prepositions and conjunctions are stripped before the check so mixed-case spans like СО СМЕРТЬЮ are caught. Multi-word organisations that contain a contract-vocab token but also non-boilerplate words (Совета Безопасности, Лондонский международный арбитражный суд) survive. Surfaced on the real pilot document Рус_ДКП-2025.docx, where Natasha tagged 11 such uppercase section headers and standalone party words as COMPANY. Closes Fix #13 of the post-0.2.3 detector-improvement plan.
• COMPANY (company_ru) detector now drops bare NNN-NNN spans that Natasha occasionally emits as ORG (e.g. 503-024, 230-001 — passport-division codes). The new passport_division detector re-classifies them as NUMBER; this filter is a belt-and-braces guard so an ORG span never leaks into the token mapping even if the regex anchor misses. Closes Fix #14.
• ADDRESS (address) detector now recognises Russian addresses that start with the region / district / village block rather than the г.|город city prefix, e.g. Московская область, Клинский район, д. Коробочкино, ул. Помещичья, д. 1. The new _RU_REGION_VILLAGE_STREET_RE requires a region marker (ская область / ий край / Республика <name>) plus a district marker (ский|ой|ого район / р-н) plus a village marker (д. / деревня / с. / село / пос. / посёлок / х. / хутор) so it only fires on real addresses and does not hijack arbitrary region mentions. Optional street + house suffixes consume the rest of the address greedily on a comma boundary. Before this fix the city-anchored regex missed the entire span and only the region/district survived as separate address_ner LOC spans — the village and the street stayed unmasked. Surfaced on the pilot Рус_ДКП-2025.docx; closes Fix #18.

Fixed

• uv tool upgrade now uses the correct flag. The 0.2.6 hotfix added --refresh-package docs-anonymizer to both anonymizer.updater and the anonymize update CLI, but that flag only exists on uv tool install — uv tool upgrade rejects it with unexpected argument '--refresh-package' found. Every in-UI [Update] click on 0.2.6 / 0.2.7 wrote the rejection into ~/.anonymizer/last_update_failure.json and the next start surfaced the 🟡 "Не удалось обновить" pill instead of the upgrade banner. Both call sites now pass --reinstall-package docs-anonymizer which implies --refresh-package per uv tool upgrade --help.
• current_status no longer hides a fresh update banner behind the update_failed pill. Previously a single failed upgrade attempt routed every subsequent request into state="update_failed" even when a newer release was already on PyPI — exactly the case where the user most wants to retry. The failed pill still shows when the probe sees no newer release.

Changed

• Update banner UX overhaul:
  • Topbar pill and v<version> chip now hide while the banner is on screen — they duplicated the version pair the banner already shows ("v0.2.5 → 0.2.6").
  • Banner text is now v<current> → <latest> · Release notes, with Release notes linking to the canonical notes_url from version.json schema v1. The previous template fell back to network.banner whenever changelog was missing, printing the same text twice.

Fixed

• POST /update/apply now returns an HTML overlay instead of a JSON payload. htmx happily swapped the raw {"status":"shutting_down","message":"Updater spawned; this process will exit."} blob into the banner before. New overlay tells the user "Update started — close this tab, the new version opens in a fresh tab in a few seconds" and renders full-screen.
• _read_update_failure now drops stale last_update_failure.json entries whose reason references the pre-0.2.0 anonymizer distribution name (Caused by: \`anonymizer\` is not installed). These leftovers silently routed current_status into offline_status and kept the update banner hidden after the docs-anonymizer rename. Pilot install on 0.2.5 hit this.

Added

• anonymize update CLI subcommand. Synchronous wrapper around uv tool upgrade --refresh-package docs-anonymizer docs-anonymizer for shell users who don't want to round-trip through the in-UI [Update] banner. Returns the upgrade exit code so it composes with shell scripts.

Fixed

• All upgrade paths now pass --refresh-package docs-anonymizer to uv. Without it --force --reinstall and tool upgrade happily reinstall the same cached old version that PyPI's Simple API has already superseded — a fresh release can sit on PyPI for an hour while every install still resolves the previous version locally. Updated:
  • installer/install.sh and installer/install.ps1
  • the in-UI anonymizer.updater detached process

Fixed

• In-UI update banner now actually fires when a newer version is published. anonymizer.webapp.network_status.probe was reading the non-existent latest key from version.json and silently flipping every probe into offline_status because Version("") raised InvalidVersion. The canonical https://anonymizer.site/version.json schema v1 exposes the key as version (alongside sha256 / url / released_at / schema_version / notes_url); probe now reads version with latest kept as a backward-compat fallback for older internal mirrors. Surfaced by pilot install on 0.2.3 not seeing the banner after the site published 0.2.4. The test_network_status suite now mirrors the real site payload (full v1 keys) so this exact regression can't slip through unit tests again.

Added

• Vendor-neutral agent workflow infrastructure under .agents/skills so Claude Code, Codex, and generic agents share one canonical workflow source. .claude/skills entries are now compatibility shims that delegate to the canonical files, with adapter notes for Claude, Codex, and generic agents.

Fixed

• Agent workflow consistency tests now validate YAML frontmatter with yaml.safe_load. This prevents invalid multiline description fields from silently breaking skill discovery while still keeping Claude shims and canonical .agents skills in sync.

Fixed

• Add setuptools<81 as a hard runtime dependency in [project] dependencies. Without the runtime pin, uv tool install docs-anonymizer end-user environments resolved the latest setuptools (>=81), which dropped the bundled pkg_resources package; the first RU NER call then crashed with::

  File ".../pymorphy2/analyzer.py", line 114, in _iter_entry_points
      import pkg_resources
  ModuleNotFoundError: No module named 'pkg_resources'
  

  Surfaced in the pilot run on the first real RU .docx document. The existing [tool.uv] constraint-dependencies line was a uv-only override and did not propagate into the published wheel metadata. Lifting this pin requires upgrading natasha to a release that uses pymorphy3.

Added

• anonymize --version (and anonymize -V) flag that prints anonymize <version> and exits. Previously the installer scripts' post-install anonymize --version smoke check failed because the flag did not exist.

Fixed

• Installer scripts (installer/install.sh, installer/install.ps1, and the same pair hosted on anonymizer.site) now pin the tool environment to Python 3.12 via uv tool install --python 3.12. Without the pin uv used the user's default interpreter, which on a fresh macOS / Linux box with uv 0.9.x is 3.14 — the upper bound requires-python = ">=3.11,<3.13" added in 0.2.1 was not enough because uv tool install selects the interpreter before resolving the package and only validates the bound at resolve time, which is too late: a tool environment built on 3.14 still attempts to compile blis 0.7.11 from sdist (no cp313/cp314 wheels) and fails on the removed CPython internals.
• docs/installation.md recovery command updated to include the --python 3.12 pin and the --with model URL.

Fixed

• requires-python constrained to >=3.11,<3.13. blis 0.7.11 (transitive via spacy 3.7 → thinc 8.2) only publishes wheels for cp310..cp312; on Python 3.13 / 3.14 uv tool install docs-anonymizer previously fell back to building blis from sdist and crashed against the CPython 3.14 C API (deleted _PyDict_SetItem_KnownHash, new _PyLong_AsByteArray signature, etc.). Reported during pilot install on a machine where uv picked up its uv-managed Python 3.14. Lifting the upper bound requires a coordinated bump to spacy >= 3.8 and a refreshed en_core_web_lg model wheel.

First public release on PyPI under the neutral distribution name docs-anonymizer (the Python import name remains anonymizer). Version stays sub-1.0 because the product is still in pilot stage; promotion to 1.0.0 will follow once pilot feedback is incorporated. The previous internal 1.0.0 tag (built against name = "anonymizer") was never published to PyPI and is kept only as a historical marker. The reservation stub at docs-anonymizer 0.0.1 is superseded by this release.

Detection metrics snapshot

Systematic 62-document corpus (60 synthetic testkit generate --count 60 --seed 42

• 2 golden_partial es/it) after the Tier P0/P1 fix queue below. Baseline was overall recall 0.886 / precision 0.679 / F1 0.769; reproduce via scripts/analyze_corpus.py --corpus /tmp/corpus_systematic --corpus tests/integration/fixtures/golden_partial.

Type	Recall	Precision	F1	Baseline F1
EMAIL	1.000	1.000	1.000	1.000
NUMBER	1.000	1.000	1.000	0.870
PHONE	1.000	1.000	1.000	0.850
DATE	1.000	1.000	1.000	0.790
PERSON	1.000	0.963	0.981	0.750
COMPANY	1.000	0.992	0.996	0.710
ADDRESS	1.000	0.990	0.995	0.400
Overall	1.000	0.991	0.996	0.769

Residual FP (~7 entries) concentrate in PDF parsing artifacts (e.g. Cardiff, ЛюбовьМихайловна — ligature/glue from the parser layer, not detector logic) and are tracked separately in the detector-improvements plan as Fix #12.

Added

• Sprint 7: Feedback hardening + agent infrastructure — anonymize testkit analyze-feedback aggregates PII-free feedback logs into a ranked HTML report; feedback analyzer unit/integration coverage exercises the analyze/reproduce/improve/score loop on synthetic data; agent docs now point at current code paths; local .claude/skills/* runbooks were finalized for feedback triage, known-failure reproduction, detector improvement, regression checking, and patch release; docs/it-policy.md, docs/feedback.md, and docs/sprint-7-pilot-runbook.md document the operational close-out process.
• Sprint 6: Webapp UI redesign — full UI per docs/superpowers/specs/2026-05-18-anonymizer-ui-design.md:
  • Single-page workspace (replaces 3-screen wizard from tech-spec §9)
  • shadcn-style visual language in plain CSS (no React, no node)
  • Side-by-side preview with hover tooltips, pair highlight, sync scroll
  • Manual mask via text selection → token type dropdown
  • Unmask flow with detector_overreach PII-free feedback event
  • Stage-based processing UI ("Reading file" → "Searching PII" → …)
  • Warning banner + error states (unsupported format / corrupted / crash)
  • Settings modal: detector toggles, strict-mode, open-logs-folder
  • Feedback modal with sample/context fields
  • Network status pill (4 states) + update banner template
  • i18n: Russian (extended) + English UI translations via Babel + .po
  • Language picker in topbar (/session/lang endpoint)

Fixed

• DATE detector now drops bare numeric spans that lack any year/month signal in the surrounding 48-character window. dateparser previously tagged fragments like 7/10, 1500, or from 9/10 as DATE entities even when the document used them as ratios, amounts, or unit references; the new _has_date_context guard requires either a 4-digit 19xx/20xx year or an English/Russian month name nearby. Closes Fix #7 of the systematic corpus plan.
• ADDRESS detector now recognises Russian prospect abbreviations written as пр-т and postfixed street-type forms such as Невский пр. and Малый Знаменский пер. in structured г. <City>, <Street>, д. <House> addresses. Surfaced by the systematic testkit corpus plan.
• ADDRESS detector now recognises numbered UK postcode addresses without an explicit street type, alphanumeric UK building numbers such as 221B, and Russian implicit street forms such as г. Москва, Вавилова, д. 19. Structured address spans now outrank contained date-like fragments, so house numbers like 7/10 no longer displace the full address.
• COMPANY detector now recognises Russian legal-form prefixes АНО, АО, ЗАО, ПАО, ОАО, ООО with quoted names and ИП Фамилия И.О. patterns, covering misses from the systematic testkit corpus plan.
• COMPANY legal-form regex spans now use a dedicated conflict precedence so ИП Смирнов А.А. and quoted Russian legal entities survive contained PERSON spans and overbroad NER spans with adjacent taxpayer numbers.
• COMPANY detector now ignores Cyrillic-only spans produced by the English NER path in mixed RU contact blocks, and recognises English business-name forms such as Thornehill Estates, Cardiff Tech Partners, and Ironbridge Logistics Co. from the systematic testkit corpus.
• PERSON detector now ignores Cyrillic-only spans produced by the English NER path in mixed RU/email contact blocks, recognises inverted English names such as Evans Charlotte and King Isabella M., and adds a Russian full-name regex fallback for cases where Natasha splits surname and first-name/patronymic spans. Follow-up filters keep standalone two-word business phrases and Russian institutional title-case phrases out of PERSON. English COMPANY NER now drops unvalidated one-token ORG spans such as Evans that can displace full person names.
• ADDRESS NER fallback now drops noisy single-word Russian locative city mentions such as в г. Москве, while structured г. <City>, <Street> address regex hits remain covered. Phone detection now scans RU, GB, and US national formats with span deduplication, and generated testkit manifests enumerate raw phone/passport surfaces instead of only canonical values.
• COMPANY NER post-processing now drops unvalidated one-token spans, reducing false positives such as DOB, ДР, Evans, and standalone legal-form abbreviations while preserving legal-suffix and legal-form regex matches.
• Contract-number detection no longer treats school/building identifiers such as Школа №57 as contract numbers, preventing nested NUMBER spans from displacing full company-name detections.
• kf-0001: Russian passport detection now handles the № sign separator between series and number (extended passport_ru regex). Surfaced while annotating Рус_ДКП-2025.docx for golden_real corpus.
• kf-0002: ADDRESS detector recognises UK-format addresses (<number> <street> <Square|Crescent|...>, <City> <UK postcode>) and Russian-format addresses (г. <City>, ул. <Street>, д. <House>, кв. <Unit>, including hyphenated compound city names like "Санкт-Петербург"). Conflict-resolution precedence for structured address regex raised above company_en/company_ru, while NER fallback now uses a separate address_ner detector name with lower precedence so single-token LOC spans like "London" cannot evict legitimate company spans like "Acme London Ltd". The RU regex street group is greedy (previously lazy matched only the minimum 2 chars, leaving most of the address unmasked). Passport № separator handles the no-space variant (45 23№786140). On the 12-doc synthetic corpus this lifted ADDRESS recall 0.29 → 0.54, precision 0.23 → 0.48, F1 0.25 → 0.51; overall recall 0.886 → 0.922.

Added

• Repository skeleton: src/anonymizer/ package layout per tech-spec §2.
• Tooling: pyproject.toml with uv, ruff, pyright, pytest configuration.
• CI: pr.yml, main.yml, release.yml workflows per tech-spec §16.
• Binary guard: tests/integration/test_no_network.py enforces no-network rule for anonymizer.core (tech-spec §2, §22).
• Installer skeletons: installer/install.sh and installer/install.ps1 that fail with a clear message because MVP-0 is not yet shipped.
• Sprint 0 complete: skeleton, CI, no-network guard, installer stubs, IT handoff doc. Sprint 1 (core foundations: docx parser/writer + email/phone regex + basic Token Manager) can begin.
• Sprint 1 complete: docx parser (body + headers/footers/footnotes/endnotes; accept-all + remove comments + IRM rejection), docx writer (body/header/footer run replacement plus footnote/endnote XML replacement), EmailDetector (regex) + PhoneDetector (phonenumbers), TokenManager with canonical-key normalization for EMAIL+PHONE, process_document orchestrator, and a working anonymize <file> CLI. Sprint 2 (NER + LanguagePack + xlsx) can begin.
• Sprint 2a complete: LanguagePack abstraction with RU + EN packs (Natasha + spaCy NER); lingua-py language hint on ParsedDocument; canonical-key normalizers for PERSON/COMPANY/ADDRESS/DATE/NUMBER/DATA; 5 NER detectors (PersonRu, PersonEn, CompanyRu, CompanyEn, Address). xlsx, P0 regex detectors (Date + INN/OGRN/KPP/SNILS/PassportRu/IBAN/Card/BIC/BICRu/ContractNumber), and conflict resolution land in Sprint 2b.
• Sprint 2b complete: DateDetector + 10 Russian taxpayer/banking regex detectors (INN, OGRN, KPP, SNILS, PassportRu, IBAN, Card, BIC, BICRu, ContractNumber); xlsx parser/writer; conflict resolution per tech-spec §6; pipeline + CLI accept docx/xlsx; integration coverage for mixed docx P0 masking and xlsx token reuse/comment removal/formula preservation. Sprint 3 (PyMuPDF, metadata cleaner, coverage warnings, strict mode) can begin.
• Sprint 3 complete: PDF text-layer parsing + true redaction via PyMuPDF, metadata cleaners for docx/xlsx/pdf, format-coverage warnings with informational/risky/unsafe tiers, anonymize --strict, and PDF end-to-end coverage for redaction, AcroForm cleanup, and signature warnings.
• PR #4 review hardening: PDF redaction now uses parser geometry instead of page-wide text search, PDF coverage scanning moved out of core.detection, malformed coverage scans fail closed, warning-blocked CLI outputs are removed, and filled PDF form widgets are explicitly removed rather than value-cleared.
• Sprint 4 complete: testkit personas (20: 10 ru + 10 en, all INNs checksum-valid), Generator protocol + DocxGenerator / XlsxGenerator / PdfGenerator (text/manifest deterministic by seed; raw container bytes may carry timestamps), metrics (recall/precision/F1) and corpus runner over docx/xlsx/pdf; corpus-manifest JSONSchema (docs/agents/corpus-manifest-schema.yaml) + manifest reader/validator; feedback-log Pydantic models (ManualMaskAdded / ManualUnmask / DocumentSummary) mirroring §20 with contract test; tests/integration/known_failures/ skeleton + template + discovery harness; golden_partial es/it corpus with full-precision gate (NER silent, regex tier active); golden_generated reproducible synthetic corpus (seed=42, 24 docs); §18 warning-code end-to-end coverage via format_edge_cases integration test; anonymize testkit generate / run / list-known-failures CLI subcommands; hypothesis-driven fuzz on docx pipeline + CI gate activated. Sprint 5 (Web App + Manual mask) can begin.
• Sprint 5 complete: local FastAPI + htmx webapp on 127.0.0.1 (Upload -> Processing SSE -> Preview -> Download), session-token + CSRF + localhost-only CORS middleware, in-RAM session manager, detector toggles, offline network-status stub, manual-mask ManualOverride anchors for docx/pdf/xlsx materialized as first-class entities with conflict precedence, writer regression coverage for manual_override, PII-free crash/feedback JSONL audit logs, anonymize default webapp serve mode plus serve/stop, and integration guards for webapp E2E plus audit no-PII. Sprint 6 (update flow + distribution) can begin.
• Sprint 6 release candidate: detached updater process, real webapp version.json probe with online/update/offline states, POST /update/apply, --no-update-check, finalized install scripts, canonical AGPL-3.0 LICENSE, Sprint 6 release runbook, and acceptance benchmark notes.

Changed

• Language detection (core.language.detection) now recognises Spanish and Italian alongside Russian / English. Documents in es / it surface as language_hint="es" / "it" instead of being silently classified as en. NER detectors (person_ru, person_en, company_*, address) now respect the document-level hint via _document_language_blocks_detector and stay silent on hint mismatches, keeping the regex tier as the sole acceptance surface on partial-language documents.

• Testkit runner now preserves nested corpus output paths, rejects ambiguous duplicate-stem manifests, enforces expected_warnings, and supports multi-corpus CLI runs without writing _out/ into fixture directories.

• Parser exception surface tightened: DocxParser now wraps zipfile.BadZipFile, lxml.etree.XMLSyntaxError, and docx.opc.exceptions.PackageNotFoundError into ParseError. The "missing w:body" branch reclassifies from EncryptedFileError to ParseError (post-PR-2 review feedback).

• Pipeline now resolves cross-detector span overlap before token assignment (greedy leftmost-keep with longer-span tiebreaker on shared left edge); contract documented in pipeline.py. Prepares for Sprint 2 NER landing.

• Pipeline drops the dead AssertionError branch on detect_format: the Literal["docx"] return type already narrows to docx, anything else raises UnsupportedFormatError from within the detector.

• format_detection.detect_format reads only the first 8 magic bytes via a stream (path.open("rb").read(8)) instead of loading the full file into memory — avoids unnecessary spikes on large inputs.

• sanitize_docx_package_bytes now has a typed source: bytes | ZipFile signature (was untyped duck-typing).

• Note block_id format changes from <region>/p_<note_id>_<paragraph_idx> to <region>/p_<note_id>|<paragraph_idx> so the writer can split unambiguously without assuming integer note IDs. The | delimiter is private to footnotes/endnotes block IDs.

• Run Pydantic model marked frozen=True for symmetry with TextSpan / Entity (both already frozen).

Deferred

• Pyright strict vs standard recorded as docs/questions.md Q-025 — default proposal: strict for core/, standard elsewhere; resolve before Sprint 2 NER landing.
• golden_real hand-annotation for 4 documents from docs/examples/ — recorded as docs/questions.md Q-026. Last remaining Sprint-4 task; needs Legal Lead sign-off and lands before the MVP-0 pilot.

Deprecated

• N/A

Removed

• N/A

Fixed

• DOCX output now persists track-change accept-all/comment removal and masks PII detected in word/footnotes.xml / word/endnotes.xml.
• Sprint 5 webapp review hardening: manual-mask overrides now reprocess documents end-to-end, session reset emits the new cookie token, upload paths are sanitized, oversized uploads are rejected, and feedback context snippets redact partial entity fragments.
• PDF token replacements now preserve the source text font family and size instead of shrinking short-source replacements into mismatched labels.
• PDF lines touched by wider replacement tokens are rewritten as a line so following text shifts instead of being overprinted by the token.
• Web processing page now surfaces safe parser failures, including scanned PDFs without a text layer, instead of leaving the progress screen spinning.
• English legal PDFs now avoid masking service headings, section references, schedule labels, duration numbers, suite numbers, and BIC-like words as entities; EINs and full street addresses are masked as single spans.
• English DOCX legal agreements now mask Contract Ref. identifiers, title-case legal-suffix company names, and directional street addresses while preserving role labels, fee headings, business-day ordinals, and routine channel/payment terms.
• Sprint 5 PR review follow-up: marketing/adtech brand names are preserved in service/account contexts, processing SSE reports parse failures before generic progress, feedback responses no longer expose local log paths, and anonymize stop verifies the pidfile target before signaling.
• English company detection now preserves common privacy and data-protection law titles, including TDPSA / CCPA / GDPR-style act and regulation names.
• English company detection now also preserves common operational SaaS/tool and role/reporting terms such as DocuSign, Asana, CMS, CMO, QBR, and AAA rules.

Security

• Local CORS now pins same-origin requests to the served 127.0.0.1 port.

Added (PR #3 — markup-system-foundation)

• Corpus manifest v2 (anonymizer.testkit.manifest_v2) — Pydantic source of truth with per-entity char offsets, block_id anchors, occurrence tracking, tier classification, and JSONSchema regenerated automatically (scripts/regen_corpus_manifest_schema.py).
• anonymize annotate {prepare,compile,render} CLI for detector-first manifest authoring and synthetic-doc round-trip. compile accepts --source {real,synthetic}; synthetic source additionally requires --style-ref and --variant (plus optional --seed, --author) which populate Manifest.generator directly — no manual YAML edit needed.
• Runner v2 with overlap-based matching (≥50% rule + non_entities), CorpusMetrics carrying overall/by_tier/by_type/per-corpus breakdowns, RunReport preserving per-corpus identity.
• Acceptance gates at two layers: per-document manifest.acceptance block and per-corpus <corpus>/acceptance.yml. AcceptanceFailure exits 1; report still written.
• testkit run --only <basename> filter and testkit diff --baseline --current subcommand.
• Markdown run report in working/testkit-runs/<timestamp>/report.md with per-tier + per-type tables and a "P0 misses" section driven by _diagnose_miss heuristic.
• Skills annotate-corpus and generate-synthetic-doc for guided manifest authoring and synthetic-doc production respectively.

Changed (PR #3 — markup-system-foundation)

• Existing golden_partial manifests migrated from v1 ([TYPE, "text"] pairs) to v2 with full block_id + offset structure.
• regression-check skill updated to point at new report location.

Migration

• v1 manifests are no longer accepted by the runner. Existing manifests must be migrated via Manifest.migrate_v1 (see scripts/regen_corpus_manifest_schema.py for the migration pattern).