Changelog

Italian language support and the native macOS application. The anonymizer now detects and masks Italian personal data — names, companies, addresses, textual dates, phone numbers, and label-anchored identifiers such as codice fiscale (including C.F./cod. fisc.), Partita IVA, and cadastral references — with the it_core_news_lg model available as an opt-in download. The new native macOS app (Apple Silicon) ships as a signed and notarized DMG with on-demand language models, automatic updates, and the same fully local processing: documents never leave your machine.

New file formats and crypto-asset detection. You can now anonymize plain text (.txt), Markdown (.md), CSV tables, saved web pages and HTML files, and RTF documents (the anonymized RTF copy is delivered as .docx). Text files keep their original encoding — including legacy Windows-1251. The tool also detects and masks cryptocurrency data: wallet addresses (Bitcoin, Ethereum, Tron and others), transaction hashes, and private keys (WIF, xprv) as [CRYPTO_N] tokens. Existing formats and languages are unchanged.

Fixes a Settings → Languages glitch: after reinstalling the app, a previously downloaded language could show a permanent "restart to activate" hint even though its model was gone. The list now reflects what is actually installed and offers Download again.

German document support. Install the German model at setup time (or download it from Settings → Languages) and German contracts get full local anonymization: names (including von/zu particles and academic titles), companies with German legal forms (GmbH, AG, GmbH & Co. KG …), street addresses with PLZ, German phone formats, verbose German dates ("15. Januar 2024") — alongside the existing email, IBAN and date detection. German is fully opt-in: nothing changes unless you install the model. Russian, English and Spanish are unchanged.

Internal architecture release. Per-language name/company detection now lives entirely inside each language pack, so future languages can be added without touching the core engine. No change to detection behaviour or document output.

Hardening and quality release. The local web UI now rejects requests with a non-local Host header (defence against DNS-rebinding), and large spreadsheets resolve overlapping detections faster. No change to detection behaviour or document output.

Date detection on Russian documents is more precise: decimal numbers written with a comma (14,7), the standalone word "год"/"года" with no year attached, and a stray preposition trailing a year ("2022 год в …") are no longer masked as dates. Real dates — including bare years like "2022" — are unaffected.

This release sharpens detection quality for Spanish documents. Mexican identifiers (RFC, CURP, CLABE bank accounts) are now recognized and masked. Spanish dates written out in words ("15 de julio de 2025") are masked whole instead of leaving fragments, payment percentages are no longer mistaken for dates, and date-named streets ("Calle 5 de Mayo") stay part of the address. Company names the model previously missed (like "GreenLeaf Organics Corp., S.A. de C.V.") are now masked as one piece. Russian and English documents are unaffected.

This release adds Spanish (es) as an optional detection language. Install the Spanish model at setup time (or download it later from Settings → Languages), restart, and Spanish documents will have names (including double surnames), companies, and addresses detected — alongside the existing email, phone, IBAN and date detection. Russian and English are unchanged and Spanish is fully opt-in: nothing changes for you unless you install the Spanish model.

Minor UI cleanup: the one-time first-run banner has been removed, and the language menu in the top bar now closes when you click elsewhere or press Esc.

This release improves redaction accuracy: repeated dates, contract numbers with lowercase letters, hyphenated Russian surnames, and bank-card/IBAN numbers written with unusual spacing or dashes are now caught more reliably.

Changed

• Webapp: documents with risky/unsafe structures (e.g. DOCX custom XML parts or bookmark names containing PII) are no longer blocked from download by default. Instead, the download button carries an amber warning indicator and a tooltip explaining that some structures were not anonymized, leaving the final decision to the reviewing lawyer. A second download button is now available at the top of the preview (status row). Hard blocking still applies when the app is started in strict mode (anonymize serve --strict); in that case /download/{doc_id} returns a readable HTML page instead of a JSON error that the browser would otherwise save as a cryptic file.
• Webapp: the warning banner now lists human-readable descriptions instead of raw warning codes (e.g. "custom XML parts are not anonymized" instead of DOCX_CUSTOMXML_UNSUPPORTED), with localized ru/en strings for every DOCX/XLSX/PDF coverage warning and a graceful fallback for unknown codes.
• Webapp: toggling strict mode in Settings now updates the download button and block notice instantly via an HX-Trigger re-render, without re-processing the document, and the choice is persisted to ~/.anonymizer/settings.json so it survives restarts (overridable via ANONYMIZER_SETTINGS_PATH). A CLI --strict flag still forces strict on for that run.

Fixed

• Installer UX: declining optional Tesseract OCR no longer leaves a confusing OCR warning in the post-install doctor check. The installer now labels the slow install/first-run verification steps and ends with the command to start the app. The POSIX installer is also explicitly sh-compatible for the published curl ... | sh entrypoint.

Scanned and hybrid PDFs can now be anonymized locally when Tesseract OCR with English and Russian language packs is available. The app warns about low OCR confidence and embedded images that need manual review, and the installer and doctor command help check or set up OCR. Detection is also stronger for legal identifiers, Hong Kong address fragments, company names, and international phone separators.

Fixed

• Preview token-sources tooltip no longer disappears when the pointer crosses the gap from the token to the popup; it stays open for 220 ms so the user can reach it and click "remove mask". Crossing into a different token closes any stale popups. The host element is also taken out of the paragraph flow so its injected fragment never affects the spacing of surrounding text.

Document comparison in the preview is easier to use: the source and anonymized panes can be resized, collapsed, and scrolled horizontally for wide tables. Documents also keep KPI, fee, target, and cadence fragments visible when those values are not real dates.

Russian report documents are handled more accurately when they contain date ranges or amount rows that look similar to phone numbers. Values such as 24-28.02.2022, 13.10.2022, and 2022 - 360 000 remain visible when they are not phone numbers, while real phone numbers continue to be masked.

Internal quality checks for anonymization coverage are now stricter. The release keeps critical entity categories at the required priority level and verifies fixed regression cases against the exact intended synthetic entity.

• Russian person names: the detector no longer absorbs the K(F)X farm-organisation prefix into the entity surface. In documents like "Глава К(Ф)Х Иванов И.И.", only the person's name is masked; К(Ф)Х remains visible in the text.
• Document processing: rare edge cases are handled more safely and precisely, reducing the risk of incorrect masking or unstable output when documents contain unusual structure.

Fixed a Windows-only crash when anonymizing documents containing company names — the legal-forms dictionary failed to load on Russian Windows (cp1251 default codepage) with UnicodeDecodeError. The file is now explicitly read as UTF-8 regardless of system locale.

English company-name detection now correctly skips "Google Looker Studio" in marketing/adtech contexts. Previously the prefixed form leaked through and was masked as a counterparty name; now it stays visible like the bare "Looker Studio" form already did.

Tables in documents (signature blocks, registers, schedules) now render in the preview as real tables instead of a flat list of paragraphs — easier to review the anonymization row by row. Russian civil-act registry series like "II-МО" / "I-МК" are now masked as NUMBER; previously they slipped through to the output DOCX while the trailing certificate number was masked.

DOCX files with tables (signature blocks, registers, schedules) are now anonymized correctly. Previously, a parser dedup bug could silently drop most cells of large tables, letting PII through to the output untouched. The webapp also now shuts itself down automatically when you close the browser tab, and upload progress now starts immediately after dropping a file with visible stages instead of a false failure state.

The per-stage upload progress indicator now animates visibly while documents are being processed. Previous versions delivered all stages as a single update at the end, so small documents jumped directly to the final result with no visible progression.

The upload progress indicator now stays visible through all processing stages, including for small documents that finish quickly. Earlier versions sometimes showed a misleading "failed" banner on slow page loads when processing was actually still running successfully.

PDF documents containing pre-marked redaction areas are now handled safely: covered content is fully redacted and any embedded replacement text on those areas is dropped before anonymization.

Russian bank accounts, SNILS, and ИНН are now masked in pilot documents, while public regulators and sanctions bodies are left readable so sanctions clauses stay intelligible. A new offline diagnostic command reports installation health. Document carriers and diagnostic logs no longer retain raw entity text after anonymization.

Added

• passport_division detector — context-anchored Russian regex that recognises NNN-NNN passport division codes (код подразделения) as NUMBER tokens. Anchored on код подразделения / подразделение / выдан so bare NNN-NNN sequences (product SKUs, tariff codes) are not hijacked. Registered with conflict precedence 95 — same tier as passport_ru — so the code wins over company_ru even before the post-filter below drops the ORG span. Closes Fix #20 of the post-0.2.3 detector-improvement plan.
• AnonymizerConfig.mask_countries (default False) — opt-in flag for masking country names. In legal documents the country of citizenship is typically NOT what lawyers want masked (it provides useful jurisdiction context, never identifies anyone uniquely), so country GPE spans like Россия / Российская Федерация / Англия / Уэльс / United Kingdom / USA are dropped from address_ner output by default. Set the flag to True for paranoia workflows that want everything geographic anonymised. Closes Fix #19. PR #19 review pass: the country opt-in now runs before the noisy-location filter, so single-word inflected country forms ending in "е" (Уэльсе / Королевстве) are not eaten by the bare-prepositional RU LOC filter when mask_countries=True.

Fixed

• DATE (date) detector now drops Russian <spelled-number> (digit) legal phrasing as a false positive. The _DATEPARSER_SPELLED_NUMBER pattern previously listed only English cardinals (one..ninety), so the equivalent Russian forms used in contracts — в течение трёх (3) лет, в течение пяти (5) рабочих дней, в двух (2) экземплярах, десяти (10) дней, тридцать (30) календарных дней — were misclassified as DATE by dateparser.search. The extended regex enumerates the main inflected forms of one .. ninety (nominative + most common oblique cases — RU nominative один added in PR #20 review pass) so the FP guard fires for both languages. The guard also no longer over-fires: a real long-form RU/EN date that sits next to a duration phrase (14 октября 2025 года ... в течение трёх (3) лет) survives because the spelled-number filter now only triggers when raw lacks a month name. Surfaced on Рус_ДКП-2025.docx, where 5 such phrases were tagged as DATE; pipeline now emits 6 dates (the real ones) on that document. Closes Fix #17.
• ADDRESS (address_ner) detector now drops Natasha LOC/GPE/FAC spans that consist only of Russian contract / boilerplate vocabulary. Adds a _RU_LOC_STOP_TOKENS set covering contract roles (покупатель, продавец, сторона), structural terms (договор, запись, приложение) and Natasha noise tokens from ALL-CAPS section headers (что, включено, указанные, кроме, соответствующая, незамедлительно, исключается). Prepositions / conjunctions are stripped before the check so multi-word noise spans like "соответствующая Запись незамедлительно исключается из" are caught. The stop-list now runs against ANY language pack's NER output (not just ru), because spaCy-EN occasionally emits Cyrillic noise on mixed-script blocks too. On the pilot Рус_ДКП-2025.docx this cut address_ner output from 47 → 10 entries (only legitimate regions / raions remain). Closes Fix #15.
• PERSON (person_ru) detector now drops Natasha PER spans whose tokens are all contract-role labels (Продавца, Покупателю, Стороны). The set is symmetric with the address / company stop-lists. Closes Fix #16.
• COMPANY (company_ru) detector now drops Natasha ORG spans that consist exclusively of generic Russian contract / boilerplate vocabulary (СТОРОНЫ, СТОРОН, СПОРОВ, ГАРАНТИИ, ПОДПИСИ СТОРОН, СО СМЕРТЬЮ, ЕДИНОГО ГОСУДАРСТВЕННОГО, standalone Банк, Стороны предпринимают, …). All inflected forms of each base word are enumerated in _RU_CONTRACT_TERMS; short prepositions and conjunctions are stripped before the check so mixed-case spans like СО СМЕРТЬЮ are caught. Multi-word organisations that contain a contract-vocab token but also non-boilerplate words (Совета Безопасности, Лондонский международный арбитражный суд) survive. Surfaced on the real pilot document Рус_ДКП-2025.docx, where Natasha tagged 11 such uppercase section headers and standalone party words as COMPANY. Closes Fix #13 of the post-0.2.3 detector-improvement plan.
• COMPANY (company_ru) detector now drops bare NNN-NNN spans that Natasha occasionally emits as ORG (e.g. 503-024, 230-001 — passport-division codes). The new passport_division detector re-classifies them as NUMBER; this filter is a belt-and-braces guard so an ORG span never leaks into the token mapping even if the regex anchor misses. Closes Fix #14.
• ADDRESS (address) detector now recognises Russian addresses that start with the region / district / village block rather than the г.|город city prefix, e.g. Московская область, Клинский район, д. Коробочкино, ул. Помещичья, д. 1. The new _RU_REGION_VILLAGE_STREET_RE requires a region marker (ская область / ий край / Республика <name>) plus a district marker (ский|ой|ого район / р-н) plus a village marker (д. / деревня / с. / село / пос. / посёлок / х. / хутор) so it only fires on real addresses and does not hijack arbitrary region mentions. Optional street + house suffixes consume the rest of the address greedily on a comma boundary. Before this fix the city-anchored regex missed the entire span and only the region/district survived as separate address_ner LOC spans — the village and the street stayed unmasked. Surfaced on the pilot Рус_ДКП-2025.docx; closes Fix #18.

Fixed

• uv tool upgrade now uses the correct flag. The 0.2.6 hotfix added --refresh-package docs-anonymizer to both anonymizer.updater and the anonymize update CLI, but that flag only exists on uv tool install — uv tool upgrade rejects it with unexpected argument '--refresh-package' found. Every in-UI [Update] click on 0.2.6 / 0.2.7 wrote the rejection into ~/.anonymizer/last_update_failure.json and the next start surfaced the 🟡 "Не удалось обновить" pill instead of the upgrade banner. Both call sites now pass --reinstall-package docs-anonymizer which implies --refresh-package per uv tool upgrade --help.
• current_status no longer hides a fresh update banner behind the update_failed pill. Previously a single failed upgrade attempt routed every subsequent request into state="update_failed" even when a newer release was already on PyPI — exactly the case where the user most wants to retry. The failed pill still shows when the probe sees no newer release.

Changed

• Update banner UX overhaul:
  • Topbar pill and v<version> chip now hide while the banner is on screen — they duplicated the version pair the banner already shows ("v0.2.5 → 0.2.6").
  • Banner text is now v<current> → <latest> · Release notes, with Release notes linking to the canonical notes_url from version.json schema v1. The previous template fell back to network.banner whenever changelog was missing, printing the same text twice.

Fixed

• POST /update/apply now returns an HTML overlay instead of a JSON payload. htmx happily swapped the raw {"status":"shutting_down","message":"Updater spawned; this process will exit."} blob into the banner before. New overlay tells the user "Update started — close this tab, the new version opens in a fresh tab in a few seconds" and renders full-screen.
• _read_update_failure now drops stale last_update_failure.json entries whose reason references the pre-0.2.0 anonymizer distribution name (Caused by: \`anonymizer\` is not installed). These leftovers silently routed current_status into offline_status and kept the update banner hidden after the docs-anonymizer rename. Pilot install on 0.2.5 hit this.

Added

• anonymize update CLI subcommand. Synchronous wrapper around uv tool upgrade --refresh-package docs-anonymizer docs-anonymizer for shell users who don't want to round-trip through the in-UI [Update] banner. Returns the upgrade exit code so it composes with shell scripts.

Fixed

• All upgrade paths now pass --refresh-package docs-anonymizer to uv. Without it --force --reinstall and tool upgrade happily reinstall the same cached old version that PyPI's Simple API has already superseded — a fresh release can sit on PyPI for an hour while every install still resolves the previous version locally. Updated:
  • installer/install.sh and installer/install.ps1
  • the in-UI anonymizer.updater detached process

Fixed

• In-UI update banner now actually fires when a newer version is published. anonymizer.webapp.network_status.probe was reading the non-existent latest key from version.json and silently flipping every probe into offline_status because Version("") raised InvalidVersion. The canonical https://anonymizer.site/version.json schema v1 exposes the key as version (alongside sha256 / url / released_at / schema_version / notes_url); probe now reads version with latest kept as a backward-compat fallback for older internal mirrors. Surfaced by pilot install on 0.2.3 not seeing the banner after the site published 0.2.4. The test_network_status suite now mirrors the real site payload (full v1 keys) so this exact regression can't slip through unit tests again.

Added

• Vendor-neutral agent workflow infrastructure under .agents/skills so Claude Code, Codex, and generic agents share one canonical workflow source. .claude/skills entries are now compatibility shims that delegate to the canonical files, with adapter notes for Claude, Codex, and generic agents.

Fixed

• Agent workflow consistency tests now validate YAML frontmatter with yaml.safe_load. This prevents invalid multiline description fields from silently breaking skill discovery while still keeping Claude shims and canonical .agents skills in sync.

Fixed

• Add setuptools<81 as a hard runtime dependency in [project] dependencies. Without the runtime pin, uv tool install docs-anonymizer end-user environments resolved the latest setuptools (>=81), which dropped the bundled pkg_resources package; the first RU NER call then crashed with::

  File ".../pymorphy2/analyzer.py", line 114, in _iter_entry_points
      import pkg_resources
  ModuleNotFoundError: No module named 'pkg_resources'
  

  Surfaced in the pilot run on the first real RU .docx document. The existing [tool.uv] constraint-dependencies line was a uv-only override and did not propagate into the published wheel metadata. Lifting this pin requires upgrading natasha to a release that uses pymorphy3.

Added

• anonymize --version (and anonymize -V) flag that prints anonymize <version> and exits. Previously the installer scripts' post-install anonymize --version smoke check failed because the flag did not exist.

Fixed

• Installer scripts (installer/install.sh, installer/install.ps1, and the same pair hosted on anonymizer.site) now pin the tool environment to Python 3.12 via uv tool install --python 3.12. Without the pin uv used the user's default interpreter, which on a fresh macOS / Linux box with uv 0.9.x is 3.14 — the upper bound requires-python = ">=3.11,<3.13" added in 0.2.1 was not enough because uv tool install selects the interpreter before resolving the package and only validates the bound at resolve time, which is too late: a tool environment built on 3.14 still attempts to compile blis 0.7.11 from sdist (no cp313/cp314 wheels) and fails on the removed CPython internals.
• docs/installation.md recovery command updated to include the --python 3.12 pin and the --with model URL.

Fixed

• requires-python constrained to >=3.11,<3.13. blis 0.7.11 (transitive via spacy 3.7 → thinc 8.2) only publishes wheels for cp310..cp312; on Python 3.13 / 3.14 uv tool install docs-anonymizer previously fell back to building blis from sdist and crashed against the CPython 3.14 C API (deleted _PyDict_SetItem_KnownHash, new _PyLong_AsByteArray signature, etc.). Reported during pilot install on a machine where uv picked up its uv-managed Python 3.14. Lifting the upper bound requires a coordinated bump to spacy >= 3.8 and a refreshed en_core_web_lg model wheel.

First public release on PyPI under the neutral distribution name docs-anonymizer (the Python import name remains anonymizer). Version stays sub-1.0 because the product is still in pilot stage; promotion to 1.0.0 will follow once pilot feedback is incorporated. The previous internal 1.0.0 tag (built against name = "anonymizer") was never published to PyPI and is kept only as a historical marker. The reservation stub at docs-anonymizer 0.0.1 is superseded by this release.

Detection metrics snapshot

Systematic 62-document corpus (60 synthetic testkit generate --count 60 --seed 42

• 2 golden_partial es/it) after the Tier P0/P1 fix queue below. Baseline was overall recall 0.886 / precision 0.679 / F1 0.769; reproduce via scripts/analyze_corpus.py --corpus /tmp/corpus_systematic --corpus tests/integration/fixtures/golden_partial.

Type	Recall	Precision	F1	Baseline F1
EMAIL	1.000	1.000	1.000	1.000
NUMBER	1.000	1.000	1.000	0.870
PHONE	1.000	1.000	1.000	0.850
DATE	1.000	1.000	1.000	0.790
PERSON	1.000	0.963	0.981	0.750
COMPANY	1.000	0.992	0.996	0.710
ADDRESS	1.000	0.990	0.995	0.400
Overall	1.000	0.991	0.996	0.769

Residual FP (~7 entries) concentrate in PDF parsing artifacts (e.g. Cardiff, ЛюбовьМихайловна — ligature/glue from the parser layer, not detector logic) and are tracked separately in the detector-improvements plan as Fix #12.

Added

• Sprint 7: Feedback hardening + agent infrastructure — anonymize testkit analyze-feedback aggregates PII-free feedback logs into a ranked HTML report; feedback analyzer unit/integration coverage exercises the analyze/reproduce/improve/score loop on synthetic data; agent docs now point at current code paths; local .claude/skills/* runbooks were finalized for feedback triage, known-failure reproduction, detector improvement, regression checking, and patch release; docs/it-policy.md, docs/feedback.md, and docs/sprint-7-pilot-runbook.md document the operational close-out process.
• Sprint 6: Webapp UI redesign — full UI per docs/superpowers/specs/2026-05-18-anonymizer-ui-design.md:
  • Single-page workspace (replaces 3-screen wizard from tech-spec §9)
  • shadcn-style visual language in plain CSS (no React, no node)
  • Side-by-side preview with hover tooltips, pair highlight, sync scroll
  • Manual mask via text selection → token type dropdown
  • Unmask flow with detector_overreach PII-free feedback event
  • Stage-based processing UI ("Reading file" → "Searching PII" → …)
  • Warning banner + error states (unsupported format / corrupted / crash)
  • Settings modal: detector toggles, strict-mode, open-logs-folder
  • Feedback modal with sample/context fields
  • Network status pill (4 states) + update banner template
  • i18n: Russian (extended) + English UI translations via Babel + .po
  • Language picker in topbar (/session/lang endpoint)

Fixed

• DATE detector now drops bare numeric spans that lack any year/month signal in the surrounding 48-character window. dateparser previously tagged fragments like 7/10, 1500, or from 9/10 as DATE entities even when the document used them as ratios, amounts, or unit references; the new _has_date_context guard requires either a 4-digit 19xx/20xx year or an English/Russian month name nearby. Closes Fix #7 of the systematic corpus plan.
• ADDRESS detector now recognises Russian prospect abbreviations written as пр-т and postfixed street-type forms such as Невский пр. and Малый Знаменский пер. in structured г. <City>, <Street>, д. <House> addresses. Surfaced by the systematic testkit corpus plan.
• ADDRESS detector now recognises numbered UK postcode addresses without an explicit street type, alphanumeric UK building numbers such as 221B, and Russian implicit street forms such as г. Москва, Вавилова, д. 19. Structured address spans now outrank contained date-like fragments, so house numbers like 7/10 no longer displace the full address.
• COMPANY detector now recognises Russian legal-form prefixes АНО, АО, ЗАО, ПАО, ОАО, ООО with quoted names and ИП Фамилия И.О. patterns, covering misses from the systematic testkit corpus plan.
• COMPANY legal-form regex spans now use a dedicated conflict precedence so ИП Смирнов А.А. and quoted Russian legal entities survive contained PERSON spans and overbroad NER spans with adjacent taxpayer numbers.
• COMPANY detector now ignores Cyrillic-only spans produced by the English NER path in mixed RU contact blocks, and recognises English business-name forms such as Thornehill Estates, Cardiff Tech Partners, and Ironbridge Logistics Co. from the systematic testkit corpus.
• PERSON detector now ignores Cyrillic-only spans produced by the English NER path in mixed RU/email contact blocks, recognises inverted English names such as Evans Charlotte and King Isabella M., and adds a Russian full-name regex fallback for cases where Natasha splits surname and first-name/patronymic spans. Follow-up filters keep standalone two-word business phrases and Russian institutional title-case phrases out of PERSON. English COMPANY NER now drops unvalidated one-token ORG spans such as Evans that can displace full person names.
• ADDRESS NER fallback now drops noisy single-word Russian locative city mentions such as в г. Москве, while structured г. <City>, <Street> address regex hits remain covered. Phone detection now scans RU, GB, and US national formats with span deduplication, and generated testkit manifests enumerate raw phone/passport surfaces instead of only canonical values.
• COMPANY NER post-processing now drops unvalidated one-token spans, reducing false positives such as DOB, ДР, Evans, and standalone legal-form abbreviations while preserving legal-suffix and legal-form regex matches.
• Contract-number detection no longer treats school/building identifiers such as Школа №57 as contract numbers, preventing nested NUMBER spans from displacing full company-name detections.
• kf-0001: Russian passport detection now handles the № sign separator between series and number (extended passport_ru regex). Surfaced while annotating Рус_ДКП-2025.docx for golden_real corpus.
• kf-0002: ADDRESS detector recognises UK-format addresses (<number> <street> <Square|Crescent|...>, <City> <UK postcode>) and Russian-format addresses (г. <City>, ул. <Street>, д. <House>, кв. <Unit>, including hyphenated compound city names like "Санкт-Петербург"). Conflict-resolution precedence for structured address regex raised above company_en/company_ru, while NER fallback now uses a separate address_ner detector name with lower precedence so single-token LOC spans like "London" cannot evict legitimate company spans like "Acme London Ltd". The RU regex street group is greedy (previously lazy matched only the minimum 2 chars, leaving most of the address unmasked). Passport № separator handles the no-space variant (45 23№786140). On the 12-doc synthetic corpus this lifted ADDRESS recall 0.29 → 0.54, precision 0.23 → 0.48, F1 0.25 → 0.51; overall recall 0.886 → 0.922.

Added

• Repository skeleton: src/anonymizer/ package layout per tech-spec §2.
• Tooling: pyproject.toml with uv, ruff, pyright, pytest configuration.
• CI: pr.yml, main.yml, release.yml workflows per tech-spec §16.
• Binary guard: tests/integration/test_no_network.py enforces no-network rule for anonymizer.core (tech-spec §2, §22).
• Installer skeletons: installer/install.sh and installer/install.ps1 that fail with a clear message because MVP-0 is not yet shipped.
• Sprint 0 complete: skeleton, CI, no-network guard, installer stubs, IT handoff doc. Sprint 1 (core foundations: docx parser/writer + email/phone regex + basic Token Manager) can begin.
• Sprint 1 complete: docx parser (body + headers/footers/footnotes/endnotes; accept-all + remove comments + IRM rejection), docx writer (body/header/footer run replacement plus footnote/endnote XML replacement), EmailDetector (regex) + PhoneDetector (phonenumbers), TokenManager with canonical-key normalization for EMAIL+PHONE, process_document orchestrator, and a working anonymize <file> CLI. Sprint 2 (NER + LanguagePack + xlsx) can begin.
• Sprint 2a complete: LanguagePack abstraction with RU + EN packs (Natasha + spaCy NER); lingua-py language hint on ParsedDocument; canonical-key normalizers for PERSON/COMPANY/ADDRESS/DATE/NUMBER/DATA; 5 NER detectors (PersonRu, PersonEn, CompanyRu, CompanyEn, Address). xlsx, P0 regex detectors (Date + INN/OGRN/KPP/SNILS/PassportRu/IBAN/Card/BIC/BICRu/ContractNumber), and conflict resolution land in Sprint 2b.
• Sprint 2b complete: DateDetector + 10 Russian taxpayer/banking regex detectors (INN, OGRN, KPP, SNILS, PassportRu, IBAN, Card, BIC, BICRu, ContractNumber); xlsx parser/writer; conflict resolution per tech-spec §6; pipeline + CLI accept docx/xlsx; integration coverage for mixed docx P0 masking and xlsx token reuse/comment removal/formula preservation. Sprint 3 (PyMuPDF, metadata cleaner, coverage warnings, strict mode) can begin.
• Sprint 3 complete: PDF text-layer parsing + true redaction via PyMuPDF, metadata cleaners for docx/xlsx/pdf, format-coverage warnings with informational/risky/unsafe tiers, anonymize --strict, and PDF end-to-end coverage for redaction, AcroForm cleanup, and signature warnings.
• PR #4 review hardening: PDF redaction now uses parser geometry instead of page-wide text search, PDF coverage scanning moved out of core.detection, malformed coverage scans fail closed, warning-blocked CLI outputs are removed, and filled PDF form widgets are explicitly removed rather than value-cleared.
• Sprint 4 complete: testkit personas (20: 10 ru + 10 en, all INNs checksum-valid), Generator protocol + DocxGenerator / XlsxGenerator / PdfGenerator (text/manifest deterministic by seed; raw container bytes may carry timestamps), metrics (recall/precision/F1) and corpus runner over docx/xlsx/pdf; corpus-manifest JSONSchema (docs/agents/corpus-manifest-schema.yaml) + manifest reader/validator; feedback-log Pydantic models (ManualMaskAdded / ManualUnmask / DocumentSummary) mirroring §20 with contract test; tests/integration/known_failures/ skeleton + template + discovery harness; golden_partial es/it corpus with full-precision gate (NER silent, regex tier active); golden_generated reproducible synthetic corpus (seed=42, 24 docs); §18 warning-code end-to-end coverage via format_edge_cases integration test; anonymize testkit generate / run / list-known-failures CLI subcommands; hypothesis-driven fuzz on docx pipeline + CI gate activated. Sprint 5 (Web App + Manual mask) can begin.
• Sprint 5 complete: local FastAPI + htmx webapp on 127.0.0.1 (Upload -> Processing SSE -> Preview -> Download), session-token + CSRF + localhost-only CORS middleware, in-RAM session manager, detector toggles, offline network-status stub, manual-mask ManualOverride anchors for docx/pdf/xlsx materialized as first-class entities with conflict precedence, writer regression coverage for manual_override, PII-free crash/feedback JSONL audit logs, anonymize default webapp serve mode plus serve/stop, and integration guards for webapp E2E plus audit no-PII. Sprint 6 (update flow + distribution) can begin.
• Sprint 6 release candidate: detached updater process, real webapp version.json probe with online/update/offline states, POST /update/apply, --no-update-check, finalized install scripts, canonical AGPL-3.0 LICENSE, Sprint 6 release runbook, and acceptance benchmark notes.

Changed

• Language detection (core.language.detection) now recognises Spanish and Italian alongside Russian / English. Documents in es / it surface as language_hint="es" / "it" instead of being silently classified as en. NER detectors (person_ru, person_en, company_*, address) now respect the document-level hint via _document_language_blocks_detector and stay silent on hint mismatches, keeping the regex tier as the sole acceptance surface on partial-language documents.

• Testkit runner now preserves nested corpus output paths, rejects ambiguous duplicate-stem manifests, enforces expected_warnings, and supports multi-corpus CLI runs without writing _out/ into fixture directories.

• Parser exception surface tightened: DocxParser now wraps zipfile.BadZipFile, lxml.etree.XMLSyntaxError, and docx.opc.exceptions.PackageNotFoundError into ParseError. The "missing w:body" branch reclassifies from EncryptedFileError to ParseError (post-PR-2 review feedback).

• Pipeline now resolves cross-detector span overlap before token assignment (greedy leftmost-keep with longer-span tiebreaker on shared left edge); contract documented in pipeline.py. Prepares for Sprint 2 NER landing.

• Pipeline drops the dead AssertionError branch on detect_format: the Literal["docx"] return type already narrows to docx, anything else raises UnsupportedFormatError from within the detector.

• format_detection.detect_format reads only the first 8 magic bytes via a stream (path.open("rb").read(8)) instead of loading the full file into memory — avoids unnecessary spikes on large inputs.

• sanitize_docx_package_bytes now has a typed source: bytes | ZipFile signature (was untyped duck-typing).

• Note block_id format changes from <region>/p_<note_id>_<paragraph_idx> to <region>/p_<note_id>|<paragraph_idx> so the writer can split unambiguously without assuming integer note IDs. The | delimiter is private to footnotes/endnotes block IDs.

• Run Pydantic model marked frozen=True for symmetry with TextSpan / Entity (both already frozen).

Deferred

• Pyright strict vs standard recorded as docs/questions.md Q-025 — default proposal: strict for core/, standard elsewhere; resolve before Sprint 2 NER landing.
• golden_real hand-annotation for 4 documents from docs/examples/ — recorded as docs/questions.md Q-026. Last remaining Sprint-4 task; needs Legal Lead sign-off and lands before the MVP-0 pilot.

Deprecated

• N/A

Removed

• N/A

Fixed

• DOCX output now persists track-change accept-all/comment removal and masks PII detected in word/footnotes.xml / word/endnotes.xml.
• Sprint 5 webapp review hardening: manual-mask overrides now reprocess documents end-to-end, session reset emits the new cookie token, upload paths are sanitized, oversized uploads are rejected, and feedback context snippets redact partial entity fragments.
• PDF token replacements now preserve the source text font family and size instead of shrinking short-source replacements into mismatched labels.
• PDF lines touched by wider replacement tokens are rewritten as a line so following text shifts instead of being overprinted by the token.
• Web processing page now surfaces safe parser failures, including scanned PDFs without a text layer, instead of leaving the progress screen spinning.
• English legal PDFs now avoid masking service headings, section references, schedule labels, duration numbers, suite numbers, and BIC-like words as entities; EINs and full street addresses are masked as single spans.
• English DOCX legal agreements now mask Contract Ref. identifiers, title-case legal-suffix company names, and directional street addresses while preserving role labels, fee headings, business-day ordinals, and routine channel/payment terms.
• Sprint 5 PR review follow-up: marketing/adtech brand names are preserved in service/account contexts, processing SSE reports parse failures before generic progress, feedback responses no longer expose local log paths, and anonymize stop verifies the pidfile target before signaling.
• English company detection now preserves common privacy and data-protection law titles, including TDPSA / CCPA / GDPR-style act and regulation names.
• English company detection now also preserves common operational SaaS/tool and role/reporting terms such as DocuSign, Asana, CMS, CMO, QBR, and AAA rules.

Security

• Local CORS now pins same-origin requests to the served 127.0.0.1 port.

Added (PR #3 — markup-system-foundation)

• Corpus manifest v2 (anonymizer.testkit.manifest_v2) — Pydantic source of truth with per-entity char offsets, block_id anchors, occurrence tracking, tier classification, and JSONSchema regenerated automatically (scripts/regen_corpus_manifest_schema.py).
• anonymize annotate {prepare,compile,render} CLI for detector-first manifest authoring and synthetic-doc round-trip. compile accepts --source {real,synthetic}; synthetic source additionally requires --style-ref and --variant (plus optional --seed, --author) which populate Manifest.generator directly — no manual YAML edit needed.
• Runner v2 with overlap-based matching (≥50% rule + non_entities), CorpusMetrics carrying overall/by_tier/by_type/per-corpus breakdowns, RunReport preserving per-corpus identity.
• Acceptance gates at two layers: per-document manifest.acceptance block and per-corpus <corpus>/acceptance.yml. AcceptanceFailure exits 1; report still written.
• testkit run --only <basename> filter and testkit diff --baseline --current subcommand.
• Markdown run report in working/testkit-runs/<timestamp>/report.md with per-tier + per-type tables and a "P0 misses" section driven by _diagnose_miss heuristic.
• Skills annotate-corpus and generate-synthetic-doc for guided manifest authoring and synthetic-doc production respectively.

Changed (PR #3 — markup-system-foundation)

• Existing golden_partial manifests migrated from v1 ([TYPE, "text"] pairs) to v2 with full block_id + offset structure.
• regression-check skill updated to point at new report location.

Migration

• v1 manifests are no longer accepted by the runner. Existing manifests must be migrated via Manifest.migrate_v1 (see scripts/regen_corpus_manifest_schema.py for the migration pattern).