Skip to content

Read Portuguese given names a particle completes, in two tiers (#1118) - #1120

Merged
derek73 merged 5 commits into
masterfrom
feat/issue-1118-portuguese-particle-given-names
Nov 6, 2026
Merged

derek73 merged 5 commits into
masterfrom
feat/issue-1118-portuguese-particle-given-names

Conversation

@derek73

@derek73 derek73 commented Nov 6, 2026 •

Copy link
Copy Markdown
Owner

Closes #1118.

rules/P10 (a given name a particle completes) read only words marked on Spanish counts, so Portuguese compound given names read the phrase as the start of the surname: Maria das Graças Silva → family das Graças Silva, Maria de Lurdes Costa → family de Lurdes Costa. #1057 had measured Portuguese-speaking countries on ORCID's capped sample (20,000 records a country) and found no word at its 90% bar. This PR re-measures over the whole extract and marks the Portuguese words in two tiers. This moves default parses.

What changes

pt.toml gains given readings written after_particle, each recording its counts. The counts compare a given box ending in a particle run plus the word against a family box opening with one and holding more words, over BR PT AO MZ CV GW ST TL.

Tier Words Read
Default: ≥90% given, ≥10 records lurdes 42:1, céu 31:1, gloria 25:1, tarso 53:3, ribamar 17:1, kássia 15:1, amparo 11:1, glória 38:4 (ceu takes céu's); plus unaccented fatima in es.toml, on its Spanish counts (91:0), as fátima is everywhere
Pack: 75% up to the bar, ≥10 records, default = false cássia 462:64, cassia 170:31, graça 87:20, graças 138:37, kassia 19:5, piedade 13:4 (graca, gracas take their accented words') under locales.PT and locales.PT_BR (a pt-BR choice selects pt.toml)
Left out: under 75% nazaré 70%, dores 66%, penha 62%, rosário 61%, assunção, carmo 53%, conceição 47%, chagas, neves, luz; aparecida (written mostly with no particle) nowhere
Name Before Default after PT / PT_BR after
Maria de Lurdes Costa family de Lurdes Costa middle de Lurdes, family Costa same
José de Ribamar Sousa family de Ribamar Sousa middle de Ribamar same
Maria das Graças Silva family das Graças Silva unchanged middle das Graças, family Silva
RITA DE CASSIA SOUZA family DE CASSIA SOUZA unchanged middle DE CASSIA
Maria das Dores Silva, José do Carmo Souza family unchanged unchanged

Why two tiers. A Brazilian filer often types a surname into the given box: silva scores 32% "given" on the same count. That pulls every Portuguese share down, which is why fátima (83%) and socorro (85%), already marked on their Spanish counts, stand under the bar here. A word at 80% is far above a surname's share but below the default's bar. So it is read where the caller has said the names are Portuguese, and nowhere else. The alternative, the default tier alone, left the issue's own das Graças a surname for every caller.

Accents. _normalize does not fold them, and Brazilian registries write unaccented capitals, so each unaccented spelling is its own entry, measured on its own in each language's countries. So it can part from its accented twin: kassia (79%) is pack-tier where kássia is default.

Measurements

These are from 2026-11-06, against e76cdad3 with --base (a git archive export), on py3.11. The word counts above are over the whole ORCID extract. The ORCID agreement rows are over the capped corpus (raw.jsonl, at most 20,000 records a country, 1,831,533 records). Only 3,223 of those hold a default-tier word and 204 a pack-tier word, so the rows count moves among those.

Population Fixed Broken Changed
local_agreement.py orcid-researchers (capped), default 29 0 14
local_agreement.py orcid-researchers --locale pt_br (capped, both tiers) 56 6 28
local_agreement.py psc-individuals, default 9 2 5
  • TSE, default: snapshot.py --local tse-donors --base moves 1,803 of 854,531 names, in family and middle only (DE FATIMA 1,466, DA GLORIA 115, DE TARSO 73, DE LURDES 47, DE RIBAMAR 43, DO AMPARO 20).
  • TSE, PT_BR: config_diff.py diff --local tse-donors default plain:PT_BR over the 1,950 names holding a pack-tier phrase moves 1,218, nearly all family → middle (CASSIA 568, GRACAS/GRAÇAS 483).
  • What breaks is P10's Accepted cost, a surname written with a marked phrase. ORCID's 6 are da Graça ×4, das Graças and De Cássia; PSC's 2 are Da Gloria Vaz Ceita Daio and De Fatima Zenti.
  • Committed corpora: they hold none of these phrases, so no existing snapshot reading moves. The four new rule examples join corpus_rules.jsonl and the snapshot as new names.

Docs and tests

  • rules/P10: a new Then bullet for the pack tier, with [pt]/[pt_br] examples and boundaries, and the marked-word bullet now names the Portuguese words. rules_budget.json is re-recorded. On axis 12, the new clause reads no new signal: it is the same lookup of the word behind the particle run, and only which readings a language choice turns on differs.
  • decisions/P10: a Standing line, and a 2026-11-06 entry with both tiers, what was left out, the measurements and the recompute recipes. decisions/O7: a head note marks its "no Portuguese-speaking word reaches the bar" clause as superseded.
  • User docs: customize.rst (the marked-word paragraph), locales.rst (a pack-tier paragraph with a doctest, and a new language-packs label), the vocab README's after_particle row, root AGENTS.md's PARTICLE_GIVEN_NAMES bullet, and the release log (folded into the Should a two-word given name like María José or Min Ho fill given, instead of given + middle? #704 bullet, since a user sees one change).
  • Tests: test_es_and_pt_pack_contents now holds the PT surface with the pack-tier words, and _ROTATORS["pt"]/["pt_br"] gain a rotator for each.

Checks: full suite, serial benchmarks, mypy, ruff, build.py --check, sphinx doctest and html (no output), and README doctest all pass. No code changed, only vocabulary, tests and docs.

Design-docs review (round 1, fixed in 6616c59)

All twelve axes ran on fa47db4b. Every count in the entry and the pt.toml notes reproduced on the whole extract, and a grid of 23 shapes × 17 words under every order, switch and pack moved only the P10 shape. Three findings, all fixed:

  • Unaccented fatima had been weighed only on Portuguese counts and sat in the pack tier. On the Spanish counts that put fátima on by default it is 91:0, so María de Fatima López read family de Fatima López beside María de Fátima López's middle de Fátima. It is now a default reading in es.toml. This adds DE FATIMA's 1,466 TSE names to the default reach.
  • Standing stated a looser cut than the Log applied. "Well above a surname's share" would also admit dores and penha, which the entry refused. Standing and the vocab README now state 75% up to the bar with ten given records, and that an unaccented spelling is measured on its own.
  • The ORCID rows didn't name their capped population. They are now labeled, with the reachable population given before the counts.

🤖 Generated with Claude Code

Design-docs review (round 2, of 6616c59; fixed in 1d30f1a)

The reviewer confirmed all three round-1 fixes and recomputed every re-measured number. Four findings, all fixed:

  • Standing ruled out the rare unaccented spellings. ceu, graca and gracas (2–4 given records) take their accented twin's reading. Standing now says a spelling under ten given records does.
  • A strict Portuguese choice reads none of the words es.toml marks. Under parser_for(locales.PT, strict=True), de Fátima, de Fatima, do Socorro, de Lourdes and do Rosario read as family. Moving fatima joined an existing class; it didn't create one. decisions/P10 now records this as accepted, and Should parser_for(locales.PT, strict=True) read Maria de Fátima Costa as middle de Fátima, as the default does? #1122 holds the question.
  • No test pinned the new fatima reading. rules/P10 now has "Maria de Fatima Costa" → middle="de Fatima", which fails with es.toml's reading removed (checked).
  • The 75% cut's stated reason didn't fit its numbers. It is now stated as a chosen threshold, with piedade (76%) and nazaré (70%) on either side.

rules/P10 read only Spanish-measured words, so Maria das Graças Silva and
Maria de Lurdes Costa read the phrase as the start of the surname.
Re-measured over the whole ORCID extract in Portuguese-speaking countries:
eight words reach the 90% bar and are marked by default (lurdes, céu,
glória, tarso, ribamar, kássia, amparo); seven more, below the bar but well
above a surname's share, are marked default = false in pt.toml, read under
locales.PT and locales.PT_BR alone (graças, graça, cássia, fatima, piedade).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@derek73 derek73 added enhancement vocabulary Per-word vocabulary model and language packs labels Nov 6, 2026
@derek73 derek73 self-assigned this Nov 6, 2026
@derek73 derek73 added this to the 3.0 milestone Nov 6, 2026
derek73 and others added 4 commits November 6, 2026 14:40
- Mark unaccented fatima on its Spanish counts (91 to 0), on by default in
  es.toml as fátima is, where it had sat in pt.toml's pack tier on its
  Portuguese counts alone; María de Fatima López read family 'de Fatima
  López' beside María de Fátima López's middle 'de Fátima'.
- State the pack tier's actual cut (75% up to the bar, ten given records)
  in decisions/P10's Standing and the vocab README, and record that an
  unaccented spelling is measured on its own (kassia parts from kássia).
- Label the ORCID agreement rows as the capped corpus and give the
  population that could move; re-measure every row after the fatima move.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
- Standing: an unaccented spelling under ten given records takes its
  twin's reading (ceu, graca, gracas), as the Log and pt.toml say.
- Record that a strict Portuguese choice does not read the words es.toml
  marks (de Fátima, de Fatima, do Socorro, de Lourdes), accepted here.
- Pin unaccented Fatima's default reading with a rules/P10 example; it
  fails with es.toml's reading removed.
- State the 75% cut as a chosen threshold, with the words either side.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…tuguese-particle-given-names

# Conflicts:
#	tests/v2/test_locales.py
@derek73
derek73 enabled auto-merge (squash) November 6, 2026 23:08
@derek73
derek73 merged commit f2afb7b into master Nov 6, 2026
13 checks passed
@derek73
derek73 deleted the feat/issue-1118-portuguese-particle-given-names branch November 6, 2026 23:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement vocabulary Per-word vocabulary model and language packs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Should Maria das Graças Silva read middle das Graças, family Silva, as María del Carmen García does?

1 participant