core/persian.py
Rama main · réplica de lectura del repositorio original.
# SPDX-License-Identifier: BUSL-1.1
# Copyright (c) 2024-2026 Ali Sadeghi Aghili
"""Canonical Persian text normalisation for matching and cache-key equality.
Why this module exists
-----------------------
Persian text reaches this codebase through several input methods, and two
spellings of the same word are common:
* **Digits**: Persian (۰-۹, U+06F0-06F9), Arabic-Indic (٠-٩, U+0660-0669),
and ASCII (0-9) digits all appear depending on keyboard/IME.
* **ي/ك vs ی/ک**: the Arabic-form YEH (ي, U+064A) and KAF (ك, U+0643) are
frequently typed in place of their Persian equivalents (ی U+06CC, ک
U+06A9) -- both are valid input, neither is "wrong".
* **ZWNJ** (U+200C, zero-width non-joiner): used inside Persian compound
words (میخواهم) but often omitted or replaced with a plain space
(می خواهم / میخواهم) depending on typing habits.
* **Whitespace**: run-length and placement varies with no change in
meaning.
* **Unicode form**: the same visible word can arrive as different
codepoint sequences -- a composed letter (U+0622 ALEF WITH MADDA ABOVE)
vs. its decomposed form (ALEF + COMBINING MADDA ABOVE), or as an Arabic
*presentation form* / ligature (U+FE8D ALEF ISOLATED FORM, U+FDF2 the
ALLAH ligature) rather than the plain base letters. Presentation forms
and ligatures are not exotic here -- they are exactly what text copied
out of a PDF or an older Windows application (this project's actual
input environment) tends to contain.
Before this module existed, each call site that needed to compare Persian
text for equality (the query cache's key, the value retriever's filter
extractors, the schema retriever's TF-IDF tokenizer) carried its own
partial translation table, and no two of them agreed. Concretely: the
cache folded ي/ك and ZWNJ so two spellings hashed to the *same* cache key,
while the value retriever did not fold them when extracting filters --
so the *same* cached answer could be served for two spellings that the
retriever itself would have turned into two different SQL queries. This
module is the single place that decides what "the same Persian text"
means, so every call site agrees.
Two entry points, not one
--------------------------
:func:`normalize_for_matching` collapses whitespace to single spaces --
it is for comparing text where word *boundaries* still matter (a
multi-word cache key, a multi-word alias such as "تالار پتروشیمی" checked
as a substring of a question).
:func:`normalize_compact` goes one step further and removes whitespace
entirely. It exists for matching a *fixed, known* vocabulary token
(a month, weekday, or season name) that a user may type as one word, two
words, or ZWNJ-joined -- "پنج شنبه" / "پنجشنبه" / "پنجشنبه" must all
resolve to the same weekday, and none of the three spellings is any more
"correct" than the others. Using the whitespace-collapsing form here
would leave "پنج شنبه" unmatched against the canonical "پنجشنبه".
Both build on the same folding rules; :func:`normalize_compact` simply
strips what :func:`normalize_for_matching` leaves as a single space.
Note on scope: this module folds only the rules already present
somewhere in this codebase before unification, plus Unicode
NFKC normalisation (added after it was measured that plain NFC misses
Arabic presentation forms and ligatures -- see the ordering note on
:func:`normalize_for_matching`). It is deliberately not the place to add
further coverage (e.g. diacritic stripping) -- that is a separate
decision with its own tradeoffs.
Helpers that are not normalisation
-----------------------------------
Two later helpers live here because they are about Persian text, but they are
**not** part of :func:`normalize_for_matching` and change none of its output
(so :data:`NORMALIZER_VERSION` and the query cache's keys are untouched):
* :func:`letter_form_key` folds only the letter variants a stored value can
differ from a typed one by (Arabic ي/ى/ك, presentation forms, ZWNJ), for
deciding which stored value a filter literal was meant to be
(:mod:`retrieval.filter_spelling`). It never folds case, digits or spaces.
* :func:`strip_plural_suffix` reduces a word written with an attached plural
(تالارها) to its stem, for matching a question against aliases and trigger
phrases. Callers use the stem *in addition to* the word, never instead.
Cache-key version
------------------
:data:`NORMALIZER_VERSION` is bumped whenever this module's OUTPUT changes
for any input (not merely its API surface). Any consumer that builds a
persistent or cross-run key from :func:`normalize_for_matching`'s output
must fold this into that key, so a version bump can never let a stale
entry, produced under the old folding rules, collide with a new one --
see ``api.query_cache.QueryCache._question_key``.
"""
from __future__ import annotations
import re
import unicodedata
from collections.abc import Iterable
#: Persian digits (U+06F0-06F9) and Arabic-Indic digits (U+0660-0669)
#: mapped to ASCII 0-9, so "۱۴۰۲", "١٤٠٢", and "1402" all fold alike.
#: Unaffected by NFKC (verified: NFKC leaves both digit scripts alone),
#: so this table's correctness does not depend on running before or
#: after the NFKC step below.
_DIGIT_MAP = {
**{chr(0x06F0 + i): str(i) for i in range(10)}, # Persian
**{chr(0x0660 + i): str(i) for i in range(10)}, # Arabic-Indic
}
#: Arabic-form letters folded to their Persian equivalents: ي (U+064A) ->
#: ی (U+06CC), ك (U+0643) -> ک (U+06A9). Both spellings are common in
#: Persian text depending on input method/keyboard.
_LETTER_MAP = {"ي": "ی", "ك": "ک"}
#: The one and only translation table in this codebase (digit folding and
#: letter folding combined into a single `str.translate` pass -- the two
#: maps share no keys, so combining them is equivalent to applying either
#: order separately).
_TRANSLATION_TABLE = str.maketrans({**_DIGIT_MAP, **_LETTER_MAP})
#: Zero-width non-joiner -- common inside Persian compound words
#: (میخواهم) but irrelevant to text equality; stripped outright.
#: Unaffected by NFKC (verified by execution), so, like the digit map,
#: its position relative to the NFKC step does not matter.
_ZWNJ = ""
_WHITESPACE_RE = re.compile(r"\s+")
#: Bumped whenever normalize_for_matching's OUTPUT changes for any input.
#: See the module docstring's "Cache-key version" section.
NORMALIZER_VERSION = "2"
def normalize_for_matching(text: str) -> str:
"""Fold *text* to a canonical form for equality/substring comparison.
Applies, in order: Unicode NFKC normalisation, Persian/Arabic-Indic
digit folding, ي/ك -> ی/ک letter folding, ZWNJ removal, whitespace
collapsing (runs of whitespace become a single space), and ASCII
lowercasing. None of this changes the *meaning* of the text -- it
only removes differences a reader would never notice between two
spellings of the same question or name.
This is the form used for: cache-key equality, substring/alias
matching (e.g. ring names), and any other comparison where word
boundaries (single spaces) still matter.
Ordering is load-bearing: NFKC MUST run before the ي/ك letter fold,
not after. NFKC maps an Arabic *presentation form* of YEH (e.g.
U+FEF1 ARABIC LETTER YEH ISOLATED FORM) to the plain *Arabic* base
letter ي (U+064A) -- never directly to the Persian ی. Only a fold
applied AFTER NFKC reaches ی. Folding first and running NFKC second
leaves a presentation-form YEH as Arabic ي, permanently unmatched
against Persian-letter text. Verified by execution: with the correct
order, U+FEF1 folds to U+06CC (ی); reversed, it lands on U+064A (ي).
Do not reorder these two steps.
Examples
--------
>>> normalize_for_matching(" خرید در ۱۴۰۲ ")
'خرید در 1402'
>>> normalize_for_matching("خرید در ١٤٠٢") == normalize_for_matching("خرید در 1402")
True
>>> normalize_for_matching("علي") == normalize_for_matching("علی")
True
>>> normalize_for_matching("كارگزار") == normalize_for_matching("کارگزار")
True
>>> normalize_for_matching("میخواهم") == normalize_for_matching("میخواهم")
True
>>> normalize_for_matching("Top Customers") == normalize_for_matching("top customers")
True
NFKC folds a composed letter and its decomposed (base + combining
mark) spelling to the same form -- e.g. ALEF WITH MADDA ABOVE
(U+0622) vs. ALEF (U+0627) + COMBINING MADDA ABOVE (U+0653):
>>> normalize_for_matching("\u0622") == normalize_for_matching("\u0627\u0653")
True
It also folds an Arabic presentation form / ligature -- text pasted
from a PDF or an older Windows application often arrives this way --
to its plain spelled-out equivalent, e.g. the isolated ALEF
presentation form (U+FE8D) vs. plain ALEF (U+0627), and the ALLAH
ligature (U+FDF2) vs. "الله" spelled out with four base letters:
>>> normalize_for_matching("\ufe8d") == normalize_for_matching("\u0627")
True
>>> normalize_for_matching("\ufdf2") == normalize_for_matching("الله")
True
"""
text = unicodedata.normalize("NFKC", text)
text = text.translate(_TRANSLATION_TABLE)
text = text.replace(_ZWNJ, "")
text = _WHITESPACE_RE.sub(" ", text).strip()
return text.lower()
def normalize_compact(text: str) -> str:
"""Like :func:`normalize_for_matching`, but removes ALL whitespace.
For matching a fixed vocabulary token (month/weekday/season name)
that may be typed as one word, two words, or ZWNJ-joined -- spacing
is not meaningful for these tokens, only the letters are.
Examples
--------
>>> normalize_compact("پنج شنبه") == normalize_compact("پنجشنبه")
True
>>> normalize_compact("پنجشنبه") == normalize_compact("پنجشنبه")
True
"""
return _WHITESPACE_RE.sub("", normalize_for_matching(text))
#: The letters a stored value and a typed one can differ by, one tuple per
#: letter, the Persian form first. ي (U+064A) and ى (U+0649, alef maksura) for
#: ی (U+06CC); ك (U+0643) for ک (U+06A9). Deliberately short: ة/ه, ۀ/هٔ and the
#: alefs are different letters or optional marks, not two spellings of one
#: letter, and a digit script is a separate decision
#: (:mod:`retrieval.filter_spelling` explains each exclusion).
LETTER_FORM_GROUPS: tuple[tuple[str, ...], ...] = (
("\u06cc", "\u064a", "\u0649"),
("\u06a9", "\u0643"),
)
_LETTER_FORM_TABLE = str.maketrans({
variant: group[0] for group in LETTER_FORM_GROUPS for variant in group[1:]
})
#: Arabic Presentation Forms-A (U+FB50-FDFF) and -B (U+FE70-FEFF): what text
#: pasted out of a PDF or an old Windows application contains.
PRESENTATION_FORMS_RE = re.compile("[\ufb50-\ufdff\ufe70-\ufeff]")
def letter_form_key(text: str) -> str:
"""The key under which two spellings of one stored value are equal.
Folds, and only folds: Arabic presentation forms and ligatures to their
base letters (NFKC, applied to those characters alone, so nothing else in
the text is rewritten), canonical composition (NFC), ي/ى -> ی and ك -> ک
(:data:`LETTER_FORM_GROUPS`), and ZWNJ removed. Case, digits, whitespace and
every other character are kept as they are, which is what makes this
narrower than :func:`normalize_for_matching`: two texts with the same key
are the same words spelled with other letter forms, nothing more.
Parameters
----------
text:
A stored value or a typed literal.
Returns
-------
str
The key. Equal keys mean "the same text up to letter form".
Examples
--------
>>> letter_form_key("علي") == letter_form_key("علی")
True
>>> letter_form_key("كتاب") == letter_form_key("کتاب")
True
>>> letter_form_key("می\u200cخواهم") == letter_form_key("میخواهم")
True
>>> letter_form_key("\ufef1") == letter_form_key("ی")
True
Nothing else is folded:
>>> letter_form_key("Ali") == letter_form_key("ali")
False
>>> letter_form_key("۱۴۰۲") == letter_form_key("1402")
False
>>> letter_form_key("تالار ها") == letter_form_key("تالارها")
False
>>> letter_form_key("ة") == letter_form_key("ه")
False
"""
text = PRESENTATION_FORMS_RE.sub(lambda m: unicodedata.normalize("NFKC", m.group()), text)
text = unicodedata.normalize("NFC", text)
return text.translate(_LETTER_FORM_TABLE).replace(_ZWNJ, "")
#: Attached plural endings, longest first: the plural alone, the plural with
#: the indefinite or the ezafe ی, and the plural with a possessive pronoun.
_PLURAL_SUFFIXES = (
"هایشان", "هایمان", "هایتان", "هایش", "هایم", "هایت", "هایی", "های", "ها",
)
#: A stem shorter than this is not a word the suffix was attached to.
_MIN_STEM = 3
def strip_plural_suffix(token: str) -> str:
"""The stem of a word written with an attached Persian plural, else *token* itself.
For a word of the **normalised** text (the half-space is already gone, so
``تالارها`` arrives as ``تالارها``). Strips one ending,
``ها``, ``های``, ``هایی`` or ``ها`` with a possessive pronoun (``هایم``,
``هایت``, ``هایش``, ``هایمان``, ``هایتان``, ``هایشان``), and only when
what is left is at least three characters, all of them letters. Not stripped, on purpose:
``ان`` and ``ات``, which end names and ordinary words that are no plural
(``تهران``, ``ایران``, ``حیات``), and anything in Latin script. A word that
ends in ``ها`` without being a plural but has a long enough stem is
reduced all the same (``انتها`` -> ``انت``); that is why the stem is only
ever used **in addition to** the word, where an extra token that matches
nothing costs nothing.
Parameters
----------
token:
One word, already normalised (:func:`normalize_for_matching`).
Returns
-------
str
The stem, or *token* unchanged when no rule applies.
Examples
--------
>>> [strip_plural_suffix(t) for t in ("تالارها", "تالارهای", "تالارهایی", "تالارهایش", "تالارهایشان")]
['تالار', 'تالار', 'تالار', 'تالار', 'تالار']
The half-space form is the same word once normalised:
>>> strip_plural_suffix(normalize_for_matching("تالار\u200cها"))
'تالار'
A word that is not a plural, a stem too short to be a word, and Latin script
are left alone:
>>> [strip_plural_suffix(t) for t in ("تالار", "تهران", "ایران", "حیات", "ها", "بها", "های", "tables", "ها1")]
['تالار', 'تهران', 'ایران', 'حیات', 'ها', 'بها', 'های', 'tables', 'ها1']
"""
if token.isascii():
return token
for suffix in _PLURAL_SUFFIXES:
if token.endswith(suffix):
stem = token[: -len(suffix)]
return stem if len(stem) >= _MIN_STEM and stem.isalpha() else token
return token
def strip_plural_suffixes(tokens: Iterable[str]) -> list[str]:
"""Each of *tokens* reduced by :func:`strip_plural_suffix`, in order.
The result has as many words as *tokens* and equals it when no word
carries an attached plural, so a caller can compare the two and add the
second only when they differ. Phrases of several words keep their
adjacency (``تالارهای کالا`` -> ``تالار کالا``), which is what a bigram or a
phrase match needs.
Parameters
----------
tokens:
Words of the normalised text.
Returns
-------
list[str]
The stems, same length and order.
Examples
--------
>>> strip_plural_suffixes(["تالارهای", "صادراتی", "تهران"])
['تالار', 'صادراتی', 'تهران']
>>> strip_plural_suffixes(["total", "sales"])
['total', 'sales']
"""
return [strip_plural_suffix(token) for token in tokens]