Skip to content

English TN: "per" + a unit symbol has no word boundary: "per day" -> "- per dalton y", "per hour" -> "- per H our" #403

Closed

Description

@dacy

Version: WeTextProcessing 1.0.4.1 (Linux, as locked by IndexTTS 2.5). The code is unchanged in v1.3.0
(tn/english/rules/measure.py, unit_only_graph: pynini.accep("per") + pynutil.delete(" ") followed by
graph_unit).

What happens

tn.english.normalizer.Normalizer().normalize() reads "per" followed by a unit symbol as a bare rate, even in
ordinary prose with no number, and it does not require the unit to be the whole word. "per day" matches the
unit "da" (dalton) and leaves "y" as a separate word. The tagger also prints a stray "-" in front.
We found it through IndexTTS 2.5, whose English front end calls this normaliser and reads the result aloud.

Exact output (captured from IndexTTS's text after normalization: log line on a Linux worker; input
"It happens per ."):

day -> It happens - per dalton y.
hour -> It happens - per H our.
second -> It happens - per S econd.
month -> It happens - per M onth.
minute -> It happens - per M inute.
page -> It happens - per pascal ge.
week -> It happens per week.
year -> It happens per year.
person -> It happens per person.

(A local build with pynini 2.1.7 gives the same shape: "- per dalton y", "- per hour our", "- per second econd",
"- per minute onth" for month, "- per pascal ge".) Sentences from real scripts: "The CI system does about 25
million of them per day." -> "... twenty five million of them - per dalton y."; "10 GB per second" -> "ten
gigabytes - per second econd".

Two causes

  1. No word boundary. DELETE_SPACE is delete(SPACE).star (zero or more), and the tagger is
    (token) + (punct.plus | DELETE_SPACE) repeated, so a token may end in the middle of a word: after the
    measure token "per da" the rest "y" is read as the next token. The bare per + unit branch (weight 1) beats
    "per" + "day" as two plain words (2 x weight 100).
  2. The stray "-". The tagger marks a unit-only measure integer: "-" units: "...". The verbalizer has two ways
    to print it that tie: graph |= delete('integer: "-"') + unit (intended) and (graph_cardinal | ...) + " " + unit
    (prints the "-" as if it were the number). So even a correct "price per kg" comes out "price - per kilogram".

Expected

"It happens per day." stays "It happens per day."; "price per kg" -> "price per kilogram" (no dash);
"5 km/h" and "5 km per hour" unchanged.

A fix that works (tested on 1.0.4.1, 582 real English sentences: 12 change, all repairs)

In measure.py, give the bare per + unit tagger a weight above a plain word but below two plain words, so it wins
only when the unit is the whole word (nothing left over), and make the unit-only verbalizer win the tie:

unit_graph = pynutil.add_weight(<the existing unit_graph expression>, 150)
...
graph = pynutil.add_weight(graph, 1)          # before: graph |= (delete('integer: "-"') + DELETE_SPACE + unit)
graph |= (pynutil.delete("integer: \"-\"") + self.DELETE_SPACE + unit)

Then "per day/hour/second/month/minute/page" stay as written, "per kg/km/ms/min" still expand to kilogram,
kilometer, millisecond, minute, and the dash is gone. A real word boundary (the unit must be followed by a space,
punctuation or the end) would be the cleaner fix; we could not express it as a lookahead in the FST.

Activity

  1. pengzhendong commented on Nov 6, 2026

    @pengzhendong
    Member

    Thanks for the detailed report and reproduction. I verified this with independent PyPI installs, rebuilding the graphs using Python 3.11 / Pynini 2.1.7:

    Input 1.0.4.1 1.3.0
    It happens per day. It happens - per dalton y. It happens per day.
    It happens per hour. It happens - per hour our. It happens per hour.
    price per kg price - per kilogram price per kilogram
    5 km per hour five kilometers - per hour our five kilometers per hour

    Both problems are already fixed in released 1.3.0 and current master. Although the unit-only expression in measure.py remains similar, the surrounding sentence token boundaries and raw-field verbalization have changed, so that expression alone does not reflect the current pipeline behavior.

    For IndexTTS deployments pinned to 1.0.4.1, please update the dependency pin/lock to 1.3.0 and reinstall. I also added 21 targeted regression cases in #405 (now merged), covering prose boundaries, valid rate forms, absence of the spoken dash, n-best outputs, and span mappings. The English TN/ITN suite passes all 676 tests.

    Closing as resolved in the current release. If you can reproduce this with a fresh 1.3.0 installation and rebuilt graphs, please share the input and environment details so we can investigate further.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions