Preply — language tutors
Personally, I use Preply for language learning. If you want to try an online tutor, our link includes 70% off the trial lesson. We earn a small bonus if you go on to paid lessons - it helps to keep the site running.
Example
Tagalog compounds and hyphenated words
Keep hyphens inside Tagalog compounds (bahay-kubo, tabing-ilog) instead of splitting them at every dash: a tokenizer override demo.
What this example shows
Some languages write a single word with a hyphen inside it. Tagalog does this for compounds like bahay-kubo (a nipa hut) and tabing-ilog (riverbank). This example shows how to keep those compounds whole instead of letting the hyphen break them apart, so the alignment stays clean.
The tokenizer problem
When a hyphen counts as a split character, one word turns into several boxes:
- bahay-kubo nipa+hut
- Split on the hyphen, bahay-kubo becomes three boxes (bahay, the dash, kubo), and the link to English breaks. Kept whole, it is one box that maps to one idea.
- tabing-ilog riverbank
- The same compound pattern: one Tagalog word, one meaning, one box.
The fix: tokenizer override
Turn hyphen off as a split character while keeping the other defaults, so compounds stay as single boxes. On the English side, the join character “+” ties two words into one box, so “nipa+hut” aligns as a unit to bahay-kubo. Both controls live under Settings → Tokens, where you set split characters, the join marker, and how punctuation is treated.
Boundaries and the gloss line
Tokenization decides what counts as one word; morpheme glossing decides how to label its parts. If you add a gloss line for a compound, mark internal morpheme boundaries with a separate character such as the pipe, and keep the hyphen for the compound itself.
For the gloss labels:
See also
More Tagalog and tokenization examples:
- Tagalog reduplication shows a copied syllable marked with a tilde.
- Tagalog verbal aspect paradigm lays out several verb forms together.
Open Preply