Building TagLID exposed a problem I didn’t expect. Its language identifier relied on dictionary lookups to tell Tagalog from English, and worked well until I tested words people actually use. The dictionary contains sulat, not nagsulat, so a lookup treats nagsulat as unknown even though it’s just an inflected form of the same root. Tagalog words change heavily through prefixes, suffixes, infixes, repeated syllables, and other morphological rules, so this kept happening.
I looked for a Tagalog stemmer that could recover the root word before lookup. I couldn’t find one, so I built TglStemmer.
TglStemmer is a Python library that recovers the root form of Tagalog words through rule-based transformations. It removes affixes, reduces repeated syllables, applies phonological rules, and filters the results against a dictionary to find the most likely stem.
Why I built it
TglStemmer started as a supporting tool for TagLID, built to recognize inflected words for our Grade 12 research project. Expanding the dictionary to cover every word form wasn’t practical, so recovering the root word first was the better approach. As it grew, I split it into its own library so it could be useful outside TagLID.
How I built it
At a high level, TglStemmer works by generating possible root words instead of assuming the input already exists in a dictionary.
Each input first goes through tokenization using NLTK so words can be processed individually. From there, the stemmer applies a sequence of rule-based transformations, each responsible for handling a different part of Tagalog morphology.
Removing affixes
The first step is stripping prefixes, infixes, and suffixes. This isn’t just deleting characters, since attaching an affix often changes the spelling of the original word, so the stemmer applies transformation rules to recover the root correctly.
Prefixes, involving phoneme changes and assimilation:
parami→damipangailangan→kailanganpamigay→bigaypamagitan→pagitanpanamit→damitpanigarilyo→sigarilyopanahi→tahi
Infixes such as <in>:
sinulat→sulat
Suffixes, beyond removing endings like -an and -in, also involve contractions, phoneme changes, vowel loss, and metathesis:
bayaran→bayadtauhan→taoinuman→inomkingkihan→kingkepaitin→paetbuksan→bukastamnin→tanim
Handling reduplication
Another feature of Tagalog is reduplication, where part or all of a word is repeated. The stemmer recognizes both forms.
Partial reduplication:
aalis→alisbibili→bili
Full reduplication:
ano-ano→anoanu-ano→anoiba't-iba→iba
Choosing the best stem
These transformations often produce more than one possible root. pinakamahusay could produce husay, mahusay, or pinakamahusay. Instead of picking the first result, the stemmer ranks every candidate with a custom scoring system, favoring stems that need fewer transformations, remove more meaningful affixes or reduplication, and produce more morphologically plausible words. The highest score wins.
How it helped TagLID
TagLID could now recover root forms before dictionary lookup instead of treating inflected words as unknown, and it handled mixed Taglish surprisingly well: nagwork becomes work after stripping the Tagalog prefix. It also kept TagLID simple, relying on a separate component for stemming instead of an ever-growing dictionary of inflected words.
Using TglStemmer
TglStemmer can be used as a Python library by importing it into a Python script.
from tglstemmer import stemmer
stem = stemmer.get_stem("nagsulat")
print(stem.word)
# sulat
print(stem.pre)
# nag
print(stem.suf)
# NoneWhat I learned
This project changed how I thought about language processing. I assumed stemming mostly meant chopping off prefixes and suffixes, but the more examples I tested, the more exceptions I found. It also changed how I think about building software: TglStemmer could have stayed an internal utility, but splitting it into its own library made both projects better, letting the stemmer handle Tagalog morphology while TagLID focused on identification.
Recently a researcher opened a pull request after using TglStemmer in their own work, a nice reminder that a tool built for one problem can end up helping someone else with theirs.