• 4 min read

Finding root words in a language no NLP library supports

Table of Contents

Building Aklish uncovered a familiar problem. Its dictionary-based spell checker worked well until I tested inflected words. The dictionary contains sueat, not nagsueat, so a lookup treats nagsueat as unknown even though it’s just an inflected form of the same root. Aklanon, like many Philippine languages, changes words through prefixes, suffixes, infixes, and reduplication, so this kept happening.

I had already solved this for Tagalog by building TglStemmer for TagLID. I looked for an existing Aklanon stemmer to integrate into Aklish. There wasn’t one, so I built AklStemmer.

AklStemmer is a Python library that recovers the root form of Aklanon words through rule-based transformations. It removes affixes, reduces repeated syllables, applies phonological rules, and filters the results against a dictionary to find the most likely stem.

Why I built it

AklStemmer started as an internal component of Aklish. Dictionaries mostly contain root words, not every inflected form, and expanding one to cover every combination wasn’t realistic, so recovering the root word first was the cleaner fix. As the project grew, I split the stemmer into its own library so it could be reused outside Aklish.

Aklanon is also my mother tongue, and a low-resource one with almost no NLP tools. I like building software for Filipino languages beyond Tagalog, and this felt like a small step toward that.

How I built it

The overall architecture came from TglStemmer, but I couldn’t just copy the rules. Aklanon has its own morphology, phonological changes, and affix system, so most of the stemming rules had to be rewritten. At a high level, AklStemmer generates possible root words instead of assuming the input already exists in a dictionary.

Each input first goes through tokenization using NLTK so words can be processed individually. From there, the stemmer applies a sequence of rule-based transformations based on Aklanon grammar.

Most of the linguistic rules came from an Aklanon grammar reference. Because there were no existing stemmers to build on, I had to translate those linguistic descriptions into code myself.

Removing affixes

The first step is stripping prefixes, infixes, and suffixes. Like Tagalog, this often requires reversing spelling changes introduced during inflection, so the stemmer applies phonological transformation rules alongside affix removal.

Prefixes:

  • parayaw → dayaw
  • pangablit → kablit
  • pamahaw → bahaw
  • pamasyar → pasyar
  • panumdum → dumdum

Infixes like <in>:

  • sinueat → sueat

Suffixes, which also involve contractions, phoneme changes, vowel loss, and metathesis:

  • eot → eon
  • bayari → bayad
  • sugilanon → sugid
  • agyan → agi
  • tubwan → tubo
  • buksa → bukas
  • islan → ilis

Handling reduplication

The stemmer also recognizes reduplication, another common feature of Aklanon:

Partial reduplication:

  • aabot → abot
  • babakae → bakae

Full reduplication:

  • ano-ano → ano
  • anu-ano → ano
  • ibat-iba → iba

Choosing the best stem

Applying these rules often produces multiple candidates. bukot could generate bukot, buko, or bukon. Instead of taking the first result, AklStemmer scores every candidate, favoring stems that need fewer transformations, remove more meaningful affixes or reduplication, and produce more morphologically plausible words. The highest score wins.

How it improved Aklish

Aklish could now recover root forms before checking the dictionary instead of treating inflected words as unknown, which let the spell checker recognize many words it used to flag incorrectly. Splitting the stemmer into its own library also kept the architecture cleaner: Aklish handles spell checking, AklStemmer handles the morphology.

What I learned

Supporting a low-resource language is a different kind of hard. With TglStemmer I had papers and prior work to lean on. With Aklanon there was nothing, so most of the work was reading a grammar reference and translating it into code by hand, which surfaced rules in my own language I’d never consciously noticed despite speaking it my whole life.

What I’m proudest of isn’t the algorithm, it’s that I built an NLP tool for my own mother tongue. Tagalog already has a much bigger NLP ecosystem than Aklanon. This is a small step toward closing that gap for regional languages.