Telling Tagalog and English Apart Inside the Same Sentence

Taglish is messy.

A single sentence can switch between Tagalog and English several times.

Most language detection tools assume an entire sentence belongs to one language, so they fall apart as soon as the two start mixing.

I ran into this during Grade 12 while working on our Practical Research 2 paper. We’d asked over a hundred students to fill in comic strip dialogue in their own words, and the responses were naturally in Taglish. Part of our analysis required counting how often students switched languages, which we could have labeled by hand, but that would have been painfully slow. Instead, I wrote TagLID.

TagLID is a Python library and command-line tool that identifies the language of each individual word in Taglish text. It uses dictionary lookups as its foundation, then adds rules to handle the kinds of things people actually write: slang, abbreviations, contractions, misspellings, inflected words, interjections, names, and numbers.

From a research script to a reusable tool

The first version wasn’t meant for anyone else, just a script to automate part of our research. Once the research was done, I realized it could be useful beyond that one project, so I cleaned it up, packaged it, added a CLI, and published it as a library anyone could install.

Building a rule-based language identifier

The core idea is simple: if a word appears in the Tagalog dictionary, classify it as Tagalog, and same for English. Reality wasn’t that simple. I built the dictionaries by scraping Pinoy Dictionary and parsing the GCIDE English dictionary, and added frequency lists from the Leipzig Corpora Collection for another signal.

That mattered for words that exist in both languages. Take “at.” In English it’s a preposition, in Tagalog it means “and,” and a dictionary lookup alone can’t tell which one applies since both are technically correct. TagLID compares how frequently the word appears in each language instead and picks the more likely match.

function LangIdentify(word):
    lang = None
    if word in EngDict and word in TglDict:
        if EngFreq[word] > TglFreq[word]:
            lang = "ENG"
        else if TglFreq[word] > EngFreq[word]:
            lang = "TGL"
    else if word in EngDict:
        lang = "ENG"
    else if word in TglDict:
        lang = "TGL"
    return lang

Even that wasn’t enough. People don’t always write dictionary words: they shorten them, misspell them, mix languages, invent slang. Before classification, TagLID runs preprocessing rules to normalize words, skip tokens that don’t belong to either language, and catch cases plain dictionary lookups miss.

Using TagLID

TagLID can be used as either a Python library or a command-line tool.

from taglid.lid import lang_identify, simplify

labeled_text = lang_identify("hello, mundo")
print(simplify(labeled_text))

Output:

[('hello', 'eng'), ('mundo', 'tgl')]

Or from the command line:

python -m taglid.lid

Example input:

hello, mundo

Output:

word      eng    tgl  flag    correction
hello       1      0  DICT
mundo       0      1  DICT

What I learned

This was one of the first projects where programming solved a real problem I had: repetitive work nobody wanted to do, solved by writing a script for it.

On the technical side, it taught me to scrape data, parse XML, package a Python library, and build a CLI. I also learned how fast a simple idea turns complicated. I figured language detection would mostly be dictionary lookups, then kept running into ambiguous words, slang, misspellings, and inflections.

TagLID matters to me because it was the first time I built software not because someone asked me to, but because I wanted to make my own work easier. That was the first time I felt the leverage programming actually gives you.