Stemming of Assamese

Stemming is the process of automatically extracting the base form of a given word of a language. Assamese is a morphologically rich, relatively free word order, Indo-Aryan language spoken in North-Eastern part of India that uses Assamese-Bengali script for writing. As it is among the less computationally studied languages, our aim is to extract stem from a given word. We adopt the suffix stripping approach along with a rule engine that generates all the possible suffix sequences.

Algorithm-I

1. Read a line from the corpus file. 
2. Extract words (from this point we called it as token) from the line, clean the token, that is remove punctuation marker attached with token if there is one.
3. Look up suffix-list generated manually from the end of the token. If matched with the suffix-list extract and exit. 
4. Go to step 1 until the end of the corpus.

Algorithm-II

1. Read a  line from the corpus file.
2. Extract words (from this point we called it as token from the line, clean the token, that is remove punctuation marker attached with token if there is one. 
3. Check the dictionary. If a dictionary entry matches with the token, mark token as root word and exit otherwise execute the next step. 
4. Look up suffix-list generated manually from the end of the token. If there is a match with the suffix-list extract and exit.  
5. Go to step 1 until the end of the corpus.

We found 82% accuracy with the suffix stripping approach after adding a root-word list of size 20,000 approximately.

For more information Paper