Skip to main content

Stemming of Assamese

Stemming is the process of automatically extracting the base form of a given word of a language. Assamese is a morphologically rich, relatively free word order, Indo-Aryan language spoken in North-Eastern part of India that uses Assamese-Bengali script for writing. As it is among the less computationally studied languages, our aim is to extract stem from a given word. We adopt the suffix stripping approach along with a rule engine that generates all the possible suffix sequences.

Algorithm-I
  1. Read a line from the corpus file. 
  2. Extract words (from this point we called it as token) from the line, clean the token, that is remove punctuation marker attached with token if there is one.
  3. Look up suffix-list generated manually from the end of the token. If matched with the suffix-list extract and exit. 
  4. Go to step 1 until the end of the corpus.

Algorithm-II
  1. Read a  line from the corpus file.
  2. Extract words (from this point we called it as token from the line, clean the token, that is remove punctuation marker attached with token if there is one. 
  3. Check the dictionary. If a dictionary entry matches with the token, mark token as root word and exit otherwise execute the next step. 
  4. Look up suffix-list generated manually from the end of the token. If there is a match with the suffix-list extract and exit.  
  5. Go to step 1 until the end of the corpus.
We found 82% accuracy with the suffix stripping approach after adding a root-word list of size 20,000 approximately.

For more information Paper

 

 

Labels:

Comments

Post a Comment

Popular posts from this blog

Assamse Loan words : অসমীয়া ধাৰ কৰা শব্দ

Loanwords are words adopted by the speakers of one language ( target language ) from a different language (the source language ). A loanword can also be called a borrowing words . It is a consequence of sociocultural contact between two language communities. Borrowing of words can go in both directions between the two languages in contact, but often there is an asymmetry, such that more words go from one side to the other. In this case the source language community has some advantage of power, prestige and/or wealth that makes the objects and ideas it brings desirable and useful to the borrowing language community. The following are some example of Assamese loan words and source language. TRANSLATED LOAN WORD বাতৰি কাকত bat ɔri kakɔt (news paper) স্বৰ্ণ যুগ sɔbərnɔ zug (golden era) ৰূপালী জয়ন্তী rupali zɔjənti (silver jubilee) কন্দুৱা গেছ kɔndua ges (tear gas) দূৰদৰ্শন durdɔrχɔn (television) MARATHI WORDS তাঙৰণ taŋɔrɔn (edition) বৰঙণি bɔr
Post a Comment
অধিক পঢ়ক »

Word Formation Process of Assamese

Word formation is the process of creating new words. In other words the ways in which new words are made on the basis of other words or morphemes. Most English vocabulary arises by making new lexemes out of old ones--either by adding an affix to previously existing forms, altering their word class, or combining them to produce compounds. These processes of construction are of interest to grammarians [1]. In this report I will try to write something on “Word formation of Assamese (Asomiya)”. The word Assamese is an English one, built on the same principle like Japanese, Chinese etc. The history and philology of the Assamese language was scientifically studied and presented for the first time by Dr. Banikanta Kakati[2]. The first Assamese grammar, “A Grammar of the Assamese Language” by William Robinson was published in 1839. In 1848, Nathan Brown published an Assamese Grammar, and in 1894, Prof. Nicholl published his “Sketch of Assamese Grammar”. After the work of Dr. Banikant Kakati, a
Post a Comment
অধিক পঢ়ক »

How to write Unicode Assamese (Indian language) text using Latex

TeX হৈছে টাইপছেটিং ছফ্টৱেৰ। ডোনাল্ড নুথে কম্পিউটাৰ বিজ্ঞানৰ চাৰিখন কালজয়ী কিতাপ(দ্যা আৰ্ট অৱ কম্পিউটাৰ প্ৰগ্ৰেমিং, খণ্ড ১,২,৩,৪) লিখিবৰ বাবে এই কম্পিউটাৰ টাইপছেটিং ছফ্টৱেৰটো সাজি উলিয়াইছিল ১৯৭৮চনত। TeX শব্দটো তেওঁলোকে উচ্ছাৰণ কৰে 'টেক' হিচাবে (Techical শব্দটো উচ্ছাৰণ কৰা ধৰণে)। Metafont আৰু Typefaceৰ সহায় লৈ তৈয়াৰ কৰা এই ছফ্টৱেৰটোৰ প্ৰধান উদ্দেশ্য আছিল কম কষ্টত ধুনীয়া কিতাপ লিখা। TeXৰ উত্তৰসূৰী হিচাবে LaTeX, XeTeX, TeTeX, LuaTex ইত্যাদি বহুতো ছফ্টৱেৰ তৈয়াৰ হয়। এই আটাইবোৰ TeXৰ উত্তৰসূৰী বা TeX কৰমুক্ত ছফ্টৱেৰ। এই প্ৰৱন্ধটোত Xetex ব্যৱহাৰ কৰি কেনেকৈ লিখিব পাৰি, তাৰ উদাহৰণ দিয়া হব। (আপুনি Latex ব্যৱহাৰকাৰী বুলি ধৰি লোৱা হব ) XeTeX ব্যৱহাৰ কৰিবৰ বাবে কি কি Package লাগিব?   (Linux ব্যৱহাৰকাৰীৰ বাবে) TeX সম্পাদনা কৰিব পৰা সজুলি, যেনে - LyX  (ঐচ্ছিক)। একো ন'হলে আপুনি vim বা আন Text Editor  যেনে - gEdit, kWrite আদিও ব্যৱহাৰ কৰিব পাৰে।  PDF, dvi আদি  ফাইল চাব পৰা সঁজুলি, যেনে - evince, xPDF ইত্যাদি। tex ফাইলটোত কি কি নতুন অংশ যোগ কৰিব? \usepackage{fontspec} \font\as=&q
4 comments
অধিক পঢ়ক »