0

I am working on a python NLTK tagging program. My input file is Hindi text containing several lines. On tokenizing the text and using pos_tag the output I get is with NN tag only. but with English sentence as input it does proper tagging. Kindly Help. Version - Python 3.4.1, from NLTK 3.0 documentation

Kindly help! here is what I tried.

word_to_be_tagged = u"ताजो स्वास आनी चकचकीत दांत तुमचें व्यक्तीमत्व परजळायतात."

from nltk.corpus import indian

train_data = indian.tagged_sents('hindi.pos')[:300] 
test_data = indian.tagged_sents('hindi.pos')[301:] 

print(word_to_be_tagged)
print (train_data)

and the output I get is different.

ताजो स्वास आनी चकचकीत दांत तुमचें व्यक्तीमत्व परजळायतात.
[[('पूर्ण', 'JJ'), ('प्रतिबंध', 'NN'), ('हटाओ', 'VFM'), (':', 'SYM'), ('इराक', 'NNP')], [('संयुक्त', 'NNC'), ('राष्ट्र', 'NN'), ('।', 'SYM')], ...]
Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
ASHAY
  • 43
  • 1
  • 3
  • 8
  • My code is - for input as a single sentence `import nltk s=nltk.pos_tag(nltk.word_tokenize("ताजो स्वास आनी चकचकीत दांत तुमचें व्यक्तीमत्व परजळायतात.")) print(s)` Output - >>> [('ताजो', 'NN'), ('स्वास', 'NN'), ('आनी', 'NN'), ('चकचकीत', 'NN'), ('दांत', 'NN'), ('तुमचें', 'NN'), ('व्यक्तीमत्व', 'NN'), ('परजळायतात', 'NN'), ('.', '.')] – ASHAY May 31 '15 at 08:36

1 Answers1

3

The problem is that you should use hindi POS Tagger:

from nltk.corpus import indian
from nltk.tag import tnt

train_data = indian.tagged_sents('hindi.pos')
tnt_pos_tagger = tnt.TnT()
tnt_pos_tagger.train(train_data) #Training the tnt Part of speech tagger with hindi data

print tnt_pos_tagger.tag(nltk.word_tokenize(word_to_be_tagged))

The problem is that a Part Of Speech tagger is accurate in a specific domain (mostly combination of language and topic). In English, most of the words the tagger haven't seen yet are Nouns (NN), it tags you data with NN only.

If you train it with the same domain you want it to tag after (Hindi), it should be OK.

See this for more explanations.

Community
  • 1
  • 1
AvidLearner
  • 4,123
  • 5
  • 35
  • 48
  • On trying the same, the get the following error File "C:\Python34\lib\encodings\cp1252.py", line 23, in decode return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 21: character maps to – ASHAY May 31 '15 at 09:18
  • Well its not solved yet....I am facing difficulty in training the data set.I want some help in that regard.coz depending on the data training, i will be able to tokenise the text.. that is post_tag. Kindly help.! – ASHAY Jun 09 '15 at 14:47