4

Asiya is the machine translation evaluation toolkit to score machine translation outputs (http://asiya.lsi.upc.edu/). It is largely written in Perl.

How do I use Asiya to perform BLEU metrics?

I have followed the youtube introduction video: https://www.youtube.com/watch?v=rA5De9Z4uWI

And created a config file (Asiya.config):

input=raw
srclang=en
srccase=en
trglang=ja
trgcase=ja

src=corpus.tok/test.tok.en
ref=corpus.tok/hyp.tok.ja
sys=corpus.tok/test.tok.ja

some_metrics= BLEU NIST METEOR-ex Ol

My machine translation output file is in corpus.tok/hyp.tok.ja, the source file is in corpus.tok/test.tok.en and the reference file (correct translation) is at corpus.tok/test.tok.ja. They are tokenized plain text files, each line is a sentence.

And when I ran:

/home/expert/asiya/bin/Asiya.pl -eval single -g all -metric_set some_metrics Asiya.config

I got this error:

Smartmatch is experimental at /home/expert/asiya/bin/../lib/IQ/Common.pm line 784.
Smartmatch is experimental at /home/expert/asiya/bin/../lib/IQ/Common.pm line 791.
Smartmatch is experimental at /home/expert/asiya/bin/../lib/IQ/Scoring/ESA.pm line 339.
Use of uninitialized value in concatenation (.) or string at /home/expert/asiya/bin/../lib/IQ/Config.pm line 251.
[ASIYA] directory </tools> does not exist!

The tools directory does exists as a subdirectory in the current directory that I ran the command. What went wrong? Is there a parameter that I could add for the Asiya tools directory?

How do I use Asiya to perform BLEU evaluation?

If I don't use Asiya, how else can I get the BLEU score per sentence and system BLEU scores for my machine translation output?

(more details on http://nlp.lsi.upc.edu/redmine/boards/11/topics/138)

alvas
  • 115,346
  • 109
  • 446
  • 738
  • 1
    The messages about smartmatch and use of uninitialized value are just warnings, although the latter occurring in IQ/Config.pm suggests that something may be missing in your config file. Beyond that, it's hard to diagnose without seeing the actual code...I would look at line 251 of IQ/Config.pm and see if the text "does not exist!" occurs nearby. If you can find the relevant section of code but aren't sure how to debug it, feel free to edit a snippet into your question. – ThisSuitIsBlackNot May 03 '15 at 13:01
  • Did you try setting the `ASIYA_HOME` environment variable like the project developer told you to on their [support forum](http://nlp.lsi.upc.edu/redmine/boards/11/topics/138)? – ThisSuitIsBlackNot Aug 13 '15 at 15:56

0 Answers0