The METEOR MT Evaluation System, Version 0.4.3

April 18, 2005

Satanjeev "Bano" Banerjee (satanjeev AT cmu.edu)
Alon Lavie (alavie AT cs.cmu.edu)

Carnegie Mellon University
Pittsburgh, PA, USA



1. Introduction:
================

METEOR is a system that automatically evaluates the output of machine
translation engines by comparing to them to (one or more) reference
translations. For a given pair of hypothesis and reference strings,
the evaluation proceeds in a sequence of stages, with different
criteria being used at each stage to find and score unigram
matches. By default, at the first stage all exact matches are detected
between the two strings, while in the second stage the words not
matched in the first stage are stemmed using the Porter stemmer and
then matches are found between these stemmed words. 

The matching system is written in Perl, and each matching stage is
implemented as a separate Perl module. In addition to the two default
matching modules (exact matching and stemmed matching), a WordNet
based stemmed matching module and a WordNet based synonym matching
module are also provided with this distribution. METEOR can be run
with the default modules, or the user can override the defaults, and
use one or more of the given modules in any order of
preference. Further, the user can write his own matching module and
plug it into the generic matching system.

METEOR's input file format is exactly the same as those of Bleu and
NIST's Machine Translation Evaluation system. Thus all translation
data that can be evaluated using Bleu (such as the TIDES data) can
also be directly evaluated using METEOR.



2. How to Run METEOR: 
=====================

The default way to run the system is as follows: 

perl meteor.pl -s <system ID> -t <hypothesis file> -r <reference file>

For example, using the sample files included with this distribution,
you can run as follows: 

perl meteor.pl -s SAMPLE_SYSTEM -t test.sgm -r ref.sgm 

By default:
   * the modules being run are "exact" and "porter_stem", in that
	 order
   * the words in the "default_stop_list.txt" file are used as a stop
     list to remove stop-words from both the hypothesis and reference
	 strings


You can change what modules you want to run the matcher with. For
example, to only perform exact matching, run the program as follows: 

perl meteor.pl -s SAMPLE_SYSTEM -t test.sgm -r ref.sgm --modules "exact"

Similarly to use only porter_stem, run the program as follows: 

perl meteor.pl -s SAMPLE_SYSTEM -t test.sgm -r ref.sgm --modules "porter_stem"

To use the WordNet based modules (wn_stem and wn_synonymy) you must
have the following software installed on your machine: 
* WordNet version 2.0 (not higher) (can be downloaded from: http://wordnet.princeton.edu)
* WordNet::QueryData (can be downloaded from: http://search.cpan.org/dist/WordNet-QueryData)

Once these pieces of software are installed in their default
locations, you can use the four modules like so: 

perl meteor.pl -s SAMPLE_SYSTEM -t test.sgm -r ref.sgm --modules "exact porter_stem wn_stem wn_synonymy"


You can also change which file to use as a stop list. For example, if
you have your own stop list in a file called "custom_stop_list.txt",
run the system as follows: 

perl meteor.pl -s SAMPLE_SYSTEM -t test.sgm -r ref.sgm --stop custom_stop_list.txt

If you do not want to run with any stop list at all, not even the
default one, run the system as follows: 

perl meteor.pl -s SAMPLE_SYSTEM -t test.sgm -r ref.sgm --noStop


The meteor.pl program needs to access the various Perl modules (.pm
files) - the simplest way to do this is to always run meteor.pl from
the current directory, and have the modules be in that same
directory. If however you need to put the program and the perl modules
in a different directory than your current working directory, you have
two options: 
1. Either copy the .pm files into Perl's standard library
   directories. Typically these are within Perl's install directory:
   Perl/lib and Perl/site/lib.
2. If you can't or don't want to put .pm files in the above standard
   Perl library directories, you can put them in a different
   directory, and set the environment variable "PERL5LIB" to the path
   to that directory. This has been tested to work on a machine
   running the Windows XP operating system and Perl 5.8.3.



3. Input/Output Format of METEOR:
=================================

The input format of the test and hypothesis files must be the same as
those in the example files included with this distribution. As
mentioned above, this is the same format used in the TIDES data that
the Bleu program uses as input. METEOR expects one hypothesis per
"segment", but can handle multiple reference translatations for each
segment. Segments should be grouped into documents. METEOR will
sequentially compare the hypothesis for each segment of each document
in the test file with all the references for the corresponding segment
in the corresponding document in the reference file. For each such
comparison, METEOR outputs the number of matches found and the number
of chunks found. Once all segments have been scored, overall
statistics are output, including overall precision, recall, 1-factor,
fmean, penalty and score.



4. The mStageMatcher Module:
============================

METEOR performs its stage-by-stage matching using the mStageMatcher.pm
perl module. Specifically, the METEOR program itself simply reads in
all the inputs (commandline arguments, hypothesis/reference
translations, etc) and feeds each hypothesis/reference pair to
mStageMatcher.pm to perform the stage-by-stage matching algorithm
using the input (or default) matching modules.

This mStageMatcher module can be used outside of METEOR. This
distribution provides the standAloneMatcher.pl program that shows how
to use mStageMatcher.pm outside METEOR. Read README_mStageMatcher.txt
for more details. 



5. Licensing:
=============

METEOR is distributed under the following license: 

License Start: 
                   Carnegie Mellon University                      
                     Copyright (c) 2004                            
                      All Rights Reserved.                         
                                                                   
Permission is hereby granted, free of charge, to use and distribute
this software and its documentation without restriction, including 
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of this work, and to    
permit persons to whom this work is furnished to do so, subject to 
the following conditions:                                          
 1. The code must retain the above copyright notice, this list of  
    conditions and the following disclaimer.                       
 2. Any modifications must be clearly marked as such.              
 3. Original authors' names are not deleted.                       
 4. The authors' names are not used to endorse or promote products 
    derived from this software without specific prior written      
    permission.                                                    
                                                                   
CARNEGIE MELLON UNIVERSITY AND THE CONTRIBUTORS TO THIS WORK       
DISCLAIM ALL WARRANTIES WITH REGARD TO THIS SOFTWARE, INCLUDING    
ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS, IN NO EVENT 
SHALL CARNEGIE MELLON UNIVERSITY NOR THE CONTRIBUTORS BE LIABLE    
FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES  
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN 
AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION,        
ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF     
THIS SOFTWARE.                                                     

Author: Satanjeev "Bano" Banerjee satanjeev@cmu.edu
Author: Alon Lavie alavie@cs.cmu.edu

License End.



6. Acknowledgements:
====================

The following researchers have contributed to the implementation of
the METEOR system (all at Carnegie Mellon University): 

Rachel Reynolds
Kenji Sagae
Jeremy Naman
Shyamsundar Jayaraman 
