***********************************************************************
TOOL:        IQMT Framework for MT Evaluation
AUTHOR(s):   Jess Gimnez and Enrique Amig
DATE:        14/03/2006
VERSION:     1.2
DESCRIPTION: Inside Qarla MT evaluation
***********************************************************************


Contents
--------

      README _ this file.
     INSTALL _ instructions to install this software.
 Makefile.PL _ configuration script.
    MANIFEST _ list of software components included in the package.
     CHANGES _ list of changes w.r.t the previous version.
    LGPL.txt _ license terms.
       ./lib _ IQ library.
       ./bin _ executable files:
               * IQsetup  _ setup component.
               * IQeval   _ evaluation component.
   ./samples _ sample files:
               * IQsetup.config file sample.
               * IQeval.config file sample.
               * IQREPORT XML representation format sample.

Description
-----------

The IQMT package is a Machine Translation Evaluation Software which
evaluates automatic translations according to their 'human-likeness'.

IQMT is based on the QARLA Framework (Amig et al., ACL'2005), a
probabilistic framework originally designed for the evaluation of text
summarization systems. QARLA automatically identifies the features
that distinguish human translations from automatic ones. It permits
metric combinations, without any a-priori weighting of their relative
importance. Besides, no training or adjustment of parameters is
required, there is no need for human assessments, and it does not
depend on the scale properties of the metrics being evaluated.

QARLA provides i) a measure to evaluate the quality of any set of
similarity metrics (KING), ii) a measure to evaluate the quality of a
translation using a set of similarity metrics (QUEEN), and iii) a
measure to evaluate the reliability of a test set (JACK).  The first
release of the IQMT package is freely available for public use.

Using IQMT offers a number of advantages over previous MT evaluation
packages. The most important is that it you can define a set of metrics and
then combine them into a single measure of MT quality, in a robust and
elegant manner, avoiding scaling problems and metric weightings.
This permits to avoid the 'metric bias' problem, by allowing you to tune
your system on a combination of metrics instead of on a single metric.

Several well-known and freely available current MT evaluation tools have
been incorporated so far:

  - ROUGE v 1.5.5 (http://www.isi.edu/~cyl/ROUGE/)
  - GTM v 1.2 (http://nlp.cs.nyu.edu/GTM/)
  - METEOR v 0.4.3 (http://www.cs.cmu.edu/afs/cs.cmu.edu/user/alavie/www/METEOR/)
  - BLEU/NIST mteval-kit v 11b (http://www.nist.gov/speech/tests/mt/resources/scoring.htm)


Moreover, IQMT allows the user to supply its own metrics. For future
releases, we are working on the design of new metrics that are able to
capture linguistic aspects of translation beyond lexical ones.

IQMT COMPONENTS:
================

IQMT consists of two main components:

    * IQsetup
    * IQeval


---------------------------------------------------------------------
(1) IQsetup
---------------------------------------------------------------------

The IQsetup component is responsible for applying a given set of
metrics to a given set of translations by different systems. Thus it
requires a set of human references.

Given a 'setup' config file, it generates a series of XML files
contaning MT evaluation scores for each metric and each pair:

  * SYSTEM*-REFERENCE*
  * REFERENCE*-SYSTEM*
  * SYSTEM*-SYSTEM*
  * REFERENCE*-REFERENCE*

and an 'evaluation' config file in a format convenient for the IQeval
component.


---------------------------------------------------------------------

Usage : IQsetup  [options]  <IQsetup.config>  <IQeval.config>
                                (input)          (output)
 
  - print         : print similarities onto IQeval.config (default disabled)
  - remake        : remake reports
  - V <0|1|2>     : verbosity
                    0 - non-verbose (default)
                    1 - low verbosity
                    2 - medium verbosity
 
Example: IQsetup IQsetup.config IQeval.config

---------------------------------------------------------------------

The Setup Config File must specify several variables:


# EXPERIMENT NAME ---------------------------
NAME=TCSTAR_dev1_spa2eng

# IQMT LOCATION -----------------------------
IQMT=/home/users/me/IQMT/

# FILES -------------------------------------
source=source_file.txt
ref=reference_file.txt.1
...
ref=reference_file.txt.M
system=system_output_file.txt.1
...
system=system_output_file.txt.N

# AVAILABLE METRICS -------------------------
doBLEU
doNIST
doGTM
doMETEOR
doROUGE
#doBLEU BLEU-1 BLEU-2 BLEU-3 BLEU-4
#doNIST NIST-1 NIST-2 NIST-3 NIST-4 NIST-5
#doGTM GTM-1 GTM-2 GTM-3
#doMETEOR MTR-exact MTR-stem MTR-wnstm MTR-wnsyn
#doROUGE RG-1 RG-2 RG-3 RG-4 RG-L RG-W-1.2 RG-S* RG-SU*
#--------------------------------------------

You may find an example in the './samples' directory.

Notice that you can select which of the available metrics
you want to use among a set of 24 variants from 5 different
metric families:

* doBLEU (n-gram levels = 1..4)
* doGTM (e=1..3)
* doNIST (n-gram levels = 1..5)
* doMETEOR ('exact', + 'stemming', + 'Wordnet stemming', + 'WordNet synonyms'
* doROUGE (n-gram levels = 1..4, as well as LCS, S*, SU*, W-1.2 variants)

---------------------------------------------------------------------
(2) IQeval
---------------------------------------------------------------------

The IQeval component computes QUEEN / (IQ) scores on top of the scores
generated by IQsetup.It allows the user to choose the set of
metrics, systems, references and segments to be used.
The user may define these sets in the IQeval config file.

Two levels of granularity are allowed. Results may be shown at the
system level (useful for tuning and system comparison) and at the
segment level (crucial for error analysis). Moreover, two kinds of
reports (score matrix / ranking lists) may be generated. The 'score
matrix' report allows to compare the scores by different metrics on
the same system/segment. The 'ranking lists' report provides lists of
rankings according to each metric individually. It is specially useful
for the study of correlations.


---------------------------------------------------------------------

Usage : IQeval  [options]  <IQeval.config>
 
  - doKING              : compute KING score (default disabled)
  - doKINGtt            : compute KING-TT score (default disabled)
  - doQUEEN             : compute QUEEN score (default disabled)
  - doQUEENtt           : compute QUEEN-TT score (default disabled)
  - doJACK              : compute JACK score (default disabled)
  - doJACKtt            : compute JACK-TT score (default disabled)
  - doIQ                : compute IQ score (default disabled)
  - doIQtt              : compute IQ-TT score (default disabled)
  - doOQ                : compute individual metric scores outside QARLA (default disabled)
  - optimizeKING        : find a metric set of suboptimal KING score (default disabled)
  - optimizeKINGtt      : find a metric set of suboptimal KING-TT score (default disabled)
  - doPking             : compute P_King (default disabled)
  - doPorange           : compute P_Orange (default disabled)
 
  - M <metrics>         : set of metrics (setname according to IQeval.config)
                          ['metrics' by default]
  - S <systems>         : set of systems (setname according to IQeval.config)
                          ['systems' by default]
  - R <references>      : set of references (setname according to IQeval.config)
                          ['references' by default]
  - T <segments>        : set of segments (setname according to IQeval.config)
                          ['segments' by default]
  - G <sys|seg>         : level of granularity
                          sys - system-level scores (default)
                          seg - segment-level scores
 
  - doref               : include reference scores as well (default disabled)
  - remake              : remake metric computations
  - O <0|1>             : output format
                          0 - score matrix (default)
                          1 - ranking lists (on a metric basis)
  - V <0|1|2>           : verbosity
                          0 - non-verbose (default)
                          1 - low verbosity
                          2 - medium verbosity
 
Example: IQeval -V 1 -doOQ -doQUEEN -M metrics_BLEU-4 -S systems_S0 -G seg -O 1 IQeval.config
 

---------------------------------------------------------------------

The Eval Config File is generated automatically.  It is intended for
the user to define new metric sets, system sets, and reference sets,
to be used by the IQeval component. You may find an example in the
'./samples' directory.

---------------------------------------------------------------------


ADDING NEW METRICS
==================

Additionally, the IQMT package allows the user to supply its own
user-defined metrics by means of the IQREPORT XML representation
schema. See an example (you may find it in the './samples' directory, too):

<?xml version="1.0"?>
<!DOCTYPE iqmt SYSTEM "iqmt.dtd" []>
<IQREPORT metric="MTR-wnsyn" ref="R2" score="0.6307" target="R0">
  <S n="1">0.9960</S>
  <S n="2">0.6250</S>
  <S n="3">0.8519</S>
  (...)
  <S n="502">0.7216</S>
  <S n="503">0.5222</S>
  <S n="504">0.5962</S>
</IQREPORT>


Filenames are important. They must follow this format:

./<TARGET>/<REFERENCE>/<metric>.xml

The user must provide an IQREPORT file for each pair of:

  * REFERENCE*-REFERENCE*
  * SYSTEM*-REFERENCE*

Similarities when <TARGET> and <REFERENCE> are the same item
are not necessary.

For instance, suppose you have a working set consisting of
two systems ('S0' and 'S1') and three references ('R0', 'R1'
and 'R2'). if you add a new metric called 'NEWMETRIC', you
must supply 15 XML files:

./R0/R1/NEWMETRIC.xml
./R0/R2/NEWMETRIC.xml
./R1/R0/NEWMETRIC.xml
./R1/R2/NEWMETRIC.xml
./R2/R0/NEWMETRIC.xml
./R2/R1/NEWMETRIC.xml

./S0/R0/NEWMETRIC.xml
./S0/R1/NEWMETRIC.xml
./S0/R2/NEWMETRIC.xml
./S1/R0/NEWMETRIC.xml
./S1/R1/NEWMETRIC.xml
./S1/R2/NEWMETRIC.xml

That works for the QUEEN (and IQ) components. In the future
we plan to add the KING and JACK components, which additionally
require some more pairs:

  * SYSTEM*-SYSTEM*
  * REFERENCE*-SYSTEM*

For instance, in the case seen before, 10 more XML files would
become necessary:

./R0/S0/NEWMETRIC.xml
./R0/S1/NEWMETRIC.xml
./R1/S0/NEWMETRIC.xml
./R1/S1/NEWMETRIC.xml
./R2/S0/NEWMETRIC.xml
./R2/S1/NEWMETRIC.xml

./S0/S1/NEWMETRIC.xml
./S1/S0/NEWMETRIC.xml
./S2/S0/NEWMETRIC.xml
./S2/S1/NEWMETRIC.xml

Moreover, if you plan to use the "-doOQ" option with the new metric,
remember to provide results outside QARLA for all the systems in a
multiple reference setting:

  * SYSTEM*-REFERENCE'0_..._REFERENCE'N

Again, filenames are important:

./<TARGET>/<<REFERENCE'0>_...<REFERENCE'i>..._<REFERENCE'N>>/<metric>.xml

In our example, you should provide two extra files:

./S0/R0_R1_R2/NEWMETRIC.xml
./S1/R0_R1_R2/NEWMETRIC.xml

Finally, remember to properly edit the IQeval config file,
so you can play with your new metric:

metrics_NEWMETRIC= NEWMETRIC

metrics=BLEU-1 BLEU-2 BLEU-3 BLEU-4 GTM-1 GTM-2 GTM-3 MTR-exact MTR-stem MTR-wnstm MTR-wnsyn NIST-1 NIST-2 NIST-3 NIST-4 NIST-5 RG-1 RG-2 RG-3 RG-4 RG-L RG-SUs RG-Ss RG-W-1.2 NEWMETRIC


INSTALL
=======

(See ./INSTALL' file)

This software requires several Perl packages which can be obtained at the CPAN
website (http://search.cpan.org/):

  * XML management:  XML::Twig 3.22 (http://www.xmltwig.com/xmltwig/)
                     XML::DOM 1.43
                     XML::Parser 2.34
                     XML::RegExp 0.03
  * DB_File 1.814
  * Data::Dumper 2.12
  * strict 1.02
  * IO 1.20
  * IO::File 1.09
  * POSIX 1.08
  * Unicode::String 2.07
  * File::ReadBackwards 1.04

You may also find most of them inside the ./perl-cpan directory.
Moreover, since IQMT may call external software packages, before
starting to play with IQMT, if you plan to use these metrics (BLEU,
GTM, METEOR, NIST, ROUGE) you first might want to check they
work properly.

notes: METEOR requires WordNet 2.0 (http://wordnet.princeton.edu)
       You may need to properly set the WNHOME variable.
       (e.g. setenv WNHOME /usr/local/WordNet-2.0/bin)


IQMT Discussion Group
========================

Discussion on features and bugs of this software as well as
information about oncoming updates takes place on the IQMT group, to
which you can subscribe at:

http://groups-beta.google.com/group/IQMT

and post messages at:

IQMT@googlegroups.com

Any feedback (comments, suggestions, bug reporting) from the nlp
community members will be highly appreciated.


CONTRIBUTING:
=============

The IQMT package is licensed under LGPL , which means that it may be
linked to and used by commercial software packages. But the license
also enforces that any changes or improvements made to the library
must be redistributed under LGPL terms.

Thus, if you improve the software or data, either adding new
functionalities, fixing bugs, or adding new metrics, you can not
distribute them under different conditions than those stated in the
license (i.e. freely and with no usage restrictions).

If you want that your changes and improvements become useful to many
other people using this free software, please contact us (
jgimenez@lsi.upc.edu ).


REFERENCES:
==========

Please reference this tool in your academic works citing the following
paper:

* Jess Gimnez and Enrique Amig.
  IQMT: A Framework for Automatic Machine Translation Evaluation.
  Proceedings of the 5th International Language Resources and Evaluation Conference (LREC'06).
  Genoa, Italy, 22-28 May, 2006.

@inproceedings{IQMT'06,
            author = {Jess Gimnez and Enrique Amig},
            title = {IQMT: A Framework for Automatic Machine Translation Evaluation},
            booktitle = {Proceedings of the 5th LREC},
            year = {2006}
}
