***********************************************************************
TOOL:        IQMT Framework for MT Evaluation
AUTHOR(s):   Jess Gimnez and Enrique Amig
DATE:        15/11/2005
VERSION:     1.0
DESCRIPTION: Inside Qarla MT evaluation
***********************************************************************


Contents
--------

      README _ this file.
     INSTALL _ instructions to install this software.
 Makefile.PL _ configuration script.
    MANIFEST _ list of software components included in the package.
    LGPL.txt _ license terms.
       ./lib _ IQ library.
       ./bin _ executable files:
               * IQsetup  _ setup component.
               * IQeval   _ evaluation component.
   ./samples _ sample files:
               * IQsetup.config file sample.
               * IQeval.config file sample.
               * IQREPORT XML representation format sample.

Description
-----------

The IQMT package is a Machine Translation Evaluation Software based on
the QARLA Framework (Amig et al., ACL'2005). Rather than defining a
new supermetric 'XXX' our tool follows a 'divide and conquer'
strategy. You can define a set of metrics and then combine them into a
single measure of MT quality, in a robust and elegant manner, avoiding
scaling problems and metric weightings.

Using IQMT offers a number of advantages over previous MT evaluation
packages. First, individual metrics improve their level of correlation
with human judgements when they are applied inside QARLA. Second, it
permits to avoid the 'metric bias' problem, by allowing you to tune
your system on a combination of metrics instead of on a single metric.
Third, it allows you to define a set of subtle metrics focusing on
partial aspects of MT quality possibly at different linguistic levels,
and then combine them into a single measure.

Several well-known and freely available current MT evaluation tools have
been incorporated so far:

  - ROUGE v 1.5.5 (http://www.isi.edu/~cyl/ROUGE/)
  - GTM v 1.2 (http://nlp.cs.nyu.edu/GTM/)
  - METEOR v 0.4.3 (http://www.cs.cmu.edu/afs/cs.cmu.edu/user/alavie/www/METEOR/)
  - BLEU/NIST mteval-kit v 11b (http://www.nist.gov/speech/tests/mt/resources/scoring.htm)

We plan to incorporate new metrics working at linguistic levels other
than lexical overlap in a near future.

However, the main feature of the IQMT package is that user-supplied
evaluation metrics can be easily incorporated to the system. Do not
hesitate to try your own metrics inside QARLA!

IQMT COMPONENTS:
================

IQMT consists of two main components:

    * IQsetup
    * IQeval


---------------------------------------------------------------------
(1) IQsetup
---------------------------------------------------------------------

The IQsetup component is responsible for applying a given set of
metrics to a given set of translations by different systems. Thus it
requires a set of human references.

Given a 'setup' config file, it generates a series of XML files
contaning MT evaluation scores for each metric and each pair:

  * SYSTEM*-REFERENCE*
  * REFERENCE*-SYSTEM*
  * SYSTEM*-SYSTEM*
  * REFERENCE*-REFERENCE*

and an 'evaluation' config file in a format convenient for the IQeval
component.


---------------------------------------------------------------------

Usage : IQsetup  [options]  <IQsetup.config>  <IQeval.config>
                                (input)          (output)
 
  - print         : print similarities onto IQeval.config (default disabled)
  - remake        : remake reports
  - V <0|1|2>     : verbosity
                    0 - non-verbose (default)
                    1 - low verbosity
                    2 - medium verbosity
 
Example: IQsetup IQsetup.config IQeval.config

---------------------------------------------------------------------

The Setup Config File must specify several variables:


# EXPERIMENT NAME ---------------------------
NAME=TCSTAR_dev1_spa2eng

# IQMT LOCATION -----------------------------
IQMT=/home/users/me/IQMT/

# FILES -------------------------------------
source=source_file.txt
ref=reference_file.txt.1
...
ref=reference_file.txt.M
system=system_output_file.txt.1
...
system=system_output_file.txt.N

# AVAILABLE METRICS -------------------------
doBLEU
doNIST
doGTM
doMETEOR
doROUGE
#doBLEU 1 2 3 4
#doNIST 1 2 3 4 5
#doGTM 1 2 3
#doMETEOR exact stem wnstm wnsyn
#doROUGE 1 2 3 4 L W S SU
#--------------------------------------------

You may find an example in the './samples' directory.

Notice that you can select which of the available metrics
you want to use among a set of 24 variants from 5 different
metric families:

* doBLEU (n-gram levels = 1..4)
* doGTM (e=1..3)
* doNIST (n-gram levels = 1..5)
* doMETEOR ('exact', + 'stemming', + 'Wordnet stemming', + 'WordNet synonyms'
* doROUGE (n-gram levels = 1..4, as well as LCS, S*, SU*, W-1.2 variants)

---------------------------------------------------------------------
(2) IQeval
---------------------------------------------------------------------

The IQeval component computes QUEEN / (IQ) scores on top of the scores
generated by IQsetup.It allows the user to choose the set of
metrics, systems, references and segments to be used.
The user may define these sets in the IQeval config file.

Two levels of granularity are allowed. Results may be shown at the
system level (useful for tuning and system comparison) and at the
segment level (crucial for error analysis). Moreover, two kinds of
reports (score matrix / ranking lists) may be generated. The 'score
matrix' report allows to compare the scores by different metrics on
the same system/segment. The 'ranking lists' report provides lists of
rankings according to each metric individually. It is specially useful
for the study of correlations.


---------------------------------------------------------------------

Usage : IQeval  [options]  <IQeval.config>
 
  - M <metrics>         : set of metrics (setname according to IQeval.config)
                          ['metrics' by default]
  - S <systems>         : set of systems (setname according to IQeval.config)
                          ['systems' by default]
  - R <references>      : set of references (setname according to IQeval.config)
                          ['references' by default]
  - T <segments>        : set of segments (setname according to IQeval.config)
                          ['segments' by default]
  - G <sys|seg>         : level of granularity
                          sys - system-level scores (default)
                          seg - segment-level scores
  - doOQ                : compute individual metric scores outside QARLA (default disabled)
  - remake              : remake reports
  - O <0|1>             : output format
                          0 - score matrix (default)
                          1 - ranking lists (on a metric basis)
  - V <0|1|2>           : verbosity
                          0 - non-verbose (default)
                          1 - low verbosity
                          2 - medium verbosity
 
Example: IQeval -V 1 -doOQ -G seg -O 1 IQeval.config


---------------------------------------------------------------------

The Eval Config File is generated automatically.  It is intended for
the user to define new metric sets, system sets, and reference sets,
to be used by the IQeval component. You may find an example in the
'./samples' directory.

NOTE: The KING and JACK components are not yet ready for release. They will
be included in future versions.

---------------------------------------------------------------------


ADDING NEW METRICS
==================

Additionally, the IQMT package allows the user to supply its own
user-defined metrics by means of the IQREPORT XML representation
schema. See an example (you may find it in the './samples' directory, too):

<?xml version="1.0"?>
<!DOCTYPE iqmt SYSTEM "iqmt.dtd" []>
<IQREPORT metric="MTR-wnsyn" ref="R2" score="0.6307" target="R0">
  <S n="1">0.9960</S>
  <S n="2">0.6250</S>
  <S n="3">0.8519</S>
  (...)
  <S n="502">0.7216</S>
  <S n="503">0.5222</S>
  <S n="504">0.5962</S>
</IQREPORT>


Filenames are important. They must follow this format:

<TARGET>-<REFERENCE>.<metric>.xml

The user must provide an IQREPORT file for each pair of:

  * REFERENCE*-REFERENCE*
  * SYSTEM*-REFERENCE*

Similarities when <TARGET> and <REFERENCE> are the same item
are not necessary.

For instance, suppose you have a working set consisting of
two systems ('S0' and 'S1') and three references ('R0', 'R1'
and 'R2'). if you add a new metric called 'NEWMETRIC', you
must supply 15 XML files:

R0-R1.NEWMETRIC.xml
R0-R2.NEWMETRIC.xml
R1-R0.NEWMETRIC.xml
R1-R2.NEWMETRIC.xml
R2-R0.NEWMETRIC.xml
R2-R1.NEWMETRIC.xml

S0-R0.NEWMETRIC.xml
S0-R1.NEWMETRIC.xml
S0-R2.NEWMETRIC.xml
S1-R0.NEWMETRIC.xml
S1-R1.NEWMETRIC.xml
S1-R2.NEWMETRIC.xml

That works for the QUEEN (and IQ) components. In the future
we plan to add the KING and JACK components, which additionally
require some more pairs:

  * SYSTEM*-SYSTEM*
  * REFERENCE*-SYSTEM*

For instance, in the case seen before, 10 more XML files would
become necessary:

R0-S0.NEWMETRIC.xml
R0-S1.NEWMETRIC.xml
R1-S0.NEWMETRIC.xml
R1-S1.NEWMETRIC.xml
R2-S0.NEWMETRIC.xml
R2-S1.NEWMETRIC.xml

S0-S1.NEWMETRIC.xml
S1-S0.NEWMETRIC.xml
S2-S0.NEWMETRIC.xml
S2-S1.NEWMETRIC.xml

Moreover, if you plan to use the "-doOQ" option with the new metric,
remember to provide results outside QARLA for all the systems in a
multiple reference setting:

  * SYSTEM*-REFERENCE'0_..._REFERENCE'N

Again, filenames are important:

<TARGET>-<<REFERENCE'0>_...<REFERENCE'i>..._<REFERENCE'N>>.<metric>.xml

In our example, you should provide two extra files:

S0-R0_R1_R2.NEWMETRIC.xml
S1-R0_R1_R2.NEWMETRIC.xml

Finally, remember to properly edit the IQeval config file,
so you can play with your new metric:

metrics_NEWMETRIC= NEWMETRIC

metrics=BLEU-1 BLEU-2 BLEU-3 BLEU-4 GTM-1 GTM-2 GTM-3 MTR-exact MTR-stem MTR-wnstm MTR-wnsyn NIST-1 NIST-2 NIST-3 NIST-4 NIST-5 RG-1 RG-2 RG-3 RG-4 RG-L RG-SUs RG-Ss RG-W-1.2 NEWMETRIC


INSTALL
=======

(See ./INSTALL' file)

This software requires several Perl packages which can be obtained at the CPAN
website (http://search.cpan.org/):

  * XML management:  XML::Twig 3.22 (http://www.xmltwig.com/xmltwig/)
                     XML::DOM 1.43
                     XML::Parser 2.34
                     XML::RegExp 0.03
  * DB_File 1.814
  * Data::Dumper 2.12
  * strict 1.02
  * IO 1.20
  * IO::File 1.09
  * POSIX 1.08
  * Unicode::String 2.07

Moreover, since IQMT may call external software packages, before
starting to play with IQMT, if you plan to use these metrics (BLEU,
GTM, METEOR, NIST, ROUGE) you first might want to check they
work properly.


IQMT Discussion Group
========================

Discussion on features and bugs of this software as well as
information about oncoming updates takes place on the IQMT group, to
which you can subscribe at:

http://groups-beta.google.com/group/IQMT

and post messages at:

IQMT@googlegroups.com

Any feedback (comments, suggestions, bug reporting) from the nlp
community members will be highly appreciated.


CONTRIBUTING:
=============

The IQMT package is licensed under LGPL , which means that it may be
linked to and used by commercial software packages. But the license
also enforces that any changes or improvements made to the library
must be redistributed under LGPL terms.

Thus, if you improve the software or data, either adding new
functionalities, fixing bugs, or adding new metrics, you can not
distribute them under different conditions than those stated in the
license (i.e. freely and with no usage restrictions).

If you want that your changes and improvements become useful to many
other people using this free software, please contact us (
jgimenez@lsi.upc.edu ).


REFERENCES:
==========

Please reference this tool in your academic works citing the following
paper:

* Jess Gimnez, Enrique Amig and Chiori Hori.
  Machine Translation Evaluation Inside QARLA.
  In Proceedings of the International Workshop on Spoken Language Technology (IWSLT'05).
  Pittsburgh, PA, USA October 24-25, 2005.


@inproceedings{IQMT,
  author = {Jess Gimnez and Enrique Amig and Chiori Hori},
  title = {Machine Translation Evaluation Inside QARLA},
  booktitle = {Proceedings of the International Workshop on Spoken Language Technology (IWSLT)},
  year = {2005}
}
