CoNLL-2005 Shared Task:

Semantic Role Labeling: Data & Software

Introduction F.A.Q. References CoNLL Conferences

CoNLL-2005 : Description&Goal Examples Data&Software Systems&Results

CoNLL-2004 : Summary Page (data, systems & results)

Data

The current data distribution (released on March 4th, 2005) corresponds to the Wall Street sections 02-21 as training set, and section 24 as development set.

Each dataset is distributed split into many separate folders, each grouping files of different annotations (see details in the README file):

props : Target verbs and correct propositional arguments.
synt.upc : PoS tags, and partial parses by the UPC processors
synt.col2 : PoS tags, and full parses of Collins', with WSJ-style Non-Terminals
synt.col2h : PoS tags, and full parses of Collins', with Collins-style Non-Terminals (include lexical heads and enriched tags)
synt.cha : PoS tags and full parses of Charniak
ne : Named Entities.
senses : VerbNet senses of target verbs.

Standard UNIX commands can be used to merge all the necessary files into a single training/development complete file with all the information. As an example, we distribute the make-trainset.sh shell script, which constructs a complete training set (20 sections) containing words, syntax (Collins full parses), named entities, verb senses, and propositions.

Words and gold parse trees aligning with the previous annotations are to be extracted from the Penn Treebank II collection distributed through LDC or from the CoNLL-2005 shared task collection freely distributed through LDC under an evaluation license. Use 'wsj-removetraces.pl' and 'wsj-to-se.pl' utilities (from the srlconll software package) to convert Penn Treebank annotation style into the column-based format of the shared task (details in the README file).

CoNLL-2005 Shared Task Release
released: March 04, 2005
last update: March 14, 2005

NOTE:

The data of this distribution is to be completed with the Wall Street Journal sections 02-21 and 24 of the Penn Treebank II collection. For participants not owning a valid license on the Penn Treebank II collection, LDC is providing an evaluation license allowing to freely download and use the part corresponding to the CoNLL-2005 shared task datasets during competition time.

    Distribution Package, containing all annotation layers, README file, and scripts

    conll05st-release.tar.gz    last updated on March 14, 2005


    Files of the Distribution (those not forming part of annotation layers)

    README                Description file 
    CHANGES               Changes on the release since March 04, 2005
    make-trainset.sh      Script for constructing a complete training set 
    make-devset.sh        Script for constructing a complete development set


    Annotation Layer   Status  Released     Description
 
    props.tar.gz       stable  2005/03/04   Correct props for the sample (targets + args)

    synt.upc.tar.gz    stable  2005/03/11   Partial parses of UPC processors [GM03] and [CM03]

    synt.col2.tar.gz   stable  2005/03/10   Full syntax of Collins' [Col99], model2, w/ WSJ NTs 

    synt.col2h.tar.gz  stable  2005/03/04   Full syntax of Collins' [Col99], model2, w/ lexicalized NTs 

    synt.cha.tar.gz    stable  2005/03/14   Full syntax of Charniak' [Cha00] 
                             
    ne.tar.gz          stable  2005/03/11   Named Entities of (Chieu and Ng 03)
                                             
    senses.tar.gz      stable  2005/03/04   Correct senses of target verbs, extracted from Propbank 
                                             These will not be provided for the test set.

    CoNLL-2005 SHARED TASK : TEST DATA OF WSJ AND BROWN (posted February 2006)

    conll05st-tests.tar.gz : includes correct propositions and input annotations

Software

If you find bugs in the distributed software, please get in touch asap with the organizers at srlconll <at> lsi.upc.edu.

srl-eval

The srl-eval.pl program is available for download. (last updated: April 21, 2005)

The srl-eval.pl program is the official script for evaluation of CoNLL-2005 Shared Task systems. It expects two parameters: The first is the name of the file containing correct propositions; the second is the name of the file containing predicted propositions. Both files are expected to follow the format of "props" files (first column: target verbs; remaining columns: args of each target verb). It is required that both files contain the same sentences and the same target verbs. The program outputs performance measures based on precision, recall and F1. The overall F1 measure will be the measure used to compare the performance of systems.

Use the option "-latex" to produce a table of results in LaTeX.

The program is implemented in Perl, object oriented, with each object embedded in the script. It is part of the srlconll software package.

srlconll scripts

The srlconll package contains a suite of scripts to process the data of the CoNLL-2004 and CoNLL-2005 Shared Tasks.
The package includes the srl-eval.pl script, baseline systems, and other scripts to change formats and perform other utilities with data files.

Package : srlconll-1.1.tgz
Posted on December 1, 2005. For description and usage, see the README file included in the package.
This version includes the script to convert from PropBank-1.0 to the CoNLL-2005 formats. It is also being updated to process NomBank-0.8 (very preliminary). See README-databuilt for more information.

Package : srlconll-1.0.tgz
Posted on March 4, 2005. For description and usage, see the README file included in the package.
(This is an updated version of the srlconll-beta.tgz package released on February 17, 2005)

The package is implemented in Perl, and contains a library of objects which might be useful to participants. Feel free to use the code and develop further functionalities. Feedback is welcome!

Baseline

For now, the baseline system of CoNLL-2004 is available : srl-baseline04.pl

We expect to come up with alternative baseline systems. If you have a baseline system, please let us know.

collins-fixp.pl

In Collins' trees the punctuation terminals are included within the preceding phrases. This issue makes the correspondance between syntactic nodes and propositional arguments not direct at all.

On April 19, we've made available a script for fixing some of the punctuation terminals. It solves most of the simple cases, although, obviously, it fails at the complex ones. Download the script from :

http://www.lsi.upc.edu/~srlconll/collins-fixp.pl (April 19, 2005)

Once downloaded, read the documentation by executing :

 $ perl collins-fixp.pl -h

Example of usage:

 $ gunzip -c devel/synt.col2/devel.24.synt.col2.gz | collins-fixp.pl -s

If you find bugs or inconsistencies, report them to us as soon as possible.
Ideas/code to improve the script are more than welcome.

Official Resources

PropBank Frames. Scheme of rolesets for English verbs. A roleset, defined for a verb acting in one of its verb senses, specifies the arguments that a verb accepts, and defines which semantic role each argument is playing with respect to the verb.
[ PropBank Frames ] (last updated: May 03, 2005)
Verb Senses. Manually annotated verb senses. The format of these files is just a column which aligns with the column specifying target verbs of the data. For each target verb, either the verb sense or a non-disambiguated "XX" tag appears. Note that this information will not be provided for the final test set.
/ note that verb senses are released together with the training data /

Last update: February 23, 2006. Xavier Carreras, Lluís Màrquez.