CoNLL-2005 Shared Task:
Semantic Role Labeling: Data & Software
The current data distribution (released on March 4th, 2005) corresponds
to the Wall Street sections 02-21 as training set, and section 24 as
Each dataset is distributed split into many separate folders, each
grouping files of different annotations (see details in the README file):
Standard UNIX commands can be used to merge all the
necessary files into a single training/development complete file with
all the information. As an example, we distribute the make-trainset.sh
shell script, which constructs a complete training set (20 sections)
containing words, syntax (Collins full parses), named entities, verb
- props : Target verbs and correct propositional arguments.
- synt.upc : PoS tags, and partial parses by the UPC processors
- synt.col2 : PoS tags, and full parses of Collins', with WSJ-style
- synt.col2h : PoS tags, and full parses of Collins', with
Collins-style Non-Terminals (include lexical heads and enriched tags)
- synt.cha : PoS tags and full parses of Charniak
- ne : Named Entities.
- senses : VerbNet senses of target verbs.
Words and gold parse trees aligning with the previous annotations are
to be extracted from the Penn Treebank II collection distributed
through LDC or from the CoNLL-2005 shared task collection freely
distributed through LDC under an evaluation
'wsj-removetraces.pl' and 'wsj-to-se.pl' utilities (from the srlconll software package) to convert Penn
Treebank annotation style into the column-based format of the shared
task (details in the README file).
|CoNLL-2005 Shared Task Release
released: March 04,
last update: March 14, 2005
The data of this distribution is
to be completed with the Wall Street
Journal sections 02-21 and 24 of the Penn Treebank II collection. For
participants not owning a valid license on the Penn
Treebank II collection, LDC is providing an evaluation license allowing
to freely download and use the part corresponding to the CoNLL-2005
shared task datasets during competition time.
Distribution Package, containing all annotation layers, README file, and scripts
conll05st-release.tar.gz last updated on March 14, 2005
Files of the Distribution (those not forming part of annotation layers)
README Description file
CHANGES Changes on the release since March 04, 2005
make-trainset.sh Script for constructing a complete training set
make-devset.sh Script for constructing a complete development set
Annotation Layer Status Released Description
props.tar.gz stable 2005/03/04 Correct props for the sample (targets + args)
synt.upc.tar.gz stable 2005/03/11 Partial parses of UPC processors [GM03] and [CM03]
synt.col2.tar.gz stable 2005/03/10 Full syntax of Collins' [Col99], model2, w/ WSJ NTs
synt.col2h.tar.gz stable 2005/03/04 Full syntax of Collins' [Col99], model2, w/ lexicalized NTs
synt.cha.tar.gz stable 2005/03/14 Full syntax of Charniak' [Cha00]
ne.tar.gz stable 2005/03/11 Named Entities of (Chieu and Ng 03)
senses.tar.gz stable 2005/03/04 Correct senses of target verbs, extracted from Propbank
These will not be provided for the test set.
CoNLL-2005 SHARED TASK : TEST DATA OF WSJ AND BROWN (posted February 2006)
conll05st-tests.tar.gz : includes correct propositions and input annotations
If you find bugs in the
distributed software, please get
touch asap with the organizers at srlconll <at> lsi.upc.edu.
program is available for download. (last
updated: April 21, 2005)
The srl-eval.pl program is the official script for evaluation of
CoNLL-2005 Shared Task systems. It expects two parameters: The first is
the name of the file containing correct propositions; the second is the
name of the file containing predicted propositions. Both files are
expected to follow the format of "props" files (first column: target
verbs; remaining columns: args of each target verb). It is required
both files contain the same sentences and the same target verbs. The
program outputs performance measures based on precision, recall and F1.
The overall F1 measure will be the measure used to compare the
performance of systems.
Use the option "-latex" to produce a table of results in LaTeX.
The program is implemented in Perl, object oriented, with each object
embedded in the script. It is part of the srlconll
The srlconll package contains
a suite of scripts to process the data of the CoNLL-2004 and CoNLL-2005
The package includes the srl-eval.pl script, baseline systems, and
other scripts to change formats and perform other utilities with data
Package : srlconll-1.1.tgz
Posted on December 1, 2005. For description and
usage, see the README file included
in the package.
This version includes the script to convert from PropBank-1.0 to the
CoNLL-2005 formats. It is also being updated to process NomBank-0.8
(very preliminary). See README-databuilt
for more information.
Package : srlconll-1.0.tgz
Posted on March 4, 2005. For description and
usage, see the README file included
in the package.
(This is an updated version of the srlconll-beta.tgz
package released on February 17, 2005)
The package is implemented in Perl, and contains a library of objects
which might be useful to participants. Feel free to use the code and
develop further functionalities. Feedback is welcome!
For now, the baseline system of CoNLL-2004 is available : srl-baseline04.pl
We expect to come up with alternative baseline systems. If you have a
baseline system, please let us know.
In Collins' trees the punctuation terminals are included within the
preceding phrases. This issue makes the correspondance between
syntactic nodes and propositional arguments not direct at all.
On April 19, we've made available a script for fixing some of the
punctuation terminals. It solves most of the simple cases, although,
obviously, it fails at the complex ones.
Download the script from :
(April 19, 2005)
Once downloaded, read the documentation by executing :
$ perl collins-fixp.pl -h
Example of usage:
$ gunzip -c devel/synt.col2/devel.24.synt.col2.gz | collins-fixp.pl -s
If you find bugs or inconsistencies, report them to us
as soon as possible.
Ideas/code to improve the script are more than welcome.
- PropBank Frames. Scheme
rolesets for English verbs. A roleset, defined for a verb acting in one
of its verb senses, specifies the arguments that a verb accepts, and
defines which semantic role each argument is playing with respect to
[ PropBank Frames ] (last updated: May 03, 2005)
- Verb Senses. Manually
annotated verb senses. The format of these files is just a column which
aligns with the column specifying target verbs of the data. For each
target verb, either the verb sense or a non-disambiguated "XX" tag
appears. Note that this information will not be provided for the final
/ note that verb senses are
released together with the training data /
Last update: February 23,