CoNLL-2005 Shared Task: 

Semantic Role Labeling: Systems & Results


Introduction    F.A.Q.    References    CoNLL Conferences


CoNLL-2005 :       Description&Goal       Examples       Data&Software      Systems&Results 


CoNLL-2004 :      Summary Page (data, systems & results)



The CoNLL-2005 Shared Task on Semantic Role Labeling took place during the period January to May of 2005. Participant systems and results were presented at the CoNLL-2005 conference in Ann Arbor (Michigan, USA) in June 30, 2005.

Nineteen systems participated in the closed challenge. No system was presented at the open challenge. A description of the task, with a discussion of systems and results, can be found below in the introduction paper [CM05]. Furthermore, this webpage includes the papers and outputs of all systems, as well as the slides of the talks presented at the Shared Task session.


Talks at the CoNLL-2005 Shared Task Session



Results

                 +-----------------------+-----------------------+-----------------------+---------------------------+
| Development | Test WSJ | Test Brown | Test WSJ+Brown |
+-----------------------+-----------------------+-----------------------+---------------------------+
| P R F1 | P R F1 | P R F1 | P R F1 |
+---------------+-----------------------+-----------------------+-----------------------+---------------------------+
| punyakanok | 80.05 74.83 77.35 | 82.28 76.78 79.44 | 73.38 62.93 67.75 | 81.18 74.92 77.92 ±0.7 |
| pradhan (*) | 81.91 75.08 78.34 | 82.95 74.75 78.63 | 74.49 63.30 68.44 | 81.87 73.21 77.30 ±0.7 |
| haghighi | 77.66 75.72 76.68 | 79.54 77.39 78.45 | 70.24 65.37 67.71 | 78.34 75.78 77.04 ±0.7 |
| marquez | 78.39 75.53 76.93 | 79.55 76.45 77.97 | 70.79 64.35 67.42 | 78.44 74.83 76.59 ±0.7 |
| pradhan | 80.90 75.38 78.04 | 81.97 73.27 77.37 | 73.73 61.51 67.07 | 80.93 71.69 76.03 ±0.7 |
| surdeanu | 79.14 71.57 75.17 | 80.32 72.95 76.46 | 72.41 59.67 65.42 | 79.35 71.17 75.04 ±0.7 |
| tsai | 81.13 72.42 76.53 | 82.77 70.90 76.38 | 73.21 59.49 65.64 | 81.55 69.37 74.97 ±0.6 |
| che | 79.65 71.34 75.27 | 80.48 72.79 76.44 | 71.13 59.99 65.09 | 79.30 71.08 74.97 ±0.7 |
| moschitti | 74.95 73.10 74.01 | 76.55 75.24 75.89 | 65.92 61.83 63.81 | 75.19 73.45 74.31 ±0.7 |
| tjongkimsang | 76.79 70.01 73.24 | 79.03 72.03 75.37 | 70.45 60.13 64.88 | 77.94 70.44 74.00 ±0.7 |
| yi | 75.70 69.99 72.73 | 77.51 72.97 75.17 | 67.88 59.03 63.14 | 76.31 71.10 73.61 ±0.7 |
| ozgencil | 73.57 71.87 72.71 | 74.66 74.21 74.44 | 65.52 62.93 64.20 | 73.48 72.70 73.09 ±0.8 |
| johansson | 73.40 70.85 72.10 | 75.46 73.18 74.30 | 65.17 60.59 62.79 | 74.13 71.50 72.79 ±0.8 |
| cohn | 73.51 68.98 71.17 | 75.81 70.58 73.10 | 67.63 60.08 63.63 | 74.76 69.17 71.86 ±0.7 |
| park | 72.68 69.16 70.87 | 74.69 70.78 72.68 | 64.58 60.31 62.38 | 73.35 69.37 71.31 ±0.7 |
| mitsumori | 71.68 64.93 68.14 | 74.15 68.25 71.08 | 63.24 54.20 58.37 | 72.77 66.37 69.43 ±0.8 |
| venkatapathy | 71.88 64.76 68.14 | 73.76 65.52 69.40 | 65.25 55.72 60.11 | 72.66 64.21 68.17 ±0.7 |
| ponzetto | 71.82 61.60 66.32 | 75.05 64.81 69.56 | 66.69 52.14 58.52 | 74.02 63.12 68.13 ±0.8 |
| lin | 70.11 61.96 65.78 | 71.49 64.67 67.91 | 65.75 52.82 58.58 | 70.80 63.09 66.72 ±0.8 |
| sutton | 64.43 63.11 63.76 | 68.57 64.99 66.73 | 62.91 54.85 58.60 | 67.86 63.63 65.68 ±0.8 |
+---------------+-----------------------+-----------------------+-----------------------+---------------------------+
| baseline | 50.00 28.98 36.70 | 51.13 29.16 37.14 | 62.66 33.07 43.30 | 52.58 29.69 37.95 ±0.8 |
+---------------+-----------------------+-----------------------+-----------------------+---------------------------+

Results of CoNLL-2005 evaluation, on the four evalutation sets. The WSJ+Brown test is actually a concatenation of
two tests sets (WSJ and Brown).

Systems are ranked by their performance on the WSJ+Brown test. The significance intervals for the F rates in
that column have been obtained with bootstrap resampling [Nor89].  F rates outside of these intervals are
assumed to be significantly different from the related F rate (p<0.05).

Systems marked with (*) correspond to post-conference updates.


Post-conference contributions: If you have a system developed in the setting of the CoNLL-2005 Shared Task, and a description of it in an official document (technical report or published paper/article), please contact us at srlconll <at> lsi.upc.edu. We'll be glad to add an entry for your system in the results table.



CoNLL-2005 Introduction Paper:

CoNLL-2005 Papers :


Last Update: September 16, 2005. Xavier Carreras, Lluís Màrquez.