CALL FOR NTCIR-11 TASK PROPOSALS
(DEADLINE: May 10, 2013)
NTCIR (NII Testbeds and Community for Information access Research)is a
sesquiannual series of evaluation conferences that mainly focuses on
Asian language information access. The first NTCIR conference
(NTCIR-1) took place in August/September 1999, and the tenth
(NTCIR-10) will take place in June 2013. Research teams from all over
the world participate in one or more of the NTCIR tasks to advance the
state of the art and to learn from one another's experiences.
http://research.nii.ac.jp/ntcir/ntcir-10/index.html
As the tasks being run at NTCIR-10 are reaching the final stages, it
is time to call for tasks for the next NTCIR (NTCIR-11) which will be
concluded in December 2014. Task proposals will be reviewed by the
NTCIR Programme Commitee, following the schedule below:
IMPORTANT DATES:
May 10, 2013 Deadline for task proposals (new and existing tasks)
June 7, 2013 Notification of accepted tasks
June 18-21, 2013 Task planning sessions at the NTCIR-10 conference
January 31, 2014 Deadline for additional pilot task proposals
February 28, 2014 Notification of accepted additional pilot tasks
Organisers of exising NTCIR-10 tasks (CrossLink, INTENT, 1CLICK,
PatentMT, RITE, SpokenDoc, MATH and MedNLP) are required to submit a
new proposal if they wish to continue them for NTCIR-11. New task
proposals are also very welcome. To organise an evaluation task is to
identify important research problems, tackle them strategically by
collaborating with other researchers (participants), build the
necessary evaluation framework to advance the state of the art, and
make an impact to the research community and to the future.
We will accept two types of tasks:
- Core challenge task: this is for fostering research on a particular
information access problem by providing researchers with a common
ground for evaluation. New test collections and evaluation methods
may be developed through the collaboration between task organisers
(proposers) and the task participants. At NTCIR-10, the core
challenge tasks were: CrossLink, INTENT, 1CLICK, PatentMT, RITE and
SpokenDoc.
- Pilot task: this is recommended if the information access problem to
be tackled is new and there are uncertainties as to how to evaluate
it. It may focus on a subproblem of an information access
problem. It may attract a smaller group of participating teams than
core challenge task, but may grow into a core challenge task in the
next round of NTCIR. At NTCIR-10, the pilot tasks were: MATH and
MedNLP.
NTCIR will provide a "seed funding" to each accepted task, which can
be used for some limited purposes such as hiring relevance assessors.
However, organisers of each task are expected to make the task as
self-sustaining as possible. The amount allocated to each task will
vary depending on requirements and the total number of accepted tasks,
but typical cases would be: around 1000000JPY/year for a core
challenge task and around 500000JPY/year for a pilot task.
Please submit your task proposal as a pdf to the following easychair site
by May 10, 2013 (Japan Time: UTC+9).
https://www.easychair.org/conferences/?conf=ntcir11tp
As indicated in the important dates, there will be another call for
additional pilot tasks after the NTCIR-10 conference, to accomodate
"late-breaking information access problems."
TASK PROPOSAL FORMAT:
Main part (a single A4 page)
- Task name and short name
- Task type (core challenge or pilot)
- Abstract
- Motivation
- Methodology
- Expected results
Appendix
- Names and contact information of the organisers
- Prospective participants
- Data to be used and/or constructed
- Budget planning
- Schedule
- Other notes
REVIEW CRITERIA:
- Importance of the task to the information access community and to the society
- Timeliness of the task
- Organisers' commitment in ensuring a successful task
- Financial sustainability (self-sustainable tasks are encouraged)
- Soundness of the evaluation methodology
- Language scope
NTCIR-11 PROGRAMME COMMITTEE:
Hsin-Hsi Chen (National Taiwan University, Taiwan)
Charles Clarke (University of Waterloo, Canada)
Kalervo Jarvelin (University of Tampere, Finland)
Gareth Jones (Dublin City University, Ireland)
Gary Geunbae Lee (POSTECH, South Korea)
Maarten de Rijke (University of Amsterdam, The Netherlands)
Stephen Robertson (Microsoft Research Cambridge, UK)
Ian Soboroff (NIST, US)
Hideo Joho (Co-chair, University of Tsukuba, Japan)
Tetsuya Sakai (Co-chair, Microsoft Research Asia, PRC)
NTCIR General Chairs:
Noriko Kando (NII, Japan)
Tsuneaki Kato (The University of Tokyo, Japan)
Douglas W. Oard (University of Maryland, USA)
Mark Sanderson (RMIT, Australia)