Submission Format

Each participant is allowed to submit 3 runs for evaluation. The required format and maximum length for each goal is provided below.

Task Understanding

The goal of this task is to test whether systems can understand the possible tasks users might be trying to achieve given a query. For this task, the participants have to submit keyphrases that represent the possible task user is trying to achieve given the query.
Run Format : TOPIC-ID RANK SCORE RUN-ID KEY-PHRASE
Maximum Number of keywords per topic : 1000

TREC submission system requires you to submit keywords for every topic. If there are no phrases for a certain topic, please insert 'NA' as KEY-PHRASE with a dummy score.

Example Run
Following rows indicate sample run submission for Task understanding. The RUN-ID is 1, with three key phrases for TOPIC-ID 1, i.e. query 'getting organized at work'.
TOPIC-ID RANK SCORE RUN-ID KEY-PHRASE
1 1 4.5 1 time management
1 2 3.4 1 stress managment
1 3 1.4 1 organization skills


Task Completion

Participants have to submit a ranked list of documents that could be relevant to any task a user may be trying to achieve given the query.
Run Format : TOPIC-ID Q0(unused) DOC-ID RANK SCORE RUN-ID
Maximum Number of documents per topic : 1000

TREC submission system requires you to submit documents for every topic. If there are no documents for a certain topic, please insert 'clueweb12-0000wb-00-00000' as DOC-ID with a dummy score.

Example Run
Following rows indicate sample run submission for Task completion. The RUN-ID is 1, with three documents for TOPIC-ID 1, i.e. query 'getting organized at work'.
TOPIC-ID Q0 DOC-ID RANK SCORE RUN-ID
1 Q0 clueweb12-en0040-54-00001 1 -5.2 1
1 Q0 clueweb12-en0043-54-00101 2 -7.2 1
1 Q0 clueweb12-en0040-57-01001 3 -12.2 1


Ad-hoc Task

This task is a continuation of the traditional TREC Web Track ad-hoc task, thus its identical to the previous years’ Web Track ad-hoc tasks.
Run Format : TOPIC-ID Q0(unused) DOC-ID RANK SCORE RUN-ID
Maximum Number of documents per topic : 1000

TREC submission system requires you to submit documents for every topic. If there are no documents for a certain topic, please insert 'clueweb12-0000wb-00-00000' as DOC-ID with a dummy score.

Example Run
Following rows indicate sample run submission for Ad-hoc Task. The RUN-ID is 1, with three documents for TOPIC-ID 1, i.e. query 'getting organized at work'.
TOPIC-ID Q0 DOC-ID RANK SCORE RUN-ID
1 Q0 clueweb12-en020-54-00001 1 -15.2 1
1 Q0 clueweb12-en0083-54-02010 2 -17.2 1
1 Q0 clueweb12-en1040-57-00701 3 -21.2 1