November 19, 1999

Berry Debruijn
School of Information Technology and Engineering
University of Ottawa

Evaluation of Interactive Information Retrieval

Our experimental platform - a full-text information retrieval system - supports Query Expansion (QE) by Relevance Feedback (RF). That is, the system suggests additional query terms that are typical for relevant documents and typical for irrelevant or unretrieved documents. Evaluation of QE/RF is slightly problematic. A user study is costly, and due to tensions between experiment size, expected variations between users and variations between retrieval tasks, effects are difficult to establish statistically and the results are difficult to generalize. A system study lacks the input of users and disregards the interactive character of the system, while a system study with a simulated user lacks the credibility essential to any user evaluation. A system study with a "cohort" of simulated users - each one behaving slightly differently - could be a step to solve the evaluation problems.

Today's presentation describes: