November 19, 1999
4:30pm

Berry Debruijn
School of Information Technology and Engineering
University of Ottawa

Evaluation of Interactive Information Retrieval

Our experimental platform - a full-text information retrieval system - supports Query Expansion (QE) by Relevance Feedback (RF). That is, the system suggests additional query terms that are typical for relevant documents and typical for irrelevant or unretrieved documents. Evaluation of QE/RF is slightly problematic. A user study is costly, and due to tensions between experiment size, expected variations between users and variations between retrieval tasks, effects are difficult to establish statistically and the results are difficult to generalize. A system study lacks the input of users and disregards the interactive character of the system, while a system study with a simulated user lacks the credibility essential to any user evaluation. A system study with a "cohort" of simulated users - each one behaving slightly differently - could be a step to solve the evaluation problems.

Today's presentation describes:

• the place of this new experimental method among existing methods;
• the design of the study;
• the design of the group of simulated users;
• the results of experiments;
• the restrictions and assumptions that apply; and
• examples of possible future studies within (and outside of) the framework of the Intelligent Information Access project.

  ---

  Back to the TAMALE home page