April 30, 1999
4:30

Joel Martin
Institute for Information Technology
National Research Council, Ottawa

Which System Configuration is Best?

Many complex systems have parameters that allow them to be endlessly configured, but not even the designers know which configurations are the best for a new application. Each configuration of the system can be simulated (or run if it a computer system) on a representative set of problems. One way to find the best configuration is to perform a randomized or a hill-climbing search trying to optimize the simulation results. Although this technique will find an optimal configuration, it does not guarantee that the observed optimal is the only one or that the observed optimal will still be best for a different representative sample of problems. Another approach is to simulate as many different configurations as possible from across the entire space. These simulation results would then be analyzed to make general statements about which configurations are reliably better than others. This talk will discuss how to analyze the results of simulating a large number of system configurations. These techniques have been applied to human-computer systems, machine learning systems, and traditional discrete-event systems.