Generic Methods for Multi-criteria Evaluation

Document type: Conference Papers
Peer reviewed: Yes
Full text:
Author(s): Niklas Lavesson, Paul Davidsson
Title: Generic Methods for Multi-criteria Evaluation
Translated title: Generiska metoder för multikriterie-baserad utvärdering
Conference name: SIAM International Conference on Data Mining
Year: 2008
ISBN: 978-0-89871-654-2
Publisher: SIAM Press
City: Atlanta, Georgia, USA
Organization: Blekinge Institute of Technology
Department: School of Engineering - Dept. of Systems and Software Engineering (Sektionen för teknik – avd. för programvarusystem)
School of Engineering S- 372 25 Ronneby
+46 455 38 50 00
Authors e-mail:,
Language: English
Abstract: When evaluating data mining algorithms that are applied to solve real-world problems there are often several, conflicting criteria that need to be considered. We investigate the concept of generic multi-criteria (MC) classifier and algorithm evaluation and perform a comparison of existing methods. This comparison makes explicit some of the important characteristics of MC analysis and focuses on finding out which method is most suitable for further development. Generic MC methods can be described as frameworks for combining evaluation metrics and are generic in the sense that the metrics used are not dictated by the method; the choice of metric is instead dependent on the problem at hand. We discuss some scenarios that benefit from the application of generic MC methods and synthesize what we believe are attractive properties from the reviewed methods into a new method called the candidate evaluation function (CEF). Finally, we present a case study in which we apply CEF to trade-off several criteria when solving a real-world problem.
Summary in Swedish: Traditionellt utvärderas inlärningsalgoritmer och klassificerare avseende ett enskilt kriterium. Detta har visats sig för begränsat för många inlärningsproblem. Denna studie undersöker existerande alternativ för generisk multikriteria-utvärdering och presenterar en ny metod tillsammans med en fallstudie.
Subject: Computer Science\Artificial Intelligence
Keywords: multi-criteria, classifier, evaluation, machine, learning