The Normalised Compression Distance as a File Fragment Classifier

Document type: Journal Articles
Article type: Original article
Peer reviewed: Yes
Full text:
Author(s): Stefan Axelsson
Title: The Normalised Compression Distance as a File Fragment Classifier
Journal: Digital Investigation
Year: 2010
Volume: 7
Issue: Suppl 1
Pagination: S24-S31
ISSN: 1742-2876
Publisher: Elsevier
URI/DOI: 10.1016/j.diin.2010.05.004
ISI number: 000281010700004
Organization: Blekinge Institute of Technology
Department: School of Computing (Sektionen för datavetenskap och kommunikation)
School of Computing S-371 79 Karlskrona
+46 455 38 50 00
Authors e-mail:
Language: English
Abstract: We have applied the generalised and universal distance measure NCD—Normalised Compression Distance—to the problem of determining the type of file fragments. To enable later comparison of the results, the algorithm was applied to fragments of a publicly available corpus of files. The NCD algorithm in conjunction with the k-nearest-neighbour (k ranging from one to ten) as the classification algorithm was applied to a random selection of circa 3000 512-byte file fragments from 28 different file types. This procedure was then repeated ten times. While the overall accuracy of the n-valued classification only improved the prior probability from approximately 3.5% to circa 32%–36%, the classifier reached accuracies of circa 70% for the most successful file types.
A prototype of a file fragment classifier was then developed and evaluated on new set of data (from the same corpus). Some circa 3000 fragments were selected at random
and the experiment repeated five times. This prototype classifier remained successful at classifying individual file types with accuracies ranging from only slightly lower than 70% for the best class, down to similar accuracies as in the prior experiment.
Subject: Computer Science\Electronic security
Computer Science\Artificial Intelligence