Licentiate Thesis: Classification of Potentially Unwanted Programs Using Supervised Learning
Defense: April 2013

Malicious software authors have shifted their focus from illegal and clearly malicious software to potentially unwanted programs (PUPs) to earn revenue. PUPs blur the border between legitimate and illegitimate programs, and thus fall within a grey zone. Existing anti-virus and anti-spyware software are, in many instances, unable to detect previously unseen or zero-day attacks and separate PUPs from legitimate software. Many tools also require frequent updates to be effective. By predicting the class of a particular piece of software, users can get support before taking the decision to install the software. This Licentiate thesis introduces approaches to distinguish PUP from legitimate software based on the supervised learning of file features represented as n-grams.

The overall research method applied in this thesis is experiments. For these experiments, malicious software applications were obtained from anti-malware industrial partners. The legitimate software applications were collected from various online repositories. The general steps of supervised learning, from data preparation (n-gram generation) to evaluation, were followed. Different data representations, such as byte codes and operation codes, with different configurations, such as fixed-size, variable-length, and overlap, were investigated to generate different n-gram sizes. The experimental variables were controlled to measure the correlation between n-gram size, the number of features required for optimal training, and classifier performance.

The thesis results suggest that, despite the subtle difference between legitimate software and PUP, this type of software can be classified accurately with a low false positive and false negative rate. The thesis results further suggest an optimal size of operation code-based n-grams for data representation. Finally, the results indicate that classification accuracy can be increased by using a customized ensemble learner who makes use of multiple representations of the data set. The investigated approaches can be implemented as a software tool with a less frequently required update in comparison to existing commercial tools.

Papers Included in licentiate thesis.

Paper I:
Raja Khurram Shahzad, Syed Imran Haider, and Niklas Lavesson, "Detection of Spyware by Mining Executable Files," Proc. International Conference on Availability, Reliability and Security, IEEE Press, pp. 295-302, 2010.

Paper II:
Raja Khurram Shahzad and Niklas Lavesson, "Detecting Scareware by Mining Variable Length Instruction Sequences," Proc. Tenth Annual Information Security South Africa Conference, IEEE Press, 2011.

Paper III:
Raja Khurram Shahzad, Niklas Lavesson, and Henric Johnson, "Accurate Adware Detection using Opcode Sequence Extraction," Proc. Sixth International Conference on Availability, Reliability and Security, IEEE Press, 2011.

Paper IV:
Raja Khurram Shahzad, Niklas Lavesson, "Comparative Analysis of Voting Schemes for Ensemble-based Malware Detection", in press, Journal of Wireless Mobile Networks, Ubiquitous Computing, and Dependable Applications (JoWUA), Vol. 4, No. 1, 2013.

Papers not Included in the licentiate thesis:
Raja Khurram Shahzad and Niklas Lavesson, "Detecting Scareware by Mining Variable Length Instruction Sequences," Extended abstract, Proc. 11th Scandinavian Conference on Artificial Intelligence, pp. 195-197, IOS Press, 2011.

Raja Khurram Shahzad and Niklas Lavesson, "Veto-based Scareware Detection," Proc. Seventh International Conference on Availability, Reliability and Security, IEEE Press, 2012.

Share Dela