Malware Detection in Encrypted TLS Traffic Through Machine Learning

The proliferation of TLS across the Internet leads to a safer environment for the end user but a more obscure setting for the network defender. This research demonstrates what can be learned using Machine Learning analysis of TLS traffic without decryption. It applies a novel approach to TLS analysis by analyzing data available in the unencrypted portion of the handshake combined with Open-source Intelligence (OSINT) data about Internet Protocol (IP) addresses and domain names. The metadata is then analyzed using three different machine learning algorithms: Support Vector Machine (SVM), One-Class SVM (OC-SVM), and an Autoencoder Neural Network. This research also addresses the imbalanced data distribution between malicious and benign traffic with the OC-SVM and the Autoencoder Neural Network. Finally, this research demonstrates that when using the correct header data the SVM and OC-SVM classify malware with a more than 99% F2 score and the Autoencoder approximately 95% F2.