Analysis and classification of malware based on symbolic execution and machine learning
(2024)
Files
thesis_charleshenryBertrandVO.pdf
Open Access - Adobe PDF
- 3.12 MB
Details
- Authors
- Supervisors
- Legay, Axel
- Abstract
- In recent years, the development of new information and communication technologies has been accompanied by the emergence of new cyber threats. Malware, in particular, poses significant challenges to cybersecurity engineers. New obfuscation and evasion techniques are constantly being developed to modify existing malware and bypass security solutions. Security researchers must therefore constantly improve their detection tools to protect cyberspace. New approaches based on machine learning could help researchers develop new security tools. However, most of these techniques rely on features that can be easily modified by malware authors. It is therefore necessary to explore new ways of representing malware behavior. Symbolic execution seems to be a good candidate to explore the different execution paths of malware and to reveal its malicious behavior. This thesis is dedicated to the exploration of symbolic execution and machine learning for malware analysis. After reviewing existing related approaches, we introduce SEMA (Symbolic Execution for Malware Analysis), which uses symbolic execution to explore malware execution paths and extract System Call Dependency Graphs (SCDGs). SEMA uses various machine learning techniques to detect and identify malware based on SCDGs. We then provide a detailed analysis of machine learning for packing detection, a common obfuscation technique used by malware. Finally, we extend SEMA to support federated learning, enabling collaborative model training without direct dataset sharing.
- Affiliations
Citations
Bertrand Van Ouytsel, C.-H. (2024). Analysis and classification of malware based on symbolic execution and machine learning. https://hdl.handle.net/2078.5/31148