Semester of Graduation

Spring 2020

Degree

Master of Science in Computer Science (MSCS)

Department

Computer Science and Engineering

Document Type

Thesis

Abstract

Advancements in malware development, including the use of file-less and memory-only payloads, have led to a significant interest in the use of volatile memory analysis by digital forensics practitioners. Memory analysis can uncover a wealth of information not available via traditional analysis, such as the discovery of injected code, hooked APIs, and more. Unfortunately, the process of analyzing such malicious code is largely left to analysts who must manually reverse engineer the code to discover its intent. This task is not only slow and error-prone, but is also generally left only to senior-level analysts to perform, given that significant reverse engineering skills are required. This work focuses on the use of code emulation to automatically complete one of the most common tasks of malware analysis -discovering a malware sample’s network activity. Our tool automatically discovers the locations where malware uses networking APIs, emulates the network operations, and records the parameters passed to those functions. Through the monitoring of such parameters, this work enables the automatic discovery of the IP addresses, domain names, and network ports utilized by malware to connect to remote command-and-control (C2) servers as well as accept incoming connections. This novel use of emulation applied to in-memory code provides significant benefits compared to traditional whole-system emulation, which requires a full executable to run and does not match the environment that malware executed during a live incident. In contrast, our approach can emulate any code in memory, including inside of shellcode buffers and memory-only libraries. The novel network API monitoring capabilities developed for this research project were written as an extension to HookTracer, which is an plugin for the Volatility memory analysis framework. HookTracer provides emulation of API hooks in memory, but does not target any specific network activity. The contribution of this work is the incorporation of network API monitoring into HookTracer, development of a test suite that ensures the parameter monitoring is correct, and the evaluation of the techniques we have developed against real-world malware.

Committee Chair

Richard, Golden G III

Available for download on Friday, March 10, 2023

Share

COinS