System-Level Behavior Analysis for Detecting Advanced Persistent Threats (APTs)
DOI:
https://doi.org/10.47672/ejt.2724Abstract
Purpose: Advanced Persistent Threats pose a serious threat in cybersecurity because of their stealth, long presence, and ability to hide. Most organizations placed considerable emphasis on signature-based detection techniques, which were effective against known malware but often failed to detect novel, targeted, or user-specific threats with undefined signatures. This study investigates system-level behavioral analysis as a dynamic alternative for detecting APTs, shifting focus from static indicators to the real-time behavior of processes and applications interacting with the operating system. It emphasizes the importance of identifying abnormal activities such as atypical system call usage, unauthorized process creation, memory injection, and unpredictable modifications to the registry or file system.
Materials and Methods: The research outlines several practical tools and methods used to capture behavioral data, including system call monitoring with strace and Sysmon, process and memory analysis via Process Monitor and Volatility, and registry inspection with Autoruns and Rekall. While these techniques lack automation and often require significant technical expertise, they offer valuable insights into threats that evade conventional antivirus solutions. Findings: The study acknowledges the challenges posed by high false positives, manual rule creation, and scalability limitations but underscores their critical role in laying the groundwork for modern cybersecurity practices.
Unique Contribution to Theory, Practice and Policy: Based on these findings, the study recommends the integration of behavioral detection capabilities into advanced, automated platforms that leverage machine learning and cloud-based analytics. It advocates for a behavior-first approach that prioritizes system-wide visibility and proactive threat hunting over reactive, signature-matching strategies. These recommendations aim to inform the development of AI-driven security solutions capable of detecting complex, evasive threats like APTs in real time and at scale.
Downloads
References
A. J. C. Lima, Advanced Persistent Threats, M.S. thesis, Univ. de Lisboa, Portugal, 2015.
S. Singh et al., "A comprehensive study on APT attacks and countermeasures for future networks and communications: challenges and solutions," J. Supercomputer., vol. 75, pp. 4543–4574, 2019.
P. Bhatt, E. T. Yano, and P. Gustavsson, "Towards a framework to detect multi-stage advanced persistent threats attacks," in Proc. 2014 IEEE 8th Int. Symp. Service Oriented System Engineering, 2014.
B.I.T.S. Forensics, "SANS Institute," 2019.
J. Samuel et al., "Survivable key compromise in software update systems," in Proc. 17th ACM Conf. Computer and Communications Security, 2010.
F. Scrinzi, Behavioral Analysis of Obfuscated Code, M.S. thesis, Univ. of Twente, 2015.
M. Ussath et al., "Advanced persistent threats: Behind the scenes," in Proc. 2016 Annu. Conf. Information Science and Systems (CISS), 2016.
A. Matrosov et al., "Stuxnet under the microscope," ESET LLC, vol. 6, pp. 23, Sept. 2010.
H. Mwiki et al., "Analysis and triage of advanced hacking groups targeting western countries critical national infrastructure: Apt28, Red October, and Regin," in Critical Infrastructure Security and Resilience: Theories, Methods, Tools and Technologies, 2019, pp. 221–244.
B. E. Strom et al., "MITRE ATT&CK: Design and philosophy," Tech. Rep., the MITRE Corporation, 2018.
R. Nikolaev and G. Back, "VirtuOS: An operating system with kernel virtualization," in Proc. 24th ACM Symp. Operating Systems Principles, 2013.
G. Damri and D. Vidyarthi, "Automatic dynamic malware analysis techniques for Linux environment," in Proc. 2016 3rd Int. Conf. Computing for Sustainable Global Development (INDIACom), 2016.
S. Miclea, "Windows and Linux security audit," J. Appl. Bus. Inf. Syst., vol. 3, no. 4, pp. 117, 2012.
F. Nilsson et al., "SysMon–A framework for monitoring and measuring real-time properties," 2012. [Online]. Available: https://www.diva-portal.org/smash/get/diva2:535850
C. M. Anderson and D. Kincaid, "Applying behavior analysis to school violence and discipline problems: Schoolwide positive behavior support," The Behavior Analyst, vol. 28, pp. 49–63, 2005.
F. Apap et al., "Detecting malicious software by monitoring anomalous Windows registry accesses," in Proc. Recent Advances in Intrusion Detection (RAID 2002), Zurich, Switzerland, Oct. 2002.
A. Case and G. G. Richard III, "Memory forensics: The path forward," Digital Investigation, vol. 20, pp. 23–33, 2017.
T. Liggett, Evolution of Endpoint Detection and Response Platforms, M.S. thesis, Utica College, 2018.
M. Holkovič, O. Ryšavý, and J. Dudek, "Automating network security analysis at packet-level by using rule-based engine," in Proc. 6th Conf. Engineering of Computer Based Systems, 2019.
W. Forstmeier, E.-J. Wagenmakers, and T. H. Parker, "Detecting and avoiding likely false-positive findings–a practical guide," Biol. Rev., vol. 92, no. 4, pp. 1941–1968, 2017.
M. Xie, J. Hu, and J. Slay, "Evaluating host-based anomaly detection systems: Application of the one-class SVM algorithm to ADFA-LD," in Proc. 11th Int. Conf. Fuzzy Systems and Knowledge Discovery (FSKD), Xiamen, China, Aug. 2014, pp. 978–982. doi: [10.1109/FSKD.2014.6980965]
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2020 Khaja Kamaluddin
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution (CC-BY) 4.0 License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
