Introduction

‍

Invary validates the Runtime Integrity of systems, centered around the OS.  By verifying the operating system at runtime, we establish confidence in other attestations we perform across the compute stack.   Together this provides assurance to our customers that their runtime environment, and its security posture, are what they expect it to be;  critical to any Zero Trust and Confidential Computing environments. 

‍

For an in-depth look at how we do that, check out this article by Invary CTO Dr. WesleyPeck: https://invary.com/articles/runtime-integrity-measurement-overview

‍

We have seen a myriad of systems lacking Runtime Integrity from the millions of appraisals we have performed across organizations big and small.  This post will examine two Runtime Integrity appraisals that indicated a compromised system; one caused by kernel level malware and the other caused by misconfiguration of popular EDR/XDR software.  Quick spoiler - they look similar. 

‍

We’ll discuss what that means to you, the negatives and positives from it, and what it means for the future of endpoint security.  

Note: Some details redacted for privacy and security

What does it mean when an OS lacks Runtime Integrity?  

‍

Lacking Runtime Integrity can indicate that a threat actor has control over the operating system via a rootkit or other kernel level malware, altering the kernel in memory at runtime to hide their activity. However, it generally means that something has altered the behavior of a kernel in a way that was not intended by its design.  Interestingly, there are use cases where software with the best of intentions, or even updates to the kernel itself, violate the kernel’s integrity.  

‍

The worst-case scenario is that your operating system kernel has been compromised, giving a threat actor complete control over your system. This compromised kernel is remarkably adept at concealing malicious activity from both you and your security software. You will be at the mercy of the threat actor—unless you are actively validating your Runtime Integrity.   When we demonstrate Invary, we do so by attacking a machine with a rootkit to showcase how it can alter & hide files, processes, logs, and network activity; often with various Endpoint Security software running on the machine.

‍

The best case is that legitimate software that was purposefully installed is violating the kernel’s Runtime Integrity, often so it can examine system activity in an attempt to find indicators of compromise.  Unfortunately this expands the kernel’s attack surface and can negatively affect performance and stability of your kernel. In some environments this is difficult to avoid when your only choice to protect the kernel and machine is to “infect” it. Fortunately frameworks like eBPF in the Linux world (and hopefully a similar implementation for Windows in the future) remove the need to impact Runtime Integrity. 

‍

The anatomy of an Invary Appraisal 

‍

Before showing you the two failed appraisals, it’s helpful to talk a little bit about what an appraisal consists of.  

‍

For Linux kernels, we appraise millions of data points from kernel data structures, objects, function pointers, code sequences, and their relationships.  In general, a failed appraisal highlights the specific locations in memory where the kernel’s integrity has been violated.  We do this across a set of categories we break the kernel into, partly based on intellectual property licensed from the NSA. 

‍

For a simple example:

Here are the high level categories we place failed verifications into

• Required Nodes
• Task Tree
• Task Gates (for older kernels) 
• Data Nodes
• NOPS Table
• Jump Tables 
• Function Pointers 
• Txt Section

These represent common areas where threat actors interpose themselves between the user and the kernel, the hardware and the kernel, or insert their code directly into the kernel. 

‍

The Rootkit

‍

The rootkit shown here is one we modified for testing purposes. Below is a portion of an appraisal taken immediately after the rootkit is implanted.

‍

The rootkit is hooking several system calls, using the “kill” command to receive input.   

‍

With these simple changes to the operating system kernel, this rootkit is now able to: 

• Hide specific files on disk 
• Change the data fetched from file reads to obfuscate logs, change configuration, alter user space executables 
• Hide existing kernel modules and allow the silent addition of new kernel modules 
• Hide network ports, network activity, and by-pass firewalls
• Hide specific running processes and start new processes that are hidden

‍

In addition to placing itself between the user and various system calls, the rootkit also hijacks the data structure responsible for ext4 file operations to both obfuscate and create the ability to interact with specific files or directories in a format unique to it.  

‍

The threat actors goal here is persistence, lateral movement, and exfiltration while remaining undetected.  Despite its damaging nature, it has a comparatively small footprint compared to an EDR solution.  

‍

There are many rootkits in the wild that accomplish similar goals but do not require the above hooking techniques.  These often employ methods like hijacking interrupts or altering in-memory instructions.  If you are not verifying your kernel's Runtime Integrity, you are likely overlooking these threats. Once a threat actor gains control of your kernel, your system is completely compromised, and you can no longer trust the information from your security software.

‍

The Endpoint Security software

‍

This appraisal has around 128 failed checks.

Here is a subset of them: 

‍

1.   Missing @ffffffff8de5ca10:[SyS_read,sys_read], found @ffffffffc06c7460 instead

2.   Missing @ffffffff8de5cae0:[SyS_write,sys_write], found @ffffffffc06c9f40 instead

3.   Missing @ffffffff8de5acf0:[SyS_open,sys_open], found @ffffffffc06c9b90 instead

4.   Missing @ffffffff8de5ad60:[SyS_close,sys_close], found @ffffffffc06c9e20 instead

5.   Missing @ffffffff8dc33190:[SyS_mmap,sys_mmap], found @ffffffffc06c7870 instead

6.   Missing @ffffffff8de71bf0:[SyS_ioctl,sys_ioctl], found @ffffffffc06c68d0 instead

7.   Missing @ffffffff8de5cbb0:[SyS_pread64,sys_pread64], found @ffffffffc06c71c0 instead

8.   Missing @ffffffff8de5cc70:[SyS_pwrite64,sys_pwrite64], found @ffffffffc06c8060 instead

9.   Missing @ffffffff8de5d1c0:[SyS_readv,sys_readv], found @ffffffffc06c7310 instead

10. Missing @ffffffff8de5d2c0:[SyS_writev,sys_writev], found @ffffffffc06c81b0 instead

11. Missing @ffffffff8de5d9e0:[SyS_sendfile64,sys_sendfile64], found @ffffffffc06c7b50 instead

12. Missing @ffffffff8e260860:[SyS_setsockopt,sys_setsockopt], found @ffffffffc06c5230 instead

13. Missing @ffffffff8dcb9370:[SyS_kill,sys_kill], found @ffffffffc06c67a0instead

14. Missing @ffffffff8dcbdbb0:[SyS_newuname,sys_newuname], found @ffffffffc06c5740 instead

15. Missing @ffffffff8de59f50:[SyS_truncate,sys_truncate], found @ffffffffc06c7dd0 instead

16. Missing @ffffffff8de59f90:[SyS_ftruncate,sys_ftruncate], found @ffffffffc06c7ca0 instead

17. Missing @ffffffff8de6ff90:[SyS_rename,sys_rename], found @ffffffffc06c9680 instead

18. Missing @ffffffff8de6fa30:[SyS_mkdir,sys_mkdir], found @ffffffffc06c9290 instead

19. Missing @ffffffff8de6fa50:[SyS_rmdir,sys_rmdir], found @ffffffffc06c6f40 instead

20. Missing @ffffffff8de5ad40:[SyS_creat,sys_creat], found @ffffffffc06c9ce0 instead

21. Missing @ffffffff8de6ff20:[SyS_link,sys_link], found @ffffffffc06c8d80instead

22. Missing @ffffffff8de6fac0:[SyS_unlink,sys_unlink], found @ffffffffc06c9900 instead

23. Missing @ffffffff8de6fbf0:[SyS_symlink,sys_symlink], found @ffffffffc06c6e10 instead

24. Missing @ffffffff8de5a690:[SyS_chmod,sys_chmod], found @ffffffffc06c8ae0 instead

25. Missing @ffffffff8de5a550:[SyS_fchmod,sys_fchmod], found @ffffffffc06c89a0 instead

26. Missing @ffffffff8de5a7d0:[SyS_chown,sys_chown], found @ffffffffc06c8700 instead

27. Missing @ffffffff8de5a830:[SyS_fchown,sys_fchown], found @ffffffffc06c85b0 instead

28. Missing @ffffffff8de5a800:[SyS_lchown,sys_lchown], found @ffffffffc06c8460 instead

29. Missing @ffffffff8dcafbc0:[SyS_ptrace,sys_ptrace], found @ffffffffc06c55f0 instead

30. Missing @ffffffff8dcb93f0:[SyS_rt_sigqueueinfo,sys_rt_sigqueueinfo],found @ffffffffc06c63d0 instead

31. Missing @ffffffff8de93970:[SyS_utime,sys_utime], found @ffffffffc06c6150 instead

32. Missing @ffffffff8de94920:[SyS_statfs,sys_statfs], found @ffffffffc06c54c0 instead

33. Missing @ffffffff8de94960:[SyS_fstatfs,sys_fstatfs], found @ffffffffc06c5390 instead

34. Missing @ffffffff8de839e0:[SyS_mount,sys_mount], found @ffffffffc06c8fe0 instead

35. Missing @ffffffff8de81980:[SyS_umount,sys_umount], found @ffffffffc06c8eb0 instead

36. Missing @ffffffff8dd261c0:[SyS_init_module,sys_init_module], found@ffffffffc06c6b70 instead

37. Missing @ffffffff8de86db0:[SyS_setxattr,sys_setxattr], found @ffffffffc06c5c20 instead

38. Missing @ffffffff8de86fa0:[SyS_fsetxattr,sys_fsetxattr], found @ffffffffc06c5ac0 instead

39. Missing @ffffffff8de874f0:[SyS_removexattr,sys_removexattr], found @ffffffffc06c5990 instead

40. Missing @ffffffff8de876a0:[SyS_fremovexattr,sys_fremovexattr], found @ffffffffc06c5860 instead

41. Missing @ffffffff8dcb93c0:[SyS_tkill,sys_tkill], found @ffffffffc06c6670instead

42. Missing @ffffffff8dcb9390:[SyS_tgkill,sys_tgkill], found @ffffffffc06c6520 instead

43. Missing @ffffffff8de93bf0:[SyS_utimes,sys_utimes], found @ffffffffc06c6020 instead

44. Missing @ffffffff8de5ad20:[SyS_openat,sys_openat], found @ffffffffc06c9a20 instead

45. Missing @ffffffff8de6f920:[SyS_mkdirat,sys_mkdirat], found @ffffffffc06c9140 instead

46. Missing @ffffffff8de5a6b0:[SyS_fchownat,sys_fchownat], found @ffffffffc06c8300 instead

47. Missing @ffffffff8de93ae0:[SyS_futimesat,sys_futimesat], found @ffffffffc06c5ed0 instead

48. Missing @ffffffff8de6fa70:[SyS_unlinkat,sys_unlinkat], found @ffffffffc06c97b0 instead

49. Missing @ffffffff8de6ff70:[SyS_renameat,sys_renameat], found @ffffffffc06c9530 instead

50. Missing @ffffffff8de6fc10:[SyS_linkat,sys_linkat], found @ffffffffc06c8c20 instead

51. Missing @ffffffff8de6fae0:[SyS_symlinkat,sys_symlinkat], found @ffffffffc06c6cc0 instead

52. Missing @ffffffff8de5a5d0:[SyS_fchmodat,sys_fchmodat], found @ffffffffc06c8850 instead

53. Missing @ffffffff8de92410:[SyS_splice,sys_splice], found @ffffffffc06c7700 instead

54. Missing @ffffffff8de93a10:[SyS_utimensat,sys_utimensat], found @ffffffffc06c5d80 instead

55. Missing @ffffffff8de59fd0:[SyS_fallocate,sys_fallocate], found @ffffffffc06c75b0 instead

56. Missing @ffffffff8de5d3c0:[SyS_preadv,sys_preadv], found @ffffffffc06c7060 instead

57. Missing @ffffffff8de5d4b0:[SyS_pwritev,sys_pwritev], found @ffffffffc06c7f00 instead

58. Missing @ffffffff8dcb9430:[SyS_rt_tgsigqueueinfo,sys_rt_tgsigqueueinfo], found @ffffffffc06c6280 instead

59. Missing @ffffffff8dd26300:[SyS_finit_module,sys_finit_module], found @ffffffffc06c6a20 instead

60. Missing @ffffffff8de6ff50:[SyS_renameat2,sys_renameat2], found @ffffffffc06c93d0 instead

61. Missing @ffffffff8de5dc20:[SyS_copy_file_range,sys_copy_file_range],found @ffffffffc06c79e0 instead

‍

This section describes a piece of software injecting itself into most of the syscall table.  We found this on a virtual machine that was thought to be running the same SBOM as other VMs that were passing their Runtime Integrity verifications.  Given that, we were naturally suspicious for a brief period of time that this may be a result of malware, but it quickly became clear that this was EDR software resulting from misconfigured SELinux on the machine.

‍

Why was it so easy to diagnose?   It is unusual for a rootkit to target so much of the kernel systematically, as that level of modification would risk exposure.  Additionally, the calls being hooked are all related to security operations and was likely caused by using well-known interfaces that perform this function for an app. Finally, we had observed this pattern before on other systems known to run this version of the EDR, even though it was not supposed to be enabled here. 

‍

The magnitude of the hooking is still a bit alarming; effectively, this creates a bespoke kernel. This means the performance, stability, and security characteristics of this kernel differ from those of the unmodified kernel itself. In other words, you are using a custom kernel provided by your EDR without knowing it.

‍

Utilizing frameworks like eBPF in Linux is a great way to avoid this, it allows software to analyze system activity without violating the Runtime Integrity of the kernel itself (in most cases, a topic for an upcoming blog).  Invary utilizes eBPF on Linux whenever possible in a unique way that avoids harming the kernel’s performance, expanding the kernel’s attack surface, or impacting its integrity.

‍

There are environments where an eBPF approach is not possible, for example Windows (for now) and some Linux distributions that consider eBPF potentially disruptive  - forcing users who want to protect those systems to make more intrusive changes to the kernel.  In such environments we are required to consider a direct kernel approach. 

‍

What It All Means

‍

It is not a coincidence that both threat actors and security actors alter the kernel, as they are both seeking control of the system despite working toward different outcomes.  Knowing this helps you ask the right questions to protect your system and implement better controls when operating or updating Endpoint Security software, especially if it involves kernel updates. 

‍

Invary can ensure your kernel maintains its integrity across both types of actors, identifying both hidden rootkits as well as unintended vulnerabilities created by EDR and other software packages (even kernel updates themselves).  

Contact us at [email protected] for more info and a demonstration of a rootkit attack

‍