Linux - Hunting of the Shellz with System Tap

If there is one binary that's called the most on Unix platforms it's probably the bash shell. This obviously makes perfect sense given that pulling up a single shell also pulls thousands of tools to your finger tips. Legitimate programs make calls to the shell to execute commands instead of writing their own logic which saves a ton of time and keeps programs lightweight. The ability to pull up a shell is also perhaps one of the most important and consistent features across all malware. Since both legitimate and malicious programs are constantly pulling up shells, how do we know what we should actually be looking for? We'll discuss that here in what is likely to be the first of many different shell hunting blog posts.

A great place to start is by looking for new ttys. A lot of us ignore the importance of the tty even though the creation of one tells us there is hands on activity happening somewhere on the system. The only question that needs to be asked after seeing a new tty get created is do we trust the process that created it? For those unfamiliar with what exactly a tty is, just picture it as the underlying program that's allowing you talk with a shell. For instance, when you open the Terminal application, a new tty is opened allowing you to interact fully with the stdin and stdout. Ttys are actually just special character devices stored in the dev folder. You can view your current tty with the 'tty' command.

There are also items called 'PTY's which are Pseudo-Terminals. This can be thought of shell being used over something like SSH. Where the user is connected to tty device, but that device exists somewhere outside of the local system. We will be able to look at both tty an pty creation using the System Tap language as we did in the last blog post.  Sure, you can craft up a nice little script that looks for tty files being created in the /dev folder, but we could also be more thorough by monitoring for new tty devices at the kernel level using System Tap. Here's the basic code we can use to see when a new tty is created. Upon the creation of a tty the script will return the timestamp, the UID of the user responsible for opening the tty, the name of the process that opened the tty, and its process id.


Let's run this and open a new terminal. We will also login to the system via SSH to ensure that both generate a notification.

Opening a terminal shows that gnome-pty-helpe (presumebly gnome-pty-helper) is the responsible process when opening a terminal via the GUI. When we login via SSH we see that sshd is the responsible process. Don't be confused by the UID 0. This does not mean I logged in as the root user. It means the process that created the TTY is running as root. This makes sense since we're dealing with a daemon.

Based on my system, I would expect to see both of these processes spawning ttys. Therefore, a notification would not be incredibly helpful. Don't get me wrong, logs are always handy for post-mordem analysis, but what I want is a tool that provides notifications when suspicious shells spawn. One approach to take here is to look for ttys spawning from oddball processes on your system. In my use case I'd simply call it normal for "gnome-pty-helpe" and "sshd" to create ttys. Here's my System Tap code to do just that.

Cool. We now have a working script to notify us of ttys from processes we aren't expecting them from.  Nothing too tricky here.

Before we get ahead of ourselves, let's make sure to note the use cases where this script doesn't apply. The last thing you want is some pentester who thinks he's God's gift to hacking telling you he has a way around your detections without understanding what it's actually looking for. Let's demonstrate using a little netcat example. We'll start a listener, connect to it, and run some commands...

Ah. What is this madness? Our tty tracker can't even catch a simple netcat listener? No. THIS IS NOT ITS PURPOSE. The purpose of our script is to catch shells with ttys which netcat does not use. This tty tracker script is not meant to catch all malicious shells, rather it's suppose to catch shells that are guaranteed to be suspicious. If we were looking for all shells without ttys we would have caught this netcat use case, but in the course of about 10 minutes we probably would have caught about a thousand other processes.

However, if the attacker refuses to use a tty he does lose the availability to use interactive programs. For instance, let's say our attacker is in a situation where his backdoor is running as a standard user, but he has figured out the root password and wishes to escalate using sudo.

When running the tty command via netcat we see that we are in fact lacking a tty. When running sudo to escalate to root, the password prompt ends up appearing in the terminal that has the tty. This isn't to say that nothing can be accomplished. We can basically still manage the same amount of damage without a tty, we will just have to work harder for it. But a tty makes life simpler by providing full functionality and therefore is a desirable built in feature for a backdoor.  We can actually get ourselves a tty through this netcat session multiple different ways. Perhaps the easiest is spawning a bash shell using python.




Here we see that the attacker was successfully able to escalate to root using sudo since he had a tty. Of course, now we should have a heads up from our system tap script.




So again, although there are many ways to run malicious commands from shells, this is a way to catch highly interesting hands on activity. It wouldn't be hard to modify this script  to climb process trees allowing us to look for ttys spawning under  web server processes which would also be incredibly abnormal and interesting, but maybe more on that later. :D

Linux - Hunting for Beaconing Using System Tap

I have challenged myself to get myself back into writing consistently after taking a break from it. Hopefully I can keep this blog going longer than one post... so.... here goes.

Ever find yourself dealing with a Linux server that's beaconing to a known bad IP address or domain but shows no other symptoms of being compromised? A lot of attacker malware behaves this way even in the Unix world. When the attacker is not active on the system, the backdoor checks in with the Command and Control server at a specified interval so the attacker still knows it's available. Once you confirm a system is talking to a known bad IP address the goal is generally to get it off the network before the attacker becomes operational again. The downside of pulling the network plug so quickly is that you might find it harder to pinpoint the process that is making the malicious network connections.  Sometimes you just want a little bit more detail before making the decision to bring an important system offline.

The difficult part about trying to detect beaconing is that it happens incredibly quickly. A script junkie may consider throwing a tool like netstat in a while loop and running diff between each execution. Although this isn't a terrible idea, it's very possible that the C2 beacon could occur between netstat executions given that it occurs so quickly.  Enter System Tap. This is a language that gives you access to low level hooks on the operating system and then compiles your script into a kernel module and runs it. Kernel modules that might take you days to write normally can be written in minutes which is perfect for incident response.

In the System Tap documentation there are a lot of examples on how to trace TCP connections. I've also come into contact with Linux malware that beacons to its C2 using UDP packets. So I'm going to take the focus off TCP an put it on UDP.  Here's as basic as it gets if you just want to track UDP connections in real time using System Tap.

For those that have never used System Tap, know that it operates with two primary items. Events and Handlers. In this case our event is udp.sendmesg which occurs when... wait for it.... UDP messages are sent. Inside the brackets we specify that we want to retrieve some data back when this happens. Most data is returned back automatically but we call ctime() to make our timestamp pretty and execname() to automatically look up the process name associated with the UDP call. We can test our little script by running a ping command.

Perfect. By using hardly any lines of code we are able to see the UDP connections our system is making as well as the process responsible!  Let's take this thought a step further.  What if instead of viewing all UDP connections we wanted to write a tool that detects processes beaconing at a steady rate.  To do this, we just have to keep a list of timestamps in which each IP is contacted per process. If the delta is consistent between each timestamp then we will consider the process suspicious and report it to the user.

For those who are familiar with Python you know that calculating something like this wouldn't be too difficult. You could easily record data in a format like the following.

{procname:[timestamp1,timestamp2,timestamp3,timestamp4] }

You could then calculate the difference between each timestamp and determine if the beaconing is steady or not. Unfortunately, System Tap won't be quite as user friendly when it comes to scripting this process out.   I've drafted up a small tool titled "beaconator.stp" which will print out UDP calls and then every 30 seconds check the beaconing to see if there are any processes showing consistent calls. Since many don't have a Linux UDP based backdoor handy, we can use ping and sleep inside a bash script to imitate such a behavior. I use a four second beacon here for testing, but a real beacon is likely to be far more spread out



Checkout the code below or at github. I've commented it as best I could.  I'd be hesitant to run this on any important servers as it's just a PoC. I haven't been highly thorough in my cleanup or tidiness. but take the code and modify it yourself and make it work in your best interest! Also note that this will be far noisier if you're running it on a server of some type.

A few things I love about this approach.

1. It's done 100% from the host level.
2. We can tie any beacon to the process that's performing it. Sometimes this can be difficult even with a memory dump.
3. It's faster than dumping memory and performing analysis.
4. System Tap also allows for cross-instrumentation which under the right circumstances will let you compile your code into a kernel module and use that kernel module on a separate system instead of having to install the dependancies every time.