Ever find yourself dealing with a Linux server that's beaconing to a known bad IP address or domain but shows no other symptoms of being compromised? A lot of attacker malware behaves this way even in the Unix world. When the attacker is not active on the system, the backdoor checks in with the Command and Control server at a specified interval so the attacker still knows it's available. Once you confirm a system is talking to a known bad IP address the goal is generally to get it off the network before the attacker becomes operational again. The downside of pulling the network plug so quickly is that you might find it harder to pinpoint the process that is making the malicious network connections. Sometimes you just want a little bit more detail before making the decision to bring an important system offline.
The difficult part about trying to detect beaconing is that it happens incredibly quickly. A script junkie may consider throwing a tool like netstat in a while loop and running diff between each execution. Although this isn't a terrible idea, it's very possible that the C2 beacon could occur between netstat executions given that it occurs so quickly. Enter System Tap. This is a language that gives you access to low level hooks on the operating system and then compiles your script into a kernel module and runs it. Kernel modules that might take you days to write normally can be written in minutes which is perfect for incident response.
In the System Tap documentation there are a lot of examples on how to trace TCP connections. I've also come into contact with Linux malware that beacons to its C2 using UDP packets. So I'm going to take the focus off TCP an put it on UDP. Here's as basic as it gets if you just want to track UDP connections in real time using System Tap.
For those that have never used System Tap, know that it operates with two primary items. Events and Handlers. In this case our event is udp.sendmesg which occurs when... wait for it.... UDP messages are sent. Inside the brackets we specify that we want to retrieve some data back when this happens. Most data is returned back automatically but we call ctime() to make our timestamp pretty and execname() to automatically look up the process name associated with the UDP call. We can test our little script by running a ping command.
Perfect. By using hardly any lines of code we are able to see the UDP connections our system is making as well as the process responsible! Let's take this thought a step further. What if instead of viewing all UDP connections we wanted to write a tool that detects processes beaconing at a steady rate. To do this, we just have to keep a list of timestamps in which each IP is contacted per process. If the delta is consistent between each timestamp then we will consider the process suspicious and report it to the user.
For those who are familiar with Python you know that calculating something like this wouldn't be too difficult. You could easily record data in a format like the following.
{procname:[timestamp1,timestamp2,timestamp3,timestamp4] }
You could then calculate the difference between each timestamp and determine if the beaconing is steady or not. Unfortunately, System Tap won't be quite as user friendly when it comes to scripting this process out. I've drafted up a small tool titled "beaconator.stp" which will print out UDP calls and then every 30 seconds check the beaconing to see if there are any processes showing consistent calls. Since many don't have a Linux UDP based backdoor handy, we can use ping and sleep inside a bash script to imitate such a behavior. I use a four second beacon here for testing, but a real beacon is likely to be far more spread out
Checkout the code below or at github. I've commented it as best I could. I'd be hesitant to run this on any important servers as it's just a PoC. I haven't been highly thorough in my cleanup or tidiness. but take the code and modify it yourself and make it work in your best interest! Also note that this will be far noisier if you're running it on a server of some type.
Checkout the code below or at github. I've commented it as best I could. I'd be hesitant to run this on any important servers as it's just a PoC. I haven't been highly thorough in my cleanup or tidiness. but take the code and modify it yourself and make it work in your best interest! Also note that this will be far noisier if you're running it on a server of some type.
A few things I love about this approach.
- It's done 100% from the host level.
- We can tie any beacon to the process that's performing it. Sometimes this can be difficult even with a memory dump.
- It's faster than dumping memory and performing analysis.
- System Tap also allows for cross-instrumentation which under the right circumstances will let you compile your code into a kernel module and use that kernel module on a separate system instead of having to install the dependancies every time.
