Building a Virtualized Cybersecurity Lab: Splunk SIEM Setup and Log Forwarding

In this third part of my virtualized cybersecurity lab series, I move beyond infrastructure and identity to focus on visibility—installing Splunk, configuring log forwarding, and building the monitoring layer required for SOC-style analysis and detection.

A Need For Robust Logs

With my Active Directory environment fully deployed and all client systems joined to the domain, the next logical step in my homelab journey is establishing visibility. Identity services create the foundation of authentication and access, but without centralized logging, it’s impossible to understand how the environment behaves, detect unusual activity, or analyze the impact of simulated attacks. That’s where a SIEM comes in.

In this third post of the series, I focus on installing and configuring Splunk as my primary SIEM platform, then enabling log forwarding from my Ubuntu Desktop, Windows 11, and Windows Server 2025 VM. My goal is to build a realistic monitoring stack that mirrors what security teams rely on in production environments—capturing authentication logs, system activity, and security-relevant events in one place.

As with all parts of this homelab, I want a system I can continuously modify and learn from. Splunk gives me an excellent starting point for understanding enterprise log ingestion, parsing, and detection logic. In future iterations, I may migrate this VM to Wazuh or run a separate Wazuh deployment alongside Splunk so I can compare SIEM platforms and expand my defensive skill set.

This post represents the beginning of the “visibility” phase of the project—turning raw system activity into actionable insights.

Lab Architecture Reminder

Before diving into configuration, here’s the high-level lab design as of this stage in the project.

Current environment includes:
pfSense (router/firewall)
lab-LAN internal network
Windows Server 2022 (Domain Controller + DNS)
Windows 11 endpoint
Ubuntu Desktop endpoint
With networking complete, the identity layer sits on top of this controlled environment.

Process Overview

For readers who want insight into how I approached the build, I’ve included an overview of the major configuration steps I performed. Readers don't have to read every detail, this is more a summary of the important actions supported by screenshots to document the process.

Step 1 - Installing Splunk on Ubuntu

Install Splunk on the VM:

Example for Ubuntu

sudo dpkg -i splunk-9.0.x-linux-x64.deb
sudo /opt/splunk/bin/splunk start --accept-license
sudo /opt/splunk/bin/splunk enable boot-start

Screenshot below I am using the above commands

Access Splunk WebGUI:
Go to https://Splunk_VM_IP:8000
Default admin login: admin / password you set

Screenshot after logging in

Step 2 - Configure Inputs/Listeners in Splunk

Universal Forwarder Input (for Windows & Linux agents):
Go to Settings → Data Inputs → Forwarded Data / UDP / TCP
Set up:
UDP 514 (for Syslog) for Linux and pfSense
TCP 9997 (default for Splunk forwarders)
Enable receiving logs:

sudo /opt/splunk/bin/splunk enable listen 9997

Below you will see a screenshot of the ports being listened to

Logs we plan to collect

• Security logs
• Sysmon operational logs
• Authentication events
• DNS logs (from DC)
• Linux auth logs
• pfSense firewall logs

Step 3 — Forward Windows Logs

Install Splunk Universal Forwarder on Windows 11 & Server
Download the Universal Forwarder from Splunk website

Install on both Windows VMs
Configure Forwarder to send logs to Splunk server:

Example configuration

c:\Program Files\SplunkUniversalForwarder\bin\splunk add forward-server [Splunk_IP]:9997
c:\Program Files\SplunkUniversalForwarder\bin\splunk add monitor "C:\Windows\System32\winevt\Logs\Security.evtx"
c:\Program Files\SplunkUniversalForwarder\bin\splunk add monitor "C:\Windows\System32\winevt\Logs\System.evtx"
c:\Program Files\SplunkUniversalForwarder\bin\splunk start

Below I am listing the forward server and testing the connection to the Ubuntu Server IP

For richer telemetry, I also installed Sysmon using the SwiftOnSecurity configuration.

Step 4 — Forward Linux Logs

On Ubuntu VMs:
Install rsyslog

sudo apt install rsyslog

Configure rsyslog to forward logs to Splunk:

sudo vim /etc/rsyslog.d/60-splunk.conf

Add:
. @@[Splunk_IP]:514
Restart rsyslog:

sudo systemctl restart rsyslog

This is the content after editing the file

Step 5 — Forward pfSense Logs

Log into pfSense WebGUI
Go to Status → System Logs → Settings
Under Remote Syslog Servers, add the Splunk IP and UDP 514
Select all log types we want forwarded

pfSense sends logs in BSD-style syslog format, which Splunk parses differently than Linux logs.

Step 6 — Verify Logs in Splunk

Go to Splunk Web → Search & Reporting
Search for logs:

index=* | stats count by host, sourcetype

we see entries from Ubuntu Desktop, Windows 11, Server 2022, and pfSense in that order

Key Takeaways / Lessons Learned

Implementing Splunk and configuring log forwarding across every layer of the lab marked a major shift from building infrastructure to actively observing and understanding it. Once identity and authentication services were in place, centralized logging quickly became the backbone of the environment. Working through Splunk Forwarders, Sysmon, rsyslog, and pfSense log forwarding reinforced how critical consistent telemetry, accurate timestamps, and structured data inputs are for effective security monitoring. This phase made it clear that visibility is what transforms a functional network into a defensible one.

• Visibility starts with identity. Once AD was deployed, it became clear how critical centralized logging is for understanding authentication patterns, user behavior, and privilege boundaries.
• Forwarders require careful planning. Splunk Universal Forwarders, Sysmon, and rsyslog each have unique configuration requirements, reinforcing the importance of consistency in log transport and timestamping.
• Telemetry sources differ widely. Windows event logs, Sysmon telemetry, Linux auth logs, and pfSense firewall logs each reveal different layers of system and network activity.
• Search, filtering, and verification matter. Using Splunk queries like index=* stats count by host or looking for EventCode 4624/4625 helped validate that ingestion was functioning end-to-end.
• This stage transforms the lab into a real defensive environment. With logs centralizing into Splunk, the environment has the foundational visibility needed for detection engineering, threat simulation, and SOC-style analysis.

Next Steps / Learning Opportunities

With centralized logging now in place, the lab has reached a point where activity can be observed, recorded, and reviewed across every major system. Rather than immediately expanding detection logic, the next step is to introduce a realistic source of adversarial behavior so that this telemetry has meaningful context.

In the next blog post, I will connect my physical Kali Linux ThinkPad to the lab environment through a VPN tunnel terminating on the pfSense firewall. This will allow the Kali system to function as a controlled attack machine—either simulating an external threat actor accessing the network or a device that has gained internal access through a trusted entry point.

By integrating Kali through pfSense, I can safely generate attack traffic against domain-joined systems while keeping the lab fully isolated from my real network. This step completes the foundational loop of the environment: networking, identity, visibility, and now an adversarial perspective to validate the logging and monitoring already in place.