Saturday, July 13, 2013

Virtual Security Onion via Ubuntu, KVM & Openvswitch port mirror

I've had a desire to play with KVM and openvswitch for a while.  I've been curious about how capable openvswitch's port mirroring functionality might be (could I mirror one source to multiple destinations for instance). This combined with some recent hardware purchases and the annoyance of booting into windows to manage my ESXi setup at home provided me the motivation this weekend to finally get the ball rolling on redoing the home network altogether with migrating to KVM as the first step.  So I took a new AMD 8 core cpu box with 24 megarams and blew on Ubuntu 13.04 - just because.  This box will host my pfSense firewall that sits between the big bad intertubes and my home network as well as the Security Onion sensor guest to watch traffic that passes through both the inside and outside interfaces.  Here's a crappy diagram. Ignore the FreeNAS junk or check out Smooth's earlier post.



Here's some notes on what I did.  Cobbled together from various blogs and documentation:

1.) I chose the "this machine will be a virtual host" or whatever during the initial install.  Update, install some more stuffs and reboot:


apt-get install openvswitch-datapath-source openvswitch-common openvswitch-switch virt-manager

2.) Modify /etc/network/interfaces:

# The INSIDE Physical Interface
auto eth0
iface eth0 inet manual
up ifconfig $IFACE 0.0.0.0 up

down ifconfig $IFACE down
post-up echo 1 > /proc/sys/net/ipv6/conf/$IFACE/disable_ipv6
post-up ethtool -G $IFACE rx 4096; for i in rx tx sg tso ufo gso gro lro; do ethtool -K $IFACE $i off; done
#
# Management interface for hypervisor
auto br0
iface br0 inet static
 address 10.101.1.5
 gateway 10.101.1.1
 netmask 255.255.255.0
 broadcast 10.101.1.255
post-up echo 1 > /proc/sys/net/ipv6/conf/$IFACE/disable_ipv6
post-up ethtool -G $IFACE rx 4096; for i in rx tx sg tso ufo gso gro lro; do ethtool -K $IFACE $i off; done

# The OUTSIDE Physical Interface
auto eth1
iface eth1 inet manual
up ifconfig $IFACE 0.0.0.0 up

down ifconfig $IFACE down
post-up echo 1 > /proc/sys/net/ipv6/conf/$IFACE/disable_ipv6
post-up ethtool -G $IFACE rx 4096; for i in rx tx sg tso ufo gso gro lro; do ethtool -K $IFACE $i off; done
#

3.) Modify /etc/init/failsafe.conf timeouts to 1

$PLYMOUTH message --text="Waiting for network configuration..." || :
sleep 1

$PLYMOUTH message --text="Waiting up to 60 more seconds for network configuration..." || :
sleep 1
$PLYMOUTH message --text="Booting system without full network configuration..." || :
# give user 1 second to see this message since plymouth will go
# away as soon as failsafe starts.
sleep 1

4.) Create openvswitch bridges and add ports.  Br0 is my home network, Br1 the internet side.

ovs-vsctl add-br br0 
ovs-vsctl add-port br0 eth0
ovs-vsctl add-br br1
ovs-vsctl add-port br1 eth1

5.) Reboot and check openvswitch is running/configured with these commands:

service openvswitch-switch status
ovs-vsctl show
ovs-vsctl list bridge

6.) Create pfSense and Security Onion VMs. Note that virt-manager doesn't currently support creating the network types needed.  I just created the guests using the default network and then edited each virtual machine's interface settings via virsh (sudo virsh edit <name of VM>.  Here is a sample of what my interface settings look like for my Security Onion sensor:

    <interface type='bridge'>
      <mac address='52:54:00:ec:ac:6f'/>
      <source bridge='br0'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='e9e255c6-84ca-2f76-991a-543547638960'/>
      </virtualport>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

After firing up your VM you'll notice new interfaces on your host that correspond to interfaces on your guest.  For instance I have vnet0 that corresponds to the management interface of my sensor while vnet1 and vnet2 are my inside and outside monitoring interfaces respectfully Keep track of these for your port mirror setup.

7.) Create port mirrors.  Here's my inside port mirror config, all one line:


ovs-vsctl -- set Bridge br0 mirrors=@m -- --id=@eth0 get Port eth0 -- --id=@vnet1 get Port vnet1 -- --id=@m create Mirror name=mymirror select-dst-port=@eth0 select-src-port=@eth0 output-port=@vnet1

A few things to note here.  You can remove your port mirror config like this:

ovs-vsctl clear bridge br0 mirrors

You can also add a script to /etc/libvirt/hooks if you so wish.  I modified this dude's script so that it worked for me.

Also, it turns out that you can indeed mirror traffic from one source port to multiple destinations.  Might be handy for dev/testing NSM sensors.  Here's what I tested, all one line:


ovs-vsctl -- set Bridge br0 mirrors=@m,@n -- --id=@eth0 get Port eth0 -- --id=@vnet3 get Port vnet3 -- --id=@vnet5 get Port vnet5 -- --id=@m create Mirror name=mymirror select-dst-port=@eth0 select-src-port=@eth0 output-port=@vnet3 -- --id=@n create Mirror name=secondmirror select-dst-port=@eth0 select-src-port=@eth0 output-port=@vnet5

8.) tcpdump on your Security Onion monitoring interfaces, you should see traffic. Now you can run through your Security Onion setup and rock some NSM.


Helpful links:
Virtio pfSense stuff: http://doc.pfsense.org/index.php/VirtIO_Driver_Support