Virtual network testbed
From FBSD_tips
DRAFT - DRAFT - DRAFT -
Contents |
[edit] Rationale
Sometimes there is a need to quickly test a network configuration, e.g. a firewall rule set, but setting up the configuration on real equipment is too time consuming (e.g. physically wiring, installing multiple OSes), expensive (e.g. multiple hosts and switches) and inconvenient (e.g. 'locking yourself out' of routers, out of band access). One workable solution is employing a Virtual Machine software in tandem with FreeBSD's ample palette of network plumbing. This article will detail such an arrangement.
[edit] Design
Diagram of VM and network on host machine :
This is an actual network map built with Tkined, part of the scotty package (in ports). It can be used in conjunction with snmpd either on the host machine or within the guest OSes (as long as they are reachable network-wise).
[edit] VM Software
I selected Qemu as the virtual host platform because :
- It is a full on emulation of the entire machine.
- Multiple copies can run at once on the host OS
- With the KQemu kernel module performance is acceptable
- Arbitrary numbers of NICs can be configured for the guest OSes
[edit] Network Building Blocks
Freebsd has quite a few building block elements to construct virtual networks. These include (but are not limited to) :
- Tap interface
- Tun interface
- Bridge interface (2 varieties)
- In kernel bridging
- VLANs
I have chosen to connect VMs together using in kernel bridge groups and tap drivers for this implementation. This method is simple and effective.
[edit] References
Scotty/Tkined references :
Qemu :
[edit] Implementation
[edit] Directory Hierarchy
Create a directory hierarchy like this :
/usr/qemu/ /usr/qemu/router/ /usr/qemu/client1/ /usr/qemu/client2/
Download ftp ftp://ftp.freebsd.org/pub/FreeBSD/ISO-IMAGES-i386/6.2/6.2-RELEASE-i386-disc1.iso into /usr/qemu/.
[edit] Software Set Up
Ports you will need installed :
# Vnc # Qemu (built with the KQemu option) # iftop (or another interface tapping and graphing app)
Kernel modules you will need loaded :
# kqemu # aio # bridge # if_tap
[edit] Router Set Up
In /usr/qemu/router/ put this script file.
setup.sh: #!/bin/sh qemu-img create -f qcow2 /usr/qemu/router/fbsd_router.img 10G qemu -boot d \ -hda /usr/qemu/router/fbsd_router.img \ -m 128 \ -cdrom /usr/qemu/6.2-RELEASE-i386-disc1.iso \ -vnc localhost:0 \ -daemonize \ -no-reboot vncviewer localhost:0
Make this file executable and run it. This will first set up a 10 GB virtual disk, then install to it. Once you reboot at the end of the install it will exit the VM (as per the -no-reboot option).
During the install you should
- Partition one 8GB root and one 2GB swap partitions
- Select the developer package (with all sources)
- Add ports tree
- Network interface ed0 :
- Host name : router
- IP address : 10.1.1.1
- Netmask : 255.255.255.0
- Network interface ed1 :
- Host name : router
- IP address : 10.1.2.1
- Netmask : 255.255.255.0
- Add a user account and put in the wheel group
- Password root.
- Exit installer
You now have a router VM disk image made.
[edit] Client1 Set Up
In /usr/qemu/client1 put this script file.
setup.sh: #!/bin/sh qemu-img create -f qcow2 fbsd_client1.img 10G qemu -boot d \ -hda fbsd_client1.img \ -m 128 \ -cdrom /usr/qemu/6.2-RELEASE-i386-disc1.iso \ -vnc localhost:1 \ -no-reboot vncviewer localhost:1
Make this file executable and run it. This will first set up a 10 GB virtual disk, then install to it. Once you reboot at the end of the install it will exit the VM (as per the -no-reboot option).
During the install you should
- Partition one 8GB root and one 2GB swap partitions
- Select the developer package (with all sources)
- Add ports tree
- Network interface ed0 :
- Host name : client1
- IP address : 10.1.1.2
- Netmask : 255.255.255.0
- Add a user account and put in the wheel group
- Password root.
- Exit installer
You now have the client1 VM disk image made.
[edit] Client2 Set Up
In /usr/qemu/client2 put this script file.
setup.sh:
#!/bin/sh
qemu-img create -f qcow2 fbsd_client2.img 10G
qemu -boot d \
-hda /usr/qemu/client2/fbsd_client2.img \
-m 128 \
-cdrom /usr/qemu/6.2-RELEASE-i386-disc1.iso \
-vnc localhost:2 \
-daemonize \
-no-reboot
vncviewer localhost:2
Make this file executable and run it. This will first set up a 10 GB virtual disk, then install to it. Once you reboot at the end of the install it will exit the VM (as per the -no-reboot option).
During the install you should
- Partition one 8GB root and one 2GB swap partitions
- Select the developer package (with all sources)
- Add ports tree
- Network interface ed0 :
- Host name : router
- IP address : 10.1.2.2
- Netmask : 255.255.255.0
- Add a user account and put in the wheel group
- Password root.
- Exit installer
You now have the client2 VM disk image made.
[edit] Running The Virtual Network
[edit] Booting The Router
In /usr/qemu/router/ put this script file and make it executable :
run.sh: #!/bin/sh touch /dev/tap0 dd if=/dev/tap0 of=/dev/null count=0 2>/dev/null ifconfig tap0 up touch /dev/tap1 dd if=/dev/tap1 of=/dev/null count=0 2>/dev/null ifconfig tap1 up touch /dev/tap4 dd if=/dev/tap4 of=/dev/null count=0 2>/dev/null ifconfig tap4 up qemu -boot c -hda /usr/qemu/router/fbsd_router.img \ -m 128 -vnc localhost:0 \ -pidfile /usr/qemu/router/router.pid \ -daemonize \ -no-reboot \ -net nic,macaddr=00:00:00:00:00:00 -net tap,fd=6,name=tap0,script=no 6<>/dev/tap0 \ -net nic,macaddr=00:00:00:00:00:01 -net tap,fd=7,name=tap1,script=no 7<>/dev/tap1 \ -net nic -net tap,fd=8,name=tap4,script=no 8<>/dev/tap4 vncviwer localhost:0
Because DUMMYNET is part of the rationale for this lab, a new kernel that includes that option will need to be built. Instructions for that are at :
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-building.html
The reason for the 3rd tap interface on the router is to be able to connect the router to the real ethernet for outside access. There are several reasons for this, testing outgoing NAT from the client hosts, adding software from ports or packages, updating the OS, etc. This is how you would connect it to the physical ethernet :
sysctl net.link.ether.bridge_cfg=tap4:3,rl0:3
Assuming that your NIC is an rl0 and you have no bridge group 3 already. Then, in the router virtual machine get an IP address, for instance via dhcp like this :
dhclient ed2
The ed2 interface will be the 3rd interface configured on the qemu command line and associate with the tap4 interface in th host environment.
[edit] Booting Client1
In /usr/qemu/client1/ put this script file and make it executable :
run.sh: #!/bin/sh touch /dev/tap2 dd if=/dev/tap2 of=/dev/null count=0 2>/dev/null ifconfig tap2 up qemu -boot c -hda /usr/qemu/client1/fbsd_client1.img \ -m 128 -vnc localhost:1 \ -pidfile /usr/qemu/client1/client1.pid \ -daemonize \ -no-reboot \ -net nic,macaddr=00:00:00:00:00:02 -net tap,fd=4,name=tap2,script=no 4<>/dev/tap2 vncviewer localhost:1
[edit] Booting Client2
In /usr/qemu/client1/ put this script file and make it executable :
run.sh: #!/bin/sh touch /dev/tap2 dd if=/dev/tap2 of=/dev/null count=0 2>/dev/null ifconfig tap2 up qemu -boot c -hda /usr/qemu/client2/fbsd_client1.img \ -m 128 -vnc localhost:2 \ -pidfile /usr/qemu/client1/client1.pid \ -daemonize \ -no-reboot \ -net nic,macaddr=00:00:00:00:00:03 -net tap,fd=4,name=tap2,script=no 4<>/dev/tap2 vncviewer localhost:1
[edit] Building The Bridges
Now that you have all the machines running it is time for the bridging. In /usr/qemu/ put this script :
bridge_up.sh : #!/bin/sh sysctl net.link.ether.bridge.enable=1 sysctl net.link.ether.bridge_cfg=tap0:1,tap2:1,tap1:2,tap3:2
You should already have the bridge kld loaded, or bridge built into your kernel. Once you run this script you should be able to login to any of the VMs and ping all the interfaces.
[edit] Tapping The Virtual Network
Find an interface in the virtual network that you are interested in the traffic on and tap it.
iftop tap0
[edit] Examples
[edit] Discussion
[edit] Stuff
This is just one archetype configuration: 1 router between 2 hosts, others could include multiple routers, more bridge groups per router, DMZ arrangements, 'attacking' hosts, etc.
The same way that the router can be brought on to a real network for adding software or other tasks that require actual network access the clients can be as well. You can replace the virtual bride with a bridge to the actual network temporarily and reconnect once you are done. I included a 3rd interface on the router because it is more common for it to have this connection at the same time as the virtual networks than the client hosts.
DUMMYNET could also be used in the host's environment to simulate various other conditions, e.g. an asymetric ISP uplink for instance.
Other OSes that Qemu supports could be used, e.g. a version of windows with a known exploit could be attacked using nessus or other tool, intrusion detection could be employed.
[edit] Stuff that is ugly
- All the : "touch /dev/tap2; dd if=/dev/tap2 of=/dev/null count=0 2>/dev/null; ifconfig tap2 up" Stuff sould really be : "ifconfig tap2 create; ifconfig tap2 up". But on my (6.2-release system) it gives me "ifconfig tap6 create; ifconfig: SIOCIFCREATE: Invalid argument".
- The setup and run scripts should be 1 or 2 script with command line parameters for the different virtual hosts. I will get around to this.
- Qemu seems to only like absolute paths.
- On the router, IPFW can be loaded but DUMMYNET must be compiled in to a new kernel.
- Qemu will occaisionally corrupt it's virtual disk, so MAKE FREQUENT BACKUPS.
- the stock VNC seems not to like QEMU's vnc server, TightVNC seems to work fine tho.
