Virtual network testbed II
From FBSD_tips
DRAFT - INCOMPLETE - DRAFT - INCOMPLETE - DRAFT - INCOMPLETE
Contents |
[edit] Rationale
Sometimes there is a need to quickly test a network configuration, e.g. a firewall rule set, but setting up the configuration on real equipment is too time consuming (e.g. physically wiring, installing multiple OSes), expensive (e.g. multiple hosts and switches) and inconvenient (e.g. 'locking yourself out' of routers, out of band access). One workable solution is employing a Virtual Machine software in tandem with FreeBSD's ample palette of network plumbing. This article will detail such an arrangement.
[edit] Design
Diagram of VM and network on host machine :
This is an actual network map built with Tkined, part of the scotty package (in ports). It can be used in conjunction with snmpd either on the host machine or within the guest OSes (as long as they are reachable network-wise).
[edit] VM Software
I selected Qemu as the virtual host platform because :
- It is a full on emulation of the entire machine.
- Multiple copies can run at once on the host OS
- With the KQemu kernel module performance is acceptable
- Arbitrary numbers of NICs can be configured for the guest OSes
- RAW drive type can be built and manipulated by the host OS
This last point is a big change from the virtual network lab #1, which used Qemu COW virtual drives. This meant that all work had to be done inside the boot VM, e.g. adding packages or compiling a kernel. Building a kernel is quite intensive to be doing inside an emulation and adding ports and packages required auxiliary network configuration. The raw disk can be mounted built using host OS tools, chrooted into to configure and have files copied into and out of directly. Additionally I have a centralized and option driven script to commit these actions.
[edit] Network Building Blocks
Freebsd has quite a few building block elements to construct virtual networks. These include (but are not limited to) :
- Tap interface
- Tun interface
- Bridge interface (2 varieties)
- In kernel bridging
- VLANs
I have chosen to connect VMs together using in kernel bridge groups and tap drivers for this implementation. This method is simple and effective.
[edit] References
Scotty/Tkined references :
Qemu :
[edit] Implementation
[edit] Directory Hierarchy
Create a directory hierarchy like this :
/usr/local/qemu/ /usr/local/qemu/router/ /usr/local/qemu/client1/ /usr/local/qemu/client2/ /usr/local/qemu/configs/ /usr/local/qemu/scripts/
[edit] Software Set Up
[edit] Host OS
[edit] Packages
Ports you will need installed :
# Vnc # Qemu (built with the KQemu option) # iftop (or another interface tapping and graphing app)
[edit] Modules
Kernel modules you will need loaded :
# kqemu # aio # bridge # if_tap
[edit] OS sources
You will also need a kernel build with IPFW and DUMMYNET ready to be installed. Make a ne kernel config in /usr/src/sys/i386/conf/ called VNET_IPFW and put these options in it :
include GENERIC options IPFIREWALL options DUMMYNET
Now build this kernel and it will be ready to install in the VMs.
cd /usr/src/ make buildkernel KERNCONF=VNET_IPFW
Next, buildworld :
cd /usr/src/ make buildworld
[edit] Scripts
In the scripts directory put vhost_image.sh. It is the script that will create and configure the virtual hard drives.
vhost_image.sh :
#!/bin/sh
args=`getopt mrtlsp:n:k:c: $*`
if [ $? -ne 0 ]
then
echo 'Usage: ...'
echo '-m : make the image from scratch'
echo '-r : remake the existing image'
echo '-t : mount and chroot the image'
echo '-l : mount the image and start a shell'
echo '-s [FILE] : script to pipe into the chroot'
echo '-p [QDIR] : path to image'
echo '-n [FILE] : name of image'
echo '-k [KERN] : install named kernel'
echo '-c [CONF] : Directory the support files are in'
exit 2
fi
set -- $args
for i
do
case "$i"
in
-m)
ACTION=MAKE
shift;;
-r)
ACTION=REMAKE
shift;;
-t)
ACTION=MOUNT_CHROOT
shift;;
-l)
ACTION=MOUNT_SHELL
shift;;
-s)
ACTION=MOUNT_SCRIPT
SCRIPT=$2
shift; shift;;
-p)
QDIR=$2
shift; shift;;
-n)
FILE=$2
shift; shift;;
-k)
KERN=$2
shift; shift;;
-c)
CONF=$2
shift; shift;;
--)
shift
break;;
esac
done
if [ X${ACTION} = X ]; then echo "action not speficied (-m or -r)"; exit; fi
if [ ! -r ${SCRIPT} ]; then echo "script for chroot does not exist"; exit; fi
if [ X${QDIR} = X ]; then echo "path to image not set"; exit; else echo "QDIR is ${QDIR}"; fi
if [ X${FILE} = X ]; then echo "image FILE not set"; exit; else echo "image is ${FILE}"; fi
if [ X${KERN} = X ]; then KERN=GENERIC; fi
if [ X${CONF} = X ]; then echo "config directory no set"; exit; fi
case "${ACTION}"
in
MAKE|REMAKE)
# Make the QDIR if it doesn't exist
if [ ! -d ${QDIR} ]; then mkdir ${QDIR}; fi
if [ ${ACTION} = "MAKE" ]
then
# Make the file:
dd if=/dev/zero of=${QDIR}/${FILE} bs=1M count=10240
fi
# Make an md device :
MD=`mdconfig -a -t vnode -f ${QDIR}/${FILE}`
# put the boot block on the md device - where is a better source for this?
dd if=${CONF}/boot_block of=/dev/${MD}
# Slice it - slice 1 is the target
fdisk -v -f ${CONF}/fdisk.cfg /dev/${MD}
# Label the disk slice 1
disklabel -R -B /dev/${MD}s1 ${CONF}/disklabel.txt
# make the filesystem
newfs /dev/${MD}s1a
# Mount it somewhere, maiing it if ti doesn't exist
if [ ! -d ${QDIR}/mnt ]; then mkdir ${QDIR}/mnt; fi
mount /dev/${MD}s1a ${QDIR}/mnt
# Install the OS
(
cd /usr/src
echo "installing world"
make installworld DESTDIR=${QDIR}/mnt/
# It needs a device.hints - where is a better source for this?
cp /boot/device.hints ${QDIR}/mnt/boot/device.hints
echo "installing kernel"
make installkernel DESTDIR=${QDIR}/mnt/ KERNCONF=${KERN}
)
# Install all the needed configs
mergemaster -ia -D ${QDIR}/mnt/
chroot ${QDIR}/mnt /usr/sbin/pwd_mkdb -d /etc -p /etc/master.passwd
chroot ${QDIR}/mnt /usr/bin/cap_mkdb /etc/login.conf
# Set up to mount the file systems on boot :
cp ${CONF}/fstab.txt ${QDIR}/mnt/etc/fstab
# clean up
umount ${QDIR}/mnt
mdconfig -d -u ${MD}
;;
MOUNT_*)
# Make an md device :
MD=`mdconfig -a -t vnode -f ${QDIR}/${FILE}`
# Mount it somewhere
mkdir -p ${QDIR}/mnt 2> /dev/null
mount /dev/${MD}s1a ${QDIR}/mnt
# Setup the boot with sysinstall
case $ACTION
in
MOUNT_CHROOT)
echo "entering a chroot shell, on exit it will unmount"
chroot ${QDIR}/mnt/
;;
MOUNT_SHELL)
echo "starting a shell, exitting will unmount"
/bin/sh
;;
MOUNT_SCRIPT)
echo "reading $SCRIPT into a chrooted /bin/sh"
cat $SCRIPT | chroot ${QDIR}/mnt/ /bin/sh
;;
esac
# clean up
umount ${QDIR}/mnt
mdconfig -d -u ${MD}
;;
esac
[edit] Configuration files
Put in /usr/local/qemu/configs/ the following files :
This is used to set up the users in the VMs.
add_users.txt: ------------- echo test | pw useradd -g wheel -d /home/test -s /bin/sh -n test -h0 echo test | pw usermod -n root -h0
This is user to set up the router's rc.conf.
router_rc.txt: ------------- echo ifconfig_ed1=\"inet 10.1.2.1 netmask 255.255.255.0\" > /etc/rc.conf echo ifconfig_ed0=\"inet 10.1.1.1 netmask 255.255.255.0\">> /etc/rc.conf echo sshd_enable=\"YES\" >> /etc/rc.conf echo hostname=\"gate.test.com\" >> /etc/rc.conf
This is used to set up the client1's rc.conf.
client1_rc.txt: -------------- echo ifconfig_ed0=\"inet 10.1.1.2 netmask 255.255.255.0\">> /etc/rc.conf echo sshd_enable=\"YES\" >> /etc/rc.conf echo hostname=\"client1.test.com\" >> /etc/rc.conf
This is used to set up the client2's rc.conf.
client2_rc.txt: -------------- echo ifconfig_ed0=\"inet 10.1.2.2 netmask 255.255.255.0\">> /etc/rc.conf echo sshd_enable=\"YES\" >> /etc/rc.conf echo hostname=\"client1.test.com\" >> /etc/rc.conf
The fdisk data to set up the disk slices.
fdisk.cfg: --------- #start 63, size 20971377 p 1 165 63 20971377
The dislabel.
disklabel.txt: ------------- 8 partitions: # size offset fstype [fsize bsize bps/cpg] a: 18867594 2097152 4.2BSD 2048 16384 28552 b: 2097152 0 swap c: 20964746 0 unused 0 0 # "raw" part, don't edit
The fstab entry, so the VM knows what to mount.
fstab.txt: --------- /dev/ad0s1b none swap sw 0 0 /dev/ad0s1a / ufs rw 1 1
And, put a copy of the boot loader for an IDE drive in "boot_block".
[edit] Router Set Up
Run the vhost_image.sh script in /usr/local/qemu/scripts/ to set up the base router image like so :
/usr/local/qemu/scripts/vhost_image.sh -m \ -p /usr/local/qemu/router/ \ -n router.img \ -k VNET_IPFW \ -c /usr/local/qemu/configs/
Now we will set up the network and hostname.
/usr/local/qemu/scripts/vhost_image.sh \ -s /usr/local/qemu/configs/router_rc.txt \ -p /usr/local/qemu/router/ \ -n router.img \ -k VNET_IPFW \ -c /usr/local/qemu/configs/
Now we will add the test account and password root.
/usr/local/qemu/scripts/vhost_image.sh -s /usr/local/qemu/configs/add_users.txt \ -p /usr/local/qemu/router/ \ -n router.img \ -k VNET_IPFW \ -c /usr/local/qemu/configs/
You now have a router VM disk image made.
[edit] Client1 Set Up
Run the vhost_image.sh script in /usr/local/qemu/scripts/ to set up the base router image like so :
/usr/local/qemu/scripts/vhost_image.sh -m \ -p /usr/local/qemu/client1/ \ -n client1.img \ -k VNET_IPFW \ -c /usr/local/qemu/configs/
Now we will set up the network and hostname.
/usr/local/qemu/scripts/vhost_image.sh \ -s /usr/local/qemu/configs/client1_rc.txt \ -p /usr/local/qemu/client1/ \ -n client1.img \ -c /usr/local/qemu/configs/
Now we will add the test account and password root.
/usr/local/qemu/scripts/vhost_image.sh -s /usr/local/qemu/configs/add_users.txt \ -p /usr/local/qemu/client1/ \ -n client1.img \ -c /usr/local/qemu/configs/
You now have the client1 VM disk image made.
[edit] Client2 Set Up
Run the vhost_image.sh script in /usr/local/qemu/scripts/ to set up the base router image like so :
/usr/local/qemu/scripts/vhost_image.sh -m \ -p /usr/local/qemu/client2/ \ -n client2.img \ -k VNET_IPFW \ -c /usr/local/qemu/configs/
Now we will set up the network and hostname.
/usr/local/qemu/scripts/vhost_image.sh \ -s /usr/local/qemu/configs/client2_rc.txt \ -p /usr/local/qemu/client2/ \ -n client2.img \ -c /usr/local/qemu/configs/
Now we will add the test account and password root.
/usr/local/qemu/scripts/vhost_image.sh -s /usr/local/qemu/configs/add_users.txt \ -p /usr/local/qemu/client2/ \ -n client2.img \ -c /usr/local/qemu/configs/
You now have the client2 VM disk image made.
[edit] Running The Virtual Network
[edit] Booting The Router
In /usr/qemu/router/ put this script file and make it executable :
run.sh: #!/bin/sh touch /dev/tap0 dd if=/dev/tap0 of=/dev/null count=0 2>/dev/null ifconfig tap0 up touch /dev/tap1 dd if=/dev/tap1 of=/dev/null count=0 2>/dev/null ifconfig tap1 up touch /dev/tap4 dd if=/dev/tap4 of=/dev/null count=0 2>/dev/null ifconfig tap4 up qemu -boot c -hda /usr/qemu/router/fbsd_router.img \ -m 128 -vnc localhost:0 \ -pidfile /usr/qemu/router/router.pid \ -daemonize \ -no-reboot \ -net nic,macaddr=00:00:00:00:00:00 -net tap,fd=6,name=tap0,script=no 6<>/dev/tap0 \ -net nic,macaddr=00:00:00:00:00:01 -net tap,fd=7,name=tap1,script=no 7<>/dev/tap1 \ -net nic -net tap,fd=8,name=tap4,script=no 8<>/dev/tap4 vncviwer localhost:0
Now, run it by typing :
cd /usr/local/qemu/router ./run.sh
[edit] Booting Client1
In /usr/qemu/client1/ put this script file and make it executable :
run.sh: #!/bin/sh touch /dev/tap2 dd if=/dev/tap2 of=/dev/null count=0 2>/dev/null ifconfig tap2 up qemu -boot c -hda /usr/qemu/client1/fbsd_client1.img \ -m 128 -vnc localhost:1 \ -pidfile /usr/qemu/client1/client1.pid \ -daemonize \ -no-reboot \ -net nic,macaddr=00:00:00:00:00:02 -net tap,fd=4,name=tap2,script=no 4<>/dev/tap2 vncviewer localhost:1
Now, run it by typing :
cd /usr/local/qemu/client1 ./run.sh
[edit] Booting Client2
In /usr/qemu/client1/ put this script file and make it executable :
run.sh: #!/bin/sh touch /dev/tap2 dd if=/dev/tap2 of=/dev/null count=0 2>/dev/null ifconfig tap2 up qemu -boot c -hda /usr/qemu/client2/fbsd_client1.img \ -m 128 -vnc localhost:2 \ -pidfile /usr/qemu/client1/client1.pid \ -daemonize \ -no-reboot \ -net nic,macaddr=00:00:00:00:00:03 -net tap,fd=4,name=tap2,script=no 4<>/dev/tap2 vncviewer localhost:1
Now, run it by typing :
cd /usr/local/qemu/client2 ./run.sh
[edit] Building The Bridges
Now that you have all the machines running it is time for the bridging. In /usr/local/qemu/scripts/ put this script :
bridge_up.sh : #!/bin/sh sysctl net.link.ether.bridge.enable=1 sysctl net.link.ether.bridge_cfg=tap0:1,tap2:1,tap1:2,tap3:2
You should already have the bridge kld loaded, or bridge built into your kernel. Once you run this script you should be able to login to any of the VMs and ping all the interfaces.
[edit] Tapping The Virtual Network
Find an interface in the virtual network that you are interested in the traffic on and tap it.
iftop tap0
[edit] Examples
[edit] Discussion
[edit] Stuff
This is just one archetype configuration: 1 router between 2 hosts, others could include multiple routers, more bridge groups per router, DMZ arrangements, 'attacking' hosts, etc.
The same way that the router can be brought on to a real network for adding software or other tasks that require actual network access the clients can be as well. You can replace the virtual bride with a bridge to the actual network temporarily and reconnect once you are done. I included a 3rd interface on the router because it is more common for it to have this connection at the same time as the virtual networks than the client hosts.
DUMMYNET could also be used in the host's environment to simulate various other conditions, e.g. an asymetric ISP uplink for instance.
Other OSes that Qemu supports could be used, e.g. a version of windows with a known exploit could be attacked using nessus or other tool, intrusion detection could be employed.
[edit] Stuff that is ugly
- All the : "touch /dev/tap2; dd if=/dev/tap2 of=/dev/null count=0 2>/dev/null; ifconfig tap2 up" Stuff sould really be : "ifconfig tap2 create; ifconfig tap2 up". But on my (6.2-release system) it gives me "ifconfig tap6 create; ifconfig: SIOCIFCREATE: Invalid argument".
- The setup and run scripts should be 1 or 2 script with command line parameters for the different virtual hosts. I will get around to this.
- Qemu seems to only like absolute paths.
- On the router, IPFW can be loaded but DUMMYNET must be compiled in to a new kernel.
- Qemu will occaisionally corrupt it's virtual disk, so MAKE FREQUENT BACKUPS.
