Advanced FreeBSD Installation
From FBSD_tips
WARNING: THESE METHODS ARE DESTRUCTIVE. BACKUP ALL DATA BEFORE YOU START, OR IT WILL BE LOST
This is some of my instructions for 'common' methods of installation that require more work.
[edit] Using a seperate /boot partition
[edit] Creating the partitions
I use a GParted Live CD for this step, as it can handle resizing better. First create a small (256MB) primary partition of type unformatted (to use as /boot), and then a large (a few GB) logical partition of type ext2 (for /). After they are applied, close gparted and run cfdisk from the console. Change the type to a5 (so loader won't get confused) and mark it bootable., write, and boot up a FreeBSD install CD.
Also, while in a console, give the ext2 partition a label, like so (assuming it's hda5, check cfdisk for the actual partition):
tune2fs -L root /dev/hda5
This will be important later.
From now on, we'll assume it's ad0, CHECK THIS FIRST, else you will overwrite the wrong drive.
Use Fixit/CD to get a command prompt, and first, we'll chroot to /mnt2, so we get a complete FreeBSD environment.
chroot /mnt2 mount -t devfs devfs /dev
And load geom label:
geom label load
Now, let's rewrite bootcode with fdisk:
fdisk -iB /dev/ad0
Note the case, it matters. all you need to do is just type n to everything but 'change boot code'
Now, let's make the primary partition bootable:
bsdlabel -wB /dev/ad0s1
And give it a filesystem:
newfs -L boot /dev/ad0s1a
Now, let's mount both drives (and create the mountpoint for /boot):
mount -t ext2fs /dev/ext2fs/root /mnt mkdir /mnt/boot mount /dev/ufs/boot /mnt/boot
Set DESTDIR and extract the distfiles as in Helio/man-install
After they've all been extracted, we'll first prepare the boot system, First by creating a symlink in /boot/boot that points to tbe root of the partition.
cd /mnt/boot/ rm boot ln -s . boot
And Point boot2 to the right loader:
echo "/loader" >boot.config
Now, fill /mnt/boot/loader.conf with the following:
kernel="GENERIC" geom_label_load="YES" ext2fs_load="YES" vfs.root.mountfrom="ext2fs:ext2fs/root"
And /mnt/etc/fstab:
/dev/ext2fs/root / ext2fs rw,noatime 1 1 /dev/ufs/boot /boot ufs ro,noatime 2 2
[edit] Quickly creating a FreeBSD-Based Product
[edit] What is a FreeBSD-Based Product?
Simply, any product or embedded device that uses FreeBSD as the host OS, for example, Juniper Routers, m0n0wall, pfsense, FreeNAS etc. These instructions are a quick (And surprisingly flexible) way of creating a FreeBSD product, which also contains alot of the non-obvious 'gotchas' in the FreeBSD system, and the ways of working around them.
After we go through the basics, there will be some examples of using this method to make:
- PXE bootable varnish cache array
- 'Instant' Redundant Firewalls
- Network Oriented Disk Imager
- Web Browser Kiosk
This not even close to all the uses, these are just common uses for such an arrangement.
[edit] Starting out
This requires you to have an already existing FreeBSD installation, to create the bootable drive; However, we will make sure it's usable no matter /where/ it actually is (ie, you can install it to a USB drive, a eSATA drive, an internal SATA, an internal PATA, an internal CD drive, an external CD drive, a PXE boot server, a zip drive, an MO drive... pretty much anything your BIOS properly supports.
You will also need sysutils/makefs installed.
[edit] Extracting Distfiles
First, let's set DESTDIR to a scratch directory
export DESTDIR=/usr/obj/scratch
and let's create the destination:
mkdir $DESTDIR
Now, let's get a base FreeBSD system (that's clean) to work with.
Note: you could just as easily do this with just a
make installworld && make installkernel
if you decide to, just skip past this part.
First, insert the FreeBSD CD 1, and mount it (let's say, under /dist)
mount -t cd9660 /dev/acd0 /dist
now, cd to /dist/RELEASENAME, for our release:
cd /dist/6.2-RELEASE
this directory contains the distribution sets and the installer files.
the two we definately need are base, and kernels.
first:
cd base
then:
./install.sh
It'll prompt for an 'are you SURE', then begin copying all the system binaries to the hard drive.
now:
cd ../kernels
all dists are installed using a ./install.sh script, some, like kernels', need an extra option. for example, the name of the kernel you want to install. 6 has SMP and GENERIC on the cd, so pick the kernel and type:
./install.sh KERNELNAME
[edit] Scratch directory (Or, messing with hier for fun and profit)
You may be asking yourself now, why did we install everything to scratch instead of the root of the drive? Well, I'll be honest, this is my personal preference. We are going to create what's called an MD Preloaded root filesystem. It'll be heavy (it will take a few MBs of ram, and it will cause a bit more RAM usage then a normal FreeBSD install) but it will have a few added bonuses.
- Booting the product will only require BIOS support, no FreeBSD device support needed.
- You can run FreeBSD on hardware that most FreeBSD users (and a few developers) will think it's impossible to. For example, using this method I booted up FreeBSD 6.2 on an intel DP35M board, using a USB thumb drive with my /boot, and running my personal network disk imaging product to a firewire drive :)
- You can, using this method, boot this product, using the same set of files, from any of the possible ways of booting FreeBSD. This includes booting off a CD/DVD and booting off PXE. The only downside is /usr, the sheer size makes it impractical to add to a filesystem that, by nature, stays entirely in RAM. However, since most products use little in /usr, this is negligable. (And I'll explain how to get the things you /require/ from /usr onto your product)
- Your libs (and your basic binaries) can be safely versioned, and even, upgraded without any fear. The nice side effect is that you can completely remove /rescue from the rootfs, as your libs will be guarenteed to stay in sync with the binaries, and guaranteed to work. You also gain the capability of having multiple products on the same CD, and just editing the bootloader menu to choose between them.
- You only need the boot device for a few seconds on, for example, a single usb thumb drive only needs to be in the machine until the FreeBSD copyright splash after the boot menu, then you can move on to the next machine; no hard drives are needed whatsoever; in fact, the only thing that needs to really work is the RAM and CPU.
[edit] Populating the boot filesystem
Now, you'll want to create two seperate directories at this point, we'll call them BOOTDIR and ROOTDIR
export ROOTDIR=/usr/obj/rootdir && mkdir ${ROOTDIR}
and
export BOOTDIR=/usr/obj/bootdir && mkdir ${BOOTDIR}
what's the difference? Well, BOOTDIR is going to be the actual product; it'll have the kernel, bootloader, and the root fs. ROOTDIR is going to be the contents of the root fs, before it's compressed and put in BOOTDIR.
now, let's copy the boot directory:
cp ${DESTDIR}/boot ${BOOTDIR}/
Why copy? Why not just mv? If you copy, the old version stays in the scratch directory; to roll security updates, all that is required is to use freebsd-update on the scratch directory, and redo the ROOTDIR and BOOTDIR generation (which is explained in the section on #Maintaining A Product
Now we have ourselves the kernel, and bootloader.
[edit] Populating the root filesystem
First, we'll copy /lib to the root filesystem, as it is. I wouldn't recommend attempting to strip anything out of /lib, as binaries are alot more likely to link against them then anything else.
cp ${DESTDIR}/lib ${ROOTDIR}/
remember to copy /libexec too! The linker resides here, without it you can't run most binaries.
cp ${DESTDIR}/libexec ${ROOTDIR}/
now, let's make a /dev directory, as it's needed by the kernel (and you get a nasty kernel panic without it)
mkdir ${ROOTDIR}/dev
let's copy /bin too, as most of the binaries there are close to required anyway:
cp ${DESTDIR}/bin ${ROOTDIR}/
Now, what about /sbin? Normally, you'd copy that to the root directory, however, my personal preference is to just do:
cp ${DESTDIR}/sbin/* ${ROOTDIR}/bin/
then:
ln -s /bin ${ROOTDIR}/sbin
Why? Because it keeps all the binaries in one directory, while still preserving the normal UNIX filesystem (since most products rarely have multiple user's logging in, it's kind of pointless to have a seperate directory anyway). It also simplifies things, as you can just keep PATH=/bin and leave it at that.
Now, let's create some directories a normal FreeBSD install will need:
mkdir ${ROOTDIR}/tmp
mkdir ${ROOTDIR}/var
mkdir ${ROOTDIR}/boot
mkdir ${ROOTDIR}/etc
The first two will stay empty (we'll handle them later), but why /boot? Well if you were to try to use fdisk or bsdlabel's -B flags, they would fail, because it reads the boot sectors from /boot. In fact, while we're here, let's add them now.
cp ${DESTDIR}/boot/boot ${ROOTDIR}/boot/
cp ${DESTDIR}/boot/mbr ${ROOTDIR}/boot/
Now, if you want to use "full screen" apps, you'll need to add the termcap, however, there are some problems with it. Different programs handle termcap differently, so, there's a specific way to handle it.
First, create the directory termcap will be in:
mkdir -p ${ROOTDIR}/usr/share/misc/
and copy the termcap to this directory:
cp ${DESTDIR}/usr/share/misc/termcap ${ROOTDIR}/usr/share/misc/
And lastly, create a symlink in /etc to termcap:
ln -s /usr/share/misc/termcap ${ROOTDIR}/etc/termcap
Now full screen apps will work.
Now, on to the meat and potatoes, /etc. While we could just as well just copy all of /etc, and configure it normally; however, in most cases it's overkill. And since we're loading all of this into ram to stay in ram, I'll just show you what you need to get init to quit complaining, and to have a pretty usable system :)
First, let's copy the passwd and group files, so programs can at least map uid 0 to 'root'. We'll also throw the capability database so that login classes will work properly.
cp ${DESTDIR}/etc/master.passwd ${ROOTDIR}/etc/
cp ${DESTDIR}/etc/passwd ${ROOTDIR}/etc/
cp ${DESTDIR}/etc/group ${ROOTDIR}/etc/
cp ${DESTDIR}/etc/spwd.db ${ROOTDIR}/etc/
cp ${DESTDIR}/etc/pwd.db ${ROOTDIR}/etc/
cp ${DESTDIR}/etc/login.conf ${ROOTDIR}/etc/
Ok, Now what?
Now you get to write the rc script that will run the main app in your product. While you could just as well use the normal rc.d system, for most applications, it's sheer overkill. Instead, I'll show you what you /need/ to do to get a usable system, and you can go from there.
To start, open ${ROOTDIR}/etc/rc with your favorite text editor. We'll start with:
#!/bin/sh VERSION="0.1r1" trap : 2 trap "echo 'Boot Interrupted'" 3
The first line is the start of a normal shell script, the second line is the version number of the product (not needed, but it's easier to have it here and extract it in a build script then any other way), and then two lines trapping signals that might be bad to ignore.
Next, we'll add:
export PATH=/bin export TERM=cons25
This sets our environment to something usable from here on out.
Now, we'll create some directories we'll need for various programs, like /var and /tmp:
mdmfs -s 512k md /var/ mkdir /var/db mkdir /var/db/entropy mkdir /var/db/freebsd-update mkdir /var/db/ipf mkdir /var/db/pkg mkdir /var/db/ports mkdir /var/db/portsnap mkdir /var/empty mkdir /var/log mdmfs -s 1m md /tmp mount -o rw /dev/md0 /
Depending upon your application, you might want to tweak the sizes of the memory filesystems. However, this is sufficient for our task.
Now, you run the commands you need for your product (which is outside the scope of this document)
but remember, to add a reboot command at the end of /etc/rc, so the machine shuts down properly, and to add any commands to clean up after your product in ${ROOTDIR}/etc/rc.shutdown. It will not be cleaned up for you.
[edit] Finishing up BOOTDIR
Let's generate the root.fs image, and compress it with gzip so it doesn't take up that much space:
makefs ${BOOTDIR}/boot/root.fs ${ROOTDIR}
gzip -9 ${BOOTDIR}/boot/root.fs
The kernel is going to need some configuration, so open up ${BOOTDIR}/boot/loader.conf with your favorite text editor.
First, we need to tell loader to load the root filesystem, and point the kernel to the right device name.
mfsroot_load="YES" mfsroot_type="mfsroot" mfsroot_name="/boot/root.fs" vfs.root.mountfrom="ufs:md0"
Note that the name lacks the trailing .gz, loader will check to see if a .gz exists before reading without it.
Next, we'll tell the boot loader which kernel to load, and, any modules we'd like to load too.
kernel="GENERIC" ext2fs_load="YES" pf_load="YES"
Anything that requires a module to be loaded, will have to be done in loader.conf. This includes filesystems, pf, sound card drivers, whatever.
[edit] Making a Bootable Device
[edit] Making a bootable 'Disk'
A 'Disk' on freebsd applies to alot of things, it works for internal hard drives, external hard drives, floppy drives, and zip drives. We'll ignore installing to a floppy, as right now it's far too big to do so.
First, let's write boot code to the target drive, and add a few labels.
You'll need to find out the actual name for the device, in this example, we will use da0 (which just so happens to be my CF card)
fdisk -IB /dev/da0 bsdlabel -wB /dev/da0s1
Now we have a bootable disk on da0, easy, right? Now, let's give it a filesystem so it can boot something useful.
First, however, let's get a 'low collision' name for the filesystem, and add it to an environment variable.
export ROOTNAME=$(dd if=/dev/random bs=8m count=1 2>/dev/null | md5 -q | cut -b1-31)
Kinda complicated, right? Well, it could be less complicated, however, that has a high probability that the device name will be unique (it would be 'mathematically provable to be collision resistant', but, due to a bug in newfs, it can't)
now, let's format it and label it with that name:
newfs -L $ROOTNAME /dev/da0s1a
now, mount it; in our example we'll use /mnt.
mount /dev/ufs/$ROOTNAME /mnt
now, just copy the contents of ${BOOTDIR} like so:
cd $(BOOTDIR} tar -cf - * | tar -xf - -C /mnt/
unmount, and congrats, you just created a bootable drive.
[edit] Making a Product ISO image
This is easier, just do:
mkisofs -b boot/cdboot -no-emul-boot -r -J -V "Product Name" -publisher "Companies Name" -o /path/to/product.iso ${BOOTDIR}
Now burn it with your favorite cd recording tool, or place it on your website.
[edit] PXE Booting
Even easier, just point your TFTP server to use ${BOOTDIR} as the root (and NFS as well), and that's all. If you'd like to get rid of the NFS requirement, you'll need to patch loader and pxeboot: NFS-less pxeboot and loader
[edit] Maintaining a Product
[edit] Automating the steps
[edit] Upgrading FreeBSD in the product
The simplest step, merely do freebsd-update -b ${DESTDIR} fetch install, and re-roll the product. Easy, eh? When you don't need to recompile, this is alot more efficient then building world or running make release.
[edit] Adding ports or programs in /usr
If you try to run a shell from the product, you may notice some tools missing, like for example, more, grep, and awk. These programs are in /usr/bin of a normal FreeBSD system, so they weren't added to your ROOTDIR. I'll show you how to add these to your ROOTDIR, without bringing in the large amount of files in /usr.
First, let's copy the binaries themselves.
cp ${DESTDIR}/usr/bin/more ${DESTDIR}/usr/bin/grep ${DESTDIR}/usr/bin/awk ${ROOTDIR}/bin/
We'll copy them to /bin because we really don't need a second binary file hierarchy, it's just not needed.
Now, we need a list of all the libraries they link against, so that they are in /lib (and the program can actually be run).
Let's start with more:
ldd /usr/bin/more
Which returns:
/usr/bin/more:
libncurses.so.6 => /lib/libncurses.so.6 (0x28091000)
libc.so.6 => /lib/libc.so.6 (0x280d0000)
Look at the list, this is why we just copied all of /lib when we created ${ROOTDIR}, that way we don't have to ldd every single program and figure out what libs it needs and what it doesn't. It greatly simplifies matters.
Now, let's do grep:
ld /usr/bin/grep
which returns:
/usr/bin/grep:
libgnuregex.so.3 => /usr/lib/libgnuregex.so.3 (0x2808b000)
libbz2.so.2 => /usr/lib/libbz2.so.2 (0x2809b000)
libz.so.3 => /lib/libz.so.3 (0x280ac000)
libc.so.6 => /lib/libc.so.6 (0x280bd000)
grep has two libs that aren't in /lib, so we'll copy them now:
cp /usr/lib/libgnuregex.so.3 /usr/lib/libbz2.so.2 ${ROOTDIR}/lib/
And finally, awk:
/usr/bin/awk:
libm.so.4 => /lib/libm.so.4 (0x28095000)
libc.so.6 => /lib/libc.so.6 (0x280ab000)
As you can see, this too doesn't have any libs outside of /lib, so nothing needs to be done.
Now that you know, you can add /usr/bin/sysinstall to your product, and make any of your products usable to install a FreeBSD system. The actual steps will be left as an exercise to the reader.
[edit] Multiple versions on a single drive
This is fairly simple, after you copy boot to the ${ROOTDIR}, you can merely just mv it from GENERIC to PRODUCTv1.0, etc, and use that for the kernel="GENERIC" line in loader.conf.
And when you generate the root image, use a different name (in all parts) for the root.fs, for example, root1.0.fs and root1.0.fs.gz.
In this way your PXE server (or CD) can have multiple versions on the same drive, without having to worry about mixing and matching. In the next section I'll go over how to edit the FreeBSD boot menu, where you can add the 'version chooser'.
[edit] Fun with forth
[edit] Setting product options on the PXE server, and the boot menu
in /etc/rc, you'll need some glue, in our example we'll use the variable 'autostart', and set it to an environment variable in /etc/rc:
export AUTOSTART=$(kenv -q autostart)
It's up to you to make this environment variable usable with your product, for example, you'd have to add some logic to the product to check to see if this environment variable exists, and if it doesn't, make a pretty menu to select or an input box to manually specify.
Now, to set the product option on the PXE server, just add:
autostart="blah"
to the ${BOOTDIR}/boot/loader.conf.
Let's get to the good stuff, and add a menu item for 'autostart'
A word of caution, the bootloader's forth implementation does not handle errors well, if you mistype, the bootloader will show a nasty panic and not let you correct it. So it's not recommended at all to change beastie.4th without testing it in a proper environment first.
First, open ${BOOTDIR}/boot/beastie.4th in your favorite text editor.
Now, if you aren't used to forth, this can be a little daunting, and since this isn't a forth language primer, I'll just show you the necessary things you need to change to edit the boot menu.
Scroll down to this portion of the file:
variable bootsinglekey variable escapekey variable rebootkey
These are variable declarations for each key, add one right afterwards for your option, like so:
variable autostartkey
Now, we use the variable in the menu; scroll down farther to this:
printmenuitem ." Boot FreeBSD in Safe Mode" bootsafekey !
printmenuitem ." Boot FreeBSD in single user mode" bootsinglekey !
printmenuitem ." Boot FreeBSD with verbose logging" bootverbosekey !
printmenuitem ." Escape to loader prompt" escapekey !
And add your own menu item:
printmenuitem ." Autostart Mode Enabled" autostartkey !
Now, we need to add one more thing, the actual 'action' the key takes.
Scroll down further to this:
dup bootsinglekey @ = if
s" YES" s" boot_single" setenv
0 boot
then
dup escapekey @ = if
2drop
s" NO" s" autoboot_delay" setenv
exit
then
Right after the then, add the following:
dup autostartkey @ = if
s" YES" s" autostart" setenv
0 boot
then
Remember to keep a leading " ", and keeping the s" and " portions. The forth interpretor will bomb itself out if they are missing, and you won't be able to boot at all.
You can add as many of these as you want, just remember to use different names for variables in each place. You can also, if you choose, to remove or change some of the other menu items. Just remember to be very careful, and preserve as much of the file as you can.
