I’ll start with a little bit of back story, if you’re just interested in the tech skip ahead…

FreeBSD and Me

FreeBSD and I have a long history together, in fact just about as long as I’ve been playing with computers. It was ~1996, I had just graduated from high school and managed to find the account sign up page for the University of Kentucky. I signed up for all of the accounts it would give me having no idea what they were since I didn’t start school for another 4 months or so. One of those accounts turned out to be email, pop.uky.edu. The server provided telnet access so I logged on and was greeted with a motd including FreeBSD. Up until that point I’d only been exposed to Windows. I knew that UNIX systems existed, but didn’t really have access to one. I could tell this was a UNIX system and the word Free piqued my interest. It turned out that it was a dual-P2 with a few hundred megabytes of RAM. I was pretty amazed, a pretty modest system hosting 40k users email accounts. A few minutes later I was downloading several (probably 8-12) floppy images and on my way to having a UNIX of my own. I’ve haven’t liked windows since.

I recently purchased a new system primarily intended as a network storage server. It’s a fairly modest, or at least cheap, system in all respects except hard drive space, I have 3 identical Segate drives weighing in at 1.5 TB each. I knew I wanted to give ZFS a try so I started looking at Open Solaris. I never really liked the Solaris env and quickly found Nexenta which seemed to address that complaint. Problem turned out to be that Nexenta is only avaliable in 32-bit flavors, wtf is up with that, and therefore didn’t cope with 4G of RAM or 1.5TB HD (I can’t remember which.) In reading around about ZFS I had run across something talking about ZFS support in FreeBSD. In reality I should of started there, but I wanted to mess around with DTrace as well, little did I know that also has been ported to FreeBSD. So an hour or so later I had a working FreeBSD system, fully running on ZFS.
</backstory>

Requirements

• A system with a spare 3G or more in its own partition (though you’ll want way more for it to be interesting.)
• FreeBSD bootonly iso, burnt on to a CDROM (or another installation method)
• A couple hours to play around with it all

I initally worked off of the process described here, what follows is pretty similar and changes only where I’ve optimized out a step or two or just done things according to my situation, ymmv. As always please read my disclaimer thoroughly.

My Setup

• 3 1.5TB hard drives in fully dedicated disk mode (the only part that’s really relevant)
• New Egg shared wish list for the full details.
• Old PCI 3-com Ethernet card as the NIC on the motherboard didn’t seem to get along with FreeBSD.

FreeBSD Install

I won’t walk through every detail of my process, but some of the big ones include

1. Use entire disk for all 3 disks, installing bootloader on all 3
2. Create a 1G partition on each disk, and allocate the rest to a 2nd partition, the important thing here, since I’m using raidz, is that the involved partitions are identical in size.
• ad4s1a: 1G UFS, / (during install anyway)
• ad4s1d: rest of drive, initially set to mount at /zfs0, but immeidately removed the mount point (takes a long time to format the file system and we don’t need it to be)
• ad6s1a: 1G UFS, /r2, will eventually be used as a redundant/backup root
• ad6s1d: same as ad4s1d
• ad8s1b: 1G swap
• ad8s1d: same as ad4s1d
3. Otherwise install whatever you like, I always start with a minimal install and add things as I need/want them.

Post Install – The Real Stuff

After removing the disk and rebooting select single user mode:

mount -w /

Create the raid pool, allocating whatever partitions you have to use, my command looked like:

zpool create tank raidz /dev/ad3s1d /dev/ad6s1d /dev/ad8s1d

We don’t want to mount the pool as a whole, so unset it’s mountpoint

zfs set mountpoint=none tank

Now we’ll create our root filesystem in the tank pool. We’re specifying a temporary mount point that we can use for now, later we’ll disable mounting for this filesystem as it will be mounted before booting from it.

zfs create -o mountpoint=/tank tank/root

This is just an example of how you would create other filesystems inside of the tank pool. Its safe to create home and go ahead and mount it in its final location so long as you didn’t create users during the intial install process. If you did then you need to do something similar to the root copy we’ll be doing in a second to copy the data from the old partition to the new.

zfs create -o mountpoint=/home tank/home

Check out what you’ve created with

df -h
zfs list

Now enable ZFS on boot

echo 'zfs_enable="YES"' >> /etc/rc.conf

And then copy all of the current, non-ZFS, root data over to the ZFS root partition.

find -x / | cpio -pmd /tank 

No remove the boot directory we just copied over

rm -rf /tank/boot

And then create a point at which the UFS boot directory will be mounted so that we can update it later if need be.

mkdir /tank/bootdir
cd /tank
ln -s bootdir/boot boot

Now we need to tell the UFS bootloader to enable ZFS and to boot from our root ZFS filesystem. We also need to do some ZFS tuning based on information found in the wiki at ZFSTuningGuide, check it out for more info on what’s being set there. The specific values depend mainly on the amount of RAM you have and what (else) you’ll be using the box for.

echo '
vm.kmem_size="1024M"
vm.kmem_size_max="1024M"
#vfs.zfs.arc_max="100M"
zfs_load="YES"
vfs.root.mountfrom="zfs:tank/root"
' >> /boot/loader.conf

Edit the ZFS root’s fstab, /tank/etc/fstab, so that the UFS bootdir is mounted where we pointed the symlink up above, the dev portion of this may vary if you’ve

/dev/ad4s1a /bootdir	ufs	rw	1	1

And finally change the root filesystem’s mountpoint so that ZFS won’t try to mount it automatically.

cd /
zfs set mountpoint=legacy tank/root

Thanks pretty much it, just reboot and check it out.

The last step is to mirror the UFS boot parition on to a 2nd drive in case the first dies, to do this

find -x /bootdir | cpio -pmd /r2

Any time you want to add a new filesystem, they can be very handy for bookkeeping and concise snapshotting, you just do something like the following

zfs create -o mountpoint=/media/music tank/music