Building a diskless FreeBSD 10 Jail server
------------------------------------------
Last edited: $Date: 2013/12/13 14:36:38 $
## Building a new diskless FreeBSD 10.0 Jail server
> FreeBSD Jail is a great lightweight virtualisation method.
> See https://en.wikipedia.org/wiki/FreeBSD_jail
Our project of the day is to build a new diskless system.
The goal is to have a setup on the PXE-server that will allow a
diskless motherboard to run as a FreeBSD 10.0 server. The rootfs
will be mounted over NFS. Also the diskless server mounts a iscsi
virtual disk on which the jails are hosted.
Our diskless server will be running on an old VIA EPIA 600 MHz mini-
ITX board. This means a low power processor and not that much RAM.
In the network there is a SAN that serves both NFS shares as well as
iscsi drives. The root filesystem of our new diskless server will be
mounted over NFS on this SAN. This makes it easy to access the
rootfs from outside the server.
For our jails NFS is not a good choice because starting and running
jails requires file locks. Because of this we choose to mount a
iscsi drive which will hold our jails.
## Steps
* Get FreeBSD 10.0 on a virtual machine
* Get the latest source code
* Build server rootfs with buildworld
* Build kernel for diskless boot with rootfs over NFS
* Mount rootfs over NFS
* Install world and kernel on rootfs
Administrating FreeBSD means practicing patience. We have to
checkout the source code and build a new environment so several
steps will take a lot of time. It helps to plan to do steps during
the night...
## Get FreeBSD 10.0 on a virtual machine
At the time of writing this page, FreeBSD 10.0-BETA4 was available.
By using this beta we get all the new goodies FreeBSD 10 has to
offer.
FreeBSD offers several possibilities to get it up and running, I
downloaded the KVM qcow2 image and started from there.
This is how I started the virtual machine on my Debian kvm-server:
kvm -vnc 192.168.1.25:0 \
-net nic,model=e1000,vlan=0 \
-net tap,ifname=tap1,script=/etc/qemu-ifup-br0\
-m 2048 -hda FreeBSD-10.0-BETA4-i386-20131130-r258774.qcow2
This virtual machine will be used to build the rootfs and the kernel
for our new diskless server.
The time needed to create our new diskless server is mostly
determined by the duration of compiling world and compiling the
kernel. So the computing power of the virtual machine has a lot of
impact on the duration of building the new diskless system.
## Fetch the source code
> svnlite is the new cvsup
Next we get the latest source code to build FreeBSD 10.0. In the
virtual machine we fetch this code with the new svnlite that is the
replacement for cvsup.
svnlite checkout https://svn0.us-west.freebsd.org/base/releng/10.0/ /usr/src
Choose a mirror site close to you. see
http://www.freebsd.org/doc/handbook/svn- mirrors.html
The checkout seems to take ages, so do this at the end of the day
and continue with the following steps the next morning.
### Build server rootfs with buildworld
First, set up /etc/src.conf.
WITHOUT_ATM=yes
WITHOUT_BIND=yes
WITHOUT_BLUETOOTH=yes
WITHOUT_FLOPPY=yes
WITHOUT_GAMES=yes
WITHOUT_INET6=yes
WITHOUT_IPX=yes
WITHOUT_PPP=yes
WITHOUT_SENDMAIL=yes
WITHOUT_SHAREDOCS=yes
WITHOUT_WIRELESS=yes
WITHOUT_ZFS=yes
The VIA EPIA motherboard does not support that much RAM, so we will
not use ZFS. I have not yet played with IPV6, so we first build
without IPV6 support. After having set up /etc/src.conf we can start
building the root filesystem.
Probably we could have disabled more parts of the build to create a
smaller rootfs, but this way we have something to start with that
can be optimized in a later phase.
mount server:/path/to/nfsshare /mnt
setenv NFSROOTDIR /mnt
cd /usr/src
make buildworld
make installworld DESTDIR=${NFSROOTDIR}
### Build kernel for diskless boot with rootfs over NFS
Next we have to build a kernel that allows diskless booting.
Create /usr/src/sys/i386/conf/MYKERNEL as a config file for the
configuration of the kernel.
make buildkernel KERNCONF=MYKERNEL
make installkernel KERNCONF=MYKERNEL DESTDIR=${NFSROOTDIR}
make distribution DESTDIR=${NFSROOTDIR}
## Prepare new environment
Go into the newly created environment, set password and create a new
user, that can be used for ssh login.
chroot ${NFSROOTDIR}
passwd
adduser
echo 'sshd_enable="YES"' > /etc/rc.conf
echo 'defaultrouter="192.168.1.254"' >> /etc/rc.conf
mkdir /iscsi
mkdir -p conf/base
mkdir -p conf/base/iscsi
tar -c -v -f conf/base/etc.cpio.gz --format cpio --gzip etc
tar -c -v -f conf/base/var.cpio.gz --format cpio --gzip var
exit
The /conf/base/var and /conf/base/etc directories become read-write
mounted as /var and /etc on a memory file system in our diskless
system.
The conf/base/iscsi directory becomes read-write mounted as /iscsi
on our diskless system. We need this to be able to mount an
iscsi-drive on which we can install our jails.
## Prepare PXE server
On the PXE/DHCP server we now have to set up an entry for our
diskless server. I use the isc-dhcp-server on a Debian-box. This is
what is needed in /etc/dhcp/dhcpd.conf:
host disklessfreebsdbox {
hardware ethernet 12:34:56:78:90:ab;
fixed-address 192.168.1.9;
filename "freebsd10/boot/pxeboot";
next-server 192.168.1.2;
option domain-name "tradesystem.nl";
option domain-name-servers 192.168.1.1;
option subnet-mask 255.255.255.0;
option root-path "192.168.1.2:/rootfs/freebsd10";
}
You have to change this according to your local situation.
Here is some explanation on the entry above:
* hardware ethernet: the mac address of your diskless system
* fixed-address: this is the ip address to be asigned to your diskless system
* filename "freebsd10/boot/pxeboot": the location of the file named
"pxeboot" which was installed during the execution of the make
commands above, the path is relative to the tftpboot-path on the
PXE-server
* next-server 192.168.1.2: the ip-address of the NFS-server
* option domain-name-servers 192.168.1.1: your DNS-server (mine is local)
* option subnet-mask 255.255.255.0: the netmask for your diskless system
* option root-path "192.168.1.2:/rootfs/freebsd10": the location of
the root filesystem which was installed during the execution of
the make commands above, that is going to be mounted over NFS by
the diskless system.
## Boot diskless
After these preparations are done we can do a test boot of the.
system Solve any problems before going to the next part .
Enter your fresh diskless system and have a look around. Notice that
the root filesystem is mounted read-ony. This is done by the script
inside /etc/rc.initdiskless, this is part of the standard FreeBSD
environment. This is why we need the /conf/base directory on the
NFS- server. The contents of this directory will be mounted in
read-write mode on a memory filesystem.
## Install ezjail
Stop the diskless system. In the rootfilesystem we create a
/etc/iscsi.conf file and run the tar command again to re-create the
cpio format etc.cpio.gz file.
After that chroot again into the root filesystem and install ezjail.
cd ${NFSROOTDIR}
cd etc
cp /etc/resolv.conf .
chroot ${NFSROOTDIR}
pkg install ezjail
Now the pkg repository catalogue will be initialised. Have a cup of
tea, it might take a while.
The total amount of data on the ${NFSROOTDIR} for our diskless
system will now be grown to about 520 Mb.
## Prepare ezjail
### Setup iscsi, mount iscsi drive
Go into the diskless system.
Use the created /etc/iscsi.conf file to connect to the iscsi-server
and mount the iscsi-share on /iscsi on the diskless system.
mkdir -p /iscsi/jails/etc/ezjail
On the virtual machine create a symlink in the NFS-share
chroot ${NFSROOTDIR}
cd /usr/local/etc
rmdir ezjail
ln -s /iscsi/jails/etc/ezjail .
In /usr/local/etc/ezjail.conf set the ezjail_jaildir to the /iscsi
directory
ezjail_jaildir=/iscsi/jails
This way when we create jails in the diskless server, these will be
created on the iscsi-drive, which is the only rewritable partition
on a harddisk.
### Initialize ezjail
ezjail-admin install
After this, create a first jail to test
ezjail-admin create -f example example.com 10.0.0.2
Make sure you have created an alias on your network interface.
This can be done on the virtual machine in the rc.conf on the
${NFSROOTDIR}.
## Using the system
One of the quirks of FreeBSD jails is that for every jail that
is created a file named /etc/fstab. is created, where
is the name of the jail with periods replaced by
underscores.
In our diskless system the /etc directory lives in a memory
filesystem. This means any run-time changes are lost on shutdow of
the system. I have not found a way to change this path. So we have
to conserve these files on a writable harddisk.
What i do is copy the /etc/fstab.* files to /iscsi/jails/etc/ and
after a reboot and a remount of /iscsi copy the files back to /etc.
It is ugly, but it works...
## Resources
To build this server I used a tons of info from the internet. Below
are some very helpfull pages.
* PXE Booting with an NFS Root File System, a chapter from the excellent
FreeBSD handbook, http://www.freebsd.org/doc/handbook/network-pxe-nfs.html
* [Chapter 9.4 about configuring the FreeBSD kernel (chapter from the
excellent FreeBSD handbook,
http://www.freebsd.org/doc/en/books/handbook/kernelconfig-config.htm
$Id: freebsd10_disklessjailserver.txt,v 1.2 2013/12/13 14:36:38 matto Exp $