FreeBSD build in WPICS

Quick and dirty:

• Boot usb stick to shell.
• make partitions and filesystem in a way of your choosing, appropriate to the size and purpose of the machine being built (VMs won't use ZFS, for instance).
  • in ZFS I make separate areas for root, /var, /tmp, /usr, /usr/local, /home, and a temporary /workspace area for compiling stuff
• mount the destination dir-tree at /mnt
• extract kernel and base from /usr/freebsd-dist or /usr/share/freebsd-dist:
  • cd /mnt
  • tar xvfUp /usr/freebsd-dist/kernel.txz
  • tar xvfUp /usr/freebsd-dist/base.txz
• chroot to the installed system and do configuration:
  • passwd root
  • adduser (add a user for your use, make sure it is in wheel group or you won't be able to SU to root)
  • tzsetup
  • edits to /etc/rc.conf for
    • hostname
    • network setup for DHCP on primary interface (consider synchronous DHCP)
    • NTP/NTPDATE startup (edit conffile for ntp)
  • PKG repo setup, if necessary
  • sendmail SMARThost setup, if necessary
  • syslog redirection to cslog.cs.wpi.edu
  • NFS mounting of /FreeBSD directory tree
    • remove /usr/src and /usr/obj, replacing with symlinks into the /FreeBSD area.
  • /etc/make.conf containing
    • WRKDIRPREFIX set to some local disk area
    • CCACHE_BUILD setting (you'll need to install ccache of course)
      • ccache should have a quick working area to store its cache in, perhaps in /workspace.
  • /etc/hosts.allow setup (restriction of incoming SSH)
  • /etc/periodic.conf which enables:
    • periodic clearing of temporary areas
    • periodic ZPOOL scrubbing if you're using ZFS.
• If using ZFS don't forget to enable it in /boot/loader.conf and /etc/rc.conf
• install the proper boot loader (pmbr and gptzfsboot or gptboot)

This quick-list surely forgets some items. Once this is all done, use the source tree to rebuild the system, making it up to date. Then start installing the packages you want.

• AMD64 build requires 1G of space in root, because the kernel is larger. Some older versions of FreeBSD don't allocate enough space by default.
• You MUST have at least as much swapspace as RAM on the machine, otherwise it will be impossible to take a crash-dump of the machine when/if it has a kernel panic. With insufficient swap the reboot following a system crash may not succeed, leaving your machine hung. If you don't care about crashdumps they can be disabled in /etc/rc.conf (if they aren't disabled by default).
• Don't think about ZFS unless you have a 64-bit machine (i.e., AMD64) and a lot of RAM. ZFS likes RAM. If you're doing a lot of small file ops on the same machine, esp. operations between ZFS and UFS, be prepared for some fireworks. TEST YOUR MACHINE BEFORE DEPLOYING. Suggestions for ZFS tuning can be located in the FreeBSD Wiki.

• build machine (default partition sizes but double root partition size on amd64; 4 or 5G /usr, all other space to /home)
• install “minimal” set of software.
• map /FreeBSD share from imladris
  • add the host you're building to /etc/exports on imladris
  • mkdir /FreeBSD on the host you're building
  • add this text to the build-host's /etc/fstab:

imladris.wpi.edu:/home/FreeBSD /FreeBSD nfs     rw,noauto,async,nolockd,tcp 0 0

• run sudo mount /FreeBSD
• make symlinks and compile-workspaces:

cd /usr
rm -rf src obj
ln -s /FreeBSD/src-releng_9/src .
ln -s /FreeBSD/ports .
mkdir /home/workspace
mkdir /home/workspace/portwork
mkdir /home/workspace/usr-obj
ln -s /home/workspace/usr-obj ./obj

• use this NFS drive to rebuild the box and make it current.
  • occasionally if you're upgrading from one version of FreeBSD to another, a parallel build will fail. Before deciding that you can't do a build, nuke the /usr/obj/* directories and try the build again without doing a parallel build, i.e., one job at a time (make -j 1). That will sometimes solve problems.

cd /
nice /FreeBSD/build-script >& /tmp/bso &

• the build process will take a while, especially if you're on old hardware.
• In FreeBSD-9 and above, consider using CLANG to do the build, it is faster and less memory-hungry than GCC. But some packages fail to build using CLANG, annoyingly. So if you don't want to worry, just use GCC….
• setup NTP, Email.
• ntp.conf points to a number of FreeBSD NTP machines originally, all we really need is ntp.cs.wpi.edu and perhaps one other on important machines.

• for email, cd to /etc/mail and run “make” to generate a host-specific sendmail.mc file. Edit that file and find the SMART_HOST definition. Uncomment that line and then replace the entry “your.isp.mail.server” with “smtp.wpi.edu”. Then run make install, and place a line into /etc/rc.conf that reads: sendmail_enable=“YES”.

• add packages and other fun stuff. Without /usr/ports/x11/xauth, x11 forwarding via SSH will fail.
  • a good package list: sudo, emacs, bash, portupgrade-devel, libchk, pkg_cutleaves, munin-node, rdiff-backup, nrpe2 (for nagios), mc (for DMagid), MORE??

• setup in /etc/periodic.conf:
  • weekly package checks, clearing /tmp, portaudit.
  • Servers should get a monthly zfs scrub

daily_clean_tmps_days=5                               # If not accessed for
daily_clean_tmps_dirs="/tmp /var/tmp"                   # Delete under here
daily_clean_tmps_enable="YES"                           # Delete stuff daily
weekly_status_pkg_enable="YES"                           # Find out-of-date pkgs

• Setup syslog forwarding in /etc/syslog.conf

*.*                      @cslog.cs.wpi.edu
*.*                      @rous.wpi.edu

• Consider restricting SSH access to on-campus machines only, via /etc/hosts.allow
• Consider adding “vfs.read_max=64” or “vfs.read_max=128” to /etc/sysctl.conf.

• in /etc/pam.d, edit the “passwd” file and uncomment the line with “passwdqc” this will force users to select secure passwords when they change theirs.
• Use Securelevels
• Consider upgrades using the package systrem so that you can react to security patches in the ports collection faster?

Recent, hopefulyl useful instructions can be found in these places:

• http://www.aisecure.net/2011/11/28/root-zfs-freebsd9/
• http://ximalas.info/2011/10/17/zfs-root-fs-on-freebsd-9-0/

Give these a try and see ifthe instructions are viable.

Cut-n-paste until I can do formatting. This procedure is fora two disk system with disks named /dev/da0 and /dev/da1. As of 20130327 I've got this whole process scripted, but the scripts aren't listed here. This example below is for reference, and is not a script.

gpart destroy -F da[01]
gpart create -s gpt da[01]
gpart add -b 34 -s 94 -t freebsd-boot -l boot-da[01] da[01]
gpart add -t freebsd-zfs -l zfs-da[01] da[01]
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da[01]

zfs load (...prints garbage but does load the zfs modules)
zpool create pool mirror /dev/gpt/zfs-da0 /dev/gpt/zfs-da1 (...ignore
      error) ... if you're trying on the Nth time on the same machine,
      you may get an error here saying that one of your disks is "part
      of potentially active pool..." ... repeat the zpool create
      command with the -f flag to override this error... IF there
      aren't actually any other active zpools on the machine.
zfs set checksum=fletcher4 pool
zfs set mountpoint=none pool
zfs set compression=on pool
zfs create pool/root
zfs set mountpoint=/mnt pool/root
zpool set bootfs=pool/root pool
zpool export pool
zpool import -o cachefile=/var/tmp/zpool.cache pool

zfs create -V 4G pool/swap
zfs set compression=off pool/swap
zfs set checksum=off pool/swap
zfs set org.freebsd:swap=on pool/swap
zfs create pool/var
zfs create pool/tmp
zfs create pool/usr
zfs create pool/home

zfs set mountpoint=/mnt/var pool/var
zfs set mountpoint=/mnt/tmp pool/tmp
zfs set mountpoint=/mnt/usr pool/usr
zfs set mountpoint=/mnt/home pool/home
zfs mount -a
mkdir /mnt/var/tmp
chmod 1777 /mnt/tmp /mnt/var/tmp

cd /mnt
tar --unlink -xpJf /usr/freebsd-dist/base.txz
tar --unlink -xpJf /usr/freebsd-dist/kernel.txz

cp /var/tmp/zpool.cache /mnt/boot/zfs/
cd /mnt
vi etc/rc.conf boot/loader.conf :

   add to rc.conf:
    zfs_enable="YES"

   add to loader.conf:
    zfs_load="YES"
    vfs.root.mountfrom="zfs:pool/root"

write these two files and make sure they have no errors, spelling etc.
touch /mnt/etc/fstab
cd /
zfs umount -a
zfs set mountpoint=legacy pool/root
zfs set mountpoint=/var pool/var
zfs set mountpoint=/tmp pool/tmp
zfs set mountpoint=/usr pool/usr
zfs set mountpoint=/home pool/home

Now: reboot onto the disks.  Set root PW, and run tzsetup, etc.