Debian GNU/kFreeBSD in a FreeBSD Jail - part 2

Wednesday, February 29, 2012

Previously I wrote about getting Debian GNU/kFreeBSD working in a jail. I've worked on it a bit more, polishing things up so I've got it working pretty seamlessly with my existing ezjail FreeBSD jails, so everything starts automatically, and you can use the ezjail commands to stop/restart the jail.

Here are a few more notes about how things got setup for my jail I named debian:

Kernel Modules

In /boot/loader.conf, I added these lines:

fdescfs_load="YES"
linprocfs_load="YES"
linsysfs_load="YES"
tmpfs_load="YES"

Mounting Filesystems

Created /etc/fstab.debian and populated with:

linproc     /jails/debian/proc      linprocfs       rw 0 0
linsys      /jails/debian/sys       linsysfs        rw 0 0
tmpfs       /jails/debian/lib/init/rw   tmpfs       rw 0 0

ezjail Config

Created /usr/local/etc/ezjail/debian with these contents:

export jail_debian_hostname="debian"
export jail_debian_ip="127.0.0.6"
export jail_debian_interface="lo0"
export jail_debian_rootdir="/jails/debian"
export jail_debian_mount_enable="YES"
export jail_debian_devfs_enable="YES"
export jail_debian_devfs_ruleset="devfsrules_jail"
export jail_debian_fdescfs_enable="YES"
export jail_debian_exec_start="/etc/init.d/rc 3"
export jail_debian_flags="-l -u root"

I also tried adding an IPv6 address to the jail, and that seems to work OK

So you can now stop/start with jail with

service ezjail.sh stop debian
service ezjail.sh start debian

Connect to the jail console

If you create a symlink for login (so that from the jail's POV there's a /usr/bin/login, like there would be on a FreeBSD jail)

cd /jails/debian/usr/bin/
ln -s ../../bin/login .

then you can use the ezjail-admin command to get a console in the jail, with:

ezjail-admin console debian

Otherwise, I've been using my own script to get a console (which assumes bash is installed in the jail), named /usr/local/sbin/jlogin

#!/bin/sh
#
# log into a jail, running bash
#
JID=`jls | grep " $1 " | awk '{print $1}'`
exec jexec $JID env -i PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin TERM=$TERM EDITOR=$EDITOR LANG=$LANG HOME=/root bash -l

That runs as:

jlogin debian

Debian GNU/kFreeBSD in a FreeBSD Jail

Sunday, February 26, 2012

I've been a FreeBSD user for quite some time, going back to 3.3 or so, and for the last serveral years have also been working a lot with Ubuntu Linux. So when I ran across Debian GNU/kFreeBSD, which provides a Debian environment on top of a FreeBSD kernel, I was somewhat intrigued. It got even more interesting when I found a tutorial on setting up GNU/kFreeBSD in a jail. The notion of having a Debian environment on my home FreeBSD server without having to get something like VirtualBox running was just too good to pass up.

I got it running fairly decently, but along the way ran into some small problems - and thought I'd jot down what they were and what the fixes were.

FreeBSD Update

At first, I was using FreeBSD 8.2-RELEASE, and used debootstrap to install Debian Squeeze, as the tutorial showed. Once inside the jail, things sort of worked, but most commands, aptitude especially, would die with:

User defined signal 1

It turns out you need a newer kernel than 8.2 to run kFreeBSD in a chroot, as is mentioned in the FAQ. I upgraded my FreeBSD kernel/world to 8.3-PRERELEASE (2012-02-22), and the "signal 1" problem went away.

Debian Update

The next problem was that aptitude would still die, with:

Uncaught exception: Unable to read from stdin: Operation not permitted

After reading about this bug in cwidget, it seemed an upgrade to Wheezy was needed to fix the problem - and sure enough that problem went away.

kbdcontrol and /dev/console

The upgrade to Wheezy didn't go entirely smoothly, mainly due to the kbdcontrol package (required by sysvinit) being unable to access /dev/console in the jail. I wasn't worried about keeping things in the jail isolated for security reasons, so I went ahead and added /dev/console on-the-fly to the running jail by running outside the jail:

devfs -m /jails/debian/dev rule add path 'console*' unhide
devfs -m /jails/debian/dev rule applyset

After that, the kbdcontrol package was able to be upgraded, and I seem to have a Wheezy FreeBSD jail now. Very cool.

UPDATE: A followup talks more about the actual file changes made to run as an ezjail

Automounting ISO images in FreeBSD

Monday, September 26, 2011

Since I've been playing with ISO images a lot lately (see posts tagged: pxe), I thought I'd take a look at making it easier to access their contents, since manually mounting and unmounting them gets to be a drag. It turns out than an Automounter is just what the doctor ordered - a service than will mount a filesystem on demand.

Typically, you'd see automounters mentioned in conjunction with physical CD drives, floppy drives, or NFS mounts - but the idea works just as well for ISO files. This way you can have available both the original ISO image and its contents - but without the contents taking up any additional space.

For FreeBSD, the amd utility will act as our automounter, on Linux systems amd is an option too, but another system called autofs seems to be widely used there - perhaps I'll take a look at that in another post.

Let's start with the desired end result ...

Directory Layout

On my home server I'd like to have this directory layout:

/data/iso/
    images/
        openbsd-4.9-i386.iso                    
        ubuntu-10.04.3-server-amd64.iso
        ubuntu-11.04-server-amd64.iso                    
            .
            .
            .

/data/iso/contents will be where the image contents will be accessible on-the-fly, by directory names based on the iso file names, for example:

/data/iso/
    contents/
        openbsd-4.9-i386/    
            4.9/
            TRANS.TBL
            etc/
        ubuntu-10.04.3-server-amd64/
            README.diskdefines
            cdromupgrade
            dists/
            doc/
            install/
            isolinux/
            md5sum.txt
            .
            .
            .
        ubuntu-11.04-server-amd64/             
        .
        .
        .

Mount/Unmount scripts

amd on FreeBSD doesn't deal directly with ISO files, so we need a couple very small shell scripts than can mount and unmount the images. Let's call the first one /local/iso_mount :

#!/bin/sh
mount -t cd9660 /dev/`mdconfig -f $1` $2

It does two things: first creating a md device based on the given iso filename (the first argument), and mounting the md device at the specified mountpoint (the second argument). Example usage might be:

/local/iso_mount /data/iso/images/ubuntu-11.04-server-amd64.iso /mnt

The second script we'll call /local/iso_unmount

#!/bin/sh
unit=`mdconfig -lv | grep $1 | cut -f 1`
num=`echo $unit | cut -d d -f 2`
umount /dev/$unit
sleep 10
mdconfig -d -u $num

It takes the same parameters as iso_mount. (the sleep call is a bit hackish, but the umount command seems a bit asychronous, and it doesn't seem you can destroy the md device immediately after umount returns - have to give the system a bit of time to finish with the device) To undo our test mount above would be:

/local/iso_unmount /data/iso/images/ubuntu-11.04-server-amd64.iso /mnt

amd Map File

amd is going to need a map file, so that when given a name of a directory that something is attempting to access, it can lookup a location of where to mount it from. For our needs, this can be a one-liner we'll save as /etc/amd.iso-file

*   type:=program;fs:=${autodir}/${key};mount:="/local/iso_mount /local/iso_mount /data/iso/images/${key}.iso ${fs}";unmount:="/local/iso_unmount /local/iso_unmount /data/iso/images/${key}.iso ${fs}"

A map file is a series of lines with

<key> <location>[,<location>,<location>,...]

In our case we've got the wildcard key *, so it'll apply to anything we try to access in /data/iso/contents/, and the location is a semicolon-separated series of directives. type:=program indicates we're specifying mount:= and unmount:= commands to handle this location. ${key} is expanded by amd to be the name of the directory we tried to access.

amd Config File

I decided to use a config file to set things up rather than doing it all as commandline flags, so this is my /etc/amd.conf file:

[ global ]
log_file = syslog

[ /data/iso/contents ]
map_name = /etc/amd.iso-file

Basically telling amd to watch the /data/iso/contents/ directory, and handle attempts to access it based on the map file /etc/amd.iso-file. Also set logging to go to syslog (typically you'd look in /var/log/messages)

Enable it and start

Added these lines to /etc/rc.conf

amd_enable="YES"
amd_flags="-F /etc/amd.conf"

Fire it up with:

service amd start

You should be in business. Unfortunately, if you try

ls /data/iso/contents

the directory will initially appear empty, but if you try

ls /data/iso/contents/openbsd-4.9-i386

you should see a listing of the image's top-level contents (assuming you have a /data/iso/images/openbsd-4.9-i386.iso file). Once an image has been automounted, you will see it in ls /data/iso/contents

Check the mount

If you try:

mount | grep amd

you'll probably seem something like:

/dev/md0 on /.amd_mnt/openbsd-4.9-i386 (cd9660, local, read-only)

The cool thing is, after a couple minutes of inactivity, the mount will go away, and /data/iso/contents will appear empty again.

Manually unmount

The amq utility lets you control the amd daemon, one possibility being to request an unmount to happen now, with for example:

amq -u /data/iso/contents/openbsd-4.9-i386

Conclusion

That's the basics. Now if you're setting up PXE booting and point your Nginx server for example to share /data/iso, you'll be able to reference files within the ISO images, and they'll be available as needed.

Setting up a PXE environment for OS installations

Thursday, September 1, 2011

If you're fooling around with various OSes, installing them by first burning CDs or DVDs gets to be a drag - and you end up with piles of old discs that just go into a landfill. Sure, there are rewritable disks, but they wear out and get scratched eventually. USB memsticks can be painful too - sometimes difficult to create and with different BIOSes having different levels of support.

A slick way to go is to set yourself up to do PXE (Preboot eXecution Environment) installations over a network. Most network cards have had PXE support included for many years now. If you have a machine handy that can act as a simple server, you can have an enviroment where you boot a machine, select the OS you want to install from a menu, and everything will just be pulled over your local network.

There are plenty of writeups on how to PXE install Ubuntu from an Ubuntu server, or FreeBSD from a FreeBSD server - but to make things more interesting and explicit I'll go cross-platform and talk about deploying Ubuntu Server 11.04 from a FreeBSD 8.2 server, and try to make it general enough so that later on we can add other OSes to the menu such as CentOS or OpenBSD.

Requirements

PXE booting a machine requires two basic services be present on your network:

OSes such as Ubuntu or CentOS require a third service:

PXELINUX

For the Network Bootstram Program, we'll use PXELINUX, which is available as part of the SYSLINUX project. The name SYSLINUX is a bit misleading in that it's not actually Linux, but rather a collection of bootloaders that are often used with Linux, and capable of loading other OSes as well. Think of something more along the lines of GRUB, than an actual Linux distro.

To start off with, I'll create a /tftpboot directory, download syslinux-4.04.tar.gz from here, extract and copy two files we want:

mkdir /tftpboot
fetch http://www.kernel.org/pub/linux/utils/boot/syslinux/syslinux-4.04.tar.gz
tar xzvf syslinux-4.04.tar.gz
cp syslinux-4.04/core/pxelinux.0 /tftpboot
cp syslinux-4.04/com32/menu/menu.c32 /tftpboot

We're done with the syslinux download now, so you could clean it up if you want with:

rm -rf syslinux-4.04*

Next, create a configuration directory

mkdir /tftpboot/pxelinux.cfg

and in that directory create a file named default with these initial contents:

DEFAULT menu.c32
PROMPT 0
TIMEOUT 200

LABEL local                           
    MENU LABEL Local Boot
    LOCALBOOT 0

That should be enough to get us a barebones menu when we PXE boot a machine, with a single option to boot off the local harddisk (we'll get to Ubuntu later).

Enable TFTP

TFTP is already included in FreeBSD, just need to make sure it's enabled.

In /etc/inetd.conf make sure this line has the default # removed from the front (so it's not commented out)

tftp   dgram   udp     wait    root    /usr/libexec/tftpd      tftpd -l -s /tftpboot

In /etc/rc.conf, make sure inetd is enabled, adding if necessary:

inetd_enable="YES"

Depending on what you had to do above, start, or reload the inetd daemon with:

service inetd start

or

service inetd reload

Check that the machine is now listing on UDP port 69

sockstat | grep :69

See if you can fetch the NBP using the tftp utility (assuming your server's IPv4 address on the network you'll be doing PXE boots is 10.0.0.1)

cd /tmp
tftp 10.0.0.1
tftp> get /pxelinux.0
tftp> quit
rm pxelinux.0

If it works you should have seen somthing like:

Received 26443 bytes during 0.1 seconds in 53 blocks

Tweak DHCP Server

For this part I'm assuming you're running an ISC dhcpd server (if not, we'll have to cover that in another post). You basically just need to add two lines to /usr/local/etc/dhcpd.conf telling a client what server to use for TFTP and what NBP to fetch:

next-server 10.0.0.1;
filename "/pxelinux.0";

On my server, I just wanted to do this on one particular subnet, so there's a chunk that looks something like this now:

subnet 10.0.0.0 netmask 255.255.255.0 
    {
    range 10.0.0.127 10.0.0.250;
    option routers 10.0.0.1;

    next-server 10.0.0.1;
    filename "/pxelinux.0";
    }

Restart dhcpd

service isc-dhcpd restart

Give it a try

On your client machine, you may have to poke around in the BIOS to enable PXE booting. You'll have to figure out this part for yourself. If you can select your Network Card as the boot device, and everything else is working right, you should see a simple menu something like this:

Initial success

OK! we're at the "Hello World" stage, we know the client and server are doing the bare minimum necessary for PXE to function at all. Time to move on to the good stuff.

Ubuntu Server 11.04

For this next step, I'll assume you've downloaded an ISO into say /foo/ubuntu-11.04-server-amd64.iso The specific version shouldn't matter too much, so if you want to do 10.04 LTS or something else, it should all be about the same.

Mount the ISO image, so we can copy a couple files into /tftpboot and share the rest with a web server.

mkdir -P /iso_images/ubuntu-11.04-server-amd64
mount -t cd9660 /dev/`mdconfig -f /foo/ubuntu-11.04-server-amd64.iso` /iso_images/ubuntu-11.04-server-amd64
mkdir /tftpboot/ubuntu-11.04-server-amd64
cp /iso_images/ubuntu-11.04-server-amd64/install/netboot/ubuntu-installer/amd64/linux /tftpboot/ubuntu-11.04-server-amd64
cp /iso_images/ubuntu-11.04-server-amd64/install/netboot/ubuntu-installer/amd64/initrd.gz /tftpboot/ubuntu-11.04-server-amd64

So now our /tftpboot directory has these five files underneath it:

pxelinux.0
pxelinux.cfg/default
menu.c32
ubuntu-11.04-server-amd64/linux
ubuntu-11.04-server-amd64/initrd.gz

To the /tftpboot/pxelinux.cfg/default file append

LABEL ubuntu-11.04-server-amd64-install             
    MENU LABEL Ubuntu 11.04 Server AMD64 Install
    kernel ubuntu-11.04-server-amd64/linux
    append vga=788 initrd=ubuntu-11.04-server-amd64/initrd.gz

Try PXE booting your client again, this time you'll have "Ubuntu 11.04 Server AMD64 Install" as one of your choices, select that, cross your fingers, and if all goes well in a few seconds you should see:

Initial success

and you can go through and answer the initial questions about the install.

If you're OK with pulling the bulk of the OS over the internet from the official Ubuntu mirrors, it should work although it might be slow. Since we have a nice server sitting on our LAN with a copy of the ISO, we should setup to use that and do a much faster install.

Web Server

For this example, I'll assume nginx has been installed as the webserver (any one will do though, so if you've already got apache installed - that'll work fine too).

The default nginx install uses /usr/local/www/nginx as its docroot, lets put a symlink to our mounted ISO image in there:

ln -s /iso_images/ubuntu-11.04-server-amd64 /usr/local/www/nginx

and also put in a minimal Debian Installer "preseed" file in there that'll help things along by telling the installer to use our webserver for the installation packages. Create a text file named /usr/local/www/nginx/ubuntu-11.04-server-amd64.txt with these contents:

d-i mirror/country string manual
d-i mirror/http/hostname string 10.0.0.1
d-i mirror/http/directory string /ubuntu-11.04-server-amd64
d-i mirror/http/proxy string

Check that you can fetch that file with the URL: http://10.0.0.1/ubuntu-11.04-server-amd64.txt

Edit the /tftpboot/pxelinux.cfg/default file and append

url=http://10.66.0.1/ubuntu-11.04-server-amd64.txt

to the end of the append line of our Ubuntu section, so it now looks like:

LABEL ubuntu-11.04-server-amd64-install             
    MENU LABEL Ubuntu 11.04 Server AMD64 Install
    kernel ubuntu-11.04-server-amd64/linux
    append vga=788 initrd=ubuntu-11.04-server-amd64/initrd.gz url=http://10.66.0.1/ubuntu-11.04-server-amd64.txt

Try PXE booting the Ubuntu install again. You'll still get some initial questions about language and keyboard (we can deal with those in another post), but you shouldn't be asked about mirrors - the installer will know to pull files from your local webserver.

Go through the install on the client, watch the /var/log/nginx-access.log file on the server, you'll see the installer fetching all kinds of files, so you'll know it's all working.

You're in business

So at this point you've got yourself a working PXE installation environment and can do a basic Ubuntu server install.

By adding a few more parameters to your seed file and the PXE configuration you can eliminate some of the installer questions. I'll probably write about that in another post, but if you want to figure it out yourself, check out the Ubuntu Installation Guide - Appendix B. Automating the installation using preseeding

There's so many things you can do with the PXE menus, kernel options, and so on - it can't all be covered in one place. But hopefully you've got a good starting point now, if you know all the basic services are in place and working.