Converting thin jails to thick jails

Leave a Comment / ezjail, FreeBSD, iocage, Jails, Open Source, ZFS / By

I have been using ezjail since at least 2008 (see earlier blog post). A few years ago, I started deploying iocage on new servers. About three months ago, I starting converting systems from ezjail to iocage.

When I converted my first system, I found that the existing documentation for conversion was incomplete. Specifically, symlinks were a problem.

I raised an issue and wrote a better script which I have since used on a few hosts. Today, as I convert my last ezjail host, I will outline what I’m doing in a blog post so it is easier to follow.

HEADS UP:

  • This was run on a FreeBSD 11.2 system.
  • The thin_to_think script requires rsync to be installed.
  • To automatically start your iocage jails, run this command to add the approprate setting to /etc/rc.conf: 
    sudo sysrc iocage_enable="YES"
  • When you are done with ezjail, remove that setting from /etc/rc.conf with this command: 
    sudo sysrc -X ezjail_enable

My main examples are with simple jail names, no periods, no hyphens. If you have more complex name, please refer to the Other things I had to do for other jails.

Why go thick?

I want to go to thick jails and away from thin jails for several reasons. The biggest of which is: I don’t want to upgrade all of my jails at once. I want to be able to pick and choose when some jails are upgraded. Going thick also allows me to run jails which are purposely on different versions from the host, should I want do so.

Getting iocage ready

Here is what I did before running my scripts.

$ sudo iocage fetch
********************************************************************************
fdescfs(5) is not mounted, performance may suffer. Please run:
mount -t fdescfs null /dev/fd
You can also permanently mount it in /etc/fstab with the following entry:
fdescfs /dev/fd  fdescfs  rw  0  0
********************************************************************************

Creating system/iocage
Creating system/iocage/download
Creating system/iocage/images
Creating system/iocage/jails
Creating system/iocage/log
Creating system/iocage/releases
Creating system/iocage/templates
Default configuration missing, creating one
[0] 11.2-RELEASE
[1] 12.0-RELEASE

Type the number of the desired RELEASE
Press [Enter] to fetch the default selection: (11.2-RELEASE)
Type EXIT to quit: 0

I selected 0 because this host is running 11.2-RELEASE.

Fetching: 11.2-RELEASE

Downloading: MANIFEST [####################] 100% 
Downloading: base.txz [####################] 100% 
Downloading: lib32.txz [####################] 100% 
Downloading: src.txz [####################] 100% 
Extracting: base.txz... 
Extracting: lib32.txz... 
Extracting: src.txz... 

* Updating 11.2-RELEASE to the latest patch level... 
Looking up update.FreeBSD.org mirrors... 3 mirrors found.
Fetching public key from update2.freebsd.org... done.
Fetching metadata signature for 11.2-RELEASE from update2.freebsd.org... done.
Fetching metadata index... done.
Fetching 2 metadata files... done.
Inspecting system... done.
Preparing to download files... done.
Fetching 137 patches.....10....20....30....40....50....60....70....80....90....100....110....120....130... done.
Applying patches... done.
Fetching 2 files... . done.
The following files will be added as part of updating to
11.2-RELEASE-p9:
...

The fetch process continued until:

/usr/src/usr.sbin/bhyve/fwctl.c

WARNING: FreeBSD 11.2-RELEASE is approaching its End-of-Life date.
It is strongly recommended that you upgrade to a newer
release within the next 2 months.
Installing updates... done.

This is OK, no big deal I am moving to iocage before upgrading this host to FreeBSD 12.0 at a later date.

Here is what was created:

$ zfs list -r system/iocage
system/iocage                             1.27G   172G   100K  /system/iocage
system/iocage/download                     271M   172G    88K  /system/iocage/download
system/iocage/download/11.2-RELEASE        270M   172G   270M  /system/iocage/download/11.2-RELEASE
system/iocage/images                        88K   172G    88K  /system/iocage/images
system/iocage/jails                         88K   172G    88K  /system/iocage/jails
system/iocage/log                           88K   172G    88K  /system/iocage/log
system/iocage/releases                    1.01G   172G    88K  /system/iocage/releases
system/iocage/releases/11.2-RELEASE       1.01G   172G    88K  /system/iocage/releases/11.2-RELEASE
system/iocage/releases/11.2-RELEASE/root  1.01G   172G  1.01G  /system/iocage/releases/11.2-RELEASE/root
system/iocage/templates                     88K   172G    88K  /system/iocage/templates

I don’t want to use that mountpoint, I want /iocage instead:

 $ sudo zfs set mountpoint=/iocage system/iocage

And because I know that mountpoint is not otherwise in use:

$ sudo rmdir /system

Now we have this:

$ zfs list -r system/iocage    
NAME                                       USED  AVAIL  REFER  MOUNTPOINT
system/iocage                             1.27G   172G   100K  /iocage
system/iocage/download                     271M   172G    88K  /iocage/download
system/iocage/download/11.2-RELEASE        270M   172G   270M  /iocage/download/11.2-RELEASE
system/iocage/images                        88K   172G    88K  /iocage/images
system/iocage/jails                         88K   172G    88K  /iocage/jails
system/iocage/log                           88K   172G    88K  /iocage/log
system/iocage/releases                    1.01G   172G    88K  /iocage/releases
system/iocage/releases/11.2-RELEASE       1.01G   172G    88K  /iocage/releases/11.2-RELEASE
system/iocage/releases/11.2-RELEASE/root  1.01G   172G  1.01G  /iocage/releases/11.2-RELEASE/root
system/iocage/templates                     88K   172G    88K  /iocage/templates

Repeat for each jail

The following steps are repeated for each jail.

Define what jail we are converting

This is the first part of the example1 script.

export JAIL=ns2

All the folowing steps depend upon ${JAIL}.

Create the new thick jail

I created each new jail like this:

sudo iocage create -r 11.2-RELEASE --thickjail --name ${JAIL}

Now we have this:

$ zfs list -r system/iocage/jails/ns2
NAME                           USED  AVAIL  REFER  MOUNTPOINT
system/iocage/jails/ns2        292K   172G    92K  /iocage/jails/ns2
system/iocage/jails/ns2/root   200K   172G  1.01G  /iocage/jails/ns2/root

Looking inside that root directory, we see what looks like an entirely vanilla FreeBSD filesystem.

$ ls -l /iocage/jails/ns2/root
total 77
-r--r--r--   1 root  wheel  6197 Jun 22  2018 COPYRIGHT
drwxr-xr-x   2 root  wheel    47 Apr  7 14:25 bin
drwxr-xr-x   9 root  wheel    52 Apr  7 14:25 boot
dr-xr-xr-x   2 root  wheel     2 Jun 22  2018 dev
drwxr-xr-x  27 root  wheel   105 Apr  7 15:13 etc
drwxr-xr-x   4 root  wheel    56 Apr  7 14:25 lib
drwxr-xr-x   3 root  wheel     5 Jun 22  2018 libexec
drwxr-xr-x   2 root  wheel     2 Jun 22  2018 media
drwxr-xr-x   2 root  wheel     2 Jun 22  2018 mnt
drwxr-xr-x   2 root  wheel     2 Jun 22  2018 net
dr-xr-xr-x   2 root  wheel     2 Jun 22  2018 proc
drwxr-xr-x   2 root  wheel   148 Apr  7 14:25 rescue
drwxr-xr-x   2 root  wheel     6 Jun 22  2018 root
drwxr-xr-x   2 root  wheel   137 Jun 22  2018 sbin
lrwxr-xr-x   1 root  wheel    11 Apr  7 14:23 sys -> usr/src/sys
drwxrwxrwt   2 root  wheel     2 Jun 22  2018 tmp
drwxr-xr-x  14 root  wheel    14 Jun 22  2018 usr
drwxr-xr-x  24 root  wheel    24 Jun 22  2018 var

Grab information from the old jail and try assigning it into the new jail

I took the example1 script and put it into a note application, and adjusted it for my needs. All I did was set the JAIL variable. Then I copied and pasted part of it into a shell (the part down to and including grep devfs_rule.

NOTE: the example2-hypens.txt script is actually more flexible. I suggest using it instead. Set both JAIL and JAIL_FIXED to the same value until you hit trouble, then see the Other things I had to do for other jails.

Here is what I ran and I have added my comments along the way:

[dan@zuul:~] $ grep USERLAND_VERSION= /usr/jails/basejail/bin/freebsd-version
USERLAND_VERSION="11.2-RELEASE-p8"
[dan@zuul:~] $

Yes, that’s expected, good. Nothing to do there.

[dan@zuul:~] $ grep ip= /usr/local/etc/ezjail/${JAIL}
export jail_ns2_ip="10.80.0.85,162.208.116.85,2610:1c1:0:4:e6aa:8980:e324:a3e9"
[dan@zuul:~] $ grep ip= /usr/local/etc/ezjail/${JAIL} | cut -f 2 -d '"'
10.80.0.85,162.208.116.85,2610:1c1:0:4:e6aa:8980:e324:a3e9
[dan@zuul:~] $ 
[dan@zuul:~] $ # NOTE this is not useful for IPv6 addresses for address dynamically assigned such as em0|10.0.0.1
[dan@zuul:~] $ IPV4=`grep ip= /usr/local/etc/ezjail/${JAIL} | cut -f 2 -d '"'`
[dan@zuul:~] $ 
[dan@zuul:~] $ sudo iocage set ip4_addr="${IPV4}" ${JAIL}
Please provide a valid ip: Expected 4 octets in '2610:1c1:0:4:e6aa:8980:e324:a3e9'

Yes, I will have to manually assign IP addresses because I didn’t want to get too fancy in this script.

[dan@zuul:~] $ 
[dan@zuul:~] $ grep hostname /usr/local/etc/ezjail/${JAIL}
export jail_ns2_hostname="ns2.unixathome.org"
[dan@zuul:~] $ 
[dan@zuul:~] $ JAIL_HOSTNAME=`grep hostname /usr/local/etc/ezjail/${JAIL} | cut -f 2 -d '"'`
[dan@zuul:~] $ 
[dan@zuul:~] $ sudo iocage set host_hostname=${JAIL_HOSTNAME} ${JAIL}
Property: host_hostname has been updated to ns2.unixathome.org

Good, this is automated and what i want.

[dan@zuul:~] $ 
[dan@zuul:~] $ cat /etc/fstab.${JAIL}
/usr/jails/basejail /usr/jails/ns2/basejail nullfs ro 0 0

Nothing special there, no action required.

[dan@zuul:~] $ 
[dan@zuul:~] $ grep devfs_rule /usr/local/etc/ezjail/${JAIL}
export jail_ns2_devfs_ruleset="devfsrules_jail"
[dan@zuul:~] $

Good, simple, no action required.

Action required for that jail

Based on the output of the script, I did the following steps manually:

  1. Set the IP4 IP address[es]: 
    [dan@zuul:~] $ sudo iocage set ip4_addr="10.80.0.85,162.208.116.85" ${JAIL}                                                                                   
Property: ip4_addr has been updated to 10.80.0.85,162.208.116.85
  2. Set the IPv6 IP address[es]: 
    [dan@zuul:~] $ sudo iocage set ip6_addr="2610:1c1:0:4:e6aa:8980:e324:a3e9" ${JAIL}                                                                            
Property: ip6_addr has been updated to 2610:1c1:0:4:e6aa:8980:e324:a3e9

That’s it.

Other things I had to do for other jails

This section shows some of the special things I had to do for certain jails.

More complex jail names

When I went to convert svn.pgcon.org I hit this snag:

[dan@zuul:~] $ export JAIL=svn.pgcon.org
[dan@zuul:~] $ grep USERLAND_VERSION= /usr/jails/basejail/bin/freebsd-version
USERLAND_VERSION="11.2-RELEASE-p8"
[dan@zuul:~] $ 
[dan@zuul:~] $ grep ip= /usr/local/etc/ezjail/${JAIL}
grep: /usr/local/etc/ezjail/svn.pgcon.org: No such file or directory

The errors continued from there. It was because of special case like this that I created example2-hypens.txt.

When I used that example, I set these values:

export JAIL=svn.pgcon.org
export JAIL_FIXED=svn_pgcon_org

With this I got:

[dan@zuul:~] $ grep USERLAND_VERSION= /usr/jails/basejail/bin/freebsd-version
USERLAND_VERSION="11.2-RELEASE-p8"
[dan@zuul:~] $ 
[dan@zuul:~] $ grep ip= /usr/local/etc/ezjail/${JAIL_FIXED}
export jail_svn_pgcon_org_ip="10.80.0.90,162.208.116.90,2610:1c1:0:4::90"
[dan@zuul:~] $ grep ip= /usr/local/etc/ezjail/${JAIL_FIXED} | cut -f 2 -d '"'
10.80.0.90,162.208.116.90,2610:1c1:0:4::90
[dan@zuul:~] $ 
[dan@zuul:~] $ IPV4=`grep ip= /usr/local/etc/ezjail/${JAIL_FIXED} | cut -f 2 -d '"'`
[dan@zuul:~] $ 
[dan@zuul:~] $ sudo iocage set ip4_addr="${IPV4}" ${JAIL}
Please provide a valid ip: Expected 4 octets in '2610:1c1:0:4::90'
[dan@zuul:~] $ 
[dan@zuul:~] $ grep hostname /usr/local/etc/ezjail/${JAIL_FIXED}
export jail_svn_pgcon_org_hostname="svn.pgcon.org"
[dan@zuul:~] $ 
[dan@zuul:~] $ JAIL_HOSTNAME=`grep hostname /usr/local/etc/ezjail/${JAIL_FIXED} | cut -f 2 -d '"'`
[dan@zuul:~] $ 
[dan@zuul:~] $ sudo iocage set host_hostname=${JAIL_HOSTNAME} ${JAIL}
Property: host_hostname has been updated to svn.pgcon.org
[dan@zuul:~] $ 
[dan@zuul:~] $ cat /etc/fstab.${JAIL_FIXED}
/usr/jails/basejail /usr/jails/svn.pgcon.org/basejail nullfs ro 0 0
[dan@zuul:~] $ 
[dan@zuul:~] $ grep devfs_rule /usr/local/etc/ezjail/${JAIL_FIXED}
export jail_svn_pgcon_org_devfs_ruleset="devfsrules_jail"

Which is much easier to deal with. Now I’m back in the same situation as before.

Simple jail name, more complex hostname

Even with this, sometimes the jail short name does not match the hostname and you get this situation:

[dan@zuul:~] $ sudo ~/thin_to_thick.sh /usr/jails/newjail /usr/jails/${JAIL}/ /iocage/jails/${JAIL}/root
The destination jail ('/iocage/jails/FOO/root') must exist and must be a directory
[dan@zuul:~] $ sudo ~/thin_to_thick.sh /usr/jails/newjail /usr/jails/${JAIL}/ /iocage/jails/FOO.example.org/root/

In this case, the required iocage pathname will be obvious if you check it.

Copy over the jail

This step shuts down the old jail and copies over the files.

[dan@zuul:~] $ sudo ezjail-admin stop ${JAIL}
Stopping jails: ns2.unixathome.org.
$ sudo ~/thin_to_thick.sh /usr/jails/newjail /usr/jails/${JAIL}/ /iocage/jails/${JAIL}/root
/basejail exists, and this will interfere with the rsync process
$

Oh, that stuff is there and it will interfere with the rsync stuff. The script makes use of rsync and having that /basejail there will mess with stuff. Those directories do not exist on my other hosts, but it did on this one. Simple solution:

[dan@zuul:/] $ sudo mv basejail DELETEME.basejail
[dan@zuul:/] $ sudo mv basejail2 DELETEME.basejail2
[dan@zuul:/] $

Back to the copy process:

[dan@zuul:~] $ sudo ~/thin_to_thick.sh /usr/jails/newjail /usr/jails/${JAIL}/ /iocage/jails/${JAIL}/root
finding symlinks in new jail: /usr/jails/newjail
original NEW JAIL symlinks, full paths
/usr/jails/newjail/bin
/usr/jails/newjail/boot
/usr/jails/newjail/etc/aliases
/usr/jails/newjail/etc/rmt
/usr/jails/newjail/etc/termcap
/usr/jails/newjail/etc/unbound
/usr/jails/newjail/lib
/usr/jails/newjail/libexec
/usr/jails/newjail/rescue
/usr/jails/newjail/sbin
/usr/jails/newjail/sys
/usr/jails/newjail/usr/bin
/usr/jails/newjail/usr/include
/usr/jails/newjail/usr/lib
/usr/jails/newjail/usr/lib32
/usr/jails/newjail/usr/libdata
/usr/jails/newjail/usr/libexec
/usr/jails/newjail/usr/ports
/usr/jails/newjail/usr/sbin
/usr/jails/newjail/usr/share
/usr/jails/newjail/usr/src
/usr/jails/newjail/usr/tests/lib/libc/tls/libh_tls_dynamic.so
/usr/jails/newjail/usr/tests/lib/libthr/dlopen/h_pthread_dlopen.so
/usr/jails/newjail/usr/tests/libexec/rtld-elf/libpythagoras.so
/usr/jails/newjail/var/db/etcupdate/current/etc/aliases
/usr/jails/newjail/var/db/etcupdate/current/etc/rmt
/usr/jails/newjail/var/db/etcupdate/current/etc/termcap
/usr/jails/newjail/var/db/etcupdate/current/etc/unbound
/usr/jails/newjail/var/db/etcupdate/current/sys
/usr/jails/newjail/var/db/etcupdate/current/usr/share/man/en.ISO8859-1/man1
/usr/jails/newjail/var/db/etcupdate/current/usr/share/man/en.ISO8859-1/man2
/usr/jails/newjail/var/db/etcupdate/current/usr/share/man/en.ISO8859-1/man3
/usr/jails/newjail/var/db/etcupdate/current/usr/share/man/en.ISO8859-1/man4
/usr/jails/newjail/var/db/etcupdate/current/usr/share/man/en.ISO8859-1/man5
/usr/jails/newjail/var/db/etcupdate/current/usr/share/man/en.ISO8859-1/man6
/usr/jails/newjail/var/db/etcupdate/current/usr/share/man/en.ISO8859-1/man7
/usr/jails/newjail/var/db/etcupdate/current/usr/share/man/en.ISO8859-1/man8
/usr/jails/newjail/var/db/etcupdate/current/usr/share/man/en.ISO8859-1/man9
/usr/jails/newjail/var/db/etcupdate/current/usr/share/man/en.ISO8859-15
/usr/jails/newjail/var/db/etcupdate/current/usr/share/man/en.UTF-8/man1
/usr/jails/newjail/var/db/etcupdate/current/usr/share/man/en.UTF-8/man2
/usr/jails/newjail/var/db/etcupdate/current/usr/share/man/en.UTF-8/man3
/usr/jails/newjail/var/db/etcupdate/current/usr/share/man/en.UTF-8/man4
/usr/jails/newjail/var/db/etcupdate/current/usr/share/man/en.UTF-8/man5
/usr/jails/newjail/var/db/etcupdate/current/usr/share/man/en.UTF-8/man6
/usr/jails/newjail/var/db/etcupdate/current/usr/share/man/en.UTF-8/man7
/usr/jails/newjail/var/db/etcupdate/current/usr/share/man/en.UTF-8/man8
/usr/jails/newjail/var/db/etcupdate/current/usr/share/man/en.UTF-8/man9
/usr/jails/newjail/var/db/etcupdate/current/usr/share/nls/POSIX
/usr/jails/newjail/var/db/etcupdate/current/usr/share/nls/en_US.US-ASCII
/usr/jails/newjail/var/db/etcupdate/current/usr/share/openssl/man/en.ISO8859-1/man1
/usr/jails/newjail/var/db/etcupdate/current/usr/share/openssl/man/en.ISO8859-1/man3
/usr/jails/newjail/var/db/etcupdate/current/usr/share/openssl/man/en.ISO8859-15
/usr/jails/newjail/var/yp/Makefile
modified NEW JAIL symlinks, relative paths
/bin
/boot
/etc/aliases
/etc/rmt
/etc/termcap
/etc/unbound
/lib
/libexec
/rescue
/sbin
/sys
/usr/bin
/usr/include
/usr/lib
/usr/lib32
/usr/libdata
/usr/libexec
/usr/ports
/usr/sbin
/usr/share
/usr/src
/usr/tests/lib/libc/tls/libh_tls_dynamic.so
/usr/tests/lib/libthr/dlopen/h_pthread_dlopen.so
/usr/tests/libexec/rtld-elf/libpythagoras.so
/var/db/etcupdate/current/etc/aliases
/var/db/etcupdate/current/etc/rmt
/var/db/etcupdate/current/etc/termcap
/var/db/etcupdate/current/etc/unbound
/var/db/etcupdate/current/sys
/var/db/etcupdate/current/usr/share/man/en.ISO8859-1/man1
/var/db/etcupdate/current/usr/share/man/en.ISO8859-1/man2
/var/db/etcupdate/current/usr/share/man/en.ISO8859-1/man3
/var/db/etcupdate/current/usr/share/man/en.ISO8859-1/man4
/var/db/etcupdate/current/usr/share/man/en.ISO8859-1/man5
/var/db/etcupdate/current/usr/share/man/en.ISO8859-1/man6
/var/db/etcupdate/current/usr/share/man/en.ISO8859-1/man7
/var/db/etcupdate/current/usr/share/man/en.ISO8859-1/man8
/var/db/etcupdate/current/usr/share/man/en.ISO8859-1/man9
/var/db/etcupdate/current/usr/share/man/en.ISO8859-15
/var/db/etcupdate/current/usr/share/man/en.UTF-8/man1
/var/db/etcupdate/current/usr/share/man/en.UTF-8/man2
/var/db/etcupdate/current/usr/share/man/en.UTF-8/man3
/var/db/etcupdate/current/usr/share/man/en.UTF-8/man4
/var/db/etcupdate/current/usr/share/man/en.UTF-8/man5
/var/db/etcupdate/current/usr/share/man/en.UTF-8/man6
/var/db/etcupdate/current/usr/share/man/en.UTF-8/man7
/var/db/etcupdate/current/usr/share/man/en.UTF-8/man8
/var/db/etcupdate/current/usr/share/man/en.UTF-8/man9
/var/db/etcupdate/current/usr/share/nls/POSIX
/var/db/etcupdate/current/usr/share/nls/en_US.US-ASCII
/var/db/etcupdate/current/usr/share/openssl/man/en.ISO8859-1/man1
/var/db/etcupdate/current/usr/share/openssl/man/en.ISO8859-1/man3
/var/db/etcupdate/current/usr/share/openssl/man/en.ISO8859-15
/var/yp/Makefile
...

Insert long pause here. This is the bulk of the copying, the time will vary according to the size of the jail.

It took about 5 minutes and then I was back at the prompt.

Start the new jail

From the example script:

$ sudo iocage start ${JAIL}
* Starting ns2
  + Started OK
  + Using devfs_ruleset: 5
  + Starting services OK
  + Executing poststart OK

Checking all is well

At this point, I run two checks:

  1. ssh – if I can ssh in, that’s a good sign.
  2. nagios – if nagios gives the all green, I’m good to proceed.

Once everything is good, I proceed to the next step.

But what could go wrong?

Perhaps you missed a mount point, or an IP address. Check them all.

Disable the old jail, enable the new jail

This will prevent the old jail from starting and make sure that the new one starts. You don’t want both running.

$ sudo ezjail-admin config -r norun ${JAIL}
$ sudo iocage set boot=on ${JAIL}
Property: boot has been updated to on

Until all jails are completed

The above steps are performed for each jail.

Current status

As I type this, one jail is being copied, and I have seven jails remaining on ezjail:

  1. zuul-pg01 – production PostgreSQL 11.2 database server for this jail server
  2. bsdcan.org – I want to be particular about when I migrate this one: registration for BSDCan 2019 has just opened
  3. pgcon.org – same criteria as bsdcan.org

I want to choose the times for those transitions careful. With registration freshly open, I’d like to do it at a quiet time.

Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive

Leave a Comment

Scroll to Top