How to chroot

My colleague setup a jenkins to do nightly and on demand build, it works perfect till last week, whenever we decided to move some components to rpm based packaging so to ease the production deployment and to reproduce the bug.

The problem is, whenever you have multiple jobs in jenkins, you cannot yum/rpm install packages in different versions. A simple case for us is that we are running boost-1.44, while some team who replies on us requested a new version of boost, we cannot build two set of jobs at same time.

So I came up with a idea of setting something that can islolate jenkins jobs. Actually I was thinking of OpenVZ at very beginning, but later on I figured out it may be too complicate to us at least at this stage. I’m still keeping an eye on OpenVZ, just in case we meet some chroot-cannot-solve problem I may give it a try.

Back to original topic – I need to have a way to setup an isolated environment for jenkins, it should be easy to create/destroy, while use all standard (CentOS) tools to manage packages. I guess what I did works in Debin/Ubuntu as well – you just need to replace all yum with apt, and rpm with dpkg 😛

First let’s create a snapshot of CentOS image first, you don’t have to create the image if you don’t care of the speed and network traffic too much, what’s being done here can put into jenkins jobs but in my case the speed is about 10 times faster by using a snapshot, and definitely it reduce the network data flow (my image has minimum set of development packages, even it installed limited packages, it still need to pull ~300M data from CentOS repo, but sure if you have setup your own repo, that’s another story).

You should have some common sense of replacing versions/locations below to meet your need, and of course all commands need to be run as root (sudo please):

1. create a directory to be the chroot’s root, let’s assume it is in $ROOT_DIR

   mkdir -p $ROOT_DIR

2. bookstrap yum, you can choose other versions as you like (check 5on6 sections below)

   mkdir -vp $ROOT_DIR/var/lib/rpm
   rpm -v --rebuilddb --root=$ROOT_DIR
   rpm -Uhv --root=$ROOT_DIR --nodeps http://mirror.centos.org/centos/6/os/x86_64/Packages/centos-release-6-3.el6.centos.9.x86_64.rpm
   yum --installroot=$ROOT_DIR install -y yum

3. do other necessary configurations – this is to avoid warning/errors during yum in step #4

   cp -v /var/lib/random-seed $ROOT_DIR/var/lib/random-seed
   touch $ROOT_DIR/etc/fstab
   cp -v /etc/resolv.conf $ROOT_DIR/etc/resolv.conf

4. this is the image for jenkins, so most likely you need all dev tools

   yum --installroot=$ROOT_DIR groupinstall -y "Development tools"

5. Most likely you need to copy your repo file to $ROOT_DIR/etc/yum.repo.d/
6. Some file systems need to be mounted (check comment #1 to see a better solution for this)

   mount -o bind /proc $ROOT_DIR/proc
   mount -o bind /dev $ROOT_DIR/dev

7. Now you can step into the root and do some other stuffs you want, to me the most important thing is clean up yum cache so that everytime jenkins will pull the most recent package (for our own repo only) instead of using cache

   chroot $ROOT_DIR
   [in chroot now] yum clean all
   [in chroot now] exit

8. unmount what we mounted before

   umount $ROOT_DIR/proc
   umount $ROOT_DIR/dev

9. take an snapshot

   cd $ROOT_DIR
   tar zcf /tmp/centos6-20121212-01.tgz *

10. now we have a snapshot under /tmp, named centos6-20121212-01.tgz
    hey, it’s 12-12-12 now, nice

Once you have the snapshot, it becomes pretty easy to use it in jenkins, you have to solve sudo problem first, which allows jenkins user (either jenkins, or apache, or nobody, or tomcat) sudo to run the build.sh without password (ping me if you don’t know how to), and here are some codes copied from my build.sh:

mkdir -vp $ROOT_DIR
tar zxf /tmp/centos6-20121212-01.tgz -C $ROOT_DIR
mount -vo bind /proc $ROOT_DIR/proc
mount -vo bind /dev $ROOT_DIR/dev
mount -vo bind /dev/pts $ROOT_DIR/dev/pts
mount -vo bind /sys $ROOT_DIR/sys
chroot $ROOT_DIR yum install -y home-grown-packages
cat > $ROOT_DIR/tmp/build.sh << EOF
/bin/mkdir -p /tmp/build
cd /tmp/build
mkdir -vp BUILD BUILDROOT RPMS SOURCES SPECS SRPMS
/usr/bin/svn co svn+ssh://blah-blah-blar/foo/bar
cd /tmp/build/BUILD
rpmbuild --define '_topdir /tmp/build' -bb path/to/jenkins.spec
EOF
chmod +x $ROOT_DIR/tmp/build.sh
chroot --userspec=nobody $ROOT_DIR /tmp/build.sh
# some more operations here, copy the final rpm to central repository, copy the test result back to jenkins workspace, etc
umount -v $ROOT_DIR/sys
umount -v $ROOT_DIR/dev/pts
umount -v $ROOT_DIR/dev
umount -v $ROOT_DIR/proc
rm -fr $ROOT_DIR

Pretty much the commands explained what they do …

There are something more to come soon, people are asking if build CentOS5 packages over CentOS6 is possible or not – theortically it should work, but I haven’t tested and I’m pretty sure there will be some troubles here and there, I will give a shoot tomorrow.

Will post more here in case I find anything interesting.