BuildBot

The current status of our buildbot can be seen in its ​web interface.

Setting up a new build slave

Buildbot consists of a central "build master", and a series of build slaves. The master farms out builds to the slaves, at frequencies, and with configuration options, specified in the central configuration. Build slaves are relatively lightweight, but must have all the software installed on them which is necessary to perform a build of Xapian.

In addition, at present, the buildbot client software must run on the slave. We are working on making buildbot able to run all commands over ssh to avoid the need to run a buildbot server on the slave, but until this is done, build slaves will require a buildbot slave server to be running.

In buildbot 0.8.x, the buildslave code has been split off into a separate package, so what you need to install and how you set up the slave depends on the version of buildbot you have.

Setting up a Linux slave

Install commands and package names are for Debian wheezy, but Ubuntu should be very similar, and this should be at least some help for other distros.

apt-get install buildbot-slave
apt-get install build-essential ccache valgrind
apt-get install zlib1g-dev uuid-dev
# xapian-core docs:
apt-get install texlive-latex-base texlive-extra-utils texlive-base-bin help2man python-docutils pngcrush
apt-get install texlive-base-bin texlive-fonts-extra texlive-fonts-recommended texlive-latex-extra
# For omega:
apt-get install libpcre3-dev libmagic-dev
# For the bindings:
apt-get install python-dev php5-dev php5-cli lua5.2 liblua5.2-dev ruby-dev tcl-dev openjdk-6-jdk mono-devel
# FIXME: other stuff needed too...
# for building from git:
apt-get install git m4 flex bison graphviz

Creating the slave (buildbot 0.8+)

To create a slave working in directory buildbot_slave:

buildslave create-slave buildbot_slave buildbot.xapian.org:9989 <name> <password>
echo 'Ann Example <ann@example.org>' > buildbot_slave/info/admin
echo 'Debian 7.0 (wheezy)' > buildbot_slave/info/host
buildslave start buildbot_slave

Creating the slave (buildbot 0.7.x)

To create a slave working in directory buildbot_slave:

buildbot create-slave buildbot_slave buildbot.xapian.org:9989 <name> <password>
echo 'Ann Example <ann@example.org>' > buildbot_slave/info/admin
echo 'Debian 6.0 (squeeze)' > buildbot_slave/info/host
buildbot start buildbot_slave

Setting up a windows Cygwin build slave

To set up a new windows Cygwin build slave, take the following steps (assuming a fresh installation of windows) (Note that we have experienced problems on machines which have old copies of cygwin1.dll lying around - check c:\WINDOWS for this):

• Download and run the Cygwin setup.exe (from ​http://www.cygwin.com/setup.exe). Install a minimal set of packages. If you understand cygwin, check that the directory /etc/alternatives exists after the installation - I've observed some problems with /etc/alternatives not getting created correctly. If it doesn't exist, try using a different mirror to get a newer version of the alternatives package.
• Run setup.exe a second time. This time select and install the following essential packages (these will pull in other packages as dependencies; it's fine to install other stuff too if you want it):
  • autoconf2.5 (or later)
  • automake1.10 (or later)
  • bash
  • bison
  • bzip2
  • ccache
  • coreutils
  • doxygen
  • gcc-g++
  • gcc-mingw-g++
  • git
  • help2man
  • libtool
  • make
  • pcre-devel
  • perl
  • python
  • tar
  • zlib
• Enable the compiler cache (ccache). This will use more disk space, but speed up rebuilds; you can skip it if this doesn't sound helpful. There are several ways to enable it, but the following worked well for me:
  • create a file named /usr/local/bin/gcc containing

    #!/bin/sh
    ccache /usr/bin/gcc "$@"
    

  • create a second file named /usr/local/bin/g++, with similar contents, but with gcc replaced by g++.
• Install easy_install. This can be done by going to ​http://pypi.python.org/pypi/setuptools and downloading the latest setuptools egg (currently ​http://pypi.python.org/packages/2.5/s/setuptools/setuptools-0.6c8-py2.5.egg). Then, open a cygwin command prompt, and run the egg using sh. For example, if the egg was saved in "C:\", you would run the command "sh /cygdrive/c/setuptools-0.6c8-py2.5.egg".
• Install twisted and buildbot: from the cygwin shell, simply run "easy_install twisted" and then "easy_install buildbot".
• Get a password and name for your slave from the xapian buildbot maintainers (contact us on IRC or via the xapian-devel mailing list).
• Create your build slave, using the following command (in a newly opened cygwin shell), where "<name>" is the name for your build slave, and "<password>" is the password:
  • "buildbot create-slave buildbot_slave buildbot.xapian.org:9989 <name> <password>"
• Edit the information files for the build slave:
  • from the cygwin prompt run "notepad.exe buildbot_slave/info/admin", and adjust the file to contain your contact details.
  • from the cygwin prompt run "notepad.exe buildbot_slave/info/host", and adjust the file to contain a brief description of the host (eg, rough specifications, what OS it is running, whether it is permanently connected).
• Start your build slave:
  • from the cygwin prompt run "buildbot start buildbot_slave". You should see, after a few other lines of text pop up, "The buildslave appears to have (re)started correctly."

Ideally, you will need to ensure that the buildbot restarts automatically when the build slave machine is restarted. To do this, add a new String value to the Windows Registry at HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run - name it 'Buildbot' and give it a path to a batch file that will start a Cygwin shell and run "buildbot start buildbot_slave". Such a batch file follows:

@echo off
C:
chdir C:\cygwin\bin
bash --login -c "/usr/bin/buildbot start /home/Administrator/buildbot_slave"

Setting up a Mac OS slave

(This should probably all be done in a virtualenv, but I last set one up from scratch before that was terribly practical.)

You want the buildbot python package, which may or may not install cleanly. Then you want to add the following into launchd and set it up to start on boot; more details when someone does this again from scratch:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
        <key>StandardOutPath</key>
        <string>/Users/buildbot/slave/twistd.log</string>
        <key>StandardErrorPath</key>
        <string>/Users/buildbot/slave/twistd-err.log</string>
        <key>EnvironmentVariables</key>
        <dict>
                <key>PATH</key>
                <string>/opt/local/bin:/sbin:/usr/sbin:/bin:/usr/bin:/Users/buildbot/local/bin</string>
        </dict>
        <key>GroupName</key>
        <string>daemon</string>
        <key>KeepAlive</key>
        <dict>
                <key>SuccessfulExit</key>
                <false/>
        </dict>
        <key>Label</key>
        <string>org.xapian.buildbot.broteas</string>
        <key>ProgramArguments</key>
        <array>
                <string>/usr/bin/twistd</string>
                <string>-no</string>
                <string>-l</string>
                <string>-</string>
                <string>--pidfile=/var/run/org.xapian.buildbot.broteas.pid</string>
                <string>-y</string>
                <string>./buildbot.tac</string>
        </array>
        <key>RunAtLoad</key>
        <true/>
        <key>UserName</key>
        <string>buildbot</string>
        <key>WorkingDirectory</key>
        <string>/Users/buildbot/slave</string>
</dict>
</plist>