This is the first part of many posts in a series of blog posts where I want to document how the MepSQL packages were built. By doing that, I will also end up covering the MariaDB build system (which this is based on), some of BuildBot, Amazon EC2 cloud and packaging DEBs and RPMs just in general, so it could be interesting from many perspectives. In this first part I'll simply scribble some notes about reviewing the OpenSuse Build System, Launchpad PPA service vs using your own servers and automating the builds with BuildBot.
Originally I just wanted to work on some new ideas on the automated build and QA system used by MariaDB. But since leaving Monty Program I didn't have access to any of those servers anymore, so as a first step I had to look into what alternatives there are for building binary packages for many operating systems and hardware platforms. In fact, this was another thing I had wanted to learn more about for a while. For instance Michal Hrušecký uses OpenSuse Build Service to build both MySQL and MariaDB packages for all RPM based distributions in the blink of an eye - I was interested to find out what's behind that magic.
The OpenSuse Build Service is a free service that allows you to submit an RPM source package (and separately a Debian source package to build Debs for Debian and Ubuntu) and it will automatically build packages for recent versions of all the popular (and even not so popular anymore) RPM based distributions. It also hosts the repository which is another important service - a co-located server is actually quite expensive to rent if the project in question is just a hobby of a private person.
While OBS is an OpenSuse creation and service, the code is available as open source, and in fact other projects like MeeGo are using it for their own build service.
The only thing missing with the OBS is one pretty important one: I didn't find a way to also test the packages it produces. So I could use OBS to build packages, but then to see if they actually work I'd still need access to an instance of all versions of Red Hat, Fedora, CentOS, Suse, Mandriva, Debian and Ubuntu - in both 32-bit and 64-bit variations even! Since publishing untested binaries isn't really an option, this omission in OBS is quite severe and for me - a showstopper.
Launchpad provides Personal Package Archives service, where you can submit a DEB source package and get packages built for all recent Ubuntu versions. Also PPA will host the debs so people can download them.
While it's understandable that Canonical focuses on making it easy to build packages for Ubuntu, it's not sufficient as a general solution. PPA also doesn't include a testing service, just like OBS. I had a look at how the guys at Canonical build the official Ubuntu debs, and it seems they run "make test" as part of the build. This however seems pointless to me: yes, it tests that the MySQL code passes the tests, but it doesn't test the very thing they should be testing: whether their own packaging script works!
So the conclusion on reviewing OBS and PPA: Please include a possibility to test whether the packages actually work, that would make your service much more usable!
Update: Andreas Jaeger comments below, that OBS already does a test install of the packages but doesn't then run any tests (that's already something!) and the implementation of actually testing the installation is in a planning and discussion phase (URL below).
So this left me with the option of staying with the BuildBot based system running on my own servers, since it is the only alternative where I can also configure the system to actually test the packages being built. This introduced a new problem: As I didn't have a server available, should I buy one or should I use the trendy cloud services? This is the topic for the next post.
PS: A word about Jenkins is in order: In addition to BuildBot I'm aware of many open source projects using Jenkins (formerly known as Hudson) for the same purpose - Drizzle is one of them. From the point of view I had in this post Jenkins can be considered equivalent to BuildBot: Both include running the system on your own servers. One consideration I had was that "all else being equal" I would choose to use the same system as MariaDB uses, so that any improvements could potentially also benefit MariaDB (and any other MySQL fork as well). Of course, Drizzle is also a MySQL fork so theoretically I could have still gone with Jenkins, but with Drizzle being more different than the other MySQL forks are from each other, the likelihood for more synergy was larger with deriving from the MariaDB system. (Plus, I was already familiar with that system.)