Dealing with the Cambrian Explosion 2/2: Parameterizing the package name in DEB files

Yesterday I wrote down the approach used in the MepSQL build system to parameterize the TAR package name produced. Today I will follow up with how the same was done for building DEBs. The motivation is to create a system that can be used flexibly to create packages of any MySQL fork, with any brand name: mepsql-server-*.deb, percona-server-server-*.deb, mariadb-server-*.deb or even just mysql-server-*.deb (which I might do some day).

While yesterday's tricks with the TAR files were rather straightforward, with the process of building DEBs this turns out to be much more challenging. But not to worry, like my former collague Bernhard Ocklin used to say: This is software, anything is of course possible.

I used the debian source archive from Ubuntu as a starting point, as official MySQL doesn't build DEBs at all. When you correctly unpack the weird format this .diff.gz file is in, you end up with a debian/ directory which is supposed to reside at the root of your MySQL source tree. (Note: Not the bzr source tree, but the one unpacked from a released source tar file, the one we built yesterday.) This directory has quite a lot of content:

$ ls -F|cat

For each DEB, there are files named *.dirs and *.files which are lists of what files from the MySQL build should be put into that package, as well as pre- and post-installation scripts. You also see other Debian/Ubuntu specific files to be added, like an upstart script. There are additions and patches to be applied to the upstream source TAR.

The control file lists the meta-data for each package to build, descriptions and dependencies. And there is the rules file, which is like a makefile. It is what makes everything happen.

In all of these files, the string "mysql" or "MySQL" is found 3855 times:

$ grep -i mysql * */*|wc -l

Most refer to file names and directories which needn't be changed. Almost half are in the translations:

$ grep -i mysql po/*|wc -l

While it was not done in MepSQL, most of the strings found in the po translations could potentially be "rebranded" to your fork name.

Out of these, the string "mysql" was replaced in 28 places in various files.

$ grep XmysqlforkX * */*|wc -l

And 37 filenames:

$ ls *XmysqlforkX*|wc -l

Rather than changing the string "mysql" to "mepsql", a generic placeholder "XmysqlforkX" was inserted in those places.

Finally, the control file contains so many references to package names, 50 to be precise:

$ grep mepsql control.mepsql |wc -l

...and also considering the fact that when creating a fork one might actually want to edit or rewrite the description texts which are in this file, another approach was needed. Rather than writing XmysqlforkX everywhere in the file, it is possible to just have the complete file customized for your own fork, and save it as control.mepsql (where "mepsql" matches the value of $MYSQLFORK).

In the MepSQL Bakery scripts these template files are stored in a directory debian-5.1/. (The 5.1 is the MySQL version number to build.) Before the build starts, they are copied to the actual debian/ directory, and in the process all occurences of XmysqlforkX are replaced with the value of $MYSQLFORK. The sed and other shell magic needed was eventually quite straightforward, once I had figured out what needs to be changed where:

# New approach: filenames and file contents in the template debian-5.1 dir may
# contain the string "XmysqlforkX". This should be replaced with value of $MYSQLFORK.
mkdir debian
for f in $(find "${BAKERY_BASE_DIR}/debian-${MYVER}" -type f); do
   # Manipulate filename
   newf=$(basename $f | sed "s/XmysqlforkX/$MYSQLFORK/g")
   # Manipulate file contents, save with new filename
   sed "s/XmysqlforkX/$MYSQLFORK/g" $f > debian/$newf
# Other directories can be copied as is, there's nothing to manipulate
cp -rf ${BAKERY_BASE_DIR}/debian-${MYVER}/additions ${BAKERY_BASE_DIR}/debian-${MYVER}/patches ${BAKERY_BASE_DIR}/debian-${MYVER}/po debian/
# Except sed is also needed for po/
sed "s/XmysqlforkX/$MYSQLFORK/g" ${BAKERY_BASE_DIR}/debian-${MYVER}/po/ > debian/po/
# Finally, it is possible to specify your own files that are not manipulated with
# sed, rather used as is. For instance, you can have control.mepsql, control.mariadb
# etc... and the following will use the file whose suffix matches $MYSQLFORK
for f in $(find "debian/" -name "*.$MYSQLFORK"); do
   # $f includes debian/, but basename strips it away for the second argument.
   mv -f $f debian/$(basename "$f" ".$MYSQLFORK")

Going forward...

I have not yet built any RPM files in MepSQL. I expect that the $MYSQLFORK trick will be easier to do in the .spec file than it was in the rather complex debian build system. It remains to be seen.

Working on a build system for MySQL forks, let alone blogging about it, isn't perhaps the sexiest thing there is. But in the work presented in this pair of blog posts I've tried to introduce some new thinking into how packages are built, with the motivation to enable easy code sharing and co-maintenance across the different forks we are working on. I hope that this could encourage others to think of similar ideas in other parts of the code base, and I will of course continue to do so myself too.

Next week I will return with one more post that gives a high level overview of all the layers, from buildbot automation to makefiles, that you come across when building your own MySQL source tree with the MepSQL Bakery.

Add new comment

The content of this field is kept private and will not be shown publicly.
  • No HTML tags allowed.
  • External and mailto links in content links have an icon.
  • Lines and paragraphs break automatically.
  • Web page addresses and email addresses turn into links automatically.
  • Use [fn]...[/fn] (or <fn>...</fn>) to insert automatically numbered footnotes.
  • Each email address will be obfuscated in a human readable fashion or, if JavaScript is enabled, replaced with a spam resistent clickable link. Email addresses will get the default web form unless specified. If replacement text (a persons name) is required a webform is also required. Separate each part with the "|" pipe symbol. Replace spaces in names with "_".
About the bookAbout this siteAcademicAmazonBeginnersBooksBuildBotBusiness modelsbzrCassandraCloudcloud computingclsCommunitycommunityleadershipsummitConsistencycoodiaryCopyrightCreative CommonscssDatabasesdataminingDatastaxDevOpsDrizzleDrupalEconomyelectronEthicsEurovisionFacebookFrosconFunnyGaleraGISgithubGnomeGovernanceHandlerSocketHigh AvailabilityimpressionistimpressjsInkscapeInternetJavaScriptjsonKDEKubuntuLicensingLinuxMaidanMaker cultureMariaDBmarkdownMEAN stackMepSQLMicrosoftMobileMongoDBMontyProgramMusicMySQLMySQL ClusterNerdsNodeNoSQLodbaOpen ContentOpen SourceOpenSQLCampOracleOSConPAMPPatentsPerconaperformancePersonalPhilosophyPHPPiratesPlanetDrupalPoliticsPostgreSQLPresalespresentationsPress releasesProgrammingRed HatReplicationSeveralninesSillySkySQLSolonSunSybaseSymbiansysbenchtalksTechnicalTechnologyThe making ofTungstenTwitterUbuntuvolcanoWeb2.0WikipediaWork from HomexmlYouTube