5 years of MySQL
5 years of MySQL
People often write a blog post when they reach some nice anniversary since they joined MySQL community. Well, for those old enough it usually means when they joined MySQL AB as employee. For me this was January 2008. Because I didn't remember the month correctly, I haven't blogged anything then, but decided to save it for a better opportunity - now.
TL;DR Starting this week I will be working for 10gen, selling MongoDB to the Nordics. This blog post is really long - even then it doesn't contain the most interesting stories, I'm not sure if they can ever be published. Sorry for the length, but remember you don't need to read all at once. This is my last MySQL post so save some of it for cold winter days!
2008 - Sun acquisition
I joined MySQL AB at an all company meeting in Orlando, two days before the acquisition by Sun was announced. Wednesday morning meeting we wondered if this was just some joke Mårten was using as introduction. But then Jonathan Schwartz appeared on a live video screen. Clearly it was real.
After a couple of hours listening to Rich Green and other Sun executives, an army of waiters started coming in with 400+ glasses of chilled vodka to commemorate the acquisition. Kaj Arnö sang the introduction to Helan går, the famous Swedish toasting song:
A close friend of Mårten Mickos asked a couple weeks ago what I think about Mårten. Without hesitation I can say he is obviously the best CEO I've ever worked for. But it was more than that. I joined MySQL quite late, but because I belong to this small Swedish speaking minority population in Finland, I was immediately treated like family. Later that day Mårten was drinking a Heineken with me, revealing that "Heineken" was the project code name for the Sun acquisition.
I think Mårten was like that to everyone, even if they didn't speak Swedish. Even last week I was discussing with others at Percona Live how to handle some routine community fuckup, and my advise was to think about what would Mårten do. Of course it's not like he is Jesus, but he was a good leader.
There was a karaoke bar there and I sang "House of the rising Sun". I had difficulty finding the right tune, so Richard Mason helped me out from the audience. He remains at Oracle today, heading all of the MySQL and Linux sales worldwide.
When I came home from Orlando my son was 6 weeks old. He was sleeping as I took him into my arms, opened his eyes a little and smiled at me, then closed his eyes again. It was his first smile.
Back then I was following the birth of open source business models a lot, so I was familiar with Sun as the owner of Java, OpenOffice and even Solaris was open source by then. Later that week I got to meet James Gosling and Ian Murdock, creators of Java and Debian respectively. So it was a company with a lot of sexy open source stuff going on and I was excited about the news. A few months later I listened to an internal conference call about that quarter's financial results and realized that over 90% of the revenues were coming in from selling Sparc servers, and basically no revenue was coming from all the sexy open source stuff. I understood already then that it wasn't going to end well...
My job was to sell MySQL to telecommunications companies. In the beginning it mostly meant equipment manufacturers and MySQL Cluster. This had already gathered momentum the year before, with 2 consecutive MySQL Cluster deals that were the biggest in MySQL history, the second of them was the first 7 figure deal ever. We continued to build it from there and made lots of small and big OEM deals.
It was a dream job for me. I was paid to travel to customers to help them migrate from proprietary Oracle RAC (and sometimes Sybase) to open source MySQL Cluster. At this point everyone was already using Linux (well, Solaris was popular in telco, but at this point it was open source too) and JBoss and Tomcat, and the database tier was "the final frontier" in the software stack yet to go open source. Being owned by Sun (ironically) gave us a lot of credibility with the customers, and the downturn was gaining momentum adding pressure to save money. The biggest savings I'm aware of was a project spending 50 million euro per year on Oracle RAC, migrating to MySQL Cluster and saving more than 90% of that money. In fact they saved more because MySQL Cluster didn't need expensive SAN storage, I think this may have saved them another 50 million.
Beginning 2009 the momentum increased more, with service providers and other "enterprise customers" starting large projects to introduce a dual vendor strategy by migrating to MySQL. We sold them on a three year plan leading to 50% or more of the database tier migrated to MySQL. Only a year earlier MySQL had made the first 7 figure deal, now every senior sales rep was discussing such. In terms of migrations we were talking a total of thousands of servers. Eventually only a few of them would close before Oracle acquisition happened. That of course changed the story of a dual vendor strategy a bit...
2009 - the Oracle acquisition and Monty Program
While our team doubled our sales, Sun overall was going downhill and we were under a hiring freeze. So all of us had to work twice as hard to keep up.
Sun was organizing a large training event in Las Vegas for all technical field engineers. It was mostly on Solaris and Java stuff. So essentially a free trip to Vegas with nothing important to do! I had already booked (and company had paid) both hotel and flights. I have never been to Vegas so I was quite excited.
But then the head of the Nokia account team started pushing me to attend an important customer meeting in Finland that week. Nokia, Ericsson, Orange... these guys always had important meetings with CTOs and whatever, and they always begged MySQL guys to attend because it was basically the only sexy product that Sun had. But eventually we found out (independently from the head of the account team) that there was indeed an important prospect attending the meeting. So I didn't go to Vegas although everything was already paid for. In a way it was nice to stay a week in Finland, as we were so busy those days. I met the prospect and we started to discuss yet another Oracle to MySQL migration.
Then one morning we were all gathered for yet another MySQL user conference in Santa Clara. The biggest ever. I got a gentle wakeup call informing me that Oracle had acquired Sun. Only a few days earlier we had digested the news that IBM wouldn't buy Sun after all. Now this.
My first thought as I got up from bed: I should have gone to Vegas! Indeed, during the same week we had lots of meeting canceled. The proposition of a dual vendor strategy had kind of lost something overnight, and customers had lost interest.
Soon after this I was offered the opportunity to join the MariaDB team as COO for Monty Program. The company was only a few months old. So my tasks ranged from introducing the routine of weekly meetings to initiating proceedings to get back montyprogram.com from a Chinese cybersquatter.
Quite unplanned we got much entangled with the EU Commission proceedings to approve the Oracle Sun merger. There's not enough space to write anything meaningful about that. Suffice to say I've always been impressed by the way Oracle was able to turn a difficult situation into their victory. Having essentially lost on factual arguments, Oracle within 1-2 weeks contacted their 500 biggest European customers (the number is my own estimate) to invite customers to write letters to the EU in support of Oracle. This included VPs based in Redwood Shores, 9 timezones away, waking up during European business hours to attend the second phase of such calls. They got 59 US senators to sign a letter in their support (which must have cost a lot of campaign contributions, im told!) All of this is a remarkable logistical feat in a very short time. Neither Sun nor Nokia would be able to pull it off.
We created the Open Database Alliance as a Swiss association to make MariaDB a foundation governed open source project. Percona had sent over Tom Basil - this must have been a non-trivial investment for what was then a small consulting company. We took a long walk and we have been close friends forever after. Unfortunately Monty Program board in the end decided against moving MariaDB assets to the association. I had worked for 2 months and spent 20kEUR for nothing...
But something good did come out of it for my part. To argue the business case for why a foundation would be good for the project, I collected some statistics that compared the success of foundation led vs vendor owned open source projects. While failing to convince the Monty Program board, the results have later been widely cited. I got to share them with a Rackspace VP - I don't know what impact they eventually had in the birth of the OpenStack Foundation. The 451 Group research published last week certainly vindicated my case a lot.
2010 ...and SkySQL
I had previously managed a paid-by-the-hour business and it was quickly clear to me that the original Monty Program business model, even in the best case, wouldn't be sustainable. In any case the customers looking MySQL migrations didn't need developers, they needed consultants and support engineers and maybe even strategic advice. At this time Percona had not yet launched a subscription business, and they didn't really have any ground troops in Europe.
I was already on paternity leave when SkySQL made its public launch during our joint all company meetings in Istanbul. One of my best memories of all these five years is when we finally stepped into the bus where the founding SkySQL team welcoming us. We had worked hard for over a year to create MariaDB and SkySQL. Almost 50 happy faces and friends rejoined, the atmosphere and energy is something I'll never forget. It gives me similar goose bumps as watching videos of people flooding through the cracks of the Berlin wall in 1989. Cheesy, but it's how I feel.
(Video appears to be from the meeting room, don't know why nobody posted toast from the dinner where the atmosphere was much more intimate and energetic.)
Both the Chairman and the CEO were poker guys. At 5 AM it was me and Ulf remaining at the poker table and we agreed to split 80 for me and 20 for him. So the first person to get any payback from SkySQL was me!
Olivier Beutels had finished 3rd - all he got was a hairy companion in his king size bed, as navigating back to the MP hotel at that hour was not something I was interested in. Ulf was up 9 AM to present the SkySQL strategy and after him I got to do an unplanned stand-in once more as Monty Program COO and present the MariaDB roadmap to the new sales team, as my successor to the COO throne was caught in traffic. That Istanbul traffic can be a bitch - especially if you're hangover and oversleep... Having been 4 months away, I got 2 of the upcoming features mixed up, but Igor gently corrected me.
People asked me last week many times what I think about the Monty Program and SkySQL merger. It is a natural outcome. Monty had committed to a 5 year plan and we are now into year 4 of it. SkySQL had all the time provided most of the revenue to MP, for a long time both boards were majority controlled by the same people and MP executives even owned stock in SkySQL. So it is very much non news to me.
I don't think anyone asked me what I think about MariaDB Foundation but I'll tell you anyway. The mere announcement of a foundation in November - even before it really existed - unplugged a lot of pent up momentum for MariaDB adoption. I'm really happy for the MariaDB developers - it always made me sad to think of all the great work they did on the product while they really underachieved in "go to market" strategy. Finally all of your work is going onto people's servers!
The foundation is of course much a compromise. SkySQL remains owner of the MariaDB brand, while the foundation gets to own mariadb.org. Personally I'm not good at compromises, but a foundation is of course better than no foundation. (As 451 statistics show.) Note that while this is not comparable to a Linux Foundation or OpenStack Foundation, it is not unprecedented. Wordpress and Ubuntu foundations have similar relationships with a single corporate parent.
My prediction for the MariaDB Foundation is that it will be a lot like Ubuntu Foundation. It will of course not become the only, maybe even not the dominating, MySQL variant (ie Ubuntu is not Red Hat) but it can very well nurture a vibrant community of its own. Last week I already met AccelerationDB, a company who is betting their strategy heavily around MariaDB and contributing to the foundation. And probably they will benefit a lot on building on the MariaDB momentum.
I owe Monty a lot of gratitude for everything that I got to do the past 5 years. Working as his COO I got to meet and befriend a lot of "important" people in the MySQL space, from Tom Basil to Mark Callaghan. I even got to meet with Neelie Kroes and took a piss with Thomas Vinje and Edward Screven.
I also respect you a lot for eventually creating the MariaDB Foundation, as you always said you would, and not giving in to pressure to the contrary. Same is true for my stake in Monty Program, I know there too there's been pressure to the contrary but you vigorously defended your promise that all of your employees were entitled to what was said in the hacking business model. For me MariaDB and Monty Program was never about the money, and still today I have no idea what my stake may be worth, but I respect you for defending your own principles.
And I get some twisted satisfaction from the fact that I now finally ended up owning SkySQL stock after all! Overall, I felt it was a nice conclusion also for me personally that the foundation and the SkySQL merger were announced this month, as I will now leave for new challenges.
These years were of course also very stressful to everyone. Me and others got a lot of shit for suggesting that a lot of MySQL people planned to leave Oracle. Anders, Giuseppe and Domas published (well intended) posts that this is FUD and Oracle is great and nobody is planning to leave. A couple months later they themselves had all resigned Oracle. At the MySQL conference another member of MySQL support team yelled "traitor" at Bryan Alsdorf, jokingly perhaps but hurt his feelings nevertheless. Only weeks later this engineer too resigned Oracle and joined the DBA team at a large social networking company.
Personally I have never said that Oracle would kill MySQL. I can say this because I know I have never believed it, as it doesn't make any business sense to do so. The MySQl migrations already stopped the same day the acquisition was announced - Oracle didn't need to kill anything to get what they wanted. I had worked together with Tomas Ulin around MySQL Cluster and I all the time expected MySQL engineering to do well under his rule. Of course, they have done even better than I expected too!
We've overcome quite many obstacles during these 5 years. Probably that's why all the friendships are so strong too, they're galvanized by all the experiences we've had together.
It wasn't just acquisitions and community drama. In 2010 almost all Europeans were stuck in San Francisco after the MySQL conference due to the Eyjafjallajokull volcano causing all of European air traffic to shut down.
Several of us also had pregnant wives at home. My daughter was born 6 weeks after I got back home. She is quite a fiery one too. The "volcano babies club" we call them.
A week later I got home via Mexico City and Madrid. Spain was the first country to get clear skies, and I figured just getting onto the right continent made a huge difference. I had 13 hours to spend in Mexico City. Turns out I don't need a visa for Mexico, so I went to a random underground station to visit a bazaar. You never know where MySQL will take you.
2011 - MepSQL, Drizzle, Nokia and Galera
I still had months of paternity leave left as I resigned Monty Program. I had been intrigued by the state of the art continuous integration and build system Kristian Nielsen had built for MariaDB and started playing with it. As a first step I made it use a buildbot module that would launch AWS EC2 instances on demand. (MariaDB uses on-premise, always on, hardware).
From there I actually got to a state where I could use the same scripts to build both vanilla MySQL, Percona Server and MariaDB and any other MySQL branch using any other name I wanted. A unified build system for all MySQL forks!
This is actually not as easy as you'd think, it turns out especially DEB build scripts are very hard wired to use the given name of a project and it was a lot of work to parameterize this. The first and only release from this project was to package and publish the Facebook branch of MySQL.
As it had become clear that MariaDB and the Open Database Alliance wouldn't become the broad and unifying projects I had hoped them to become, I had committed to find myself a neutral space to continue working on all of the forks in the MySQL ecosystem. I didn't want to upload my work to launchpad as a "mariadb" branch, nor of course as a "mysql" branch either. So to create a neutral, non-vendor zone I named it "mepsql", which is upside down of "websql".
I had created my own fork and I wrote some blogs about the "Cambrian explosion" of forks as explained by Stephen O'Grady. When Drizzle published a GA release I felt that there were too many forks and mine was just tongue-in-cheek, so I silently withdrew from MepSQL to not create unnecessary noise. I wanted to give the Drizzle guys a fair chance after all the work they had done. But MepSQL was eventually mentioned briefly in the 3rd edition of "High Performance MySQL" book. Nokia DBA guys found it there and they were proud to work with a famous person :-)
The MepSQL work was very useful when I interviewed for jobs at Nokia. They asked me questions about how to do this and that in bash and what I knew about EC2. Well bring it on! I did very well in that interview.
Having worked at 2 MySQL vendors already, Nokia provided me with a "neutral territory". They didn't care what I was blogging about MySQL. Over time also the thumbs-down on planet mysql disappeared, as I was no longer associated with one of the warring forks. The content on my blog of course was the same as always. Only three times have I seen people actually voting thumbs down based on the content. Maybe it is time for them to go?
I'm a database consultant so I've never written as much C++ as I did those months. Another thing I learned were the JSON query languages of MongoDB and Couchbase which I used for inspiration to Drizzle's JSON API. Will kind of be useful in my next job...
I was hired to Nokia to solve the problem with MySQL high availability and backups. After evaluating for example Pacemaker for a week, I became convinced that failover-based systems can never be really reliable. (Today I clarify this to say that systems where replication and failover are handled by independent pieces of software are broken by design. For example MySQL Cluster does failover internally, handled by the NDB protocol, and this has never failed anyone. While I have less experience, I believe the same is true for MongoDB Replica Sets.)
This is how I found Galera. I was inspired by reading about the Amazon Dynamo protocol, where node failures have no impact on anything. While that is not possible for a relational database, Galera at least provides similar properties.
I've done a lot of interesting things the past 5 years, but purely technically I believe advocating and helping Galera gain mainstream adoption is the one thing where I have helped MySQL take a technical leap forward. Of course, as the developers are Finland based, it's another team I've become close friends with too.
Percona and the MySQL Conference
In addition to helping to get SkySQL started, it's been great to help other entrepreneurs getting started too. Small things like just sharing my thoughts where they asked for advice, maybe arranging a customer gig or two, or mentioning someone on my blog or arranging a recruitement or partnership.
In this respect I had relatively little to do with Percona. They've grown and hired lots of talented people and I simply had nothing to do with it. Funnily enough more than once I was suspected of being a "Percona guy". I suppose as I did various things I thought were the best for the community as a whole, my motives were often aligned with Percona's business motives.
Indeed, early on Percona had realized that it benefits them to support the promote the entire MySQL ecosystem, including their so-called competitors SkySQL and Oracle. To some this was counter intuitive and people were simply puzzled by it. But actually it makes perfect business sense: As Percona was growing faster than the market on average, it made sense to support also the growth of other vendors in the market. Baron even blogged publicly that he thinks it's great that SkySQL is doing well in Europe, as Percona didn't have much presence there. A few years later Percona has been able to grow both their technical and sales team and now they capitalize on the fact that the MySQL market has grown and is still growing here. Without SkySQL and MariaDB, who knows where those customers would be now? Probably using PostgreSQL or even SQL Server. (Those were the most popular alternatives when people doubted MySQL's future.)
Percona's growth has been very encouraging because it has confirmed 2 principles that I believe in.
The first is the rule we know from other open source ecosystems, that the "more open" alternative usually ends up winning. For example Red Hat over Suse or OpenStack over Eucalyptus. I recently wrote about this phenomenon separately. Since MySQL AB was perhaps the most significant company to challenge this rule, Percona's victory has been very important to open source overall.
More importantly, Percona's winning on being more open has also created a marketplace for others that choose to be more open rather than closed. This was very apparent when I worked together with Codership to sell Galera. In the clustering space we've had half a dozen startups trying to enter the market with a MySQL HA solution: Schooner, GenieDB, ScaleDB, Clustrix, Xeround... Each of them has had tens of millions in VC funding. One of them even OEMed Galera, so it was technically the same solution, with just one difference, they tried the closed source approach. Instead this market was taken over by 4 penniless guys from Finland. They were open source and the partnership with Percona meant a lot - Vadim's first blog was like a blessing from the Pope! I'd like to think that Tokutek's open sourcing their product will end up like a similar success story. Vadim has already published a first alpha build with Tokutek and Percona Server.
The other principle was something my parents thaught me to believe in: that competence (or "education", as that was often the topic) is a form of capital. When I look at Percona's success and compare with some other vendors competing in the same space, it is safe to say that if I could choose to have competence or 10 million in VC funding, I'd rather choose competence. Well done Peter, Vadim and Baron! (That said, good luck Baron in also finding that funding for your next venture!)
In the end I grew quite close with Peter, Vadim and Baron due to working on keeping the MySQL conference alive. This was an issue already in 2009, even before the acquisition had been cleared by regulators. "Whistleblowers" inside Oracle contacted myself and others at Monty Program and told us that Oracle had no intention of continuing with the conference in 2010. We passed this information on to O'Reilly who decided to go forward alone instead. After dragging their feet both years, Oracle did end up participating both in 2010 and 2011.
With Oracle not promoting the conference to their MySQL users, and Percona and SkySQL still rather small players, attendance dwindled and O'Reilly had to face the fact it wasn't a justifiable business anymore. In 2009 Percona had been shut out by Sun/MySQL from the conference which prompted them to start organizing their own Percona Live conferences instead. In 2012 they had grown confident enough that they took over the big conference itself from O'Reilly. This is a good example of how obstacles may turn out to be blessings in the end!
Percona has been the best host the conference could get. They've gone out of their way to make it an inclusive conference that is open to all. Especially in 2012 many people emailed them to demand that Oracle should not be invited to the conference because they were evil, had sued Google, were killing MySQL, or some other reasons. Baron responded to all of those demands (by private emails which I've seen) that everyone will be welcome at their conference and they will never censor anyone because they know first hand how that feels. In 2013 Oracle responded to this invitation and their talks were very well received. Both years the conference has also been open to other Percona competitors or just people with a personal grudge, who have at times tried to undermine or boycott the conference for who know what reasons. Percona even have promoted the free SkySQL Day that happens adjacent to the main conference. Again, this makes business rational sense, but not every business executive always understands such things. It's been truly remarkable to witness their commitment to 100% openness and inclusiveness in this area.
On this note I think it should be said that this year a line was crossed. While I admire the ability of Percona to forgive and continuously invite back those attacking them, I think a line needs to be drawn when attendees (and non-attendees, even) have attacked other attendees. We must not allow open source conferences to become places where attendees need to fear losing their jobs (or worse) due to reckless tweeting or blogging from people they've never even met before. The good thing is that since Percona has already proven their commitment to inclusiveness and tolerance during the past 2 years, they are now in a strong position to enforce an appropriate amount of discipline to ensure that the conference remains such an inclusive and positive place as they've built it to become.
A tradition somewhat linked to the conference is the annual MySQL awards. We took over this tradition too when Oracle wasn't supporting the 2010 conference. Colin found a place that sells nice goblets and he has been importing them from Malaysia every year. I arranged within 36 hours for a community panel to choose some winners. Shlomi was a panelist (he had won the previous year) and stepped up to build a wordpress based voting system that we still use today. They will be continuing this tradition as I step aside.
I think the awards have really gained in prestige when they are decided by an independent community panel. Winners are always so proud to get them, it feels good to be able to make someone so happy. Even those that are repeat winners make a big deal out of it, because it's different to win this award - they say.
I've always asserted that the panel is free to award whoever they want for whatever reason they want. There's one exception though, I've managed to talk myself out of myself being voted to win, with the explanation it would feel weird to stand alone on stage and hand myself an award. But here's a little secret: I haven't gone empty handed out of all this! Both in 2010 and 2012 we had some logistical challenges and ended up with one goblet too much. I of course kept them for myself. So I've been plenty awarded after all:
The next step
The theme for the past 5 years has been one of transformation. In 2009 Baron made the famous keynote "MySQL has a community, but PostgreSQL is a community".
From my point of view this was all the time the root cause to all the problems. The problem was not that MySQL was acquired by Oracle, the problem was that MySQL was vulnerable to such an acquisition in the first place. (Also problems with engineering process and performance issues back then can be attributed to same lack of diversity.) If anything, we should thank Oracle for acting as a catalyst in this transformation. I think we can now safely say that MySQL is a community!
The uncertainty around the future of the MySQL conference was really the last thing that needed to be settled. Last year around this time I came home from the first Percona Live Worldwide MySQL Conference feeling that even that issue was now settled and the conference was in safe hands with energy returning.
As Nokia was still going through the layoffs, although my team was unaffected, it was natural to look around for opportunities. I decided that after so many years focusing on what was good for the MySQL community, now would be a good time to finally look for what was good for myself.
I of course talked to various startups in the MySQL space and got some really good offers. Still, one unsolicited and unexpected offer was better than anyone else. It was a bit surprising but I decided that ok, then so be it. I've accepted the offer from 10gen and will be back to doing pre-sales consulting and selling... only this time it is MongoDB.
Last week's conference was a perfect culmination of everything I've been part of during these 5 years. I got to meet all the friends one more time, enjoy the Sun of Santa Clara and we enjoyed maybe the best conference ever. Put in historical perspective the atmosphere was great - I think everyone else is tired of the history too and just wanted to leave those things behind.
SkySQL again arranged their own "add on" day on Friday and I went there for the evening reception. With Kaj and Patrick (Backman) and Ulf no longer present, they realized they didn't have anyone there who could sing the introduction to their by now famous toasting song. (En liten fågel satt en gång...) So Max Mether invited me to lead the singing. This is something I've never done before in any context - usually this introduction is sung by something like a choir's lead tenor, or otherwise most senior attendee of the party. It was a small thing, but for me it felt a big honor and quite sentimental.
And so my 5+ years with MySQL ended as they had begun, with singing Helan går.
(Watch this space, there should be a video of me singing coming online soon.)
It was planned I would speak at the SkySQL day, but I had yielded my spot to Seppo who spoke about MariaDB Galera Cluster. They had still prepared a thank you gift with my name on it, so I got to take home the beer glass to the right in the below picture.
In my first day at MySQL AB new employee training in Orlando, I "won" the competition of who knows the most languages (4). Actually I tied with Pasi Vuorela (now at SkySQL) but he yielded the beer glass to me (left). I was well prepared for a tie break round of who knows the most programming languages...
There was a time when I stopped using this glass and some other Sun branded items I've got. To me the Oracle acquisition wasn't just a change in employer, it was as if MySQL and Sun had failed the whole open source movement. Now enterprise users would turn to Microsoft SQL Server for their dual vendor strategy. (Also EnterpriseDB at that time was nonexistant in Europe.) Watching this glass made me feel sick. But that scenario didn't happen, people are using MySQL and other open source databases and I'm again using my beer glass.
Two glasses, one half empty, one half full. Cheers!