We are pleased to announce that our paper Automated System Performance Testing at MongoDB will be presented at DBTest 2020. (A workshop in conjunction with SIGMOD/PODS.) It is available today on Arxiv.org.

This paper presents the framework we developed to do automated performance testing on realistic MongoDB clusters. We have used and evolved this system to run hundreds of benchmarks every day as part of our Continuous Integration system. In conjunction with publishing this paper, we have also finally open sourced the Python code to this framework. The framework is called Distributed Systems Infrastructure 2.0, or DSI for short.

A trilogy of publications

One more reason I'm really glad about this paper is that it completes a trilogy of articles that describe the work I have done during 4 years in MongoDB R&D. The previous two are

• Reducing variability in performance tests on EC2
  • This one I have also presented at conferences.
• Change Point Detection in Software Performance Testing
  • David will be presenting this paper at ICPE 2020 on Thursday!

The change point detection paper is perhaps the one we are most proud of and has already attracted lots of interest from peers in the industry. Trying to automate the analysis of performance regressions is a hard problem, and while we struggle with it too, it's been energizing to see that we might actually be a step or two ahead of many others.

Back to the paper on Automated System Performance Testing - with DSI - it is a nice ending to the trilogy because it binds together all the other work we did. The performance tuning lessons learned in the "reducing variability" blog post are encapsulated in the setup phase of DSI, and the change point detection helps us process and benefit from the results produced by DSI.