Announcing rubybench.org
almost 10 years ago
A year ago I put out a call to action for long running benchmarks.
I got a great response and lots of people lended a hand trying to get a system built. The project itself was quite slow moving, we had some working bits, but nothing really complete to show.
A few months ago Alan Guo Xiang Tan contacted me about the project. In the past few months Alan has been working tirelessly on the project.
Alan rebooted the project and built a Docker backend for running our performance tests and web frontend.
The first UI tracks performance across stable releases of Ruby. The goal is to provide you with a good gauge on what will happen when you upgrade Ruby.
- Will it be faster?
- Will it consume more or less memory?
The second UI tracks Ruby’s progress through the various commits.
- Did a particular commit improve/impact performance?
- How is memory doing?
Rubybench is already helping ruby
In the very short amount of time rubybench was responsible for isolating a regression in the URI library. Our goal is for it to become an tool that the Ruby core team loves using. A tool that automatically finds regression and allows the core team to feel safer when making performance related changes.
Call for help
We still need lots of help with the project:
- Design help with home page and logo.
- Coding help with both the web and the runner.
- Testing, to confirm all our testing methodology is correct and results reproducible.
- Come to our forum and make some suggestions or ask some question.
Call for sponsors
We are in desperate need for a sponsor. We can not run rubybench without a bare metal server. We started a topic to discuss our requirements.
Contact me at sam.saffron at gmail.com, if you would like to help out!
FAQ
What about JRuby / Rubinius and other implementations?
For our initial release we focused on MRI, however if we have enough hardware and help we would love to include other implementations. We would also love to work with various implementers to ensure the bench suite is both accurate and reproducible.
Why all the focus on Docker
A guiding goal for the rubybench project is to have repeatable easy to generate test results. By using Docker we are able to ensure all our tests are repeatable.
Will you run benchmarks on cloud hosting services
We will not publish CPU results gathered on virtual hosts where we can not control our CPU allocation. The only results published on rubybench run on production level bare metal servers.
I am very excited about this project and hope it will help us all have a more awesome and faster Ruby!
A huge thank you goes to Alan for bootstrapping this project and getting it to a publishable state!