I made this tweet that got reasonably popular:

We benchmarked how long it takes to run the Ruby test suite for Discourse across our various dev machines. I can not believe what a crazy tax I have paid over the years insisting on sticking with Windows, highlighted results mine.

https://twitter.com/samsaffron/status/1111511735851081728

This evoked a lot of extremely strong emotions from various people out there. Ranging from “Sam is a fool what kind of insane benchmark is this”, “the real story is MacOS has bad Ruby perf” to a general “Oh no”.

The core point I was trying to make was that I was paying a pretty high tax for deciding to “stick with with Windows”. There are a bunch of other points hiding here that are also worth discussing.

Why are you sticking with Windows to run Linux in a VM?

https://twitter.com/shanselman/status/1111538525017530371

What I did not know is the extent of the VM tax I was paying regularly. I never dual booted my computer so I had no proper anchoring point of reference.

I very strongly believe that many Ruby/Rust/Go/Elixir/Scala and even some Node developers who end up doing the WSL dance or run Linux in a VM for development, or use Linux Docker for dev on Windows are not aware of the full extent of the tax.

On my machine the price of admission for using WSL was 25% slowdown in my day to day running of specs. And a 38% slowdown for using a VMware based VM.

I am not alone here… other team members have experienced similar slowdowns. Other people out there also experience similar slowdowns.

https://twitter.com/stevedesmond_ca/status/1111585969650712578

What I thought was inside my wonderful wish hat was that the performance hit was minor:

Yes. But that is not the question. The difference is normally negligible (1% to 5%).

If you Google, well that is the general answer you get. VMs are more or less the same as no VM, 5-10% perf tax. My reality was different. Maybe I was missing magic bios turbo settings, maybe I needed to direct mount a volume instead of using a vmdk on my dedicated NVMe Samsung 960 pro, maybe there is some magic I could do to get to this magic 1-5% number. Maybe Hyper-V is better I am not sure. All I know is that I am not alone here.

WSL is not an option for me cause Ruby likes lots of small files and lots of stats calls, WSL has terrible lots of small file performance as documented by the WSL team. How terrible you ask? As a baseline just running a rails c console without bootsnap was taking me upwards of 30 seconds. Same operations takes 4-5 seconds on Linux without bootsnap. Even with all the workarounds we could place this bad IO perf was something that I just noticed too much. In fact I preferred the 38% slowdown cause at least stuff was consistent and not wildly off balance like WSL is. Being able to launch a console or web server quickly is critical during dev. Fuse does not appear to be happening any time soon so you can not work around this using ninja tricks of block mounting a device.

So, I stuck with a VM cause it was nice not to have to constantly reboot my computer and thought the price I was paying was not that high.

I like the Windows 10 font rendering, I like the HiDPI support, I like using Lightroom on Windows and playing Rocksmith on Windows. I like the out-of-the-box experience and minimal amount of tweaking needed. I like being able to launch Skype without it segfaulting cause I was LD_PRELOADing jemalloc. I feel Windows 10 as a window manager is on par (for my usage) to my Macbook Pro running MacOS.

Dual booting is a compromise for me, some stuff I have works best in Windows. I thought the compromise I was making performance wise was worth the comfort of living in a “known OS” that I like.

I felt that if I start booting to Linux I am going to have to fight with drivers, have stability issues, not have a complete toolset and so on.

I felt comfortable at home and moving is one of the most stressful life events.

Is 2019 the year of Linux on the Desktop?

https://twitter.com/joeneville_/status/1111891763151728640

The joke goes like this. Every year a bunch of people joke about how LOL this will be the year of Linux on the Desktop. It happens every year. It starts cause someone says “hi Linux is quite good these days, could this be the year of Linux on the Desktop?”. And then a bunch of happy and well meaning trolls, say ha ha … as always you are wrong… this is not the year of Linux on the Desktop.

And so it goes…

This banter is usually well meaning and grounded in reality. However it has a very big side effect, which impacts developers in a significant manner. Developers who do not use Linux on the desktop are scared of Linux. They are scared even if their production code only deploys on Linux (and not MacOS or Windows)

I felt super scared to go down the path of Linux cause I was terrified … about drivers … font rendering… HiDPI support… multi monitor support and the list goes on.

In fact, I was not wrong to be scared. It is fiddly to get Linux going. I almost gave up after my first 4-8 hours cause Firefox on Linux is still stuck on a very sad default there is no hardware acceleration out of the box, so scrolling is mega jerky. This very simply rectifiable behavior was a deal breaker for me. If I could not get scrolling a web page to be smooth, I am out of here, not going to use Linux. Luckily the issue was resolved after tweaking 1 value in about:config.

NVIDIA does not have a great story as well, the future of Desktop on Linux is Wayland. The windows manager I wanted to try, sway, only works properly if you use the open source community provided nouveau driver. Even getting NVIDIA to work nicely involves enabling hardware compositing and fiddling with X11 config.

My position is not that Linux is poised to take over the world in a storm this year. It is a far more humble position. If you want to get the best bang for buck and want to get the best possible performance developing Discourse, or any Ruby on Rails application Linux on the Desktop/Laptop with no VM is your best bet.

It is also important to note that I opted for medium hard mode when I moved to Linux. I am only 2 neck beards away from installing Linux from scratch.

https://www.reddit.com/r/ProgrammerHumor/comments/3mun55/evolution_of_linux_neckbeardsxpost/
source

My colleagues who went through similar exercises of shifting from Windows/Mac to Linux stuck with Ubuntu and Linux Mint, they tell me they had a very smooth ride.

Have you tried running Ruby on Windows?

Avdi triggered quite a discussion about this a few days ago:

https://twitter.com/avdi/status/1109834294665592834

The point he is trying to make is that a Ruby that works well on native Windows will help Ruby adoption a lot and eliminate drain to other frameworks. Installing a whole brand new OS is just too much of a barrier. Just install Linux is not a solution.

The reality is that running MRI Ruby native on Windows hits 2 big fundamental problems:

  1. Filesystem performance characteristics of NTFS on Windows are badly suited to the current Ruby design. We love lots of small files, we love lots of stats calls.

  2. It is a gigantic effort porting various dependencies to Windows native (and maintaining them), as it stands many of the Discourse dependencies simply do not work on Windows. The gems simply will not install. The fundamental issue is that if you are writing a c extension in a gem it is extra work to get it to work on Windows. Getting stuff to work on MacOS and Linux is no extra work vast majority of the time.

(2) is a tractable problem, but I wonder if it is worth any kind of effort given WSL has far wider compatibility and should offer reasonable performance once a workaround exists for the filesystem problem (which is fundamental and not going to change on Windows native). Discourse works fine on WSL (provided you skip using unicorn) Discourse does not work at all on Ruby on Windows native. The Apple tax is similar in cost to the Windows WSL tax (except for filesystem perf). I feel that once WSL gets a bit more polish and fixes it will be competitive with the current Mac experience.

The Apple performance tax

One pretty obvious thing from the chart I provided was showing there is a pretty severe Apple performance tax as well.

When looking at user benchmarks per: UserBenchmark: Intel Core i7-8559U vs i7-8750H. We expect an 8559U to have faster single core performance (thermal locking withstanding) than the 8750H. Yet this Linux 8750H laptop is clocking a spectacular 9m13s compared to the Macbook Pro 15m16s. We are seeing poor MacOS performance across the board. And we are not alone:

https://twitter.com/dominiksander/status/1111928539580956672

https://twitter.com/zalesz/status/1111726682115899393

It appears that people insisting on the native MacOS experience are paying a significant tax for developing Ruby on Rails on a Mac.

I know that DHH loves his iMac Pro and recommends it enormously.

https://twitter.com/dhh/status/940658254501302274

Yes, the hardware is real nice, the screen is beautiful, the machine is wonderfully put together. The Window manager is nice. Zero driver problems. However, sadly, there is a significant OS tax being paid sticking on MacOS for Ruby on Rails development.

I think the Ruby community should explore this problem, document the extent of this problem and see if anything can be done to bring Darwin closer to the numbers the same machine does with Linux. Is this problem rooted in the filesystem? The OS? The llvm compile of Ruby? Security features in MacOS? Something about how Spectre+Meltdown (which is already patched in my Linux)? It is very unclear.

As it stands I would not be surprised at all if you dual booted a Mac with Windows, installed WSL and got better performance running the Discourse test suite on Mac+Windows+WSL. In fact I am willing to take bets you would.

So, to all those people who say… oh there is an alternative … just hackintosh your way out of this mess. Not only are you stuck playing Russian roulette every MacOS update, you are also paying a tax which is similar to the tax you are paying on Windows already.

What about parallel testing?

Rails 6 is just around the corner. This is the first time Rails is going to ship with officially supported and sanctioned parallel testing. When I run the Discourse spec suite on my Linux system CPU barely scratches the 10% mark for the whole 8 minutes the test suite is running, IO is not saturated.

Here I am freaking out about a measly 38% perf hit when I could be running stuff concurrently and probably be able to run our entire test suite in 2 minutes on my current machine on Windows in a VM.

It may feel a bit odd to be making such a big deal prior to taking care of the obvious elephant in the room.

I completely agree, parallel testing is an amazing thing for Rails, this is going to make a lot of developers extremely happy.

Also, profiling our test suite, eliminating and improving slow tests is super important.

We are going to adopt parallel testing for our dev environments this year.

But I guess this was not my point here. My issue is that we I was driving with the hand break on. Even when our test suite gets faster, the hand break will remain on.

Where am I headed?

I am feeling significantly happier in my Arch Linux home. In a pretty surprising turn of events not only is stuff much faster for me, I also feel significantly more productive at work due to having a windows manager that works much better for me than my Mac or Windows setups ever did. Yes there are compromises, I need to get my hands far dirtier than I had to in the past. However the payoff has been huge.

I have been a long time I3wm user, however I never got the proper experience being straddled in the middle of 2 windows managers. Now that i3 is my only windows manager I am unlocking tremendous amount of value out of it.

Why, you ask? Well I plan to write a bit about my experience over the next few weeks. My plan is to try out a different tiling windows manager every month for the next few months to find the flavor that fits me best.

I stuck with Windows for 6 years developing an application that works best on Linux because I was comfortable in Windows. Habits are incredibly hard to break. I was not fully aware what price I was paying. I can also assure you many other developers are in the same boat as I was.

If I have one piece of advice here, it is … don’t be afraid to experiment. Linux on the desktop is getting better, it is reasonably straight forward to re-partition a drive and setup a dual booting system. If you are in the same boat as I was, living between 2 worlds, especially if you are on a desktop and not a laptop, take a break and experiment.

Please feel free to post any of your experiences or benchmarks here, I will try to answer every post on my blog carefully. I am curious to see more benchmarks from more people comparing MacOS to Linux on the same computer or Windows+WSL / VM and Linux.

And as always … enjoy.

Comments

Sam Saffron over 5 years ago
Sam Saffron

short addendum… Linux doing so well is not due to meltdown/spectre shortcuts on my computer:

SUMMARY: CVE-2017-5753:OK CVE-2017-5715:OK CVE-2017-5754:OK CVE-2018-3640:OK CVE-2018-3639:OK CVE-2018-3615:OK CVE-2018-3620:OK CVE-2018-3646:OK

When running: GitHub - speed47/spectre-meltdown-checker: Spectre, Meltdown, Foreshadow, Fallout, RIDL, ZombieLoad vulnerability/mitigation checker for Linux & BSD

Eddie J over 5 years ago
Eddie J

Same experience and similar solution. I think you hit on one of the biggest contributing factors to why Node.js has grown so rapidly. The Node mentality of pure JS is best, means 95% all projects work seamlessly across platforms. Platform independence was an early feature and the Jimp module is a beautiful example. In contrast, other languages like PHP and ROR have been relying on os level packages like imagemagick. Once those biases last for decades, becomes a big challenge to solve vendor lock in. For new projects, the solution is to use the latest and greatest, for existing projects, the solution is not so clear.

Sam Saffron over 5 years ago
Sam Saffron

I think a big contributing factor with node being so “multi platformy” is that it is rooted in v8 (and v8 being so fast). Clearly google need Chrome to work great everywhere, so node start from a position of strength.

node still has a bit of juggling you need to do rarely with: The Future of Native Modules in Node.js - NearForm , eg: GitHub - sass/node-sass: Node.js bindings to libsass

It admirable that the node team are careful to supply cross platform primitives such as https://nodejs.org/docs/latest/api/fs.html#fs_class_fs_fswatcher something for example that does not exist in Ruby core.

Additionally, stuff like https://nokogiri.org/ in Ruby simply leans on libxml and requires a native compile whereas the node equivs GitHub - cheeriojs/cheerio: The fast, flexible, and elegant library for parsing and manipulating HTML and XML. and GitHub - jsdom/jsdom: A JavaScript implementation of various web standards, for use with Node.js etc are simply written in JS.

In node you tend to need to reach out to c extensions a lot less for pure performance cause V8 is simply so fast.

Paul Ceccato over 5 years ago
Paul Ceccato

Dude, you’re playing Rocksmith too! I have to play it on my mac as I can’t figure out what audio driver-fu is required to get it working on Windows.

Tony Murray over 5 years ago
Tony Murray

Welcome to Arch Linux, glad you are enjoying it.

As for video cards, AMD has a lot better experience because they decided to embrace open source several years ago and it is really paying off. I suspect it was a large factor for them being selected for Google Stadia.

Sam Saffron over 5 years ago
Sam Saffron

Odd, it seemed to work just fine on my Windows with no magic, just plugged in the USB cable and it worked.

I really enjoy Rocksmith I am not sure if I am really getting much better with my guitar, but it sure is good fun.

Sam Saffron over 5 years ago
Sam Saffron

Thanks Tony,

Yeah, my understanding is that Intel are still the best and then AMD are a second and NVIDIA a distant third.

If I was building a Linux desktop from scratch I would absolutely go with AMD over NVIDIA.

Eddie J over 5 years ago
Eddie J

Yeah, that is true about the v8 engine being fast. Also, very interesting stuff about n-API. I guess that’s a product of not having a bunch of legacy code. Node is relativity new despite the JS API being ancient. Remember Dhtml? Js was historically developed at a snails pace. Then again, chrome has been working on their engine for a long time, that must be secret sauce. Hopefully it inspires ROR core to reevaluate biases at a lower level to keep things going.

Paul Ceccato over 5 years ago
Paul Ceccato

We should have a rocksmith jam someday!

Alex over 5 years ago
Alex

I have a similar experience, but I didn’t take that long to abandon Windows
as a dev machine, I fought about 1 year, ended using a VM and as I spent less and less time on
Windows native tools, I asked myself: What if I use Linux with a Windows on a VM?

And that’s what I did, I’m using Linux Mint for about 7 yrs now, and I’m very happy with it.

The jump you do in productivity when you learn to use the many available command line *nix tools on your work ( grep, awk, zsh to name a few ) is unparallel.

Tom over 5 years ago
Tom

Unlike you, I looked at these numbers and thought “I have to get off this MacBook Pro ASAP”. The keyboard issue didn’t help, but 2/3 the time to run tests - I’ll take that every day of the week.

I just ordered a “gaming” “laptop” with a i9-9900K desktop CPU.

What’s sad is, 10 years ago, I built a Ruby on Rails business, deployed with Linux, but developed on Windows. Git was tricky to set up right, but Ruby and Rails mostly worked great!

Sam Saffron over 5 years ago
Sam Saffron

I am very curious if you can report some before / after numbers once you are done with your upgrade?

Tom over 5 years ago
Tom

I have no final numbers yet, but this is much faster at running the tests, but, in WSL, when running 8 parallel processes, the File Load time that RSpec reports is over 1 minute per process and is half the overall time to run the specs.

Running that same code in a Hyper-V VM of Ubuntu runs the specs in the same time, but the File Load time is only 8 seconds per process.

What’s interesting is that there is clearly contention in the file system, as the more processes I use, the slower the File Load total gets. 1 process loads in 30 seconds, 8 processes take 1M+ per process (so over 8 minutes when added up)

WSL is magical, except for this file access issue.

Sam Saffron over 5 years ago
Sam Saffron

I have some great news for WSL users.

I have recently tested the alpha release of WSL2 and am happy to report the slow filesystem issue is completely gone. The performance characteristics of WSL2 are extremely similar to Hyper-V. The custom kernel in WSL2 conserves memory really nicely, often Linux thought I had 1.5G in use when taskmgr reported half.

You get the same trivial setup and ultra fast boot you did with WSL1 and no longer pay a huge performance cost for file system access.

Once released I would very much recommend WSL2 as the first port of call people should try when they want to contribute to Discourse under Windows. It means you don’t need to fiddle with Virtual Box / VMWare or Hyper-V anymore to get best in class Linux on Windows performance.

The performance hit you would expect is in the range of 0-20% depending on your workload and virtually identical to what other Hyper-V offers (except that memory will be better on WSL2 due to custom kernel)

Tom over 5 years ago
Tom

I saw this announcement!

Does this also mean that FUSE and DTrace will be supported too? I know regular Windows recently added DTrace support, and the Linux kernel does too. And I have some ideas I want to play with

The tricky part (for me) of Ubuntu/Windows over Hyper-V has been the file sharing. If they can nail that in WSL 2, and make it faster, I will be much happier in my choice to switch to Windows.

Sam Saffron over 5 years ago
Sam Saffron

Yes, this is a proper install of the Linux kernel with a few patches, as far as I can tell there are 3 patches (at least) involved

  • Auto mounting of /mnt/c which gives you access to windows file system
  • Automatic SMB like protocol to talk to the VM (I have not got this to work yet but I am on alpha)
  • Proper comms with hypervisor so memory behavior is far more efficient and WSL2 uses less

I am sure there is more to it, but we can all review this code in a month to get the full scope.

This really is a killer set of features, over the years I fussed so much getting shares to work between VMs and host, its just one of those fiddly things.

With WSL2 after the 2 second boot everything is ready to rock and roll.


comments powered by Discourse