Many developers spend a fair amount of the day waiting:

waiting place

Waiting for a build to end
Waiting for search results to come up
Waiting for an editor to launch
Waiting for tests to run
Waiting for a boss to give instructions

There is also a more insidious form of waiting. The zero value work you tend to do repetitively, also known as trivial inconveniences.

These issues don't stop you from working, they just make your job slightly more annoying.

For example:

  • Every time you hit the "refresh" button to see how your CSS change modified the site you are working on.
  • Every time you manually run your test suite.
  • Every time you "compile" your project and wait for it to finish.

Computers could do these tasks automatically; instead, like trained monkeys, we often repeat these pointless tasks tens and hundreds of times daily.

While developing Discourse I pay very special attention to waiting and trivial inconveniences. I work tirelessly to clear friction. In fact, I will spend days to eliminate particularly bothersome bottleneck.

I use Vim

Vim is tough editor to pick up, especially late in your career. It is notoriously difficult to master.

Fortunately, people getting started today with Vim have some amazing resources such as Practical Vim by Drew Neil, Vimcasts and VimGolf that make the process less painful.

Vim helps me eliminate many trivial inconveniences.

  • It launches instantly, I am never waiting for my editor to start up.
  • I use a fair amount of Vim plugins. Syntastic, rails.vim, NERDTree, surround.vim, tcomment and many others. The plugins I use help meld Vim so it works really well with my workflow.
  • I have some Discourse specific bindings.
    • I bind CTRL-A to touch a restart file, I can bounce my web server in a key stroke if needed.
    • I bind CTRL-S to dump a message in the Discourse messaging bus that forces all web browsers pointing at local Discourse to refresh.
    • I have a variety of navigation hot keys to cut through the JavaScript files, :Rdmodel t<tab> ... takes me to the topic js model.

I am not afraid to switch tools

I have used Ack for a very long time to search through code files, recently I discovered the silver searcher. Silver searcher is faster than Ack, to me this makes all the difference. To quote Geoff Greer, the author.

I can already hear someone saying, “Big deal. It’s only a second faster. What does one second matter when searching an entire codebase?” My reply: trivial inconveniences matter.

I am not afraid to spend money on software

I have Windows 8 Desktop (with 2 monitors) and a MacBook Pro Retina on my desk. Initially, I used to reach over to the laptop to type in commands in order to test stuff out. Reaching over to the laptop is not a mammoth effort but it is very inconvenient. Due to this trivial inconvenience I rarely used my laptop during the day.

I downloaded Synergy and tried sharing my keyboard with my Mac laptop. It worked... sometimes. It crashed and failed a lot and was a monster to configure. Many developers would stop here. Free does not work, go shopping, for something else free. I disagree with this attitude.

There are a couple of paid alternatives. Share mouse worked very well for me. In a blink I spent the 50 or so bucks to eliminate these inconveniences. Why save 50 bucks just to have a $2000 laptop sit on your desk collecting dust?

Live CSS refresh

Whenever I save a CSS file my browsers automatically refresh with the amended visual style. This is accomplished with a very Spartan amount of code.

In our Guardfile:

module ::Guard
  class AutoReload < ::Guard::Guard

    require File.dirname(__FILE__) + '/config/environment'
    def run_on_change(paths)
      paths.map! do |p|
        hash = nil
        fullpath = Rails.root.to_s + "/" + p
        hash = Digest::MD5.hexdigest(File.read(fullpath)) if File.exists? fullpath
        p = p.sub /\.sass\.erb/, ""
        p = p.sub /\.sass/, ""
        p = p.sub /\.scss/, ""
        p = p.sub /^app\/assets\/stylesheets/, "assets"
        {name: p, hash: hash}
      end
      # target dev
      MessageBus::Instance.new.publish "/file-change", paths
    end

    def run_all
    end
  end
end

guard :autoreload do
  watch(/tmp\/refresh_browser/)
  watch(/\.css$/)
  watch(/\.sass$/)
  watch(/\.scss$/)
  watch(/\.sass\.erb$/)
  watch(/\.handlebars$/)
end

In our devlopment js file it does something like:

return Discourse.MessageBus.subscribe("/file-change", function(data) {
  return data.each(function(me) {
    var js;
    if (me === "refresh") {
      return document.location.reload(true);
    } else {
      return $('link').each(function() {
        if (this.href.match(me.name) && me.hash) {
          if (!$(this).data('orig')) {
            $(this).data('orig', this.href);
          }
          this.href = $(this).data('orig') + "&hash=" + me.hash;
        }
      });
    }
  });
});

}

There are plenty of solutions out there that achieve the same goal, there is LiveReload. The folks at Live Reload also documented some other options.

I chose to roll out our own solution cause I wanted better control. Our code knows how to reload Ember templates and will grow to do more sophisticated things.

I spend time making the development environment fast

Discourse has a very large number of JavaScript and CSS files. This means that during development the web server has to serve out over 370 assets to the clients. This is certainly an edge case for Rails. However, as client side frameworks like Ember and Angular become more popular this is going to be a problem more people are going to face.

I committed a fix to Discourse that cuts down the time it takes to refresh a page down from 4.5 seconds to 1 second in Development. I also opened an issue to help decide if we want to include the optimisation in Rails or not. Unlike more common attempts, that bundle up all assets in Dev to "work around" the problem, this fix is transparent and does not compromise debugging. In future when source maps are more pervasive, hacks like this may not be needed. I am a pragmatist though, this solves our problem now. Why wait?

I also committed a fix that dramatically improved sprockets (the Rails Asset Pipeline) in dev mode.

I have effective Rails development environment

I have a reasonably fast workstation, though I am due for a Haswell upgrade next month. I have been running dedicated SSDs for development for years now. I have a multi-monitor setup primary one is a 30 inch Dell.

my desktop
larger image

Most Rails developers out there are probably using Ruby 1.9.3 (or 1.8.7) untuned to work locally. If I start a Rails console to Discourse on an untuned stack it takes 16 seconds to boot up. I run Ruby 2.0 in Development with the following environment vars:



export RUBY_GC_MALLOC_LIMIT=1000000000
export RUBY_HEAP_SLOTS_GROWTH_FACTOR=1.25
export RUBY_HEAP_MIN_SLOTS=800000
export RUBY_FREE_MIN=600000
export LD_PRELOAD=/usr/lib/libtcmalloc_minimal.so

This reduces Discourse startup time to just under 4 seconds. That is a 4x improvement just by tuning my dev machine. Additionally, I have experimented with Zeus and Spring (Discourse is pre-configured to work with Spring) with varying degrees of success. When these forking apps work you get 1 second startups.

We use better errors that works many times better than the default Rails error page and MiniProfiler that allows me to quickly debug through issues.

I really hate waiting for tests

When I started working with Discourse we had a lightning fast test suite. I could easily work on an area of code and get immediate feedback about my specs.

Fast forward a few months, we now have a test suite that takes 3 minutes to run, after you pull some ninja moves to get it that fast.

I agree it takes too long, I would like to refactor and speed it up. However, this is not the point of this section.

There are currently 2 common spec runners used by the Ruby community, autotest and Guard. Guard is more modern and actively developed.

Trouble is both runners are very troublesome for my workflow.

Both runners insist on running every single spec in the queue before obliging and running the spec you want them to run. So, my typical workflow used to be.

  1. Launch bundle exec guard
  2. Write my broken test
  3. Save
  4. Wait up to 3 minutes for the old run to finish so I can see it really failed
  5. Fix it
  6. Save
  7. See it passed
  8. GOTO (2)

So, a lot of waiting. But its critical I run the entire test suite often cause many of our tests unfortunately integrate.

To resolve my large amount of pain I took the time to write a test runner that works really well for me. https://github.com/discourse/discourse/blob/master/lib/autospec/runner.rb .

  • It will interrupt the current testing when it detects I want to run a spec (cause I saved a file)
  • It will resume the old testing after it finished running the spec I wanted it to run.
  • It will automatically focus on a broken test (only run a single test) something that allows me to spring puts statement around my code to quickly inspect state for the broken spec.
  • It allows me to switch focus to another spec (by saving a spec) even in a failed state.

For me, having a test runner that operates as I would expect it to has completely transformed the way I work. I no longer approach testing as a drudge, it is fun again.


Eliminating trivial inconveniences makes me a happier developer. Go ahead, find one inconvenience you have and eliminate it. You will not regret it.

Comments

Vlad over 1 year ago
Vlad

Nice reading, thank you.
I’m really interested (not holy war or trolling) about why Windows, and why Windows 8. And can you please tell more about Rails environment inside of your Win box – vagrant? cygwin?

Harry over 1 year ago
Harry

I’m also really curious about your rails environment on Windows, could you expand on it a bit?

Also, very nice article, thanks!

Sam Saffron over 1 year ago
Sam Saffron

Thank You!

I run a local Ubuntu VM for dev, the terminal you are seeing is gnome-terminal running tumx. I have a cygwin X Server running on my Windows box (startx). I then ssh -Y into my Ubuntu machine to get a proper terminal allowing my to fire off gVim and other X apps.

If you are interested in a slightly simpler Windows setup you could look at http://mobaxterm.mobatek.net/

I do not use Vagrant, all my code lives inside the VM, its portable, I can take it on any laptop. I prefer developing in Linux as opposed to BSD/Mac mainly cause we deploy on Linux and I prefer being closer to the production setup forces me to learn more about Ubuntu and its quirks.

I strongly recommend against trying to do any real Rails development on Windows using Windows Ruby. Its slow and you will hit gems that do not work.

Harry over 1 year ago
Harry

Thanks for sharing your setup Sam! I’m gonna take a look at Cygwin and X11 forwarding.

Thank you again for your text.

John_F over 1 year ago
John_F

This reminds me of a piece I read about Herman Miller shaving something like half a second off chair production, when they are already very fast at making chairs. I can’t find it now, but this blog post about them is pretty cool: http://www.treehugger.com/sustainable-product-design/herman-millers-greenhouse-factory-generates-15-pounds-of-landfill-waste-per-month.html

Ks over 1 year ago
Ks

if you develop on Ubuntu anyway, why not run that instead of Windows 8? Sounds like a lot of hassle.

Sam Saffron over 1 year ago
Sam Saffron

I really like my tricked out windows 8, its super stable has all the latest browsers and a good enough windows manager. This setup also allows me to easily experiment with my dev environment. I can install latest GCC test something out and then revert to snapshot.

Anthony_D over 1 year ago
Anthony_D

Always interesting to see how top devs go about their workflow.

What is this Discourse MessageBus of which you speak? Is the existence of this message bus the reason why you didn’t use something like guard-livereload for browser refreshing?

I don’t use a VM, i develop on the desktop (Macbook, vim, git) and then rsync with one command to the dev box (Linux, web server, db server, web framework). I wonder could I use your guard technique to ping changes back to my browser? That’s a pain point I never considered removing and I have had to disable caching in the developer tools part of Chrome and/or use query-string date and time stamped CSS html tricks. What do you reckon?

Sam Saffron over 1 year ago
Sam Saffron

The message bus deserves its own blog post, in a nutshell it is very similar to faye. Unlike faye it is tightly coupled to redis and allows for reliable recoverable channels. The server does not need to keep track of the clients connected, instead it keeps track of the channels. clients can always issue a command to catch up on messages. It lives here: https://github.com/discourse/discourse/tree/master/vendor/gems/message_bus

It allows you to do stuff like this
The real power is that in a single line of code you can activate code on the client side from the server.

If you want live CSS changes I would recommend checking out guard livereload first. We already use the message bus to notify users of new posts or message, adding another listener is totally trivial.

Rick over 1 year ago
Rick

@KS I also run windows as my main, and develop in linux VM. I tried Ubuntu & Debian, Centos & Redhat and I would always run into some issue that required reading configuring downloading special drivers etc etc, so much lost productivity. To steal from Mac, Windows just works for me. Ubuntu is still way behind OSX/Windows for desktop management, all I want from linux is the command line and services, which the VM provides nicely. In addition I can experiment carefree with my linux and packages and if I goof up or go down a path of regret, rolling back my work 3 hours ago takes all of 10 seconds. I can mimic any production environment we have or clients use exactly. I will never NOT develop on a VM. Once you see the light then your choice becomes what windows manager do you prefer the most? OSX Windows FlavorOfLinux, and from there everything will be the same in VM. I prefer windows myself as games are still important to me off hours and there is far better support for various apps on windows

@OP synergy works great for me, been using it for years.

host = windows
left = mac
right = laptop running linux

I use it less however, osx and linux are now VMs on my windows

Speaking of VMs, I also have an i7 and cores are really overrated. I have osx and 2 linux vms going nearly all the time, it always saddens me when I see “parked” cores.

I leave all of that open and play games in a window for breaks, still doesnt tax all the CPUs. Probably should have went i5 and spent more on GPU :)

Xr09 over 1 year ago
Xr09

Just post this to say, Synergy works, stop spreading FUD.

The config couldn’t be easier. Put something like this in the server and execute synergys, then execute synergyc server-ip on the clients. https://gist.github.com/xr09/5508785

Sam Saffron over 1 year ago
Sam Saffron

It works for you, great! Unfortunately, it did not work for me. Maybe related to the fact I am on Windows 8, maybe something about my setup, who knows. It certainly was not simple to configure, involved firewall editing and other nasties.

Charles_Feduke over 1 year ago
Charles_Feduke

This is a great article about actually improving workflow as a developer. While I’ve been using all of the tools you mention the Rails improvements and configuration will certainly help speed up my work. (I’ve actually stopped developing new projects with Rails for the most part because of startup time! Maybe I can give it another shot.) Thanks for taking the time to share your experience.

Pawe_Go_cicki over 1 year ago
Pawe_Go_cicki

Really liked reading about your setup. Especially the parts about running your specs. Latest ‘guard-rspec’ has an option to re-run the failed spec, called :focus_on_failed => true which works pretty well.

I wonder if it could be possible to implement “interrupt currently running spec after file save” in guard-rspec as well.

Sam Saffron over 1 year ago
Sam Saffron

Been discussing this with the guard team, we would like this done. Sure know about focus_on_failed ... I wrote it smile

Matt_Jibson over 1 year ago
Matt_Jibson

I just committed a one-key user reset that I perform many times based on this advice. Thanks.

Andrew_Vit over 1 year ago
Andrew_Vit

Do you actually work strictly inside the VM or do you share a folder from your host OS into it? On my Mac, I like to have my normal GUI tools for working with the code (MacVim, SourceTree, etc.) but have everything running inside the VM environment using Vagrant & Virtualbox. One major obstacle is with detecting file changes in the project folder, since inotify or fsevents don’t work over the virtualization boundary. VMs are great but it feels like a compromise sometimes.

Sam Saffron over 1 year ago
Sam Saffron

I do all my work strictly in the VM, I use gitg/git gui and gvim. The big advantages are that its faster cause the FS is faster and you get to use inotify.

I find gvim pretty much indistinguishable from mvim. SourceTree is nice, but you can always keep using it by having an smb/nfs share out of your VM. Personally I use gitg and git gui most of the time.

Dvr over 1 year ago
Dvr

refresh … css

thanks youve given me a very good idea, Ill share it later if I can get it to work…

Paul_Gatt over 1 year ago
Paul_Gatt

Hi Sam,

I was at RORO Syd last night and watched your talk.

Just wondering where do you place these lines:

export RUBY_GC_MALLOC_LIMIT=1000000000
export RUBY_HEAP_SLOTS_GROWTH_FACTOR=1.25
export RUBY_HEAP_MIN_SLOTS=800000
export RUBY_FREE_MIN=600000
export LD_PRELOAD=/usr/lib/libtcmalloc_minimal.so

Thanks for the talk it was really interesting.

Sam Saffron over 1 year ago
Sam Saffron

simplest spot would be .rvmrc if you are on rvm ... or perahps .bashrc

also be sure to have the right tcmalloc installed if you are adding it

Paul_Gatt over 1 year ago
Paul_Gatt

what’s the right tcmalloc? I am on Mac

Sam Saffron over 1 year ago
Sam Saffron

Honestly I am not sure, you may have to hand compile it. I think its safe to skip.

Jorge_Castro over 1 year ago
Jorge_Castro

Thanks for writing this up Sam.

Are you using any tools to rev quickly from local to a real server? We’re working to solve this problem directly in Ubuntu with Juju and doing some Vagrant integration work.

Your problems seem to be the exact target audience we’re trying to fix. Rev quickly locally and then push to remote quickly.

Any workflow painpoints from running locally to remote I’m particularly interested in as it’s a common problem with a bunch of people we talk to.

Sam Saffron over 1 year ago
Sam Saffron

Internally we use jenkins to validate the builds and push. So it automatically fires off a build when source changes.

Traditionally many Ruby projects will use capistrano http://meta.discourse.org/t/deploy-discourse-to-an-ubuntu-vps-using-capistrano/6353

The huge pain point with Ruby apps is daemonizing, its the #1 issue. You have to run daemons for your web, sidekiq (job queue) and clockwork (ruby style cron) for discourse. Figuring out which is the right way to daemonize and monitor (monit / upstart / bluepill / foreman) it a huge pita.

Can you pop on to http://meta.discourse.org to discuss some of this ?

Kev over 1 year ago
Kev

Nice article and thanks for the MobaXterm tip-off. Took me all of 2 mins to get PyCharm (installed on my Fedora 18 box) to launch and appear on my Win7 desktop…was really quite gobsmacked that it worked first time. I’ve tried and failed so many times to make this happen with Cygwin’s X bits on Windows, MobaXterm “just works”.

I totally agree with your “I am not afraid to spend money on software” sentiments. It’s crazy that some folks get hung up on ideology and won’t spend $50 on a commercial/closed source tool to help them get the job done more efficiently. I’ll happily pay for closed source tools as much as I am happy to donate money to open source projects that I find essential.

Alex over 1 year ago
Alex

Using a VM “steals” many seconds from you.

I had a similar setup, Win7 and Ubuntu vm some years ago, now I have Win7 on a vm that I don’t remeber the last time I’ve used.

Jack over 1 year ago
Jack

A vim alternative to file watchers like Guard is the auto command hook BufWritePost. I’ve been using this to kick off my css preprocessor and trigger a css refresh.

One commonly overlooked impedance to development flow is typos. Most of the programmers I know are touch typists when it comes to letters and common punctuation, but the slew of symbols in code is a constant source of interruptions to trains of thought. The lack of good typing tutors for programmers annoyed me enough that I developed typing.io, a typing program that lets programmers practice typing code in multiple languages from open source projects.

Jon_Burgess 11 months ago
Jon_Burgess

Thanks Sam, you have inspired me (finally) to create a vagrant/berkshelf template project!


comments powered by Discourse