In late May 2016 Chrome pushed out Chrome 51, unlike many releases of Chrome which are complete non-events, this release has an enormous impact. Google turned on Brotli support – and they promptly backported it into Chrome 50.

Firefox added support for Brotli in September 2015. They even blogged about it. 8 months later, thanks to Google, Brotli went from a compression format supported in less than 10% of global browsers to nearly 50% global adoption!

Brotli is HTTPS only

If you visit a site over HTTP your browser will not accept the br encoding. The reasoning for this is documented at the end of the chromium issue.

pdansk

I have a question. What does intermediates refer to in the reason to restrict this feature to HTTPS only? Caches? If so, this may or may not be solved by adding no-transform to Cache-Control when you send Content-Encoding: br. I don’t know.

I think that restricting to HTTPS is regrettable, as what you save with brotli you lose double by not having caching.

kenjibaheux@chromium

Intermediaries (or “middle boxes”) refers to companies/infra/software meddling with the data transfer between you (the user) and the webserver.

One example from SDCH that was mentioned to me (the name of the company is not relevant to the discussion so I’m hiding it):

"The most extreme case of middle box was Company AcmeTelecom (fictitious name but true story), that tried to make things better when faced with unknown content encodings, by doing the following things:

a) Remove the unrecognized content encoding
b) Pass the (already compressed!) content through another gzip encoding pass
c) Claim that the content was merely encoded as gzip"

Lack of Brotli support over unencrypted HTTP is no mistake. The world is full of terribly non-compliant HTTP proxies:

https://twitter.com/samsaffron/status/514538810894340097

Brotli is yet another reason for you to push for that HTTP/2 change you have been holding out on.

Brotli: The New Pied Piper?

When I first heard of Brotli, there were a lot of stats being published about how much better the compression is. The Next Web said it is 26% better. and Akamai blogged about its huge potential. I was thinking to myself… did we just uncover a real world pied piper?

In my measurements, the best case compression for Brotli really does blow away the competition:

(percentages are expressed as savings compared to gzip -9)

• Even at level 5, Brotli is better than gzip -9
• Worst case savings is 9.29% (for jQuery)
• Best case savings is 29% (for our Discourse application js)
• The larger the file, the better Brotli fares.
• You can get a free 3-4% savings by using zopfli which is compatible with gzip decompression.

Comparing Brotli compression levels

Brotli, like other compression formats, offers a choice of compression levels, from 1 all the way to 11. Level 5 is where Brotli starts using context modelling, one of the more advanced features of the format.

Compressing Brotli at the highest setting is so slow gzip -9 is practically invisible from the graph!

• Brotli at level 11 is not feasible for dynamic content
• Brotli at level 5 is competitive with gzip and produces files that are even smaller than gzip -9
• Zopfli is really slow, for large files it can take 4× the time of Brotli 11.

Brotli is not easy to adopt today

Another complication with Brotli is that it is missing-in-action.

Ubuntu 16.04 is the first Ubuntu distro with apt-get install brotli, if you want to play around on earlier distros you have to find a third party apt repo or compile it yourself.

NGINX has a Brotli extension, this is not distributed with the official NGINX package, forcing you to compile NGINX by yourself if you want brotli today.

sidenote: nginx -VV makes it significantly easier to self-compile NGINX without getting into a big mess.

Apache only recently got a Brotli extension

When it comes to the various asset pipelines out there, none of them support pre-compressing Brotli assets, so you have to bolt it in yourself.

Even with all these rather big hurdles you can have it today, and it does work!

Brotli decompression is fast

I did not include any graphs or stats for gzip and brotli decompression, they are simply so fast that I would need to build a rather elaborate test harness to get an accurate measure of the difference. It is simply so fast that it does not really matter. I found that the IO of writing the file is the bottleneck not the decompression.

CDN support for Brotli is uneven

Of the 4 CDNs I checked:

Fastly and MaxCDN had stripped out accept encoding for Brotli, falling back to gzip, CDN77 appears to be letting Brotli through. KeyCDN allow Brotli through (they also provide a test to show CDN is allowing Brotli).

Fastly have a documented workaround, but support is not built in.

This uneven support is understandable. CDNs usually normalize the Accept-Encoding header. This happens cause there is a huge matrix of Accept-Encoding in the wild, if this is not normalized the same gzipped resource can be stored 44 times.

If a CDN allows brotli through it can quite easily double all the requests to your website for the origin resource. It has to check if you have both Brotli and gzip for every asset.

Considering that the vast amount of resources you are serving are probably not Brotli compressed, this can become very wasteful, especially if the first request to the asset comes from a browser that does not accept the br encoding like Safari or IE.

Ideally, CDNs should provide you with a way to whitelist some (or all) assets to opt out of Accept Encoding normalization.

Brotli is about to become an RFC

The magic date, May 25, 2016, was not randomly picked by Google for enabling Brotli in Chrome. It was the date IANA finally stamped the ietf draft with RFC Ed Queue. That is the final state the document enters prior to becoming an RFC and getting an RFC number. The approval announcement is already out.

The draft’s authors submitted it to the Independent Stream in April 2014. It took the ISE rather a long time to find suitable reviewers for a new compression technique, but, it finally got a brief review from Ulrich Speidel(U Auckland, a coding theory researcher), and a more thorough review from Jean-Loup Gailly (the author of gzip). The authors made changes based on those reviews, and it is now ready for publication.

Google did the right thing in not rushing Brotli to the public, it is excellent that Jean-loup Gailly and Mark Adler the original authors of the gzip format had a chance to properly review the draft.

What can we do?

Proxy and web server vendors should make it easier to experiment with Brotli

I would like to see NGINX, Apache, HAProxy etc. make it easier for people to experiment with Brotli. For NGINX this means shipping with Brotli support in an optional official extension.

The biggest challenge ahead is figuring out a smart heuristic for figuring out which compression level to use balancing compression with CPU cost and current load on dynamic content. This is very critical for Brotli adoption, there is huge variance between maximum and minimum compression for both size and time. This was never the case for gzip.

Microsoft’s IIS has special settings that allow it to disable gzip compression if CPU goes higher than a certain amount. Ideally NGINX brotli extensions can learn from it and enable a setting that backs off on compression if load is too high due dynamic assets compression.

Overall, dealing with brotli compression requires that our web servers get smarter and more understanding of the huge variance CPU with between the various brotli compression levels.

CDNs should advertise their Brotli support

CDNs these days publish support for HTTP/2, usually on the front page. This is excellent and I am impressed that so many CDNs these days ship with H2 support.

Brotli support on the other hand is a completely hidden item. It is not advertised anywhere. Given the huge compression benefits for the web, I hope CDNs can begin address advertising their Brotli support soon. I hope the community can start collating a list, say on http://cdncomparison.com/ where people shopping for CDNs can decide based on Brotli support.

Additionally there is a big untapped market in the CDN world for adding Brotli compression to assets, like many CDNs do with gzip.

You can experiment with Brotli support for static assets today

At Discourse we now allow you to optionally precompile all your static assets as .br files. You can see this in action when you visit https://discuss.samsaffron.com on latest Chrome.

These are assets that never change between deploys and are always consumed on the front page. You are very likely to also have these kind of assets; your common css file and common js file.

If you have control over your asset pipeline adding support is quite simple. NGINX support though slightly more complicated is straight forward. There are huge wins to be had with the new format and adopting it for “precompiled” assets is a huge win, especially on Android mobile.

If you can perform brotli 11 compression upfront on all your most common static assets you will see very large network performance gains on supporting browsers.

###Hold off on fighting with Brotli support for dynamically generated requests unless you fit the correct profile

On-the-fly Brotli compression is an area that needs a lot of experimentation. It is not enough to flick the switch at level 5 and forget about it.

There is a clear win enabling brotli level 1 if you had gzip level 1 enabled.

There is a clear win enabling brotli level 5 if you were comfortable with the CPU cost of gzip level 9 on dynamic assets and have capacity. (in this case you can expect a 5-10% win size wise)

In some cases there is no clear win enabling brotli on dynamic content.

I will follow up this article with another properly addressing the question of on-the-fly brotli compression.

Lobby Microsoft and Apple to add Brotli support

It is unclear when Microsoft and Apple plan to adopt Brotli in their browsers. This is particularly important for mobile browsers where every byte count quite a lot!

Brotli is an exciting technology, which you can start adopting today, big thanks to Google, Jyrki Alakuijala and Zoltán Szabadka!

Big thank you to Zoltán, Ilya and Jeff for reviewing this this article.