Jekyll and it's alternatives from a site generation point of view

As I have previously mentioned this site runs jekyll, which works with reasonable speed however my personal blog which contains a few more pictures creates a problem. The cause is mainly slow regeneration times, something that also affects this site. I believe that the main performance missuses stems from two main sources, mainly jekyll’s single threaded approach which does suffer on a laptop cpu like the one I use, another issue is with the image plugins used that offers nice features but also creates a performance loss. With this in mind I have decided to benchmark the site generation between a few static generators to evaluate the competition and possibly change engine. The static site generators I will focus on is Wintersmith, which is built on node.JS and Hugo, which is built on GO.

The three generators are similar in usage but to compare performance I will need to create sites with similar features on all three of them. To simplify performance comparison I will omit the picture_tag plugin described in a previous post, I will however still include the full scale images to determine how large resources are handled. To eliminate further contamination of third party scripts, I will use the standard themes for each of the sites. Apart from a selection of “real” posts I have included some generated posts with large amounts of text.

To get relevant results I ran 10 builds for each site and calculated the mean value for the runs, the resulting values is presented in the following chart.

There are a few things worth mentioning about the values in this chart.

  • The measured times showed very small variations.
  • Since jekyll don’t provide build time output I used “time” to measure jekyll build times.
  • Even though Wintersmith have slow build times, the rebuilds are relatively fast, this is just based on “feeling” no measurements where done.

This is a rather basic test but my results seems logical especially since Hugo claims to focus on speed.

To further validate these results I ran test with large sites (1000 auto-generated posts) Where I basically generated posts using the following commands:

for ((n=0;n<1000;n++)); do hugo new post/test_$n \
&& cat content/post/test.md >> content/post/test_$n; done

for ((n=0;n<1000;n++)); do ./jekyll-post -D "testsite/_posts/" test_$n \
&& cat test.md >> testsite/_posts/2014-08-12-test_$n.md; done

And for Wintersmith I basically copied the jekyll posts.

To well managed data I used the following command to run the tests:

for ((n=0;n<10;n++)); do time buildCommand &> out.log; done

Which means that I use “time” for all the tests this time. The following chart shows the results in seconds.

A complete extract of the command output is at the end of this post, where it’s worth noting that Hugo uses several cores, while jekyll and Wintersmith makes do with one. Another thing worth mentioning is that the results shown above should be considered relative to each other instead of focusing on the specific timings. However in the interest of providing data, the generated site contained 1000 posts, totaling in 77mb of text.

With these results I can basically conclude that hugo is the fastest of the three.

jekyll build &> out.log  72.41s user 0.69s system 99% cpu 1:13.37 total
jekyll build &> out.log  72.02s user 0.81s system 99% cpu 1:12.93 total
jekyll build &> out.log  72.02s user 0.67s system 99% cpu 1:12.77 total
jekyll build &> out.log  97.32s user 1.32s system 99% cpu 1:39.08 total
jekyll build &> out.log  89.43s user 1.01s system 99% cpu 1:30.78 total
jekyll build &> out.log  73.79s user 0.82s system 99% cpu 1:14.72 total
jekyll build &> out.log  74.58s user 0.79s system 99% cpu 1:15.47 total
jekyll build &> out.log  75.81s user 0.81s system 99% cpu 1:16.79 total
jekyll build &> out.log  75.60s user 1.14s system 99% cpu 1:16.91 total
jekyll build &> out.log  76.62s user 0.80s system 99% cpu 1:17.66 total
hugo &> out.log  11.17s user 0.56s system 263% cpu 4.449 total
hugo &> out.log  11.25s user 0.42s system 289% cpu 4.036 total
hugo &> out.log  11.04s user 0.38s system 290% cpu 3.932 total
hugo &> out.log  11.06s user 0.36s system 290% cpu 3.932 total
hugo &> out.log  11.14s user 0.37s system 289% cpu 3.977 total
hugo &> out.log  11.06s user 0.36s system 290% cpu 3.925 total
hugo &> out.log  11.76s user 0.41s system 260% cpu 4.666 total
hugo &> out.log  11.20s user 0.37s system 286% cpu 4.042 total
hugo &> out.log  11.35s user 0.40s system 271% cpu 4.329 total
hugo &> out.log  11.55s user 0.42s system 270% cpu 4.423 total
wintersmith build &> out.log  47.02s user 3.51s system 98% cpu 51.353 total
wintersmith build &> out.log  41.86s user 2.23s system 101% cpu 43.516 total
wintersmith build &> out.log  41.17s user 1.95s system 101% cpu 42.396 total
wintersmith build &> out.log  41.40s user 2.02s system 101% cpu 42.603 total
wintersmith build &> out.log  46.20s user 2.41s system 101% cpu 48.025 total
wintersmith build &> out.log  40.77s user 1.89s system 101% cpu 42.094 total
wintersmith build &> out.log  45.93s user 2.20s system 101% cpu 47.213 total
wintersmith build &> out.log  47.15s user 2.37s system 101% cpu 48.648 total
wintersmith build &> out.log  47.05s user 2.62s system 101% cpu 49.035 total
wintersmith build &> out.log  48.66s user 2.81s system 101% cpu 50.829 total

Update

As mentioned in the comments the output from my builds is somewhat unclear, to remidy this I have re-run the tests, this time in maner that’s easier to recreate.

The following test output is generated on a A3 Basic linux vm in Azure, the machine is running ubuntu 14.10, I am using the latest stable version of jekyll, hugo and wintersmith.

The last one have caused some issues, mainly caused by out of memory exceptions this have forced me to limit the amount of pages to 100, for the record though both jekyll(444s) and hugo(115s) managed to generate 1000 pages. The usage of the latest version of hugo (v.0.12) have caused some issues with performance which seems to have taken a pretty decent hit with since v.0.11, this does change the outcome of the tests considerably. I ran a quick test of hugo v0.11 and the build time difference is massive v.0.11 (12s) vs v0.12 (51s).

With the updated data I can conclude the following

  • Hugo v.0.11 is still the fastest, by a wide margin
  • Hugo v.0.12 is slightly slower than wintersmith, however wintersmith does not handle massive sites (500+ posts) at all
  • Jekyll is still the slowest but have a massive community which is a plus

To run the tests I have used the following bash script, it assumes that all “generators” is available in path.

#!/bin/bash
rm -rf HTest JTest WTest jekyll-post testData
wget https://s3-eu-west-1.amazonaws.com/imagesfredrikme/testData
echo "Creating jekyll site"
wget https://raw.githubusercontent.com/jbranchaud/mybin/master/jekyll-post
chmod 755 jekyll-post
jekyll new JTest
for ((n=0;n<100;n++))
do
    ./jekyll-post -D "JTest/_posts" test$n > /dev/null
done
for f in JTest/_posts/*.md
do
    cat testData >> $f
done

cd JTest
echo "Jekyll tests"
for ((n=0;n<10;n++)); do /usr/bin/time -f %e jekyll build >> ../JekyllOut.log; done
cd ..
rm -rf JTest

echo "Creating hugo site"

hugo new site HTest
cd HTest
for ((n=0;n<100;n++))
do
    hugo new post/test_$n.md > /dev/null
    cat ../testData >> content/post/test_$n.md
done
git clone https://github.com/spf13/hyde.git themes/hyde
cd ..

cd HTest
echo "Hugo tests"
for ((n=0;n<10;n++)); do /usr/bin/time -f %e hugo -t hyde >> ../HugoOut.log; done
cd ..
rm -rf HTest

echo "Creating winthermsith site"
wintersmith new WTest
for ((n=0;n<100;n++))
do
        mkdir WTest/contents/articles/article$n
        echo "---
title: article$n
date: 2012-12-12 12:12
author: Fredrik <[email protected]>
template: article.jade
----" > WTest/contents/articles/article$n/index.md
        cat testData >> WTest/contents/articles/article$n/index.md
done

cd WTest
echo "Wintersmith tests"
for ((n=0;n<10;n++)); do /usr/bin/time -f %e wintersmith build >> ../WintersmithOut.log; done
cd ..
rm -rf WTest

Finally, this is the raw data from my runs, all measurements is in seconds, and the data used is available at this link

Jekyll tests         Hugo tests            Hugo V.0.11 Tests
159.46              63.84                       10.98
161.55              64.36                       12.57
162.12              62.52                       12.14
161.50              57.23                       11.67
160.89              57.49                       12.04
160.11              58.72                       9.91
159.94              59.74                       11.68
160.87              61.76                       10.93
160.27              62.86                       12.70
160.57              64.07                       12.72

Wintersmith tests
47.27
47.51
47.81
47.48
47.33
47.64
47.08
47.45
48.38
47.95