As I have previously mentioned this site runs jekyll, which works with reasonable speed however my personal blog which contains a few more pictures creates a problem. The cause is mainly slow regeneration times, something that also affects this site. I believe that the main performance missuses stems from two main sources, mainly jekyll’s single threaded approach which does suffer on a laptop cpu like the one I use, another issue is with the image plugins used that offers nice features but also creates a performance loss. With this in mind I have decided to benchmark the site generation between a few static generators to evaluate the competition and possibly change engine. The static site generators I will focus on is Wintersmith, which is built on node.JS and Hugo, which is built on GO.
The three generators are similar in usage but to compare performance I will need to create sites with similar features on all three of them. To simplify performance comparison I will omit the picture_tag plugin described in a previous post, I will however still include the full scale images to determine how large resources are handled. To eliminate further contamination of third party scripts, I will use the standard themes for each of the sites. Apart from a selection of “real” posts I have included some generated posts with large amounts of text.
To get relevant results I ran 10 builds for each site and calculated the mean value for the runs, the resulting values is presented in the following chart.
There are a few things worth mentioning about the values in this chart.
- The measured times showed very small variations.
- Since jekyll don’t provide build time output I used “time” to measure jekyll build times.
- Even though Wintersmith have slow build times, the rebuilds are relatively fast, this is just based on “feeling” no measurements where done.
This is a rather basic test but my results seems logical especially since Hugo claims to focus on speed.
To further validate these results I ran test with large sites (1000 auto-generated posts) Where I basically generated posts using the following commands:
for ((n=0;n<1000;n++)); do hugo new post/test_$n \ && cat content/post/test.md >> content/post/test_$n; done for ((n=0;n<1000;n++)); do ./jekyll-post -D "testsite/_posts/" test_$n \ && cat test.md >> testsite/_posts/2014-08-12-test_$n.md; done
And for Wintersmith I basically copied the jekyll posts.
To well managed data I used the following command to run the tests:
for ((n=0;n<10;n++)); do time buildCommand &> out.log; done
Which means that I use “time” for all the tests this time. The following chart shows the results in seconds.
A complete extract of the command output is at the end of this post, where it’s worth noting that Hugo uses several cores, while jekyll and Wintersmith makes do with one. Another thing worth mentioning is that the results shown above should be considered relative to each other instead of focusing on the specific timings. However in the interest of providing data, the generated site contained 1000 posts, totaling in 77mb of text.
With these results I can basically conclude that hugo is the fastest of the three.
jekyll build &> out.log 72.41s user 0.69s system 99% cpu 1:13.37 total jekyll build &> out.log 72.02s user 0.81s system 99% cpu 1:12.93 total jekyll build &> out.log 72.02s user 0.67s system 99% cpu 1:12.77 total jekyll build &> out.log 97.32s user 1.32s system 99% cpu 1:39.08 total jekyll build &> out.log 89.43s user 1.01s system 99% cpu 1:30.78 total jekyll build &> out.log 73.79s user 0.82s system 99% cpu 1:14.72 total jekyll build &> out.log 74.58s user 0.79s system 99% cpu 1:15.47 total jekyll build &> out.log 75.81s user 0.81s system 99% cpu 1:16.79 total jekyll build &> out.log 75.60s user 1.14s system 99% cpu 1:16.91 total jekyll build &> out.log 76.62s user 0.80s system 99% cpu 1:17.66 total
hugo &> out.log 11.17s user 0.56s system 263% cpu 4.449 total hugo &> out.log 11.25s user 0.42s system 289% cpu 4.036 total hugo &> out.log 11.04s user 0.38s system 290% cpu 3.932 total hugo &> out.log 11.06s user 0.36s system 290% cpu 3.932 total hugo &> out.log 11.14s user 0.37s system 289% cpu 3.977 total hugo &> out.log 11.06s user 0.36s system 290% cpu 3.925 total hugo &> out.log 11.76s user 0.41s system 260% cpu 4.666 total hugo &> out.log 11.20s user 0.37s system 286% cpu 4.042 total hugo &> out.log 11.35s user 0.40s system 271% cpu 4.329 total hugo &> out.log 11.55s user 0.42s system 270% cpu 4.423 total
wintersmith build &> out.log 47.02s user 3.51s system 98% cpu 51.353 total wintersmith build &> out.log 41.86s user 2.23s system 101% cpu 43.516 total wintersmith build &> out.log 41.17s user 1.95s system 101% cpu 42.396 total wintersmith build &> out.log 41.40s user 2.02s system 101% cpu 42.603 total wintersmith build &> out.log 46.20s user 2.41s system 101% cpu 48.025 total wintersmith build &> out.log 40.77s user 1.89s system 101% cpu 42.094 total wintersmith build &> out.log 45.93s user 2.20s system 101% cpu 47.213 total wintersmith build &> out.log 47.15s user 2.37s system 101% cpu 48.648 total wintersmith build &> out.log 47.05s user 2.62s system 101% cpu 49.035 total wintersmith build &> out.log 48.66s user 2.81s system 101% cpu 50.829 total
As mentioned in the comments the output from my builds is somewhat unclear, to remidy this I have re-run the tests, this time in maner that’s easier to recreate.
The following test output is generated on a A3 Basic linux vm in Azure, the machine is running ubuntu 14.10, I am using the latest stable version of jekyll, hugo and wintersmith.
The last one have caused some issues, mainly caused by out of memory exceptions this have forced me to limit the amount of pages to 100, for the record though both jekyll(444s) and hugo(115s) managed to generate 1000 pages. The usage of the latest version of hugo (v.0.12) have caused some issues with performance which seems to have taken a pretty decent hit with since v.0.11, this does change the outcome of the tests considerably. I ran a quick test of hugo v0.11 and the build time difference is massive v.0.11 (12s) vs v0.12 (51s).
With the updated data I can conclude the following
- Hugo v.0.11 is still the fastest, by a wide margin
- Hugo v.0.12 is slightly slower than wintersmith, however wintersmith does not handle massive sites (500+ posts) at all
- Jekyll is still the slowest but have a massive community which is a plus
To run the tests I have used the following bash script, it assumes that all “generators” is available in path.
#!/bin/bash rm -rf HTest JTest WTest jekyll-post testData wget https://s3-eu-west-1.amazonaws.com/imagesfredrikme/testData echo "Creating jekyll site" wget https://raw.githubusercontent.com/jbranchaud/mybin/master/jekyll-post chmod 755 jekyll-post jekyll new JTest for ((n=0;n<100;n++)) do ./jekyll-post -D "JTest/_posts" test$n > /dev/null done for f in JTest/_posts/*.md do cat testData >> $f done cd JTest echo "Jekyll tests" for ((n=0;n<10;n++)); do /usr/bin/time -f %e jekyll build >> ../JekyllOut.log; done cd .. rm -rf JTest echo "Creating hugo site" hugo new site HTest cd HTest for ((n=0;n<100;n++)) do hugo new post/test_$n.md > /dev/null cat ../testData >> content/post/test_$n.md done git clone https://github.com/spf13/hyde.git themes/hyde cd .. cd HTest echo "Hugo tests" for ((n=0;n<10;n++)); do /usr/bin/time -f %e hugo -t hyde >> ../HugoOut.log; done cd .. rm -rf HTest echo "Creating winthermsith site" wintersmith new WTest for ((n=0;n<100;n++)) do mkdir WTest/contents/articles/article$n echo "--- title: article$n date: 2012-12-12 12:12 author: Fredrik <[email protected]> template: article.jade ----" > WTest/contents/articles/article$n/index.md cat testData >> WTest/contents/articles/article$n/index.md done cd WTest echo "Wintersmith tests" for ((n=0;n<10;n++)); do /usr/bin/time -f %e wintersmith build >> ../WintersmithOut.log; done cd .. rm -rf WTest
Finally, this is the raw data from my runs, all measurements is in seconds, and the data used is available at this link
Jekyll tests Hugo tests Hugo V.0.11 Tests 159.46 63.84 10.98 161.55 64.36 12.57 162.12 62.52 12.14 161.50 57.23 11.67 160.89 57.49 12.04 160.11 58.72 9.91 159.94 59.74 11.68 160.87 61.76 10.93 160.27 62.86 12.70 160.57 64.07 12.72 Wintersmith tests 47.27 47.51 47.81 47.48 47.33 47.64 47.08 47.45 48.38 47.95