Hugo, another performance perspective

During an update to my previous post regarding performance in static site generators I discovered a major performance difference between Hugo v.0.11 and v.0.12 putting the latest Hugo version at the same build times as jekyll.

This require further testing since one of the main reasons for me to use Hugo is build speed and since Hugo just released v.0.13 I decided it was a good time to do another performance test, this time HugoVHugoVHugo as odd as it sounds. I will compare the three latest versions of Hugo and try to determine if v.0.12 was just a momentary lapse in reason or if shiny new features have been allowed to shift focus from performance.

These tests will basically be done as before, with the small difference that I due to an issue with v.0.11 have to use an older version of the chosen theme (herring-cove).

Method

The test focuses entirely on build performance, I will therefore create a large (1000 post) site where each post consists of a relatively large amount of random text (348Kb). The site will then be built 10 times by each version of hugo with execution time captured by time, finally the average execution time is calculated from the results and the comparison can begin.

After this I will run the same process for sites containing 10 and 100 posts, this is done to get a good idea of both the time required for each post and if there have been any improvements in either small site performance or large site performance between the versions.

A full list of raw data and the commands used is available at the end of this post.

To keep the results somewhat consistent with the results from my previous post I am once again using an Azure A3 Basic linux vm with running a clean install of Ubuntu server.

Results

As you can see in the following charts the build performance have dropped considerably from V.0.11 to V.0.13. This drop in performance is probably caused by the rather massive increase in features between the versions. However for a basic site like this which does not utilize all of the new features the performance drops is worrying.

The build time per post showed in the chart below does offer some relief even though the build times have gone up from v.0.11 we can see that the latest release offers some improvements for smaller sites. This is probably caused by improvements in the rendering engine creating lower overhead.

Raw data and commands

The following commands was used to create the site. The test data is available at this link, and the raw output from the tests is available at this link.

# Folder creation
mkdir HugoTest
cd HugoTest
# Download hugo binaries
wget https://github.com/spf13/hugo/releases/download/v0.11/hugo_0.11_linux_amd64.tar.gz
tar -xvf hugo_0.11_linux_amd64.tar.gz
mv hugo_0.11_linux_amd64/hugo_0.11_linux_amd64 hugo11
wget https://github.com/spf13/hugo/releases/download/v0.12/hugo_0.12_linux_amd64.tar.gz
tar -xvf hugo_0.12_linux_amd64.tar.gz
mv hugo_0.12_linux_amd64/hugo_0.12_linux_amd64 hugo12
wget https://github.com/spf13/hugo/releases/download/v0.13/hugo_0.13_linux_amd64.tar.gz
tar -xvf hugo_0.13_linux_amd64.tar.gz
mv hugo_0.13_linux_amd64/hugo_0.13_linux_amd64 hugo13
rm -rf *.tar.gz

# Create site
./hugo11 new site HTest

# Move binaries to site
mv hugo* HTest/

cd HTest
## Get correct version of theme
git clone https://github.com/spf13/herring-cove.git themes/herring-cove
cd themes/herring-cove
git checkout 72a84112f063c36a70c6b0e1d9427e9f5807da39
cd ../../
# Download test data
wget https://s3-eu-west-1.amazonaws.com/imagesfredrikme/testData
# Create posts
for ((n=0;n<1000;n++)); do ./hugo11 new post/test_$n.md;cat testData >> content/post/test_$n.md;done;

# Run tests pipe hugo output to log
echo "V.0.11"; for ((n=0;n<10;n++)); do /usr/bin/time -f %e ./hugo11 -D -t herring-cove >> log1; done;echo "V.0.12"; for ((n=0;n<10;n++)); do /usr/bin/time -f %e ./hugo12 -D -t herring-cove >> log2; done;echo "V.0.13"; for ((n=0;n<10;n++)); do /usr/bin/time -f %e ./hugo13 -D -t herring-cove >> log3; done;

# Redo from Create site for each of the post counts