Data-driven content with GoHugo.io

Update: 8. Feb. 2015 See the Notes section at the end.

Update: 27. Dec 2014

What if you want to create simple pseudo dynamic content within a page or a layout with Hugo, the static site generator?

My idea is: Import any JSON or CSV from any local file or URL and make the JSON or CSV content available in a shortcode or directly in the layout files.

The Go code is currently in my template_resources.go.

How to implement?

Local JSON or CSV files must reside inside Hugos working directory.

As an example I’m using the JSON from my GitHub Stars.

In your markdown template you can e.g. add a short code like:

{{< demoJsonGH url="static/wp-content/uploads/hugo/starred.json" >}}
{{< demoJsonYT url="http://gdata.youtube.com/feeds/users/useryt/uploads?alt=json&max-results=10" >}}

The demoJsonGH short code template is:

<ul class="pinglist">
  {{ $url := .Get "url" }}
  {{ range getJSON $url }}
    {{ $p := . }}
    <li>
      {{$p.language}}: <strong>{{ $p.name }}</strong>
      <a href="{{ $p.html_url }}" target="_blank">{{ $p.full_name }}</a>
      <br>
      Stars: {{$p.stargazers_count}} | Watchers: {{$p.watchers_count}}<br>
      {{ $p.description }}
    </li>
    {{ end }}
</ul>

Parsing JSON results

  • Go: dl nf/dl
    Stars: 54 | Watchers: 54
  • Go: aws-go stripe/aws-go
    Stars: 641 | Watchers: 641
    An incredibly experimental, automatically generated set of AWS clients in Go.
  • Go: lane oleiade/lane
    Stars: 77 | Watchers: 77
    A golang queues, stacks and deques implementation library
  • Go: go golang/go
    Stars: 4831 | Watchers: 4831
    The Go programming language
  • Go: goji zenazn/goji
    Stars: 1857 | Watchers: 1857
    Goji is a minimalistic web framework for Golang that's high in antioxidants
  • Go: gox mitchellh/gox
    Stars: 774 | Watchers: 774
    A dead simple, no frills Go cross compile tool
  • Go: ioprogress mitchellh/ioprogress
    Stars: 235 | Watchers: 235
    Go (golang) package for progress bars around io.Reader/Writers.
  • PHP: mpmd AOEpeople/mpmd
    Stars: 50 | Watchers: 50
    Magento Project Mess Detector (for n98-magerun)
  • Go: gometry rakyll/gometry
    Stars: 228 | Watchers: 228
    A visual interface to work with runtime profiling data from Go programs (work-in-progress)
  • Objective-C: CotEditor coteditor/CotEditor
    Stars: 1014 | Watchers: 1014
    Lightweight Plain-Text Editor for OS X
  • Go: httprouter julienschmidt/httprouter
    Stars: 1041 | Watchers: 1041
    A high performance HTTP request router that scales well
  • Go: http2 bradfitz/http2
    Stars: 474 | Watchers: 474
    HTTP/2 support for Go (in active development)
  • Go: robotstxt-go temoto/robotstxt-go
    Stars: 62 | Watchers: 62
    The robots.txt exclusion protocol implementation for Go language #golang
  • Go: vegeta tsenart/vegeta
    Stars: 2025 | Watchers: 2025
    HTTP load testing tool and library. It's over 9000!
  • Go: cron robfig/cron
    Stars: 325 | Watchers: 325
    a cron library for go
  • Go: grace facebookgo/grace
    Stars: 515 | Watchers: 515
    Graceful restart for Go servers.
  • Go: mysqltest facebookgo/mysqltest
    Stars: 17 | Watchers: 17
    Package mysqltest provides standalone instances of mysql suitable for use in tests.
  • Go: ensure facebookgo/ensure
    Stars: 18 | Watchers: 18
    Package ensure provides utilities for testing to ensure the given conditions are met and Fatal if they aren't satisified.
  • Go: cockroach cockroachdb/cockroach
    Stars: 2556 | Watchers: 2556
    A Scalable, Geo-Replicated, Transactional Datastore
  • Shell: compressed-magento-sample-data Vinai/compressed-magento-sample-data
    Stars: 14 | Watchers: 14
    A highly compressed version of the magento 1.9 sample data and a script to create it.
  • Go: delve derekparker/delve
    Stars: 1266 | Watchers: 1266
    Go debugger
  • Go: ngrok inconshreveable/ngrok
    Stars: 3863 | Watchers: 3863
    Introspected tunnels to localhost
  • JavaScript: dashboards keen/dashboards
    Stars: 4641 | Watchers: 4641
    Responsive dashboard templates for Bootstrap
  • Go: sql-migrate rubenv/sql-migrate
    Stars: 206 | Watchers: 206
    SQL schema migration tool for Go.
  • Go: boom rakyll/boom
    Stars: 1703 | Watchers: 1703
    HTTP(S) load generator, ApacheBench (ab) replacement, written in Go
  • Go: pb cheggaaa/pb
    Stars: 380 | Watchers: 380
    Console progress bar for Golang
  • Go: viper spf13/viper
    Stars: 193 | Watchers: 193
    Go configuration with fangs
  • Go: go-spew davecgh/go-spew
    Stars: 383 | Watchers: 383
    Implements a deep pretty printer for Go data structures to aid in debugging

Parsing YouTube feed url:

  • map[$t:Carols in the Domain, Sydney 2014 type:text]
  • map[$t:Blow hole at Pancake Rocks, Punakaiki, New Zealand type:text]
  • map[$t:Blue Mountains Blackheath Grand Canyon Track type:text]
  • map[$t:Sculptures by the sea 2014 type:text]
  • map[type:text $t:Blue Mountains View @ Wentworth Falls]
  • map[$t:Australian Reptile Park: Milking a Sydney Funnel Web Spider type:text]
  • map[$t:Australian Reptile Zoo: Milking a Costal Taipan type:text]
  • map[type:text $t:Australian Reptile Zoo: Koalas]
  • map[$t:Australian Reptile Zoo: Tasmanian Devil feeding an egg type:text]

One strange problem occurs when parsing the Youtube API 2.0 (deprecated):

The JSON response looks something like:

  "entry": [
    {
      "id": {"$t": "http://gdata.youtube.com/feeds/videos/XeceUDREW0U"},
      "published": {"$t": "2014-12-20T22:35:14.000Z"},
      "updated": {"$t": "2014-12-20T22:37:03.000Z"},
      "category": [
        {
          "scheme": "http://schemas.google.com/g/2005#kind",
          "term": "http://gdata.youtube.com/schemas/2007#video"
        },
        {
          "scheme": "http://gdata.youtube.com/schemas/2007/categories.cat",
          "term": "Travel",
          "label": "Travel & Events"
        }
      ],
      "title": {
        "$t": "Carols in the Domain, Sydney 2014",
        "type": "text"
      },

The YouTube short code template is:

<ul class="pinglist">
  {{ $url := .Get "url" }}
  {{ $j := getJSON $url }}

  {{ range $j.feed.entry }}
    {{ $v := . }}
    <li>
      {{index $v.title }}<br>
    </li>
  {{ end }}
</ul>

Hugo totally panics and crashes when using: {{$v.title.$t}}.

The best solution should be switching to Youtube API v3.

Or any other ideas?

Parsing CSV

The short code within your page for the CSV looks like:

{{< demoCsv url="static/wp-content/uploads/hugo/SalesJan2009.csv" sep="," >}}

The url can be a local or a remote resource. Sep is the CSV separator which can only be one character long. There is currently no possibility to provide a line separator (Default: \r\n or \n; \r does not work.).

The html of the demoCsv short code displays:

<table border="1">
  {{ $url := .Get "url" }}
  {{ $sep := .Get "sep" }}
  {{ range $i, $r := getCSV $sep $url }}

    {{ if eq $i 0 }}
      <thead>
        <tr>
          {{ range $r }}
            <th>{{ . }}</th>
          {{end}}
        </tr>
      </thead>
    {{else}}
      <tbody>
        <tr>
          {{ range  $r }}
            <td>{{ . }}</td>
          {{end}}
        </tr>
      </tbody>
    {{end}}

  {{ end }}
</table>

The final result:

Transaction_date Product Price Payment_Type Name City State Country Account_Created Last_Login Latitude Longitude
1/2/09 6:17 Product1 1200 Mastercard carolina Basildon England United Kingdom 1/2/09 6:00 1/2/09 6:08 51.5 -1.1166667
1/2/09 4:53 Product1 1100 Visa Betina Parkville MO United States 1/2/09 4:42 1/2/09 7:49 39.195 -94.68194
1/2/09 13:08 Product1 1230 Mastercard Federica e Andrea Astoria OR United States 1/1/09 16:21 1/3/09 12:32 46.18806 -123.83
1/3/09 14:44 Product1 1200 Visa Gouya Echuca Victoria Australia 9/25/05 21:13 1/3/09 14:22 -36.1333333 144.75
1/4/09 12:56 Product2 3600 Visa Gerd W Cahaba Heights AL United States 11/15/08 15:47 1/4/09 12:45 33.52056 -86.8025
1/4/09 13:19 Product1 1200 Visa LAURENCE Mickleton NJ United States 9/24/08 15:19 1/4/09 13:04 39.79 -75.23806
1/4/09 20:11 Product1 1200 Mastercard Fleur Peoria IL United States 1/3/09 9:38 1/4/09 19:45 40.69361 -89.58889
1/5/09 20:09 Product1 1200 Mastercard adam Martin TN United States 1/2/09 17:43 1/4/09 20:01 36.34333 -88.85028

Integrating into layout HTML files

An example on how I have integrated my GitHub Gists into the left sidebar can be found here: layouts/partials/sidebarLeftCategories.html#L29

Notes

Downloaded remote files will be cached in --cacheDir. The default cacheDir is set to $TMPDIR/hugo_cache/ The only cache invalidation method is left to the user: rm *. Downloaded files are always cached.

Update 8. Feb. 2015:

The parameter --ignoreCache has been added to ignore the read from the cache but writing to the cache is still happening.

getJSON and getCSV are now variadic functions. You can submit multiple parts of an URL which will be joined to the final URL. Example:

{{ $id := .Params._id }}
{{ $url_pre :=  "http://localhost:3000/db/persons/" }}
{{ $url_post := "/limit/10/skip/0" }}
{{ $gistJ := getJSON $url_pre $id $url_post }}

For getCSV the separator argument has been moved to the beginning of the function.

Futures Features

Once I have more time I would like to implement also an RSS reader and advanced authentication methods.

A far future feature would be to generated a whole category or category tree with n documents from a JSON file. That feature would allow to import categories and products from Magento or any other system.

Update 29. Jan. 2016:

Renamed the topic to Data-driven content because better wording :-)

Related posts

comments powered by Disqus