Jekyll Blogging Workflow: Link Posts

I wanted a way to write blog posts digesting the links I come across on a regular basis, much in the style of Waxy.org/links, Mike Gunderloy's A Fresh Cup, and numerous others that I follow.

I use Pinboard to collect links throughout each day. It has a really simple interface, can be posted to from all kinds of apps I use everyday (Reeder, TweetBot, etc.), and is run by a really cool dude.

Naturally I wanted something that could read my Pinboard bookmarks and automatically post to the blog. However, since I'm using GitHub Pages it can't quite be as automated as a new post everytime I bookmark because I have to create a Markdown file and then push it to GitHub to publish. This is OK, since a digest is probably a better format anyway.

Luckily Pinboard offers feeds of your bookmarks in various formats. We can combine these feeds with a little Ruby magic I whipped up I call pinboard2md:

#!/usr/bin/env ruby
require 'nokogiri'

doc = Nokogiri::XML(ARGF)

doc.search('post').each do |post|
  url = post.attr('href')
  title = post.attr('description')
  description = post.attr('extended')

  output = "* [#{title}](#{url})"
  output << " - #{description}" if description.to_s.strip != ""
  puts output
end

I run it from the command line with:

curl "https://foo:[MY_PINBOARD_PASSWORD]@api.pinboard.in/v1/posts/all?fromdt=2014-10-17" | pinboard2md

It grabs the feed via curl (with some additional options to only fetch links I posted within the last week) and pipes to our Ruby script that does some simple parsing and converts it to a bulleted list of links in Markdown format that I can copy and paste into a new post or just pipe to a new file. It's nice because it gives me a bit of time to truly "digest" the links and add additional commentary, although any descriptive text that I entered into Pinboard when I bookmarked will natually be included there as well.

This is a good example of why a Jekyll static page blog is so appealing to developers like me and why command-line workflows should be desirable to anyone who works with text.

Friday Link Review

Every week I come across a lot of links. (Hopefully) every Friday I'll post the best for you. Enjoy!

Using jq to Import JSON into Elasticsearch

Elasticsearch works really well with JSON. However, you can't just ship plain 'ol JSON, you have to use the bulk API. You could write a script in your favorite program language to do this, but you can use the incredibly-awesome jq tool piped to curl on the command line.

First, make sure you have jq installed. On a Mac, its a quick brew install jq away.

Then, do something like this:

cat file.json | jq -c '.[] | {"index": {"_index": "bookmarks", "_type": "bookmark", "_id": .id}}, .' | curl -XPOST localhost:9200/_bulk --data-binary @-

We're taking the file file.json and piping its contents to jq first with the -c flag to construct compact output. Here's the nugget: We're taking advantage of the fact that jq can construct not only one but multiple objects per line of input. For each line, we're creating the control JSON Elasticsearch needs (with the ID from our original object) and creating a second line that is just our original JSON object (.).

At this point we have our JSON formatted the way Elasticsearch's bulk API expects it, so we just pipe it to curl which POSTs it to Elasticsearch!