Friday Link Review

Every week I come across a lot of links. (Hopefully) every Friday I'll post the best for you. Enjoy!

Using jq to Import JSON into Elasticsearch

Elasticsearch works really well with JSON. However, you can't just ship plain 'ol JSON, you have to use the bulk API. You could write a script in your favorite program language to do this, but you can use the incredibly-awesome jq tool piped to curl on the command line.

First, make sure you have jq installed. On a Mac, its a quick brew install jq away.

Then, do something like this:

cat file.json | jq -c '.[] | {"index": {"_index": "bookmarks", "_type": "bookmark", "_id": .id}}, .' | curl -XPOST localhost:9200/_bulk --data-binary @-

We're taking the file file.json and piping its contents to jq first with the -c flag to construct compact output. Here's the nugget: We're taking advantage of the fact that jq can construct not only one but multiple objects per line of input. For each line, we're creating the control JSON Elasticsearch needs (with the ID from our original object) and creating a second line that is just our original JSON object (.).

At this point we have our JSON formatted the way Elasticsearch's bulk API expects it, so we just pipe it to curl which POSTs it to Elasticsearch!

Screenshot Script

I take a lot of screenshots, often wanting to share them with someone or post on this blog. I used to have a script that would automatically upload to Flickr, but I wanted to update it a bit to use S3. Here's what I came up with:

PATH=/usr/local/bin:$PATH
DATE=$(date '+%Y%m%d-%H%M%S')
FILE=~/Dropbox/Screenshots/${DATE}.png
S3_BUCKET="assets.kevinmarsh.com"
S3_URL_PREFIX="http://d1j4kwyjzsqmb8.cloudfront.net"

screencapture -i $FILE

if [ -e $FILE ]; then
  optipng -o5 $FILE
  s3cmd put -P $FILE s3://$S3_BUCKET/
  echo "$S3_URL_PREFIX/$DATE.png" | pbcopy
  osascript -e 'display notification "URL Copied to Clipboard" with title "Screenshot"'
fi

The script is triggered with โ‡งโŒ˜5 via an Alfred workflow, then does the following:

  1. Invokes screencapture allowing me to either drag a selection or press Space and enter window selection mode
  2. Optimizes the resulting PNG. This is the slowest step, taking a few seconds, but typically shaves off about 40% of the filesize. (Will PNG writers ever to this automatically?!)
  3. Uploads to S3
  4. Copies the CloudFront URL to the pasteboard
  5. Pops up a notification telling me its done

It's almost perfect. I wish there was a way to have some sort of progress indicator, but I don't think that's possible with Notification Center.