Recently

Watched

Reading

  • I’m slowly getting into Semiology of Graphics. It’s a special kind of read, because I know that a lot of the ideas I employ already are derived from this book, but I’ve never read it directly.
  • Winter is Coming (Probably) Soon. Something that everyone in the tech industry likely nervously read this month. The connection between general interest rates and a specific industry - and thinking about where that industry lies in the economy - is something new to me. Bad Notes on Venture Capital is less of an intro and I’m embarrassed not to follow a lot of it.
  • What We Talk About When We Talk About What We Talk About When We Talk About Making: it seems like everyone read this this month, and it really is quite good. Though I hadn’t read anything from Tim Maly prior, the style feels familiar, in a good way, to some other folks I admire. That said, I think I’ve realized why I don’t always like reading this type of article, because I can’t find the intended mood or actions or whatever thoughtpieces should invoke. On a similar level of micro/macro technology social thinking is a preliminary atlas of gizmo landscapes.
  • Zoe Quinn’s Depression Quest: #gamergate loomed large in the public consciousness this month, and I followed Zoe’s twitter with interest. There are lots of directions - understanding the way the media talks about 4chan and the difference between the ‘underbelly of the internet’ and the, usual internet. Or about what isn’t the underbelly of the internet: what separates Facebook and 4chan?

Listening

Recently

DCTN

The little local project I’ve been working on is DCTN. It’s a list of upcoming live music in Washington, DC.

Personal projects live on multiple axes: some flirt with global relevance, others personal connection. DCTN comes after projects that embraced the idea of developers for developers: mistakes.io, for instance, is only relevant to the JavaScript literate, and chartpipe to the information visualization set. Creating these things is easier in a way, because I’ve never had to research the people who would be interested in them. Imagining what other people think and feel is difficult. And it’s easy to anticipate people who are a lot like yourself.

DCTN lives a little outside that dynamic. I’ve been in the “DC Music Scene” for years, as part of Teen Mom and other bands. But I don’t go to shows all the time, or follow news outlets like Brightest Young Things. And the potential is limited: after all, DC is just one city, and the $8-12 indie band show market is barely staying afloat.

Local and Time

Incredibly local things are fascinating: in college, I built the Swem Signal, a way to share your location in the dungeon-like Swem library. It’s a crude PHP hack, built before Foursquare or Twitter were popular. But it still works, and students still use it. I’d like to think that it’s making nerds like me a little happier because they can tell their friends where to find them and to talk a little, before getting shushed.

Which connects to the other thing: time. Having worked on a lot of technology projects and observed the tech industry for years, it’s shocking how the usage curves vary - projects can maintain a flat curve of dedicated users for decades, and others can skyrocket to hundreds of thousands and then vanish into obscurity or shutdown. Different sorts of enjoyment, in both production and consumption.

React

DCTN is a React application. The Facebook-developed framework is pleasant to work with: its approach to building pages is not too many steps away from building in d3, but feels like it minimizes boilerplate. Each view can be written as a separate file and stitched together with browserify.

I have my gripes with React: building a touch application requires some sketchy hacking of the React source code to include a pre-release plugin. This will surely sort itself out, but wasn’t very obvious in documentation or code. React’s JSX is surprisingly nice - I’ll trade templates for source transforms any day of the week, especially when the JSX transform has decent error reporting. Unfortunately, JSHint doesn’t play nice with JSX, so I need to use a pre-release version and lots of jshint:ignore tags to avoid error noise in vim.

There’s plenty more to say about React, but I’ll leave that for a separate article.

The Scraper

DCTN has a server component. The core data - listings of concert times and details - comes directly from venue websites by the concerted work of scrapers. Each scraper is written in node.js and cheerio, and is a module with unit tests.

The server component is not client facing: it pushes data on a schedule to S3, which is then consumed by a static website. DCTN.org is hosted on Github Pages and requests data from an Amazon S3 endpoint. That data gets there by a server process, but never does a client request require anything new from the server.

The scraper framework, tonight-sources, is written as a node.js web app, running both an http server for status checks and a cron-like process that does the scraping. It also contains some simple metadata for each venue as GeoJSON.

tonight-sources currently scrapes:

There are plenty of other ones in the works and various stages of completion. Scrapers for U Hall & 9:30 Club are complete, but the venues apparently block my server.

Node.js for Scrapers

Despite the cheerio module module, this isn’t node.js’s brightest area:

  • Handling failures at the request, tokenize, parse, and output stages - all of which happen often - is cumbersome
  • Async i/o isn’t necessary or beneficial in this case, but sync request libraries are poorly maintained and tested

Go, with goquery, is extremely promising in this regard, and I’ve started to see nice scrapers being written in it.

Dokku

This is the first app I’ve built with Dokku. Dokku is a Heroku clone written in bash, on top of Docker. To review terms:

  • Heroku is a ‘platform as a service app’. Unlike traditional web servers, you don’t have to install node.js or PHP or such: you just push code to it, just like you’d push to Github, Heroku detects what that code requires, installs it, and runs it for you.
  • Docker is a lightweight virtual machine system. It’s a way of packaging small images of computers as portable files that you can run anywhere: you can make a Docker image that has Linux and PHP and MySQL, and run it as a server, just like you’d use VirtualBox to run Windows on your Mac. Unlike VirtualBox, Docker is simple, small, and efficient.
  • Dokku is a Heroku clone written with Docker. It connects magic detection & installation code with an interface that receives code, and then connects to Docker to run applications. It’s really neat.

Dokku means that when DCTN has more components, they can be packaged and run separately, as separate services. This is how I run tonight-tweets, which is the twitter bot that runs @dctnght. The twitter client is a separate node app that runs in a separate virtual server on the same machine. So, if one crashes, the other doesn’t, and I can update them individually. Since the datastore is separate - it’s S3 - they can be totally decoupled.

DCTN Today

DCTN is an ongoing project: it’s something that I’m building for myself, for friends, and for people in DC. Right now, it has a tiny handful of users, orders of magnitude less than ‘popular’ projects or established blogs. I Hope it’ll grow. Unlike other DC music outlets, it’s entirely automated, so day-to-day accuracy doesn’t rely on my assistance.

I’d love help - this project lives in my limited free-time and has lots of unexplored potential.

Code