The little local project I’ve been working on is DCTN. It’s a list of upcoming live music in Washington, DC.
Personal projects live on multiple axes: some flirt with global relevance,
others personal connection. DCTN comes after projects that embraced the idea of
developers for developers: mistakes.io, for instance, is only relevant to the
these things is easier in a way, because I’ve never had to research the people who
would be interested in them. Imagining what other people think and feel is difficult.
And it’s easy to anticipate people who are a lot like yourself.
DCTN lives a little outside that dynamic. I’ve been in the “DC Music Scene”
for years, as part of Teen Mom and other bands. But I don’t go to shows all
the time, or follow news outlets like Brightest Young Things. And the potential
is limited: after all, DC is just one city, and the $8-12 indie band show
market is barely staying afloat.
Local and Time
Incredibly local things are fascinating: in college, I built the
Swem Signal, a way to share your location
in the dungeon-like Swem library. It’s a crude PHP hack,
built before Foursquare or Twitter were popular. But it still works, and students
still use it. I’d like to think that it’s making nerds like me a little happier
because they can tell their friends where to find them and to talk a little,
before getting shushed.
Which connects to the other thing: time. Having worked on a lot of technology
projects and observed the tech industry for years, it’s shocking how the usage
curves vary - projects can maintain a flat curve of dedicated users for decades,
and others can skyrocket to hundreds of thousands and then vanish into obscurity
or shutdown. Different sorts of enjoyment, in both production and consumption.
DCTN is a React application. The Facebook-developed
framework is pleasant to work with: its approach to building pages is
not too many steps away from building in d3, but feels like it minimizes boilerplate.
Each view can be written as a separate file and stitched
together with browserify.
I have my gripes with React: building a touch application requires some sketchy
hacking of the React source code to include a pre-release plugin. This will
surely sort itself out, but wasn’t very obvious in documentation or code.
React’s JSX is
surprisingly nice - I’ll trade templates for source transforms any day of the week,
especially when the JSX transform has decent error reporting. Unfortunately,
JSHint doesn’t play nice with JSX, so I need to use a pre-release version
and lots of
jshint:ignore tags to avoid error noise in vim.
There’s plenty more to say about React, but I’ll leave that for a separate
DCTN has a server component. The core data - listings of concert times and
details - comes directly from venue websites by the concerted work of scrapers.
Each scraper is written in node.js and cheerio, and is a module with unit tests.
The server component is not client facing: it pushes data on a schedule to S3,
which is then consumed by a static website. DCTN.org
is hosted on Github Pages and
requests data from an Amazon S3 endpoint. That data
gets there by a server process, but never does a client request require anything
new from the server.
The scraper framework,
is written as a node.js web app, running both an http server for status checks
and a cron-like process that does the scraping. It also contains some
simple metadata for each venue
tonight-sources currently scrapes:
There are plenty of other ones in the works
and various stages of completion. Scrapers for U Hall & 9:30 Club are complete,
but the venues apparently block my server.
Node.js for Scrapers
Despite the cheerio module module,
this isn’t node.js’s brightest area:
- Handling failures at the request, tokenize, parse, and output stages -
all of which happen often - is cumbersome
- Async i/o isn’t necessary or beneficial in this case, but sync request
libraries are poorly maintained and tested
Go, with goquery, is extremely
promising in this regard, and I’ve started to see nice scrapers being
written in it.
This is the first app I’ve built with Dokku.
Dokku is a Heroku
clone written in bash,
on top of Docker. To review terms:
- Heroku is a ‘platform as a service app’. Unlike traditional web servers,
you don’t have to install node.js or PHP or such: you just push code to it,
just like you’d push to Github, Heroku detects what that code requires,
installs it, and runs it for you.
- Docker is a lightweight virtual machine system. It’s a way of packaging
small images of computers as portable files that you can run anywhere: you
can make a Docker image that has Linux and PHP and MySQL, and run it as a
server, just like you’d use VirtualBox to run Windows on your Mac. Unlike
VirtualBox, Docker is simple, small, and efficient.
- Dokku is a Heroku clone written with Docker. It connects magic detection
& installation code with an interface that receives code, and then connects
to Docker to run applications. It’s really neat.
Dokku means that when DCTN has more components, they can be packaged and run
separately, as separate services. This is how I run
which is the twitter bot that runs @dctnght. The
twitter client is a separate node app
that runs in a separate virtual server on the same machine. So, if one crashes,
the other doesn’t, and I can update them individually. Since the datastore is
separate - it’s S3 - they can be totally decoupled.
DCTN is an ongoing project: it’s something that I’m building for myself, for
friends, and for people in DC. Right now, it has a tiny handful of users, orders
of magnitude less than ‘popular’ projects or established blogs. I Hope it’ll
grow. Unlike other DC music outlets, it’s entirely automated,
so day-to-day accuracy doesn’t rely on my assistance.
I’d love help - this project lives in my limited free-time and has lots of