Canvas animations on maps

Earlier this month, I discussed techniques for animation with canvas, sketching out how you can simulate motion and produce graphics, videos, and GIFs. You can draw anything with Canvas's primitives - lines, polygons, images, text - but I didn't explain how to do maps. Maps are just glorified charts, but you'll need the right tools and concepts to create them in Canvas. Let's see how.

The literal projection

What cartographers term the equirectangular projection is the most literal transform from coordinates as we typically record them to an image. Longitude and latitude are numbers from -180 to 180 and -90 to 90 respectively, and images are plotted as pixels from 0 to width and 0 to height.

The equirectangular projection requires only arithmetic: to project longitude and latitude values onto a 360x180 canvas, you would write

// longitude and latitude onto a 360x180 canvas
var x = longitude + 180;
var y = 90 - latitude;

// scaling them up to fit a 640x360 canvas
var x = (longitude + 180) * 2;
var y = (90 - latitude) * 2;

// to pan the map, you can add to one of the numbers
var x = (longitude + 180 + 50) * 2;
var y = (90 - latitude) * 2;

The equirectangular projection doesn't have much cartographic value: mostly it's popular because it's incredibly simple. This is how I created a video of the centers of all OpenStreetMap edits:

You might notice that that video starts from nothing: there's no basemap under the points. This is because most maps you'll see on the internet aren't in the equirectangular projection, so I don't have many go-to sources for tiles. On the satellite side, you can find plenty of NASA composite images like Blue Marble that are in equirectangular, but Mapbox, OpenStreetMap, and the vast majority of other mapping services are available only in the Spherical Mercator projection.

Spherical mercator

To get more context in your canvas geo-visualizations, you'll want an actual map behind it. Let's review the tools of the trade.

  • node-sphericalmercator implements the transformation between spherical mercator and WGS84 (longitude & latitude)
  • geo-viewport transforms bounding boxes into zoom + centerpoint combinations

For Python fans, mercantile solves the same problems as node-sphericalmercator.

For projections in general, proj4 and proj4js are the comprehensive solutions and d3 projections are also very strong. But the transform from spherical mercator to and from longitude & longitude is one of the simplest possible, so using a lightweight, purpose-built tool like node-sphericalmercator makes a lot of sense.

  1. We'll need a dataset and a basemap
  2. To frame the visualization, choose a geograhical extent and image size
  3. Then decide how the animation will look across time

I'll use the NYC Taxi Trips dataset famously liberated by Chris Whong.

To determine the desired geographical extent, you can either find the actual extent of the data, or choose one yourself. I tend to do the latter, since most datasets have outliers - one taxi that drives out to New Jersey shouldn't mean that your map should be state-level rather than city-level.

Okay, so the taxi dataset is a really big CSV file. I'll be doing this in node as usual. First thing is to set up CSV parsing and file reading: with a file this large, use a streaming parser so that you never hold gigabytes of data in memory.

Make a directory to play around in:

mkdir taxi-map
cd taxi-map

Initialize a package.json file. This makes your experiment replicable: with the dependencies recorded in this file, others can easily run npm install and get the same software you used.

npm init .

Install the csv-parser module. There are lots of fancy CSV parsers: this is a pretty good one.

npm install --save csv-parser

Then start writing draw.js:

var fs = require('fs'),
  csv = require('csv-parser');

fs.createReadStream('trip_data_1.csv')
  .pipe(csv())
  .on('data', function(data) {
    console.log('row', data);
  });

Run node draw.js to make sure that the CSV is being parsed properly. A row looks like:

{ medallion: '1CF8717030F447204CCBD67F812CD426',
  hack_license: '5292D80EDB27399CEDA641A9241593DF',
  vendor_id: 'VTS',
  rate_code: '1',
  store_and_fwd_flag: '',
  pickup_datetime: '2013-01-13 10:38:00',
  dropoff_datetime: '2013-01-13 10:39:00',
  passenger_count: '1',
  trip_time_in_secs: '60',
  trip_distance: '.26',
  pickup_longitude: '-74.002815',
  pickup_latitude: '40.749241',
  dropoff_longitude: '-74.002258',
  dropoff_latitude: '40.751831' }

Keeping this simple, we'll only pay attention to the pickup location, at least at first. The 's around it mean that it's being parsed as a string. So we'll use parseFloat to make it a number before doing anything else.

Using a static map for a background

Let's pull up and get that extent in place. I use geojson.io for this sort of thing: zoom into the map and draw a box around New York. Then go to Meta -> Add bboxes to add a bbox property to each feature. For the box I drew, that adds:

"bbox": [
  -74.09111022949219,
  40.60456943720527,
  -73.76014709472656,
  40.8626410807892
]

geo-viewport gives you the magic code to figure out, for a bounding box and desired pixel bounding box, what zoom level & centerpoint will properly contain the bounding box.

Install geo-viewport

npm install --save geo-viewport

And then create a file download_map.js:

var geoViewport = require('geo-viewport');

console.log(geoViewport.viewport([
  -74.09111022949219,
  40.60456943720527,
  -73.76014709472656,
  40.8626410807892
], [640, 640]));

This outputs

{ center: [ -73.92562866210938, 40.73360525899724 ], zoom: 11 }

Okay: let's get a map there. I'll use the Mapbox Static API for this. Since I'm only doing it once, I'll use Katy Decorah's Static Map Maker and plug in the numbers from above.

© Mapbox © OpenStreetMap

Since this map includes OpenStreetMap data and Mapbox design, it'll need that © Mapbox © OpenStreetMap attribution anywhere it goes.

So I'll download the static map and call it background.png. Let's start drawing points.

Projecting points on a static map

First we'll just get this background.png image to pass through node-canvas, the library that I covered in the last post about animation.

Install node-canvas:

npm install --save canvas

And comment out the main bit of the draw.js script, but start initializing a canvas that's 640x640 to fit the image, draw the image on it, and then save it to a file.

var fs = require('fs'),
  Canvas = require('canvas');

var canvas = new Canvas(640, 640);
var ctx = canvas.getContext('2d');

var background = new Canvas.Image();
background.src = fs.readFileSync('./background.png');
ctx.drawImage(background, 0, 0);

fs.writeFileSync('frame.png', canvas.toBuffer());

Now it's time to combine the background layer with our datasource. At this point you know there's a 640x640 image with a zoom level and centerpoint: this is enough to calculate where a given longitude, latitude point should fall.

This is where node-sphericalmercator comes in: the function you'll use is called px: it translates longitude, latitude, and zoom into pixel coordinates: x & y.

Maps are often thought of in tile coordinates, rather than pixel coordinates. The two are basically interchangeable: tile coordinates are pixel coordinates divided by 256, and vice-versa.

Install sphericalmercator: npm install sphericalmercator.

Create a new projection instance:

var projection = new (require('sphericalmercator'))();

So, for instance: at zoom level 0, 0°, 0° is halfway in the middle of a single tile of 256x256px. -180°, 85° is in the top-left corner and 180°, -85° is in the bottom-left.

> projection.px([0, 0], 0)
[ 128, 128 ]
> projection.px([-180, 85], 0)
[ 0, 0 ]
> projection.px([180, -85], 0)
[ 256, 256 ]

So we have a projection, a static map, and a bunch of data at different longitude, latitude positions. To figure out where in the image to draw the data, let's figure out the pixel location of its center, and then the pixel location of its top-left corner, which we'll call the 'origin', and then for each data point we can find their pixel locations, subtract the origin, and have a proper relative value.

var center = projection.px([
  -73.92562866210938,
  40.73360525899724], 11);

// The image is 640x640, so to get from the center
// to the top-left, we go up by half the width & height
var origin = [
  center[0] - 320,
  center[1] - 320
];

Now that we have the origin value, we can create a function that gives pixel values relative to the static map, rather than the top-left corner of the entire world.

function positionToPixel(coord) {
  var px = projection.px(coord, 11);
  return [
    px[0] - origin[0],
    px[1] - origin[1]
  ];
}

positionToPixel will transform longitude, latitude points to pixel coordinates used in node-canvas. Consult the previous post for the details of the leftpad module: let's just dive into animation.

ctx.fillStyle = '#84014b';
var frame = 0, perFrame = 100;
fs.createReadStream('trip_data_1.csv')
  .pipe(csv())
  .on('data', function(data) {
    var pixel = positionToPixel([
      parseFloat(data.pickup_longitude),
      parseFloat(data.pickup_latitude)]);
    ctx.fillRect(pixel[0], pixel[1], 2, 2);
    frame++;
    if (frame > 0 && frame % perFrame === 0) {
      // draw an image every time we've drawn 100 points.
      fs.writeFileSync('frames/' + leftpad(frame / perFrame, 5) + '.png',
        canvas.toBuffer());
      // and then draw, at a low opacity, the original static
      // map over it. this makes the points fade out.
      ctx.globalAlpha = 0.4;
      ctx.drawImage(background, 0, 0);
      ctx.globalAlpha = 1;
    }
  });

Run this for a while, and frames will be generated in the frames directory, and then use ffpmeg to combine them into a video, and you'll get:

Onwards

The full source code for this example is on GitHub: feel free to use it and the concepts introduced in any way you'd like.

Practical ramda - functional programming examples

Ramda is a second-generation JavaScript library for functional programming. We're using it in Mapbox Studio to great effect, and given that functional programming libraries are often documented with mathematical or abstract examples, I think it would be useful to give some examples of how.

Second generation

So Ramda is a second generation functional programming library.

Arguably the first generation was underscore and lodash. In the era when Internet Explorer 8 was still a consideration in JavaScript programming, underscore had a clear purpose: to make JavaScript's functional methods cross-browser compatible. JavaScript has great, built-in functionality for functional iteration, mapping, and reduction, but these methods, like Array.map and Array.reduce, have been slow to arrive in Internet Explorer. underscore gave the world reliable _.map and _.reduce methods - along with many others - that worked the same in every browser.

In the time since, Internet Explorer 8 has faded into the past and native ES5 methods are expected for most applications. So the role of functional programming libraries has shifted from a compatibility layer to adding new, foundational methods for functional programming. In Mapbox Studio, we started with 101, a library that is "maintained to minimize overlap with vanilla JS." 101 provides a lot of great functionality, especially with support of keypaths throughout, but we found that composing 101's methods was a bit lacking.

Ramda is a library built for composition and higher-level functional programming. It tends to flip method arguments, so instead of _.map([], function() {}), ramda exposes R.map(function() {}, []), which more often makes it possible to reuse the map function. And functions in ramda are curried by default, so the R.map method could be called like R.map(function() {}, []) or R.map(function() {})([]) interchangeably.

The third generation of functional programming libraries in JavaScript is probably something like trine, which uses future syntax to write functional application in a chaining style. But ES6 is already a bit of a gamble, so Mapbox Studio isn't ready to heavily use a strawman spec.

Currying

One of the major strengths of ramda is its approach to currying methods. Ramda's curried methods mean that a method that looks like

foo(1, 2, 3);

Can be called with each argument in steps:

foo(1)(2)(3);
// which means
var curried = foo(1);
var alsoCurried = curried(2);
var finallyEvaluated = alsoCurried(3);

This is really nice in the case of a method like Array.map because you can partially apply a method and use it as a method in Array.map. "Partially applying the method" means you give it fewer than the full set of required arguments and supply the rest of the arguments later.

For instance, let's take an array of objects with an 'id' parameter:

var data = [{ id: 'foo' }, { id: 'bar' }];

The R.prop method will give you the value of a property of a single object:

var id = R.prop('id', { id: 'foo' }); // returns 'foo'

But what about an array? The desired end result is ['foo', 'bar']. It would be great if we could just use R.prop, but have it apply against an array. Luckily, thanks to currying, we can:

var ids = data.map(R.prop('id'));

You provide the method R.prop('id') as the argument to .map: it is a function that, given an object, returns its id property.

Enough talk. Let's see some code.


Problem: given an array of objects called layers, create an object called layerMap that has the 'id' property of a layer as a key and the layer itself as a value.

Solution:

var layerMap = R.zipObj(R.pluck('id', layers), layers);

Problem: you want to dynamically load a JavaScript library to power a function. In this case, we want to load the AWS SDK when someone wants to upload a file. But we want to make sure that if the method is called many times in a row, we don't load the library as many times.

Solution: the R.once method ensures the function is only evaluated once, and on successive calls returns the same value.

var getAWS = R.once(function() {
  return new Promise(function(resolve, reject) {
    loadScript('/bundle-aws.min.js', function(err) {
      if (err || typeof AWS === 'undefined') {
        return reject(err || new Error('An error occurred'));
      } else resolve(AWS);
    });
  });
});

Problem: given a list of layers from a Mapbox GL Style Spec style, find groups of more than one layer that have a specific set of properties in common.

Solution:

var refProps = ['type', 'source', 'source-layer',
  'minzoom', 'maxzoom', 'filter', 'layout'];

// takes a layer object, returns a stripped-down version
var pickRequired = R.pick(refProps);

// compose creates a function like JSON.stringify(pickRequired(layer))
var getRefHash = R.compose(JSON.stringify.bind(JSON), pickRequired);

var candidates = R.values(R.groupBy(getRefHash, layers))
  // this is like function(l) { return l.length > 1 }, except point-free
  .filter(R.compose(R.lt(1), R.length))
  // turn a list of objects into a list of their id properties
  .map(R.pluck('id'));

Problem: given a geographical extent as an array of floating-point numbers, return an array of numbers with fixed decimal precision

Solution: note how the ramda way lets us express problems like this in point-free style.

var fixedBounds = bounds.map(R.invoker(1, 'toFixed')(1));

We use ramda in plenty of other places. Often lets us express logic that would require anonymous functions, but without using explicit functions at all: for instance R.always(1) is the same thing as function() { return 1; }, but simpler and more literal.

Try out ramda: it's well-documented, practical, and lets you brag about functional programming to your friends.

Mapnik's Second Act

Mapnik is a tool for rendering maps. Early in the history of Mapbox, Mapnik was the 'beautiful' in the 'beautiful custom maps' slogan. Unlike many of its competitors, the project emphasizes details like anti-aliasing quality and Photoshop-like compositing filters to yield results usable by professional cartographers.

The direction of geo is toward vector maps, and the renderers on that side are more like Mapbox GL or Mapzen's Tangram than Mapnik: WebGL/OpenGL-based systems with an emphasis on frame-by-frame rendering. Mapbox's new map designs don't use CartoCSS or Mapnik XML - we're switching to Mapbox GL Style Spec, which opens up whole new possibilities.

But what of Mapnik? What we're finding out is that the years spent on Mapnik have yielded a project that's pretty remarkable in many ways in addition to map rendering.

Packaging

Photo by Mapbox

mapbox

One of the things that has always made Mapbox different from other node.js shops is how heavily we need to use native modules. node-mapnik, node-sqlite3, node-zipfile, bridge the JavaScript-C++ divide for good reason: interacting with large binary files and connecting to established, complex codebases is tough.

And once we develop these native extensions, they need to be deployed to hundreds of EC2s quickly and without a long compile step. To make this happen Dane Springmeyer and others built mason, a magical build system that yields S3-hosted easy-to-install binary packages.

The end result is that you can install Mapnik with npm install mapnik, and on 90% of computers it'll download a binary and be done in a few seconds.

I can remember days of struggling to compile Boost headers and link everything correctly. The world is a better place now, and the same magic can be applied to other node modules.

Image Processing

Rendering thousands of tiles a second and paying for the bandwidth is a powerful motivator to optimize image encoding. As a result, Mapnik has raster-image processing superpowers that have been reused in quite a few projects:

  • node-blend composites multiple images into one, with offsetting and compression tweaks. It's incredibly fast, and used in production for map compositing. Under the hood, it's node-mapnik.
  • spritezero renders SVG icons into raster sprite sheets for usage as map icons. It's what powers the new icons interface in Mapbox Studio, and it's made of node-mapnik.
  • assert-http is a testing framework for REST APIs. It has the ability to test image outputs and do fuzzy comparisons to account for random values in antialiasing - thanks to node-mapnik.
  • node-fontnik uses a port of Mapnik's rendering stack to rasterize SDF fonts for Mapbox GL maps.

Mapnik's image API covers much of the surface area of Imagemagick but unlike many of the Imagemagick bindings for node, using node-mapnik doesn't mean shelling out to some other process, and the image APIs play nice with node.js Buffer data.

Data

Mapmaking is a process of simplification: given the limits of resolution, size, and cognition, maps always include a subset of available information. In its current place in the Mapbox stack, Mapnik is the engine that turns raw data into visualization-appropriate tiles. As part of this task, it has grown impressive data interfaces: in addition to OGR bindings, Mapnik has its own incredibly fast interfaces to CSV, Shapefiles, TopoJSON, GeoJSON, and SQLite datasources.

This means that you can use Mapnik's incredibly battle-tested code to implement tools like format converters:

var mapnik = require('mapnik');
mapnik.register_default_input_plugins();

var source = new mapnik.Datasource({
  type: 'csv', file: './csv_to_geojson.csv'
});
// get meta-information about this datasource
console.log(source.describe());

// convert this CSV file into GeoJSON
var featureset = source.featureset();
var featureCollection = {
  type: 'FeatureCollection',
  features: []
};
var feature;
while (feature = featureset.next()) {
  featureCollection.features.push(JSON.parse(feature.toJSON()));
}
console.log(JSON.stringify(featureCollection, null, 2));

source