Practical ramda - functional programming examples

Ramda is a second-generation JavaScript library for functional programming. We're using it in Mapbox Studio to great effect, and given that functional programming libraries are often documented with mathematical or abstract examples, I think it would be useful to give some examples of how.

Second generation

So Ramda is a second generation functional programming library.

Arguably the first generation was underscore and lodash. In the era when Internet Explorer 8 was still a consideration in JavaScript programming, underscore had a clear purpose: to make JavaScript's functional methods cross-browser compatible. JavaScript has great, built-in functionality for functional iteration, mapping, and reduction, but these methods, like Array.map and Array.reduce, have been slow to arrive in Internet Explorer. underscore gave the world reliable _.map and _.reduce methods - along with many others - that worked the same in every browser.

In the time since, Internet Explorer 8 has faded into the past and native ES5 methods are expected for most applications. So the role of functional programming libraries has shifted from a compatibility layer to adding new, foundational methods for functional programming. In Mapbox Studio, we started with 101, a library that is "maintained to minimize overlap with vanilla JS." 101 provides a lot of great functionality, especially with support of keypaths throughout, but we found that composing 101's methods was a bit lacking.

Ramda is a library built for composition and higher-level functional programming. It tends to flip method arguments, so instead of _.map([], function() {}), ramda exposes R.map(function() {}, []), which more often makes it possible to reuse the map function. And functions in ramda are curried by default, so the R.map method could be called like R.map(function() {}, []) or R.map(function() {})([]) interchangeably.

The third generation of functional programming libraries in JavaScript is probably something like trine, which uses future syntax to write functional application in a chaining style. But ES6 is already a bit of a gamble, so Mapbox Studio isn't ready to heavily use a strawman spec.

Currying

One of the major strengths of ramda is its approach to currying methods. Ramda's curried methods mean that a method that looks like

foo(1, 2, 3);

Can be called with each argument in steps:

foo(1)(2)(3);
// which means
var curried = foo(1);
var alsoCurried = curried(2);
var finallyEvaluated = alsoCurried(3);

This is really nice in the case of a method like Array.map because you can partially apply a method and use it as a method in Array.map. "Partially applying the method" means you give it fewer than the full set of required arguments and supply the rest of the arguments later.

For instance, let's take an array of objects with an 'id' parameter:

var data = [{ id: 'foo' }, { id: 'bar' }];

The R.prop method will give you the value of a property of a single object:

var id = R.prop('id', { id: 'foo' }); // returns 'foo'

But what about an array? The desired end result is ['foo', 'bar']. It would be great if we could just use R.prop, but have it apply against an array. Luckily, thanks to currying, we can:

var ids = data.map(R.prop('id'));

You provide the method R.prop('id') as the argument to .map: it is a function that, given an object, returns its id property.

Enough talk. Let's see some code.


Problem: given an array of objects called layers, create an object called layerMap that has the 'id' property of a layer as a key and the layer itself as a value.

Solution:

var layerMap = R.zipObj(R.pluck('id', layers), layers);

Problem: you want to dynamically load a JavaScript library to power a function. In this case, we want to load the AWS SDK when someone wants to upload a file. But we want to make sure that if the method is called many times in a row, we don't load the library as many times.

Solution: the R.once method ensures the function is only evaluated once, and on successive calls returns the same value.

var getAWS = R.once(function() {
  return new Promise(function(resolve, reject) {
    loadScript('/bundle-aws.min.js', function(err) {
      if (err || typeof AWS === 'undefined') {
        return reject(err || new Error('An error occurred'));
      } else resolve(AWS);
    });
  });
});

Problem: given a list of layers from a Mapbox GL Style Spec style, find groups of more than one layer that have a specific set of properties in common.

Solution:

var refProps = ['type', 'source', 'source-layer',
  'minzoom', 'maxzoom', 'filter', 'layout'];

// takes a layer object, returns a stripped-down version
var pickRequired = R.pick(refProps);

// compose creates a function like JSON.stringify(pickRequired(layer))
var getRefHash = R.compose(JSON.stringify.bind(JSON), pickRequired);

var candidates = R.values(R.groupBy(getRefHash, layers))
  // this is like function(l) { return l.length > 1 }, except point-free
  .filter(R.compose(R.lt(1), R.length))
  // turn a list of objects into a list of their id properties
  .map(R.pluck('id'));

Problem: given a geographical extent as an array of floating-point numbers, return an array of numbers with fixed decimal precision

Solution: note how the ramda way lets us express problems like this in point-free style.

var fixedBounds = bounds.map(R.invoker(1, 'toFixed')(1));

We use ramda in plenty of other places. Often lets us express logic that would require anonymous functions, but without using explicit functions at all: for instance R.always(1) is the same thing as function() { return 1; }, but simpler and more literal.

Try out ramda: it's well-documented, practical, and lets you brag about functional programming to your friends.

Mapnik's Second Act

Mapnik is a tool for rendering maps. Early in the history of Mapbox, Mapnik was the 'beautiful' in the 'beautiful custom maps' slogan. Unlike many of its competitors, the project emphasizes details like anti-aliasing quality and Photoshop-like compositing filters to yield results usable by professional cartographers.

The direction of geo is toward vector maps, and the renderers on that side are more like Mapbox GL or Mapzen's Tangram than Mapnik: WebGL/OpenGL-based systems with an emphasis on frame-by-frame rendering. Mapbox's new map designs don't use CartoCSS or Mapnik XML - we're switching to Mapbox GL Style Spec, which opens up whole new possibilities.

But what of Mapnik? What we're finding out is that the years spent on Mapnik have yielded a project that's pretty remarkable in many ways in addition to map rendering.

Packaging

Photo by Mapbox

mapbox

One of the things that has always made Mapbox different from other node.js shops is how heavily we need to use native modules. node-mapnik, node-sqlite3, node-zipfile, bridge the JavaScript-C++ divide for good reason: interacting with large binary files and connecting to established, complex codebases is tough.

And once we develop these native extensions, they need to be deployed to hundreds of EC2s quickly and without a long compile step. To make this happen Dane Springmeyer and others built mason, a magical build system that yields S3-hosted easy-to-install binary packages.

The end result is that you can install Mapnik with npm install mapnik, and on 90% of computers it'll download a binary and be done in a few seconds.

I can remember days of struggling to compile Boost headers and link everything correctly. The world is a better place now, and the same magic can be applied to other node modules.

Image Processing

Rendering thousands of tiles a second and paying for the bandwidth is a powerful motivator to optimize image encoding. As a result, Mapnik has raster-image processing superpowers that have been reused in quite a few projects:

  • node-blend composites multiple images into one, with offsetting and compression tweaks. It's incredibly fast, and used in production for map compositing. Under the hood, it's node-mapnik.
  • spritezero renders SVG icons into raster sprite sheets for usage as map icons. It's what powers the new icons interface in Mapbox Studio, and it's made of node-mapnik.
  • assert-http is a testing framework for REST APIs. It has the ability to test image outputs and do fuzzy comparisons to account for random values in antialiasing - thanks to node-mapnik.
  • node-fontnik uses a port of Mapnik's rendering stack to rasterize SDF fonts for Mapbox GL maps.

Mapnik's image API covers much of the surface area of Imagemagick but unlike many of the Imagemagick bindings for node, using node-mapnik doesn't mean shelling out to some other process, and the image APIs play nice with node.js Buffer data.

Data

Mapmaking is a process of simplification: given the limits of resolution, size, and cognition, maps always include a subset of available information. In its current place in the Mapbox stack, Mapnik is the engine that turns raw data into visualization-appropriate tiles. As part of this task, it has grown impressive data interfaces: in addition to OGR bindings, Mapnik has its own incredibly fast interfaces to CSV, Shapefiles, TopoJSON, GeoJSON, and SQLite datasources.

This means that you can use Mapnik's incredibly battle-tested code to implement tools like format converters:

var mapnik = require('mapnik');
mapnik.register_default_input_plugins();

var source = new mapnik.Datasource({
  type: 'csv', file: './csv_to_geojson.csv'
});
// get meta-information about this datasource
console.log(source.describe());

// convert this CSV file into GeoJSON
var featureset = source.featureset();
var featureCollection = {
  type: 'FeatureCollection',
  features: []
};
var feature;
while (feature = featureset.next()) {
  featureCollection.features.push(JSON.parse(feature.toJSON()));
}
console.log(JSON.stringify(featureCollection, null, 2));

source

Map Codeflow

For quite a while now, I've been working on the new Mapbox Studio, a visual map styling tool. It's based on Mapbox GL and writes styles in the GL Style Spec, a dialect of JSON.

What we've been working on is a visual tool: you can import and export JSON, but unlike previous products, the focus is more on a user interface than it is on a tricked-out CodeMirror-based code editor. This is indicative of some deeper changes.

Vector tiles split data processing and rendering

Mapbox GL being in-browser makes it possible to build a much simpler style editor. CartoCSS, our previous styling langage, compiled down to Mapnik XML in a rather slow build step. And then the styles needed a Mapnik install to run.

Mapnik does many jobs: it parses geospatial data formats, projects that data, applies fancy styles and anti-aliasing, and then creates tiny and efficient PNG & JPG files.

In stark contrast, the world of Mapbox Vector Tiles and Mapbox GL detaches the rendering from the data processing: one tool turns geodata into tiles, another tool turns tiles into images.

Newness

While the MBTiles & CartoCSS standards have had time to grow, and now have quite a few implementations, vector tiles are admittedly bleeding edge, and the projects we're starting to release are some of the first examples of what they can do.

From building the new Mapbox Studio first as a code interface, and then as a UI, I'm entirely convinced that visual is the way forward. We can make a new promise to people: that they will no longer run into silly syntax errors, or have three abstractions to debug simultaneously. There'll no longer be a language that's a bit like CSS, but not exactly: the GL Style Spec opens us up to a whole new universe of tooling that works extremely well with JSON - something that would never happen with a custom syntax. And with no parse errors in-between changes, we can smoothly transition between differerent variations of a style and save every change in an undo/redo history.

The code flow

What's really exciting (besides, of course the new Mapbox Studio) is that Mapbox GL makes it simpler to build powerful software with maps. There should be lots of good options, in many different styles.

So: the mapbox-gl-codeflow-example shows how you develop Mapbox GL styles in code. In around 60 lines of simple JavaScript, it runs a little gulp command that props up a server, reloads a page on changes, and moves around files. It's like TileMill, without the UI: as we eventually learned, a lot of people technical enough to write code are also very partial to their text editors, so you can use Sublime/vim/emacs/Atom or whatever you'd like.

There's one bonus feature, inspired by a few people's comments about JSON being a rather strict language. The example includes converters for YAML, toml, and JSON5, so you can use a terse, comment-supporting language as input, if you'd like.

A layer in YAML

  - id: 'background'
    type: 'background'
    paint:
      'background-color': '#f00'

In TOML:

[[layers]]
  id = 'background'
  type = 'background'
  [layers.paint]
    background-color = '#f0f'

In JSON5:

{
  id: 'background',
  type: 'background',
  paint: {
    'background-color': '#ff0'
  }
}

For even more power, it also supports JavaScript applications as input, so you can actually author styles in JavaScript and it'll run them: so you can use JavaScript's wide range of libraries and native support for variables and operations. In my example, I'm using chroma-js as a subsitute for CartoCSS's color operations.

Errors

I think a lot about errors: it seems like error-handling is one of the most important parts of any development environment.

So this example also includes error handling: for syntax errors in JavaScript, semantic errors in stylesheets, and parse errors in JSON, it shows the error in a nicely-formatted browser window. It's inspired by React Native's awesome error-display style.

Just an example

This is just one example of a tool for a GL style development workflow: there are many other ways to do it, in other languages with different wheelies. The whole of the code is shorter than most Gulp build scripts.

Stay tuned for Mapbox Studio, and check out the codeflow example on GitHub.