shutterbits

At Shutterstock we recently went through the process of settling on a preferred templating language. We have lots of projects across different languages and platforms, and it was clear to the front end team that we would gain efficiency by investing in a single templating approach.

There are lots of templating languages out there, but two flavors stood out as obvious prospects: the Mustache family, and the Django family. There are strong pros and cons for each. Mustache has unmatched cross-platform support, while Django-inspired templating languages provide a more robust feature set.

We set out to compare implementations of slightly tricky tasks in both settings. Would we miss the extra features from the Django family? We used Node.js along with Swig (Django family) and Hogan (Mustache family) for our comparison.

Task #1: Position within a list

It happens from time to time that we’d like to know our index within a loop while in the context of templating. Let’s say we’re implementing some sort of thing with drag-and-drop, and we want to add a data attribute with the initial position of the element:

Position in a List with Swig

{% for name in names %}
  <li data-position="{{ loop.index }}">{{ name }}</li>
{% endfor %}

Swig makes this easy. We have access to the special loop property and its attributes. This situation was anticipated.

Position in a List with Mustache

With Mustache we have to massage the data beforehand. Somewhere in code before sending to the template, we populate an index attribute:

var names = [ 'Jacob', 'Sophia', ...];
names.forEach(function(name, index) {
  names[index] = { name: name, index: index };
});

Then finally in the template:

{{#names}}
  <li data-position="{{ index }}">{{ name }}</li>
{{/names}}

This is a little rough. The template itself looks great, but it’s regrettable to have to write the code that transforms names to add index alongside. It’s also a bit of a shame to have to mix formatting code in with business logic or controller logic.

Task #2: Currency Formatting

Often we want to show dollar amounts on the page. Let’s see how we do here with a very basic example where we want to show a list of items and a total price.

Formatting with Swig

Swig comes with some built-in filters, but not exactly any that would get us going here. That’s okay–we can define our own filter and register it with Swig:

var usd = function(value) {
  return '$' + value.toFixed(2) + ' USD';
};
swig.init({ filters: { usd: usd } })

Prepare our data and render:

var invoice = {
  items: [...],
  price: 100
};
res.render("checkout", invoice);

Apply filters by specifying them after a pipe for interpolated values:

{% for item in items %}
  <div class="item">{{ item }}</div>
{% endfor %}

<div class="price">{{ price|usd }}</div>

Formatting with Mustache

Mustache requires a similar approach. We define a lambda to pass along with the data to render. Since we want to act on the rendered value here, we need to render the input first and then return a transformation of that.

var usd = function() {
  return function(template) {
    var value = Hogan.compile(template).render(this);
    return '$' + Number(value).toFixed(2) + ' USD';
  }
};
var invoice = {
  items: [...],
  price: { value: 100, usd: usd }
};
res.render("checkout", invoice);

Later in the template:

{{#items}}
  <div class="item">{{ item }}</div>
{{/items}}

<div class="price">
  {{#price}}{{#usd}}{{ value }}{{/usd}}{{/price}}
</div>

This works, but it’s bulky and verbose having to set the price context, and then having to wrap the value with the formatter. One way or another you have to do some massaging before you pass the data to the template. We could also have just done the formatting right within the data structure too, but it would seem a shame to have to mix display formatting with data access.

Task #3: Place CSS at the top and JavaScript at the bottom

There’s a fairly common pattern where on a given page we may want to include a page-level CSS stylesheet, generate some markup, and then add event listeners at the end. From within a page template, ideally we’d like to put a <link> tag in the head and inject some JavaScript just before the closing </body> tag.

CSS at the top and JS at the bottom with Swig

Swig supports block-level inheritance. In our layout template we specify the document structure:

<!-- layout.swig -->
<!doctype html>
<html>
  <head>
    <link rel="stylesheet" type="text/css" href="/global.css">
    {% block css %}{% endblock %}
  </head>
  <body>
    {% include 'header.swig' %}
    {% block content %}{% endblock %}
    {% include 'footer.swig' %}
    {% block footer_js %}{% endblock %}
  </body>
</html>

Now from within a page template we can override those blocks to get our bits to be where we want them within the document:

<!-- page.swig -->
{% extends 'layout.swig' %}

{% block css %}
  <link rel="stylesheet" type="text/css" href="/page.css">
{% endblock %}

{% block content %}
  <div id="main">Wonderful content here...</div>
{% endblock %}

{% block footer_js %}
  <script>
    var main = document.querySelector('#main');
    main.addEventListener('click', function(e) {
      e.target.setAttribute('contenteditable', true);
    });
  </script>
{% endblock %}

CSS at the top and JS at the bottom with Mustache

Mustache doesn’t directly support inheritance so we have to fake it. There are different ways to do it, none of them especially elegant. One way is to just always piece together the whole document using partials.

<!-- css.mustache -->
<link rel="stylesheet" type="text/css" href="/global.css">

<!-- page.mustache -->
<!doctype html>
<html>
  <head>
    {{>css}}
    <link rel="stylesheet" type="text/css" href="/page.css">
  </head>
  <body>
    {{>header}}
    <div class="main">Wonderful content here...</div>
    {{>footer}}
    <script>
      var main = document.querySelector('#main');
      main.addEventListener('click', function(e) {
        e.target.setAttribute('contenteditable', true);
      });
    </script>
  </body>
</html>

This is nicely explicit, but has its obvious drawbacks too. There are other approaches to consider as well. Some implementations support the concept of standard layouts. There’s also a proposal to update the Mustache spec to support inheritance. Hogan actually has unadvertised support for the proposed implementation. Yet another way is to use carefully crafted helpers in Handlebars. But basically when it comes to inheritance, Mustache support is a bit of a mess at the moment.

Other Considerations

Cross-Platform Support

We want to avoid having our templates lock us into a backend language or framework. Ideally if we want to migrate a project from Ruby/Sinatra to Node/Express, the last thing in the way should be the templates.

On a more practical level, we also want the flexibility to be able to share templates between the server and the client. For example, we may want to serve the first page of search results as rendered markup from the server, but then serve subsequent pages through AJAX, sending back JSON for the browser to interpolate into the same template that the server used. Both Mustache and Swig have solid in-browser implementations, so either fits the bill there.

Mustache actually has a formal spec, so you can be fairly sure that mustache templates that render with one implementation will likely render the same way in another. In real life though, since Mustache is very minimal, different implementations often add extra improvised features. Hogan adds nested accessors; Handlebars adds alternative control-flow syntax and helpers.

The Django family has decent cross-platform support, but implementations take wider liberties deviating from the original Django version. There’s Swig for JavaScript, Jinja for Python, Liquid for Ruby, DTL for Perl, and Twig for PHP. These implementations are all very strongly Django inspired, but again they diverge here and there.

Learning Curve

At Shutterstock we like to encourage people to reach outside their core disciplines and think holistically about what we’re all trying to do. For example, front-end developers might weigh in heavily on the direction of the product; back-end developers might suggest design directions; ops folk might suggest new business strategies, etc, etc.

So when it comes to templating, we want anyone to be able to dive in and find their way quickly, whether or not they have a background in front end development. A quick qualitative survey around the office found that relatively technical people had a much easier time picking up the Django style than they did learning the terse Mustache semantics.

Conclusion

In the end we went with the Django family — Swig, Twig, Liquid, etc. We still use Mustache here and there, but for projects of any decent size we find we really enjoy having the extra features and robustness in the Django templating languages.

We’re happy to share a project we’ve developed that helps us understand how our customers are interacting with our site.  It’s called Lil Brother — it tracks clicks and other events in the browser, and reports back in real time.

How does it work?

First, add some client code to pull in the library and initialize a Lil Brother observer:

<script src="http://server:8000/lilbro.js"></script>

<script>
var lilBro = new LilBro({
    element: document.body,
    server: 'server:8000'
});
</script>

Then start up the node listener on the server side:

$ node bin/lilbro --output-file events.log

Lil Brother attaches a single event handler to a top-level container and listens for events.  As users click on links or focus inputs or check boxes, etc, the client sends those events over the wire to a listener on the server where they get recorded.

Can I have some context?

When a click happens, we gather what context we can and send that along too.  If the target element has an id and/or a class, we note that.  Otherwise, we traverse up the DOM until we find a parent’s id or class.  We also grab the element tag name, X and Y mouse coordinates relative to the element and to the page, scroll positions, and input values if the element happened to be some sort of input field.

In addition to metadata around the event, we discover other attributes too: browser version, operating system, viewport width and height, request path, and some other bits.

Visits and visitors

Of course clicks are part of a larger hierarchy.  There are users behind these clicks, and users browse in sessions.  To tie events together, Lil Brother sets two cookies: a long-lived visitor cookie, and a short-lived visit cookie.  We send the values of these cookies along so that we can string events together and aggregate later.

What next?

With the data captured on the client, sent over the wire, and recorded one way or another, Lil Brother’s job is done, and there’s much left to do yet.  From here, aggregation, visualization, analysis are now possible, and left to you.  Imagine what you can do with this data: build heat maps, visualize traffic patterns, discover customer friction points, measure behavior in A/B tests, and on and on. Have fun!

As coders, we usually begin our careers by throwing together commands sprinkled with single-letter variable names and wild contortions of logic. The code usually does the job inefficiently and fails in catastrophic ways in unexpected situations. As we progress, our code becomes more robust and correct, but for a while, we continue to put together code that isn’t very well-designed. Code like this is “underengineered.” Underengineering is bad, unless it’s the result of a conscious decision to get the job done faster by skipping most of the design phase and taking on some “technical debt” (AKA “design debt”).

After a couple years of unintentionally writing underengineered code and dealing with the fallout, we compensate by planning ahead and designing better systems which anticipate future needs. Our code becomes easier to read, understand, and work on. That might be the sweet spot, because the next step for most of us is overengineering.

Once we get good at designing systems in abstract ways, we often proceed to develop them in the most abstract ways possible. Overengineered projects often take months or years to complete. It’s difficult to get consensus when every last detail has to be perfect. When we overengineer, it’s likely that by the time we’re done, the business or technology environment will have changed in ways that render much of that excessively well-planned work irrelevant. And, we can’t see the future, no matter how hard we try. When we overengineer, despite all of our planning and abstractions, we often fail to anticipate the abstractions we’ll actually need. Voltaire is credited with the saying, “The perfect is the enemy of the good.”

Why do we overengineer? It seems smart at first. If some abstractions are good, then lots of abstractions are better! Programmers also get bored. We like to solve new problems and develop new skills. A coder friend both wise and tired beyond his years once told me, “There are only seven problems in software engineering, and we just solve them over and over again.” It took me years to understand what he was saying, because I’ve always said, “Software engineering is the best job in the world, because it’s never the same job twice. If it is, you’re doing it wrong!” In fact, we’re both right. His point was that although we may be solving new problems with new languages, abstractions, techniques, and technologies, at the bottom of it all will always lie just a few fundamental solutions. Loops. Conditionals. Functions. Reading from and writing to files and databases. Because of this, programmers can get bored even while engaged in one of the most interesting jobs in the world.

Overadoption is a similar problem that doesn’t get nearly as much attention as it deserves. Programmers love new technology. But the newest, coolest technologies are also the most buggy. Last year I found and fixed a bug in our payment system that could have caused us to approve all payments of a certain type without actually checking to see if those payments were authorized. The bug was not introduced directly through the fault of any of our coders. It had been made possible long ago by the inclusion of an open-source software package that we were using before it was quite ready for enterprise use. After we installed it, it went through so many changes so quickly that we couldn’t keep up. Updating the package would have broken our code, so we were stuck living with a bug that had been fixed years ago. We were early adopters. But it’s possible to adopt too early. We’ve also quickly adopted and trashed various wiki systems, file systems, project management software systems, and version control schemes. We’ve gotten to a pretty good place today, but several of the steps in between could have been skipped entirely if we’d just been a little more patient and reflective before jumping on those rickety bandwagons.

Engineers sometimes, usually unintentionally, make themselves indispensable by hoarding knowledge about arcane legacy systems. It’s quite possible to become indispensable for the opposite reason as well. If the latest, greatest technology is installed the moment it’s released, it will be difficult for everyone in the organization to keep up. We all enjoy learning new things, and building our skills helps the organization as a whole. But sometimes the best thing we can do for the company is to go to work and solve those seven boring problems all over again for a few hours. Sometimes we just have to sit down and bang out some boilerplate.

William F. Buckley, Jr. said, “A Conservative is a fellow who is standing athwart history yelling ‘Stop!’” I’m not a conservative. I’m just standing athwart technology calmly advising, “Slow down a little.”

Developing solutions to complex technical problems as a team can be fun and challenging, but also highly contentious as every great developer has their own Grand Plan for how things Should Work. For every developer in the room there will be just as many proposed solutions to any given problem. They may be based on competing philosophies, varieties of foresight, attention to edge cases, and other considerations that provide a fertile ground for a passionate discussion, but that discussion often struggles to find focus and consensus.

So how do you streamline this process? Discussions like these can go on endlessly, go off on tangents, and nitpick on trivial details. How do you avoid hashing out solutions that are over-engineered and overly complicated?

One way to do this is to focus more on the problem rather than the solution.

When you focus solely on the problem, you’re working with facts everyone can agree on. Aim your discussion at getting everyone to fully understand the problem, and all the issues surrounding it. Once everyone is on the same page, you’re better positioned to prioritize the issues and offer specific proposals to address them. At this point, you still need to work to keep the discussion focused, so always take every solution and go back to understand the problems they address, and their side effects.

As an example here’s a set of 5 questions I like to follow when attacking a new problem. In general, it keeps me focused, while preventing me from falling in love with any one solution prematurely.

1. What is the high level problem you want to solve?

2. What impediments are currently making this a problem? i.e. Why does this problem exist?

3. For each impediment, what are the issues that need to be addressed to resolve that impediment?

4. For each issue that needs to be addressed, what are all the possible solutions you can imagine?

5. For each solution, what are each of their advantages and disadvantages? What is the scope and magnitude of each of those advantages/disadvantages?

The main things to take away from this are that in steps 1 through 3, you’re just decomposing the problem. Every complex problem is just a combination of simple problems. When you’re dealing with simple problems, you’re more likely to come up with simple solutions.

A second thing to note is that it isn’t until step 4 that you’re even thinking about solutions. Also, your goal at this point is to come up with multiple solutions to each issue. This is very useful for helping everyone keep an open mind. Each individual should make it their goal to see the advantages and disadvantages of each possible solution rather than picking one and arguing it to death.

Step 5 may still be the most contentious since it leaves the most room open for debate, however the fact that you’ve just laid out and decomposed the problem should lead to a more focused discussion on possible solutions.

So in your next planning meeting, give this a shot, and see if it leads to a faster successful outcome, or at the least, a more enjoyable meeting. The key thing is to use the forum of a group discussion to make sure ideas are added that any one person may have missed — not as a platform for everyone to argue their master plan. Ultimately every developer has the same goal of solving problems, executing that goal well as a team is the challenge. But instead of getting passionate about the solution, get passionate about understanding the problem.

Like many other Linux/open-source software tech companies, Shutterstock makes extensive use of tried-and-true technologies like MySQL.  We are always exploring different database technologies such as Riak and MongoDB, but at the core of our business is a highly available and tightly managed MySQL infrastructure.  We started on MySQL with a loosely-designed schema and have been adding to it incrementally over the years.

Some of our developers who are less comfortable on the command line use GUIs to access the database but the more bearded folk tend to use the standard mysql-client CLI tool that is the stock and trade of any LAMP stack.  vim, emacs, git and mysql are usually open in many terminals on our desktops.  But, unlike the others, the mysql shell is not the most up-to-date tool in the toolbox.  With poor pagination and output handling, no color highlighting and a somewhat irritating input prompt, the mysql client causes its fair share of frustrated “that’s not what I meant to happen” moments.

We need pagination.  With our eight-year-old schema, some of our tables span 800 columns of console output.  Simple queries like “select * from accounts limit 1” fill the screen with line after line of ASCII table rendering characters  Even restricting the output to one line per column (“\G”) makes for an impressively difficult amount of data to parse.

In response to this, we recently undertook some improvements to the tool.  With our expertise in Perl and Moose OO programming, rewriting the mysql client in Perl seemed like a simple exercise in DBI programming (which we’re very comfortable with) and some straightforward CLI tooling.  By approaching the problem iteratively, we were able to very quickly come up with a drop-in replacement to the mysql client with the majority of features we use on a daily basis implemented.  From this as a starting point, we were free to explore what we wanted to fix.

We’re very pleased with the result.  We call it AltSQL, as it’s an alternative to and improvement over some of the standard command line SQL tools.

The first and simplest change to make was to add color.  We’re used to seeing our prompts full of color.  Our bash prompt highlights the hostname name in red, ls shows directories in blue, and vim and emacs give our coding full color syntax.  Adding contextual coloring to tabular output makes sense, was a simple addition, and comes at no expense since the DBI statement handler contains a great deal of context about each result that’s delivered.

Implementing a better prompt was a simple matter of finding a suitable CPAN module, and Term::Readline::Zoid fit the bill on that measure quite well.  Offering out-of-the-box multiline editing and an extensible autocomplete and key binding interface, we were able to move quickly.

We finally had a mysql shell prompt that could abandon the statement when you typed Ctrl-C rather than exiting the program. Improving the table rendering was next.  By dropping in Text::ASCIITable we quickly had a better table renderer that properly wrapped output on newline characters, but why stop there?  All of our terminal emulators have full Unicode support, so we spent some time developing a simple but powerful low-level Unicode box formatter (Text::UnicodeBox) to make terminal table drawings more intuitive and less obtrusive.

Adding horizontal and vertical pagination was a simple change, but a powerful one.  By checking the output width and height of the table to be printed, we are able to conditionally use the less pager.  This feature finally made “select * from accounts limit 1” a command we could type without worry.  No matter what the terminal size, you’ll be able to see the data in a usable format.

This is just the beginning.  By choosing Moose, all the features of the tool are extendable by other modules.  We’ve written it from the ground up to be pluggable.  In fact, most of the features mentioned above aren’t a  part of the core code, but instead written as modules to modify the behavior.

We hope that other people can benefit from this.  Regardless of if you use Perl or not, we think this is a useful tool that could make your job easier.  Install it from CPAN or Github and try it out.