For the geeks: Github data viz!

I got to work on this awesome project for the last couple weeks. I had a ton of fun data-mining and playing with d3. Check it out:

gitviz.jit.su

Also, check out some of the questions I approached, and the solutions I came up with.

  • How do I not go into callback hell while doing multiple API calls?
  • How do I mine and then return a large amount of data to a client?

Solutions:

Using Node’s EventEmitter library

GitViz uses the EventEmitter library for two main purposes:

  • Preventing callback messes.
  • It makes streaming data incrementally to the client clean and simple

Here is how I did it:

Require the library:

EventEmitter = require('events').EventEmitter

Create a unique instance of event emitter. I built an ‘initialize’ function that wraps the creation process. I export this to my express server file.

init = () ->
#this 'reset' method re-initializes global constants used in the script.
  reset()
  eventMaker = new EventEmitter
  eventMaker.get = get
  return eventMaker

When I run init() in the express server, it gives me both the “get” function, which makes the necessary API calls, and the ability to emit events, and set triggers.

Here is the relevant server code:

app.get '/query/:user/repo/:repo', (req, res) ->
  repoRoute = req.params.user + '/' + req.params.repo
  db.Commit.findOne { 'repo': repoRoute }, 'commits', (err, commitList) ->
    throw err if err
    if commitList
      res.send commitList.commits
    else
      res.write '['
#getCommits.init() is the same as init() above.
      commitStream = getCommits.init()
      commitStream.on 'commit', (commit) ->
        res.write commit
      commitStream.on 'end', (string) ->
        res.end ']'
      commitStream.get req.params.user, req.params.repo

After the event emitter has been initialized, I now have access to setting triggers with “commitStream.on”

These line up with when I emit a ‘commit’ event from inside the worker, after I have the latitude and longitudes for any given commit:

currentRequest.emit 'commit', commitLocation

This event emitter was easy to implement, but I got snagged temporarily by not having an initialize function wrapping around it.

Before I wrapped the instantiation of the event emitter (‘new EventEmitter’) in an initialize function, I was adding events and triggers to the same event emitter on every ‘get’ request from the client to the server. This resulted in pre-mature triggering of the events after the first time the eventEmitter is set.

Be careful of premature event emission!

Using res.write and res.end

Grabbing all the commits on a large repository can take more than 2 minutes. When deciding between sockets or get requests, it felt intuitive to just use a socket and emit an event to the client when all the commits were ready, and then just send them all at once.

Instead of implementing sockets, I just stuck with node’s native ‘write’ and ‘end’ function, combined with the event emitter.

Every time this line ran on the node worker:
currentRequest.emit ‘commit’, commitLocation

It triggered this event on the server:

#this line opens an array to place commits into
res.write '['
#this event places events into the array
commitStream.on 'commit', (commit) ->
    res.write commit

By using res.write instead of res.send or res.end, I allow the connection to receive a small amount of the data. This prevents the browser from shutting down a hanging get request.

When the last commit is ready, the worker emits this event:
currentRequest.emit ‘end’, ‘done!’

Resulting in ending the response and closing the array on the client:

commitStream.on 'end', (string) ->
    res.end ']'

res.end tells the client to close the response.

The Dwarven Node Miner

I had to write a bio for myself recently, and was asked this question…It was also April Fool’s.

What is the most exciting piece of code you’ve written?

The console logs “saved” and my eyes light up with joy. I am a bearded dwarf, my node worker is my pick axe, and I am deep in the mines of Nested API Calls. I open my mongoDB shell and I look into my collections. Precious JSON jewels everywhere!

After hours of toil and sweat dripping from my long dwarf-beard, I have finally found it.

I…

  • pulled every commit, ever, on a large open-source repo on Github,
  • plucked the author of every commit,
  • sent a second request for more details on each author,
  • got their latitude and longitude from the google maps API,
  • sent one commit with geo-data at a time to the client using events

…then ended the response after the last shiny JSON object was sent up the mineshaft.

I stare lovingly at my trusty Node pick-axe. It is adorned with events to keep nested callback goblins at bay, and crystal clear method names and responsibilities gild it’s edges. The shiniest axe I have ever crafted.

I do a dwarven diddy and song to celebrate.

I built Deface.meteor.com!

I had a ton of fun building this Meteor app. Here’s a link to my app!!, and the description:

Deface is a collaborative draw-on-someone’s-face app.

  • Provide a link to any jpeg on the internet, and it will pop up on the screen shortly.
  • Add SVG images of cats to the photo.
  • Use the draw tool to upgrade the photo with MS Paint optimization.

I ran into a few technical challenges when working on this app, and I made some simple explanations that will hopefully save people time. The topics are:

  1. Futures
  2. Adding Node Modules to Meteor
  3. Public Collaboration

Also, a link to the Github repo: https://github.com/jkatsnelson/deface-meteor

Futures!

The server side component of this app involves one method:

get_image: (url, id) ->
    fut = new Future()
    options =
        url: url
        encoding: null
    # Get raw JPG binaries
    request.get options, (error, result, body) ->
        if error then return console.error error
        jpeg = body.toString('base64')
        fut.ret jpeg
    # pause until binaries are fully loaded
    return fut.wait()

If the future wasn’t present in this code, I would be implicitly returning the request object, rather than fut.wait().

To make your code block and wait for an asynchronous function to return, simply:

1) Create a Future

fut = new Future()

2) Tell the future what you want to return

fut.ret jpeg

3) Tell the function you are writing to wait until the future returns!

fut.wait()

Adding Node Modules to Meteor!

I used the [Request][request] node module to grab jpg images as raw binary.
(I couldn’t use Meteor’s http library to do this– it wasn’t returning the correct binaries. Like a good dev, I opened a bug report.)

It’s slightly tricky to add node modules to Meteor AND successfully deploy to Meteor’s hosting platform.

Here is what I used:

require = __meteor_bootstrap__.require;
fs = require('fs')
path = require('path')
base = path.resolve '.'
isBundle = fs.existsSync base + '/bundle'
modulePath = base + (if isBundle then '/bundle/static' else '/public') + '/node_modules'
request = require(modulePath + '/request')

So what is going on here?

Meteor provides require.js with this call to meteor bootstrap:

require = __meteor_bootstrap__.require;

Just for getting node modules to work, require the filesystem module (fs) and path module.

fs = require('fs')
path = require('path')

Using Path, find the base directory of the project, whether we are in deployment or production:

base = path.resolve '.'

Check if we are in production by looking for a “bundle” file

isBundle = fs.existsSync base + '/bundle'

Use what we’ve learned about whether we are in production or development to create the proper path to the node modules:

modulePath = base + (if isBundle then '/bundle/static' else '/public') + '/node_modules'

Voi la, and then simply use the correct path to require request:

request = require(modulePath + '/request')

Public Collaboration

Leveraging Meteor’s observer helpers and [Fabric]’s event system and serialization methods made it really easy to add collaborative drawing.

Here’s how I did it:

First, I added an event on mouse up that turns the entire canvas into JSON and adds it to MongoDB.

window.canvas.on 'mouse:up', () ->
    image = window.canvas.toJSON()
    Images.insert image

On start-up:

Meteor.startup ->

I initiate a new Fabric canvas

  window.canvas = new fabric.Canvas 'c'

Create a query on the images database, and watch for the ‘added’ event

  query = Images.find({})
  query.observe
    added: (image) ->
      image.objects && window.canvas.loadFromJSON image

When an image is added, I use fabric’s ‘loadFromJSON’ method to put the added image in the canvas, overwriting the current canvas.

I like my coffee how I like my implicit returns and my backbone renders…

I won’t be continuing that joke, I promise– but I will be talking about implicit returns in Coffeescript, with an example of Backbone.js biting me in the Backside (I won’t be continuing that joke, either).

Specifically, I’d like to explain a situation I found myself in recently while building a Blackjack game in Backbone.js.

Let’s begin with a brief tour.

This is in my index.html. Notice that the result of the BlackjackView’s ‘render’ function is appended to the body of the html.

And this is the view being instantiated.

     tl;dr: just read the render function from line 27-30.

So, looks pretty good, right? Here’s the pseudocode for what my render does:

The render:

  • Detaches all children from the body, making sure I’m starting with a clean slate as I render my main view.
  • Renders the main view’s template, creating the skeleton for my blackjack game.
  • Appends the player’s hand, showing their cards.
  • Appends the dealer’s hand.

What it really did was render only the dealer’s hand.

The difference between the results and my expectations can be explained by two things:

  1. Backbone made a design decision that somewhat decouples the term “render” from what it really does, since it does not necessarily “render” a view to the DOM directly when called.
  2. I need to be really careful with implicit returns in CoffeeScript.

Explanation:

  • Let’s examine this line of the render, second from last:

@$('.player-cards').append @playerHandView.render()

On first glance, it looks like “append the player hand view to a jquery selector, which is in the DOM”

On second glance, it looks like “append the player hand view to a jquery selector, which is attached to @(keyword ‘this’ in javascript). Since @ is the BlackjackView, which is being rendered to the DOM in index.html, then it’s basically the same as my first inference.”

So, why the heck would the player hand not be on the DOM, but the dealer hand would be? They are nearly the same line.

The biggest thing I should have paid attention to right away, is a question to always be asking yourself in coffeescript:

“Where is the implicit return?”

Here is what the render function looks like, compiled into Javascript:


render: function() {
 this.$el.children().detach();
 this.$el.html(this.template());
 this.$('.player-cards').append(this.playerHandView.render());
 return this.$('.dealer-cards').append(this.dealerHandView.render());
 }

So, I was actually returning the append to the dealer cards.

But why would that matter? Look at all those other appends I did before that…Didn’t I just render the template, and the player hand? Why wouldn’t they be on the DOM too?

Well, it all comes down to what is going on in index.html.

When I call the “render” function on the BlackjackView, I’m not actually ‘rendering’ anything. I am returning a DOM element, which can then be used by the append function in the BlackjackView. Render doesn’t actually “render” anything – it returns a DOM element.

Even though this is misleading, I don’t think it’s a bug in Backbone. I think it’s an intelligent design decision with some unfortunate side effects, like misleading me.

It makes sense that a view should have a “render” function, and that the render function should ultimately return a DOM element. Many Backboners (hehe) choose to use a Singleton pattern like the one I show on index.html– where everything ultimately happens in one jQuery append call, right on the body. That way, everything is encapsulated inside of Backbone’s eco-system. If each view’s “render” function were actually appending to the DOM, things could easily become disordered.

In fact, they often do with people who do not use Backbone idiomatically. When I first began learning Backbone, I immediately started NOT using Backbone INSIDE OF Backbone. I would just use the “views” and “models” as rough sketches of what I wanted to do, while actually just appending things to the DOM using direct jQuery calls, and modifying attributes to my models without ‘get’ and ‘set’. If you weren’t aware you shouldn’t be doing this– let me know, I’ll write about it!

What do you want me to write about?

Well, I’m almost half-way through Hack Reactor’s (formerly Catalyst’s) 12 week immersion program.

I’ve learned a lot, that’s for sure. I plan on blogging about javascript, and generally about what it’s like to be at a 66-hour a week programming immersion school.

Here are some thoughts on potential topics. Please let me know if you’d find them interesting:

General School Related Stuff:

  • What’s it like being in an immersion programming school for 66 hours a week? How do you find time to do anything else?
  • How hard was it to get into the school?
  • Do you believe in love?
  • How optimistic are you about finding an (awesome) job after class?

Javascript/Programming Related:

  • A 5-step model to answering CS questions on job interviews, or in general.
  • How can you learn data structures in javascript?!
Design Patterns:
  • What are the major Javascript design paradigms?
  • What are good sub-classing methods when using pseudo-classical instantiation?
  • How not to use Backbone
Common mistakes:
  • What’s up with the keyword “this”?
  • Say I have an object with a key value pair like this: object = { 1: ‘meow’ }. Why can’t I access the value with object.1?!?!
  • Coffeescript– how to not get bit by implicit returns.

Design patterns

For the last few days of my class, my biggest take aways have been on design patterns. Here are the big ones I’ve been playing with and understanding better:

  • Pseudo classical instantiation pattern
  • Prototypal Chaining
  • Function sharing
  • Models and Views with pure jQuery.

I may elaborate on these sometime, or if someone asks me to.

Lots of New Things: and a description of what its like to go to code school with some experience under your belt

Here is a list of things that have happened lately:

Finished my internship at Splurgy!

My final project for the company was building an internal statistics dashboard to help the marketing team’s job become easier.

I used d3 to build 5 graphs, displaying different items from the database (number of users per day, number of campaigns started, etc…).

I learned a lot of javascript, and a bit of SQL, plus I learned how to wrap it all in rails. It was awesome.

Then…

I decided to go to coding school at Catalyst!

For the past 4 days I have been here all day, every day.

Our weeks start on Fridays, with just Thursdays off…Insane amounts of time spent working.

Coming in, I was a bit nervous about the fact that others will be starting from 0 while I’ve already worked in a development shop. The main reason I was nervous is because Michi at Splurgy told me that I was probably hirable already, and to spend a sizable chunk of money after this on classes was unnecessary if my goal was getting a job.

What I know now, after a few days, is that I’ve never been able to spend so much sustained time coding.

Working in pairs with tight supervision has really prevented a lot of the frustration parts about working on problems yourself– getting stuck for an hour in a sink-hole of stackoverflow posts, not knowing how to really identify your issue, or not knowing what to focus on and having no one help hone you into your goals. To some degree, I had these things taken care of at my internship– but the team couldn’t really coddle me in the same way teachers I am paying can. Definitely a good environment to blaze ahead in.

I also know that having such a social atmosphere for coding means I am really learning how to verbalize the patterns and thought processes that go into problem solving in web development.

Even if I could have gone straight to a job, I wouldn’t be able to express myself well enough in an interview to get the job I really want. With some practice, I probably could have– but it’s hard to imagine ending up with as holistic a skill set as I am going to receive here.

Today we spent some time implementing basic data structures. I honestly feel like I didn’t have any true conception of what a data structure was until this talk, and now that I know, it feels simple.

I keep wanting to weigh “could I have done this myself?” against everything I’ve learned. And in these first few days– probably. But I wouldn’t have learned this as fast, or had as much fun.

I think this method of education should become more pervasive. It’s highly effective for most. I think I am an edge case, and have had to spend more time parsing whether it was a good choice. I think at the end I’ll be happy about it.