Hipsters Inc

I've been busy.

Sometime back in February or March, I took up tenure at Invite Media in Philadelphia. Remember all that pissing and moaning I was doing about startups in Philadelphia? Yeah. Turns out I was wrong. This operation is a full-out Paul Graham party. There's not all too many people working here, but rest assured they're all bell-curve-busting smart. The kind of people where your jaw drops just a bit more every day you work with them. I had to interview and hire a lot of people at my last job, and I'm kind of in awe that these guys were able to pull that off. I was certainly never able to do so.

It's a nice place. I take smoke breaks on the roof. I see a lot of stray balloons. Here's a picture.

The work I do is the kind of work I dream of doing in my free time. They buy all the food. They buy whatever beers I want to drink. Now all I have to is make a convincing argument for the need of data visualization for the scale of raw data we are crunching. One track mind? Perhaps.

We have interesting traffic patterns. These are really quick sketches, but here's how I see it in my mind:

And here's how I want other people to see it:

Or something close to that. People will sometimes call Tufte out for one reason or another. I often have a hard time disagreeing. He is a bit of an idealist. Regardless, visualization is the quickest and most efficient route between mounds of data and a concrete understanding. Often times, rules of business (read: over-the-top presentation) come in to play, and they can't be ignored on any practical scale. Tufte is a lever to pull -- an invaluable counterbalance you can't ignore while pursuing the flashier side of the art.

That's just kind of rambling garbage, though. I'm still piecing together my thoughts on the subject. I believe the top 10% of people who have honed the skill of properly visualizing massive sets of data are going to be in a pretty sweet boat in the coming years, and I've been spending a lot of time sharpening that axe.

But, mostly, there are a lot of posts I wanted to make, but felt that I shouldn't post them until remarking on the new job. A hasty, rambling post is sometimes needed.

Now, I'm not claiming to be a Scala master, but hey. If I can half-heartedly try to tech people to code using Processing, maybe I can eat my own dog food. Scala, by the way, runs in the Java VM, and seems to integrate with Java mighty well. If functional languages make no sense to you, Scala is a nice stepping stone.

Aaaanyway...If you'd like to hop aboard my train of Scala education, here's how I'll be paying my way.

Path to Awesome

• Snag Processing. Snag Scala.
• Make a new directory. Copy core.jar from Processings's lib folder.
• On a Unix-y system, run something like: export CLASSPATH=.:core.jar
• If you're Windows, I think it goes: set CLASSPATH=.;core.jar -- but I could be wrong
• Crack open a new file. Let's call it processingtest.scala. Punch in the following:

object ProcessingTest extends processing.core.PApplet {
  import processing.core._;

  override def setup(): unit = {
    size(400, 300, PConstants.P3D)
    noLoop
  }

  override def draw(): unit = {
    background(51)

    var i=0
    while(i < 100) {
      pushMatrix
      fill(random(255))
      translate(random(width), random(height), random(-30, 30))
      box(random(10, 70));
      popMatrix
      i += 1
    }
  }

  def main(args: Array[String]): unit = {
    var frame = new javax.swing.JFrame("Test")
    var applet = ProcessingTest
    frame.getContentPane().add(applet)
    applet.init
    frame.pack
    frame.setVisible(true)
  }
}

• Run something along the lines of: scalac processingtest.scala && scala ProcessingTest

Lovely. If you take out that noLoop statement, you'll see that the ol' girl is actually pretty quick. As for me, the big thing I'm getting out of this: Mixins. Holy crap. That's going to be awesome. And the ability to move back and forth...man.

If/When this Scala Eclipse plugin gets semi-decent auto-completion, I'll probably just start screaming. Probably pretty loud. Auto-completion in my IDE is about the only reason I'll still defend Java.

Oh, and sorry if I owe you an email. I've been all over the place. She's-a-comin'

A week or so ago, this picture was making the rounds. A couple different people had smaller copies of it, but I was unable to find out who actually made it. Because the instances of the picture I saw were quite small, I assumed it was assembled by some crazy algorithm with a huge corpus of art to compare.

So, of course, I spent the next day or two thinking about what that algorithm would look like. My friend, John, lent me a number of really interesting Computer Vision books when he moved over the summer. I remembered reading a chapter on template matching. Turns out that won't really do the trick, but it's close.

Long story short: The original picture that was floating around was actually just planned out and painted, so I tried to implement my algorithm.

The test picture was taken by Steh. It's pretty high up on the list when you look for "interesting" photos tagged "face" on Flickr.

Anyway, the corpus of search images was entirely too small for this undertaking. However, after browsing through the output, I believe it's probably close to correct. If not, it's on the right path. If you click on the image above, you can see some neat things going on. For example, on both sides of her neck, the algorithm found hair in a similar alignment. Man. That sounded a bit creepier than I wanted.

But! The fun part! I've been looking for a good excuse to start playing with Hadoop for a while. I've gone through their grep example and all that, but that just doesn't give the kind of joy I was expecting. Hadoop, by the way, is more-or-less a Java implementation of Google's Map Reduce algorithm. And, it's sweet.

There is a level of nirvana you never know until you get 12 CPUs absolutely pegged. I can only imagine the euphoria of having access to a proper cluster.

So, I'm going to keep working on this. Right now, I'm trying to have transforms not love contrast so damn much. I think a once-over with the ol' profiler could also do some good.

So, I read that The New Yorker was having some manner of cover competition. Seeing as my illustrative talents end with Wolf Blizter's glorious, glorious face, I though I'd try something a bit more algorithmic. Whelp.

It's all the New Yorker covers arranged chronologically. There are important historical events along the left spine, and little lines connecting them to the proper issues. See?

I know it doesn't look like I spent the whole weekend on it, but hey. We all make mistakes. Did you know there's somewhere around 3,700 issues of the New Yorker? Did you know they skip a semi-random number of issues per year in some semi-random fashion? Given a flat set of images of issue covers, how would you figure out which issues fell on which dates? I'll give you a hint: It's really fun.

I felt like blowing off some thinking-too-hard steam by whipping up a little gravity/flocking sketch. After using Processing for a little under a year now, it's amazing how quickly sketches like this come together. For example, I have a Target3D interface with a couple different implementations...this one just uses some Brownian motion. It gives the particles something to steer towards. There's the hacked up Vector3 class, and an often-reached-for RingList.java. It's a few lines of glue, and you're off.

It's just nice to crack open Eclipse, and go straight to tweaking The Interesting Stuff.

I tried focusing only on color for the first one.

Then, I tried focusing only on shape for the second fellow. Totally inspired by Robert Hodgin's batshit prints shown here (the non-butterfly ones).

That shallow depth of field really gets me going. These are both the same chunk of code, with the same random seed, so there's some spot where the line up, but I didn't really feel like tracing that down. I have a feeling I edited that out of the color one.

Yeah, it's quick. I think I'm entirely incapable of waiting longer than an evening for something to render.

Oh, yeah. And careful with that RingList. It's barely implemented. I just thought I'd share. I'm using it for the tails in this code.

Ladies and Gentleman, your 2008 wallpaper.

I was working on tracking down the memory leak in P5Sunflow, so I needed some large input. Well, that and I wanted to dust off the printer from the days of yore. So, what would a ball of squiggles look like when the target is 24 by 24 inches at 300 DPI? Well...

And even the detailed box is scaled down quite a bit. Here's a neat Google Maps style thing to play with:

Same thing, full page.

2,500 ribbons. The scene file was just under 300M. It took Sunflow about 7 hours to do the whole thing. I have to say, that is a fine piece of software. I was watching its memory very closely. I gave it 2 gigs of RAM to use, it shot up to 908M after a minute or two, and just stayed there. I don't know if you've ever seen the tell-tale sawtooth of your average Java program's memory usage, but this wasn't it. Sunflow, literally, throughout the 7 hours, didn't deviate more than a megabyte. There is truly some magic going on in there.

So, I just printed this fella out. After seeing it on paper, looking at it on a computer feels pretty stupid. There's something neat about huge high-resolution prints. You can so easily juggle the scale and the detail at the same time. It kind of seems impossible on a monitor. That, and anything will look good on watercolor paper. You can quote me on that.

Also, I fiddled with the RSS feed the other day. Sorry if I spammed you.

I just got an email from a fine fellow by the name of Drew. He sent me a link to one of his pieces, Not a problem.

Fucking freaky. There's something about that one in particular that seems so personal to me.

That Internet. That shit'll get you every time. Whenever you think you did something neat, someone will always remind you that you're just dicking around.

If none of this makes any sense to you, it's in reference to yesterday's post.