Hipsters Inc

Been Busy

Fri, 11 Apr 2008 04:00:00 +0000

I've been busy.

Sometime back in February or March, I took up tenure at Invite Media in Philadelphia. Remember all that pissing and moaning I was doing about startups in Philadelphia? Yeah. Turns out I was wrong. This operation is a full-out Paul Graham party. There's not all too many people working here, but rest assured they're all bell-curve-busting smart. The kind of people where your jaw drops just a bit more every day you work with them. I had to interview and hire a lot of people at my last job, and I'm kind of in awe that these guys were able to pull that off. I was certainly never able to do so.

It's a nice place. I take smoke breaks on the roof. I see a lot of stray balloons. Here's a picture.

The work I do is the kind of work I dream of doing in my free time. They buy all the food. They buy whatever beers I want to drink. Now all I have to is make a convincing argument for the need of data visualization for the scale of raw data we are crunching. One track mind? Perhaps.

We have interesting traffic patterns. These are really quick sketches, but here's how I see it in my mind:

And here's how I want other people to see it:

Or something close to that. People will sometimes call Tufte out for one reason or another. I often have a hard time disagreeing. He is a bit of an idealist. Regardless, visualization is the quickest and most efficient route between mounds of data and a concrete understanding. Often times, rules of business (read: over-the-top presentation) come in to play, and they can't be ignored on any practical scale. Tufte is a lever to pull -- an invaluable counterbalance you can't ignore while pursuing the flashier side of the art.

That's just kind of rambling garbage, though. I'm still piecing together my thoughts on the subject. I believe the top 10% of people who have honed the skill of properly visualizing massive sets of data are going to be in a pretty sweet boat in the coming years, and I've been spending a lot of time sharpening that axe.

But, mostly, there are a lot of posts I wanted to make, but felt that I shouldn't post them until remarking on the new job. A hasty, rambling post is sometimes needed.

New Toy: Scala and Processing

Wed, 23 Jan 2008 05:00:00 +0000

Now, I'm not claiming to be a Scala master, but hey. If I can half-heartedly try to tech people to code using Processing, maybe I can eat my own dog food. Scala, by the way, runs in the Java VM, and seems to integrate with Java mighty well. If functional languages make no sense to you, Scala is a nice stepping stone.

Aaaanyway...If you'd like to hop aboard my train of Scala education, here's how I'll be paying my way.

Path to Awesome

Snag Processing. Snag Scala.
Make a new directory. Copy core.jar from Processings's lib folder.
On a Unix-y system, run something like: export CLASSPATH=.:core.jar
If you're Windows, I think it goes: set CLASSPATH=.;core.jar -- but I could be wrong
Crack open a new file. Let's call it processingtest.scala. Punch in the following:

object ProcessingTest extends processing.core.PApplet {
  import processing.core._;

  override def setup(): unit = {
    size(400, 300, PConstants.P3D)
    noLoop
  }

  override def draw(): unit = {
    background(51)

    var i=0
    while(i < 100) {
      pushMatrix
      fill(random(255))
      translate(random(width), random(height), random(-30, 30))
      box(random(10, 70));
      popMatrix
      i += 1
    }
  }

  def main(args: Array[String]): unit = {
    var frame = new javax.swing.JFrame("Test")
    var applet = ProcessingTest
    frame.getContentPane().add(applet)
    applet.init
    frame.pack
    frame.setVisible(true)
  }
}

Copy/Paste

object ProcessingTest extends processing.core.PApplet {
  import processing.core._;

  override def setup(): unit = {
    size(400, 300, PConstants.P3D)
    noLoop
  }

  override def draw(): unit = {
    background(51)

    var i=0
    while(i < 100) {
      pushMatrix
      fill(random(255))
      translate(random(width), random(height), random(-30, 30))
      box(random(10, 70));
      popMatrix
      i += 1
    }
  }

  def main(args: Array[String]): unit = {
    var frame = new javax.swing.JFrame("Test")
    var applet = ProcessingTest
    frame.getContentPane().add(applet)
    applet.init
    frame.pack
    frame.setVisible(true)
  }
}

Run something along the lines of: scalac processingtest.scala && scala ProcessingTest

Lovely. If you take out that noLoop statement, you'll see that the ol' girl is actually pretty quick. As for me, the big thing I'm getting out of this: Mixins. Holy crap. That's going to be awesome. And the ability to move back and forth...man.

If/When this Scala Eclipse plugin gets semi-decent auto-completion, I'll probably just start screaming. Probably pretty loud. Auto-completion in my IDE is about the only reason I'll still defend Java.

Oh, and sorry if I owe you an email. I've been all over the place. She's-a-comin'

An Interesting Algorithm: Pattern Matching Mosaic

Fri, 18 Jan 2008 05:00:00 +0000

A week or so ago, this picture was making the rounds. A couple different people had smaller copies of it, but I was unable to find out who actually made it. Because the instances of the picture I saw were quite small, I assumed it was assembled by some crazy algorithm with a huge corpus of art to compare.

So, of course, I spent the next day or two thinking about what that algorithm would look like. My friend, John, lent me a number of really interesting Computer Vision books when he moved over the summer. I remembered reading a chapter on template matching. Turns out that won't really do the trick, but it's close.

Long story short: The original picture that was floating around was actually just planned out and painted, so I tried to implement my algorithm.

The test picture was taken by Steh. It's pretty high up on the list when you look for "interesting" photos tagged "face" on Flickr.

Anyway, the corpus of search images was entirely too small for this undertaking. However, after browsing through the output, I believe it's probably close to correct. If not, it's on the right path. If you click on the image above, you can see some neat things going on. For example, on both sides of her neck, the algorithm found hair in a similar alignment. Man. That sounded a bit creepier than I wanted.

But! The fun part! I've been looking for a good excuse to start playing with Hadoop for a while. I've gone through their grep example and all that, but that just doesn't give the kind of joy I was expecting. Hadoop, by the way, is more-or-less a Java implementation of Google's Map Reduce algorithm. And, it's sweet.

There is a level of nirvana you never know until you get 12 CPUs absolutely pegged. I can only imagine the euphoria of having access to a proper cluster.

So, I'm going to keep working on this. Right now, I'm trying to have transforms not love contrast so damn much. I think a once-over with the ol' profiler could also do some good.

Tilley

Sun, 13 Jan 2008 05:00:00 +0000

So, I read that The New Yorker was having some manner of cover competition. Seeing as my illustrative talents end with Wolf Blizter's glorious, glorious face, I though I'd try something a bit more algorithmic. Whelp.

It's all the New Yorker covers arranged chronologically. There are important historical events along the left spine, and little lines connecting them to the proper issues. See?

I know it doesn't look like I spent the whole weekend on it, but hey. We all make mistakes. Did you know there's somewhere around 3,700 issues of the New Yorker? Did you know they skip a semi-random number of issues per year in some semi-random fashion? Given a flat set of images of issue covers, how would you figure out which issues fell on which dates? I'll give you a hint: It's really fun.

Getting some flocking out of my system

Fri, 04 Jan 2008 05:00:00 +0000

I felt like blowing off some thinking-too-hard steam by whipping up a little gravity/flocking sketch. After using Processing for a little under a year now, it's amazing how quickly sketches like this come together. For example, I have a Target3D interface with a couple different implementations...this one just uses some Brownian motion. It gives the particles something to steer towards. There's the hacked up Vector3 class, and an often-reached-for RingList.java. It's a few lines of glue, and you're off.

It's just nice to crack open Eclipse, and go straight to tweaking The Interesting Stuff.

I tried focusing only on color for the first one.

Then, I tried focusing only on shape for the second fellow. Totally inspired by Robert Hodgin's batshit prints shown here (the non-butterfly ones).

That shallow depth of field really gets me going. These are both the same chunk of code, with the same random seed, so there's some spot where the line up, but I didn't really feel like tracing that down. I have a feeling I edited that out of the color one.

Yeah, it's quick. I think I'm entirely incapable of waiting longer than an evening for something to render.

Oh, yeah. And careful with that RingList. It's barely implemented. I just thought I'd share. I'm using it for the tails in this code.

Wolf Blitzer Speaks

Fri, 28 Dec 2007 05:00:00 +0000

Ladies and Gentleman, your 2008 wallpaper.

Big Squiggle

Mon, 24 Dec 2007 05:00:00 +0000

I was working on tracking down the memory leak in P5Sunflow, so I needed some large input. Well, that and I wanted to dust off the printer from the days of yore. So, what would a ball of squiggles look like when the target is 24 by 24 inches at 300 DPI? Well...

And even the detailed box is scaled down quite a bit. Here's a neat Google Maps style thing to play with:

Same thing, full page.

2,500 ribbons. The scene file was just under 300M. It took Sunflow about 7 hours to do the whole thing. I have to say, that is a fine piece of software. I was watching its memory very closely. I gave it 2 gigs of RAM to use, it shot up to 908M after a minute or two, and just stayed there. I don't know if you've ever seen the tell-tale sawtooth of your average Java program's memory usage, but this wasn't it. Sunflow, literally, throughout the 7 hours, didn't deviate more than a megabyte. There is truly some magic going on in there.

So, I just printed this fella out. After seeing it on paper, looking at it on a computer feels pretty stupid. There's something neat about huge high-resolution prints. You can so easily juggle the scale and the detail at the same time. It kind of seems impossible on a monitor. That, and anything will look good on watercolor paper. You can quote me on that.

Also, I fiddled with the RSS feed the other day. Sorry if I spammed you.

A Little Freaky

Thu, 20 Dec 2007 05:00:00 +0000

I just got an email from a fine fellow by the name of Drew. He sent me a link to one of his pieces, Not a problem.

Fucking freaky. There's something about that one in particular that seems so personal to me.

That Internet. That shit'll get you every time. Whenever you think you did something neat, someone will always remind you that you're just dicking around.

If none of this makes any sense to you, it's in reference to yesterday's post.

Reverse Engineering Visuals

Wed, 19 Dec 2007 05:00:00 +0000

My obsessions ebb and flow like the goddamn moon was tugging at them. It's not that I don't have the attention span, it's just that the further you dig into something, the more you discover. The deeper you dive, the greater the chance you'll find some gem that puts everything else on the back burner. Maybe not the back burner, but certainly applies a gaussian blur with a sizable kernel.

I've become enthralled with reverse engineering visuals. You know, looking at some nondescript collection of pigment or concrete, and feeling an overwhelming need to pick it apart. Not in a critical sense, just in an algorithmic sense.

During a cleaning spree in the not-too-distant past, I started scanning my old notebooks so I could throw them away. The high-school era ones were filled with squiggles like the following.

They're pretty ugly, and scream of being bored to tears, but I can't stop myself from dissecting their rules. The bands have a particular z-ordering. The curves have a pattern. The pen on paper produces a certain bit of semi-random noise. The kind of thing you start drawing on napkins and discussing with your girlfriend over dinner.

As silly of an exercise as it may be, it needed to get worked out. And, of course, once you've approximated the algorithm, you can take it far beyond the original.

When it comes to free time, I'm a big fan of taking something only as far as it makes you happy. This is such an example. It's not exact, but I've gotten the enjoyment of looking at it.

I'm thinking about writing a book about reverse engineering things. Mainly visuals. Not because I have any talent for it, but because I find it so interesting. It also seems to be one of the few things I've ended up working on that I can explain to anyone at all.

I don't really want to sell a book, I just want to write it. Cohesively. Seeing as my writing skills are so poor, I'd need an editor. I can dump information to a certain extent, but I'd never be able to assemble it.

For example, I wear a lot of hats at my current place of employment. I gave lookin' around notice pretty recently. Among other things, I started assembling a huge document of everything one would need to know to do my job. It's an absolute mess. All cross-references and bullshit.

Ramble.

P5Sunflow Things

This is the first project where I've had to eat my own dog food in regards to P5Sunflow. Let it be known that, embarrassingly enough, it leaks massive amounts of memory between frames. If you're rendering one frame, there's not too big of a difference. If you have rendered multiple in one shot, I apologize profusely. That slightly-more-than-linear growth in your render time wasn't ray-tracing magic, it was your garbage collector. My bad.

Another glaring bug...if you were getting out-of-bounds exceptions occasionally, that's also my fault. After looking at the code again, it's a really, really obvious bug that no one could have ever conceivably written. The most astonishing part is that it ran at all.

I wanted to have a lot more glitter and glitz when moving the code from proof-of-concept to 0.0.1, but this is going to have to be a premature release. I'm going to try to find the time this weekend to track down the former. Observe! Subversion prowess! Regardless, I thank you for your enthusiasm up to this point.

Meg's Pretty Awesome

Sun, 16 Dec 2007 05:00:00 +0000

So, after a long night of drinking and juvenile fun with friends, I head back to Meg's place. We're just sitting in her room settling down. I'm at her desk with a pen trying to draw her some Perlin noise displaced line thing I was screwing with earlier in the day. Romantic, no?

Anyway, she suddenly springs into action. "Hold on! I made you something!"

And she hands me this:

So, I get a good chuckle just because it's some silly thing. I'm playing with it in my hands, and she's still giving me a look like I'm missing something. She says "Look at the end."

A Markov Chain. I probably woke up the neighbors with the laugh that followed. Sorry fellas, she's spoken for.

Graffiti Input Particles

Mon, 10 Dec 2007 05:00:00 +0000

As noted all too often, I'm working a (what turns out to be very large) ruleset for graffiti. I'm not a huge fan of graffiti or anything, but it seems interesting. If you see generative graffiti on television, you'll immediately notice that it looks, well, generative. It's this neat little corner of art that people just can't seem to get right. How do the angles interact with each other? How often to the character glyphs alternate which angle the lean? What is the exact probability that a terminating line will end with an arrow?

I've been measuring images for the past couple weeks trying to get the ruleset down.

Anyway, to test this code, I needed some incoming paths. I set up a quick particle system that respondeds to music. Turns out, I liked how that looked on its own. This isn't supposed to look like graffiti, mind you. Just the shapes and motions being described.

Unrelated: Vimeo is really nice. The UI is incredible. They'll even let you upload crap in HD (as the above video has been). They have forums where they respond (and sometimes fix) your problems. Often within the day. I can't get that from places I pay money to. I don't have a point. I was just impressed.

Also Unrelated: I'll be seeking employment around the new year.