Rogueleaderr

Postgres Fuzzy Search Using Trigrams (+/- Django)

Thu, 16 May 2013 10:00:31 -0400

When building websites, you’ll often want users to be able to search for something by name. On LinerNotes, users can search for bands, albums, genres etc from a search bar that appears on the homepage and in the omnipresent nav bar. And we need a way to match those queries to entities in our Postgres database.

At first, this might seem like a simple problem with a simple solution, especially if you’re using the ORM; just jam the user input into an ORM filter and retrieve every matching string. But there’s a problem: if you do

Bands.objects.filter(name="beatles")

You’ll probably get nothing back, because the name column in your “bands” table probably says “The Beatles” and as far as Postgres is concerned if it’s not exactly the same string, it’s not a match.

Users are naturally terrible at spelling, and even if they weren’t they’d be bad at guessing exactly how the name is formatted in your database. Of course you can use the LIKE keyword in SQL (or the equivalent ‘__contains’ suffix in the ORM) to give yourself a little flexibility and make sure that “Beatles” returns “The Beatles”. But 1) the LIKE keyword requires you to evaluate a regex against every row in your table, or hope that you’ve configured your indices to support LIKE (a quick Google doesn’t tell me whether Django does that by default in the ORM) and 2) what if the user types “Beetles”?

Well, then you’ve got a bit of a problem. No matter how obvious it is to human you that “beatles” is close to “beetles”[1], to the computer they’re just two non-identical byte sequences. If you want the computer to understand them as similar you’re going to have to give it a metric for similarity and a method to make the comparison.

There are a few ways to do that. You can do what I did initially and whip out the power tools, i.e. a dedicated search system like Solr or ElasticSearch. These guys have notions of fuzziness built right in (Solr more automatically than ES). But they’re designed for full-text indexing of documents (e.g. full web pages) and they’re rather complex to set up and administer. ES has been enough of a hassle to keep running smoothly that I took the time to see if I could push the search workload to Postgres, and hence this article.

Unless you need to do something real fancy, it’s probably overkill to use them for just matching names.

Instead, we’re going to follow Starr Horne’s advice and use a Postgres EXTENSION that lets us build fuzziness into our query in a fast and fairly simple way. Specifically, we’re going to use an extension called pg_trgm (i.e. “Postgres Trigram”) which gives Postgres a “similarity” function that can evaluate how many three-character subsequences (i.e. “trigrams”) two strings share. This is actually a pretty good metric for fuzzy matching short strings like names.

To use pg_trgm, you’ll need to install the “Postgres Contrib” package. On ubuntu:

sudo apt-get install postgres-contrib

**WARNING: THIS WILL TRY TO RESTART YOUR DATABASE**

then pop open psql and install pg_trgm (NB: this only works on Postgres 9.1+; Google for the instructions if you’re on a lower version.)

psql

CREATE EXTENSION pg_trgm;

\dx # to check it's installed

Now you can do

SELECT *

FROM types_and_labels_view

WHERE label % 'Mountain Goats'

ORDER BY similarity(label, 'Mountain Goats')

DESC LIMIT 100;

And out will pop the 100 most similar names. This will still take a long time if your table is large, but we can improve that with a special type of index provided by pg_trgm:

CREATE INDEX labels_trigram_index ON types_and_labels_table USING gist (label gist_trgm_ops);

CREATE INDEX labels_trigram_index ON types_and_labels_table USING gin (label gin_trgm_ops);

(GIN is slower than GIST to build, but answers queries faster.

That’ll take a while to build (possibly quite a while), but once it does you should be able to fuzzy search with ease and speed. If you’re using Django, you will have to drop into writing SQL to use this (until someone, maybe you, writes a Django extension to do this in the ORM.)

And as a frustrating finishing note, my attempt to implement this on LinerNotes was not ultimately succesful. It seems that that index query performance is at least O(n) and with 50 million entities in my database queries take at least 10 seconds. I’ve read that performance is great up to about 100k records then drops off sharply from there. There are some apparently additional options for improving query performance, but I’ll be sticking with ElasticSearch for now.

[1] Sorry, Googlebot! Not sorry, Bingbot.

Song of the day, Bored Nothing —“Get Out of...

Wed, 24 Apr 2013 20:14:20 -0400

Song of the day, Bored Nothing —“Get Out of Here”. HT @niralshah

How to notify/email yourself when an EC2 instance terminates

Wed, 24 Apr 2013 16:23:17 -0400

I make pretty heavy use of EC2 spot instances, which as you know can terminate at any time with no warning.

In order to get my spots back up ASAP, I’d like be notified when they terminate.

This turned out to be much harder than I expected. I thought I’d be able to add a simple script that would send me an email when the instance shuts down (Amazon is nice enough to send a shutdown command on termination instead of just pulling the plug.)

But that approach has a couple of problems.

First, you can’t easily send email from EC2 instances (because of spammers) and have to manually get instances whitelisted by elastic IP which is a pain.

Second, it’s not the easiest thing to write a shutdown hook in Linux.

So, here’s a solution for both of those problems.

The email problem

Amazon has a service called CloudWatch that seems great for this, except that it can only monitor metrics emitted by running instances. So you can’t set it to alert you on a system shutdown, nor on a metric polling failure because it randomly misses packets all the time.

So the solution is a different Amazon service called SNS (simple notification service) that will let you trigger an event that can be configured to send you an email. So we’re going to write a script that tells SNS to send us an email.

(SNS is free for ~200k requests/month, so unless you’re planning on doing something nuts this approach should have no marginal costs.)

To do that, you first need to set up a “topic” in SNS. Go to the SNS dashboard and

1) click “create topic”.
2) Create a topic called “instance_down” or whatever you like.
3) Click the topic and click “create subscription”
4) Choose protocol “email” and enter your email address as the endpoint

To use SNS in a script, we need the AWS command line tools.

If you already have pip installed, just

pip install awscli

Then:

touch ~/.awsconfig
emacs ~/.awsconfig

Make it say:

[default]
AWS_ACCESS_KEY_ID=« YOUR AWS ID»
AWS_SECRET_ACCESS_KEY=« YOUR AWS SECRET KEY»
region=« YOUR REGION »

Create an init.d script:

sudo emacs /etc/init.d/ec2-shutdown

Make it say:

#! /bin/sh
### BEGIN INIT INFO
# Provides: ec2-terminate
# Required-Start: $network $syslog
# Required-Stop:
# Default-Start:
# Default-Stop:
# Short-Description: restart
# Description: send termination email
### END INIT INFO
#

export AWS_CONFIG_FILE={{ YOUR CONFIG FILE }}
export AWS_DEFAULT_REGION={{ YOUR REGION }} # config file not picking up region for some reason
sudo -E aws sns publish —topic-arn {{ YOUR SNS ARN }} —message “ec2 ser\
ver {{ YOUR IDENTIFIER }} went down at $(date)”
sleep 3 # make sure the message has time to send

exit 0

That script will tell SNS to send you a email saying your server is down and giving the time. You can customize the message however you like.

The Shutdown Script Problem

So how do we make this run on shutdown?

We’re going to use init.d scripts. It’s taken me a little while to get my head around how this works, but in a nutshell…

Linux has a concept of “runstates”, which include things like “shutdown” and “logged in”. You can tell Ubuntu to run shell scripts when it changes into a runstate by placing scripts in certain folders in side of /etc. The two states that concern us are rc0 and rc6, i.e. “shutdown” and “reboot”, which correspond to folders /etc/rc0.d and /etc/rc6.d respectively.

Ubuntu has a command line tool, update-rc.d that will automatically symlink scripts into the appropriate folders depending on the CL paramaters you pass it. It’s all a bit complicated, but all you need to do here is:

sudo update-rc.d ec2-shutdown start 10 0 6 .

(This says run the script on entering run states 0 and 6, shutdown and reboot. And put it 10th in the list of things to do)

And that should do it! Now your instance should send you an email whenever it shuts down, terminates or reboots. I’m not sure how limit to just EC2 termination, but please comment if you know!

If you have any problems, leave a comment.

And if you find this useful, consider following me on Twitter.

Links to all the parts of my "Everything About Economics" Blog Series

Mon, 22 Apr 2013 00:22:00 -0400

Back linking has gotten unwieldy, so I’m going to just update links on this page to all the parts from now on:

Intro: The future of the economy, in bite-sized chunks.

Part 1: What is An Economy?

Part 2(a): Rich Hypothetical Society, Poor Hypothetical Society

Part 2(b): Productivity is Prosperity

Part 3: How Productivity Grows

More Thoughts on Free Will

Sun, 21 Apr 2013 23:58:46 -0400

A couple of weeks ago, I left a comment on Albert Wenger’s excellent blog “Continuations” about Free Will. The comment was rather long, so I reposed it here on my blog. Somewhat amazingly, Albert read my comment and left a thoughtful reply.

I’m not sure of the etiquette of reposting Albert’s comment, but to summarize he said that my argument was off-base because the “information layer” I described is just an abstraction and doesn’t interact with the physical world. No interaction means no influence, and that means that the stimulus->thought->action sequence remains a deterministically closed loop.

I, logorrheic that I am, responded with another rather long comment. This is that comment.

Well, actually this is:

If it’s true that the “information layer is entirely abstract” (i.e. completely non-interacting with the tangible world) then your whole argument is clearly right.

But I’m not sure that premise is true. It seems that the informational world is constantly interacting with the physical world in the form of “physical laws being followed.”

Metaphorically, particles in the world are “data” and the laws of physics (i.e. the information layer) are “source code.” Every particle interaction is (metaphorically) a computation. Now all of our observation to date suggests that the “source code” of the universe is stable and fixed. But any honest physicist will tell you that our (extremely elegant) mathematically descriptions of how objects have behaved in the past tell us nothing about why they behaved that way.

I raised GEB to show that “laws” aren’t necessarily well-behaved in edge cases.[1] And so similarly, the fact that we’ve observed that some parts of the “Ur-program” work one way does NOT demonstrate conclusively that there aren’t other parts of the program that introspectively alter the Ur-program’s source code and lead to measurable, experimentally verifiable physical effects.

You raise one potential “point of strangeness” in your PS — quantum waveform collapse. My knowledge of quantum mechanics and neuroanatomy is limited, but my best understanding is that neural interactions are sensitive enough that quantum effects can make the difference between whether a neuron fires or not. And we know from computers (cf. my Caesar’s cipher example above) that changing one bit in a computation chain can radically alter the downstream result, so any slight (non-random!) bias in the quantum coin-flip could be enough to completely alter behavior. (And as far as I know would not violate the conservation of energy or any other physical laws except our observation that quantum collapse is usually random.)

Could patterns of thought directly alter the wave-form collapse? Well, we already know that the quantum world is altered by observation per se. And conscious introspection is fundamentally self-observation. So it is possible that when the wetware finds itself in physical states that mirror certain “information patterns” (e.g. “contemplating two choices”), the presence of those patterns triggers an alteration in the source code that directly influences the way quantum collapse happens in connected synapses, leading a neuron to fire that otherwise wouldn’t. And from the GEB problem, we might expect the “meta program” to be especially poorly behaved (perhaps even fundamentally indeterminate, i.e. “free”) in this circumstance.

So is this explanation likely? That’s hard to say. It’s neither confirmed nor contradicted by existing data. It’s counter-intuitive, but so is every other part of quantum mechanics.

But best of all, this theory is testable. We just need to carefully measure the statistical in vivo quantum behavior of synapses during decision-making and I can be proven wrong once and for all. :)

[1] I admit I’m not sufficiently demonstrating that “logic laws” work the same ways as physical laws but there is certainly a remarkable symmetry. It doesn’t seem at all fundamentally necessary that physical laws should be mathematically expressible and elegant. I’m happy to write more on this point if anyone is curious.

Everything you need to know about economics in 400 words, Part 3

Fri, 05 Apr 2013 14:59:00 -0400

It seems I only write on this topic while flying. So I guess you can expect this series to wrap up sometime in 2015.

But to recap, last time I told you that productivity is prosperity. So all increases in per capita wealth come from increases in the amount of economic goods a single worker can produce in an hour.[1]

So how does productivity increase? Let’s start with the obvious. We get the stuff we have by making complicated stuff out of simpler stuff. We start with iron and chemicals, apply some process over a period of time, and end up with steel. Visually:

(1 unit of Time + 1 unit of Iron + 1 unit of Chemicals) => 1 unit Steel

The productivity of this process is simply the ratio of inputs to outputs. If we rejiggered the above process to produce 2 units of steel from the same input, we would have doubled productivity. Ditto if we halved the amount of time.

Physical productive processes are closely analogous to computer algorithms. We could just as easily talk about the “productivity” of a bitcoin mining operation in terms of how many BC a computer can generate per hour.

So how do we get more productivity out of our processes (which, to reiterate, is the only way we get richer overall)? Well, just like with computer algorithms we have two options:

Do the same thing faster (e.g. get a faster CPU)

Swap in a better algorithm (e.g. use quick sort instead of bubble sort)

“Do more faster” is an appealing tagline, but unfortunately human land-speed doesn’t follow Moore’s law. So our best option is to find better algorithms for our productive processes.

So transitively, the best way for society to get richer is to devise and select better processes. By and large, we call process improvements technology. The cotton gin and the Bessemer process were technologies that allowed radically more efficient transformation of a raw commodity into a finished product.

One of the most important stories of the last 30 years has been the way that computers have inserted themselves deeply into our economy’s various productive processes. But most of what we’ve seen so far are “do the same thing faster” improvements. The real productivity revolution will come once computers consistently help us choose better algorithms for our processes and for our lives.

[1] Okay, all increases except one-off “gold strikes” that shower us with random natural riches.

Is Free Will a Buffer Overflow?

Tue, 02 Apr 2013 13:55:00 -0400

[Once again, I’ve been inspired by a post on Albert Wenger’s blog, this time about whether or not free will exists.]

I’ve spent a long time thinking about this one as well. I think you’re probably right, but here’s the best alternative I’ve been able to come up with. These ideas are heavily inspired by having finally finished Gödel, Escher, Bach.

Basically, it’s clearly true that determinism holds over the space of physical objects if you assume that a continuous set of physical laws governs all physical interactions. If my brain is just a physical object, then the arrangement of its atoms at present is exclusively a function of their arrangement in the past.

But how justified are we in making that assumption about the continuity and coverage of physical laws?

Let’s start with coverage: it’s actually not hard to think of something that doesn’t interact with the physical world in quite the same way as an apple — a computer program.

Yes, a program is represented by marks on a disk, and yes it is executed by a set of magnets that write and erase those marks using electricity routed through a CPU. But ultimately those purely physical components produce an informational state projected through my screen. And as I write I am interacting not only with the keys of my keyboard but also but with the abstract state of the system. And the state per se has causal power: perform a Caesar cipher on your blog post and it becomes virtually meaningless to me despite having extremely similar statistical properties and on-disk representation.

So then where does the meaningfulness of the program state reside? Can we point to a memory address that distinguishes “meaningful” from not meaningful (even in principle)? Can we point to an atom in my brain that makes the text meaningful to me, even though it would remain meaningful to you even if my brain we destroyed (and vice versa) but become meaningless if all brains were destroyed?

The point here is that even in closed physical systems, there is an information “layer” that has both its own rules (i.e. “logic”) and yet tangibly influences the physical world. This is the layer where the program-ness of a program resides, or the law-ness of physical laws, or the meaningful-ness of a sentence.

Normally this layer is well behaved — I encode my experiences into words that you understand; when fundamental particles interact they “check” which laws to follow and then they follow them.

But the layer has known flaws, specifically when self-reference enters the picture. Gödel’s incompleteness theorem establishes that there are questions which can be precisely formulated but not decided, e.g whether the statement “this statement is a lie” is true or not (or the somewhat related question of whether an arbitrary computer program will ever terminate.) These questions aren’t undecidable because of a deficit in our ability to measure (a la Heisenberg uncertainty) but rather because of an irresolvable quirk in the laws of the information layer.

The interesting thing about consciousness (the domain within which free will would presumably operate) is that it is deeply and inherently self-referential. The very exercise of writing this comment is an example of an information state reflecting on its own statefulness.

Now let’s circle back to the question of consistency in physical laws. Usually it seems like the surest bet: no one has ever been fired for predicting that an apple will obey gravity.

But when we start talking about the way that particles in our brain interact with the information layer (in ways we understand only loosely, at best) and we focus in on an area (self-reference) where the information layer is already know to misbehave, how confident can we be that the rules we’ve observed elsewhere operate the same way here?

Or more colorfully, how certain can we free will isn’t some sort of buffer overflow in the operating system of the universe?

[Edit: if you like this post, please upvote on Hacker News! Look for the post titled “Is Free Will a Buffer Overflow?”]

To welcome in April, here’s a playlist of my favorite...

Mon, 01 Apr 2013 19:02:19 -0400

To welcome in April, here’s a playlist of my favorite songs I found in March. Some extra SXSW goodness in this one.

The Game Theory of Google Reader

Sun, 31 Mar 2013 15:57:58 -0400

Google Reader is one of top five most-used websites. I literally always have a reader tab open in Chrome. So when I heard that Reader is shutting down on July 1st, I was like

But after a while I was like

And now I’m like…

…because I’ve concluded that I’m glad Google is shutting down Reader.

Why, dear reader? Because of Game Theory (obviously)! By shutting down Reader, Google is breaking a bad Nash Equilibrium which has been hobbling the progress of web content distribution.

Whoa, what does that mean?

Well, “Game Theory” is simply a mathematical examination of the ideal strategy for playing games. Unsurprisingly it often applies to business and other competitive situations.

Let’s consider the canonical example Game Theorists love to use: “the prisoner’s dilema”. In this scenario, the cookie monster and his brother (Charleston) have both been arrested for, you guessed it, home invasion and murder. The cops put them into two separate cells and start questioning them.

“Now look, cookie monster, I’m not going to lie. Our evidence here is weak. If neither of you confess then we’re going to have to take this to trial, and you might both get off. But I’ll offer you a deal. If you testify against Charleston, we’ll give you immunity and five cookies. Unless, of course, he testifies against you in which case you’ll be learning to count to twenty to life.”

The police say the same thing to Charleston. So what is the cookie monster to do? He has two choices: “cooperate” with Charleston and go to trial, or “defect” and rat him out in exchange for sweet cookies (or long jail time if Charleston also defects.) Charleston has the same choices.

To figure out what both should do, we can make a simple matrix:

Cookie Monster is Player A, Charleston is Player B, and the numbers in each square are the number of cookies each gets in each situation, respectively. (No, the numbers don’t match up; I’m borrowing the diagram.)

Obviously, both monsters are better off if they both cooperate. But that won’t happen, at least not assuming that both care exclusively about maximizing their individual cookie intake.

Why not? Remember that our monsters are in separate rooms and cannot coordinate. Also, notice that each monster is individually best off in the situation where the other monster cooperates but he defects. So if either monster suspects that the other one will be generous and cooperate, his individual narrowly rational best option is to defect. And so he will.

Now the phrase “Game Theory” might quite appropriately make you think of the movie “A Beautiful Mind”. And if you saw that movie, you might think that Russell Crowe (a.k.a. “John Nash”) got academically famous for figuring out how to use game theory to pick up women in bars. But actually he got famous for inventing the concept of a “Nash Equilibrium”.

To over-simplify, a Nash Equilibrium is a set of player choices in a game (e.g. “Cookie monster -> defect”, Charleston -> defect” in the prisoner’s dilema) that will produce an outcome which neither player can improve by changing his choice unilaterally.

Take another look at the prisoner’s dilema diagram. The bottom right square is a Nash Equilibrium because, even though it is the worst outcome it’s the best that either player can do for himself. If Charleston thinks cookie monster is going to defect, he makes himself even worse off by cooperating. Same for cookie monster.

So in games where a Nash Equilibrium exists, we can generally expect that it will come to pass.[1][2]

So how does this all apply to Google Reader? Well, the existence of Google Reader has created and trapped us in a bad Nash Equilibrium. In this case, Google isn’t actually a player. Google is literally and figuratively out of the game — they haven’t cared about Google Reader in years and have let it wither on the vine.[3]

Instead, the players are consumers of RSS content and developers of alternative RSS readers.

The developers have two choices,

develop a new, better reader
do nothing, and let people continue to use Google

And the consumers have two choices:

switch to an alternative
keep using Google reader

We end up with something that looks an awful lot like the prisoner’s dilema, but with a few key differences.

It’s still the case that everyone is better off if consumers and developers cooperate on moving to a new reader. But it’s no longer the case that one side gains because the other loses. This is actually a game with two Nash Equilibria (develop, switch) and (don’t develop, don’t switch.) And instead of being played once in a jailhouse, this decision process is repeated every day.

Games with multiple equilibria (which I am of course oversimplifying here) introduce a fascinating question: “given that there’s a better state which both sides can agree is better for them individually how can move to that better world?” Remember that in a Nash Equilibrium, neither side can improve her outcome by changing her move unilaterally. But if the sides can coordinate then it’s as simple as agreeing to change their decisions at the same time.

But in the real world of RSS readers, developers can’t practically coordinate en masse with consumers. Each has to speculate in isolation about what the other will do. So we end up in a world where fear reigns and consumers don’t investigate alternative RSS readers because they fear that the other options are even more primitive. And although developers could make something much better, they don’t want to risk investing the development time and having no one show up (since GR is “good enough” for most people.)

And so for years we’ve been muddling along in the bad Nash Equilibrium of continuing to use GR and letting RSS become “uncool” relative to social sharing (despite being, IMHO, about 10x more useful.)

But when Google shuts down Reader, the whole game changes. Suddenly the bad Nash Equilibrium square is out of play. That means that coordination between developers and consumers is no longer necessary because “staying put” is no longer an option.

The immediate aftermath might be painful, but I predict that we’ll all end up with a much better system.

Also, note that this dynamic is hardly exclusive to GR. It’s all over the tech world: some company develops a product which gets popular; they lose interest and stop improving. But they keep the product alive, and the network effects and switching costs keep people using it even though much better mechanisms are theoretically possible. The two worst offenders I can think of are:

Craigslist
Javascript

Both are terrible but both are extremely widely used. And so we’re stuck in a bad Nash Equilibrium where everyone suffers because it’s too hard to get people to sell their broken furniture on a new website when they can just use Craigslist and it’s too risky for Mozilla to implement Python in the browser when developers might just keep kludging JS.

If Craigslist would just take a lesson from Google and shut down completely, it would be one of the best days ever in tech.

Can you think of better examples of “okay” products that could help the world by just disappearing? Please comment and let me know!

[1] Game theory is complicated, and there are a lot of exceptions depending on circumstances. But simplification usually gets us close enough.
[2] The prisoner’s dilema is one of the saddest abstract constructs I know. In a painful nutshell, it explains why “we can’t all just get along.”
[3] Not because they don’t care per se, but because it’s just not impactful enough relative to their other lines of business

Cover of the day — a heavy metal “barbie girl”

Sun, 24 Mar 2013 22:27:36 -0400

Cover of the day — a heavy metal “barbie girl”

Cover of the day: Chvrches “I Would Die For V”

Wed, 20 Mar 2013 15:14:25 -0400

Cover of the day: Chvrches “I Would Die For V”

LinerNotes Liveblog #7 -- SXSWi

Wed, 20 Mar 2013 08:01:00 -0400

Things in Austin got weird. There was carousing, there was trespassing, and yes, there was even laser tag. But after seven crazy days in Austin, my SXSW adventure is over. Instead of sharing a small Austin house with ten people (including two different bands), I’m now just sharing a house with my parents as I visit the Bay Area for some meetings.

SXSW was definitely fun, but it was also probably a waste of time.[1] I probably could have seen that coming — one of the clearer lessons of my time working on LinerNotes is to be very skeptical of conferences. I’ve been to several of the bigger ones (mostly because I managed to snag free tickets), and none of them have delivered enough value to clearly justify the time it took to attend, much less the the jaw-dropping ticket prices (two thousand dollars for TC Disrupt? Seriously?)

Still, out of all the hundreds of tech conferences SXSW retains a somewhat mystical reputation for “serendipity,” i.e. that simply by walking around and talking to people one might encounter the investor or business partner one needs to “kick it up a notch.” That’s a tempting offer, since I do believe in serendipity (or rather the darker flip-side that many endeavors will be destroyed by the absence of serendipity.)

My skepticism almost made me follow Danielle Morrill’s advice and skip SXSW, but a few things fell in place at the last minute (i.e. I found a place to stay for free) and so I decided to take the plunge.

And…I was disappointed.

Serendipity did not strike. I’m sure that it did for some people, but I didn’t meet even one new person with whom I plan to stay in close contact. To be fair, I didn’t do much to maximize my luck surface – I only went for the last two days, I didn’t buy a conference badge ($800?!) and I only struck up conversations with 20% of the people I randomly stood next to in lines instead of 100%. Still, the conversations I did have didn’t give me the sense that my digital soul mate was just one forced conversation away.

The basic problem is with the odds. When SXSWi was smaller, it may have been viable to talk to a large percentage of attendees and either meet someone useful or meet someone who could introduce you to someone useful they just met. But now there are 30,000 people at SXSWi. Let’s assume that’s 1.5% of the entire startup world, and let’s assume that the population is representative of the general startup population. [2]

What are my (very rough) odds of meeting someone useful? Well, I’d estimate that there are probably less than 500 viable co-founders and 500 viable investors for me in the whole world, and LinerNotes is not really at a point where I’d benefit much from any business development relationships or marketing hires. If 1.5% of those people are at SXSW, that’s approximately 30 people at SXSW worth meeting.

If I’m there for two days, I probably come into physical proximity with 300 people in a day or 600 in total. That’s 4% of the conference. 4% of 30 is 1.2, meaning that in the whole two days, I was probably near someone useful about twice.

Now how many conversations with strangers can I reasonably have in a day? If I’m really forcing it, maybe 20. So that’s (approximately) 40 dice rolls with a 599/600 odds per roll of not meeting someone useful. That’s an 93% chance of failure, which implies a 7% chance of meeting someone useful, if I’m aggressively social and if the population is really representative (which it probably isn’t).

So after all that expense, travel, and foregone working time, the odds of serendipity striking are about 1 in 15. Compared to everything else I could do with those resources, going to SXSW for networking is pretty clearly not worth it.

So why does SXSW continue to have a reputation for magical serendipity?

Simple, because only the 1 in 15 people who benefit bother to write about it or tell their friends. No one likes a party pooper and so other than me and Danielle Morrill, few people want to loudly admit to wasting their time in Austin.

To be clear, I don’t regret going. The main purpose of my trip was actually to go to SXSW Music and do market research for LinerNotes, and it was worth sacrificing a weekend of work to have some fun and confirm my suspicions about SXSW. And I did meet up with a few old friends at SXSWi and got to strengthen those relationships.

There are some people for whom SXSW still does make sense (specifically startups that sell to marketers or app makers, or VC’s and later stage startup employees who just want to relax and party.) But now I can say from experience that there are better ways to make new friends than shouting at strangers at a loud, crowded Taco Bell sponsored party.

[1] This post only applies to SXSW Interactive. The dynamics of SXSW Music were much different and probably need their own blog post.

[2] This is obviously a rough calculation. These numbers are mostly made up with the goal of being ballpark correct and having errors cancel out. And even very generous assumptions don’t change the overall picture much.

A little late, but here’s a playlist of my favorite songs...

Tue, 05 Mar 2013 18:16:40 -0500

A little late, but here’s a playlist of my favorite songs I found in February.

LinerNotes beta launch day 5 – ‘What is LinerNotes?’

Wed, 27 Feb 2013 16:47:38 -0500

Sorry for the hiatus; it was a busy weekend. I participated in the NYC Monthly Music Hackathon, continuing work on my project Videodrome (a HypeMachine/Exfm for music videos.) I’m aiming to release it publicly next month.

Much more importantly, I pressed the button on LinerNotes.com on Saturday night. The beta is now live, publicly accessible, and has gone four days without exploding.

There was a bunch of interesting process involved that I’d like to blog about, but I need to write something else first. I realized over the weekend that I obviously should have started this series by answering one simple crucial question…what the heck is LinerNotes, anyway?

Actually there’s more like three questions:

1) What is LinerNotes right now?

2) What will LinerNotes become?

3) Why bother building something like this in the first place?

I’m going to answer in reverse order.

Why?

This deserves it’s own blog post and I’ll probably write one soon. But the short answer is very simple – because I’m a curious person. I genuinely love learning things. And right now, it’s too hard for me to learn about and find music (and a great many other things.)

Many people are satisfied with whatever pops out of Google or Wikipedia or Pandora. Well, I’m not. I want more. I want a lot more. If you don’t then LinerNotes is probably not for you. At least not today.

LinerNotes is hand-tailored to the way I want to listen to music. I want my experience to be fluid, effortless, and richly contextualized. I want an endless stream of music that really affects me emotionally, and I want to know how that music and the artists who make it fit into the broad and wonderful history of music. I want to spend my time listening and learning and ruminating, not seeking and scrounging and clicking. The more I know about my music, the more I like it. And as hanging out with my friends constantly reminds me, there’s an awful lot I don’t know.

What will LinerNotes become?

I want LinerNotes to bring all information about music together in one place. All of it. But I want to keep it organized enough that you can find any fact or song just by describing it, e.g. “bands from Liverpool in the 50’s.”

LinerNotes is going to become the Virgil to your musical Dante, guiding you from the hell of tired silence up into the golden light of musical rapture. LinerNotes is going to become the Mrs. Robinson to your musical Benjamin, guiding you from music novice to musical master and then going straight up crazy when you try to date our musical daughter.

LinerNotes will be there for you on every step of your music experience, from helping you learn about and remember what you already love to helping you find what you’re going to love next. LinerNotes isn’t loyal to any label or big corporation — we’re only loyal to you.

What is LinerNotes today?

LinerNotes.com is a very early beta right now. It’s not the realization of my grand vision – it’s just the first version I can show to people without cringing from embarrassment.

Right now, LinerNotes is basically just an overlay on top of Wikipedia, but with some really useful you-friendly enhancements:

It’s condensed. Because we love the Beatles, but we don’t five-pages-of-unbroken-text love the Beatles.
We’ve got complete discographies for nearly every artist, including bootlegs and rarities. Plus pages for every song and album, often complete with credits.
Our dangerously-advanced search system lets you search using concepts like “people who play the ukulele” or “bands who’s hometown is Manchester”. Press the “hard mode” button on the homepage to give it a try.
You can browse genres and record labels to find new bands. Have you seen all the great acts on Jagjaguwar?
You can start listening immediately using Spotify or YouTube (click the “Videos” tab.)

We’ve got an extremely long list of additional features we’re going to add on top of all that, but we need your help to decide what to build first.

So welcome to LinerNotes. Please give it a try and leave feedback telling us what you like, what you don’t like, and what your musical experience today is missing.

Cover of the day: The War on Drugs covers Bruce...

Tue, 26 Feb 2013 20:34:35 -0500

Cover of the day: The War on Drugs covers Bruce Springsteen’s “The Ties That Bind.”

Song of the day: The Reflections, “Disconnected”

Sat, 23 Feb 2013 17:08:42 -0500

Song of the day: The Reflections, “Disconnected”

"Broken Tools" -- Live-blogging LinerNotes.com, Day 4

Fri, 22 Feb 2013 00:50:52 -0500

“Equipment in action operates in an inconspicuous usefulness, doing its work without our noticing it. When the tool fails, its unobtrusive quality is ruined. There occurs a jarring of reference, so that the tool becomes visible as what it is:”—Graham Harman [1]

Last live-blog, I made a plan to launch. Just the day before, I talked about how often planning goes astray. And unsurprisingly, it’s only taken two days for my plan to get off track.

This plan is about broken tools. But more broadly it’s about being sucker punched by the unexpectable. Luckily I got off pretty easy this time, but the punches come hard and fast, and they will for as long as I’m in this business.

Why write about it now? Because a lot of stuff has broken in the last few days. I’ve had to spend so much time putting out fires that my self-imposed deadline of “pushing the button” by tomorrow is probably not going to happen.

Some of the breakage has been minor and some has been major. All of it has been completely typical.

Minor example: I got a letter (delayed in the mail) urgently requiring me to scan and email a document. My scanner broke that same day. So I had to waste 20 minutes on the phone with Lexmark’s absolutely dismal customer before tracking down a fax machine in a bodega.

Major: The virus I thought I beat last week has returned, leaving me feverish and mealy-minded. So I’ve been operating at 25% productivity today as I try to force my mind to think enough thoughts to figure out why ElasticSearch has abruptly stopped working.[2]

Both of these inconveniences were effectively unpredictable. Both siphoned off real time. And like a running back’s shoe falling off mid-stride, they made themselves perilous to ignore and shattered the precious “flow” where technical productivity hides out.

As you saw on Tuesday, I tried to make conservative estimates for how long each step of preparation would take. But I’ve already handily blown through my margins. The problem is that the time a task requires is not normal distributed – it’s heteroskedastic. In other words, nine out of ten times it will take me 15 minutes to get to Union Square. But the tenth time the train will break and it will take an hour. So when I’m deciding how early to leave for a meeting, my only options are to leave an hour early (and spend hours each month alone in Starbucks, guarding my seat from 14 year old girls with their cinnamon spice lattes), or to accept that once in a while I’ll be offensively late.

Doing the former is the equivalent of buying so much health insurance that you can’t afford to eat. It’s just not practicable. On the other hand, the person you’re late for will really be offended: people tend to weight one negative interaction about as much as a dozen positive ones. I once heard noted VC Brad Feld say that he would not invest in a company unless every interaction was better than the previous one.

But think about the paradox there: the only teams that never fail are either much too conservative, absurdly lucky, or outright lying.

For me, lying is automatically off the table. And my luck is stubbornly resistant to dark magic. But I haven’t figured out another good solution. It pains me to miss my launch target, especially so soon after I’ve posted it publicly. But the alternatives (don’t be transparent, or power through and make myself sicker) seem even worse.

Entrepreneurs build with fragile tools. They’re all we can afford. But earning the confidence of employees and partners seems to require projecting a superhuman façade.

How do you reconcile that tension?

[1] I wanted to start this post off with a quote from philosopher Martin Heidegger, since I’m stealing his idea about tools. But I couldn’t find a direct quote that would require less than 500 words of jargon unpacking, so enjoy this secondary source.

[2] I skipped yesterday; these posts are taking about 75 minutes / per, so a rigid daily cadence is clearly not going to happen.

[3] Backward incompatible version upgrade, plus forgetting to update the cluster name in code upon spawning a new cluster.

Music video of the day — Nightlands, “I Fell In Love...

Thu, 21 Feb 2013 10:00:49 -0500

Music video of the day — Nightlands, “I Fell In Love With A Feeling”

Music video of the day, MS MR — ‘Fantasy’

Wed, 20 Feb 2013 23:53:30 -0500

Music video of the day, MS MR — ‘Fantasy’

LinerNotes live-blog day 3: "My planning process in action"

Wed, 20 Feb 2013 03:15:00 -0500

Last time, I said that detailed planning is mostly useless. This time, I’m going to make a detailed plan. Why? Two reasons: 1) I have a very clear and tangible goal – make LinerNotes publicly accessible, and 2) I have a long list of small, discrete tasks that need to happen first. So in this case “planning” isn’t as much about figuring out what to do as it is figuring out when to do it.

Technically, I could launch in one step. I’ve got a fully functional version of the site running on production servers, so all I have to do is change the A-Record on www.linernotes.com to a new I.P. address and you will be able to give it a whirl. But I want to make the best first impression possible, even though the main point of this beta is to test ideas and gather feedback.

Good impressions require polish. Polish is in the details. Keeping track of details means brain-strain. Plans help brain-strain. So let’s save my brain and make a plan.

When I’m in “loose planning” mode (i.e. most of the time when I’m working alone) the plan is a long list of tasks kept in Asana with a rough priority ordering kept in my head. To get more rigorous, I start mapping tasks onto discrete blocks of time. Asana has a lot of virtues, but my biggest frustration is that it doesn’t integrate well with a calendar.

I mostly use the built-in Mac apps (iMail because I have to many email accounts to use the Gmail website; iCal because I just haven’t found a good calendar app). So this planning process is going to consist of making sure I have Asana tasks for everything I want to accomplish and then manually putting those tasks into iCal. (Someone please disrupt this process. Or recommend a better flow.)

Here’s a high-level list of what I need to get done:

Update the data and streamline the pipeline. LinerNotes starts life as about 10GB of .tar.gz files. It takes about 20 hours and too much manual supervision to bloom into final form.

Clean up the CSS. I’m working with a designer on a professional-looking redesign, but that won’t be ready for a few weeks.

Implement basic unit testing. TDD + constantly changing developments = a ton of time writing unused tests. I’ve decided to backload the pain.

Come up with a solid analytics / metrics plan. I’ve got Google Analytics and Datadog, but I want to make sure I learn as much as possible from early users.

A bunch of relationship management stuff (I’m probably not going to write much about individual people, but I may write about writing about individual people.)

Fix many minor bugs

Press the button.

Based on the plan I made while writing this, it looks like Friday will be the day.

Here’s my calendar for the week: