Does genuine tech innovation happen better in a recession?

I know this may sound a little counter intuitive, but bear with me. There are some important kinds of innovation in technology which require a recession and not full employment. In the late 90’s there was the huge dot.com boom, everybody who could code at all was working from dawn to dusk. Some people made huge amounts of money, some also lost it, others just worked endless hours.

Then there was a collapse, huge numbers of geeks were unemployed, hundreds of thousands in the san francisco bay area alone. Out of that collapse there was time for people experiment, play with stuff, do things which were not required for meeting the next deadline. That recession and massive underemployment of geeks created blogging as we know it today.

It also laid the groundwork for the small agile startups which optimize the whole web 2.0 thing. Ludicorp which could go from building a multiplayer game to photosharing site. Design shops who have time to roll their own web framework in an obscure programming language from japan, and the like.

When you are busy and having paying clients banging on the door, you don’t have time to focus on exploring new areas, discovering the depth and meaning of your medium. When i was working at odeo we were all heads down, coding, fixing, building, getting the damned thing done. There was real competition, apple and yahoo were both wanting to crush us with their huge legacy advantages of userbases and platforms. It was only once we realized we’d lost, that Odeo wasn’t going to be the podcasting platform we’d envisioned, that Ev decided to do a series of hack weeks, play with new ideas, see if there wasn’t something new which could be created. That playing around seemed like floundering to me at the time. I was frustrated and burned out. But that process is what helped Jack pickup some ideas he’d had floating around in his head for years, but no time to work on. Those ideas became twitter, called twttr at the time.

The dot bomb wasn’t the only time when many techies were underemployed. I was reading a review of Matz’s keynote at Ruby Fools in denmark a few months ago and something jumped out at me.

“The recession in Japan meant that Matz – while not unemployed – didn’t have many assignments at work, so he secretly started to hack away at Ruby at the office.”

Not only was blogging the result of industry wide underemployment of geeks, and twitter the result of a similar thing within a single company, but Ruby was created in a very similar environment during the recession in Japan in the 90’s.

To the vast majority of programmers, it’s a job, they’re the %80 who are vocational programmers. They program during their work day, and go home to do other things, on the weekend they aren’t working on their own projects. Then there are the people who are obsessed, the geek’s geek as it where. We believe in learning a new programming language every year.

Real innovation in software seems to happen when we have underemployment of the true geeks. Either inside of companies where the geeks are allowed to explore, or outside of companies through economic forces. Long hours and tight focus is often needed to get a startup going, or ship software, but it’s not what creates something really new.

It’s ironic, the very forces of modern capitalism which praise the internet economy as being the ultimate in friction free globalized markets hold back real innovation. It’s the down cycles, either within companies or in the broader labor market, which give the innovators space to risk walking down dead ends. It’s a cliché to say that you need to be able to fail if you are going to take real risks. When geeks are free to hack on their own then no deadlines, just interesting spaces to explore. It’s then that we see real innovation.

I spent a year working at Yahoo Brickhouse, we were supposed to be yahoo’s ‘inner startup’. We were going to find a way to create new stuff, employed to wander in the woods and come back with amazing ideas. We were told that it was ok to fail 9 times out of 10. We built several things which never made it out the door, two which made it to beta launch, Bravo Nation, and Fire Eagle. I’m not sure what the future of Bravo Nation is, but i know Fire Eagle is going to launch and it’ll be a major part of making location based systems interoperate. Fire Eagle was a project which was started by Tom Coates, Paul Hammond, and Simon Willison, they were working from London and largely ignored by everybody at Yahoo. Before they could finish they were taken off of the project to finish higher priority things. Then Yahoo Research Berkeley said, oh there are some neat ideas here, and they played with it for a while, they were also ignored by the larger yahoo who focused on shipping things rather than playing with interesting concepts. Eventually Tom talked Brickhouse in to joining in on this crazy location broker idea and it took another 6 months of work to get a beta out the door. The whole process went round and round all the while yahoo was doing it’s own internal soul searching. At the end of the day Brickhouse had to make either BravoNation or Fire Eagle succeed or it was getting restructured in to who knows what. The day of the big layoffs back in February, Brickhouse lost it’s director, Salim Ismail, and his boss yahoo vice president Bradley Horowitz. Caterina, who had the brickhouse idea in the first place, had already left on maternity leave. Chad Dickerson was left in charge and did amazing work to help fire eagle get out the door and support Yahoo Live. Clearly we weren’t actually allowed to go off and fail 9 times out of 10 in the process of coming up with a crazy new and important idea.

What i find truly ironic about the process is we did end up coming up with a pretty damned good idea. That of using oauth and webservices to broker information between other applications and services on the web while leaving the user in control and protecting privacy. Were we the only ones who came up with it? No, innovation is about a community of thinkers, there were many people involved who work at many places including twitter, pownce, magnolia, six apart, wesabe, flickr, many others, and of course yahoo and google. You see it now with interest in federated social network, the gnip project, and other stuff which is still in the works.

Shipping a product and being innovative are often not compatible goals. The innovation happens in the downtime, then once an idea has gelled, the innovation stops and the work of creating production code begins. Sometimes it’s the same folks, most of the time it’s different folks. Often there is a long gap between the ideas being created and somebody building something useful on top of them.

We need recessions and underemployment of the innovators for them to have time to do their work. In a more ideal world, we would support this just like we should support having artists in society. Until we change it, we’re stuck with dumb brutish capitalism which has lucked in to creating these spaces for innovation.

Update: I fixed some of the spelling and grammar problems, thanks. :)