Anarchogeek - Home

The Open Web Foundation - Why Do It?

2008-07-24T17:32:00Z

So i’ve not been involved in the newly launched Open Web Foundation but i have been following along as various folks i know worked on OpenID and OAuth. Is it a great idea, or yet another structure? I’m not sure.

Here’s my understanding. Most standards traditionally have been heavy vendor driven processes. Even when you don’t need $$$ or to be a big company to participate, you do need to have the resources to fly around the world to the meetings, like IETF. It’s painful and nasty.

There are a series of standards which are created using more open source / hacker / collaborative ways which are emerging from code, like extracting a framework from your application, you extract the standard from your work and best practices. It’s easy, lightweight, great, works.

Some of these extracted open sourcey standards get picked up and big companies want to start implementing them. This is great, just look at how OAuth has gotten adopted all over the place.

But there is a problem, these big companies have lawyers and their lawyers are both stupid and evil. Lawyers created the stupid and evil system of intellectual property. These lawyers all seem to have adopted a particularly brain dead view of what my happens with related to participation and patents and intellectual property rights.

They think that unless you have a magic paragraph which says you don’t grant patents and intellectual property rights just by participating in a standards process, or even USING or READING a standard, the big corps loose their rights to enforce their patents.

Yes it’s super stupid, but that’s what you get when you let lawyers run your world.

So the Open Web Foundation is about creating something which will shut the lawyers up and let these bottom up, extracting standards folks still work without having to die the death of a thousand meetings with legal.

Beyond REST? Building Data Services with XMPP PubSub

2008-07-23T23:06:00Z

Kellan and I gave a talk today at OSCON about some ideas we've been playing with around how to make web services work as things keep getting bigger and faster. Apparently it was well received.

UPDATE: There's a great write up by Robert Kaye about the talk by on O'Reilly Radar and Joshua Schachter confirmed similar use / abuse pattern for delicious.

Beyond REST? Building Data Services with XMPP PubSub

view presentation (tags: xmpp pubsub rest oscon)

Does genuine tech innovation happen better in a recession?

2008-07-20T02:09:00Z

I know this may sound a little counter intuitive, but bear with me. There are some important kinds of innovation in technology which require a recession and not full employment. In the late 90’s there was the huge dot.com boom, everybody who could code at all was working from dawn to dusk. Some people made huge amounts of money, some also lost it, others just worked endless hours.

Then there was a collapse, huge numbers of geeks were unemployed, hundreds of thousands in the san francisco bay area alone. Out of that collapse there was time for people experiment, play with stuff, do things which were not required for meeting the next deadline. That recession and massive underemployment of geeks created blogging as we know it today.

It also laid the groundwork for the small agile startups which optimize the whole web 2.0 thing. Ludicorp which could go from building a multiplayer game to photosharing site. Design shops who have time to roll their own web framework in an obscure programming language from japan, and the like.

When you are busy and having paying clients banging on the door, you don’t have time to focus on exploring new areas, discovering the depth and meaning of your medium. When i was working at odeo we were all heads down, coding, fixing, building, getting the damned thing done. There was real competition, apple and yahoo were both wanting to crush us with their huge legacy advantages of userbases and platforms. It was only once we realized we’d lost, that Odeo wasn’t going to be the podcasting platform we’d envisioned, that Ev decided to do a series of hack weeks, play with new ideas, see if there wasn’t something new which could be created. That playing around seemed like floundering to me at the time. I was frustrated and burned out. But that process is what helped Jack pickup some ideas he’d had floating around in his head for years, but no time to work on. Those ideas became twitter, called twttr at the time.

The dot bomb wasn’t the only time when many techies were underemployed. I was reading a review of Matz’s keynote at Ruby Fools in denmark a few months ago and something jumped out at me.

“The recession in Japan meant that Matz – while not unemployed – didn’t have many assignments at work, so he secretly started to hack away at Ruby at the office.”

Not only was blogging the result of industry wide underemployment of geeks, and twitter the result of a similar thing within a single company, but Ruby was created in a very similar environment during the recession in Japan in the 90’s.

To the vast majority of programmers, it’s a job, they’re the %80 who are vocational programmers. They program during their work day, and go home to do other things, on the weekend they aren’t working on their own projects. Then there are the people who are obsessed, the geek’s geek as it where. We believe in learning a new programming language every year.

Real innovation in software seems to happen when we have underemployment of the true geeks. Either inside of companies where the geeks are allowed to explore, or outside of companies through economic forces. Long hours and tight focus is often needed to get a startup going, or ship software, but it’s not what creates something really new.

It’s ironic, the very forces of modern capitalism which praise the internet economy as being the ultimate in friction free globalized markets hold back real innovation. It’s the down cycles, either within companies or in the broader labor market, which give the innovators space to risk walking down dead ends. It’s a cliché to say that you need to be able to fail if you are going to take real risks. When geeks are free to hack on their own then no deadlines, just interesting spaces to explore. It’s then that we see real innovation.

I spent a year working at Yahoo Brickhouse, we were supposed to be yahoo’s ‘inner startup’. We were going to find a way to create new stuff, employed to wander in the woods and come back with amazing ideas. We were told that it was ok to fail 9 times out of 10. We built several things which never made it out the door, two which made it to beta launch, Bravo Nation, and Fire Eagle. I’m not sure what the future of Bravo Nation is, but i know Fire Eagle is going to launch and it’ll be a major part of making location based systems interoperate. Fire Eagle was a project which was started by Tom Coates, Paul Hammond, and Simon Willison, they were working from London and largely ignored by everybody at Yahoo. Before they could finish they were taken off of the project to finish higher priority things. Then Yahoo Research Berkeley said, oh there are some neat ideas here, and they played with it for a while, they were also ignored by the larger yahoo who focused on shipping things rather than playing with interesting concepts. Eventually Tom talked Brickhouse in to joining in on this crazy location broker idea and it took another 6 months of work to get a beta out the door. The whole process went round and round all the while yahoo was doing it’s own internal soul searching. At the end of the day Brickhouse had to make either BravoNation or Fire Eagle succeed or it was getting restructured in to who knows what. The day of the big layoffs back in February, Brickhouse lost it’s director, Salim Ismail, and his boss yahoo vice president Bradley Horowitz. Caterina, who had the brickhouse idea in the first place, had already left on maternity leave. Chad Dickerson was left in charge and did amazing work to help fire eagle get out the door and support Yahoo Live. Clearly we weren’t actually allowed to go off and fail 9 times out of 10 in the process of coming up with a crazy new and important idea.

What i find truly ironic about the process is we did end up coming up with a pretty damned good idea. That of using oauth and webservices to broker information between other applications and services on the web while leaving the user in control and protecting privacy. Were we the only ones who came up with it? No, innovation is about a community of thinkers, there were many people involved who work at many places including twitter, pownce, magnolia, six apart, wesabe, flickr, many others, and of course yahoo and google. You see it now with interest in federated social network, the gnip project, and other stuff which is still in the works.

Shipping a product and being innovative are often not compatible goals. The innovation happens in the downtime, then once an idea has gelled, the innovation stops and the work of creating production code begins. Sometimes it’s the same folks, most of the time it’s different folks. Often there is a long gap between the ideas being created and somebody building something useful on top of them.

We need recessions and underemployment of the innovators for them to have time to do their work. In a more ideal world, we would support this just like we should support having artists in society. Until we change it, we’re stuck with dumb brutish capitalism which has lucked in to creating these spaces for innovation.

Update: I fixed some of the spelling and grammar problems, thanks. :)

Dear Lazyweb: What's a good prosumer soho wifi / router / access point?

2008-07-16T16:21:00Z

So i’m sick of flaky wifi / routers / access points. I’ve had netgear’s, d-link’s, belkin’s, and linksys routers. To me they all were about the same, which is to say they worked some of the time.

Before i go back to getting one more of the same to replace my now limping netgear, i’ve thought, what if i got something more than the cheapest consumer class wifi router. I know that hotels, conference centers, and all sorts of other places have wifi which doesn’t need to be restarted on a regular basis, that doesn’t suck. Sure some have wifi which sucks, but some don’t. The same goes with companies, at Yahoo! there was good wifi coverage all over campus, their crazy vpn made it hard to use, but the coverage was there.

So then my question becomes, what are my options one level up? So far a little searching around i found the Cisco Aironet’s which start around $200 for the lower end models and the ZyXAL’s ZyWALL 2WG router which supports 3G as a backup connection in addition to being a wifi router at $250+.

Are those good routers? What are the alternatives in that quality range? Has anybody tried and used either of those?

Dear lazyweb, please help….

iPhones in Argentina, Uruguay, and Paraguay

2008-07-10T19:22:00Z

So while everybody up north is getting ready for the introduction of the 3G iPhone, it’s also arriving down at the bottom of the world. Apple chose the mexican company, Claro, formerly CTI Movil, to sell iPhones in latin america, probably because they could go with provider and get the whole region.

While Claro hasn’t been very clear about exactly how much the iPhone will cost, when it’ll be for sale, or really anything except to say that it’s coming. I do have some information.

According to their prices posted for mexico the monthly plans will cost between $45 USD per month and $85 USD per month. The rumors for chile had similar prices. The phone will cost between $450 USD and $0 USD depending on contract, 8gb vs 16gb, etc… Taxes included, which in latin america are substantial, here in Uruguay the IVA sales tax is %21.

How many iphones are coming down south? Well i don’t know all the numbers, but Claro’s confirmation page when you request notice about the phones says that they have imported 20,000 phones to Argentina, Uruguay, and Paraguay each. On the surface that seems fare, each country gets a nice shipment. The problem is, Argentina is a MUCH bigger country than the guay’s. Argentina’s population is 40.6 million, Paraguay’s 6.8, and Uruguay’s 3.4. The data i found is from 2006, but Argentina’s cell phone penetration is better than %76. Uruguay’s is %67, and Paraguay’s is %47. So not only did Argentina get fewer phones, they’ve got more people are likely to WANT one. There are 13.5 times more iphones for sale when adjusted for the cellphone ownership market in Uruguay than Argentina!

Given that it’s a couple hour ferry ride across the river from Argentina to Uruguay i expect there might be a grey market of Argentines buying their iPhone from Claro Uruguay if Claro Argentina sells out.

There have been come claims that Movistar (owned by the spanish telefonica) also will be selling iPhones in latin america, but clearly it’s not at launch, because there’s nothing on their websites about it.

I’ll followup when they actually release their prices and say when they will start selling them.

The ascendancy of Hacker News & the gentrification of geek news communities

2008-07-07T15:49:00Z

I’ve been reading the reddit clone site, Hacker News from ycombinator more lately. It’s got a nice combination of alpha geek tech and small / agile startup topics. While i like the political news on reddit, honestly it’s stopped being very topical for me… to much taken over by link spammers i think who are gaming the system for traffic. The subsites like ruby.reddit.com are still a great source of traffic.

What has surprised me is that Hacker News seems to have found me about the same time i found it. When i went to go do a post i thought might be relevant to the community, about moving email from using smtp to xmpp, i posted it on hacker news. Only after did i discover that somebody had already posted a link to it.

Then yesterday i did wrote another blog post, this time about leaving yahoo brickhouse, and again over half the traffic came from another link posted to hacker news. So what’s going on here? Well first the community of folks has shifted. It’s previous semi-private places have been lost and new places created.

For me long ago i read slashdot, then kuro5hin, then my delicious network, then , then Hacker News. There were others in there too, i get lots of links from reading topical blogs like rubyinside, activist news from indymedia, etc…

The shift of online communities resembles urban development and the gentrification of many hip neighborhoods. The artists and hackers move in first, they are in development parlance, risk tolerant. For urban neighborhoods that means they’ll deal with crime if they can get cool warehouses to take over. Then slowly the neighborhood transforms, and gets some nice cafes and clubs, gets known as the place where the hip kids play, and more people come. Rent gets driven up, the crowds come, it becomes to crowded, and the hipsters have to move on. Just replace hipsters with alpha geeks and you get the same process.

We are creating virtual communities and then by our very own actions gentrifying them!

So why do i like the small sites, not just hacker news, but dzone, rubyflow and a bunch of others? Well for one because they work well for me. I can submit a link, or in the case of hacker news, somebody else in the community links to my stuff, and then i get traffic. A lot of traffic really, I can get 7 votes, but that translates in to 300 to 3000 visits to the article. It’s much harder to get on the front page of reddit, or dare say digg where the true unwashed masses of tech news junkies hang out these days. It’s even harder to get on top of yahoo buzz, where a few hours on the top page can lead to millions of page views.

Are we doomed to keep creating these communities, enjoying them for a while, then having to abandon them and move on? When i helped start indymedia.org back in 1999, we thought open publishing, the ability to put on the internet your own articles, videos, pictures about news was revolutionary. It was a big deal, this was before you could just create a blogspot or wordpress site. Our enemy was CNN the site which only showed you the news they wanted… But today cnn has Unedited. Unfiltered. News. iReport.com which is pretty damned similar to what we were doing with indymedia. Then then take some of the news created on ireport and integrate it to cnn.com’s site and use it in the news. The BBC does something similar.

The point is, we won. We took an idea, which said that the masses should be able to make their own media, and we did it as an example and eventually the people we were fighting against started copying us. No we didn’t win all of what we wanted, we had a political agenda which we able to advance here and there, we stopped the WTO round, ended the FTAA (free trade area of the americas), but in may ways we won.

So what does that mean to online communities, generating and finding news? Well first off it turns out that we, the broader hacker community is doing a good job at coming up with models which change the world. From blogs to wikis to link voting and collaborative editing, we’re coming up with ideas which other people are copying. Or sometimes the hacker community’s tools become mainstream. But we also face the reality that there is a tremendous value in influencing what gets seen.

If you can make a website which gets a lot of traffic, there is money to be made there. That’s the attraction of the SEO / SEM world. They’re not respected by true hackers, but they are huge, and they come in and destroy communities like reddit.

One option is we just keep moving, which is what we’ve been doing. From slashdot to kuro5hin, from digg to reddit from reddit to hacker news.. The other option is we try and build in to our systems anti-SEO / SEM protections. Find ways to use emergent behavior to find real and relevant content without having it be gameable. Twitter stopped spammers by using tinyurl for all links… Delicious did it by making it so my network is people i choose. The link voting sites will have a hard time. Perhaps we’ll just switch sites every 6 months to a year, but there’s got to be a better way.

Good Bye To The Purple Mothership: Leaving Yahoo

2008-07-06T21:04:00Z

I just realized that i forgot to post anything on my blog about this… woops! I left Yahoo Brickhouse in April. I twittered about it, but wasn’t sure what to say in a blog. While at Yahoo i got to do the architecture work and a lot of the development of Yahoo! Fire Eagle. We launched the first rails app at Yahoo and were the first to release a working OAuth implementation. Working with Brickhouse was amazing, a great group of people who worked incredibly hard to release an amazing product.

Working with, or rather for Yahoo, was not so great. It was a constant painful struggle. Maybe at some point i’ll sit down and write about the relationship the reform movement part of yahoo had with the mothership. While i tried to make things better, i’m not sure that improving the quality of a fortune 500 company is really my cup of tea.

I have looked at a number of options since leaving Yahoo, and decided to spend some time coding along side the amazing geeks at entp.com. One time at a foocamp i heard somebody say, “find the smartest group of people you can and work with them.” Entp makes Lighthouse a ticket tracking system that rails itself uses, and a few other apps.

Moving to a small agile company is a breath of fresh air. Instead of we need to have a meeting about it, my questions get answered with, “sounds good, pdi”. Instead of being told that subversion is on the timeline for 2 years from now, everybody’s using git.

In addition to working with entp i’m also doing some work as an adviser to mapufacture, change.org, and others helping them with scaling, architecture and tech strategy.

The Future of Email: From SMTP to XMPP

2008-06-20T02:16:00Z

Email is dead! Long live email!

Email has long been the killer app of the internet. It has taken us to a world were everybody has an address and anybody can send an email to anybody else. Email works incredibly well.

At the same time, email is totally broken. Address books are painful to maintain, and they don’t tell us about somebody’s ability to actually reach another person. Spam is a major problem, i have no easy way of saying who i want to be able to send me messages, no way of saying, i don’t want messages from you any more. Spam, and getting around spam filters with legitimate email is a huge problem. The vast majority of email is spam.

There is a reason the myspace/facebook generation hardly use email. They’ve got a system which solves the spam issue, built in is a buddylist which lets you define who can send you messages. It’s also a realtime system rather than a store and forward system expecting users to be mostly offline. The problem is these message systems are walled gardens.

Jabber, and it’s XMPP protocol, were built for IM, but they made it super flexible. It can easily be used for email to solve the delivery permission / address book issues. Now i can easily authorize people to send me messages.

The addresses are even compatible, so you can have a bridge, attempt to deliver via xmpp and if the domain doesn’t handle xmpp roll back to smtp.

Of course i’m far from the first person to have thought of this, it’s come up in 2004, 2006 and 2007. So the question is, why hasn’t it happened. Well first off, these things don’t just happen on their own, somebody has to do the work, write software, organize it, make the change you want to see.

How could we get from here to there? Kill email so that email can live free?

Well the way i see it there are several things which need to happen. It’s a chicken and egg issue, nobody is sending email via xmpp because nobody can receive email via xmpp, the clients don’t exist. Nobody’s building xmpp email clients because nobody’s sending email via xmpp to receive.

There are some things working towards adoption of xmpp for email. Critically, the email address can stay the same. Many of the alpha geeks already use the same address for their email and their jabber IM accounts. The email address is deeply embedded in the culture and any attempt to vary from it would doom the move to email over xmpp.

The second thing which can help lead to adoption of xmpp email is the integration of email / messaging and IM. You see it in yahoo mail, facebook, and myspace. The big email / messaging providers are already routing IM messages alongside the email.

The third third thing which will help is we don’t actually have to get very many providers to adopt xmpp email to get critical mass. Despite email’s incredibly federated nature, there are a few providers who have hundreds of millions of accounts. Get one, or several of those providers to switch and provide xmpp email support in addition to smtp email, and you’d be able to use that to shift everybody over. The promise of getting out of spam filters will be enough to get the big senders to jump over, and once you’ve got both sides, then it can take years for the rest of the net to move over, it has to be a gradual process. Once you’ve got a big provider who accepts email via xmpp, then you can work on building out library support, create the email extensions to ejabberd, openfire, etc… and bridges to sendmail, exim, postfix, etc..

The obvious people to do this are the google gtalk & gmail teams. They’ve already pushed the idea that my gmail address IS both email and IM. They’ve got the servers running in parallel. It’s simply a matter of building out a test setup, defining how the standard will work, and getting gmail to support it.

This is not to say that there aren’t others who could do it. My ex-employer, Yahoo! could do it, but fixing the future of email is probably not a high priority. Microsoft could do it, but they have a hard time adopting open standards and wouldn’t be trusted by the open source developer community who maintain the current technology stack which makes email work. Other than google, the only other player who i could adopting this and pushing it forward is actually AOL. While mostly ignored, AOL has been pretty decent at adopting and pushing new technology and could see this as a way of getting back in to the lead setting trends for the future.

While we wait for somebody big to adopt it, building proofs of concept, making a system which will work, would probably push forward the case for replacing smtp with xmpp as our global email delivery system.

Speaking about Fire Eagle

2008-05-02T01:00:00Z

Last month I spoke about Yahoo! Fire Eagle at the Emerging Communications Conference. Lee did a great job putting together the conference, and had everything recorded. I feel i did an ok job explaining Fire Eagle, but perhaps i had too much coffee to make up for being on the tail end of a dozen straight days of conferences.

So if you’re interested in Fire Eagle as it relates to the mobile and telephony world, this is a good talk. The slides are online at slideshare.net.

It’s humbling to see a video of yourself speaking. Public speaking is really hard. I’ve been trying to work on my presentations, and i’m getting better over time.

Next week i’m going to be speaking at xtech in dublin in a longer talk about Fire Eagle. I’ll get in to the actual api’s and bit about building apps with Fire Eagle. In July i’ll be co-speaking with Kellan about using jabber for web services in Beyond REST? Building Data Services with XMPP PubSub at OSCON 2008. In between i’ll be at Google IO and Rails Conf / Caboose Conf, but thankfully i won’t be speaking, unless i get inspired to do a lightning talk.

Funny OLPC Story - How do you delete files?

2008-04-30T20:15:00Z

So today i sat in on a meeting with some folks in the Uruguayan government who were trying to get ebook type educational material to laptops of the kids who have them now in uruguay. There are about 200,000 laptops being distributed, and it would cost too much to print books with the material, so they figured these laptop things might be a good way of doing it. In the discussion they told me this little story about tech support and the OLPC or Project Ceibal as it’s called in Uruguay.

It seems that during the first trial in the Florida Department of Uruguay they were having a problem. The kids it seems were downloading too much stuff from the internet. The laptops have a 1 gb flash drive, so it’s pretty easy to see how it could fill up. So the teachers told the ceibal folks that this was a problem, the drives were filling up and nobody knows how to delete files.

Well this is a problem, so there were meetings, and more meetings, how to delete files, they called up the University of Uruguay’s Engineering Faculty and investigated further. After four months of back and forth they had the answer and somebody traveled up to Florida (the uruguayan florida) with the answers and a training to teach the teachers how to delete files. It was after all what they asked for.

When they got there the teachers said, “oh, that! The kids figured out how to delete the files and manage them months ago.” Which is of course the whole point of OLPC, the kids can use the tech, it’s open, hackable, and explorable.

Another story is a friend of mine was visiting his cousin and the cousin was excited because he just got one of the laptops. But he said there was a problem, some of the interface was in english. My friend, being a programer, sat down and tried to figure it out. It seems that was some problem with the packages, he’s not exactly sure what’s wrong. But at one point “Save” was translated as “Salvar” instead of “Guardar” Salvar does mean save, but in the kind of way that Jesus Saves. Not the kind of thing you’d do with files, which is Guardar which might literally be translated something more like ‘to put away’ than ‘save’. I asked the OLPC folks about it on irc, and they said that perhaps the build being shipped out in uruguay is out of date with what they currently have released. Clearly they need a good logistics person / team to do release management and handling lots of branched distributions. Not an easy task.

On the whole people seem excited about OLPC. It would be good if the Uruguayan government could do something about class size, 1 teacher for 40 students is the REAL education problem, but the laptops help.

Two conversations about tests and software development

2008-04-25T19:58:00Z

I’ve had the privilege of working with many developers over the years on a diversity of projects. I like tests, i think they make the software development process more reliable and help keep code from being unmaintainable disasters. So i find it interesting to see why inspires people to write tests, and why they give up on them.

Recently i had a very similar conversation with two very different developers, one i’ll call TestFail and the other i’ll call TestHeavy.

Rabble: “I see there are tests on this project, but they don’t seem to pass anymore.”
TestFail: “Yes, I was writing tests but then things got busy and I couldn’t keep them up. The tests feel out of sync with the application 6 months ago. I was the only developer on the project and i just couldn’t keep up the tests and getting releases out the door.”

Then i had another conversation with a different developer on the same subject.

Rabble: “I see there are a lot of tests for this project.”
TestHeavy: “Yes. The thing is when a project gets really pressured for time it’s the tests which make it possible to add features and keep moving forward. Normally I just develop on my own and don’t tend to work with other developers on projects. I found having tests means i can keep getting releases out the door.”

In both cases i’m paraphrasing from memory, but the thing which struck me is that two smart developers used the same two reasons to come up with totally different conclusions around testing. The two principal points are, i’m working on my own, and there is heavy time pressure on releases.

The code bases in question, i’ll keep this all anonymous to respect privacy, are also really different. First off, the applications do totally different things, but it’s enough to say that they are both fairly complicated web applications built in rails.

TestFail’s application has 44075 lines of application code and 778 lines of tests. Over the last year the code base has grown by over %50. On the other hand, TestHeavy’s code base is 4014 lines of application code and 4802 lines of tests (technically specs as it uses rspec). TestHeavy’s code base is younger, but over the last 6 months it’s only added a couple hundred lines of new code. Many new features have been added, but the size of the application has stated more or less the same.

While i clearly prefer TestHeavy’s approach, what puzzles me is the path taken by TestFail. There are many interesting things to be learned by watching the techniques of an effective test driven developer, but that’s kind of an easy problem. Just watch what they do and document it.

Harder is to understand why somebody fails. When somebody starts writing tests but finds they aren’t a useful part of their software development practice. It could be considered the ‘falling off the wagon’ problem. Really to me it’s a question of why is testing not the path of least resistance. What needs to happen for a developer to internalize testing.

If tests are written, and they are relevant to the the functionality of the application, the code base stays smaller, faster, and is more maintainable. But telling people that it’s a good idea to write tests, even getting them started and using tests isn’t enough. Figuring how to help people continue is even more important. I suspect that a large part of the problem is that people partially start using tests, but they don’t ever get a development environment which is setup to encourage testing. It’s painful and difficult as opposed to making debugging easier and development faster.

Does anybody have experience with trying and failing to stick with testing. Why was it?

Building flex 3 (flash) swc libraries on mac os x with ant and flex builder 3

2008-04-13T21:35:00Z

I strongly believe in learning new languages and platforms. While i’m super happy with the ruby / rails / merb world, I know it has it’s limits. Most of the time when i look farther afield i think of hacking on lua, erlang, or smalltalk… But recently i had an app idea which struck me as something which really was a good fit for Air. Adobe’s version of Flash to run desktop applications. Let me say at first I though Air was stupid, but over time i’ve come to realize that for some applications, it makes lots of sense. It’s similar to my thinking Flash was stupid until it changed the world of online video, and ActionScript was some stupid director derived toy language until I realized that these days it’s EMCA script, basically the same damned thing as Javascript.

So i set out on the path of building an Air app. I asked my old buddy, Dom and he said there were two IDE’s for Flex development, flex is the updated version of flash which is more programming like vs Director scripting like. Flex draws on the java world’s traditions more than the scripting world, a sin may be forgivable if i am able to build this cool app i’ve thought up. For Flex there are two IDE’s Apatna (free and ‘professional’ edition) and Adobe’s own Flex Builder 3 which is payware, but you get a 90 free trial to play with. The core is actually free, the SDK to build flex apps and it comes with no IDE, but just command line tools.

Flex draws on Java, there are tools like ant which are used for all the build processes. Really the IDE’s are both based on customized versions of eclipse. I’m not a fan of IDE’s they seem to be needed when the development processes has gotten to complicated to fit inside your head and try and make everything point and click. Silliness. Flex can use html to render an interface, but it also has it’s own proprietary xml for laying out interface elements. It creates a dom like thing, which you can attach action script events to… Once you get over how different it is from real web development, at some level it’s the same damned thing. It’s worth stating, that you can make flash as standards compliant, open, accessible, etc as html and unobtrusive javascript, but nobody seems to actually do it. Just yesterday i was noticing that yahoo map’s flash app wouldn’t let me paste in an address. Just like rails and django encourage you to do the right thing by default, flash defaults to clunky custom interface elements with broken accessibility.

Anyway, back to what i was saying. How to make this stuff work. There are it seems a lot of people with flash / flex development blogs which is useful, but on the whole i’ve been very unimpressed with the documentation. There’s a HUGE amount about how to do various UI widget things, and very little about toolsmithing, libraries, building out flex as a platform.

The thing is, i know how a rails app works, how php plays with apache and a load balancer. I don’t know how these flex apps come together. It found lots of tutorials on building toy apps, but not a one ever mentioned using a library you find online and incorporating it in to your application. I suspect it’s because the open source tradition is pretty week in the flash world. Just like in VB and Java, because you pay for the IDE, it creates a culture of pay to play. It stands in stark contrast to the scripting world of Perl, PHP, Python, and Ruby. In the scripting world we build libraries for our own uses, then release them for the community to use and help maintain. The only libraries i’ve found to be released so far in the flex world, come from Adobe employees themselves. Clearly it’s a failure of community that there is repository of open source libraries like CPAN, Ruby Gems, PEAR, and the Python Package Index. What the adobe does have is RIA Forge, as far as i can tell RIA is some term Macrodobe made up. The thing is, RIA forge is just a directory linking to libraries around the web, kind of like RAA. It has no build process, no clearly defined way in which libraries are included in your larger application. It solves the, how do i find what’s out there problem, but not really the how do i easily include it in to what i’m building.

Which brings me to the whole reason i wrote this blog post. I spent the last day or two trying to figure out what to do with libraries like these: as3corelib and as3awss3lib. They are the core action script 3 libraries for things like md5 hashing, json serialization, and general string, date, and time functionality. The second one is action script 3 library for working with amazon web services. Pretty useful and straight forward stuff. Don’t ask my why the as3corelib isn’t just included for use everywhere all the time, it’s an extreme case of batteries not included. In ruby i’d run ‘sudo gem install aws-s3’ and i’d magically get the libraries i need. There are pretty straight forward alternatives php, python, and perl.

But these libraries aren’t distributed as binaries, packages, nor anything easily installable. What do you do get is a note saying: “There is currently no zip archive available. Please check the code out using subversion. ” That’s pretty much it. When you do check out the svn repo you get this:

bas1:~/code/as3awss3lib-read-only rabble$ tree .
.
`-- src
    `-- com
        `-- adobe
            `-- webapis
                `-- awss3
                    |-- AWSS3.as
                    |-- AWSS3Event.as
                    |-- Bucket.as
                    `-- S3Object.as

5 directories, 4 files

For files, in a nested set of directories which seems pretty straight forward. I want to be able to access these files from my app, so i simply did the obvious thing, from a scripting background, copied the whole com dir and children over to my lib directory in the app i’m building. Needless to say that didn’t work. I tried moving around the files to various other directories and that didn’t work either. I read everything I could find in the documentation and on various community created sites, dead ends. I tried asking in the #flex irc channel, nobody seemed to even understand my question. I tried pinging friends who work on flash stuff at Adobe as well, i got blank stares and silence.

What was going on here? Clearly somebody at Adobe had build this library, clearly it was intended to be used. No where did i find any documentation on how to build these things. The as3awss3lib had more files than as3awss3lib, but still there was no clear path as to what to do with it. Adobe has spent millions of dollars on documentation, help, blogs, building libraries, and trying to create a community around their platform, and I couldn’t figure out how to use a simple library to add to my code?

I think a lot of it comes down to assumptions. People in flash/flex are mostly focused on user facing eye candy and not on the underlying libraries. There isn’t a lot of discussion about the libraries or the work which goes in to them. Many apps are build and left running as they are. They aren’t things which are maintained and run over time with updates.

Eventually i tracked down what i needed to do. It is assumed that although the repositories are called libraries, the thing you want to do is compile them in to swc files. You can’t just include other people’s code as libraries in to your own code tree, you’ve got to compile them. These compiled libraries and ‘assets’ (i guess you can have a ‘library’ of non-code things like images) are called swc files. These files need to be compiled and included in your application, before and separate from the build process for your own code. Again this is a little java like. Not necessarily wrong, but quite different from how most dynamic languages work. ActionScript is a dynamic language, somebody at Adobe clearly fell in love with java as the way enterprise professionals do things, and added some clunky extra steps to the process.

So now that we know that you can’t just include code in the lib directory of your app and have it be available to your app at run time. This is a big leap. It’s part of why i’ve found nobody who understood my questions. Once i made the leap then it was pretty easy to figure out that i needed to use ant to build the app i tried running ant and got errors. A command line utility and errors, that’s something i can debug! To run ant you simply cd to the build directory and type ant.

bas1:~/code/as3corelib-read-only/build rabble$ ant
Buildfile: build.xml

properties:

lib:

BUILD FAILED
/Users/rabble/code/as3corelib-read-only/build/build.xml:63: 
  Execute failed: java.io.IOException:
  C:/Program Files/Adobe/Flex Builder 2 Plug-in/Flex SDK 2/bin/compc.exe: not found

Total time: 0 seconds

This brings me to my other issue which i find so surreal about the Flash / Flex / Adobe world. It’s a windows world, everybody else is an after thought. It’s full of arcane paths, and executables have their own funny little suffixes so that the OS can figure out that they are executable. In the unix world, which includes linux, bsd, cygwin on windows, and of course mac os x, there are standard conventions for figuring out the paths to libraries, executables, etc…

Ant didn’t seem to like spaces in file names so i fixed that, but then there was still a big problem.

## Change this:
# The location of the Flex 2 SDK on your sytem.
flex2sdk.bin.dir = C:/Program Files/Adobe/Flex Builder 2 Plug-in/Flex SDK 2/bin
flex2sdk.lib.dir = C:/Program Files/Adobe/Flex Builder 2 Plug-in/Flex SDK 2/frameworks/libs

## To this:
# The location of the Flex 2 SDK on your sytem.
flex2sdk.bin.dir = /Applications/Adobe_Flex_Builder_3/sdks/3.0.0/bin
flex2sdk.lib.dir = /Applications/Adobe_Flex_Builder_3/sdks/3.0.0/lib

And and now ant gets a little farther down the path of working.

bas1:~/code/as3corelib-read-only/build rabble$ ant
Buildfile: build.xml

properties:

lib:
     [exec] /Applications/Adobe_Flex_Builder_3/sdks/3.0.0/bin/compc.exe: 
     /Applications/Adobe_Flex_Builder_3/sdks/3.0.0/bin/compc.exe: cannot execute binary file
     [exec] Result: 126

BUILD SUCCESSFUL
Total time: 0 seconds

Don’t let the BUILD SUCCESSFUL message trick you! It didn’t actually work. This time it found compc.exe but as you’d expect, this is a mac, and on mac (and it’s unix cousins) exe files don’t run. At first i thought maybe Adobe was doing some craziness where they just decided to make all the executables on every platform .exe. Turns out that wasn’t the case, setting the executable flag didn’t help. What adobe does is include a separate file, sans .exe, which is the version of that app for normal operating systems. So it requires another change to the build.properties file.

#change
asdoc.exe = ${flex2sdk.bin.dir}/asdoc.exe
compc.exe = ${flex2sdk.bin.dir}/compc.exe
mxmlc.exe = ${flex2sdk.bin.dir}/mxmlc.exe

#to
asdoc.exe = ${flex2sdk.bin.dir}/asdoc
compc.exe = ${flex2sdk.bin.dir}/compc
mxmlc.exe = ${flex2sdk.bin.dir}/mxmlc

Again if it wasn’t assumed that these libraries would be built in a mono-platform windows only world, then the ant build.xml file would be written in such a way to look in the obvious places. Once everything is in place, then ant works!

bas1:~/code/as3corelib-read-only/build rabble$ ant
Buildfile: build.xml
</p><p>
properties:
</p><p>
lib:
     [exec] Loading configuration file /Applications/Adobe_Flex_Builder_3/sdks/3.0.0/frameworks/flex-config.xml
     [exec] /Users/rabble/code/as3corelib-read-only/bin/corelib.swc (79242 bytes)
</p><p>
BUILD SUCCESSFUL
Total time: 7 seconds

As far as i can tell, the ./libs/ directory is a good place to put these kinds of libraries. Presumably they get bundled in to your application you build an air / flash swf for distribution. I haven’t gotten that far yet. If i stay motivated and keep moving forward with this project, i’ll keep blogging my way through this mess. I am tempted by the shiny bobble of pretty and small cross platform GUI desktop apps which lies on the other end of this journey.

It appears that some libraries need to be compiled, like corelib, while others, the as3awss3lib can just be dropped in the src directory with their full path, ./com/adobe/webapis/awss3 What’s not clear is why it’s one way or another.

txtmob gets subpoenaed - data retention in the surveillance era

2008-03-31T13:54:00Z

Tad’s TxtMob has been subpoenaed by the new york police department demanding the records of everybody using the service during the RNC protests in 2004. I got to know Tad through our collaboration on various projects related to activism and emerging telephony. I even poked around and did a little work on the txtmob code.

All that is to say, i know a thing or two about the background, how txtmob works, and issues of data retention and privacy. Nothing i’m saying here is based on privileged information you can’t get from reading the source.

The reality is today, we live in a world where much more information about our daily lives are collected than ever before. There is this scary nightmare scenario, the corporations / state know where everybody is all the time, what they are doing, who they talk to, etc… Most of the time this information is being used for marketing sell more stuff. While it’s slimy, it’s not really evil. What happens when the state starts to use the same information to suppress free speech and dissent, things get much more serious.

This is nothing new, things evolve, but it’s a world we’ve been dealing with for a while. Years ago there was one phone company, they had records of everything, you couldn’t know if somebody accessed your records because the phone company wouldn’t tell you. With the internet, and now emerging voip / open source telephony, the ability to run your own telecommunications infrastructure has emerged.

The work of the anarchist geek community for the last decade or so has been to build this autonomous infrastructure. Indymedia’s been a big part of this task, to create a privacy enhanced participatory media network. Another has been blasterisk an asterisk based phone system with dial in numbers around world for activists to use to make phone calls and do international coordination. For email there is riseup which provides email hosting and mailinglists for hundreds of thousands of activists. Social networking tools for collaboration and organizing, crabgrass. My own, protest.net, a calendaring service, etc… While txtmob has been compared to twitter, txtmob predates twitter and was very explicitly talked about as a model to be copied / learned from in the creation of twitter.

What all of these services have in common is they are providing critical infrastructure to social movements and they take privacy seriously. New tools have been build to allow for making privacy easy.

Txtmob is included in those sites because it does things right. First when you delete your account, remove your phone number, etc… it really does remove it. No record left of your messages, your login, your phone number. Secondly txtmob does not use a short code, nor is there a legal agreement between txtmob and the carriers. Rather txtmob uses the path of least resistance to deliver messages, finding holes and cracks in the sms system to let messages pour through. Tracking txtmob messages is more like tracking p2p traffic than the american idol voting via sms. This not following the rules is probably why the NYPD went to txtmob instead of the carriers. The data is many places, but txtmob is the easy place to get it, if it’s there.

The task of going to the carriers or the NSA to get the txtmob data is much harder than getting it directly from the source. First it’s a HUGE data mining task. It’d involve using something like hadoop or google’s map/reduce to load up the data, and then tracking down a few thousand sms’s out of a stream of trillions. Most of the time operators are lazy, it’s easy to get them to comply with even questionably legal orders for data, my employer is a great example. Service providers tend to log MUCH more data than they need, in the name of security, potential datamining, etc. If we don’t have that data, then we are both able to follow the law, and protect our users.

() It’s worth noting, that there are new EU rules / laws which require extensive data retention. It’s much worse than the situation in the US. So much for the EU being concerned about privacy.

Fire Eagle talk at the Emerging Communications 2008 Conference

2008-03-15T17:31:00Z

I gave a talk on Thursday to the Emerging Communications conference. It was my first chance to speak publicly about Fire Eagle.

| View

The nice folks at the made a video of the original launch announcement at etech last week by Tom Coates. Thankfully nobody made a video of my talk at ecomm. :)

It's Danger Day, Fire Eagle Flies

2008-03-05T17:41:00Z

Today we let loose the developer’s launch of fire eagle a location broker platform from yahoo brickhouse. It’s what i’ve been working on for the last 6 months and is the first publicly facing rails app to launch at Yahoo!

I’m really excited by Fire Eagle. It’s a system which lets you collect your location from any number of sources, such as your cellphone, and then provide that back out to other applications. Fire Eagle then lets you fuzz your location and control who is using it. Share the city you’re in with dopplr, the neighborhood with facebook, but let the taxi locator see your exact location.

I’m proud of the work i’ve done fire eagle and to have worked with a great team, including Tom, Seth, Jeannie, Skylar, Schuyler, Ayman, Kevin, Marc, Phil, Salim, Mor, and Sam.

We’ve got an irc channel, #fireeagle on irc.freenode.net, wiki, twitter bot, and mailinglist. Come join us and play.

Launching a ruby on rails app at yahoo has been an incredible experience. We’ve had to figure out how to marry a stable mature platform with the agile, test driven, DRY model of development which Rails has popularized. I’m particularly excited that we were able to adopt oauth and open standards like georss and geojson.