Subject: Re: [FooCampers] Open source in the cloud era
From: Philip Ashlock
Date: Wed, 17 Mar 2010 21:55:30 -0400
To: A mailing list for Foo Camp alumni

I realized that I didn't get too specific with talking about important applications of open source other than the general trends around cloud computing. My focus is on open source and open systems for municipal public sector technology and there's a lot of work to do there and a lot of untapped potential for open source. The open source paradigm makes total sense for municipalities because they are not competing with one another, they are publicly funded, and they largely share the same needs and architecture. While open source makes logical sense in the municipal world, it's often blocked by incompatible culture, bureaucracy, and procurement policy. This is starting to change and we can see some massive headway being made out in the open west, but there's still a lot of work to be done.

The ideal will be an interoperable open network of municipalities using and collaborating on open technology. With the advances being made by a few key cities we should have some strong precedents in the very near future. For now I'll leave it there. I don't want to spoil some upcoming announcements and the talks that I will be giving at OpenGov West and Where 2.0.

Some information on the precedents and thinking around municipal open source and open systems leading toward new software ecologies and "SmarterCities™" can be found below, but look forward to some significant advances and announcements regarding this in the near future. &

Philip Ashlock wrote:
I see that you're already preparing to give your OSBC talk in a moment, but maybe I can still provide some useful thoughts for the future.

Let me start with a few assumptions and generalizations:
We can see all these qualities in the transformation occurring in the cloud era and I think we can already begin to see the evolution beginning beyond the cloud era. The degree of open/closed and centralized/decentralized waxes and wanes with each new stage of technology, but it occurs at a different layer in the stack each time. Here's a rough sketch of phases in physical infrastructure. I know these are arguable, but I think it illustrates a general trend.

centralized operations: mainframe with terminals
distributed operations: pc
centralized networks: pc + pc/server (pre TCP/IP networks)
distributed networks: pc + pc/server (internet)
centralized network services: mobile + pc + centralized clouds (now)
distributed network services & more distributed clients: things + mobile voip/openbts + pc/tablet + private cloud + mesh (future)

Similar paralleled phases can also be associated with software and data.

Where we are right now is very analogous to a mainframe with dumb terminals, but I think this is changing. We are now beginning to see the potential of a more distributed infrastructure for both the web and for the internet with more open standards for web services and data as well as open standards and open source for private cloud infrastructure and telephony. Obviously there's a near religious debate about the future of the web and the role or viability of the more orthodox and academic models for a semantic web versus the more iterative scrappy and often more bootstrappable forays into potential standards (elder brilliantly wise Tim Berners Lee Vs younger brilliantly naive Tim Berners-Lee). I think lots of different strategies will have a role to play. I'm certainly no expert on this stuff, but I often group things into the traditional semantic web standards (eg RDFa/SPARQL, XMPP, etc) and the more nebulous "open stack for the web" emerging standards for web services associated with the social web (eg OpenID, Webfinger, PortableContacts, oAuth, PuSH,, Salmon, etc). Anil Dash has a nice take on these two different approaches, comparing XMPP to PuSH, but of course Google is flanking the web with these standards from both sides: Wave on one side and Buzz and other services on the other. In the end, we'll be using parts of both.

My point with all of this is that web technologies and related protocols are gaining potential to parallel the openness and distributed behavior of the underlying TCP/IP stack. I think much of the vision of the semantic web is for web technologies to abstract and act much more like internet protocols. My guess is that this is very much what will eventually happen. This is also coupled with the emerging trend for more robust private cloud computing that can interoperate with existing centralized cloud providers.

As was mentioned earlier, a lot is happening in the emergence of the open source private cloud with Ubuntu private cloud / Eucalyptus / Nasa Nebula. There's also activity around the open source abstraction layers to provide interoperable App Engine style deployment on many different cloud infrastructures, eg libcloud and Silver Lining.

There's enterprise-level private cloud and states like Michigan and Utah providing a sort of federated public-services private cloud infrastructure for their regions rather than having municipal IT rely on the big centralized providers. There are also attempts to recontextualize what the cloud is and really bring the features we've come to expect to the edges of the network in our homes and offices. Even though robust virtualization and massive scalability is not yet there, you could say that things like the Pogoplug and other uses of the Sheevaplug and plug computing are a frontier for cloud computing. Eventually cloud computing will be much more cloud like: nebulously ubiquitous. It won't be that you don't know exactly where in the AWS network your computing is, it'll be that you won't know where in the meshy network of everything your compute power and data are. Obviously, we've already seen signs of this with SETI-like grid computing, botnets, bittorrent, etc.  Ideally this transition will also be coupled with robust mesh networks and OpenBTS or whatever it'll take to evolve out of the current centralized control of the networks.

Ultimately, where open source and open systems matter most is at the edges where they're being threatened by litigation or a simple lack of established openness. This is clearly playing out with cloud computing, smart phones, network infrastructure, and the legal fights for free culture and FLOSS in general. The value of open source is always clear: distributing the value of innovation, the freedom of nuanced control, and the robustness of an open decentralized system.

Sorry I didn't get this out before the OSBC talk. Will that be available to view on the web in the near future?


Tim O'Reilly wrote:
For years, I've been saying that open source needs to adapt to the cloud era.  In pieces like The Open Source Paradigm Shift and What is Web 2.0? I argued that data driven applications in the cloud provide a fundamental challenge to open source.

Yet it's clear that open source is still extremely relevant.  For example:

The Android open source hardware and software stack is playing a major role in mobile.

Arduino is sparking an open source revolution in sensors and DIY projects based on them, and there are important open source projects in machine vision like OpenCV.  Willow Garage is building an open source robotics platform.

Open source tools are playing a major role in those aforesaid cloud data applications, in areas from hadoop to cassandra and other nosql tools.

Open source plays a major role in web operations.

Tools like Ushahidi are crowdsourcing crisis data, and starting to find new uses.

Openstreetmap ditto.

Nat Torkington just wrote a phenomenal post about the need for tools to manage open data as if it were an open source project:

I'm giving a talk on this subject at OSBC next week, and I'd love to hear from this group about what are some of the most interesting open source projects on your radar, and what makes them important.

I'm sure that there are lots of projects that I'm not aware of, and I'd love ideas for projects that I ought to feature in my talk.

Tim O'Reilly, Founder & CEO O'Reilly Media
1005 Gravenstein Highway North, Sebastopol, CA 95472,,

FooCampers mailing list