silent servers and the Goog

I earn my living from designing and developing computer systems, and I do believe that there is a lot we can learn about the nature of the New Paradigm from the computer world. For example, the moves from hierarchical control-based structures to free-form distributed relationships, and from selfish greed-motivated commercial activity towards altruistic giving and creation for the love of it, are illustrated very well in the growth and success of the open source movement. The original MIT hackers were pioneers of the New Paradigm, and the free software and open source movements, that grew out of their activities, provide clear signals as to the way ahead in other walks of life.

However, these days we all bask free in the Jupiter-sized benevolence of the Great God Google. The Goog keeps all of our data in its cloud, and provides us with all of the computer services we need, foree of charge, through our web connections and browsers... Don't get me wrong - I think Google IS great. They are on a glorious mission to free us from the evil empire of Micros&*t and I am an ardent user of their services: gmail handles all my emailing, and has done for 3 years or so now. Google docs has freed me from the horrors of MS Office, and from interminable waiting for Open Office to waken up... I get my daily RSS feeds fix via Google Reader. My web portal is iGoogle. I often go to Google News for news. I always go first to Google maps for online mapping. I love Google Earth. I blog on Blogger. I even use Google for searching (no - really!) and generally only go elsewhere when their servers break down (in which case ask.com takes the silver medal)... but there we have the rub: even the Great Google sometimes breaks. So far I have not lost any data, but I have found myself with fully functional broadband, but no email facility (for days at a time in the early days of gmail, admittedly downtime has been much more rare in recent times). With alarming frequency I have had to resort to inferior services when the GGG homepage takes a nap. The downtime is small compared to the uptime, and given the size and technical strength of the Google behemoth you might think that it is churlish, even paranoid, to be concerned about Google unavailability, but I am looking at a bigger picture...

Other people worry about what the Goog might do (or be doing) with their private data - and this is a legitimate concern as we are dealing with a commercial organization here, and these days the track record of commercial organizations when it comes to ethics and honesty is not good (as an understatement...). When bankers cannot be trusted then why should we trust financially motivated geeks? But I don't really care about this issue - I feel I have nothing to hide and so they can do what they like with my data: so long as they don't lose it, I am happy. I am not giving them vital passwords or bank account access codes. The risks from misuse of my data seem small to me compared to the advantages of free (as in beer) access to that data from any web-connected computer, along with the peace of mind that comes from not having to safeguard my data from loss. I grew up using tapes and floppy disks and then hard drives to store all my data, and all of these formats offered the feature of regular catastrophic loss of data. Accordingly, much time was spent on making multiple backups of everything, and much of my hair turned white through the stress of trying to keep some organisation and method to all of the backups, to ensure that my vital data survived the regular melt downs of media and equipment.

What concerns me about Google is not that they might lose my data, nor even that their servers may be down on occasion, but that I might lose my connection to Google for sustained periods of time. I am currently typing this on in my new home in rural Italy, where I am at the mercy of Telecom Italia for the provision of the phone line and broadband (or even dial-up) connection that I require to pursue my profession with any degree of professionalism... forget about mobile connections, as my cellphone struggles to even deliver usable voice telephony thanks to dramatic mountains and solid medieval buildings that barricade out the loutish advances of the cellular networks. I may wait many weeks before I can connect again (and upload a splurge of blogging..), but I am actually enjoying the break from emails and emailing and RSS feeds and blogging and global news trawling that had become the core of my daily existence - I am forced to go for walks in the leafy mountain lanes, rediscovering the wonders of nature.. I am having to idle away many hours basking in the mediterranean sun, instead of burning my eyes out on my monitor.... this actually counts as a major improvement in the quality of my life. My work consists of system design and programming, and my productivity in these activities (the ones that pay the rent) has gone through the roof, due to my being denied the time-sucking pleasures of online communication, internet TV, youtube, and the rest. But if I had relied on the Goog to store all of my data, instead of just my emails and RSS feeds, then I would be unable to work at all, until Telecom Italia run out of excuses for failing to turn up on appointed days and times and connecting a phone line for me... With the joys of USB sticks I can shuffle my work once or twice a week to other computers that are more connected, and thereby upload it to servers, and life goes on sweetly and quietly and increasingly chilled out....

But all of this has set me thinking:what happens when the server farms start to die? What happens when the connections start to drop? Don't be fooled by the apparent reliability of the internet and its service providers in recent years: we are now plummeting into an abyss of global economic meltdown on a scale unprecedented in the history of mankind. Organisations will die suddenly and catastrophically, en masse, in an unpredictable fashion. Our financial systems, our political systems, our cultures even, are all crumbling to dust. The oil supplies are dwindling - we have passed peak oil. We have failed time and again to moderate our squandering of the Earth's resources, and instead have ramped up our consumption in an exponential fashion. Now it starts to crash down around our ears. The old certainties mean nothing. We are entering armageddon.

The internet was designed from the start to cope with nuclear holocaust - it is armageddon-ready. But we must expect huge losses of servers and connections, and a general breakdown in reliability. Once the USA sticks the finger up at China and defaults on its debt, forcing China to nuke the USA in order to safeguard the all-important oriental honour, much of Google may disappear in smoke. Even if we still have sporadic access to the web, we cannot rely on the continuing existence of centralized data stores. The paradigm of client server is past its sell-by date. Google wants to store all of our data in its centralized storage facilities - yes they have built in redundancy and distribution of data within their server farm architectures, but the whole lot may be pulverized to dust. Even if it is not, we cannot expect to keep having 24/7 access to it. The monolithic model is approaching extinction. Google is old-paradigm and we should be very wary of getting fully on their cloud.

The internet (i.e. the network) is bomb-proof (and, it seems, anything-else-proof too!) because of its new paradigm design: it is distributed, and it assumes that data will be lost en-route. If you have two working computers and a wire (or wireless) between them, the internet can still exist. Take out, at random, 99.999999% of the computers on the internet, and the internet can still exist and can still function. It is the primary wonder of the modern world. But our storage of data has to be done according to the same paradigm: we must assume that data stores will be lost, and that data will be lost in transit, and yet still provide a reliable persistence of whatever data is deemed to be important. Distributed technologies like bit-torrent and skype show the way forward: they utilize any machines they can, and coordinate these machines in a flexible way to provide data transmission and telecommunications that are every bit as armageddon-proof as the internet infrastructure itself. These systems operate in a viral fashion, and they are not susceptible to manipulation by saboteurs, censors or other rogues.

Every machine on the internet must be seen as a node: just as the internet itself sees each machine as an IP address, the data storage and data transmission systems of the future must see every machine as a homogenous cell. Each cell must be essentially identical in capability: they must be able to store and catalog whatever data comes their way that is deemed important - to the limits of their storage capacity (which will obviously vary enormously), and they must be able and willing to pass on their contents to other cells when connected to them and asked to do so. Data that is deemed important by many people will then be duplicated in many locations, while data that is deemed unimportant will be stored sparsely, if at all. Such a system will continue to provide utility, for as long as the internet itself does so. Monolithic data stores will die. Bye bye Google. Bye bye Wikipedia (may your glorious content be cached many times).

In this context the recent hyped announcement of Google Wave shows a failure by those in positions of influence and power to understand the way the world is going. Google Wave ( wave.google.com ) is a very interesting development: it creates a new document format, the 'wave' which wraps around (or replaces) many old types of document, such as email, instant messaging, wiki pages, text files, spreadsheets, image files, video files, etc. etc. Each 'wave' is stored centrally by Goog, or another 'server engine' provider. Instead of sending emails to each other, whereby I keep a copy of the email and you receive another copy, in the Google Wave paradigm, there is one central copy which we both can see online. I can then change my message to you after I first send it (indeed the whole concept of sending is essentially eliminated by Google Wave - we can watch our correspondents in the act of writing their messages, character by character. The process can scale up to multi-user collaboration (wiki style with instant messaging overtones) on one wave. All document types other than text messages are simply dragged and dropped on the wave, and thus become part of it. Clever stuff for sure, and on first encounter it can seem beguiling. Bye bye email, bye bye wiki, bye bye Google Docs? Hello Waves :)

Yes - just what I always wanted to do with my increasingly scarce time (once I have a web connection that is, and my current idyll is destroyed): watch my friends and colleagues, and a host of spammers also, as they laboriously type out, misspell and correct, delete and rewrite their messages... even more of an idle waste of time than Twittering... and yes, just what I always wanted, for those messages sent to me by my friends and colleagues to be changing in front of my eyes, hours, days, months, years, after I thought they were finished - for comments to be added here there and everywhere, at any time, in a totally non-linear and chaotic fashion - to be able to replay the evolution of these monstrous waves step by step ad nauseam.... the prospect is enough to drive me to permanently hitting the power off button and taking up farming full time. I have no doubt that a host of ardent believers will embrace Google Wave enthusiastically, and will become addicted to its gameplay, while losing sight of any real meaning in the content. More and more noise in our lives. Less and less value. I do not think I will be waving with the rest of the surfer dudes: I am just too old and crusty and cranky for such flash without substance. I want to craft and hone my prose before publishing it for others to read - I consider the time of my readers to be valuable and worthy of my effort. Less is more (OK OK I am not applying that maxim very well to this particular post....). The process of creativity is itself seldom an edifying sight: it makes a poor spectator sport. Google Wave will have its uses in brainstorming sessions and the like, and there is an admirable commitment to open source and to fostering collaboration, but while the centralized design (no duplication of data) obeys traditional database design ideology, it will not work during the apocalypse - the chances of the wave server being up, and of you having a working connection to it, at the time you wish to compose your opus, are slim. I am sure that the Goog will provide an offline mode so you can compose a message without being connected, but then all the bells and whistles that everyone is getting so excited about will vanish, and we are back to traditional messaging, without the benefit of multiple copies being distributed across multiple machines, and without the benefit of being stable and unchanging once posted.

No - Google Wave is not the golden technology of the new age, for us all to surf in blissed-out harmony. Something more appropriate must be put together before the sands of internet stability run out. I am going to search the web for a distributed messaging system that is suitable for the times we are entering. It should be very simple and very viral. If you know of such a system then please let me know. If such a tool does not exist then I may even build it myself. Watch this space :)