Scott Hanselman

RFC: OpenTweets - Why is Microblogging centralized?

May 3, '08 Comments [21] Posted in Musings
Sponsored By

I'm thoroughly enjoying Twitter (follow me!) Apparently others are enjoying it also as there's a funny estimate that it's costing $14 billion in lost productivity (via a back of envelope calculation). There is about a million people on Twitter and maybe 60k adding each a month. Certainly it'll be blocked by most corporate firewalls soon for just this reason.

Twitter is also down ALL the time and it's been having weekly (daily?) scale problems for a YEAR, culminating in rumors that the development team is leaving Ruby on Rails that have been denied by Evan Williams (via a "Tweet," of course). However, that doesn't change the fact that Twitter is down so often there is a site dedicated to "Twitter Down Art" showcasing all the pictures that Twitter puts up when their sites are down. Google changes their art monthly and during holidays, but never goes down. Twitter goes down so often that they use that as an opportunity to change their art! So far, in this era of transparency, no one has explained in technical (or any) terms what the problem with Twitter is and folks are getting impatient. I think that it would be a great PR and Karmic move to just start a Twitter Technical Blog and share the crazy IT problem of the day. Surely someone is sleeping on a cot next to the TweetBoxes and has a story to tell.

There's another rumor that Twitter is worth about $150M. Seems to me that the second bubble hasn't popped. I'm certainly no business man, but we can build this better ourselves.

Twitter is centralized, which is insane (unless you're Google or Amazon or Live.com, etc, which are actually distributed in their own special way).

Twitter is IRC, it's IM, it's chatrooms, but ultimately it's just microblogging with a multi-format API. Here's the XML for my Twitter Account but it looks nicer as Atom.

But I already have a blog! Why use a service when I already host a blog? I hooked up Microsummaries to DasBlog two years ago. There's clearly precedent.

Two folks I know already have Twitter Backup Feeds. Both LazyCoder (backup twitstream) and Dave Winer (twitstream) have Backup Twitter Feeds. Dave takes it even farther and makes his twistream discoverable via a link in his blog:

<link rel="twitstream" href="http://twitter.scripting.com/daveRss.xml" />

With backups like these, who needs Twitter? There is already a very responsive group of Twitter client applications some of which, like twhirl cross-post to Pownce and Jaiku, two Twitter competitors. Russell Beattie has an idea for an Open Twitter Server called Peep using Jabber.

I propose that we distribute Twitter into generalized spec for microblogging and just use RSS or Jabber for the transport. I think the very-fast-rev'ing Twitter clients would support it. 

Thoughts? Should Twitter be a service or should Microblogging be an Open Thing?

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. I am a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by ORCS Web
Saturday, May 03, 2008 6:41:47 AM UTC
Awesome idea! I'm already thinking about an ASP.Net MVC Implementation!

I don't think we'd need a new spec. As you demonstrated, ATOM and RSS are perfectly capable of publishing this information. Essentially, all we'd need is client support for things like "@" replies and SMS support (that would be trickier).
Saturday, May 03, 2008 9:02:09 AM UTC
I think that a well-spec'ed system is critical here -- we really do need a few pieces that don't yet exist. While RSS or ATOM etc will help us publish the tweetfeeds, we need a good standard for addressing people (i.e. what follows the @ to create an @reply?), we need a standard protocol to exchange the replies and private messages as well.

I've typed up some thoughts on the matter.
Saturday, May 03, 2008 10:36:46 AM UTC
Just wondering how you decentralise the usernames without making them as long as email addresses.
Saturday, May 03, 2008 12:17:56 PM UTC
I'm primarily wondering how you handle the text messaging portion. I guess email is a pretty good possibility, and then you'll have a large group of users that use mobile browsers. But, I'd really want to ensure that I get tweets onto my phone via text when someone @replies to me or when someone I'm following posts.
Saturday, May 03, 2008 1:07:47 PM UTC
Yep... definitely needs to be open. As others have said, the problem is communication. It's one thing to have a micro-blog but the value of twitter is in the social connection.

As for replies... seems like there are 2 ways to go... a limited set of semi-replicated servers whereby the same service was available across all of them and really it was a distributed type of deal but somewhat limited. The other way is to make it so that anyone can have their own server... it just means replies will be longer and more complex because they've got to have some routing information. If that were the case, I'd probably vote to split out the replies into a separate field (due to length)... but that would also provide a more reasonable way to reply to multiple folks.
Saturday, May 03, 2008 1:15:02 PM UTC
You might be interested in this project: http://lagesse.org/ubertwit/. It has a lot of similarities to what you write about here!

Rob
Saturday, May 03, 2008 1:15:19 PM UTC
Scott:

Fantastic idea, I love it. Rory, use open id for usernames.
Saturday, May 03, 2008 2:01:34 PM UTC
The only big problem I see is the SMS aspect of it all, I frankly don't care for SMS, but I believe a great number if twitter users rely on SMS. Getting the telco crooks to join any open effort would be an interesting challenge to say the least.
If we exclude SMS, then yes, the whole thing looks simple enough to be made open.
But, contrasting to what you just said about lost productivity, do we want to promote more distractions and shorter, sometimes incoherent, messages?
I'd like to see some form of tagging, better than hashtags, where I could follow someone but exclude uninteresting messages like "my dog just barfed on the carpet".
Saturday, May 03, 2008 2:06:31 PM UTC
ALL twitter is a centralized 'finger': http://www.furrygoat.com/2008/04/08/is-twitter-useful-or-just-more-noise/ that provides easy (well, sort of easy) discoverability.

Twitter, in the end, should just be a 'ping' service that is hit from your blog when you have a new microcontent post.
Saturday, May 03, 2008 2:50:06 PM UTC
As Rory points out, and Dave gives a suggested workaround, you need a centralized mapping of "@nickname" and RSS feeds. I'd think about a "feedburner for microblogging" kinda service, with some extras, for tracking for example. The Jabber layer would be above it. That would be the best way to decentralize the service, and allow clients to plug into the system.
Saturday, May 03, 2008 4:45:57 PM UTC
Yes, an RFC for twitter is something that the world needs right now. Should we also standartize Pokes over the Facebook and make them open?
Alex
Saturday, May 03, 2008 9:48:43 PM UTC
It seems to me that getting two way SMS communication would prohibitively expensive to de-centralize. Usernames would also have issues, but those could be managed.

Assumeing you want to enable SMS services and have the shortest possible usernames, then it seems to me that easiest way would be to leave the SMS gateway and @username directory centralized, but de-centralize the rest of the system.
Saturday, May 03, 2008 11:29:13 PM UTC
I don't think SMS is an issue here - unless you are talking about an _immediate_ roll-over to external SMS if Twitter "died". But SMS msgs could still be queued for delivery once the Twitter API "came back to life".

Let's not try to solve _every_ potential eventuality right now. Let us focus on one thing to start - a continued conversation with your Twitter community even if Twitter is down.

Rob
Sunday, May 04, 2008 12:36:45 AM UTC
Wouldn't the right way to do this be with multi-casting? Without a centralized rendezvous service there's no other feasible way to have a generally available broadcast service which is what you'd need of you were going to make a Twitter line service sans Twitter . Relying on lots of people to out of the goodness of their hearts to set up S3-based RSS feeds or whatever is quite a stretch, and at the end of the days starts to sound a lot like USENET (minus alt.furries, hopefully).

I mean if we need such a centralized service, it's been around for ~20 yrs... it's called IRC. WorldWide, Distributed; can do both short and long form messages, you cn right programs the respond to traffics (bots) that do things (like send SMSs)..

re: SMS, A dedicated SMS service by the way, starts at about US$1,000 per month to rent the SMS short code; then there are the fees you part to inject (send) and SMS into a carriers network, if I read the docs correctly. Not a cheap proposition.

wrt the bigger picture: I am not sure I get the compelling need to disinter-mediate twitter in the first place; not all centralized services are inhenrently evil or bad and Twitters issues may or may not be caused by its architecture (since they have not published their code or a detailed architecture spec we really have no way of knowing). All popular services have growing pains.
Sunday, May 04, 2008 1:48:04 AM UTC
Frankly, if IRC had better clients (!mIRC) and permalinks (therefore reference-able and SEO friendly) we'd be done.
Sunday, May 04, 2008 2:12:00 PM UTC
Hey Scott,

For all the same reasons we don't have to worry about any one webserver going down I agree the microblogging architecture needs to be made decentralized and into a open standard.

If the Web was designed around a Twitter like architecture we all would be serving our web pages and blogs served off Tim Berners-Lee’s server farm at CERN right now.

I wrote a blog post with my thoughts - "The future of Twitter and Microblogging in education is an open protocol" @ http://hgilchrist.wordpress.com/2008/05/04/the-future-of-microblogging-in-education-is-an-open-protocol/
Sunday, May 04, 2008 9:01:48 PM UTC
Worth considering.
If the plan is to use a version of the Twitter API it will need some work: while the general REST approach is good, the XML format (when XML is being used) needs a closer look, and the response formats need proper documentation (and cleaning up, and in some cases an open version would need to add the innovation of certain features not actually being purely decorative but non-functional...such as those that always return the same result irrespective of the underlying data).
If the API is up for grabs, AtomPub might be a good choice. Anything, ANYTHING rather than XML-RPC.
Monday, May 05, 2008 1:10:50 PM UTC
James Bayer
Monday, May 05, 2008 2:40:36 PM UTC
Use XMPP/Jabber + XEP-0060 + "Atom over XMPP". That's all you need.

XEP-0060 is the PubSub extension to Jabber/XMPP
"Atom over XMPP" can be found at:
http://www.xmpp.org/internet-drafts/draft-saintandre-atompub-notify-06.html

bob wyman
Tuesday, May 06, 2008 3:14:52 AM UTC
Fo sho! I've been thinking about this lately as well. I want to forget about Twitter, along with it's horrible reputation that will evidently continue to smolder.
Wednesday, May 07, 2008 4:36:19 AM UTC
Man with 150 million we can all retire. Who cares if it is centralized or de-centralized, it is not our problem yet.
Adnan
Comments are closed.

Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.