« Long player: bonus track | Main | Invite bonanza: Pownce, Freebase, iMedix »

Skype and the hedge fund problem

August 21, 2007

Hedge funds work on a simple wisdom-of-the-crowd principle: Because they involve an extremely large number of transactions, which smooths out the vagaries of individual transactions, the movements of financial markets follow predictable patterns, which can be discerned from a study of their past behavior. Deviations from the patterns tend to be shortlived, and by making huge bets that the deviations will quickly return to the norm, you can make a whole lot of money. As we've seen recently, though, things aren't quite as simple as the hedge fund operators assume. Sometimes, very weird things happen and the deviations become either larger or longer-lived than expected, at which point the big bets can unravel in very unpleasant ways.

Peer-to-peer networks, which also involve lots and lots of different actors doing lots and lots of different things for lots and lots of different reasons, work in a similar way, and a company like Skype, whose telephone network is designed to run on many thousands of computers spread across a big P2P network, has built its system on the assumption that usage patterns are predictable - even if the actions of any individual user are not. Last week, Skype ran head-on into the hedge fund problem. The network's behavior deviated from the norm in a way that was greater than the Skype engineers had planned for, and the system crashed. The catalyst was the distribution of a routine patch from Microsoft, which led to a cascade of unanticipated effects, as Skype's Villu Arak explains:

The Microsoft Update patches were merely a catalyst — a trigger — for a series of events that led to the disruption of Skype, not the root cause of it ... The high number of post-update reboots affected Skype’s network resources. This caused a flood of log-in requests, which, combined with the lack of peer-to-peer network resources at the time, prompted a chain reaction that had a critical impact. The self-healing mechanisms of the P2P network upon which Skype’s software runs have worked well in the past ... Unfortunately, this time, for the first time, Skype was unable to rise to the challenge and the reasons for this were exceptional. In this instance, the day’s Skype traffic patterns, combined with the large number of reboots, revealed a previously unseen fault in the P2P network resource allocation algorithm Skype used. Consequently, the P2P network’s self-healing function didn’t work quickly enough. Skype’s peer-to-peer core was not properly tuned to cope with the load and core size changes that occurred on August 16.

As our economy becomes ever more tightly and intricately networked, its continued operation will hinge on the assumptions that mathematicians and software engineers embed in the code that underpins it. Usually, the assumptions will hold. But usually isn't always. Weird things happen, even in the largest of crowds.


you could not have picked on two more extreme examples of the wealth distribution spectrum...hedge funds have managed to escape regulation because they say their investors are extremley wealthy and sohisticated and don't need protection...

Skype has democratized long distance, particularly international calling...200 million global users, and they just barely report profit...

Posted by: vinnie mirchandani [TypeKey Profile Page] at August 21, 2007 05:19 PM

In this case though, the idiot hedge funds were all in the same trade. there was no math...it's called crowdsourcing I think :) .

Posted by: howardl [TypeKey Profile Page] at August 21, 2007 09:33 PM

I keep seeing Taleb Black Swans all over the place. If you've not read him, I strongly recommend it.

Posted by: Thomas Otter [TypeKey Profile Page] at August 22, 2007 11:34 AM

Post a comment

Thanks for signing in, . Now you can comment. (sign out)

(If you haven't left a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Thanks for waiting.)

Remember me?

carrshot5.jpg Subscribe to Rough Type

Now in paperback:
shallowspbk2.jpg Pulitzer Prize Finalist

"Riveting" -San Francisco Chronicle

"Rewarding" -Financial Times

"Revelatory" -Booklist

Order from Amazon

Visit The Shallows site

The Cloud, demystified: bigswitchcover2thumb.jpg "Future Shock for the web-apps era" -Fast Company

"Ominously prescient" -Kirkus Reviews

"Riveting stuff" -New York Post

Order from Amazon

Visit Big Switch site

Greatest hits

The amorality of Web 2.0

Twitter dot dash

The engine of serendipity

The editor and the crowd

Avatars consume as much electricity as Brazilians

The great unread

The love song of J. Alfred Prufrock's avatar

Flight of the wingless coffin fly

Sharecropping the long tail

The social graft

Steve's devices

MySpace's vacancy

The dingo stole my avatar

Excuse me while I blog

Other writing

Is Google Making Us Stupid?

The ignorance of crowds

The recorded life

The end of corporate computing

IT doesn't matter

The parasitic blogger

The sixth force



The limits of computers: Order from Amazon

Visit book site

Rough Type is:

Written and published by
Nicholas Carr

Designed by

JavaScript must be enabled to display this email address.