« Long player: bonus track | Main | Invite bonanza: Pownce, Freebase, iMedix »
Skype and the hedge fund problem
August 21, 2007
Hedge funds work on a simple wisdom-of-the-crowd principle: Because they involve an extremely large number of transactions, which smooths out the vagaries of individual transactions, the movements of financial markets follow predictable patterns, which can be discerned from a study of their past behavior. Deviations from the patterns tend to be shortlived, and by making huge bets that the deviations will quickly return to the norm, you can make a whole lot of money. As we've seen recently, though, things aren't quite as simple as the hedge fund operators assume. Sometimes, very weird things happen and the deviations become either larger or longer-lived than expected, at which point the big bets can unravel in very unpleasant ways.
Peer-to-peer networks, which also involve lots and lots of different actors doing lots and lots of different things for lots and lots of different reasons, work in a similar way, and a company like Skype, whose telephone network is designed to run on many thousands of computers spread across a big P2P network, has built its system on the assumption that usage patterns are predictable - even if the actions of any individual user are not. Last week, Skype ran head-on into the hedge fund problem. The network's behavior deviated from the norm in a way that was greater than the Skype engineers had planned for, and the system crashed. The catalyst was the distribution of a routine patch from Microsoft, which led to a cascade of unanticipated effects, as Skype's Villu Arak explains:
The Microsoft Update patches were merely a catalyst — a trigger — for a series of events that led to the disruption of Skype, not the root cause of it ... The high number of post-update reboots affected Skype’s network resources. This caused a flood of log-in requests, which, combined with the lack of peer-to-peer network resources at the time, prompted a chain reaction that had a critical impact. The self-healing mechanisms of the P2P network upon which Skype’s software runs have worked well in the past ... Unfortunately, this time, for the first time, Skype was unable to rise to the challenge and the reasons for this were exceptional. In this instance, the day’s Skype traffic patterns, combined with the large number of reboots, revealed a previously unseen fault in the P2P network resource allocation algorithm Skype used. Consequently, the P2P network’s self-healing function didn’t work quickly enough. Skype’s peer-to-peer core was not properly tuned to cope with the load and core size changes that occurred on August 16.
As our economy becomes ever more tightly and intricately networked, its continued operation will hinge on the assumptions that mathematicians and software engineers embed in the code that underpins it. Usually, the assumptions will hold. But usually isn't always. Weird things happen, even in the largest of crowds.
Advertisement: Are you ready for "The Big Switch"? Fast Company calls Nicholas Carr's new book "compulsively readable - for nontechies, too." Salon says it's "magisterial." Order now from Amazon.com.
Comments
you could not have picked on two more extreme examples of the wealth distribution spectrum...hedge funds have managed to escape regulation because they say their investors are extremley wealthy and sohisticated and don't need protection...
Skype has democratized long distance, particularly international calling...200 million global users, and they just barely report profit...
Posted by: vinnie mirchandani ![[TypeKey Profile Page]](http://www.roughtype.com/nav-commenters.gif)
at August 21, 2007 05:19 PM
In this case though, the idiot hedge funds were all in the same trade. there was no math...it's called crowdsourcing I think :) .
Posted by: howardl at August 21, 2007 09:33 PM
Nick
I keep seeing Taleb Black Swans all over the place. If you've not read him, I strongly recommend it.
Posted by: Thomas Otter at August 22, 2007 11:34 AM
Post a comment
Thanks for signing in, . Now you can comment. (sign out)
(If you haven't left a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Thanks for waiting.)
The Atlantic article:
Is Google Making Us Stupid?"
Nick's new book:
"Future Shock for the web-apps era" -Fast Company
"Ominously prescient" -Kirkus Reviews
"Riveting stuff" -New York Post
Greatest hits
Avatars consume as much electricity as Brazilians
The love song of J. Alfred Prufrock's avatar
Other writing
The end of corporate computing
Nick's last book:
Order from Amazon
Visit book site