The soft tyranny of the rating system

In his darkly comic 2010 novel Super Sad True Love Story, Gary Shteyngart imagines a Yelpified America in which people are judged not by the content of their character but by their streamed credit scores and crowdsourced “hotness” points. Social relations of even the most intimate variety are governed by online rating systems.

A sanitized if more insidious version of Shteyngart’s big-data dystopia is taking shape in China today. At its core is the government’s “Social Credit System,” a centrally managed data-analysis program that, using facial-recognition software, mobile apps, and other digital tools, collects exhaustive information on people’s behavior and, running the data through an evaluative algorithm, assigns each person a “social trustworthiness” score. If you run a red light or fail to pick up your dog’s poop, your score goes down. If you shovel snow off a sidewalk or exhibit good posture in riding your bicycle, your score goes up. People with high scores get a variety of benefits, from better seats on trains to easier credit at banks. People with low scores suffer various financial and social penalties.

As Kai Strittmatter reports in a Süddeutsche Zeitung article, the Social Credit System is already operating in three dozen test cities in China, including Shanghai, and the government’s goal is to have everyone in the country enrolled by 2020:

Each company and person in China is to take part in it. Everyone will be continuously assessed at all times and accorded a rating. In [the test cities], each participant starts with 1000 points, and then their score either improves or worsens. You can be a triple-A citizen (“Role Model of Honesty,” with more than 1050 points), or a double-A (“Outstanding Honesty”). But if you’ve messed up often enough, you can drop down to a C, with fewer than 849 points (“Warning Level”), or even a D (“Dishonest”) with 599 points or less. In the latter case, your name is added to a black list, the general public is informed, and you become an “object of significant surveillance.”

As Strittmatter points out, the Chinese government has long monitored its citizenry. But while the internet-based Social Credit System may be nothing new from a policy standpoint, it allows a depth and immediacy of behavioral monitoring and correction that go far beyond anything that was possible before:

The Social Credit System’s heart and soul is the algorithm that gathers information without pause, and then processes, structures and evaluates it. The “Accelerate Punishment Software” section of the system guidelines describes the aim: “automatic verification, automatic interception, automatic supervision, and automatic punishment” of each breach of trust, in real time, everywhere. If all goes as planned, there will no longer be any loopholes anywhere.

The government officials that Strittmatter talked to were eager to discuss the program and to emphasize how it would encourage citizens to act more responsibly, leading to a happier, more harmonious society. As one planning document puts it, “the system will stamp out ‘lies and deception’ [and] increase ‘the nation’s honesty and quality.'” Those sound like worthy goals, and the rhetoric is not so different from that used in the U.S. and U.K. to promote governmental and commercial programs that employ online data collection and automated “nudge” systems to encourage good behavior and social harmony. I recall something Mark Zuckerberg wrote in his recent “Building Global Community” manifesto: “Looking ahead, one of our greatest opportunities to keep people safe is building artificial intelligence to understand more quickly and accurately what is happening across our community.” I’m not suggesting any equivalence. I am suggesting that when it comes to using automated behavioral monitoring and control systems for “beneficial” ends, the boundaries can get fuzzy fast. “Of all tyrannies,” wrote C. S. Lewis in God in the Dock, “a tyranny sincerely exercised for the good of its victims may be the most oppressive.”

What’s particularly worrisome about behavior-modification systems that employ publicly posted numerical ratings is that they encourage citizens to serve as their own tyrants. Using peer pressure, competition, and status-establishing prizes to shape behavior, the systems raise the specter of a “gamification” of tyranny. Nobody wants the stigma of a low score, particularly when it’s out there on the net for everyone to see. We’ll strive for Status Credits just as we strive for Likes or, to return to Shteyngart’s world, Hotness Points. “Our aim is to standardize people’s behavior,“ a Communist Party Secretary tells Strittmatter. “If everyone behaves according to standard, society is automatically stable and harmonious. This makes my work much easier.”

On Robert Pollard: “Chicken Blows”

[No. 04 in a Series]

Take one of those short Beatles songs from the medley that closes Abbey Road, turn it inside out, fill it with nitrous oxide, and let a kindergarten class use it as a ball during recess. That’s “Chicken Blows.” A seeming throwaway that arrives near the end of the nearly endless Alien Lanes, the song reveals itself as a miniature pop masterpiece only after many listens: the exquisitely frayed melody, the trembling vocal, the aching background harmonies, all washing across the tidal pull of a hazy, hypnotic guitar line. Everything feels exhausted, out of focus, dreamlike. “Chicken Blows” is the last song you hear before you fall asleep after a night that’s gone on much too long.

Like so many Guided by Voices songs, “Chicken Blows” has a warped backstory. It was originally released in 1994, a year before Alien Lanes came out, as a track on an exceedingly obscure compilation EP called The Polite Cream Tea Corps, which was included in an issue of Ptolemaic Terrascope, an occasional British psychedelic-music magazine. But the song seems to have been written and recorded much earlier than that, perhaps even in the 1980s. It was slated to be on the aborted 1991 Guided by Voices album Back to Saturn X, which Robert Pollard “shitcanned” just as it was going into production. It was then held in suspended animation for a few years, as three seminal GBV records appeared (1992’s Propeller, 1993’s Vampire on Titus, and 1994’s Bee Thousand), before Pollard decided the time was right to release it.

What’s remarkable about “Chicken Blows” is that it sounds much more contemporary today than it did when it came out more than twenty years ago. Sonically, it anticipates the entropic, Auto-Tune experiments of Bon Iver, Kanye West, and others. It’s fitting that Frank Ocean included the number on one of his Beats I playlists earlier this year. Sometimes seeds spend a lot of time underground before they sprout.

“Chicken blows”? The lyrics are funny, but as usual they’re hiding something sad.

I’m not here to drink all the beer
in the fridge,
in the room,
in the house,
in the place
that we both so love.

“The intellect of man is forced to choose,” wrote W. B. Yeats, “perfection of the life, or of the work.” In the singleminded pursuit of his art, Pollard has had to live something of a broken life, at least when it comes to playing the domestic roles of son, husband, and father — those tireless consumers of poultry meals — and it’s this tension that gives so much of his work its heartbreaking quality. “Chicken Blows” is, among other things, a confession and an apology.

Can you sink
to the depths?
I don’t know,
I don’t even care,
and our lives
slip away.
In the end
we will probably reach
all the way
to the walls
over there.
Have you flown?

The walls of the home are the bonds of love, and it’s the sound of them slowly collapsing that gives “Chicken Blows” its poignancy.

Image: Detail of “His Beautiful Women Crying” by Robert Pollard.

The master and the machine: on AI and chess

“A Brutal Intelligence: AI, Chess, and the Human Mind,” my review of Garry Kasparov’s new book Deep Thinking: Where Machine Intelligence Ends and Human Creativity Begins, appears today in the Los Angeles Review of Books. Here’s a bit:

The contingency of human intelligence, the way it shifts with health, mood, and circumstance, is at the center of Kasparov’s account of his historic duel with Deep Blue. Having beaten the machine in a celebrated match a year earlier, the champion enters the 1997 competition confident that he will again come out the victor. His confidence swells when he wins the first game decisively. But in the fateful second game, Deep Blue makes a series of strong moves, putting Kasparov on the defensive. Rattled, he makes a calamitous mental error. He resigns the game in frustration after the computer launches an aggressive and seemingly lethal attack on his queen. Only later does he realize that his position had not been hopeless; he could have forced the machine into a draw. The loss leaves Kasparov “confused and in agony,” unable to regain his emotional bearings. Though the next three games end in draws, Deep Blue crushes him in the sixth and final game to win the match.

One of Kasparov’s strengths as a champion had always been his ability to read the minds of his adversaries and hence anticipate their strategies. But with Deep Blue, there was no mind to read. The machine’s lack of personality, its implacable blankness, turned out to be one of its greatest advantages. It disoriented Kasparov, breeding doubts in his mind and eating away at his self-confidence. “I didn’t know my opponent at all,” he recalls. “This intense confusion left my mind to wander to darker places.” The irony is that the machine’s victory was as much a matter of psychology as of skill.

Read on.

Photo: Elyktra.

Should Uber’s next CEO be a robot?

A little more than two years ago, I suggested in a post that “the killer business app for artificial intelligence may turn out to be the algorithmic CEO.” I was picking up on a point that Frank Pasquale had made in a review of The Second Machine Age:

[Thiel Fellow and Ethereum developer Vitalik Buterin] has stated that automation of the top management functions at firms like Uber and AirBnB would be “trivially easy.” Automating the automators may sound like a fantasy, but it is a natural outgrowth of mantras (e.g., “maximize shareholder value”) that are commonplaces among the corporate elite.

Now that Uber CEO Travis Kalanick has resigned, completing a meltdown of the company’s top management ranks, Uber and its investors have a perfect opportunity to disrupt the executive suite, and indeed the entire history of management, by using software to run the company. Let’s face it: Kalanick’s great failing was that he was not quite robotic enough. His flaws were not analytical but human. He was a victim of his own meat.

A fundamentally numerical company, constituted mainly of software, Uber is the perfect test bed for the robot CEO. And since its staff includes exceptionally talented programmers, it already has the skill needed to gin up the algorithms necessary to do the work Kalanick and his lieutenants did (without the attendant buffoonery).* A two-day hackathon should be more than sufficient to create a robot able to translate spreadsheet data into resource-allocation plans and use machine learning to compose forward-leaning messages that inspire staffers, drivers, and venture capitalists. And to have Uber’s robot CEO sit next to Cook, Nadella, Bezos, et al., at the next White House photo-op would be a huge PR coup.

Not only is Uber the right company for a robot CEO, but now is the right time for one. Just two months ago, Alibaba CEO Jack Ma predicted that “in thirty years, a robot will likely be on the cover of Time Magazine as the best CEO.”** As the financier Martin Hutchinson pointed out, there’s no reason to wait that long. “Human CEOs have amassed an especially dire track record in the last two decades,” he wrote. “Whereas their compensation has soared far faster than overall U.S. output, productivity growth in U.S. businesses has notably lagged, indicating their failure to invest optimally.” If there were ever a job to be automated, it’s that of the underperforming, overpaid modern CEO.

Even at this year’s World Economic Forum in Davos, the case for a robot CEO was laid out in compelling terms:

There are some distinct advantages to having a robot as your company’s CEO. Firstly, they might be able to make better, more responsible, decisions. … Robots don’t face the unpredictability we humans face, so their decisions are more likely to be consistent, based on facts. … Robots can work all day, every day. They don’t need sleep, weekends or holidays. No mere humans can say the same, however hard they may try to cultivate that impression. … And if you’ve created one CEO robot, why not create a few more? It’s not as if he or she has a unique personality. Technology allows them to interact wherever your customers are, further cutting down travel costs and helping the environment.

We may look back on Kalanick’s resignation as the most transformative act of his eventful career. He has opened the door for a robot CEO. The question now is whether the Uber board will welcome the future or resist it.


*On further thought, Uber’s coders probably have better things to do than write simple CEO algorithms. What’s really needed are cloud-based virtual CEOs. Yes: CEO-as-a-Service. Are you listening, Marc Benioff?

**Ma’s assumption that Time will still be around, with its cover intact, thirty years from now makes me question his futurist cred. But I’m going to assume he was speaking figuratively.

The robot paradox

“You can see the computer age everywhere but in the productivity statistics,” remarked MIT economist Robert Solow in a 1987 book review. The quip became famous. It crystallized what had come to be called the productivity paradox — the mysterious softness in industrial productivity despite years of big corporate investments in putatively labor-saving information technology.

I think the time has come to start talking about the robot paradox. So let me offer a new twist on Solow’s words:

You can see the robot age everywhere but in the labor statistics.

In an echo of the hype surrounding IT in the 1970s and 1980s, we’ve heard over the last decade a stream of predictions about how robots, algorithms, and other automation technologies are about to unleash an unemployment crisis. Not only will most factory jobs be handed over to automatons, but the ranks of white-collar workers will be decimated by artificial intelligence programs powered by Big Data. The end of work is nigh.

In the wake of the Great Recession, when hiring stayed stagnant for years, such predictions seemed reasonable. But recent economic statistics flat-out belie the claims. As Grep Ip, the Wall Street Journal economics columnist, wrote last week, predictions of an impending job apocalypse “would be more plausible if the evidence weren’t moving in exactly the opposite direction.” Business employment has been going up for 86 straight months, pushing the U.S. unemployment rate down to just 4.4 percent, a level many economists see as representing full employment. It’s true that a lot of workers have dropped out of the labor force, but the sustained, robust job growth makes it awfully hard to argue that advances in computer automation, which have been accelerating for a long time, are poised to create an unemployment explosion.

Even more telling is the persistently weak growth in productivity. As Ip explained: “If automation were rapidly displacing workers, the productivity of the remaining workers ought to be growing rapidly. Instead, growth in productivity — worker output per hour — has been dismal in almost every sector, including manufacturing.” You can argue that our methods of measuring productivity are imperfect, but if computers were going to obliterate workers, you should by now be seeing a strong upswing in productivity. And it’s just not there.

I’m convinced that computer automation is changing the way people work, often in profound ways, and I think it’s likely that automation is playing an important role in restraining wage growth by, among other things, deskilling certain occupations, shifting employees to more contingent positions, and reducing the bargaining power of workers. But the argument that computers are going to bring extreme unemployment in coming decades — an argument that was also popular in the 1950s, the 1960s, and again in the 1990s, it’s worth remembering — sounds increasingly dubious. It runs counter to the facts. Anyone making the argument today needs to provide a lucid and rational explanation of why, despite years of rapid advances in robotics, computer power, network connectivity, and artificial intelligence techniques, we have yet to see any sign of a broad loss of jobs in the economy.

Image: DARPA.

The digital-industrial complex

Exactly fifty years after the hippies gathered in San Francisco, another summer of love seems set to blossom. This time it’s not the flower children who are holding hands and sharing beds. It’s the titans of Big Internet.

Just this week, at its Build conference, Microsoft gave a hug to former adversaries Apple and Alphabet. “Windows PCs heart iOS and Android devices” was one of the big themes of the event — yes, the heart symbol was on display — and Microsoft announced that Apple’s iTunes app is coming to the Windows Store. Microsoft also formed a partnership with Facebook to incorporate an ad-tracking tool into Excel. Meanwhile, Apple and Amazon were engaged in their own public display of affection. They let word leak out that Amazon’s Prime Video app would soon be available on Apple TV. The once fierce rivals appear to have “reached a truce,” reported Recode.

Thanks to their technical and marketing prowess, combined with the winner-take-all dynamics of the internet, Alphabet, Amazon, Apple, Facebook, and Microsoft have emerged as the dominant companies of the consumer net (Farhad Manjoo dubs them the “frightful five”), with a combined market cap of a zillion dollars, give or take. Each now operates something of a perpetual-motion money-printing machine powered by the dollars and data that flow in such massive quantities through the net. The companies still face threats, of course, but, even as they sow disruption in other industries, their own market positions now look pretty stable and secure. They’re the winners.

While the boundaryless nature of online business means that each of the five companies competes with each of the others on many fronts, there is also now a symbiosis among them — and that symbiosis is getting stronger. Each of the five makes its profits in different ways, with Apple focusing on hardware, Google on web ads, Facebook on social-media ads, Amazon on retailing, and Microsoft on software sales and subscriptions. Their businesses overlap, but they are also complementary. And, as is often true with complementary products and services, gains by one company often help rather than hurt the businesses of the others. Each of the five is focused on expanding consumers’ dependency on the net, and as the net pie expands so does each of the five slices. At this point, being friends rather than enemies makes sense.

When it comes to business, in other words, the net is a centralizing force, not a decentralizing one as once assumed. The frightful five together form a digital-industrial complex, a nascent oligopoly set to skim the lion’s share of the profits from the consumer web for the foreseeable future. Five big pieces, loosely joined.

On Monday, the venture capitalist Jeremy Philips wrote a column intended as a rejoinder to Manjoo’s warnings about the power of the titans. Philips argued against the idea that, as he put it, “the five leading tech behemoths have turned into dangerous monopolies that stifle innovation and harm consumers.” Their businesses, he wrote, are “all converging — therefore competing — with one another.” His timing was unfortunate, as immediately after the column appeared we got the news of the new partnerships among the companies.

Philips’s argument would have sounded compelling just a few years ago. Back then, the five’s positions were not as well-established as they are now, and their relationships were defined by their skirmishes. That’s no longer the case. Yes, the businesses of the five have converged, but it’s now becoming clear that their interests have converged as well. For Big Internet, this is the dawning of the Age of Aquarius.

Image: Actors portraying hippies in “Hair.”