Reposted from the new edition of Edge:
What is the compelling urgency of the machine that it can so intrude itself into the very stuff out of which man builds his world?
– Joseph Weizenbaum
Somehow I managed to miss, until just a few days ago, the news that Joseph Weizenbaum had died. He died of cancer on March 5, in his native Germany, at the age of 85. Coincidentally, I was in Germany that same day, giving a talk at the CeBIT technology show, and — strange but true — one of the books I had taken along on the trip was Weizenbaum’s Computer Power and Human Reason.
Born in 1923, Weizenbaum left Germany with his family in 1936, to escape the Nazis, and came to America. After earning a degree in mathematics and working on programming some of the earliest mainframes, he spent most of his career as a professor of computer science at MIT. He became – to his chagrin – something of a celebrity in the 1960s when he wrote the Eliza software program, an early attempt at using a computer to simulate a person. Eliza was designed to mimic the conversational style of a psychotherapist, and many people who used the program found the conversations so realistic that they were convinced that Eliza had a capacity for empathy.
The reaction to Eliza startled Weizenbaum, and after much soul-searching he became, as John Markoff wrote in his New York Times obituary, a “heretic” in the computer-science world, raising uncomfortable questions about man’s growing dependence on computers. Computer Power and Human Reason, published in 1976, remains one of the best books ever written about computing and its human implications. It’s dated in some its details, but its messages seem as relevant, and as troubling, as ever. Weizenbaum argued, essentially, that computers impose a mechanistic point of view on their users – on us – and that that perspective can all too easily crowd out other, possibly more human, perspectives.
The influence of computers is hard to resist and even harder to escape, wrote Weizenbaum:
The computer becomes an indispensable component of any structure once it is so thoroughly integrated with the structure, so enmeshed in various vital substructures, that it can no longer be factored out without fatally impairing the whole structure. That is virtually a tautology. The utility of this tautology is that it can reawaken us to the possibility that some human actions, e.g., the introduction of computers into some complex human activities, may constitute an irreversible commitment. . . . The computer was not a prerequisite to the survival of modern society in the post-war period and beyond; its enthusiastic, uncritical embrace by the most “progressive” elements of American government, business, and industry quickly made it a resource essential to society’s survival in the form that the computer itself had been instrumental in shaping.
The machine’s influence shapes not only society’s structures but the more intimate structures of the self. Under the sway of the ubiquitous, “indispensable” computer, we begin to take on its characteristics, to see the world, and ourselves, in the computer’s (and its programmers’) terms. We become ever further removed from the “direct experience” of nature, from the signals sent by our senses, and ever more encased in the self-contained world delineated and mediated by technology. It is, cautioned Weizenbaum, a perilous transformation:
Science and technology are sustained by their translations into power and control. To the extent that computers and computation may be counted as part of science and technology, they feed at the same table. The extreme phenomenon of the compulsive programmer teaches us that computers have the power to sustain megalomaniac fantasies. But the power of the computer is merely an extreme version of a power that is inherent in all self-validating systems of thought. Perhaps we are beginning to understand that the abstract systems — the games computer people can generate in their infinite freedom from the constraints that delimit the dreams of workers in the real world — may fail catastrophically when their rules are applied in earnest. We must also learn that the same danger is inherent in other magical systems that are equally detached from authentic human experience, and particularly in those sciences that insist they can capture the whole man in their abstract skeletal frameworks.
His own invention, Eliza, revealed to Weizenbaum the ease with which we will embrace a fabricated world. He spent the rest of his life trying to warn us away from the seductions of Eliza and her many friends. The quest may have been quixotic, but there was something heroic about it too.
See other appreciations of Weizenbaum by Andrew Brown, Jaron Lanier, and Thomas Otter.
I remember Eliza, it was one of the first programs I had on my Atari that really startled me with it’s seeming complexity.
I never knew the backstory until now.
Thanks Nick.
I read Weizenbaum book when I was still a young C/Unix programmer, interested both in technology and psychology/spirituality. I consider him one of my mentors in my path toward becoming aware of how technology shapes our psyche. I wish to quote too from “Computer Power and Human Reason”: ”
The compulsive programmer is driven; there is little spontaneity in how be behaves; and he finds no pleasure in the fulfillment of his nominal wishes. He seeks reassurance from the computer, not pleasure. The closest parallel we can find to this sort of psychopathology is in the relentless, pleasureless drive for reassurance that characterizes the life of the compulsive gambler. […] Psychoanalysts, beginning with Freud, saw megalomania and fantasies of onnipotence as principal ingredients in the psychic life of the compulsive gambler. […] The compulsive gambler believes himself to be in control of a magical world to which only few men are given entrance. […] The gambler is the scientist of this magical world. He is the interpreter of the signs that Fate communicates to him, just as the scientist in the real world in an interpreter of the signs that nature communicates to everyone who cares to become sensitive to them. […] The test of the adequacy of both the scientist’s and the magician’s view of the world is its power to predict and, under suitably arranged conditions, to control. […] The test of power is control. The test of absolute power is certain and absolute control. When dealing with the compulsive programmer, we are therefore also dealing with his need to control and his need for certainty. The passion of certainty is, of course, also one of the great cornerstones of science, philosophy, and religion. And the quest for control in inherent in all technology.”
One of the first – if not the first – and the best analysis on the overpowering implication of technique (“the totality of methods rationally arrived at and having absolute efficiency (for a given stage of development) in every field of human activity”, which encompasses technology) and its pervasive rôle was that of Jacques Ellul, published in French (as La technique ou l’enjeu du siècle – Technology, the challenge of the century) in 1954. Thanks to Aldous Huxley, Ellul got recognition in the US (much more than in France, actually), and this book was translated in English in 1964 as The Technological Society.
I have no knowledge regarding the details Joseph Weizenbaum’s philosophical take on the relationship between technology and the human being. Having read the little excerpts here and in the earlier post, it’s very clear that he understands one of the deepest mysteries of all time.
The place and time where humankind and technology intersect has throughout the ages presented both the possibility of great progress and the seeds of radical change in the human condition.
After watching the link Thomas posted earlier, to a Davos panel discussion, it was very clear that the moderator, Loic Le Meur, was completely clueless. I would also suggest that he is probably not at all alone in that regard.
Technological progress quietly places an invisible curtain between the inevitable progress that it presents to our forward striving and that, which might be called our archetypal relationship to the natural world.
Where our focus on the development of consciousness needed making the curtain visible rather than looking at the symptoms thereof, our future might be a little rosier.
Alan
Thank you Alan for highlighting Alan’s link to the Davos talk.
“We are talking about promises, not facts as facts.” and “Solutions? I have no solutions. We have to learn to think critically!” [moderator smirking]
This is exactly right. And how appropriate to see it at Davos.
When I talk with money-influencing and money-wielding folks in this industry it is so often like this talk: When they sense sympathy, they pimp the promises of the latest hot investments in computing. When confronted with the risks, they throw up their arms, say “What else can we do? It grows on its own?” and then start thinking about how to decline the time of day.
Or there’s a twist on that that happens, sometimes: a serious concern discovered by thinking things through can be coopted and turned on its head if the “promise” of some next investment would “fix” it.
“There are certain dissonances here…”
-t