The new wave of computer automation has provoked much concern and debate about job losses and the future of employment. Less discussed has been the way the computer is shaping the way people work and act, both on the job and in their personal lives. As the computer becomes a universal tool for getting things done, what happens to the diverse talents that people used to develop by engaging directly with the world in all its intricacy and complexity? In “The Great Forgetting,” an essay in the new issue of The Atlantic (the online version of the article bears the title “All Can Be Lost”), I look at some of the unexpected consequences of computer automation, particularly the way that software, as currently designed, tends to steal from us the opportunity to develop rich, distinctive, and hard-earned skills. Psychologists, human-factors experts, and other researchers are discovering that the price we pay for the ease and convenience of automation is a narrowing of human possibility.
Here’s an excerpt:
Psychologists have found that when we work with computers, we often fall victim to two cognitive ailments — complacency and bias — that can undercut our performance and lead to mistakes. Automation complacency occurs when a computer lulls us into a false sense of security. Confident that the machine will work flawlessly and handle any problem that crops up, we allow our attention to drift. We become disengaged from our work, and our awareness of what’s going on around us fades. Automation bias occurs when we place too much faith in the accuracy of the information coming through our monitors. Our trust in the software becomes so strong that we ignore or discount other information sources, including our own eyes and ears. When a computer provides incorrect or insufficient data, we remain oblivious to the error.
Examples of complacency and bias have been well documented in high-risk situations — on flight decks and battlefields, in factory control rooms — but recent studies suggest that the problems can bedevil anyone working with a computer. Many radiologists today use analytical software to highlight suspicious areas on mammograms. Usually, the highlights aid in the discovery of disease. But they can also have the opposite effect. Biased by the software’s suggestions, radiologists may give cursory attention to the areas of an image that haven’t been highlighted, sometimes overlooking an early-stage tumor. Most of us have experienced complacency when at a computer. In using e-mail or word-processing software, we become less proficient proofreaders when we know that a spell-checker is at work.
The way computers can weaken awareness and attentiveness points to a deeper problem. Automation turns us from actors into observers. That shift may make our lives easier, but it can also inhibit the development of expertise. Since the late 1970s, psychologists have been documenting a phenomenon called the “generation effect.” It was first observed in studies of vocabulary, which revealed that people remember words much better when they actively call them to mind — when they generate them — than when they simply read them. The effect, it has since become clear, influences learning in many different circumstances. When you engage actively in a task, you set off intricate mental processes that allow you to retain more knowledge. You learn more and remember more. When you repeat the same task over a long period, your brain constructs specialized neural circuits dedicated to the activity. It assembles a rich store of information and organizes that knowledge in a way that allows you to tap into it instantaneously.
Whether it’s Serena Williams on a tennis court or Magnus Carlsen at a chessboard, an expert can spot patterns, evaluate signals, and react to changing circumstances with speed and precision that can seem uncanny. What looks like instinct is hard-won skill, skill that requires exactly the kind of struggle that modern software seeks to alleviate.
This is one of the themes that I’ll be exploring in my next book, The Glass Cage: Automation and Us.