Yesterday I posted an excerpt from the start of Paul Goodman’s 1969 NYRB essay “Can Technology Be Humane?” Here’s another bit, equally relevant to our current situation, from later in the piece, when Goodman turns his attention to automation, robots, and what we today call “big data”:

In automating there is an analogous dilemma of how to cope with masses of people and get economies of scale, without losing the individual at great consequent human and economic cost. A question of immense importance for the immediate future is, Which functions should be automated or organized to use business machines, and which should not? This question also is not getting asked, and the present disposition is that the sky is the limit for extraction, refining, manufacturing, processing, packaging, transportation, clerical work, ticketing, transactions, information retrieval, recruitment, middle management, evaluation, diagnosis, instruction, and even research and invention. Whether the machines can do all these kinds of jobs and more is partly an empirical question, but it also partly depends on what is meant by doing a job. Very often, e.g., in college admissions, machines are acquired for putative economies (which do not eventuate); but the true reason is that an overgrown and overcentralized organization cannot be administered without them. The technology conceals the essential trouble, e.g., that there is no community of scholars and students are treated like things. The function is badly performed, and finally the system breaks down anyway. I doubt that enterprises in which interpersonal relations are important are suited to much programming.

But worse, what can happen is that the real function of the enterprise is subtly altered so that it is suitable for the mechanical system. (E.g., “information retrieval” is taken as an adequate replacement for critical scholarship.) Incommensurable factors, individual differences, the local context, the weighting of evidence are quietly overlooked though they may be of the essence. The system, with its subtly transformed purposes, seems to run very smoothly; it is productive, and it is more and more out of line with the nature of things and the real problems. Meantime it is geared in with other enterprises of society e.g., major public policy may depend on welfare or unemployment statistics which, as they are tabulated, are blind to the actual lives of poor families. In such a case, the particular system may not break down, the whole society may explode.

I need hardly point out that American society is peculiarly liable to the corruption of inauthenticity, busily producing phony products. It lives by public relations, abstract ideals, front politics, show-business communications, mandarin credentials. It is preeminently overtechnologized. And computer technologists especially suffer the euphoria of being in a new and rapidly expanding field. It is so astonishing that the robot can do the job at all or seem to do it, that it is easy to blink at the fact that he is doing it badly or isn’t really doing quite that job.

Goodman here makes a crucial point that still gets overlooked in discussions of automation. Computers and people work in different ways. When any task is shifted from a person to a computer, therefore, the task changes in order to be made suitable for the computer. As the process of automation continues, the context in which the task is performed also changes, in order to be made amenable to automation. The enterprise changes, the school changes, the hospital changes, the household changes, the economy changes, the society changes. The temptation, all along the way, is to look to the computer to provide the measures by which we evaluate those changes, which ends up concealing rather than revealing the true and full nature of the changes. Goodman expresses the danger succinctly: “The system, with its subtly transformed purposes, seems to run very smoothly; it is productive, and it is more and more out of line with the nature of things and the real problems.”

Image: cutetape.